![\"A](\"https:\/\/storage.googleapis.com\/gweb-uniblog-publish-prod\/images\/image4_PgpO6NB.max-1000x1000.png\")
The book that led to my visit to Google.<\/p>\n<\/div>\n<\/figcaption><\/figure>\n
When I first visited Google back in 2002, I was a computer science professor at UC Berkeley. My colleague John Hennessey and I were updating our textbook on computer architecture, and Larry Page \u2014 who rode a hot-rodded electric scooter at the time \u2014 agreed to show me how his then three-year-old company designed its computing for Search. I remember the setup was lean yet powerful: just 6,000 low-cost PC servers and 12,000 PC disks answering 70 million queries around the world, every day. It was my first real look at how Google built its computer systems from the ground up, optimizing for efficiency at every level.<\/p>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n<\/div>\n
When I joined the company in 2016, it was with the goal of helping research how to maximize the efficiency of computer systems built specifically for artificial intelligence. Last year, Google set an ambitious goal of operating on 24\/7 carbon-free energy, everywhere, by the end of the decade. But at the same time, machine learning systems are quickly becoming larger and more capable. What will be the environmental impact of those systems \u2014 and how can we neutralize that impact going forward?\u00a0<\/p>\n
Today, we\u2019re publishing a detailed analysis<\/a>\u00a0that addresses both of those questions. It\u2019s an account of the energy- and carbon-costs of training six state-of-the art ML models, including five of our own. (Training a model is like building infrastructure: You spend the energy to train the model once, after which it\u2019s used and reused many times, possibly by hundreds of millions of people.) To our knowledge, it\u2019s the most thorough evaluation of its kind yet published. And while we had reason to believe our systems were efficient, we were encouraged by just how efficient they turned out to be.<\/p>\n For instance, we found that developing the Evolved Transformer model, a more efficient version of the popular Transformer architecture for ML, emitted nearly 100 times less carbon dioxide equivalent<\/a> than a widely cited estimate. Of the roughly 12.7 terawatt-hours of electricity that Google uses every year<\/a>, less than 1\/200th of a percent of it was spent training our most computationally demanding models.\u00a0\u00a0<\/p>\n What\u2019s more, our analysis found that there already exist many ways to develop and train ML systems even more efficiently: Specially designed models, processors and data centers can dramatically reduce energy requirements, while the right selection of energy sources can go a long way to reduce the carbon that\u2019s emitted during training. In fact, the right combination of model, processor, data center and energy source can reduce the carbon footprint of training an ML system by 1000 times.\u00a0<\/p>\n There\u2019s no one easy trick for achieving a reduction that large, so let\u2019s unpack that figure.\u00a0 Minimizing a system\u2019s carbon footprint is a two-part problem: First you want to minimize the energy the system consumes, then you have to supply that energy from the cleanest source possible.<\/p>\n Our analysis took a closer look at GShard and Switch Transformer, two models recently developed at Google Research. They\u2019re the largest models we\u2019ve ever created, but they both use a technique called sparse activation that enables them to only use a small fraction of their total architecture for a given task. It\u2019s a bit like how your brain uses a small fraction of its 100 billion neurons to help you read this sentence. The result is that these sparse models consume less than one tenth the energy that you\u2019d expect of similarly sized dense models \u2014 without sacrificing accuracy.<\/p>\n But to minimize ML\u2019s energy use, you need more than just efficient models \u2014 you also need efficient processors and data centers to train and serve them. Google\u2019s Tensor Processing Units (TPUs) are specifically designed for machine learning, which makes them up to five times more efficient than off-the-shelf processors. And the cloud computing data centers that house those TPUs are up to twice as efficient as typical enterprise data centers.\u00a0<\/p>\n Once you\u2019ve minimized your energy requirements, you have to think about where that energy originates. The electricity a data center consumes is determined by the grid where it\u2019s located. And depending on what resources were used to generate the electricity on that grid, this may emit carbon.\u00a0<\/p>\n The carbon intensity of grids varies greatly across regions, so it really matters where models are trained. For instance, the mix of energy supplying Google\u2019s Iowa data center produces 0.080kg of CO2e per kilowatt hour of electricity, when combining the electricity supplied by the grid and produced by Google\u2019s wind farms in Iowa. That\u2019s 5.4 times less than the U.S. average.\u00a0<\/p>\n Any one of these four factors \u2014 models, chips, data centers and energy sources \u2014 can have a sizable effect on the costs associated with developing an ML system. But their cumulative impact can be enormous.<\/p>\n When John and I updated our textbook with what we\u2019d learned on our visit to Google back in 2002, we wrote that \u201creducing the power per PC [server]\u201d presented \u201ca major opportunity for the future.\u201d Nearly 20 years later, Google has found many opportunities to streamline its systems \u2014\u00a0but plenty remain to be seized. As a result of our analysis, we\u2019ve already begun shifting where we train our computationally intensive ML models. We\u2019re optimizing data center efficiency by shifting compute tasks<\/a> to times when low-carbon power sources are most plentiful. Our Oklahoma data center, in addition to receiving its energy from cleaner sources, will house many of our next generation of TPUs, which are even more efficient than their predecessors. And sparse activation is just one example of the algorithmic ingenuity Google is using to design ML models that work smarter, not harder.<\/p>\n<\/div>\n<\/div>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"