Author: <\/p>\n
Twenty years ago, Google started using machine learning, and 10 years ago, it helped spur rapid progress in AI using deep learning. Jeff Dean and Marian Croak of Google Research take a look at how we\u2019ve innovated on these techniques and applied them in helpful ways, and look ahead to a responsible and inclusive path forward.<\/i><\/p>\n<\/p>\n<\/div>\n<\/div>\n From research demos to AI that really works<\/b><\/p>\n<\/div>\n<\/div>\n I was first introduced to neural networks \u2014 computer systems that roughly imitate how biological brains accomplish tasks \u2014 as an undergrad in 1990. I did my senior thesis on using parallel computation to train neural networks. In those early days, I thought if we could 32X more compute power (using 32 processors at the time!), we could get neural networks to do impressive things. I was way off. It turns out we would need about 1 million times as much<\/i> computational power before neural networks could scale to real-world problems.<\/p>\n<\/div>\n<\/div>\n A decade later, as an early employee at Google, I became reacquainted with machine learning when the company was still just a startup. In 2001 we used a simpler version of machine learning, statistical ML, to detect spam and suggest better spellings for people\u2019s web searches. But it would be another decade before we had enough computing power to revive a more computationally-intensive machine learning approach called deep learning. Deep learning uses neural networks with multiple layers (thus the \u201cdeep\u201d), so it can learn not just simple statistical patterns, but can learn subtler patterns of patterns \u2014 such as what\u2019s in an image or what word was spoken in some audio. One of our first publications<\/a> in 2012 was on a system that could find patterns among millions of frames from YouTube videos. That meant, of course, that it learned to recognize cats.<\/p>\n<\/div>\n<\/div>\n To get to the helpful features you use every day \u2014 searchable photo albums, suggestions on email replies, language translation, flood alerts, and so on \u2014 we needed to make years of breakthroughs on top of breakthroughs, tapping into the best of Google Research in collaboration with the broader research community. Let me give you just a couple examples of how we\u2019ve done this.<\/p>\n<\/div>\n<\/div>\n A big moment for image recognition<\/b><\/p>\n<\/div>\n<\/div>\n In 2012, a paper<\/a> wowed the research world for making a huge jump in accuracy on image recognition using deep neural networks, leading to a series of rapid advances by researchers outside and within Google. Further advances led to applications like Google Photos<\/a> in 2015, letting you search photos by what\u2019s in them. We then developed other deep learning models to help you find addresses in Google Maps<\/a>, make sense of videos on YouTube<\/a>, and explore the world around you using Google Lens<\/a>. Beyond our products, we applied these approaches to health-related problems, such as detecting diabetic retinopathy<\/a> in 2016, and then cancerous cells<\/a> in 2017, and breast cancer<\/a> in 2020. Better understanding of aerial imagery through deep learning let us launch flood forecasting<\/a> in 2018, now expanded<\/a> to cover more than 360 million people in 2021. It\u2019s been encouraging to see how helpful these advances in image recognition have been.<\/p>\n<\/div>\n<\/div>\n Similarly, we\u2019ve used deep learning to accelerate language understanding. With sequence-to-sequence learning<\/a> in 2014, we began looking at how to understand strings of text using deep learning. This led to neural machine translation<\/a> in Google Translate in 2016, which was a massive leap in quality, particularly for less prevalent languages. We developed neural language models further for Smart Reply in Gmail<\/a> in 2017, which made it easier and faster for you to knock through your email, especially on mobile. That same year, Google invented Transformers<\/a>, leading to BERT<\/a> in 2018, then T5<\/a>, and in 2021 MUM<\/a>, which lets you ask Google much more nuanced questions. And with \u201csparse\u201d models like GShard<\/a>, we can dramatically improve on tasks like translation while using less energy<\/a>.<\/p>\n<\/div>\n<\/div>\n We\u2019ve driven a similar arc in understanding speech. In 2012, Google used deep neural networks to make major improvements to speech recognition on Android<\/a>. We kept advancing the state of the art<\/a> with higher-quality, faster, more efficient speech recognition systems. By 2019, we were able to put the entire neural network on-device<\/a> so you could get accurate speech recognition even without a connection. And in 2021, we launched Live Translate on the Pixel 6<\/a> phone, letting you speak and be translated in 48 languages — all on-device, while you\u2019re traveling with no Internet.<\/p>\n<\/div>\n<\/div>\nJeff Dean<\/h3>\n<\/div>\n<\/div>\n
\n