Summary:<\/em><\/strong> \u00a0There have been several stories over the last several months around the theme that AI is about to hit a wall.\u00a0 That the rapid improvements we\u2019ve experienced and the benefits we\u2019ve accrued can\u2019t continue at the current pace.\u00a0 It\u2019s worth taking a look at these arguments to see if we should be adjusting our plans and expectations.<\/em><\/p>\n \u00a0<\/p>\n These concerns center around two issues:<\/p>\n Over Concentrated Investment<\/strong><\/span><\/p>\n It\u2019s certainly true that investment in AI is extremely concentrated in just a few cities.\u00a0 This chart from the National Venture Capital Association shows that over 80% of investment went to just six metros.\u00a0 By the way, there\u2019s a sharp break in the distribution below this level with the next best metro receiving only 2.4% of total investment (Chicago).<\/p>\n \u00a0 So what\u2019s the concern?\u00a0 The folks who have written about this see this as an over concentration of talent that increases labor costs in these areas making some types on new AI investment uneconomic.<\/p>\n It seems true that the concentration of AI talent in say San Francisco leads to the most new ideas in AI being generated there.\u00a0 But is this really a black hole with gravity so extreme that good ideas can\u2019t escape to be executed elsewhere?<\/p>\n Worriers about this concentration see these good ideas escaping to overseas low cost locations.\u00a0 I\u2019m not so sure.\u00a0<\/p>\n There is a pattern that the largest concentration of VC funded AI startups and established AI companies will draw in the most volume of new talent at the bottom.\u00a0 New graduates and young people will naturally gravitate to metros where there are the most opportunities.<\/p>\n Personally I think this concern overlooks that as these metros become more dense with opportunity they also become massively more expensive to live in and that the quality of life drops off quickly (San Francisco, I\u2019m looking at you).<\/p>\n As young professionals gain experience and also reach the life stage where they want to marry, buy homes, and start families they will naturally migrate to less expensive geographies.\u00a0 This may mean they will gravitate to the outskirts of these metros with lower cost of living where AI businesses will naturally follow to capture talent.\u00a0 It may also mean a willingness to relocate over longer distances to good quality of life locations like Austin, Seattle, Washington, Atlanta, or Miami where investment will follow talent.<\/p>\n<\/p>\n Growth in Training Compute for the Most Advanced Applications is Now Growing 7X Faster than Before<\/strong><\/span><\/p>\n This area of concern was first raised by the AI research lab OpenAI in 2018 and recently updated with new data.\u00a0 Their findings are that the amount of computational power needed to train the largest AI models has doubled every 3.4 months since 2012.<\/p>\n They see history divided into two distinct eras with the new era beginning in 2012 now growing 7X faster than the previous era.<\/p>\n \u00a0 This phenomenon has led some leading industry experts including Jerome Pesenti, the VP of Artificial Intelligence at Facebook to observe in a recent Wired article<\/u><\/em><\/a> that there may be cost\/benefit limits to what can be accomplished at the extreme limits of improving AI models.<\/p>\n Pesenti says that while scaling up is a worthwhile solution to the most difficult challenges like deep fake image detection and AI moderation of fake news and online abuse that the rate of progress is not sustainable.<\/p>\n \u201cIf you look at top experiments, each year the cost is going up 10-fold. Right now, an experiment might be in seven figures, but it\u2019s not going to go to nine or ten figures, it\u2019s not possible, nobody can afford that.\u00a0 It means that at some point we’re going to hit the wall. In many ways we already have.\u201d<\/em><\/p>\n Other advancements that are at risk would be many of the \u2018next big things\u2019 in AI<\/u><\/em><\/a> including cutting edge applications that require increased look forwards or look backs to properly classify.\u00a0 These would include many navigation apps in next gen computer vision, and contextually intelligent NLP assistants<\/u><\/em><\/a> that can relate our schedules and actions to outside events.<\/p>\n While I wouldn\u2019t want to have to do without these improvements, it\u2019s interesting what\u2019s been included in the OpenAI chart.\u00a0 For example, the most compute-hungry model is actually a reinforcement learning game playing app AlphaGoZero.\u00a0 Personally I\u2019d like to see research continue in Deep Learning RL models but the commercial applications are not as compelling as say BERT, the new NLP language encoder from Google.<\/p>\n Another issue to keep in mind is that this is an issue that impacts incremental gains in existing capabilities.\u00a0 Pesenti is right to wonder aloud if it isn\u2019t time to apply cost\/benefit criteria to these.\u00a0 This so-called wall does not prevent us in any way from continuing the rapid application of existing AI\/ML capabilities in business and harvesting the benefits currently underway.<\/p>\n In a sense this is the flying-car syndrome.\u00a0 We\u2019ve come so far so fast that if we don\u2019t get deep fake detection or fake news automatic moderation right away we\u2019ll wonder where\u2019s our flying car.<\/p>\n It\u2019s also worth noting that lots of research is being conducted in how to make the training process more efficient.\u00a0 For example, just last week researchers at Rice University announced the results of their new MACH technique (Merged Average Classifiers via Hashing).\u00a0 They report<\/u><\/em><\/a> \u201ctraining times about 7-10 times faster, and \u2026memory footprints are 2-4 times smaller\u2019 than previous large-scale deep learning techniques.<\/p>\n As for these walls, fortunately smart folks are working on both ends of this problem.\u00a0 We may need to bring some cost\/benefit discipline to our research but if the market need is there, my bet is that we\u2019ll find a way.<\/p>\n \u00a0<\/p>\n Other articles by Bill Vorhies<\/u><\/em><\/a><\/p>\n \u00a0<\/p>\n About the author:\u00a0 Bill is Contributing Editor for Data Science Central.\u00a0 Bill is also President & Chief Data Scientist at Data-Magnum and has practiced as a data scientist since 2001.\u00a0 His articles have been read more than 2 million times.<\/p>\n He can be reached at:<\/p>\n Bill@DataScienceCentral.com<\/a> or<\/span> Bill@Data-Magnum.com<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<\/a>There have been several stories over the last several months around the theme that AI is about to hit a wall.\u00a0 That the rapid improvements we\u2019ve experienced and the benefits we\u2019ve accrued can\u2019t continue at the current pace.\u00a0 It\u2019s worth taking a look at these arguments to see if we should be adjusting our plans and expectations.<\/p>\n
\n
<\/a><\/p>\n
<\/a><\/p>\n