Summary:<\/em><\/strong>\u00a0 In the literal blink of an eye, image-based AI has gone from high cost, high risk projects to quick and reasonably reliable.\u00a0 C-level execs looking for AI techniques to exploit need to revisit their assumptions and move these up the list.\u00a0 Here\u2019s what\u2019s changed.<\/em><\/p>\n \u00a0<\/p>\n For data scientists these are miraculous times.\u00a0 We tend to think of miracles as something that occurs instantaneously but in our world that\u2019s not quite so.\u00a0 Still the rate of change in deep learning, particularly in image recognition is mind boggling and way up there on the miraculous scale.<\/p>\n Buried deep in the fascinating charts and graphs of the 2018 AI Index Annual Report<\/u><\/em><\/a> is this chart.<\/p>\n \u00a0 In a year and a half, the time required to train a CNN network in the ImageNet competition has fallen from about one hour to less than 4 minutes.\u00a0 That\u2019s a 16X improvement in 18 months.\u00a0 (Keep in mind that the ImageNet dataset is comprised of 14 million labeled images in 20,000 categories.)<\/em><\/p>\n That\u2019s remarkable in itself, but compare this to our experience with training CNNs in December 2015, 36 months back when automated image detection in the ImageNet competition first exceeded human performance.\u00a0 Roughly another 18 months before this chart begins Microsoft researchers used an extraordinary 152 layer CNN which was five times the size of any previous system to beat the human benchmark for the first time (97% for the CNN, 95% for humans)<\/em>.<\/p>\n That effort took Microsoft many months of trial and error as they pioneered the techniques that led to better-than-human accuracy in image recognition.\u00a0 And while we can\u2019t fairly compare all that experimental time to today\u2019s benchmark of less than 4 minutes, it\u2019s accurate to say that in 2016 training a CNN was a task acknowledged to take weeks if not months, and consume compute resources valued in the tens-of-thousands of dollars for a single model, if it managed to train at all.<\/p>\n So 16X doesn\u2019t begin to express how far we\u2019ve come.\u00a0 For the sake of picking a starting point let\u2019s try about 40 hours of compute time then, now reduced to 4 minutes or about a 600X improvement.<\/p>\n It\u2019s worth looking at what factors brought us here.\u00a0 There are three.<\/p>\n \u00a0<\/p>\n Tuning the CNN<\/strong><\/span><\/p>\n Two years ago, setting up a CNN was still largely trial and error.\u00a0 Enough simply wasn\u2019t known about picking a starting point of layers, loss functions, node interconnections, and starting weights.\u00a0 Much less how varying any one of these factors would impact the others once launched.<\/p>\n Our understanding has advanced so far that Microsoft, Google, and several startups offer fully automated deep learning platforms<\/u><\/em><\/a> that are all but fool proof.<\/p>\n In much the same way that automated selection and hyperparameter tuning of machine learning algorithms has deemphasized the importance of deep technical knowledge of the algorithms themselves, the same has happened for deep learning.\u00a0 For some time now (OK, that means at least a year<\/em>) we\u2019ve been more concerned about adequate labeled training data than about the mechanics of the CNNs themselves.<\/p>\n \u00a0<\/p>\n Chips Are Increasingly Where It\u2019s At<\/strong><\/span><\/p>\n Our readers are most likely aware of the many software advancements that have made the techniques around implementing CNNs easier and faster.\u00a0 Fewer may be aware that an equal or greater share of the credit for this acceleration belongs to the chipmakers.\u00a0 Increasingly all types of AI mean customized silicon optimized for the many different needs of deep learning.<\/p>\n In fact, the dedicated chip track has been evolving as long as CNNs have been the algorithm of choice for image recognition given the much longer development time and much greater capital required for such an effort.<\/p>\n While the first great leap in speed and efficiency came from utilizing GPUs and FPGAs, by 2016 Google had announced its own customized chip, the TPU (Tensor Processing Unit) optimized for its TensorFlow platform.\u00a0 TensorFlow has captured by far the largest share of AI developers and the proprietary TPU chip and platform have given Google ownership of the largest share of AI cloud revenue.<\/p>\n Essentially all the other majors, both existing chipmakers and AI platform developers are following suit including Intel, NVidia, Microsoft, Amazon, IBM, Qualcomm, and a large cohort of startups.<\/p>\n This would seem to be too many competitors for this specialized area but image recognition, and more broadly deep learning-based AI applications have many ways in which to specialize.<\/p>\n Where GPUs and FPGAs are programmable, the push is specifically to AI-embedded silicon with dedicated niche applications.\u00a0 All these have contributed to the increase in speed and reliability of results in CNN image recognition applications.<\/p>\n \u00a0<\/p>\n Transfer Learning<\/strong><\/span><\/p>\n The final contributor is Transfer Learning<\/u><\/em><\/a>, though in this specific case it was not used in the ImageNet competition.<\/p>\n The central concept is to use a more complex but successful pre-trained CNN model to \u2018transfer\u2019 its learning to your more simplified (or equally but not more complex) problem.<\/p>\n The existing successful pre-trained model has two important attributes:<\/p>\n In simplified TL the pre-trained transfer model is simply chopped off at the last one or two layers. \u00a0Once again, the early, shallow layers are those that have identified and vectorized the features and typically only the last one or two layers need to be replaced.<\/p>\n The output of the truncated \u2018featurizer\u2019 front end is then fed to a standard classifier like an SVM or logistic regression to train against your specific images.<\/p>\n The resulting transfer CNN can be trained with as few as 100 labeled images per class, but as always, more is better.\u00a0 This addresses the problem of the availability and cost of creating sufficient labeled training data and also greatly reduces the compute time and accelerates the overall project.<\/p>\n Large libraries of pretrained CNNs that can be used to \u2018donate\u2019 their front ends are available as open source from the major cloud providers hoping to capture the cloud compute revenue, and also from academic sources.<\/p>\n \u00a0<\/p>\n A Note about the 2018 AI Index Annual Report<\/strong><\/span><\/p>\n This annual report is the work of Stanford\u2019s Human-Centered AI Institute who for the last few years has been publishing the encyclopedic \u2018AI Index\u2019.\u00a0 To see how the key chart above was developed and the sources that were used see the report<\/u><\/em><\/a> and its equally informative appendices.<\/p>\n \u00a0<\/p>\n Your Strategic Business Assumptions May Need to Change<\/strong><\/span><\/p>\n If you\u2019re a C-level executive looking at reasonable applications of AI to leverage in your newly digital enterprise you need to be careful who you talk to.\u00a0 Not every data scientist will be aware how rapidly image-based AI is becoming better, faster, and cheaper.<\/p>\n 24 months ago I was still advising that image-based AI was a bleeding edge technique and a project with high costs and a high risk of failure.\u00a0 In the blink of an eye that\u2019s changed.\u00a0<\/p>\n \u00a0<\/p>\n \u00a0<\/p>\n Other articles by Bill Vorhies<\/u><\/em><\/a><\/p>\n \u00a0<\/p>\n About the author:\u00a0 Bill is Contributing Editor for Data Science Central.\u00a0 Bill is also President & Chief Data Scientist at Data-Magnum and has practiced as a data scientist since 2001.\u00a0\u00a0\u00a0 He can be reached at:<\/p>\n Bill@DataScienceCentral.com<\/a> or<\/span> Bill@Data-Magnum.com<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<\/a><\/p>\n<\/p>\n
<\/a>Thanks to literally thousands of data scientists piling into deep learning at both the research and application level a much better understanding of best hyperparameter tuning has been put into practice.<\/p>\n
\n
\n