{"id":1020,"date":"2018-09-05T06:30:17","date_gmt":"2018-09-05T06:30:17","guid":{"rendered":"https:\/\/www.aiproblog.com\/index.php\/2018\/09\/05\/democratizing-deep-learning-the-stanford-dawn-project\/"},"modified":"2018-09-05T06:30:17","modified_gmt":"2018-09-05T06:30:17","slug":"democratizing-deep-learning-the-stanford-dawn-project","status":"publish","type":"post","link":"https:\/\/www.aiproblog.com\/index.php\/2018\/09\/05\/democratizing-deep-learning-the-stanford-dawn-project\/","title":{"rendered":"Democratizing Deep Learning \u2013 The Stanford Dawn Project"},"content":{"rendered":"

Author: William Vorhies<\/p>\n

\n

Summary:<\/em><\/strong>\u00a0 How about we develop a ML platform that any domain expert can use to build a deep learning model without help from specialist data scientists, in a fraction of the time and cost.\u00a0 The good news is the folks at the Stanford DAWN project are hard at work on just such a platform and the initial results are extraordinary.<\/em><\/p>\n

\u00a0<\/p>\n

<\/a>Last week we wrote that sufficient labeled training data was the single greatest cost factor<\/u><\/em><\/a> holding back wide adoption of machine learning.\u00a0 I stand corrected.\u00a0 It may be the most expensive, but it\u2019s far from the only thing holding ML back.<\/p>\n

After examining the work of the Stanford DAWN project these researchers propose that literally all the steps in the development process from data acquisition, to feature extraction, to model training, and all the way to productionizing the model are all deeply flawed.\u00a0 Each contributes significantly to holding back adoption of machine learning.<\/p>\n

The Stanford DAWN project<\/u><\/em><\/a> (Data Analytics for What\u2019s Next) is a five year project started just about a year ago by four well known Stanford CS principal investigators Peter Bailis, Kunle Olukotun, Christopher Re, and Matei Zaharia, plus their post docs and Ph.D. students.\u00a0 Together in this team of about 30+ there are plenty of academic and start-up chops to go around.\u00a0<\/p>\n

\u00a0<\/p>\n

What\u2019s Broken<\/strong><\/span><\/p>\n

What they say is wrong with the current process is pretty much what we all know.<\/p>\n

    \n
  1. Training data is too limited and too expensive to create.<\/li>\n
  2. Our algorithms are pretty much OK but it still takes a substantial team of specialist data scientists, data engineers, domain experts, and more supporting staff to get good results.<\/li>\n
  3. And if the number and cost of the expert staff were not sufficiently deterring, our understanding of how to optimize for nodes, layers, and hyperparameters is still primitive. Plus it seems that the rules change based on the type of data being analyzed.<\/li>\n
  4. Finally moving the models into production and keeping them updated is a final hurdle considering that the scoring created by the model can frequently only be achieved by continuing to use the same costly and complex architecture used for training.<\/li>\n<\/ol>\n

    Despite the huge improvements in GPU\/TPU compute, most of that machine time is wasted in trial and error.\u00a0 Our development process is not up to our hardware.<\/p>\n

    In short, it just takes too many people too darn long to create useful models, much less those that rise to the level of human performance.<\/p>\n

    We\u2019ve said it before and the DAWN PIs say it too, this phase in our development looks a lot like the dawn of the PC with lots of lab-coated specialists hovering around, all needed to keep this very young technology alive.<\/p>\n

    The major cloud providers along with everyone else understand this.\u00a0 The introduction of transfer learning<\/u><\/em><\/a> in the last year or so is a kind of stop gap that provides some simplification while reusing many of the very large and expensive pre-trained models they\u2019ve built.\u00a0<\/p>\n

    Still, if we want to get from the earliest prototype PC to today\u2019s simple and reliable laptop, we\u2019ve got a lot of simplifying and standardizing to do.<\/p>\n

    \u00a0<\/p>\n

    Measuring Success<\/strong><\/span><\/p>\n

    As the DAWN team was setting out, like all good data scientists one of the first questions they faced was how to tell if they were actually making things better.\u00a0 They decided to focus on the measure: reducing the time and cost for the end-to-end ML workflow.\u00a0 So there are these design goals:<\/p>\n