Summary:<\/em><\/strong> Recurrent Neural Nets (RNNs) are at the core of the most common AI applications in use today but we are rapidly recognizing broad time series problem types where they don\u2019t fit well.\u00a0 Several alternatives are already in use and one that\u2019s just been introduced, ODE net is a radical departure from our way of thinking about the solution.<\/em><\/p>\n \u00a0<\/p>\n In a sense, the army of data scientists has split off into two groups, each pursuing the separate applications that might be developed from these two techniques.\u00a0 In application there is essentially no overlap since image processing is about processing data that is static (even if only for a second) while RNN-NLP has always interpreted speech and text as time series data.<\/p>\n It turns out though that while RNN\/LSTMs remain the go-to technique for most NLP, the more we try to expand time series applications the more trouble we run into.\u00a0 What\u2019s on the horizon may not be so much a modification of RNNs but perhaps a hard fork to several other innovative new AI methods.<\/p>\n \u00a0<\/p>\n The First Fork<\/strong><\/span><\/p>\n The first fork that we wrote about last year is combining CNNs and RNNs in a single neural network<\/u><\/em><\/a>.\u00a0 The problem to be solved related to images that occurred in time series, that is video, and the most common tasks for this odd mashup is video scene labeling.\u00a0 Turns out this technique is also good for recognizing and labeling emotion in a video and for some types of person recognition based on having seen that person in a video before.<\/p>\n \u00a0<\/p>\n RNN Not So Good for Massive Parallel Processing (MPP)<\/strong><\/span><\/p>\n Also last year, both Google and Facebook addressed a second type of problem with RNNs.\u00a0 That is, because the data to be analyzed extends over several layers in the DNN you have to wait for all those layers to complete before calculating.\u00a0 That also means that MPP isn\u2019t really feasible.\u00a0 Yes this all still happens very fast but not fast enough for the real time language translation apps to avoid noticeable latency.<\/p>\n This second fork caused both these leaders to abandon RNNs for real time translation and adopt a variant of CNNs they labeled Temporal Convolutional Neural Nets (TCNs)<\/u><\/em>.<\/a>\u00a0 This looks a lot like a CNN with the addition of an \u2018Attention\u2019 function.\u00a0 Because they\u2019re structured as CNNs they can be easily adapted to MPP so latency disappears.\u00a0 If you\u2019d like to dig into that follow the hyperlink.<\/p>\n \u00a0<\/p>\n The Third Fork Problem \u2013 Irregular Time Series<\/strong><\/span><\/p>\n There are several other classes of time series problems that are not well addressed by RNNs.\u00a0 Mostly these are characterized by systems having continuous values (think economic or financial variables used to forecast stock prices or physically analog systems), and those in which you want to combine time series data with different frequencies, durations, and start points.<\/p>\n If this last one seems mysterious it shouldn\u2019t be.\u00a0 That describes what your medical history looks like as you visit different doctors, have appointments at different intervals, begin or stop medications at different dosages and intervals, have different physical responses (output variables) to these inputs, and simply grow older or stronger or better or worse in some measurable way.<\/p>\n This is at the core of why the vast majority healthcare applications of AI have been in image recognition.\u00a0 Because our ability to use AI with irregular time series is really deficient in its ability to predict an outcome based on these irregular separate data series.<\/p>\n One solution might be to divide up your parallel medical records into discrete steps of weeks or days or even hours (adding layers to the DNN to increase granularity).\u00a0 In theory this would adapt to the discretization required by RNNs.\u00a0 But you begin to see the problem.\u00a0 To gain maximum benefit you would have to use very fine time buckets which would increase computation cost and complexity and rapidly reach a point of impossibility.\u00a0 Then there\u2019s the issue that many of these time buckets would contain no data.<\/p>\n So both the forecasting community and the healthcare community need an AI solution superior to what RNNs can currently deliver.<\/p>\n \u00a0<\/p>\n ODE net<\/strong><\/span><\/p>\n The name of their system \u201cODE net\u201d stands for Ordinary Differential Equation net.\u00a0 Don\u2019t be misled.\u00a0 ODE net doesn\u2019t look anything like a DNN, with no nodes, layers, or interconnects.\u00a0 It is a method of using a black box differential equation solver (DES) with backpropagation and several other clever adaptions to outperform RNNs in both continuous and discrete time series problems.\u00a0 In other words, this is more like a solid slab of computation than anything that might be visualized as a neural net.<\/p>\n There are several interesting changes in mindset that come along with this method.\u00a0 For example, with an RNN you would specify layers and other hyperparameters, run the experiment and see what accuracy you achieved.\u00a0<\/p>\n With ODE net, there is a direct tradeoff between accuracy and time to train.\u00a0 You specify the desired level of accuracy and the ODE net will find the best way to achieve that, allowing training time to vary.\u00a0 If the training time is unacceptably long, specify a lower accuracy and training is faster.\u00a0 One interesting outcome might be to train at high accuracy but to speed testing by specifying a lower accuracy.<\/p>\n The paper which is available here on arXiv.org<\/u><\/em><\/a> is quite thorough and offers the results of several experiments in which the results are clearly superior to RNNs.\u00a0 It\u2019s still in its research phase, but as will most things in data science that won\u2019t necessarily be long.<\/p>\n \u00a0<\/p>\n A New Way Forward<\/strong><\/span><\/p>\n It\u2019s particularly interesting that solutions to some of these most intractable problems with our current deep dive into DNNs may not look anything like neural nets.\u00a0 It makes me also wonder for example whatever happened to that promising work in evolutionary algorithms that is no longer mainstream.\u00a0 We may be at the beginning of a very interesting hard fork into entirely new methods of AI.<\/p>\n \u00a0<\/p>\n \u00a0<\/p>\n Other articles by Bill Vorhies<\/u><\/em><\/a><\/p>\n \u00a0<\/p>\n About the author:\u00a0 Bill is Contributing Editor for Data Science Central.\u00a0 Bill is also President & Chief Data Scientist at Data-Magnum and has practiced as a data scientist since 2001.\u00a0\u00a0\u00a0 He can be reached at:<\/p>\n Bill@DataScienceCentral.com<\/a> or<\/span> Bill@Data-Magnum.com<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<\/a>Recurrent Neural Nets (RNNs) and their cousins LSTMs are at the very core of the most common applications of AI, natural language processing (NLP).\u00a0 There are far more real world applications of RNN-NLP than any other form of AI, including image recognition and processing with Convolutional Neural Nets (CNNs).<\/p>\n
<\/a>At the Neural Information Processing Systems (NIPS) conference held in Montreal last December researchers from Canada\u2019s Vector Institute presented an entirely new concept in AI time series modeling, and was named one of four best papers from the conference.<\/p>\n