Summary:<\/em><\/strong> This is the second in our \u201cOff the Beaten Path\u201d series looking at innovators in machine learning who have elected strategies and methods outside of the mainstream.\u00a0 In this article we look at Numenta\u2019s unique approach to scalar prediction and anomaly detection based on their own brain research.<\/em><\/p>\n \u00a0<\/em><\/p>\n After swimming against the tide of the \u2018narrow\u2019 or \u2018weak\u2019 AI approaches represented by deep learning\u2019s CNNs and RNN\/LSTMs his bet is starting to pay off.\u00a0 There are now benchmarked studies showing that Numenta\u2019s strong AI computational approach can outperform CNN\/RNN based deep learning (which Hawkins characterizes as \u2018classic\u2019 AI) at scalar predictions (future values of things like commodity, energy, or stock prices) and at anomaly detection.<\/p>\n \u00a0<\/p>\n How Is It Different from Current Deep Learning<\/strong><\/span><\/p>\n The \u2018strong AI\u2019 approach pursued by Numenta relies on computational models drawn directly from the brain\u2019s own architecture.\u00a0<\/p>\n \u2018Weak AI\u2019 by contrast, represented by the full panoply of deep neural nets, acknowledges that it is only suggestive of true brain function, but it gets results.\u00a0<\/p>\n We are all aware of the successes in image, video, text, and speech analysis that CNNs and RNN\/LSTMs have achieved and that is their primary defense.\u00a0 They work.\u00a0 They give good commercial results.\u00a0 But we are also beginning to recognize their weaknesses: large training sets, susceptibility to noise, long training times, complex setup, inability to adapt to changing data, and time invariance that begin to show us where the limits of their development will lead us.<\/p>\n Numenta\u2019s computational approach has a few similarities to these and many unique contributions that require those of us involved in deep learning to consider a wholly different computational paradigm.<\/p>\n \u00a0<\/p>\n Hierarchical Temporal Memory (HTM)<\/strong><\/span><\/p>\n It would take several articles of this length to do justice to the methods introduced by Numenta.\u00a0 Here are the highlights.<\/p>\n “The brain does two things: it does inference, which is recognizing patterns, and it does behavior, which is generating patterns or generating motor behavior.\u00a0 Ninety-nine percent of inference is time-based \u2013 language, audition, touch \u2013 it’s all time-based. You can’t understand touch without moving your hand. The order in which patterns occur is very important.”<\/em><\/p>\n So the \u2018memory\u2019 element of HTM is how the brain differently interprets or relates each sequential input, also called \u2018sequence memory\u2019.<\/p>\n By contrast, conventional deep learning uses static data and is therefore time invariant.\u00a0 Even RNN\/LSTMs that process speech, which is time based, actually do so on static datasets.<\/p>\n \u00a0<\/p>\n Sparse Distributed Representations (SDRs):<\/strong>\u00a0 Each momentary input to the brain, for example from the eye or from touch is understood to be some subset of all the available neurons in that sensor (eye, ear, finger) firing and forwarding that signal upward to other neurons for processing.<\/p>\n Since not all the available neurons fire for each input, the signal sent forward can be seen as a Sparse Distributed Representation (SDR) of those that have fired (the 1s in the binary representation) versus the hundreds or thousands that have not (the 0s in the binary representation).\u00a0 We know from research that on average only about 2% of neurons fire with any given event giving meaning to the term \u2018sparse\u2019.<\/p>\n SDRs lend themselves to vector arrays and because they are sparse have the interesting characteristics that they can be extensively compressed without losing meaning and are very resistant to noise and false positives.<\/p>\n By comparison, deep neural nets fire all neurons in each layer, or at least those that have reached the impulse threshold.\u00a0 This is an acknowledged drawback by current researchers in moving DNNs much beyond where they are today.<\/p>\n \u00a0<\/p>\n Learning is Continuous and Unsupervised:\u00a0<\/strong> Like CNNs and RNNs this is a feature generating system that learns from unlabeled data.<\/p>\n When we look at diagrams of CNNs and RNNs they are typically shown as multiple (deep) layers of neurons which decrease in pyramidal fashion as the signal progresses.\u00a0 Presumably discovering features in this self-constricting architecture down to the final classifying layer.<\/p>\n HTM architecture by contrast is simply columnar with columns of computational neurons passing the information on to upward layers in which pattern discovery and recognition occurs organically by the comparison of one SDR (a single time signal) to the others in the signal train.\u00a0<\/p>\n HTM has the characteristic that it discovers these patterns very rapidly, with as few as on the order of 1,000 SDR observations.\u00a0 This compares with the hundreds of thousands or millions of observations necessary to train CNNs or RNNs.<\/p>\n Also the pattern recognition is unsupervised and can recognize and generalize about changes in the pattern based on changing inputs as soon as they occur.\u00a0 This results in a system that not only trains remarkably quickly but also is self-learning, adaptive, and not confused by changes in the data or by noise.<\/p>\n Numenta offers a deep library of explanatory papers and YouTube videos for those wanting to experiment hands-on.<\/p>\n \u00a0<\/p>\n Where Does HTM Excel<\/strong><\/span><\/p>\n HTM has for many years been a work in progress.\u00a0 That has recently changed.\u00a0 Numenta has published several peer reviewed performance papers and established benchmarking in areas of its strength that highlight its superiority over traditional DNNs and other ML methods on particular types of problems.<\/p>\n In general, Numenta says that the current state of its technology represented by its open source project NuPIC (Numenta Platform for Intelligent Computing) currently excels in three areas:<\/p>\n Anomaly Detection in streaming data<\/strong>.\u00a0 For example:<\/p>\n Scalar Predictions<\/strong>, for example:<\/p>\n Highly Accurate Semantic Search<\/strong> on static and streaming data (these examples are from Corticol.Io a Numenta commercial partner using the SDR concept but not NuPICS)<\/em>.<\/p>\n \u00a0<\/p>\n Two Specific Examples of Performance Better Than DNNs<\/strong><\/span><\/p>\n Taxi Demand Forecast<\/strong><\/span><\/p>\n In this project, the objective was to predict the demand for New York City taxi service 2 \u00bd hours in advance based on a public data stream provided by the New York City Transportation Authority.\u00a0 This was based on historical streaming data at 30 minutes intervals using the previous 1,000, 3,000, or 6,000 observations as the basis for the forward projection 5 periods (2 \u00bd hours) in advance.\u00a0 The study (which you can see here<\/u><\/em><\/a>) compared ARIMA, TDNN, and LSTM to the HTM method where HTM demonstrated the lowest error rate.<\/p>\n \u00a0 \u00a0<\/p>\n Machine Failure Prediction (Anomaly)<\/strong><\/span><\/p>\n The objective of this test was to compare two of the most popular anomaly detection routines (Twitter ADVec and Etsy Skyline) against HTM in a machine failure scenario.\u00a0 In this type of IoT application it\u2019s important that the analytics detect all the anomalies present, detect them as far before occurrence as possible, trigger no false alarms (false positives), and work with real time data.\u00a0 A full description of the study can be found here<\/u><\/em><\/a>.<\/p>\n The results showed that the Numenta HTM outperformed the other methods by a significant margin.\u00a0 Even more significantly, as noted in the caption below, the Numenta HTM method identified the potential failure a full 3 hours before the other techniques.<\/p>\n \u00a0 You can find other benchmark studies on the Numenta site.<\/p>\n \u00a0<\/p>\n The Path Forward<\/strong><\/span><\/p>\n Several things are worthy of note here since as we mentioned earlier the Numenta HTM platform is still a work in progress.<\/p>\n \u00a0<\/p>\n Commercialization<\/strong><\/span><\/p>\n Numenta\u2019s business model currently calls for it to be the center of a commercial ecosystem while retaining its primary research focus.\u00a0 Currently Numenta has two commercially licensed partners, Corticol.Io which focuses on streaming text and semantic interpretation.\u00a0 The second is Grok (Grokstream.com) which has adapted the NuPIC core platform for anomaly detection in all types of IT operational scenarios.\u00a0 The core NuPICs platform is open source if you\u2019re motivated to experiment with potential commercial applications.<\/p>\n \u00a0<\/p>\n Image and Text Classification<\/strong><\/span><\/p>\n A notable absence from the current list of capabilities is image and text classification.\u00a0 There are no current plans for Numenta to develop image classification from static data since that is not on the critical path defined by streaming data.\u00a0 It\u2019s worth noting that others have demonstrated the use of HTM as a superior technique for image classification not using the NuPICs platform.<\/p>\n \u00a0<\/p>\n Near Term Innovation<\/strong><\/span><\/p>\n In my conversation with Christy Maver, Numenta\u2019s VP of Marketing she expressed that they\u00a0are confident that they will have a fairly complete framework for how the neocortex works\u00a0within the timeframe of perhaps a year.\u00a0 This last push is in the area of sensorimotor integration that would be the core concept in applying the HTM architecture to robotics.<\/p>\n For commercial development, the focus will be on partners to license the IP.\u00a0 Even IBM established a Cortical Research Center a few years back staffed with about 100 researchers to examine the Numenta HTM approach.\u00a0 Like so many others now trying to advance AI by more closely modeling the brain, IBM, like Intel and others has moved off in the direction of specialty chips that fall in the category on neuromorphic or spiking chips.\u00a0\u00a0 Brainchip out of Irvine already has a spiking neuromorphic chip in commercial use.\u00a0 As Maver alluded, there may be a silicon representation of Numenta\u2019s HTM in the future.<\/p>\n \u00a0<\/p>\n \u00a0<\/p>\n Other articles in this series:<\/p>\n Off the Beaten path \u2013 Using Deep Forests to Outperform CNNs and RNNs<\/a><\/u><\/em><\/p>\n \u00a0<\/p>\n \u00a0<\/p>\n About the author:\u00a0 Bill Vorhies is Editorial Director for Data Science Central and has practiced as a data scientist since 2001.\u00a0 He can be reached at:<\/p>\n Bill@DataScienceCentral.com<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"![](\"http:\/\/api.ning.com\/files\/4QC9xOMBQLFB32eSOd6g1EGn852ASU5W-cTwlh-E3E49ScFM5B1wgLtW2Qud3iF9pf4j9NndgldBtmRFLtBXN9XS1rnG1RYn\/strongvsweakaismall.png\")
<\/a>Numenta, the machine intelligence company founded in 2005 by Jeff Hawkins of Palm Pilot fame might well be the poster child for \u2018off the beaten path\u2019.\u00a0 More a research laboratory than commercial venture, Hawkins has been pursuing a strong-AI model of computation that will at once directly model the human brain, and as a result be a general purpose solution to all types of machine learning problems.<\/p>\n<\/a>HTM and Time:<\/strong>\u00a0 Hawkins uses the term Hierarchical Temporal Memory (HTM)<\/strong> to describe his overall approach.\u00a0 The first key to understanding is that HTM relies on data that streams over time.\u00a0 Quoting from previous interviews, Hawkins says,<\/p>\n\n
\n
\n
<\/a><\/p>\n<\/a><\/p>\n<\/a><\/p>\n