Summary:<\/em><\/strong> \u00a0IBM\u2019s Watson QAM (Question Answering Machine), famous for its 2011 Jeopardy win was supposed to bring huge payoffs in healthcare.\u00a0 Instead both IBM and its Watson Healthcare customers are rapidly paring back these projects that have largely failed to pay off.\u00a0 Watson was the first big out-of-the-box commercial application in ML\/AI.\u00a0 Has it become obsolete?<\/em><\/p>\n \u00a0<\/em>\u00a0 Kudos to IBM for being the leader in bringing us so many AI firsts including these:<\/p>\n I\u2019m sure I\u2019m leaving out many other notable firsts that IBM has scored but since it\u2019s Watson we want to talk about, we\u2019ll stop there.<\/p>\n The remarkable thing about Watson is that in 2011 the other skills that we think of as AI, image and video processing, facial recognition, text and speech processing, game play beyond chess, autonomous vehicles, all these were so primitive they were not yet close to commercial acceptance and wouldn\u2019t be for several more years.<\/p>\n Fast forward to 2013, IBM announces that healthcare and particularly cancer diagnoses and treatment recommendation will the jackpot for Watson.\u00a0<\/p>\n By 2015 IBM had already invested north of a reported $15 Billion in Watson.<\/p>\n By 2018 the press over the last two years has been heavy with hospitals scaling back or abandoning Watson altogether.\u00a0 In 2017 prestigious MD Anderson tabled their project.\u00a0 New York\u2019s Sloan Kettering Cancer Center has been helping to train Watson since 2012 but doesn\u2019t use the product on its patients.\u00a0 IBM itself announced a scaling back of its employees working on Watson healthcare.<\/p>\n Although Watson was the first large scale AI out of the box, all of this news about its shortcomings makes us ask, is Watson the first AI technique to be abandoned in this fast moving world of ever increasing AI?<\/strong><\/p>\n \u00a0<\/p>\n What exactly is Watson?<\/strong><\/span><\/p>\n Before we can address the question we have to clarify exactly what Watson is.\u00a0 The problem is that after the Watson Jeopardy win in 2011 IBM was quick to name almost every version of AI it introduced as Watson.\u00a0 This extends to CNN-based image processing and even analytic platforms for modeling that have nothing whatever to do with the original Watson, or the Watson now taking a beating.<\/p>\n \u00a0<\/p>\n Watson is a Question Answering Machine (QAM)<\/strong><\/span><\/p>\n The Watson QAM being experienced by hospitals is very much the same Watson that won Jeopardy.\u00a0 That is, using NLP text inputs and outputs the Watson QAM searches a large corpus of knowledge and provides the one answer judged most likely to be correct.<\/p>\n There\u2019s an important distinction here between a QAM and simple search.\u00a0<\/p>\n As data scientists are intimately aware, but perhaps not the doctors using Watson, all models come with errors, both false negatives and false positives.\u00a0 In cancer this is particularly problematic.\u00a0 You don\u2019t want to be the patient with the false negative whose cancer was missed any more than you want to be the healthy patient given a mis-diagnosis of cancer.<\/p>\n What doctors have been experiencing is a little more nuanced.\u00a0 As expected most of the time Watson was correct in recommending a diagnosis or treatment.\u00a0 However, on occasion Watson would recommend a clearly wrong or inadequate course of treatment.<\/p>\n During the period of optimism following launch, both the hospitals and IBM billed this as a worthwhile second opinion.\u00a0 However, as time went on doctors in the US found that they had to constantly double check the Watson recommendation and that it wasn\u2019t telling them anything they didn\u2019t already know.<\/p>\n During these last several years, IBM also did a pivot with Watson and introduced a version unique to genomics intending to identify treatments based on genomic markers in the patient.\u00a0 There are scattered anecdotal reports that on occasion Watson Genomics would find something the doctors had not anticipated.\u00a0 In limited overseas use these reports were somewhat more common.\u00a0 But in the US where IBM was reportedly charging $200 to $1,000 per patient for this review this didn\u2019t pencil out financially for hospitals.<\/p>\n The bottom line is that Watson for cancer and to a lesser extent in other healthcare applications looks to be limping toward extinction.<\/p>\n \u00a0<\/p>\n Why is this happening?<\/strong><\/span><\/p>\n It\u2019s possible that this is simply a flaw in execution.\u00a0<\/p>\n It\u2019s also possible that this indicates that the future for AI QAMs has reached a limit and won\u2019t be a major component of AI going forward.<\/p>\n It\u2019s also possible that it\u2019s a little bit of both.<\/p>\n To explore this we need to look back at how the QAM works and what it takes to set it up.\u00a0<\/p>\n \u00a0<\/p>\n The Fundamentals of QAMs<\/strong><\/span><\/p>\n \u00a0 Natural Language Processing (NLP)<\/span><\/strong>:<\/span>\u00a0 NLP is at the core of QAMs.\u00a0 NLP has been steadily advancing to be able to interpret the meaning behind a string of words and to interpret the context of those words.\u00a0 (e.g. \u201cI\u2019m feeling blue<\/em>\u201d, \u201cfind the boat with the red bow<\/em>\u201d).\u00a0 RNNs with their increasing ability to analyze strings or sequences of words both as input and output are a major driver of improvement.\u00a0 So the QAM is able to accept conversational queries (here\u2019s my patient\u2019s medical records and current status, what\u2019s the best course of action), and to provide text output.<\/p>\n Access to a Curated Knowledge Base:<\/strong><\/span> The process begins by loading a large amount of structured and unstructured source data relating to the domain to be considered (e.g. cancer diagnosis, healthcare utilization management, law, social media opinion).\u00a0 The knowledge base is human-curated and must be continuously human-updated to remove source documents that are no longer accurate or current as well as adding new material.<\/p>\n Ingestion:<\/strong>\u00a0<\/span> QAMs like Watson then start their initial exploration of the knowledge base building indices and metadata to make their subsequent processing more efficient.\u00a0 QAMs may also build graph database adjuncts to assist.<\/p>\n Initial Training:<\/strong><\/span> \u00a0QAMs require a form of supervised learning.\u00a0 Data Scientists load a large number of question and answer pairs from which the QAM learns to generalize which terms and idioms go together and also the core of logic regarding most likely answers.\u00a0 QAMs don\u2019t simply repeat these \u2018correct\u2019 sample answers, they learn to go beyond and find other correct answers based on this training data.<\/p>\n Hypothesis and Conclusion:<\/strong>\u00a0<\/span> When asked a question, the QAM will parse the question to develop a series of potential meanings or hypotheses and look for evidence in the knowledge base supporting them.\u00a0 Each hypothesis is then statistically evaluated for the QAM\u2019s confidence that it is correct and the answers are presented to the end user.\u00a0<\/p>\n Knowledge Discovery:<\/strong><\/span> \u00a0In some applications multiple answers or alternatives may actually be the goal.\u00a0 These may represent combinations of facts and circumstances that had not previously been thought of by humans, such as combinations of chemicals, drugs, treatments, materials, or chains of DNA that may represent new and novel innovations in their field.\u00a0 In some cancer applications Watson returns a prioritized list of possible treatments.<\/p>\n \u00a0<\/p>\n Where is the breakdown occurring?<\/strong><\/span><\/p>\n This isn\u2019t meant to be a forensic study of each use case but I have a strong hunch what\u2019s going on here.\u00a0 QAMs like Watson are extremely labor intensive, much more so than any other ML\/AI we\u2019re currently utilizing.<\/p>\n There is a very large amount of human labor required to identify and load all the documents and data needed to set up the original corpus of knowledge, and an on-going requirement to continuously review and remove knowledge that is out of date while keeping up with all the new findings in the field.<\/p>\n Add to this the initial and on-going training of the search relevance model which is trained off of human-generated question-and-answer pairs.<\/p>\n This is a very different model of AI\/ML implementation than we\u2019ve become used to.\u00a0 My suspicion is that the very large labor component in maintaining the database for a topic as vast and complex as cancer, or healthcare in general, has been its Achilles heel.<\/p>\n \u00a0<\/p>\n Are there examples of where Watson is being successful?<\/strong><\/span><\/p>\n I\u2019m sure there are in situations where the corpus of knowledge is more constrained and less fast changing.\u00a0 When we reviewed Watson in 2016 we listed 30 different examples<\/u><\/em><\/a> of how Watson was being employed including these:<\/p>\n Macys<\/strong> developed \u201cMacy\u2019s On Call,\u201d a mobile web app that taps Watson to allow customers to input natural language questions regarding each store\u2019s unique product assortment, services and facilities and receive a customized response to the inquiry.<\/p>\n VineSleuth<\/strong> developed its Wine4.me app to provide wine recommendations for consumers based on sensory science and predictive algorithms. The start-up uses Watson’s language classifier and translation services in kiosks in grocery stores.<\/p>\n Hilton Worldwide<\/strong> uses Watson to power \u201cConnie\u201d\u2013the first Watson-enabled robot concierge in the hospitality industry. Connie draws on domain knowledge from Watson and WayBlazer to inform guests on local tourist attractions, dining recommendations and hotel features and amenities.<\/p>\n Purple Forge<\/strong> developed a Watson based 311 Service for Surrey, Canada to answer citizens’ questions about government services. (When is recyclables pickup?) The app can answer more than 10,000 questions, more efficiently and at lower cost than humans.<\/p>\n \u00a0<\/p>\n The way forward<\/strong><\/span><\/p>\n What these examples have in common is that the knowledge base is much more limited and\/or slow to change.\u00a0 Second and perhaps more relevant is that these simpler customer-facing applications are now being addressed with the explosive adoption of chatbots.<\/p>\n So it appears that the \u2018bottom end\u2019 of this market is benefiting from the advancements in NLP via chatbots, with or more commonly without Watson QAM.<\/p>\n At the \u2018top end\u2019 of this market is where the corpus of knowledge is very large and fast changing.\u00a0 The promise was that a sophisticated QAM hypothesis\/search algorithm could combine elements of knowledge not previously combined to create unique new insights.<\/p>\n Watson has some competition in this arena coming at these complex problems from a different approach.\u00a0 For example, one hoped for outcome was the discovery of new chemicals, materials, drugs, or DNA functions.\u00a0 While Watson could conceivably still prove useful in this arena, researchers increasingly are opting for less labor intensive CNNs and RNNs for discovery<\/u><\/em>.<\/a>\u00a0 This is particularly true in the field of biology<\/u><\/em><\/a>.<\/p>\n There may still be a sweet spot for Watson in the middle of this continuum but those opportunities seem increasingly hemmed in by chatbots on the bottom and more advanced techniques on the top.\u00a0<\/p>\n It\u2019s probably not time to say that QAMs no longer have a place in the pantheon of AI\/ML, but the very high labor requirements in setup and maintenance are not as enticing as accomplishing much the same thing with deep neural nets, reinforcement learning, and tons of non-human compute power.<\/p>\n \u00a0<\/p>\n Previous Articles about Watson<\/strong><\/span><\/p>\n What Exactly is Watson?<\/u><\/em><\/a><\/p>\n What Can Modern Watson Do?<\/u><\/em><\/a><\/p>\n 30 Fun Ideas for Starting New AI Businesses and Services with Watson<\/u><\/em><\/a><\/p>\n IBM Watson Does Your Taxes: Question Answering Machine versus Expert System<\/u><\/em><\/a><\/p>\n \u00a0<\/p>\n Other articles by Bill Vorhies.<\/u><\/em><\/a><\/p>\n \u00a0<\/p>\n About the author:\u00a0 Bill Vorhies is Editorial Director for Data Science Central and has practiced as a data scientist since 2001.\u00a0 \u00a0\u00a0He can be reached at:<\/p>\n Bill@Data-Magnum.com<\/a> or<\/span> Bill@DataScienceCentral.com<\/a><\/p>\n<\/div>\n Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":" Author: William Vorhies Summary: \u00a0IBM\u2019s Watson QAM (Question Answering Machine), famous for its 2011 Jeopardy win was supposed to bring huge payoffs in healthcare.\u00a0 Instead […] Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":466,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[26],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/963"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=963"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/963\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/473"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=963"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=963"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=963"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}![](\"http:\/\/api.ning.com\/files\/weTFMmNGOTqw5hYOuCzmvnVclkwplBKE*gOaf2OfkrGLmPSwk2bwbm3IWiI8bKfVamIGDA7sk7lwBwJJWqrxqTzrI48yFtOA\/jeopardyasturingtest.jpg\")
<\/a><\/p>\n\n
\n
<\/a><\/strong><\/p>\n