{"id":2122,"date":"2019-05-10T06:36:32","date_gmt":"2019-05-10T06:36:32","guid":{"rendered":"https:\/\/www.aiproblog.com\/index.php\/2019\/05\/10\/in-big-data-its-not-the-volume-thats-important-its-the-granularity\/"},"modified":"2019-05-10T06:36:32","modified_gmt":"2019-05-10T06:36:32","slug":"in-big-data-its-not-the-volume-thats-important-its-the-granularity","status":"publish","type":"post","link":"https:\/\/www.aiproblog.com\/index.php\/2019\/05\/10\/in-big-data-its-not-the-volume-thats-important-its-the-granularity\/","title":{"rendered":"In Big Data, It\u2019s Not the Volume that\u2019s Important, It\u2019s the Granularity"},"content":{"rendered":"<p>Author: Bill Schmarzo<\/p>\n<div>\n<p>In many of my presentations and lectures, I have made the following declaration:<\/p>\n<p><em>In Big Data, it isn\u2019t the volume of data that\u2019s interesting, it\u2019s the granularity; it\u2019s the ability to build detailed analytic or behavioral profiles on every human and every device that ultimately drives monetization.<\/em><\/p>\n<p>Ever-larger volumes of aggregated data enable organizations to spot trends \u2013 what products are hot, what movies or TV shows are trendy, what restaurants or destinations are popular, etc.<span>\u00a0<\/span> This is all interesting information, <em>but how do I monetize these trends?<\/em><span>\u00a0<\/span> I can build more products or create more TV episodes or promote select destinations, but to make those trends actionable \u2013 to monetize these trends \u2013 I need to get down to the granularity of the individual.<span>\u00a0<\/span><\/p>\n<p>I need detailed, individual insights with regards to who is interested and the conditions (price, location, time of day\/day of week, weather, season, etc.) that attracts them.<span>\u00a0<\/span> I need to understand each individual\u2019s behaviors (tendencies, propensities, inclinations, patterns, trends, associations, relationships) in order to target my efforts to drive the most value at the least cost.<span>\u00a0<\/span> I need to be able to codify (turn into math) customer, product and operational behaviors such as individual patterns, trends, associations, and relationships.<\/p>\n<p>I need Analytic Profiles.<\/p>\n<h1><strong>Role of Analytic Profiles<\/strong><\/h1>\n<p><a href=\"https:\/\/www.linkedin.com\/pulse\/best-practices-analytics-profiles-bill-schmarzo\/\">Analytic Profiles<\/a>provide a structure for capturing an individual\u2019s behaviors (tendencies, propensities, inclinations, patterns, trends, associations, relationships) in a way that facilitates the refinement and sharing of these digital assets across multiple business and operational use cases.<span>\u00a0<\/span>Analytic Profiles work for any humans including customers, prospects, students, teachers, patients, physicians, nurses, engineers, technicians, mechanics, athletes, coaches, accountants, lawyers, managers, etc.<span>\u00a0<\/span> Figure 1 shows an example of a customer analytic profile.<\/p>\n<p><a href=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386161369?profile=original\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386161369?profile=RESIZE_710x\" class=\"align-full\"><\/a><\/p>\n<p><strong>Figure<\/strong> <strong><span>1<\/span><\/strong><strong>:<span>\u00a0<\/span> Customer Analytic Profile<\/strong><\/p>\n<p>In the same way that we can capture individual (human) behaviors with Analytic Profiles, we can leverage <a href=\"https:\/\/www.linkedin.com\/pulse\/me-myself-digital-twins-bill-schmarzo\/\">Digital Twins<\/a>to capture the behaviors of individual assets.<span>\u00a0<\/span> A Digital Twin is a digital representation of an asset (device, product, thing) that enables companies to better understand and predict their performance, uncover new revenue opportunities, and optimize their operations (see Figure 2).<\/p>\n<p><a href=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386166847?profile=original\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386166847?profile=RESIZE_710x\" class=\"align-full\"><\/a><\/p>\n<p><strong>Figure<\/strong> <strong><span>2<\/span><\/strong><strong>: Digital Twin<\/strong><\/p>\n<p>Analytic Profiles are much more than just a \u201c360-degree view of customer\u201d (an unactionable phrase that I loathe).<span>\u00a0<\/span> Analytic Profiles possess the ability to capture and ultimately act upon an individual\u2019s propensities, tendencies, inclinations, patterns, relationships and behaviors.<span>\u00a0<\/span> And a Digital Twin is much more than just demographic and performance data about a device. <em>To fully monetize the Digital Twin concept, one must also capture predictions about likelihood performance and behaviors that can lead to prescriptive and preventative actions.<\/em><\/p>\n<p>Again, it\u2019s the granularity of humans and devices that lead to monetization opportunities.<span>\u00a0<\/span> And I can monetize those individual humans and assets using common analytic techniques.<\/p>\n<h1><strong>Analytic Technique:<span>\u00a0<\/span>Cohort Analysis<\/strong><\/h1>\n<p>Cohort Analysis is a subset of human behavioral analytics that takes purchase and engagement data and breaks down the performance assessments into related (or \u201clike\u201d) groups of humans.<span>\u00a0<\/span> Cohort Analysis can predict the products groups of customers might want to buy and the actions they are likely to take.<span>\u00a0<\/span> Cohort analysis allows a company to see patterns and trends across the lifecycle of a group of humans. By quantifying those patterns and trends, a company can adapt and tailor its service to meet the unique needs of those specific cohorts.<\/p>\n<p>While cohort analysis is typically done for groups of clusters of humans, cohort analysis can also work for clusters of devices. Organizations can use cohort analysis to understand the trends and patterns of related or \u201clike\u201d devices over time that can be used to optimize their maintenance, repair and upgrade and end of line decisions (see Figure 3).<\/p>\n<p><strong>Figure<\/strong> <strong><span>3<\/span><\/strong><strong>: Cohort Analysis<\/strong><\/p>\n<p><a href=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386173540?profile=original\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386173540?profile=RESIZE_710x\" class=\"align-full\"><\/a><\/p>\n<p>See the blog \u201c<a href=\"https:\/\/www.linkedin.com\/pulse\/cohort-analysis-age-digital-twins-bill-schmarzo\/\">Cohort Analysis &#8211; Cohort Analysis in the Age of Digital Twins<\/a>\u201d for more details on the application of cohort analysis in the world of IoT.<\/p>\n<h1><strong>Analytic Technique: Micro-to-Macro Analytics<\/strong><\/h1>\n<p>Population health focuses on understanding the overall health and the potential health outcomes of groups of individuals.<span>\u00a0<\/span> For example, population health may predict a 25% uptick in influenza outbreaks in the Midwest in January.<span>\u00a0<\/span> Typically, the different population health organizations would advise its patients to immediately get flu shots.<\/p>\n<p>But maybe there are some important questions that need to be answered before issuing a blanket flu shot recommendation, such as:<\/p>\n<ul>\n<li>Should everyone get a shot or only those who are most likely to contract the influenza?<\/li>\n<li>Should everyone get a shot immediately, or is there a way to delay some of the shots in order to optimize the current vaccine supply while more is being manufactured?<\/li>\n<li>Are there some patients (e.g., elderly, infants, cancer patients) where the side effects of the shot are riskier than the influenza itself?<\/li>\n<\/ul>\n<p>At the macro or aggregated level, it is nearly impossible to make those decisions.<span>\u00a0<\/span> However at the granularity of the individual patient, we can leverage patient analytic profiles (with detailed care and treatment history coupled with relevant healthcare and wellness scores) that allows the healthcare providers to make specific decisions about who should get the shots and when given and the potential benefits of the shots weighted against the potential side effects.<\/p>\n<p>Those individual recommendations can then be aggregated into a group level to yield more insights and recommendations about who else should be getting shots using analytic techniques such as cohort analysis (see Figure 4).<\/p>\n<p><a href=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386177362?profile=original\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386177362?profile=RESIZE_710x\" class=\"align-full\"><\/a><\/p>\n<p><strong>Figure<\/strong> <strong><span>4<\/span><\/strong><strong>:<span>\u00a0<\/span> Micro-to-Macro Analytic Modeling Process<\/strong><\/p>\n<p>See the blog \u201c<a href=\"https:\/\/www.linkedin.com\/pulse\/challenge-macro-analytics-bill-schmarzo\/\">Micro-to-Macro Analytics &#8211; The Challenge of Macro Analytics<\/a>\u201d for more details on the Micro-to-Macro Analytics challenge.<\/p>\n<h1><strong>Analytic Technique: Predicted Customer Lifetime Value<\/strong><\/h1>\n<p>The traditional Customer Lifetime Value (LTV) score is based upon historical transactions, engagements and interactions. For example, you would leverage historical purchases, returns, market baskets, product margins and sales and support engagement data to create a \u201cCustomer LTV\u201d score. This score is a great starting point for optimizing the organization\u2019s financial and people resources around the organization\u2019s \u201cmost valuable\u201d customers.<\/p>\n<p>What if instead of using this \u201ccurrent\u201d Customer LTV, we leveraged new data sources and new analytic techniques to create a predicted Customer LTV score that predicts a customer\u2019s value to the organization? <span>\u00a0\u00a0<\/span>This predicted Customer LTV would leverage second level metrics or scores \u2013 Credit score, Retention Score, Fraud Score, Advocacy Score, Frequency Score, Likelihood to Recommend score \u2013 to create this Predicted Customer LTV (see Figure 5).<\/p>\n<p><a href=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386180881?profile=original\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386180881?profile=RESIZE_710x\" class=\"align-full\"><\/a><\/p>\n<p><strong>Figure<\/strong> <strong><span>5<\/span><\/strong><strong>:<span>\u00a0<\/span> Predicted Customer Lifetime Value<\/strong><\/p>\n<p>For example, the \u201cPredicted Customer LTV\u201d score in figure 5 highlights the following:<\/p>\n<ul>\n<li>Customers 101 and 102 are tapped out and probably should not be the focus of heavy sales and marketing efforts (you don\u2019t want to ignore Customers 101 and 102, but they should not be the primary focus of future sales and marketing initiatives).<\/li>\n<li>On the other hand, Customers 103 and 104 have significant untapped potential; these are the types of customer to target sales, marketing, support and product development investments in order to capture that untapped potential.<\/li>\n<\/ul>\n<p>This \u201cPredictive Customer LTV\u201d score would be much more effective in helping organizations optimize their sales, marketing, service, support and product development resources.<\/p>\n<p>See the blog \u201c<a href=\"https:\/\/www.linkedin.com\/pulse\/what-defines-your-most-valuable-customers-bill-schmarzo\/\">What Defines Your \u2018Most Valuable\u2019 Customers<\/a>?\u201d for more details on the concept of the Predicted Customer Lifetime Value; it\u2019s a game changer for re-framing how you value your customers!!<\/p>\n<h1><strong>IoT: Bringing It All Together<\/strong><\/h1>\n<p>IoT offers the potential to blend analytic profiles and digital twins to yield new sources of customer, product and operational value.<span>\u00a0<\/span> In particular, Edge Analytics provides the opportunity to bring a real-time or near real-time dimension into our monetization discussions; to improve our ability to catch either humans or devices in the act of doing something where augmented intelligence can make that human or device interaction more effective.<\/p>\n<p>We can leverage IoT sensors and edge analytics to transform \u201cconnected\u201d entities into \u201cSmart\u201d entities.<span>\u00a0<\/span> For example, with smart factories, integrating Analytic Profiles (for technicians, maintenance engineers, facilities managers) and Digital Twins (for milling machines, power presses, grinding machines, lathes, cranes, compressors, forklifts) can add real-time, low-latency analytics and decision optimization capabilities across each \u201csmart\u201d factory use cases (see Table 1).<\/p>\n<p>\u00a0<\/p>\n<table>\n<tbody>\n<tr>\n<td colspan=\"2\" width=\"623\">\n<p><strong>Potential \u201cSmart Factory\u201d Use Cases<\/strong><\/p>\n<\/td>\n<\/tr>\n<tr>\n<td width=\"258\">\n<p>&#8211;\u00a0 Demand Forecasting<\/p>\n<p>&#8211;\u00a0 Production Planning<\/p>\n<p>&#8211;\u00a0 Factory Yield Optimization<\/p>\n<p>&#8211;\u00a0 Facility management\u00a0<\/p>\n<p>&#8211; \u00a0Production\u00a0flow monitoring<\/p>\n<p>&#8211; \u00a0Predictive Maintenance<\/p>\n<p>&#8211; \u00a0Inventory Reduction<\/p>\n<p>&#8211; \u00a0Energy Efficiency<\/p>\n<p>&#8211; \u00a0Lead time Reduction<\/p>\n<\/td>\n<td width=\"366\">\n<p>&#8211; \u00a0Worker Scheduling Optimization<\/p>\n<p>&#8211; \u00a0Time-to-market Reduction<\/p>\n<p>&#8211; \u00a0Plant Safety and Security\u00a0<\/p>\n<p>&#8211; \u00a0Quality Cost Reduction<\/p>\n<p>&#8211; \u00a0Asset Utilization Optimization (OEE)<\/p>\n<p>&#8211; \u00a0Packaging Optimization<\/p>\n<p>&#8211; \u00a0Logistics and Supply Chain Optimization<\/p>\n<p>&#8211; \u00a0Sustainability<\/p>\n<p>&#8211; \u00a0<span>\u2026<\/span><\/p>\n<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n<p><strong>Table 1:<span>\u00a0<\/span>Potential \u201cSmart Factory\u201d Use Cases<\/strong><\/p>\n<p>To exploit the granularity and real-time\/near real-time benefits offered by IoT and Edge Analytics, we will need an IoT Analytics architecture that puts the right advanced analytic capabilities at the right points in the IoT architecture, for example (see Figure 6):<span>\u00a0<\/span><\/p>\n<ul>\n<li><strong>Reinforcement Learning<\/strong>at the edge to balance the ever-changing sets of rewards and costs to achieve optimal results by maximizing rewards (energy usage, production, cooling, heating) while minimizing costs (wear &#038; tear, depletion, overtime costs, energy consumption).<\/li>\n<li><strong>Machine Learning<\/strong><span>to uncover and quantify trends, patterns, associations and relationships buried in data that might yield better predictors of performance including clustering and classifying of similar activities and outcomes, association and correlation of related activities and outcomes, and regression analysis to quantify cause-and-effect.<\/span><\/li>\n<li><strong><span>Deep Learning<\/span><\/strong><span>to uncover dangerous or advantageous states in order to predict, prevent and monetize via determination of image contents, discovering detailed nuances of causation, supporting natural language conversations, and video surveillance to identify, count, measure and alert.<\/span><\/li>\n<\/ul>\n<p><strong><a href=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386187988?profile=original\" target=\"_blank\" rel=\"noopener noreferrer\"><img decoding=\"async\" src=\"https:\/\/storage.ning.com\/topology\/rest\/1.0\/file\/get\/2386187988?profile=RESIZE_710x\" class=\"align-full\"><\/a><\/strong><\/p>\n<p><strong>Figure<\/strong> <strong><span>6<\/span><\/strong><strong>: \u201cSmart\u201d Factories:<span>\u00a0<\/span> Edge-to-Cloud Intelligence<\/strong><\/p>\n<p><strong>Summary<\/strong><\/p>\n<p>As IT and OT professionals in organizations seek to deploy IoT, don\u2019t get seduced to start with the sensors. Start by identifying, validating, value and prioritizing the use cases that can exploit the real-time data generation and edge analytics to derive and drive new sources of customer, product and operational value.<span>\u00a0<\/span> And in the end, it isn\u2019t the volume of data that\u2019s valuable, it\u2019s the granularity.<\/p>\n<p>\u00a0<\/p>\n<\/div>\n<p><a href=\"https:\/\/www.datasciencecentral.com\/xn\/detail\/6448529:BlogPost:824074\">Go to Source<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Author: Bill Schmarzo In many of my presentations and lectures, I have made the following declaration: In Big Data, it isn\u2019t the volume of data [&hellip;] <span class=\"read-more-link\"><a class=\"read-more\" href=\"https:\/\/www.aiproblog.com\/index.php\/2019\/05\/10\/in-big-data-its-not-the-volume-thats-important-its-the-granularity\/\">Read More<\/a><\/span><\/p>\n","protected":false},"author":1,"featured_media":457,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_bbp_topic_count":0,"_bbp_reply_count":0,"_bbp_total_topic_count":0,"_bbp_total_reply_count":0,"_bbp_voice_count":0,"_bbp_anonymous_reply_count":0,"_bbp_topic_count_hidden":0,"_bbp_reply_count_hidden":0,"_bbp_forum_subforum_count":0,"footnotes":""},"categories":[26],"tags":[],"_links":{"self":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/2122"}],"collection":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/comments?post=2122"}],"version-history":[{"count":0,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/posts\/2122\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media\/459"}],"wp:attachment":[{"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/media?parent=2122"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/categories?post=2122"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.aiproblog.com\/index.php\/wp-json\/wp\/v2\/tags?post=2122"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}