Welcome!

CMS Authors: Mehdi Daoudi, Rishi Bhargava, Harry Trott, Xenia von Wedel, Carmen Gonzalez

Related Topics: @ThingsExpo, @CloudExpo, @BigDataExpo

@ThingsExpo: Blog Feed Post

Difference Between #BigData and Internet of Things | @ThingsExpo #IoT #M2M

What does it mean, as a vendor, to say that you support the Internet of Things (IoT) from an analytics perspective?

A recent argument with folks whose intelligence I hold in high regard (like Tom, Brandon, Wei, Anil, etc.) got me thinking about the following question:

What does it mean, as a vendor, to say that you support the Internet of Things (IoT) from an analytics perspective?

I think the heart of that question really boils down to this:

What are the differences between big data (which is analyzing large amounts of mostly human-generated data to support longer-duration use cases such as predictive maintenance, capacity planning, customer 360 and revenue protection) and IoT (which is aggregating and compressing massive amounts of low latency / low duration / high volume machine-generated data coming from a wide variety of sensors to support real-time use cases such as operational optimization, real-time ad bidding, fraud detection, and security breach detection)?

I don’t believe that loading sensor data into a data lake and performing data science to create predictive analytic models qualifies as doing IoT analytics.  To me, that’s just big data (and potentially REALLY BIG DATA with all that sensor data).  In order for one to claim that they can deliver IoT analytic solutions requires big data (with data science and a data lake), but IoT analytics must also include:

  1. Streaming data management with the ability to ingest, aggregate (e.g., mean, median, mode) and compress real-time data coming off a wide variety of sensor devices “at the edge” of the network, and
  2. Edge analytics that automatically analyzes real-time sensor data and renders real-time decisions (actions) at the edge of the network that optimizes operational performance (blade angle or yaw) or flags unusual performance or behaviors for immediate investigation (security breaches, fraud detection).

If you cannot manage real-time streaming data and make real-time analytics and real-time decisions at the edge, then you are not doing IOT or IOT analytics, in my humble opinion.  So what is required to support these IoT data management and analytic requirements?

The IoT “Analytics” Challenge
The Internet of Things (or Industrial Internet) operates at machine-scale, by dealing with machine-to-machine generated data.  This machine-generated data creates discrete observations (e.g., temperature, vibration, pressure, humidity) at very high signal rates (1,000s of messages/sec).  Add to this the complexity that the sensor data values rarely change (e.g., temperature operates within an acceptably small range).  However, when the values do change the ramifications, the changes will likely be important.

Consequently to support real-time edge analytics, we need to provide detailed data that can flag observations of concern, but then doesn’t overwhelm the ability to get meaningful data back to the core (data lake) for more broad-based, strategic analysis.

One way that we see organizations addressing the IoT analytics needs is via a 3-tier Analytics Architecture (see Figure 1).

Figure 1: IoT Analytics 3-Tier Architecture

We will use a wind turbine farm to help illustrate the 3-tier analytics architecture capabilities.

Tier 1 performs individual wind turbine real-time performance analysis and optimization.  Tier 1 must manage (ingest and compress) real-time data streams coming off of multiple, heterogeneous sensors. Tier 1 analyzes the data, and processes the incoming data against static or dynamically updated analytic models (e.g., rules-based, decision trees) for immediate or near-immediate actions.

Purpose-built T1 edge gateways leverage real-time data compression techniques (e.g., see the article “timeseries storage and data compression” for more information on timeseries databases) to only send a subset of the critical data (e.g., data that has changed) back to T2 and T3 (core).

Let’s say that you are monitoring the temperatures of a compressor inside of a large industrial engine.  Let’s say the average temperature of that compressor is 99 degrees, and only varies between 98 to 100 degrees within a 99% confidence level.  Let’s also say the compressor is emitting the following temperature readings 10 times a second:

99, 99, 99, 98, 98, 99, 99, 98, 99, 99, 100, 99, 99, 99, 100, 99, 98, 99, 99…

You have 10,000 of readings that don’t vary from that range.  So why send all of the readings (which from a transmission bandwidth perspective could be significant)?  Instead, use a timeseries database to only send mean, medium, mode, variances, standard deviation and other statistical variables of the 10,000 readings instead of the individual 10,000 readings.

However, let’s say that all of a sudden we start getting readings outside the normal 99% confidence level:

99, 99, 99, 100, 100, 101, 101, 102, 102, 103, 104, 104, 105, …

Then we’d apply basic Change Data Capture (CDC) techniques to capture and transmit the subset of critical data to T2 and T3 (core).

Consequently, edge gateways leverage timeseries compression techniques to drive faster automated decisions while only sending a subset of critical data to the core for further analysis and action.

The Tier 1 analytics are likely being done via an on-premise analytics server or gateway (see Figure 2).

Figure 2:  IoT Tier 1 Analytics

Tier 2 optimizes performance and predicts maintenance needs across the wind turbines in the same wind farm.  Tier 2 requires a distributed dynamic content processing rule generation and execution analytics engine that integrates and analyzes data aggregated across the potentially heterogeneous wind turbines. Cohort analysis is typical in order to identify, validate and codify performance problems and opportunities across the cohort wind turbines.  For example, in the wind farm, the Tier 2 analytics are responsible for real-time learning that can generate the optimal torque and position controls for the individual wind turbines. Tier 2 identifies and shares best practices across the wind turbines in the wind farm without having to be dependent upon the Tier 3 core analytics platform (see Figure 3).

Figure 3: Tier 2 Analytics: Optimizing Cohort Performance

Tier 3 is the data lake enabled core analytics platform. The tier 3 core analytics platform includes analytics engines, data sets and data management services (e.g., governance, metadata management, security, authentication) that enable access to the data (sensor data plus other internal and external data sources) and existing analytic models that supports data science analytic/predictive model development and refinement.  Tier 3 aggregates the critical data across all wind farms and individual turbines, and combines the sensor data with external data sources which could include weather (humidity, temperatures, precipitation, air particles, etc.), electricity prices, wind turbine maintenance history, quality scores for the wind turbine manufacturers, and performance profiles of the wind turbine mechanics and technicians (see Figure 4).

Figure 4:  Core Analytics for Analytic Model Development and Refinement

With the rapid increase in storage and processing power at the edges of the Internet of Things (for example, the Dell Edge Gateway 3000 Series), we will see more and more analytic capabilities being pushed to the edge.

How Do You Start Your IoT Journey
While the rapidly evolving expertise on the IoT edge technologies can be very exciting (graphical processing units in gateway servers with embedded machine learning capabilities with 100’s of gigabytes of storage), the starting point for the IoT journey must first address this basic question:

How effective is your organization at leveraging data and analytics to power your business (or operational) models?

We have tweaked the Big Data Business Model Maturity Index to help organizations not only understand where they sit on the maturity index with respect to the above question, but also to provide a roadmap for how organizations can advance up the maturity index to become more effective at leveraging the wealth of IOT data with advanced analytics to power their business and operational models (see Figure 5).

Figure 5:  Big Data / IoT Business Model Maturity IndexMaturity Index

To drive meaningful business impact, you will need to begin with the business and not the technology:

  • Engage the business stakeholders on day one,
  • Align the business and IT teams
  • Understand the organization’s key business and operational initiatives, and
  • Identify and prioritize the use cases (decisions/goals) that support those business initiatives.

If you want to monetize your IOT initiatives, follow those simple guidelines and you will dramatically increase the probability of your business and monetization success.

For more details on the Internet of Things revolution, check out these blogs:

The post Difference between Big Data and Internet of Things appeared first on InFocus Blog | Dell EMC Services.

More Stories By William Schmarzo

Bill Schmarzo, author of “Big Data: Understanding How Data Powers Big Business”, is responsible for setting the strategy and defining the Big Data service line offerings and capabilities for the EMC Global Services organization. As part of Bill’s CTO charter, he is responsible for working with organizations to help them identify where and how to start their big data journeys. He’s written several white papers, avid blogger and is a frequent speaker on the use of Big Data and advanced analytics to power organization’s key business initiatives. He also teaches the “Big Data MBA” at the University of San Francisco School of Management.

Bill has nearly three decades of experience in data warehousing, BI and analytics. Bill authored EMC’s Vision Workshop methodology that links an organization’s strategic business initiatives with their supporting data and analytic requirements, and co-authored with Ralph Kimball a series of articles on analytic applications. Bill has served on The Data Warehouse Institute’s faculty as the head of the analytic applications curriculum.

Previously, Bill was the Vice President of Advertiser Analytics at Yahoo and the Vice President of Analytic Applications at Business Objects.

@ThingsExpo Stories
Buzzword alert: Microservices and IoT at a DevOps conference? What could possibly go wrong? In this Power Panel at DevOps Summit, moderated by Jason Bloomberg, the leading expert on architecting agility for the enterprise and president of Intellyx, panelists peeled away the buzz and discuss the important architectural principles behind implementing IoT solutions for the enterprise. As remote IoT devices and sensors become increasingly intelligent, they become part of our distributed cloud enviro...
SYS-CON Events announced today that Auditwerx will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Auditwerx specializes in SOC 1, SOC 2, and SOC 3 attestation services throughout the U.S. and Canada. As a division of Carr, Riggs & Ingram (CRI), one of the top 20 largest CPA firms nationally, you can expect the resources, skills, and experience of a much larger firm combined with the accessibility and attent...
SYS-CON Events announced today that T-Mobile will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. As America's Un-carrier, T-Mobile US, Inc., is redefining the way consumers and businesses buy wireless services through leading product and service innovation. The Company's advanced nationwide 4G LTE network delivers outstanding wireless experiences to 67.4 million customers who are unwilling to compromise on ...
SYS-CON Events announced today that Infranics will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Since 2000, Infranics has developed SysMaster Suite, which is required for the stable and efficient management of ICT infrastructure. The ICT management solution developed and provided by Infranics continues to add intelligence to the ICT infrastructure through the IMC (Infra Management Cycle) based on mathemat...
SYS-CON Events announced today that Interoute, owner-operator of one of Europe's largest networks and a global cloud services platform, has been named “Bronze Sponsor” of SYS-CON's 20th Cloud Expo, which will take place on June 6-8, 2017 at the Javits Center in New York, New York. Interoute is the owner-operator of one of Europe's largest networks and a global cloud services platform which encompasses 12 data centers, 14 virtual data centers and 31 colocation centers, with connections to 195 add...
SYS-CON Events announced today that Cloudistics, an on-premises cloud computing company, has been named “Bronze Sponsor” of SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Cloudistics delivers a complete public cloud experience with composable on-premises infrastructures to medium and large enterprises. Its software-defined technology natively converges network, storage, compute, virtualization, and management into a ...
In his session at @ThingsExpo, Eric Lachapelle, CEO of the Professional Evaluation and Certification Board (PECB), will provide an overview of various initiatives to certifiy the security of connected devices and future trends in ensuring public trust of IoT. Eric Lachapelle is the Chief Executive Officer of the Professional Evaluation and Certification Board (PECB), an international certification body. His role is to help companies and individuals to achieve professional, accredited and worldw...
In his General Session at 16th Cloud Expo, David Shacochis, host of The Hybrid IT Files podcast and Vice President at CenturyLink, investigated three key trends of the “gigabit economy" though the story of a Fortune 500 communications company in transformation. Narrating how multi-modal hybrid IT, service automation, and agile delivery all intersect, he will cover the role of storytelling and empathy in achieving strategic alignment between the enterprise and its information technology.
Microservices are a very exciting architectural approach that many organizations are looking to as a way to accelerate innovation. Microservices promise to allow teams to move away from monolithic "ball of mud" systems, but the reality is that, in the vast majority of organizations, different projects and technologies will continue to be developed at different speeds. How to handle the dependencies between these disparate systems with different iteration cycles? Consider the "canoncial problem" ...
The Internet of Things is clearly many things: data collection and analytics, wearables, Smart Grids and Smart Cities, the Industrial Internet, and more. Cool platforms like Arduino, Raspberry Pi, Intel's Galileo and Edison, and a diverse world of sensors are making the IoT a great toy box for developers in all these areas. In this Power Panel at @ThingsExpo, moderated by Conference Chair Roger Strukhoff, panelists discussed what things are the most important, which will have the most profound e...
Keeping pace with advancements in software delivery processes and tooling is taxing even for the most proficient organizations. Point tools, platforms, open source and the increasing adoption of private and public cloud services requires strong engineering rigor - all in the face of developer demands to use the tools of choice. As Agile has settled in as a mainstream practice, now DevOps has emerged as the next wave to improve software delivery speed and output. To make DevOps work, organization...
My team embarked on building a data lake for our sales and marketing data to better understand customer journeys. This required building a hybrid data pipeline to connect our cloud CRM with the new Hadoop Data Lake. One challenge is that IT was not in a position to provide support until we proved value and marketing did not have the experience, so we embarked on the journey ourselves within the product marketing team for our line of business within Progress. In his session at @BigDataExpo, Sum...
Web Real-Time Communication APIs have quickly revolutionized what browsers are capable of. In addition to video and audio streams, we can now bi-directionally send arbitrary data over WebRTC's PeerConnection Data Channels. With the advent of Progressive Web Apps and new hardware APIs such as WebBluetooh and WebUSB, we can finally enable users to stitch together the Internet of Things directly from their browsers while communicating privately and securely in a decentralized way.
DevOps is often described as a combination of technology and culture. Without both, DevOps isn't complete. However, applying the culture to outdated technology is a recipe for disaster; as response times grow and connections between teams are delayed by technology, the culture will die. A Nutanix Enterprise Cloud has many benefits that provide the needed base for a true DevOps paradigm.
What sort of WebRTC based applications can we expect to see over the next year and beyond? One way to predict development trends is to see what sorts of applications startups are building. In his session at @ThingsExpo, Arin Sime, founder of WebRTC.ventures, will discuss the current and likely future trends in WebRTC application development based on real requests for custom applications from real customers, as well as other public sources of information,
In his General Session at 17th Cloud Expo, Bruce Swann, Senior Product Marketing Manager for Adobe Campaign, explored the key ingredients of cross-channel marketing in a digital world. Learn how the Adobe Marketing Cloud can help marketers embrace opportunities for personalized, relevant and real-time customer engagement across offline (direct mail, point of sale, call center) and digital (email, website, SMS, mobile apps, social networks, connected objects).
With the introduction of IoT and Smart Living in every aspect of our lives, one question has become relevant: What are the security implications? To answer this, first we have to look and explore the security models of the technologies that IoT is founded upon. In his session at @ThingsExpo, Nevi Kaja, a Research Engineer at Ford Motor Company, will discuss some of the security challenges of the IoT infrastructure and relate how these aspects impact Smart Living. The material will be delivered i...
"My role is working with customers, helping them go through this digital transformation. I spend a lot of time talking to banks, big industries, manufacturers working through how they are integrating and transforming their IT platforms and moving them forward," explained William Morrish, General Manager Product Sales at Interoute, in this SYS-CON.tv interview at 18th Cloud Expo, held June 7-9, 2016, at the Javits Center in New York City, NY.
SYS-CON Events announced today that Ocean9will exhibit at SYS-CON's 20th International Cloud Expo®, which will take place on June 6-8, 2017, at the Javits Center in New York City, NY. Ocean9 provides cloud services for Backup, Disaster Recovery (DRaaS) and instant Innovation, and redefines enterprise infrastructure with its cloud native subscription offerings for mission critical SAP workloads.
Your homes and cars can be automated and self-serviced. Why can't your storage? From simply asking questions to analyze and troubleshoot your infrastructure, to provisioning storage with snapshots, recovery and replication, your wildest sci-fi dream has come true. In his session at @DevOpsSummit at 20th Cloud Expo, Dan Florea, Director of Product Management at Tintri, will provide a ChatOps demo where you can talk to your storage and manage it from anywhere, through Slack and similar services ...