Your car is no longer just a transportation device. It is an internet endpoint. As is your toaster, your microwave and your refrigator. And don't even get me started on your television; it was already gathering and transmitting information before the term 'IoT' was coined. IoT, when combined with a streaming data processing pipeline, will change the way we look at everyday objects.
Where does BigData Fit in ?
There’s a perception out there that BigData is only for BIG companies, the Googles and Amazons of the world. Companies that need to store terabytes, petabytes and exabytes of data. Typical examples of BigData analysis include:
- The NSA, monitoring millions of voice and data calls every second of every day.
- Google, analyzing, in real time, everything that everyone is searching for across the world.
- Facebook, trying to recognize a face among a billion other faces.
Sure, all of the above falls under the BigData’s data crunching expertise and portfolio. However, those aren’t the only companies that can benefit from BigData.Small to midsized companies can also make a killing by using BigData – provided they ask the right questions!
Asking the Right Question - Where can I fit in IoT into my business process model?
Enter IoT, and a 2 Trillion Dollar Opportunity
Anything that MOVES is an IoT opportunity !
Vehicles can produce upwards of 560 GB data per vehicle, per day. The same can be said of almost any device with electronic components. What about things without electronic components?What about your shower, which has no electronic components? It is still a potential IoT device, with a market just waiting to be exploited. Hotels, for instance, get an annoyingly large number of complaints about shower malfunctions. Wouldn't it be nice if the shower itself informed a backend system of an issue, before a customer did?
Anything that ‘moves’ – whether it is your heart, your pulse, or the water through your showerhead - are all IoT data generating candidates.
This has major consequences when you consider the items that move along manufacturing lines.
Vehicle and Appliance Breakdowns
Say, for example, you are Toyota or Honda and are sell a few hundred million cars every year, worldwide. Now, say you want to get crash data from all the Toyotas that get into crashes, on a daily interval; i.e. Toyota would like immediate notification along with a ‘crash dump’ of all on-board data. This is a problem that BigData (combined with IoT) is already solving.This IoT plus BigData lethal combo can applied as easily to home appliance breakdowns or power blackouts or even termite infestations as to car crashes.Any product, be it a pill or a pill container – be it a soda can or potato chips - if it spends any amount of time on a manufacturing line, is a candidate for IoT and BigData.
And that is why, it is possible, that most companies who think BigData isn’t for them, are not asking the right questions. IoT combined with BigData is opening up opportunities, where none existed.
The IoT value add to all industries has been estimated by Gartner, to be a staggering $1.9 Trillion.
Google's Pixel Buds, Douglas Adams' BabelFish is a Reality!
While I was working on this article, Google announced the release of an IoT device that has AI built into it. It has a built-in natural language processor, which allows it to translate upto 40 languages on the fly! Think about that! It IS actually straight out of science fiction (all of us Hitchhiker's Guide to the Galaxy fans know what I am talking about). Douglas Adams, with his wild imagination, had conjured up a small fish (appropriately named the BabelFish), which, when inserted into one's ear (painfully), translated any alien language. If only Adams was alive today; Google's PixelBuds would have brought tears of joy to his eyes.
Sound Waves Move As Well !
Of course, the spoken language is nothing but a vibration of air; in other words, movement of air molecules. And movement, any movement, is an IoT opportunity. Google has shown the world what can be done when you combine existing NLP technologies with IoT; all of a sudden, the world has shrunk. One can travel to foreign lands without fear of a 'language barrier'.
Which BigData Engine is winning - Hadoop or Spark?
Large scale data processing engines come in two flavors – Hadoop and Spark. Hadoop is the industry standard. Created by Apache, it was the defacto implementation of the famous MapReduce search algorithm, developed at Google.Spark, though a newer entry, brought real-time data processing capability to BigData. This was to overcome MapReduce’s restricted, disk-bound, batch (no real time) processing engine.Spark, however, can run in Hadoop clusters through YARN (Yet Another Resource Negotiator), but can also run in a standalone mode. Big data practitioners expect Spark to spin-off, and perhaps replace Hadoop, especially in instances where faster access to processed data is critical.
Given that most IoT devices transmit tons of streaming data, Spark may emerge as the leader over traditional Hadoop.
Who is leading the BigData (Platforms and Products) War?
Of course, it is hard to predict, however, these companies, in my opinion have distinguished themselves through innovative, development time saving product offerings.
- Google - Google DataProc, Google TensorFlow and ML. As Google displayed with the new PixelBuds, Pixel Phone and PixelBook, their Tensorflow libraries - including natural language processing, Image Recognition and other ML technologies - are clearly a force to contend with. Google's DataProc engine, in addition, makes it relatively painless to run Hadoop / Spark as a Service. Outsourcing the big pain points in BigData to a PaaS can only be a good thing!
- Cloudera - Cloudera QuickStart VM and Cloudera's Hadoop Cluster are both huge time savers for companies looking to get up and running quickly.
- AWS / Azure - both have mature BigData ecosystems that leverage their own, individual strengths.
Who will win the IoT war?
There are more IoT startups than you can throw a brick at. Some have distinguished themselves, by endearing themselves to the developer community.
- Current Leaders - Arduino, Raspberry Pi
- Notables - Autonomo (with a Solar Panel and Long Battery Life), Wio Link
Summary
Companies that think BigData isn't for them, aren't asking the right questions. There isn't a single industry I can think of that will not be revolutionized by this coming Tsunami. As I was writing this post, Michael Dell announced a $1 Billion investmentinto IoT R and D. That's quite a powerful endorsement from a tech leader!Thoughts? Comments? BigData + IoT Success Stories? Talk to me, I'm listening..... Your car is no longer just a transportation device. It is an internet endpoint. As is your toaster, your microwave and your refrigator. And don't even get me started on your television; it was already gathering and transmitting information before the term 'IoT' was coined. IoT, when combined with a streaming data processing pipeline, will change the way we look at everyday objects.
Where does BigData Fit in ?
There’s a perception out there that BigData is only for BIG companies, the Googles and Amazons of the world. Companies that need to store terabytes, petabytes and exabytes of data. Typical examples of BigData analysis include:
- The NSA, monitoring millions of voice and data calls every second of every day.
- Google, analyzing, in real time, everything that everyone is searching for across the world.
- Facebook, trying to recognize a face among a billion other faces.
Sure, all of the above falls under the BigData’s data crunching expertise and portfolio. However, those aren’t the only companies that can benefit from BigData.Small to midsized companies can also make a killing by using BigData – provided they ask the right questions!
Asking the Right Question - Where can I fit in IoT into my business process model?
Enter IoT, and a 2 Trillion Dollar Opportunity
Anything that MOVES is an IoT opportunity !
Vehicles can produce upwards of 560 GB data per vehicle, per day. The same can be said of almost any device with electronic components. What about things without electronic components?What about your shower, which has no electronic components? It is still a potential IoT device, with a market just waiting to be exploited. Hotels, for instance, get an annoyingly large number of complaints about shower malfunctions. Wouldn't it be nice if the shower itself informed a backend system of an issue, before a customer did?
Anything that ‘moves’ – whether it is your heart, your pulse, or the water through your showerhead - are all IoT data generating candidates.
This has major consequences when you consider the items that move along manufacturing lines.
Vehicle and Appliance Breakdowns
Say, for example, you are Toyota or Honda and are sell a few hundred million cars every year, worldwide. Now, say you want to get crash data from all the Toyotas that get into crashes, on a daily interval; i.e. Toyota would like immediate notification along with a ‘crash dump’ of all on-board data. This is a problem that BigData (combined with IoT) is already solving.This IoT plus BigData lethal combo can applied as easily to home appliance breakdowns or power blackouts or even termite infestations as to car crashes.Any product, be it a pill or a pill container – be it a soda can or potato chips - if it spends any amount of time on a manufacturing line, is a candidate for IoT and BigData.
And that is why, it is possible, that most companies who think BigData isn’t for them, are not asking the right questions. IoT combined with BigData is opening up opportunities, where none existed.
The IoT value add to all industries has been estimated by Gartner, to be a staggering $1.9 Trillion.
Google's Pixel Buds, Douglas Adams' BabelFish is a Reality!
While I was working on this article, Google announced the release of an IoT device that has AI built into it. It has a built-in natural language processor, which allows it to translate upto 40 languages on the fly! Think about that! It IS actually straight out of science fiction (all of us Hitchhiker's Guide to the Galaxy fans know what I am talking about). Douglas Adams, with his wild imagination, had conjured up a small fish (appropriately named the BabelFish), which, when inserted into one's ear (painfully), translated any alien language. If only Adams was alive today; Google's PixelBuds would have brought tears of joy to his eyes.
Sound Waves Move As Well !
Of course, the spoken language is nothing but a vibration of air; in other words, movement of air molecules. And movement, any movement, is an IoT opportunity. Google has shown the world what can be done when you combine existing NLP technologies with IoT; all of a sudden, the world has shrunk. One can travel to foreign lands without fear of a 'language barrier'.
Which BigData Engine is winning - Hadoop or Spark?
Large scale data processing engines come in two flavors – Hadoop and Spark. Hadoop is the industry standard. Created by Apache, it was the defacto implementation of the famous MapReduce search algorithm, developed at Google.Spark, though a newer entry, brought real-time data processing capability to BigData. This was to overcome MapReduce’s restricted, disk-bound, batch (no real time) processing engine.Spark, however, can run in Hadoop clusters through YARN (Yet Another Resource Negotiator), but can also run in a standalone mode. Big data practitioners expect Spark to spin-off, and perhaps replace Hadoop, especially in instances where faster access to processed data is critical.
Given that most IoT devices transmit tons of streaming data, Spark may emerge as the leader over traditional Hadoop.
Who is leading the BigData (Platforms and Products) War?
Of course, it is hard to predict, however, these companies, in my opinion have distinguished themselves through innovative, development time saving product offerings.
- Google - Google DataProc, Google TensorFlow and ML. As Google displayed with the new PixelBuds, Pixel Phone and PixelBook, their Tensorflow libraries - including natural language processing, Image Recognition and other ML technologies - are clearly a force to contend with. Google's DataProc engine, in addition, makes it relatively painless to run Hadoop / Spark as a Service. Outsourcing the big pain points in BigData to a PaaS can only be a good thing!
- Cloudera - Cloudera QuickStart VM and Cloudera's Hadoop Cluster are both huge time savers for companies looking to get up and running quickly.
- AWS / Azure - both have mature BigData ecosystems that leverage their own, individual strengths.
Who will win the IoT war?
There are more IoT startups than you can throw a brick at. Some have distinguished themselves, by endearing themselves to the developer community.
- Current Leaders - Arduino, Raspberry Pi
- Notables - Autonomo (with a Solar Panel and Long Battery Life), Wio Link
Summary
Companies that think BigData isn't for them, aren't asking the right questions. There isn't a single industry I can think of that will not be revolutionized by this coming Tsunami. As I was writing this post, Michael Dell announced a $1 Billion investmentinto IoT R and D. That's quite a powerful endorsement from a tech leader! Thoughts? Comments? BigData + IoT Success Stories? Talk to me, I'm listening.....