PowerStream: Propelling Energy Innovation with Predictive Analytics
Propelling IoT Innovation with Predictive Analytics
-
Upload
memsql -
Category
Technology
-
view
232 -
download
3
Transcript of Propelling IoT Innovation with Predictive Analytics
![Page 1: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/1.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 2: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/2.jpg)
Topics• The On-Demand Economy
• From In-Memory Compu8ng to In-Memory Databases
• Renewable Energy and PowerStream
• Demo and Q&A
(c) Nikita Shamgunov and MemSQL
![Page 3: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/3.jpg)
The On-DemandEconomy
(c) Nikita Shamgunov and MemSQL
![Page 4: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/4.jpg)
We live in an on-demand economy
(c) Nikita Shamgunov and MemSQL
![Page 5: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/5.jpg)
Consumers are condi.oned to instant services, like Uber, Stripe, and Airbnb
(c) Nikita Shamgunov and MemSQL
![Page 6: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/6.jpg)
Where does that leave enterprises?
(c) Nikita Shamgunov and MemSQL
![Page 7: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/7.jpg)
Racing to meet internal and external expecta1ons for speed and
personaliza,on
(c) Nikita Shamgunov and MemSQL
![Page 8: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/8.jpg)
Batch processing is the enterprise enemy
(c) Nikita Shamgunov and MemSQL
![Page 9: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/9.jpg)
Enterprises must move from overnight to
Real-&me, intra-day opera&ons
(c) Nikita Shamgunov and MemSQL
![Page 10: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/10.jpg)
Harnessing data in real -me
(c) Nikita Shamgunov and MemSQL
![Page 11: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/11.jpg)
Achieving sub 100 ms latency• Real-'me monitoring and analy'cs on streaming video
• Proac'vely diagnose issues in real-'me
• Deliver be9er viewer experience
(c) Nikita Shamgunov and MemSQL
![Page 12: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/12.jpg)
Massive Ingest AND Analy1cs• Instant accuracy to the latest repin
• Build real-5me analy5c applica5ons
• 1 GB/sec totaling 72 TB/day
(c) Nikita Shamgunov and MemSQL
![Page 13: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/13.jpg)
What is anin-memory database?(c) Nikita Shamgunov and MemSQL
![Page 14: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/14.jpg)
In-Memory Databases...• Use memory instead of disk
(c) Nikita Shamgunov and MemSQL
![Page 15: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/15.jpg)
In-Memory Databases...• Use memory instead of disk
(c) Nikita Shamgunov and MemSQL
![Page 16: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/16.jpg)
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
(c) Nikita Shamgunov and MemSQL
![Page 17: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/17.jpg)
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
(c) Nikita Shamgunov and MemSQL
![Page 18: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/18.jpg)
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
• Put the whole dataset in memory
(c) Nikita Shamgunov and MemSQL
![Page 19: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/19.jpg)
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
• Put the whole dataset in memory
(c) Nikita Shamgunov and MemSQL
![Page 20: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/20.jpg)
In-Memory Databases...• Use memory instead of disk
• Do not (need to) save data on disk
• Put the whole dataset in memory
Well, some)mes...
(c) Nikita Shamgunov and MemSQL
![Page 21: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/21.jpg)
Wikipedia says...
In-memory databases primarily rely on main-memory for storage.
(c) Nikita Shamgunov and MemSQL
![Page 22: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/22.jpg)
In-Memory Databases• Are durable to disk (and respect ACID)
(c) Nikita Shamgunov and MemSQL
![Page 23: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/23.jpg)
In-Memory Databases• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
(c) Nikita Shamgunov and MemSQL
![Page 24: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/24.jpg)
In-Memory Databases• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
• Tradeoffs are suited to systems with lots of memory
(c) Nikita Shamgunov and MemSQL
![Page 25: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/25.jpg)
In-Memory Databases• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
• Tradeoffs are suited to systems with lots of memory
• Tend to be distributed systems
(c) Nikita Shamgunov and MemSQL
![Page 26: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/26.jpg)
In-Memory Databases• Are durable to disk (and respect ACID)
• Can spill on disk or pin data in-memory (and take advantage of it)
• Tradeoffs are suited to systems with lots of memory
• Tend to be distributed systems
• Have a different set of boClenecks
(c) Nikita Shamgunov and MemSQL
![Page 27: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/27.jpg)
Bold Claim(c) Nikita Shamgunov and MemSQL
![Page 28: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/28.jpg)
All database workloads will be running on in-memory databases
(c) Nikita Shamgunov and MemSQL
![Page 29: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/29.jpg)
Why?• Memory is ge,ng cheaper (about 40% every year)
(c) Nikita Shamgunov and MemSQL
![Page 30: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/30.jpg)
Why?• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new tape, etc)
(c) Nikita Shamgunov and MemSQL
![Page 31: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/31.jpg)
Why?• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new tape, etc)
• In-memory databases leverage SSD (no random writes)
(c) Nikita Shamgunov and MemSQL
![Page 32: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/32.jpg)
Why?• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new tape, etc)
• In-memory databases leverage SSD (no random writes)
• NVRAM is coming (and could be cheaper than SSD)
(c) Nikita Shamgunov and MemSQL
![Page 33: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/33.jpg)
Why?• Memory is ge,ng cheaper (about 40% every year)
• Cache is the new RAM (RAM is the new disk, disk is the new tape, etc)
• In-memory databases leverage SSD (no random writes)
• NVRAM is coming (and could be cheaper than SSD)
In-memory databases are tuned to modern hardware and modern workloads
(c) Nikita Shamgunov and MemSQL
![Page 34: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/34.jpg)
Renewable Energyin the News
(c) Nikita Shamgunov and MemSQL
![Page 35: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/35.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 36: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/36.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 37: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/37.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 38: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/38.jpg)
IntroducingPowerStream(c) Nikita Shamgunov and MemSQL
![Page 39: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/39.jpg)
MemSQL PowerStreamPredic'ng the global health of wind turbines
(c) Nikita Shamgunov and MemSQL
![Page 40: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/40.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 41: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/41.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 42: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/42.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 43: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/43.jpg)
Streamliner Architecture for PowerStream
(c) Nikita Shamgunov and MemSQL
![Page 44: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/44.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 45: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/45.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 46: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/46.jpg)
LiveDemo
(c) Nikita Shamgunov and MemSQL
![Page 47: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/47.jpg)
Demo Sequence• Powerstream user interface
• Showcase largest windfarms
• Real-8me simula8ons
• Witness live opera8ons
• Ease of new pipeline setup
• Ka>a subscrip8on
(c) Nikita Shamgunov and MemSQL
![Page 48: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/48.jpg)
Enabling predic.ve analy.cs• Use exis(ng models from SAS
• Create models in Spark MLlib
• Predic(ve scoring as part of the pipeline
(c) Nikita Shamgunov and MemSQL
![Page 49: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/49.jpg)
From a Real-Time Dashboard to Predic5ve Applica5ons
(c) Nikita Shamgunov and MemSQL
![Page 50: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/50.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 51: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/51.jpg)
(c) Nikita Shamgunov and MemSQL
![Page 52: Propelling IoT Innovation with Predictive Analytics](https://reader031.fdocuments.net/reader031/viewer/2022020314/587c0f931a28ab03768b64cb/html5/thumbnails/52.jpg)
(c) Nikita Shamgunov and MemSQL