3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching...
Transcript of 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching...
![Page 1: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/1.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
MEMORY-CENTRIC ARCHITECTUREIN-MEMORY PERFORMANCE, DURABILITY OF DISK
Dmitriy SetrakyanFounder, Chief Product Officer, Apache Ignite PMC
![Page 2: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/2.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Agenda
• In-Memory Computing Evolution
• Distributed Caches
• In-Memory Data Grids
• In-Memory Databases
• Distributed Databases
• Memory-Centric Databases
• Data Storage Ecosystem
![Page 3: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/3.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Distributed Caches
![Page 4: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/4.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Cache is a hardware or software component that stores data so future requestsfor that data can be served faster
![Page 5: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/5.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Local Caching
• Application In-Process Caching
– Querying Results
– Most Frequently Used Data
• Browser Caching
• Benefits
– Speed up Applications!
– Network is slow!
![Page 6: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/6.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Distributed Caches
• Share-nothing (consistent hashing or sharding)
• Simple client protocol
• Memcached, Redis, AWS ElastiCache
• Benefits
– Shared cache
– Beyond local RAM capacity
– Fault tolerant and scalable
![Page 7: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/7.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Shortcomings of Distributed Caches
• Lack of queries (SQL or JOINs)
• Lack of ACID transactions
• Lack of collocated processing
• Lack of database synchronization
– Cache-aside pattern
• Lack of native persistence
– cannot persist more than fits in RAM
• RAM warm-up on restart
![Page 8: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/8.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
In-Memory Data Grids
![Page 9: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/9.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
In-Memory Data Grids
• Intelligent Distributed Caches
• Hazelcast, GigaSpaces, Apache Ignite
• Benefits
– ACID Transactions
– Query APIs
– Event notifications
– Continuous queries
– Database Read/Write-Through
– Collocated Processing
![Page 10: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/10.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Client-Server Processing
1. Initial Request2. Fetch data from remote nodes3. Process the entire data-set
1. Initial request2. Co-locate processing with data3. Reduce multiple results into one
Co-Located Processing
![Page 11: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/11.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Shortcomings of In-Memory Data Grids
• Custom Query Languages
– No SQL, No JOINs
• Lack of native persistence
– cannot persist more than fits in RAM
• RAM warm-up on restart
![Page 12: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/12.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
In-Memory Databases
![Page 13: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/13.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
In-Memory Databases
• Primary in-RAM Storage
• In-Memory Data Grids for SQL
• VoltDB, SAP Hana
• Benefits
– Distributed SQL
– ACID Transactions
– Better Scalability
![Page 14: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/14.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Collocated Joins
1. Initial Query2. Query execution over local data3. Reduce multiple results in one
Ignite Node
Canada
Toronto
OttawaMontreal
Calgary
Ignite Node
IndiaMumbai
New Delhi
1
2
23
![Page 15: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/15.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Shortcomings of In-Memory Databases
• Rip-n-Replace existing database
– cannot keep or reuse existing database
• Lack of collocated processing
• Lack of native persistence
– cannot scale beyond in RAM
– basic memory-offload on disk
– RAM warm-up on restart
– not a system of record
![Page 16: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/16.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Distributed Databases
![Page 17: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/17.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Distributed NoSQL Databases
• Cassandra, Mongo
– Distributed persistence
– Fault tolerant and scalable
• Shortcomings
– Lack of SQL (no JOINs)
– Lack of distributed transactions
– Lack of in-memory processing
![Page 18: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/18.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Distributed SQL Databases
• CockroachDB, Cloud Spanner
– Cloud-native
– Fault tolerant and scalable
– SQL (distributed JOINS)
• Shortcomings
– Lack of in-memory processing
– Lack of key-value API
– Lack of event notifications
![Page 19: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/19.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Memory-Centric Databases
![Page 20: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/20.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Ignite Memory-Centric Architecture
![Page 21: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/21.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Memory Centric Storage
![Page 22: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/22.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Memory & Disk Utilization
Mode Description Major Advantage
In-Memory Pure In-Memory Storage Maximum perfomance possible(data is never written to disk)
In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc
Horizontal scalabilityFaster reads and writes
In-Memory + Full Copy on Disk The whole data set is stored both in memory and on disk Survives cluster failures
100% on Disk + In-Memory Cache 100% of data is in Ignite native persistence anda subset is in memory
Unlimited data scale beyond RAMCollocated memory and storage
![Page 23: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/23.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Supports HTAP Principles
• One Platform, Multiple Workloads
– OLTP and OLAP
– Real-Time Streaming and Batching
– No ETL
• Distributed Everything
– Scalability
– No Single Point of Failure
– SQL, Calculations, ML, etc.
– Cloud Native
![Page 24: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/24.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Data Storage EcosystemFeature RDBMS NoSQL IMDG IMDB GridGain and Ignite
Scale Out X ✓ ✓ ✓ ✓Availability X ✓ ✓ ✓ ✓Consistency ✓ X ✓ ✓ ✓In-Memory ✓ X ✓ ✓ ✓Persistence ✓ ✓ X X ✓SQL ✓ X X ✓ ✓Key-Value X ✓ ✓ X ✓Collocated Processing X X ✓ X ✓
![Page 25: 3 - Dmitriy Setrakyan IMCS 6-18...(data is never written to disk) In-Memory + 3rd Party DB Caching layer (aka. in-memory data grid) for existing databases – RDBMS, NoSQL, etc Horizontal](https://reader033.fdocuments.net/reader033/viewer/2022042407/5f20fcbe2a594b794b196b92/html5/thumbnails/25.jpg)
© 2018 GridGain Systems, Inc. GridGain Company Confidential
Follow the conversation.https://www.gridgain.comhttps://ignite.apache.org
Any Questions?
#apacheignite#gridgain#dsetrakyan