Apache 2.0 - Towards Converged Data Pla orm

Apache®,ApacheIgnite,Ignite®,andtheApacheIgnitelogoareeitherregisteredtrademarksortrademarksoftheApacheSo8wareFounda<onintheUnitedStatesand/orothercountries.

NIKITAIVANOVGridGainFounder&CTO

ApacheIgnitePMC

Apache2.0-TowardsConvergedDataPla>ormFastDataMeetsOpenSource

hIp://ignite.apache.org @apacheignite


Agenda

•  FastDatavsBigData–  In-MemoryDatabases–  In-MemoryDataGrids–  Hadoop&Spark

•  ConvergedDataPla>orm•  BigData+FastData

•  WhatisApacheIgnite–  BigBankUseCase–  In-MemoryDataFabric–  SharedMemoryLayer•  ShareSparkRDDs•  In-MemoryFileSystem

•  Q&A


•  VeryAc<veCommunity•  GreatWaytoLearnDistributedCompu<ng•  HowToContribute:

–  hIps://ignite.apache.org/community/

contribute.html#contribute

–  hIps://cwiki.apache.org/confluence/display/IGNITE/How+to+Contribute

ApacheIgnite:JoinUs!


•  BigData–  OLAPmostly–  LargerHistoricalDataSet–  Read-Mostly–  ThroughputNotImportant–  LowQueryLatencies–  Good-enoughforinterac<ve

analy<cs

FastDatavsBigData

•  FastData–  OLTPmostly–  SmallerOpera<onalDataSet–  HighThroughput(ops/sec)–  LowLatencies–  ConsistentandTransac<onal


•  BigData–  Hadoop•  MapReduce•  HDFS•  HBase

–  Spark•  MachineLearning•  GraphProcessing•  SQL

–  Warehouse/DBVendors

FastDatavsBigData

•  FastData–  Streaming•  Flink•  Ka^a•  Apex

–  In-MemoryDataGrid•  Ignite•  Geode(incuba<ng)

–  In-MemoryDatabase•  MemSQL•  VoltDB

–  NoSQL•  MongoDB•  Cassandra


•  In-MemoryDatabases–  MemSQL•  ClosedSource•  FreeLimitedCommunityEdi<on

–  VoltDB•  OpenSourceCommunityEdi<on(AGPL)•  ClosedSourceEnterpriseEdi<on

•  MainFeatures–  High-Throughput–  LowLatencies–  FullSQLSupport•  However,SQListheonlyAPI

–  DiskPersistence•  Diskisjustacopyofmemory

–  Completereplacementofexis<ngdatabases

FastData:In-MemoryDatabases


•  In-MemoryDataGrids–  ApacheIgnite–In-MemoryDataFabric–  ApacheGeode(incuba<ng)–  Hazelcast

•  MainFeatures–  Highthroughput–  Lowlatencies–  Key-valuestore–  Transac<ons–  Extensivedataqueryingcapability–  Diskpersistence•  Read&write-throughtodatabases•  Keepyourexis<ngdatabase

FastData:In-MemoryDataGrids


•  ApacheHadoop&ApacheSpark–  BigFamilyofProducts–  BatchProcessing–  In-MemoryProcessing(Spark)

•  MainFeatures–  Disk-basedstorage–  Interac<veAnaly<cs–  NoTransac*ons–  Read-OnlyDataSets–  StrongQueryingCapabili<es–  Rela<velyLowLatencies•  Goodenoughforhumaneye

BigDataEcosystem


•  BigData+  AddSharedMemoryStore+  AddTransac<ons

HowToBridgeTheGap?

•  FastData+  AddDisk-FirstDataSets+  AddDisk-FirstProcessing


•  FastData+BigData•  DistributedandScalable•  RealTimeData-To-Ac<on•  HybridTransac<onalandAnaly<calProcessing(HTAP)–  FastDatainMemory–  BigDataonDisk–  CombineRAM,NAND,HDD–  NoETL–  Queryhistoricalandanaly<caldata–  Transac<onsonhistoricalandanaly<caldata

WhatisaConvergedDataPla>orm?


ApacheIgniteTMIn-MemoryDataFabric:StrategicApproachtoIMC

•  Supports Applications of various types and languages

• Open Source – Apache 2.0 • Simple Java APIs • 1 JAR Dependency • High Performance & Scale • Automatic Fault Tolerance • Management/Monitoring • Runs on Commodity Hardware

• Supports existing & new data sources • No need to rip & replace


ApacheIgniteIn-MemoryDataFabric

©2014GridGainSystems,Inc.

UseCase:Largest bank in Russia and Eastern Europe, and the third largest in Europe

•  SberbankRequirements–  Migratetodatagridarchitecture–  MinimizedependencyonOracle–  Movetoopensource

•  WhyApacheIgnite–  MorethanaDataGrid–  Bestperformance

•  10+compe<torsevaluated–  Demonstratedbest

•  Faulttolerance&scalability•  ANSI-99SQLSupport•  Transac<onalconsistency

•  JointlyDeveloping•  Disk-OnlyDataSets•  QueryDisk&MemoryTogether

130

Mill

ion

Cust

omer

s

Deposit Withdrawal Statement

Disk Store

Disk Store

Disk Store

1000+ Servers

GridGain Security

Deposit Withdrawal Statement


•  BasedonJCache(JSR107)–  In-MemoryKey-ValueStore–  BasicCacheOpera<ons–  ConcurrentMapAPIs–  CollocatedProcessing(EntryProcessor)–  EventsandMetrics–  PluggablePersistence

•  IgniteDataGrid–  ACIDTransac<ons–  SQLQueries(ANSI99)–  In-MemoryIndexes–  On-Heap&Off-HeapMemory–  Automa<cRDBMSIntegra<on

ApacheIgniteDataGrid


DataGrid:DistributedCaching

Par<<onedCache ReplicatedCache


•  ANSI-99SQL•  AlwaysConsistent•  FaultTolerant•  In-MemoryIndexes(On-HeapandOff-Heap)•  Automa<cGroupBy,Aggrega<ons,Sor<ng•  Cross-CacheJoins,Unions,etc.•  Ad-HocSQLSupport

DataGrid:Ad-HocSQL(ANSI99)


SQLCross-CacheGROUPBYExample


•  IgniteRDDDeploymentModes–  ShareRDDacrosstasksonthehost–  ShareRDDacrosstasksintheapplica<on–  ShareRDDglobally–  EmbeddedvsExternalDeployments

•  FasterSQL–  In-MemoryIndexes–  SQLontopofSharedRDD

ShareRDDsAcrossSparkJobs


•  IgniteIn-MemoryFileSystem(IGFS)–  Hadoop-compliant–  EasytoInstall–  On-HeapandOff-Heap–  CachingLayerforHDFS–  Write-throughandRead-throughHDFS–  PerformanceBoost

IgniteIn-MemoryFileSystem


IgniteIn-MemoryMapReduce•  In-MemoryNa<ve

Performance•  ZeroCodeChange•  Useexis<ngMRcode•  Useexis<ngHivequeries•  NoNameNode•  NoNetworkNoise•  In-ProcessDataColoca<on•  EagerPushScheduling


•  MoreSQL–  Non-collocatedJoins–  100%DataModifica<onLanguage(DML)–  100%DataDefini<onLanguage(DDL)

•  MoreDisk–  ATMM-AdvancedTiered-MemoryModel:•  Disk-firstdatasets•  AnyDRAM/NAND/HDDmix

–  SeamlessqueryingacrossATMM

ProposedApacheIgnite2.0RoadmapConvergedDataPla>orm


ANYQUESTIONS?Thankyouforjoiningus.Followtheconversa<on.

hIp://www.ignite.apache.org

@apacheignite

Apache 2.0 - Towards Converged Data Pla orm

Documents

Transcript of Apache 2.0 - Towards Converged Data Pla orm