NVM & Implications on Data Infratsructure
-
Upload
imcpune -
Category
Technology
-
view
151 -
download
0
Transcript of NVM & Implications on Data Infratsructure
Non Volatile Memory & Implications on Data Infrastructure
Robert Geiger and Shankar Hundekar, Ampool Inc.
Memory and Storage Directions
Robert Geiger, Chief Architect and VP Engineering, Ampool Inc.
PreviousStorageClassMemory(SCM)
• SSDand“keepthepoweron”solutions
• SATASSD;NANDFlash§ NOTE:NORflashisbyteaddressablebutmoreexpensive(duetolowerdensity)
• NVMe SSD;NANDFlash(veryfast;2GBytes/seconnewMaclaptop)
• BatterybackedDRAM(replacementanissue)
• NVDIMM-F(blockdevice;highcapacity,high latency)– PCIe
• NVDIMM-N(byteaddressableatornearDRAMspeeds)– PCIe
EmergingStorageClassMemory(SCM)
• ActuallyStorageClassByteAddressable Memory
• NearDRAMlatencyandthroughput
• Basedononeofseveraltypesofmemorytechnology• MRAM(magnetic)• ReRAM (resistive)• FRAM(ferroelectric),PCM(phasechange)• XPoint (Intel/Micronwon’ttellyouexactly)
• 3-DXPoint (Intel-Micron)- DDRorPCIe
SCMintheMemory/StorageHierarchy
SCMInterfaces
• Filesystembased(standard&memorymapped)• Filesystemsgetting reallyfast
§ TestanewMaclaptopw/SSD- youcando2GBytes/secsustainedread/write• Linux(fedoracore/RedHat)andMSaddingSCMsupport in‘16• Capacitywillstillpushbigdatatodiskfordatalake,colddata• Whatdoesthismeanforin-memoryplays?
§ Beingin-memory isnolongerenough§ Sophisticationmustincrease
• PMEM.IO=>DirectaccesstoSCM• Mostlikelyinterfaceforin-memoryfocusedsolutions
In-MemoryFuture
• Linebetweenmemoryandstorageisblurring
• StorageisjustmemorywithaspecificQualityofService(QoS)
• Managingmultiplelayers/typesofmemorywilldifferentiate
• Nolongerin-memoryvs.disk
• Nowfilesystemvs.managedbyteaddressableobjectstore
WhatMattersforApp,DB,andCompute
• Theflexibilityandeaseofuseoftheinterfacesprovided
• HowmemoryismanagedtransparentlytoprovideQoS
• Theserviceguaranteeabstractions(QoS choices)provided
• Howconflictsaremanagedandprevented
WhatitMeanstoBigDataandAnalytics
• Throughput,latency,andcapacity:whatQoS isneeded?
• Avoidingpipelineconflictswithmultiplesolutionsusingmemory
• In-memorysolutionscancontrolallocations
• Thiswillallowmultipleframeworkstoco-existbetter
• “Yarnformemory”
• Moretypesofmemory:evictionalgorithmsneedadjustment• Hot,cold=>hot,warm,lukewarm,cold
Implications on Data Infrastructure
Shankar Hundekar. Senior Software Engineer, Ampool Inc.
ApacheGeodeProject
*Sourceapachegeode
Arealtimein-memorydatamanagementplatform
ApacheGeodeArchitecture
!
"
Locator
Server
Server
Server
Server +###
##
$"
$$$$
$
$ $
%%%%
$
$ $
%
' …
PersistenceinApacheGeode…
*Sourceapachegeode
� High availability
� Low latency, high throughput
� Durability
� Minimal impact to in memory speed– Minimize IO operations– Minimize locking– Minimize deserialization
…isaShared-nothingarchitecture
*Sourceapachegeode
Server 3Server 2
Bucket 3*primary*
Bucket 1secondary
Bucket 3secondary
Bucket 2*primary*
Server 1
Bucket 2secondary
Bucket 1*primary*
ampool• PoweredbyApacheGeode• BroadsupportforBigDatasolutions
• Pluggable,distributedmemorylayer
In-MemoryObjectStoreoptimized forAnalyticsampool
PLUGGABLE distributedmemorylayer
%
In-MemoryObjectStoreoptimized forAnalyticsampool
%
MANAGEDFLEXIBILITY
✅‘Bestofbreed’engines
✅ExtensibleCore
✅Pluggablestores
ampool
In-MemoryDistributedSys
Low-latencyComms.
Key-ValueStore
FunctionExecution
+
HighThroughput
TableStore
NativeInterfacePluggableStoreManager
JavaAPI
MASH(CLIExt)
JavaAPI
BuildingonPROVEN In-memoryTechnology
InmemoryStorageManagerforAnalytics
StorageManager
SCM
DRAM
• LeveragesStoragecharacteristics• Basedontheusageofdata• TieredStoragearchitecture …
Aboutthepmem.io library
• OpensourcelibrarythatsimplifiesdevelopmentofappsusingSCM
• WorkswithexistingCPUhardware
• Focusesspecificallyonhowpersistentmemory isexposedtoserver-classapplications
• Applicationswillexplicitlymanagetheplacementofdataamongthethreetiers(volatilememory,persistentmemory,andstorage)
*Sourcepmem.io
Librariesprovidedbypmem
• libpmem provides lowlevelpersistentmemorysupport
• libpmemobj libraryprovidesatransactionalobjectstore,providingmemoryallocation,transactions,andgeneralfacilitiesforpersistentmemoryprogramming.Developersnewtopersistentmemoryprobablywanttostartwiththislibrary
• libpmemblk supportsarraysofpmem-residentblocks,allthesamesize,thatareatomicallyupdated
• libpmemlog providesapmem-residentlog file.Thisisusefulforprograms likedatabasesthatappendfrequently toalogfile
*Sourcepmem.io
Conclusion
• NewStorageClassMemoriesarechangingthein-memorylandscape
• Bothopportunitiesandchallengesforin-memorysolutions
• ApacheGeodeisaprovenin-memorytechnologyplatform
• Ampool isworkingtomeetthechallengesofthesechanges!