Spark at Bloomberg: Dynamically Composable Analytics
32
Spark @ Bloomberg: Dynamic Composable Analytics Partha Nageswaran Sudarshan Kadambi BLOOMBERG L.P.
-
Upload
jen-aman -
Category
Data & Analytics
-
view
1.237 -
download
0
Transcript of Spark at Bloomberg: Dynamically Composable Analytics
Spark @ Bloomberg: Dynamic Composable Analytics
Partha NageswaranSudarshan KadambiBLOOMBERG L.P.
Bloomberg Spark Server
(Persistent)Spark
Context
Request Handler
2
SparkServerization atBloomberghasculminated inthecreationoftheBloomberg SparkServer
Function Transform
Registry (FTR)
Managed DataFrameRegistry
IngestionManager
RequestProcessor
RequestProcessor
Declarative QueryRequest
Processor
JVM
Spark Serverization – Motivation
3
• Stand-aloneSparkAppsonisolatedclustersposechallenges:
– Redundancy in:
» CraftingandManagingofRDDs/DFs
» Codingofthesameorsimilartypesoftransforms/actions
– Managementofclusters,replicationofdata,etc.
– Analyticsareconfined tospecificcontentsetsmakingCross-AssetAnalytics muchharder
– NeedtohandleReal-timeingestion ineachApp
Spark Cluster
Spark App
Spark Cluster
Spark Server
Spark App
Spark App
Spark Cluster
Spark App
Dynamic Composable Analytics• Compositional Analyticsarecommon placeintheFinancialSector
Decile Rankthe14-dayRelativeStrengthIndexofActiveEquityStocks:
DECILE(RSI(
Price,14,['IBMUSEquity','VODLNEquity',…]
))
• PriceisdataabstractedasaSparkDataFrame
• RSI,DECILEarebuilding blockanalytics,expressibleasSparktransformsandactions
4
Dynamic Composable Analytics• Anotherusecase maywanttocomposePercentilewithRSI
PercentileRankthe14-dayRelativeStrengthIndexofActiveEquityStocks:
PERCENTILE(RSI(
Price,14,['IBMUSEquity','VODLNEquity',…]
))
• OrPercentilewithROC,etc.Andthecompositionsmaybearbitrarilycomplex
5
Dynamic Composable Analyticsdef RSI(df:DataFrame,period:Int=14) : DataFrame ={
val smmaCoeff =udf((i:Double)=>scala.math.pow(period-1,i-1)/scala.math.pow(period,i))val rsi_from_rs =udf((n:Double,d:Double)=>100- 100*d/(d+n))val rsi_window=Window.partitionBy('id).orderBy('date.desc)
df.withColumn("weight",smmaCoeff(row_number.over(rsi_window))).withColumn("diff",'value- lead('value,1).over(rsi_window)).withColumn("U",when('diff>0,'diff).otherwise(0)).withColumn("D",when('diff<0,abs('diff)).otherwise(0)).groupBy('id).agg(rsi_from_rs(sum('U*'weight),sum('D*'weight))as'value)
}
def Decile(df:Dataframe) : DataFrame ={df.withColumn("value",ntile(10).over(Window.orderBy('value.desc)))
}
Ack: Andrew Foster6
Function Transform Registry• MaintainaRegistryofAnalytic functions (FTR)withfunctions expressedas
ParametrizedSparkTransformsandActions
• Functionscancompose otherfunctions,alongwithadditional transforms,withintheRegistry
• Registrysupports 'bind'and'lookup'operations
7
Function Transform Registry (FTR)
DecileFUNCTIONS SPARK IMPL.
…Percentile …
… …
Bloomberg Spark Server
(Persistent)Spark Context
Request Handler
8
Function Transform
Registry (FTR)
JVM
Request Processor• RequestProcessors (RPs)aresparkapplications thatorchestratecomposition of
analyticsonDataFrames
– RPscomplywithaspecification thatallowsthemtobehostedbytheBloombergSparkServer
– Eachrequest(suchas:computetheDecile RankoftheRSI)ishandledbyaRequestProcessorthatlooksupfunctionsfromtheFTR,Composesthemandapplies themtoDataFrames
9
Request Handler
Request Processor
.
FTR
Declarative QueryRequest
Processor
Bloomberg Spark Server
(Peristent)Spark
Context
Request Handler
10
Function Transform
Registry (FTR)
JVM
RequestProcessor
RequestProcessor
Declarative QueryRequest
Processor
Managed Data Frames
• BesideslocatingfunctionsfromtheFTR,RequestProcessorshavetopassinDataFramestothesefunctionsasinputs
• RatherthaninstantiateDataFrames,lookupDataFramesfromaDataFramesRegistry
– SuchDataFramesarecalledManagedDataFrames(MDF)
– TheRegistrythatManagestheseDataFramesistheManagedDataFrameRegistry(MDFRegistry)
11
Introducing Managed DataFrames (MDFs)
• AManagedDataFrame (MDF)isanamedDataFrame,optionallycombinedwithExecutionMetadata
– MDFscanbelocatedbynameORbyanyColumnNamedefined intheSchemaof thecorrespondingDF
• ExecutionMetadataincludes:
– DataDistribution metadatacapturesinformationabout thedatadepth, histogram information, etc.
– E.g.:AmanagedDataFrame forpricingof stocks,representing 2yearsofhistoricaldata andanotherforrepresenting 30yearsofhistoricaldata
MDF
Price DF<ID, Price>
Name: Shallow
PriceMDF
ExecutionMetadata:* 2 Yr Price
History
MDF
Price DF<ID, Price>
Name: Deep
PriceMDF
ExecutionMetadata:
* 30 Yr Price History
12
Managed DataFrames
– DataDerivationmetadatawhicharemathematicalexpressions thatdefinehowadditional columnscanbesynthesized fromexistingcolumns intheschema
– E.g.:adjPrice isaderivedColumn, definedintermsofthebasePricecolumn
– Inessence,anMDFwithdataderivationmetadatahaveaSchemathatisaunionofthecontainedDFandthederivedcolumns
MDF
Name:ShallowPriceDF
ExecutionMetadata:* 2 Yr Price
History* adjPrice =
Price – 3% of Price
Price DF<ID, Price>
MDF
Name:Deep
PriceDF
ExecutionMetadata:
* 30 Yr Price History
* adjPrice = Price – 1.75% of
Price
Price DF<ID, Price>
13
The MDF Registry
• TheMDFRegistrywithintheBloombergSparkServer providessupport for:
– BindingMDFsbyName
– LookingupMDFsbyName
– LookingupMDFbyaColumn Name(anelementoftheMDFSchema),etc.
• TheMDFRegistrymaintainsa'table'thatassociates theNameoftheMDFwiththeDFreference andColumnsintheDF
MDFRegistryName Columns DF
Ref.MetaData
ShallowPriceDF
Price,adjPrice
… …
DeepPriceDF
……
…
Price,adjPrice
14
Bloomberg Spark Server
(Peristent)Spark
Context
Request Handler
15
Function Transform
Registry (FTR)
JVM
RequestProcessor
RequestProcessor
Declarative QueryRequest
Processor
Managed DataFrameRegistry
Flow of Requests
• RequestProcessorswithintheSparkServerorchestrateanalytics
– TheseRPshaveaccesstotheRegistryandFTRs
– AreresponsibleforcomposingtransformsandactionsononeormoreMDFs
– MaydynamicallybindadditionalMDFs(materializedorotherwise)forusebyotherApps
Request Handler
Request Processor
.
MDF Registry
lookup MDFs
FTRs
applyFunction
MDFs
decoratewithTransforms
collect
16
Bloomberg Spark Server
Spark Context
Request Processor
Request Processor
Declarative QueryRequest Processor
Request Handler
MDF Registry
MDF
17
Function Transform
Registry (FTR)
RSI …
use MDF
MDF
MDF
17
Bloomberg Spark Server
Spark Context
Request Processor
Request Processor
Declarative QueryRequest Processor
Request Handler
MDF Registry
18
Function Transform
Registry (FTR)
RSI …
use
18
Ingestion Manager
MDF1
MDF2
1 2
1 2
Schema Repository
19
• Enterprise-widedatapipeline
• External(toSpark)schemarepositoryandservice
• EnablesMDFlookupbyadatasetschemaelement
• Analyticexpressionscannowbecomposedoverdataelements
Execution Metadata
20
• DatasetSourceConnection Identifiers
• BackingStores
• Real-time Topics
• StorageLevel&RefreshRate
• SubsetPredicate,etc.
Ad-hoc Cross-Domain Analytics
21
• Registrationofpre-materializedDataFrames
• Collaborativeanalyticsbetweenapplicationworkflows
• DynamiccreationofManagedDataFrames
• SparkServershavedatapertainingtoasingledomainmaterialized
• Ad-hoc cross-domainanalyticsrequirescapabilitytosynthesizeMDFsondemand
Content Subsetting
22
• Highvaluedatasub-settedwithinSpark
• Reducecostofqueryingexternaldatastore
• Specifiedasafilterpredicateattimeofregistration
• E.g.Membercompaniesofpopularindices[Dow30,S&P500,…]haverecordsplacedwithinSpark
Content Subsetting
23
• SeamlessunificationofdatainSpark(DFsubset)andbackingstore(DFsubset’)
(DFsubset UDFsubset’).filter(query)= DFsubset.filter(query)UDFsubset’.filter(query)
• Datasetownersprovidedknobsforcostvsperformance.
• LRUcachelikemechanismplannedinthefuture
• MakesenseasacapabilitynativetoSparkdataframes
Ingestion: Periodic Refresh
24
• PeriodicdatapullintoSparkfromthebackingstore
• Subsetcriteriaappliedduringdataretrieval
• Usedwhenadatasethasabackingstore,butnorealtimeupdatestreamthatwecantapinto
• Datasetownershavecontroloverstoragelevelofthedataframes createdwithinagivenMDF
Ingestion: Stream Reconciliation
25
• Analyticsneedstobelow-latencywithrespecttoqueries,butalsodatafreshness
• Sincedataisbeingsub-settedwithinSpark,needtokeepthesubsetuptodate
• DatasetspublishedtodifferentKafkatopics.
• 1:1mappingbetween datasets,topicsandDStreams.
Ingestion: Stream Reconciliation
26
Backing Store
U1 U2 U3 UN DFsubset
S1 S2 S3 SNDFN
MDF -PriceHistory
Real-Time Stream
(update state)
(Avro Deserialize, Subset Predicate)
(convert to DF-seq)
Similar intent as Structured Streaming, to be introduced in Spark 2.0
Ingestion: Data Transformation• Datainbackingstoresmayneedrepresentationtransforms
beforebeingusedinqueries
• Datainmultipletablesdenormalized intoasingleDFwithinSpark
• Or,quicklyseeeffectofdifferentstoragerepresentationsonperformance,withoutchangingtherepresentationinthebackingstore
• Implementedvia.usertransformsassociatedwithagivenMDF
Spark Server: Memory Management
28
• AnMDFcontainsmultiplegenerationofDFs,beinggeneratedanddestroyed
• MultiplegenerationsoperateduponbyRPsatgivenpointintime
• ReferencecountingtokeeptrackofwhatDFsarebeingusedandbywhom
• Longrunningqueriesabortedforforcedreclamation
Query Consistency
29
• Multiplequeriesneedtooperateonsamesnapshotofdata
• Howtoachieve,ifdataconstantlychangingunderneath?
• EachDFwithinMDFassociatedwithtimeepoch
• Registrylookupwithareferencetime
• Time-alignsub-setted dataframeswithdatainbackingstore
Spark for Online Analytics
30
– HighAvailabilityofSparkDriver• Highbootstrapcosttoreconstructingclusterandcachedstate• NaïveHAmodels(suchasmultipleactiveclusters)surfacequeryinconsistency
– HighAvailabilityofRDDPartitions• Withsubset oruniversecached,lostRDDpartitionskillqueryperformance
– PerformanceConsistency• Performancegatedbyslowestexecutor• HighAvailabilityandLowTailLatencycloselyrelated
– Interactionseffectsbetweenlow-latencyqueriesandlow-latencyupdates• NotoMinimalsandboxingbetweenjobssharingexecutorJVMs
FirstBloombergcontribution:SPARK-15352
Spark Server Acknowledgements
Andrew Foster Joe Davey Shubham Chopra
Nimbus Goehausen Tracy Liang
THANK [email protected]@bloomberg.net
