What’s new in Spark 2.0?files.meetup.com/19070069/161129 - Michael... · ©2016 Couchbase Inc. 3...

What’snewinSpark2.0?


SparkOverview

ApacheSparkisafastandgeneralengineforlarge-scaledataprocessing.


SparkOverview


Spark2.0

§ Largelycompatiblewith1.x§ SimplifiesAPI§ 2000patchesfrom280contributors

http://www.slideshare.net/SparkSummit/simplifying-big-data-applications-with-apache-spark-20


Spark2.0

§ StructuredAPIImprovements§ Whole-stagecodegeneration§ StructuredStreaming§ SimplerSetup§ SQL2003Support§ MLlibenhancements§ EnhancedRsupport§ …


StructuredAPIImprovements

§ Dataset(typed)andDataFrame(untyped)arenowunified§  DataFrame==Dataset<Row>

§ AlsousedforStructuredStreaming

https://databricks.com/blog/2016/07/14/a-tale-of-three-apache-spark-apis-rdds-dataframes-and-datasets.html


Whole-StageCodegen

§ Second-generationTungstenengine§ DepartingfromVolcanoIteratorModel§ Also:Vectorizationformoreefficientbatch-processing

https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html


Whole-StageCodegen

https://databricks.com/blog/2016/05/23/apache-spark-as-a-compiler-joining-a-billion-rows-per-second-on-a-laptop.html


StructuredStreaming(Experimental)

§ Tackling“continuousapplications”§  IntegratedAPIwithbatchjobs§ Betterinteractionwithstoragesystems§ RichIntegrationintotherestofSpark



https://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html


SimplerSetup

§ SparkSessionsubsumesSQLContext,HiveContext,…§ Onecommonentrypoint


SQL2003Support

§ SupportsSQL2003Standard

§ ReworkednativeSQLparser§ NativeDDLcommandimplementations§ Allkindsofsubqueriesnowsupported§ Cannowrunall99TPC-DSqueries


MLlibEnhancements

§ DataFrameasprimaryMLAPI§ ModelPersistence§  SupportforalllanguageAPIsinSpark:Scala,Java,Python&R§  SupportfornearlyallMLalgorithmsintheDataFrame-basedAPI§  SupportforsinglemodelsandfullPipelines,bothunfitted(a“recipe”)andfitted(aresult)

§  Distributedstorageusinganexchangeableformat

QAThanks!

Couchbase&Spark


EcommerserunsonCouchbase

6 10 ECOMMERCE COMPANIES

IN THE UNITED STATES

of the TOP

Online shopping

Omni channel services


TravelrunsonCouchbase

3 3

GLOBAL DISTRIBUTION SYSTEMS WORLDWIDE

of the TOP

3 10

AIRLINES

of the TOP


OnlineVideoStreamingrunsonCouchbase

6 10 NORTH AMERICAN AND

EUROPEAN BROADCAST TELEVISION COMPANIES

of the TOP


Sports&CasinoGamingrunsonCouchbase

6 10 ONLINE SPORTS AND

CASINO GAMING COMPANIES

of the TOP


FinancialServicesrunonCouchbase

3 3 CREDIT REPORTING

COMPANIES

of the TOP

WhySparkandCouchbaseOverview&Use-Cases


UseCases

Operations Analytics

CB

§  Recommendations§  Predictiveanalytics§  Frauddetection

§  Catalog§  Personalization§  Mobileapplications


UseCase:OperationalizeAnalytics/ML

Hadoop

MLModel

Data Warehouse

Training Data

CB

Model Online Data

Serving

Predictions


UseCase:DataIntegration

RDBMS S3 HDFS ES

NoSQL


StandaloneDeployment


Side-By-SideDeployment

AccessPatternsFromSparktoCouchbaseandBackAgain


Key-Value

Fetch/StorebyDocumentID


Key-Value


N1QLQuery

FetchbyCriteria“SQL”


Key-Value


N1QLQuery


Map-ReduceViews

MaterializedIndexes

(Aggregation)


Key-Value


N1QLQuery


Map-ReduceViews

MaterializedIndexes

(Aggregation)

Streaming

MutationStreamsForProcessing


Key-Value


N1QLQuery


Map-ReduceViews

MaterializedIndexes

(Aggregation)

Streaming


FullText

SearchonFreeformText


Key-Value


N1QLQuery


Map-ReduceViews

MaterializedIndexes

(Aggregation)

Streaming



CouchbaseDataPartitioning


DataLocality

§ RDDLocationHintsbasedontheClusterMap

§ NotavailableforN1QLorViews§  Roundrobin-can’tgivelocationhints§  Backendisscattergatherwith1noderesponding


N1QLQuery

§ N1QLisaSQLservicewithJSONextensions

§ UsesCouchbase’sGlobalSecondaryIndexes

§ Canrunonanynodeswithinthecluster

§ Nodeswithdifferingservicescanbeaddedandremovedasneededonthefly


DataService

Projector&Router

CouchbaseQueryArchitecture

QueryService

IndexService

SupervisorIndexmaintenance&Scancoordinator

Index#2Index#1

QueryProcessorcbq-engine

Bucket#1 Bucket#2

DCPStreamIndex#4Index#3

...Bucket#2

Bucket#1

ForestDBStorageEngine


SparkSQLSources

TableScanScanallofthedataandreturnit

PrunedScanScananindexthatmatchesonlyrelevantdatatothequeryathand.

PrunedFilteredScanScananindexthatmatchesonlyrelevantdatatothequeryathand.


PredicateConversion


SchemaInference


SchemaInference

N1QLRelation:28 - Inferring schema from bucket travel-sample with query 'SELECT META(`travel-sample`).id as `META_ID`, `travel-sample`.* FROM `travel-sample` WHERE `type` = 'airline' LIMIT 1000'

N1QLRelation:28 - Executing generated query: 'SELECT `name`,`callsign` FROM `travel-sample` WHERE `type` = 'airline''


SchemaInference


DCPandSparkStreaming

ReplicaIndexing

…


StructuredStreamingSource

50

Adaptedfromhttps://databricks.com/blog/2016/07/28/structured-streaming-in-apache-spark.html

DCPStream UnboundedTable


(Un)StructuredStreaming?

51


StructuredStreamingSource

52


StructuredStreamingSink

53


CouchbaseSparkConnector1.2.1

§ Spark1.6.xsupport,includingDatasets§ DCPFlowControl§ EnhancedJavaAPIs

54


CouchbaseSparkConnector2.0.0

•  Spark2.0.xSupport•  EnhancedDCPClient•  ExperimentalStructuredStreaming

55


Resources

§  SparkPackageshttps://spark-packages.org/package/couchbase/couchbase-spark-connector

•  Docshttp://docs.couchbase.com

§  Sourcehttps://github.com/couchbase/couchbase-spark-connector

§  Bugshttps://issues.couchbase.com/browse/SPARKC

56

QAThanks!

What’s new in Spark 2.0?files.meetup.com/19070069/161129 - Michael... · ©2016 Couchbase Inc. 3...

Documents

Transcript of What’s new in Spark 2.0?files.meetup.com/19070069/161129 - Michael... · ©2016 Couchbase Inc. 3...