BUILDING APACHE SPARK APPLICATION PIPELINES FOR THE KUBERNETES ECOSYSTEM · 2017-12-14 ·...

BUILDINGAPACHESPARKAPPLICATIONPIPELINESFORTHEKUBERNETESECOSYSTEM

MichaelMcCune

14November2016

1

INTRODUCTION

Alittleaboutme

EmbeddedtoOrchestrationRedHatemergingtechnologiesOpenStackSaharaOshinkoprojectforOpenShift

#ApacheBigDataEU2016 2

OVERVIEW

BuildingApplicationPipelines

CaseStudy:Ophicleide

Demonstration

LessonsLearned

NextSteps


INSPIRATION

Largerthemes

DeveloperempowermentImprovedcollaborationOperationalfreedom


CLOUDAPPLICATIONS

Whatarewetalkingabout?

MultipledisparatecomponentsRequiredeploymentflexibilityChallengingtodebug

#ApacheBigDataEU2016

Spark

HTTP

Node.jsMongoDB

Python

MySQL

RubyKafka

HDFS

ActiveMQ

PostgreSQL

5

PLANNING

Beforeyoubeginengineering

IdentifymovingpiecesStoryboardthedataflowVisualizesuccessandfailure


Spark HTTPNode.js MongoDBPython

6

PLANNING

Insightfulanalytics

Whatdataset?Howtoprocess?Wherearetheresults?



Ingest Process Publish

7

BUILDING

Decomposeapplicationcomponents

NaturalbreakpointsBuildformodularityStatelessversusstateful



8

BUILDING

Focusonthecommunication

CoordinateinthemiddleNetworkresiliencyKubernetesDNS


COLLABORATING

Buildingasateam

TherighttoolsModularprojectsIterativeimprovementsCoordinatingactions



10

CASESTUDY:OPHICLEIDE



Whatdoesitdo?

Word2VecmodelsHTTPavailabledataSimilarityqueries


SparkBrowser Node.js

MongoDB

Python

Kubernetes

Spark

Spark

Spark

Spark

TextData

TextData

TextData

12


Buildingblocks

ApacheSparkWord2VecKubernetesOpenShiftNode.jsFlaskMongoDBOpenAPI


DEEPDIVE

OpenAPI

SchemaforRESTAPIsWealthoftoolingCentraldiscussionpoint


OPENAPI


paths:/:get:description:|-Returnsinformationabouttheserverversionresponses:"200":description:|-Validserverinforesponseschema:

app=connexion.App(__name__,specification_dir='./swagger/')app.add_api('swagger.yaml',arguments={'title':'TheRESTAPIfortheOphicleide''Word2Vecserver'})app.run(port=8080)

15

DEEPDIVE

ConfigurationData

Whatisneeded?Howtodeliver?


Node.js

Python

Kubernetes

REST_ADDR=127.0.0.1

REST_PORT=8080

MONGO=mongodb://admin:admin@mongodb

16

CONFIGURATIONDATA


spec:containers:-name:${WEBNAME}image:${WEBIMAGE}env:-name:OPH_TRAINING_ADDRvalue:${OPH_ADDR}-name:OPH_TRAINING_PORTvalue:${OPH_PORT}-name:OPH_WEB_PORTvalue:"8081"ports:-containerPort:8081protocol:TCP

17

CONFIGURATIONDATA


vartraining_addr=process.env.OPH_TRAINING_ADDR||'127.0.0.1';vartraining_port=process.env.OPH_TRAINING_PORT||'8080';varweb_port=process.env.OPH_WEB_PORT||8080;

app.get("/api/models",function(req,res){varurl=`http://${training_addr}:${training_port}/models`;request.get(url).pipe(res);});

app.get("/api/queries",function(req,res){varurl=`http://${training_addr}:${training_port}/queries`;request.get(url).pipe(res);});

app.listen(ophicleide_web_port,function(){console.log(`ophicleide-weblisteningon${web_port}`);});

18

SECRETS

NotusedinOphicleide,butworthmentioning


volumes:-name:mongo-secret-volumesecret:secretName:mongo-secretcontainers:-name:shiny-squirrelimage:elmiko/shiny_squirrelargs:["mongodb"]volumeMounts:-name:mongo-secret-volumemountPath:/etc/mongo-secretreadOnly:true

19

SECRETS

Eachsecretexposedasafileinthecontainer


MONGO_USER=$(cat/etc/mongo-secret/username)MONGO_PASS=$(cat/etc/mongo-secret/password)

/usr/bin/python/opt/shiny_squirrel/shiny_squirrel.py\--mongo\mongodb://${MONGO_USER}:${MONGO_PASS}@${MONGO_HOST_PORT}

20

DEEPDIVE

Sparkprocessing

ReadtextfromURLSplitwordsCreatevectors


SPARKPROCESSING


defworkloop(master,inq,outq,dburl):sconf=SparkConf().setAppName("ophicleide-worker").setMaster(master)sc=SparkContext(conf=sconf)

ifdburlisnotNone:db=pymongo.MongoClient(dburl).ophicleide

outq.put("ready")

whileTrue:job=inq.get()urls=job["urls"]mid=job["_id"]model=train(sc,urls)

items=model.getVectors().items()words,vecs=zip(*[(w,list(v))forw,vinitems])

22

SPARKPROCESSING


deftrain(sc,urls):w2v=Word2Vec()rdds=reduce(lambdaa,b:a.union(b),[url2rdd(sc,url)forurlinurls])returnw2v.fit(rdds)

defurl2rdd(sc,url):response=urlopen(url)corpus_bytes=response.read()text=str(corpus_bytes).replace("\\r","\r").replace("\\n","\n")rdd=sc.parallelize(text.split("\r\n\r\n"))rdd.map(lambdal:l.replace("\r\n","").split(""))returnrdd.map(lambdal:cleanstr(l).split(""))

23

SPARKPROCESSING


defcreate_query(newQuery)->str:mid=newQuery["model"]word=newQuery["word"]model=model_cache_find(mid)ifmodelisNone:msg=(("notrainedmodelwithID%ravailable;"%mid)+"check/modelstoseewhenoneisready")returnjson_error("NotFound",404,msg)else:#XXXw2v=model["w2v"]qid=uuid4()try:syns=w2v.findSynonyms(word,5)q={"_id":qid,"word":word,"results":syns,"modelName":model["name"],"model":mid}(query_collection()).insert_one(q)

24

DEMONSTRATION

seeademoat


https://vimeo.com/189710503

25

https://vimeo.com/189710503

LESSONSLEARNED

Thingsthatwentsmoothly

OpenAPIDockerfilesKuberenetestemplates


LESSONSLEARNED

Thingsthatrequiregreatercoordination

APIcoordinationComputeresourcesPersistentstorageSparkconfigurations


LESSONSLEARNED

Computeresources

CPUandmemoryconstraintsLabelselectors


Kubelet

Node

Pod Pod

Pod

Pod

Pod

Kubelet

Node

Pod Pod

Pod

Kubelet

Node

Pod

28

NEXTSTEPS

Wheretotakethisproject?

MoreSpark!SeparatequeryserviceDevelopmentversusproduction


PROJECTLINKS

Ophicleide

ApacheSpark

Kubernetes

OpenShift


https://github.com/ophicleide

https://spark.apache.org

https://kubernetes.io

https://openshift.org

30

https://github.com/ophicleide

https://spark.apache.org/

https://kubernetes.io/

https://openshift.org/

THANKS!elmiko

https://elmiko.github.io@FOSSjunkie

31

BUILDING APACHE SPARK APPLICATION PIPELINES FOR THE KUBERNETES ECOSYSTEM · 2017-12-14 ·...

Documents

Transcript of BUILDING APACHE SPARK APPLICATION PIPELINES FOR THE KUBERNETES ECOSYSTEM · 2017-12-14 ·...