Those 10000 classes - Lambda Days · 2018-02-26 · Those 10000 classes I never wrote The power of...

Those 10000 classes I never wrote

The power of recursion schemes applied to data engineering

greetings

• The name’s Valentin (@ValentinKasas)

• Organizer of the Paris Scala User Group

• Member of the ScalaIO conference crew

• Cook

• Wannabe marathon runner

disclaimer

• Too much to show in a short time

!=> no time for cat pictures or funny gifs

• A lot of code that is

• potentially scary

• probably doesn’t compile

the story of a data engineering problem

Problem statement

• Building a « meta-Datalake »

• ~100 data sources w/ ~100 tables each

• From None to production in 6 months

• 5 developers

More fun

• Data coming in batches and streams

• Privacy by design

• Data sources enrolled on a rolling basis

requirements

• Pipeline must be configured at runtime

• Needs a specific schema (!<- privacy)

• Must work with different input & output formats

• JSON, CSV, Parquet, Avro, …

Solution

one schema to rule them all

• The only way to have a generic Pipeline is to build it around a schema

• That schema must contain enough information to allow the required « privacy-by-design »

schemas are cool

With a schema we can generically

• validate data

• generate random test data

• translate data between formats

Schemas are recursive

• a schema is composed of smaller schemas

• that are composed of smaller schemas

• … etc

• … and so is the data they represent

explicit recursion is bad

• The compiler doesn’t help much

• StackOverflowError is around the corner

• Mixed concerns

• Not reusable code

Recursion schemes FTW !

• Originate in « Functional programming with Bananas, Lenses, Envelopes and Barbed Wire »

• Decouple the how and the what

• Scala implementation :

github.com/slamdata/matryoshka

families of schemes

• folds: destroy a structure bottom-up

ex: cata

• unfolds: build-up a structure top-down

ex: ana

• refolds: combine an unfold w/ a fold

ex: hylo

recursion scheme recipe

To use any recursion scheme, you need:

• a Functor (called « pattern-functor »)

• a Fix-point type (most of the time)

• an Algebra and/or a Coalgebra

pattern functors

• idea : replace the recursive « slot » by a type parameter

sealed trait Tree final case class Node(left: Tree, right: Tree) extends Tree final case class Leaf(label: Int) extends Tree

pattern functors

sealed trait TreeF[A] final case class Node[A](left: A, right: A) extends TreeF[A] final case class Leaf[A](label: Int) extends TreeF[A]

object TreeF {

implicit val treeFunctor = new Functor[TreeF] { def map[A, B](fa: TreeF[A])(f: A !=> B): TreeF[B] = fa match { case Node(l, r) !=> Node(f(l), f(r)) case Leaf(i) !=> Leaf[B](i) } } }

pattern functors

• Problem : different shapes of trees have different types

val tree1 = Node(Node(Leaf(1), Leaf(2)), Node(Leaf(3), Leaf(4))) !// tree1: Node[Node[Leaf[Nothing]]] = !!...

val tree2 = Node(Leaf(0), Leaf(-1)) !// tree2: Node[Leaf[Nothing!]] = !!...

val tree3 = Node(Leaf(42), Node(Leaf(12), Leaf(84))) !// tree3: Node[TreeF[_ !<: Leaf[Nothing]]] = !!...

Fix-point types• Solution : use a fix-point type to

« swallow » recursion

final case class Fix[F[_!]](unFix: F[Fix[F!]])

Fix-point types• Solution : use a fix-point type to

« swallow » recursion

final case class Fix[F[_!]](unFix: F[Fix[F!]])

val tree2 = Fix[TreeF](Node( Fix[TreeF](Leaf(0)), Fix[TreeF](Leaf(-1)))) !// tree2: Fix[TreeF] = !!...

val tree3 = Fix[TreeF](Node( Fix[TreeF](Leaf(42)), Fix[TreeF](Node( Fix[TreeF](Leaf(12)), Fix[TreeF](Leaf(84)))))) !// tree3: Fix[TreeF] = !!...

(co)Algebras

• Define what to do with a single layer of your recursive structure

• Algebras: collapse, 1 layer at a time

F[A] !=> A

• Coalgebras: build-up, 1 layer at a time

A !=> F[A]

• A is referred to as the (co)algebra’s carrier

hylomorphisms

Pattern-functor

Algebra

Coalgebra

def hylo[F[_], A, B](a: A) (alg: Algebra[F, B], coalg: Coalgebra[F, A]): B =

alg(coalg(a) map (hylo(_)(alg, coalg)))

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

coalg( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylo( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

coalg( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylo( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

coalg( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

alg( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylo( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

coalg( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

alg( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

alg( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylo( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

alg( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

hylomorphisms

Pattern-functor

Algebra

Coalgebra

alg( )

hylomorphisms

Pattern-functor

Algebra

Coalgebra

GEtting real

one schema to rule one schema to rule them all…

Input Schema

Parquet

SchemaF[A]

Input Schema

Parquet

SchemaF[A]

Input Schema

Parquet

SchemaF[A]

Input Schema

Parquet

Step 1 : pattern functor

sealed trait SchemaF[A]

final case class StructF[A](fields: Map[String, A]) extends SchemaF[A] final case class ArrayF[A](elementType: A) extends SchemaF[A] final case class IntF[A]() extends SchemaF[A] final case class StringF[A]() extends SchemaF[A] !// etc !!...

object SchemaF { implicit val schemaFunctor = new Functor[SchemaF] {

def map[A, B](fa: SchemaF[A])(f: A !=> B): SchemaF[B] = fa match { case StructF(fields) !=> StructF(fields.mapValues(f)) case ArrayF(elem) !=> ArrayF(f(elem)) case IntF() !=> IntF[B]() case StringF() !=> StringF[B]() } } }

STEP 2 : Algebras

val schemaToParquet: Algebra[SchemaF, DataType] = {

case StructF(fields) !=> StructType(fields.map((StructField.apply _).tupled)) case ArrayF(elem) !=> ArrayType(elem) case IntF() !=> IntegerType case StringF() !=> StringType

fixSchema.cata(schemaToParquet)

step 2.1: coalgebras

val avroToSchema: Coalgebra[SchemaF, Schema] = { avro !=> avro.getType match { case RECORD !=> StructF(avro.getFields.map(f !=> f.name !-> f.schema)) case ARRAY !=> ArrayF(avro.getElementType()) case INT !=> IntF() case STRING !=> StringF() } }

avroSchema.ana[Fix](avroToSchema)

avroSchema.hylo(schemaToParquet, avroToSchema)

validating data• we used github.com/jto/validation

• defines the Rule[I, O] type

• we need a single ADT to represent data

• otherwise we couldn’t write algebras

sealed trait DataF[A] final case class GStruct[A](fields: Map[String, A]) extends DataF[A] final case class GArray[A](elements: List[A]) extends DataF[A] final case class GInt[A](value: Int) extends DataF[A] final case class GString[A](value: String) extends DataF[A]

producing validators

val validator: Algebra[SchemaF, Rule[JsValue, Fix[DataF]]] = { case StructF(fields) !=> fields.toList.traverse{ case (name, validation) !=> name !-> (JsPath \ name).read(validation) }.map(fs !=> Fix(GStruct(fs.toMap)))

case ArrayF(element) !=> JsPath.pickList(element).map(es !=> Fix(GArray(es)))

case IntF() !=> JsPath.read[Int].map(v !=> Fix(GInt(v)))

case StringF() !=> JsPath.read[String].map(v !=> Fix(GString(v))) }

less easy stuff

Schema to Avro

• Impossible to write directly an Algebra[SchemaF, Schema]

• In an avro schema, each type must have an unique name

• Solution:

• use the path of each node to name types

• store previously built schemas in a registry

Solution 1 : keep track of the path

final case class EnvT[E, F[_], A](run: (E, F[A]))

type WithPath[A] = EnvT[String, SchemaF, A]

val schemaWithPath: Coalgebra[WithPath, (String, Fix[SchemaF])] = { case (path, Fix(StructF(fields))) !=> EnvT(( path, StructF(fields.map{case (k, v) !=> (s"$path.$k", Fix(v))}) )) case (path, Fix(ArrayF(e))) !=> EnvT(( path, ArrayF((path, Fix(e))) )) !// !!... }

keeping track of the path (2)

def withPathToAvro(namespace: String) : Algebra[EnvT[String, SchemaF, ?], Schema] = { case EnvT((path, StructF(fields))) !=> SchemaBuilder .namespace(namespace) .record(path) .... }

("", fixSchema).hylo(withPathToAvro, schemaWithPath)

solution 2 : use a registry

• Schemes come in different flavours : « classic », monadic, generalized

• Here we need the monadic flavour :

AlgebraM[M[_], F[_], A] !== F[A] !=> M[A]

type FingerPrinted = (Long, Schema)

type Registry[A] = State[Map[Long, Schema], A]

solution 2 : using a registry

val reuseTypes: AlgebraM[Registry, SchemaF, FingerPrinted] = { avro !=> case StructF(fields) !=> val fp = fingerPrint(fields) State.get.flatMap{ knownTypes !=> if(knownTypes.contains(fp)) State.state(fp !-> knownTypes(fp)) else { val schema = SchemaBuilder .newRecord(fp) !!... State.put( knownType + (fp !-> schema)) .map(State.state(fp !-> schema)) } } }

fixSchema.cataM(reuseTypes)

writing transformed data

• Once we get a Fix[DataF], we need to serialize it

• to Parquet, using spark’s Row

• to Avro using GenericRecord

serializing to parquet

• We can nest Rows, so we can use an algebra with Row as the carrier

• But, we don’t want our « simple » columns to be wrapped

• If we have a 2 columns table we want to output Row(col1, col2), not Row(Row(col1), Row(col2))

paramorphism

• We need to know when we’ve reached the « bottom » of our tree

• para is a scheme that gives us the previous « tree » it has consumed

• It uses a GAlgebra[(T, ?), F, A]

GAlgebra[W[_], F[_], A] !== F[W[A!]] !=> A

building rows

val toRow: GAlgebra[(Fix[DataF], ?), DataF, Row] = { case GStruct(fields) !=> val values = fields.map{ case (_, (fix, value)) !=> if (isSimpleType(fix)) value.get(0) !// unwrap the row else value } Row(values: _*)

case GInt(value) !=> Row(value)

!// etc… }

fixData.para[Row](toRow)

to be continued …

well, actually …

inspired from real facts… but inspired only

It might seem hard at first …

… but it gets better

Those 10000 classes - Lambda Days · 2018-02-26 · Those 10000 classes I never wrote The power of...

Documents

Transcript of Those 10000 classes - Lambda Days · 2018-02-26 · Those 10000 classes I never wrote The power of...

Simulation of Wireless Communication Systems using …pparis/classes/resources_732/simulation.pdf · 8 % communications parameters Parameters.T = 1/10000; % symbol period ... Digital

Lioa 10000 model drii 10000 bh 4 năm-lioa.net

10000 клиентов

BisonBiogeme: syntaxofthemodeling language · ASC4 -0.51122 -10000 10000 0 ASC5 0.718513 -10000 10000 0 1Bounds speciﬁcation is mandatory in BisonBiogeme. If you do not want bounds,

Document3 - scstwelfare.bih.nic.in · 2225 2011—12 2011—12 21.2.2012 9306 85 43 10000 10000 10000 10000 7807 91 41 10000 000 10000 10000 67 47 10000 88 MO 91.60 79m 0 00 10000

€¦ · Web view2016 ICD-10 to ICD-9 GEMs A000 0010 00000 A001 0011 00000 A009 0019 00000 A0100 0020 10000 A0101 0020 10000 A0102 0020 10000 A0103 0020 10000 A0104 0020 10000 A0105

Photoshop 10000

PEDESTRIAN- AND TRANSIT-FRIENDLY … and transit-friendly design fea-tures fall into three classes: those deemed es-sential; those deemed highly desirable; and those deemed nice but

Cisco 10000

10000 facebookreport

HÉRCULES 15000 INOX HÉRCULES 10000 INOX€¦ · HÉRCULES 10000 Inox HÉRCULES 10000 Inox HÉRCULES 10000 Arrozeiro Inox - Capacidad volumétrica: 7.300 l - Capacidad de carga:

· XLS file · Web view · 2017-08-291 20000 10000 30000. 2 13360 10000 23360. 3 13360 10000 23360. 4 20000 10000 30000. 5 20000 10000 30000. 6 20000 10000 30000. 7 20000 4500 24500.

10000 Cigarettes

10000 Lives, 10000 Stories

Scanned by CamScanner · Md. Taijul Islam Rumina Khatun Mriclul Baruah Khairul Islam 10000 10000 10,000 20,000 0000 10000 5000 5,000 10000 10000 10000 5,000 10,000 5,000 Central Bank

Sapore Restaurant - TECHNIQUES CLASSES · 2021. 1. 18. · TECHNIQUES CLASSES These hands-on classes are ideal for both novice cooking students and those experienced students seeking

10000 words

Č 7 SMERALOVA 10 10000 34 10 10000 1786 10000

THE IDENTIFICATION PLATE OF lHE CRANEDOC. LRl 65.GB 006. l I RI rev. 00 / 2014 10.0, 39.0 33.5 G 10000 10000 10000 10000 10000 10000 10000 10000 10000 10000 10000 10000 10000 10000

Vinn 10000