Beating Python's GIL!to

Max Out Your CPUsAndrew Montalenti!

CTO, Parse.ly @amontalenti

Scaling Python!to

3,000 CoresAndrew Montalenti!

CTO, Parse.ly @amontalenti

OR:

What happens when you have 153 TB of compressed customer data that may need to be reprocessed at any time,

and it’s now growing at 10-20TB per month?

@dabeaz = “the GIL guy”

Is the GIL a feature, not a bug?!

In one Python process,

at any one time,

only one Python bytecode instruction

is executing at once.

should we just rewrite it in Go?

fast functions!running in parallel

Python

State

Code

Server 1

Core 2

Core 1

Server 2

Core 2

Core 1

Server 3

Core 2

Core 1

from urllib.parse import urlparse urls = ["http://arstechnica.com/", "http://ars.to/1234", "http://ars.to/5678", ...]

Python

State

Code

Server 1

Core 2

Core 1

Server 2

Core 2

Core 1

Server 3

Core 2

Core 1

map(urlparse, urls)

from urllib.parse import urlparse urls = ["http://arstechnica.com/", "http://ars.to/1234", "http://ars.to/5678", ...]

Cythonspeeding up functions on a single core

concurrent.futuresgood map API, but odd implementation details

Python

State

Code

Server 1

Core 2

Core 1

Server 2

Core 2

Core 1

Server 3

Core 2

Core 1

executor = ThreadPoolExecutor() executor.map(urlparse, urls)

Python

State

Code

Server 1

Core 2

Core 1

Server 2

Core 2

Core 1

Server 3

Core 2

Core 1

executor = ProcessPoolExecutor() executor.map(urlparse, urls)

Python subprocess

State

Code

Python subprocess

State

Code

pickle.dumps()

os.fork()

joblibmap functions over local machine cores

by cleaning up stdlib facilities

Python

State

Code

Server 1

Core 2

Core 1

Server 2

Core 2

Core 1

Server 3

Core 2

Core 1

par = Parallel(n_jobs=2) do_urlparse = delayed(urlparse) par(do_urlparse(url) for url in urls)

Python subprocess

State

Code

Python subprocess

State

Code

pickle.dumps()

os.fork()

ipyparallelmap functions over a pet compute cluster

Python

State

Code

Server 1

Core 2

Core 1

Server 2

Core 2

Core 1

Server 3

Core 2

Core 1

rc = Client() rc[:].map_sync(urlparse, urls)

Python

State

Code

Python

State

Code

ipengine

Python

State

Code

Python

State

Code

Python

State

Code

ipengine

ipengine

ipcontroller

Python

State

Code

pickle.dumps()

pickle.dumps()

pykafkamap functions over a multi-consumer log

Python

State

Code

Server 1

Core 2

Core 1

Server 2

Core 2

Core 1

Server 3

Core 2

Core 1

consumer = ... # balanced while True: msg = consumer.consume() msg = json.loads(msg) urlparse(msg["url"])

Python

State

Code

Python

State

CodePython

State

Code

Python

State

Code

Python

State

Code

pykafka.producer

Python

State

Code

pystormmap functions over a stream of inputs

to generate a stream of outputs

Python

State

Code

Server 1

Core 2

Core 1

Server 2

Core 2

Core 1

Server 3

Core 2

Core 1

Python

State

Code

Python

State

CodePython

State

Code

Python

State

Code

pykafka.producer

Python

State

Code

multi-lang json protocol

class UrlParser(Topology): url_spout = UrlSpout.spec(p=1) url_bolt = UrlBolt.spec(p=4, input=url_spout)

pysparkmap functions over a dataset representation

to perform transformations and actions

Python

State

Code

Server 1

Core 2

Core 1

Server 2

Core 2

Core 1

Server 3

Core 2

Core 1

Python

State

Code

Python

State

CodePython

State

Code

Python

State

Code

pyspark.SparkContext

sc = SparkContext() file_rdd = sc.textFile(files) file_rdd.map(urlparse).take(1)

cloudpickle

py4j and binary pipes

"lambda architecture"

Parse.ly "Batch Layer" Topologies with Spark & S3

Parse.ly "Speed Layer" Topologies with Storm & Kafka

Parse.ly Dashboards and APIs with Elasticsearch & Cassandra

Parse.ly Raw Data Warehouse with Streaming & SQL Access

Technology Component Summary

parting thoughts

the free lunch is over, but not how we thought

multi-process, not multi-thread multi-node, not multi-core

message passing, not shared memory !

heaps of data and streams of data

GIL: it's a feature, not a bug.

help us!

pystorm pykafka

streamparse

Questions?tweet at @amontalenti