Python concurrency: libraries overview
-
Upload
andrii-mishkovskyi -
Category
Technology
-
view
1.976 -
download
4
Transcript of Python concurrency: libraries overview
![Page 1: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/1.jpg)
Built-in solutions3-rd Party Solutions
Summary
Concurrency in PythonOverview of current solutions
Andrii V. Mishkovskyi
December 28, 2009
Andrii V. Mishkovskyi Concurrency in Python
![Page 2: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/2.jpg)
Built-in solutions3-rd Party Solutions
Summary
Based on system threadsProcess-basedCoroutines
threading module
Kernel threadsSimilar to JVM’s threading moduleLots of synchronization primitives
Andrii V. Mishkovskyi Concurrency in Python
![Page 3: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/3.jpg)
Built-in solutions3-rd Party Solutions
Summary
Based on system threadsProcess-basedCoroutines
Global Interpreter Lock
Simple locking mechanismDoesn’t allow execution of morethan 1 thread at a timeUnless it’s IO-bound threadCPU-bound threads don’t benefitfrom parallel execution
Andrii V. Mishkovskyi Concurrency in Python
![Page 4: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/4.jpg)
Built-in solutions3-rd Party Solutions
Summary
Based on system threadsProcess-basedCoroutines
multiprocessing module
API similar to multithreading
Operates on process levelNice way of avoiding GIL issues
Andrii V. Mishkovskyi Concurrency in Python
![Page 5: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/5.jpg)
Built-in solutions3-rd Party Solutions
Summary
Based on system threadsProcess-basedCoroutines
PEP 342
Simple support through “enhancedgenerators”yield expressionNo built-in library or framework
Andrii V. Mishkovskyi Concurrency in Python
![Page 6: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/6.jpg)
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
Twisted
Emacs of Python concurrencyTons of unit testsYears of active developmentSupports almost every protocolthere is
Andrii V. Mishkovskyi Concurrency in Python
![Page 7: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/7.jpg)
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
![Page 8: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/8.jpg)
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
![Page 9: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/9.jpg)
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
![Page 10: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/10.jpg)
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
![Page 11: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/11.jpg)
Twisted example
from twisted.internet import reactor, protocolclass Echo(protocol.Protocol):
def dataReceived(self, data):self.transport.write(data)
factory = protocol.ServerFactory()factory.protocol = Echoreactor.listenTCP(8000, factory)reactor.run()
![Page 12: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/12.jpg)
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
Stackless Python
Patch to CPython to not use C stackAbility to monkeypatch existingcodebase with “stackless sockets”100% compatible with CPython
Andrii V. Mishkovskyi Concurrency in Python
![Page 13: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/13.jpg)
Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):
def __init__(self, conn):self.serversocket = \
socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)()
![Page 14: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/14.jpg)
Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):
def __init__(self, conn):self.serversocket = \
socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)()
![Page 15: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/15.jpg)
Stackless Example, part Iimport stacklessimport stacklesssocketstacklesssocket.install()import socketclass Server(object):
def __init__(self, conn):self.serversocket = \
socket.socket(socket.AF_INET,socket.SOCK_STREAM)
self.serversocket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
self.serversocket.bind(conn)self.serversocket.listen(5)stackless.tasklet(self.accept)()
![Page 16: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/16.jpg)
Stackless Example, part II
def accept(self):while self.serversocket.accept:
(clientsocket, address) = \self.serversocket.accept()
stackless.tasklet(self.manage)(clientsocket,
address)stackless.schedule()
def manage(self, clientsocket, address):clientsocket.send(
cliensocket.recv(4096))s = Server((’0.0.0.0’, 8000))stackless.run()
![Page 17: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/17.jpg)
Stackless Example, part II
def accept(self):while self.serversocket.accept:
(clientsocket, address) = \self.serversocket.accept()
stackless.tasklet(self.manage)(clientsocket,
address)stackless.schedule()
def manage(self, clientsocket, address):clientsocket.send(
cliensocket.recv(4096))s = Server((’0.0.0.0’, 8000))stackless.run()
![Page 18: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/18.jpg)
Stackless Example, part II
def accept(self):while self.serversocket.accept:
(clientsocket, address) = \self.serversocket.accept()
stackless.tasklet(self.manage)(clientsocket,
address)stackless.schedule()
def manage(self, clientsocket, address):clientsocket.send(
cliensocket.recv(4096))s = Server((’0.0.0.0’, 8000))stackless.run()
![Page 19: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/19.jpg)
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
Kamaelia
Developed by BBC ResearchFollows UNIX principlesComponents all the way round
Andrii V. Mishkovskyi Concurrency in Python
![Page 20: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/20.jpg)
Kamaelia Example
import Axonfrom Kamaelia.Chassis.ConnectedServer\
import SimpleServerclass Echo(Axon.Component.component):
def main(self):while 1:
while self.dataReady("inbox"):self.send(self.recv("inbox"),
"outbox")yield 1
SimpleServer(protocol=Echo, port=1500).run()
![Page 21: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/21.jpg)
Kamaelia Example
import Axonfrom Kamaelia.Chassis.ConnectedServer\
import SimpleServerclass Echo(Axon.Component.component):
def main(self):while 1:
while self.dataReady("inbox"):self.send(self.recv("inbox"),
"outbox")yield 1
SimpleServer(protocol=Echo, port=1500).run()
![Page 22: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/22.jpg)
Kamaelia Example
import Axonfrom Kamaelia.Chassis.ConnectedServer\
import SimpleServerclass Echo(Axon.Component.component):
def main(self):while 1:
while self.dataReady("inbox"):self.send(self.recv("inbox"),
"outbox")yield 1
SimpleServer(protocol=Echo, port=1500).run()
![Page 23: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/23.jpg)
Kamaelia Example
import Axonfrom Kamaelia.Chassis.ConnectedServer\
import SimpleServerclass Echo(Axon.Component.component):
def main(self):while 1:
while self.dataReady("inbox"):self.send(self.recv("inbox"),
"outbox")yield 1
SimpleServer(protocol=Echo, port=1500).run()
![Page 24: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/24.jpg)
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
cogen
Coroutine-basedExtensible schedulerQuite low-level (if compared toKamaelia)
Andrii V. Mishkovskyi Concurrency in Python
![Page 25: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/25.jpg)
cogen example, part I
from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():
srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:
conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))
![Page 26: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/26.jpg)
cogen example, part I
from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():
srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:
conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))
![Page 27: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/27.jpg)
cogen example, part I
from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():
srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:
conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))
![Page 28: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/28.jpg)
cogen example, part I
from cogen.core import socketsfrom cogen.core import schedulersfrom cogen.core.coroutines import coroutine@coroutinedef server():
srv = sockets.Socket()srv.bind((0.0.0.0’, 8000))srv.listen(10)while True:
conn, addr = yield srv.accept()m.add(handler, args=(conn, addr))
![Page 29: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/29.jpg)
cogen example, part II
@coroutinedef handler(sock, addr):
data = yield sock.read()yield sock.write(data)sock.close()return
m = schedulers.Scheduler()m.add(server)m.run()
![Page 30: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/30.jpg)
cogen example, part II
@coroutinedef handler(sock, addr):
data = yield sock.read()yield sock.write(data)sock.close()return
m = schedulers.Scheduler()m.add(server)m.run()
![Page 31: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/31.jpg)
cogen example, part II
@coroutinedef handler(sock, addr):
data = yield sock.read()yield sock.write(data)sock.close()return
m = schedulers.Scheduler()m.add(server)m.run()
![Page 32: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/32.jpg)
cogen example, part II
@coroutinedef handler(sock, addr):
data = yield sock.read()yield sock.write(data)sock.close()return
m = schedulers.Scheduler()m.add(server)m.run()
![Page 33: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/33.jpg)
Built-in solutions3-rd Party Solutions
Summary
TwistedStackless PythonKamaeliacogenOthers
OthersConcurrence – nice asynchrounousframework with async MySQL drivereventlet – based on coroutines,developed and used by “SecondLife”py.execnet – distribute tasks overnetwork
Andrii V. Mishkovskyi Concurrency in Python
![Page 34: Python concurrency: libraries overview](https://reader031.fdocuments.net/reader031/viewer/2022020115/55492989b4c905b44c8bdc39/html5/thumbnails/34.jpg)
Built-in solutions3-rd Party Solutions
Summary
GIL doesn’t harm 90% of theexisting codeBut Twisted, Kamaelia and othersare often cleanerPython is not Erlang, it’s better ;)
Andrii V. Mishkovskyi Concurrency in Python