or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip...

70
tulip or not tulip Jose Ignacio Galarza @igalarzab

Transcript of or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip...

Page 1: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

tulip or not tulip

Jose Ignacio Galarza @igalarzab

Page 2: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

what is tulip? generators coroutines tulip components

Index

Page 3: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

tulip

asyncio

Page 4: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

“asynchronous IO support rebooted”

Page 5: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

Python >= 3.3

Page 6: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

event loop coroutines futures tasks transports protocols

Page 7: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

some necessary history

Page 8: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

python includes generators

PEP 0255

Page 9: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

yield

Page 10: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

def work_hard_normal(): results = [] ! for i in range(1, 10): print('Working very hard %d times...' % i) results.append(i) ! return results !def working_hard_generator(): for i in range(1, 10): print('Working very hard %d times...' % i) yield i !if __name__ == '__main__': for result in work_hard_normal(): if result % 5 == 0: print('Eureka!') break ! for result in working_hard_generator(): if result % 5 == 0: print ('Eureka!') break

Page 11: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

$ python3 001-generator.py ! Working normal 1… Working normal 2… […] Working normal 10… Eureka! Generatoorrr…1 Generatoorrr…2 […] Generatoorrr…5 Eureka!

result

Page 12: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

python includes coroutines

PEP 0342

Page 13: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

send values to

a generator

Page 14: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

generators

vs coroutines

Page 15: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

def list_dir(path, target): for dirpath, dirnames, filenames in os.walk(path): for filename in filenames: target.send(filename) !@coroutine def filter_str(pattern, target): while True: filename = (yield) if pattern in filename: target.send(filename) !@coroutine def print_match(): while True: result = (yield) print(result) !if __name__ == '__main__': list_dir('.', filter_str('py', print_match()))

Page 16: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

def list_dir(path, target): for dirpath, dirnames, filenames in os.walk(path): for filename in filenames: target.send(filename) !@coroutine def filter_str(pattern, target): while True: filename = (yield) if pattern in filename: target.send(filename) !@coroutine def print_match(): while True: result = (yield) print(result) !if __name__ == '__main__': list_dir('.', filter_str('py', print_match()))

Page 17: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

def list_dir(path, target): for dirpath, dirnames, filenames in os.walk(path): for filename in filenames: target.send(filename) !@coroutine def filter_str(pattern, target): while True: filename = (yield) if pattern in filename: target.send(filename) !@coroutine def print_match(): while True: result = (yield) print(result) !if __name__ == '__main__': list_dir('.', filter_str('py', print_match()))

Page 18: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

def list_dir(path, target): for dirpath, dirnames, filenames in os.walk(path): for filename in filenames: target.send(filename) !@coroutine def filter_str(pattern, target): while True: filename = (yield) if pattern in filename: target.send(filename) !@coroutine def print_match(): while True: result = (yield) print(result) !if __name__ == '__main__': list_dir('.', filter_str('py', print_match()))

Page 19: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

def coroutine(func): """ Decorator to auto-start coroutines. Got it from: PEP-0342 """ ! def wrapper(*args, **kwargs): gen = func(*args, **kwargs) next(gen) return gen ! wrapper.__name__ = func.__name__ wrapper.__dict__ = func.__dict__ wrapper.__doc__ = func.__doc__ return wrapper

what the hell is that decorator?

Page 20: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

$ python3 003-coroutines.py ! El Fari - La Mandanga.mp3 Julito Iglesias - Grandes exitos […]

result

Page 21: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

python enhances generators

PEP 0380

Page 22: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

“A syntax is proposed for a generator to delegate part of its

operations to another generator”

Page 23: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

yield from

Page 24: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

class TreeBasic: ! def __init__(self, data, left=None, right=None): self.left = left self.data = data self.right = right ! def __iter__(self): if self.left: for node in self.left: yield node ! yield self.data ! if self.right: for node in self.right: yield node

without yield from

Page 25: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

class TreeYieldFrom: ! def __init__(self, data, left=None, right=None): self.left = left self.data = data self.right = right ! def __iter__(self): if self.left: yield from self.left ! yield self.data ! if self.right: yield from self.right

with yield from

Page 26: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

let’s do an scheduler

Page 27: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

class Scheduler: ! def __init__(self): self.tasks = deque() ! def schedule(self, task): self.tasks.append(task) ! def run(self): while self.tasks: task = self.tasks.popleft() ! try: task.run() except StopIteration: print('Task %s has finished' % task) else: self.tasks.append(task)

declaring the scheduler

Page 28: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

class Task: ! ID = 0 ! def __init__(self, runner): Task.ID += 1 self.id = Task.ID self.runner = runner ! def __str__(self): return str(self.id) ! def run(self): result = next(self.runner) print('[%d] %s' % (self.id, result))

declaring what’s a task

Page 29: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

def list_dir(directory): for item in os.listdir(directory): yield item !!def echo_text(number_times): for i in range(number_times): yield 'Hi dude!'

some tasks examples

Page 30: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

if __name__ == '__main__': s = Scheduler() ! s.schedule(Task(list_dir('.'))) s.schedule(Task(echo_text(5))) s.schedule(Task(echo_text(3))) ! s.run()

creating the tasks…

Page 31: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

result…$ python3 004-scheduler.py ! [1] 001-generator.py [2] Hi dude! [3] Hi dude! [1] 002-pipeline.py [2] Hi dude! [3] Hi dude! [1] 003-coroutine.py [2] Hi dude! [3] Hi dude! [1] 004-tree.py [2] Hi dude! Task 3 has finished [1] 005-scheduler.py [2] Hi dude! [1] 00X-scheduler.pyc Task 2 has finished […]

Page 32: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

python introduces tulip

PEP 3156

Page 33: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

event loop

Page 34: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

the event loop multiplexes a

variety of events

Page 35: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

IO events use the best possible *selector for

the platform

* new module in Python 3.4 epoll, kqueue, IOCP

Page 36: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

interoperability with other frameworks is one of the main

focuses

Page 37: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

how to run the event loop?# Get the main event loop loop = asyncio.get_event_loop() !# Execute it until the future returns loop.run_until_complete(future) !# Run forever (until stop() is called) loop.run_forever()

Page 38: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

how to run callbacks?# Run the callback as soon as possible loop.call_soon(callback, *args) !# Run the callback in at least delay seconds loop.call_later(delay, callback, *args) !# Run the callback at the provided date loop.call_at(when, callback, *args)

Page 39: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

much more about this in @saghul’s talk

check his slides!

Page 40: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

coroutines

Page 41: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

it’s not mandatory to use them, but tulip does it really well

Page 42: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

we already know what’s a coroutine

@coroutine def get_url(url): r, w = yield from open_connection('google.es', 80) ! w.write(b'GET / HTTP/1.0\r\n\r\n') result = yield from r.read() print(result) !!if __name__ == '__main__': loop = asyncio.get_event_loop() loop.run_until_complete(get_url())

Page 43: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

futures

Page 44: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

promises to return a result or an

exception sometime in the future

Page 45: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

they are really *similar to

concurrent.futures

* almost the same API

Page 46: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

use yield from with futures!

generators!

Page 47: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

an easy [email protected] def wait_and_resolve_future(future): for i in range(3): print('Sleeping 1 second') yield from asyncio.sleep(1) ! future.set_result('Future is done!') !!if __name__ == '__main__': loop = asyncio.get_event_loop() ! future = asyncio.Future() asyncio.Task(wait_and_resolve_future(future)) ! loop.run_until_complete(future) print(future.result())

Page 48: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

tasks

Page 49: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

it’s a coroutine *wrapped in a future

* in fact, it’s a subclass

Page 50: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

tasks can make progress alone,

unlike coroutines

Page 51: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

why?

Page 52: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

the __init__ schedules a callback with the next step

of the coroutine

# asyncio/task.py:110 !class Task(futures.Future): ! def __init__(self, coro, *, loop=None): # . . . self._loop.call_soon(self._step) # . . .

Page 53: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

_step runs the generator# asyncio/task.py:246 !def _step(self, value=None, exc=None): # . . . try: if exc is not None: result = coro.throw(exc) elif value is not None: result = coro.send(value) else: result = next(coro) except StopIteration as exc: self.set_result(exc.value) except futures.CancelledError as exc: super().cancel() except Exception as exc: self.set_exception(exc) # . . .

Page 54: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

awesome :D

Page 55: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

transports and

protocols

Page 56: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

transports and protocols are used in

pairs

Page 57: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

“the transport is concerned about

how bytes are transmitted”

Page 58: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

“the protocol determines which bytes to transmit”

Page 59: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

protocol calls transport methods (TCP)

# Write data to the transport write(data) !# Write data using an iterator writelines(list_of_data) !# Checks if the protocol allows to write EOF can_write_eof() !# Close the writing end write_eof() !# Close the connection close()

Page 60: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

protocol callbacks (TCP)

# A new connection has been made connection_made(transport) !# New data has been received data_received(transport) !# EOF received (not all protocols support it) eof_received(transport) !# Broken connection connection_lost(exc)

Page 61: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

simple ECHO protocol using TCP

as the transport

Page 62: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

class EchoServer(asyncio.Protocol): ! def connection_made(self, transport): print('Connected') self.transport = transport ! def data_received(self, data): print('[R] ', data.decode()) print('[S] ', data.decode()) self.transport.write(data) ! def eof_received(self): pass ! def connection_lost(self, exc): print('Connection lost')

Page 63: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

class EchoClient(asyncio.Protocol): ! def connection_made(self, transport): self.transport = transport self.transport.write(b'Hola caracola') print('[S] ', 'Hola caracola') ! def data_received(self, data): print('[R] ', data) ! def eof_received(self): pass ! def connection_lost(self, exc): print('Connection lost') asyncio.get_event_loop().stop()

Page 64: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

def start_client(event_loop): task = asyncio.Task( event_loop.create_connection( EchoClient, '127.0.0.1', 8080 ) ) event_loop.run_until_complete(task) !!def start_server(event_loop): server = event_loop.create_server( EchoServer, '127.0.0.1', 8080 ) event_loop.run_until_complete(server)

Page 65: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

if __name__ == ‘__main__': ! if len(sys.argv) != 2: print('Call with --server or --client flag') sys.exit() ! loop = asyncio.get_event_loop() loop.add_signal_handler(signal.SIGINT, loop.stop) ! if sys.argv[1] == '--server': start_server(loop) else: start_client(loop) ! loop.run_forever()

Page 66: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

and with UDP?

Page 67: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

*almost the same!

* check the examples!

Page 68: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

demo?

Page 69: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

questions?

Page 70: or not tulip - PyCon España 20132013.es.pycon.org/media/tulip-or-not-tulip.pdf · or not tulip Jose Ignacio Galarza @igalarzab. what is tulip? generators coroutines tulip components

thank you!