Python: Design and Implementation

Guido van Rossumguido@google.com

CS242, Stanford University, 10/9/06

10/9/06 2

Overview

• Brief history lesson• Python’s design and design process• Python’s implementation• Conclusion; questions

10/9/06 3

Timeline

• Project started, name picked - Dec 1989

• First public release (USENET) - Feb 1991

• python.org website - 1996 or 1997• 2.0 released - 2000• Python Software Foundation - 2001• …• 2.4 released - 2004• 2.5 released - 2006

10/9/06 4

Why I Designed Python

• ABC - teaching language created at CWI– designed in late 70s, early 80s (not me)– I was on the implementation team until 86– had my own thoughts about its failing

• Amoeba - distributed microkernel OS– almost completely, but not quite, unlike Unix

– only languages available were C and sh– I decided to try to bridge the gap

• Other influences: C, Modula-3, Icon, A68

10/9/06 5

Original Design Process

• Consider a desirable feature (either user feedback or my own needs while using it)

• Only if there is absolutely no way to implement it as a library module or C extension, consider changing the compiler

• Make the absolutely minimal changes necessary to the compiler; reuse existing functionality as much as possible

• This gave us explicit self, for example

10/9/06 6

Modern Design Process

• Pretty much the same, but also worrying about backwards compatibility

• Set the bar really high by only accepting new language features that will be of use to a wide variety of users

• Prevent repeating early mistakes, where sometimes shortcuts were taken that were too extreme, and had to be changed at great cost (e.g. int/long, classic classes)

10/9/06 7

Counterforce: Readability

• No hack so clever or it must be readable!

• Require at least one of:– Does it read naturally, like English?– Does it resemble well-known math notation?

– Does it resemble a similar feature in another (popular) programming language?

– Is it a logical extension of existing syntax?

• Of course this is highly subjective…

10/9/06 8

Community Process

• PEP: Python Enhancement Proposal• Like JSR (Java), RFC (IETF)

– Though a bit less formal (smaller community)

• Not just for language changes– Common APIs (e.g. db-API, WSGI)– Sometimes library design (often not needed)– Bookkeeping (e.g. release schedules)– Typically not for performance enhancements

• Not a democracy!

10/9/06 9

Original Design Goals

• Simple implementation (1 person)• Typical Very-High-Level Language goals• Cross-platform (hardware & OS)• Readability and expressive power• Easy to learn and remember; predictability

• Safety: bugs don’t crash interpreter• Don’t compete with C; complement it

– Extensibility through C extension modules– This makes Python an ideal glue language

10/9/06 10

“Standard” VHLL Features

• Automatic memory management• Small number of powerful data types• Bytecode interpreter• Built-in serialization (marshal, pickle)

• KISS

• Even indentation was borrowed!– Lots of other stuff, too– Python doesn’t have NIH syndrome :-)

10/9/06 11

Python’s “Big Ideas”

• No type checking in the compiler• Dynamic typing, static scoping• Everything is an object• Everything has a namespace (or multiple!)

• Everything can be introspected, printed• System has few privileges over user• Interactive prompt >>>• Simplicity of implementation still matters!

10/9/06 12

How Python Is Compiled

• Lexer -> token stream– Uses a stack to parse whitespace!

• Parser -> concrete syntax tree– Simple & stupid LL(1) parsing engine

• Filter -> abstract syntax tree (uses ASDL)

• Pass 1 -> symbol table• Pass 2 -> bytecode

– Bytecode is in-memory objects– Saving to disk (.pyc file) is just a speed hack

10/9/06 13

How Python Is Executed

• Virtual machine executes bytecode• Simple stack machine• Stack frames allocated on the heap

– C stack frames point to heap stack frames– Additional stack for try blocks

• VM calls out to “abstract” operations– e.g. “PyNumber_Add(a, b)”

• Some bytecodes represent “support” calls– e.g. import, print (since these have syntax)

10/9/06 14

Example Bytecode

def gcd(a, b):while b: a, b = b, a%breturn a

>>> dis.dis(gcd)

2 0 SETUP_LOOP 29 (to 32)

>> 3 LOAD_FAST 1 (b)

6 JUMP_IF_FALSE 21 (to 30)

9 POP_TOP

3 10 LOAD_FAST 1 (b)

13 LOAD_FAST 0 (a)

16 LOAD_FAST 1 (b)

19 BINARY_MODULO

20 ROT_TWO

21 STORE_FAST 0 (a)

24 STORE_FAST 1 (b)

27 JUMP_ABSOLUTE 3

>> 30 POP_TOP

31 POP_BLOCK

4 >> 32 LOAD_FAST 0 (a)

35 RETURN_VALUE

>>>

10/9/06 15

Separation of Concerns

• VM knows very little about objects– Just their abstract API– Occasionally “cheats” for performance hacks

•e.g. int+int, list[int], function calls– Some objects are “part” of the VM

•e.g. code, frame, traceback; not function!– Namespaces implemented as dictionaries

• Objects know nothing about VM• Support infrastructure ties things together– import bookkeeping, parser, interactive prompt

10/9/06 16

Object Header

• typedef struct { int ob_refcnt; PyTypeObject *ob_type;} PyObject;

• Example (in reality this uses macros):– typedef struct { int ob_refcnt; PyTypeObject *ob_type; long ob_ival; PyIntObject;

10/9/06 17

Type Object

• typedef struct { <HEAD> char *tp_name; … destructor tp_dealloc; printfunc tp_print; getattrfunc tp_getattr; setattrfunc tp_setattr; … } PyTypeObject;

10/9/06 18

Other Python Implementations

• Jython - compiles to Java bytecode• IronPython - compiles to .NET ILWhy?• Because we can!• Keeps Python language definition honest

– No implementation specific hacks allowed

• Side bet in case Java or .NET “wins”• Fills important gap on those platforms

10/9/06 19

Jython Details

• 100% Pure Java™ certified• Language differs in few details

– e.g. strings are always Unicode (UTF-16)

• Library differs where C code is wrapped– e.g. no Tkinter

• Fully automated wrapping of Java classes– thanks to Java’s reflection API

• Needs to load significant “Jython Runtime”• Executes 2-5x slower than CPython

10/9/06 20

IronPython Details

• Very similar in architecture to Jython

• Sometimes faster than CPython!• Supported by Microsoft

– but open source license!

10/9/06 21

Questions

• And answers

• PS skip to next slide now!

10/9/06 22

[Google Events at Stanford]

• Career Fair: October 10• Tech Talk: October 18• On-campus interviews:

– November (multiple dates)– Submit your resume by October 19

• Apply through:– www.google.com/jobs/students/fulltime

• Questions:– Davidson Young <davidsony@google.com>