The State of Zope 3

The State of Zope 3Location: Plone Conference 2008, Washington, DCDate: October 10, 2008

Presenter: Stephan Richter, Keas Inc.

Changing Dynamics

The Pre-historic PastStarted development of Zope 3 in January 2002Used 42 Sprints to complete itRelease of Zope X3.0.0 on November 6, 2004Other Releases

Zope 3.1.0: October 2, 2005Zope 3.2.0: March 16, 2006Zope 3.3.0: September 27, 2006Zope 3.3.1: January 14, 2007

The Last 20 MonthsZope Explosion - Packages and the KGSAPI StabilizationGrokLots of Add-on Packages

(since February 1, 2007)

The Last 20 Months in NumbersContributors: 149 (total: 290)New Contributors: 77Checkins: 27187 (total: 91893)Total Amount of Packages: 581New Packages: 372 (exluding explosion: 283)Package Releases: 1331

The State of Zope 3 file:///Users/aclark/Desktop/StateOfZope3-PC08.html

1 of 8 8/3/09 10:03 AM

(Statistics include work on Zope 2)

Packages By PyPI CategoriesPackages in PyPI: 5199Buildout: 112 (2.2%)Plone: 375 (7.2%)ZODB: 9 (0.2%)Zope2: 328 (6.3%)Zope3: 617 (11.9%)Total Zope-related: 1441 (27.7%)

(Based on Trove Environment classifiers. Some packages might be double counted, if multipleenvironments were specified.)

Packages By NamespacesNew packages by namespace (since April 1, 2007)

cc : 2 lovely : 10 zam : 4gocept : 3 megrok : 12 zamplugin : 8grokcore : 4 plone : 2 zc : 32grokui : 2 vanguardistas : 1 zope : 28gzo : 2 z3 : 1 zopeorg : 2hurry : 6 z3c : 99 zopyx : 2jquery : 7 z3ext : 17 five : 5keas : 2

Top Contributors By Releasessrichter: 241 gary : 36 baijum : 13 shane : 7ctheune : 129 jens : 34 rossp : 12 schwendi:6andreasj : 87 faassen : 32 mkerrin : 12 shh : 5philikon : 82 dobe : 22 zagy : 11 nathan : 5jim : 82 berndroe : 19 hannosch:10 jackie : 5fafhrd : 72 benji_yo : 19 tlotze : 9 satchit : 4

The State of Zope 3 file:///Users/aclark/Desktop/StateOfZope3-PC08.html

2 of 8 8/3/09 10:03 AM

fdrake : 59 mgedmin : 17 pcardune : 9 nadako : 4rogerine : 58 batlogg : 16 janwijbr : 9 mauritsv:4jukart : 58 aaron : 14 ulif : 8 malthe : 4tseaver : 45 ccomb : 13 jinty : 8

Top Contributors By Checkinsphilikon : 1555 hannosch : 330 blackburnd : 243 ccomb : 111ctheune : 1405 wichert : 328 luciano : 237 gotcha : 109srichter : 1180 jens : 319 benji... : 235 thefu...:103andreas...:1163 mgedmin : 305 yuppie : 207 kteague : 102ulif : 1102 zagy : 301 adamg : 185 chrism : 102jim : 1099 gary : 300 regebro : 172 batlogg : 95rogeri...:756 darrylc...:297 mkerrin : 170 shane : 93faassen : 723 janwijb...: 285 alex : 170 witsch : 79malthe : 507 pcardune : 280 rossp : 164 nikhil_n : 78dobe : 460 jukart : 274 shh : 150 jinty : 78tseaver : 448 fafhrd : 257 tlotze : 143 icemac : 76fdrake : 433 baijum : 255 maurits... : 139 paran... : 75

(Data for the entire repository from February 1, 2007 to October 8, 2008.)

Eggification - An Explosive StoryGoal: Split up Zope 3 source tree into small, reusable packagesEnabled by packaging work: distutils and setuptoolsWork began in February 2006 (as far as I can tell)Final packages converted in March 2007

Explosive Hazards (1)No tools for project/package management

zc.buildout to provide a deployment solutionToday there are 112 buildout related packages on PyPI, mostly recipeszope3project/grokproject to assemble minimum setup

Unreliable package server

The State of Zope 3 file:///Users/aclark/Desktop/StateOfZope3-PC08.html

3 of 8 8/3/09 10:03 AM

The python.org PyPI server had frequent outages, stalling developmentz3c.pypimirror/zc.mirrorcheeseshopslashsimple provide scripts to produce mirrorsof the PyPI package index.

Explosive Hazards (1)Complex package inter-dependencies

Requiring one package pulls in about 60 other packages.It's very tedious to unscrew dependenciesTools like gtkeggdeps help with discoveryzc.configuration (<zope:exclude> directive) and ZAM (zam.* and zamplugin.*)help

zc.configuration enables a developer to ignore all browser-related ZCML declarations of apackage, thus usally reducing the list of dependencies. ZAM, on the other hand, is a pluggablereplacement for the management aspects of the Rotterdam skin, which reduces the dependencytree drastically.

No version management

A typical Zope application uses about 200 packagesPackage-tests themselves have about 80 packagesEveryone uses different sets of package releases for tests and application developmentVersion-dependencies are hard-coded in setup.py causing version conflicts and stalleddevelopmentCaused huge disarray in the community in October 2007

The Known Good Set (KGS)Goal: Provide a standard set of packages and their versions that is guaranteed to work togetherServes as a base for everyone to develop on

A starting point for custom applicationsWorks together with existing infrastructure and software

Creates an index with limited version availability of controlled packagesCreates an index containing only the controlled packagesCreates "version"-block for usage in buildout configurationCreates a package links pageCreates a complete buildout configuration for testing

Zope 3 KGS

The State of Zope 3 file:///Users/aclark/Desktop/StateOfZope3-PC08.html

4 of 8 8/3/09 10:03 AM

A KGS for all Zope 3 related packages maintained in http://svn.zope.org

Introduction URL: http://download.zope.org/zope3.4/intro.html

Zope 3's KGS serves as a base to Grok's or Plone's KGS (for example)

Managed Packages: 170

Combined tests: 12217

Currently all tests are passing: http://zope3.pov.lt/buildbot/

(except for some random non-deterministic ZODB-related failures)

Plans for Zope 3.4Yes, there is a plan!

Work on a Zope 3 KGS based release since November 2007 (3.4.0b2)

Most recent release: September 5, 2008 (3.4.0c7)

Commitment to produce a tree-based source release

We have a script to do this for us.

No commitment on a time scale due to lack of interest

Zope 3.4 ReleaseWork done

Create a KGS with all tests passingRelease processBackward-compatibility

Work to be doneScript to produce change logTweaks to the KGS site (i.e. include dates in release, improve wording, maybe a skin)Cut the release and announcement

TrendsStabilityPerformanceScalability

The State of Zope 3 file:///Users/aclark/Desktop/StateOfZope3-PC08.html

5 of 8 8/3/09 10:03 AM

PortabilitySecurityDeployment

StabilityOnly few enhancements to the core packages' feature setsLots of bug fixesSeveral buildout recipes provide features to maintain code

Performancelovely.responsecache / z3c.versionedresource / z3c.resourcecollectorZODB BLOB supportzc.zodbactivitylog / z3c.zodbtracing / z3c.profilerRelStorage (RDB storage backend for ZODB)ZODB Object Cache improvementsz3c.pt from Malthe Borch improves rendering 18-20 times

Keas pays Malthe to port z3c.form*, z3c.template, z3c.pagelet, z3c.macro to z3c.pt

Scalabilitylovely.memcached / lovely.session (memchached integration)lovely.remotetask / zc.async (async-based task execution)gocept.zeoraid (A RAID controller system for ZODB)z3c.extfile (Storing files external to Zope App)Soon: keas.messaging, an Amazon Simple Queue Service connectorSoon: keas.sharding, a sharding solution for the ZODB

PortabilityGood support for WSGISupport for PasterGoogle Summer of Code project to port Zope libraries to JythonInitial Google App Engine work

Securityz3c.password (Advanced password generation and verification utility)

The State of Zope 3 file:///Users/aclark/Desktop/StateOfZope3-PC08.html

6 of 8 8/3/09 10:03 AM

z3c.tan (Temporarily granting additional permissions)z3c.securitytool (Discovery of security settings)gocept.registration (Hash-based registration confirmation)keas.kmi (Key Management Infrastructure - PCI and HIPAA compliant)Unfortunately Common Criteria certification cancelled

Deploymentbuildoutmany recipeszc.sourcerelease (creates complete archives)

Relevance to PloneAll Zope 3 packages can be used within Plone

some integration (Five) code might be neededZope 3 packages usually do not have a UIfor example: z3c.form --> plone.z3cform

buildout also changed the deployment story for PloneScalability and performance tuning benefits Plone directly

The FuturePackage dependency resolutionEasier entry into Zope 3 worldStablize core furtherConcentrate on scalabilityBuild more add on packages

ChallengesRich client-side apps lessens importance of heavy app serversOpportunity and ChallengeWeb 2.0 and RESTMuch more JS support neededZope becomes a middleware serverDocumentation

In case you wonder ...

The State of Zope 3 file:///Users/aclark/Desktop/StateOfZope3-PC08.html

7 of 8 8/3/09 10:03 AM

I joined a startup called Keas, Inc.Preliminary site: keas.comWe are in stealth mode, so I cannot give many specifics yetWe are looking for Zope 3 and other Web developers in SF and NY

Questions and Comments

The State of Zope 3 file:///Users/aclark/Desktop/StateOfZope3-PC08.html

8 of 8 8/3/09 10:03 AM