Diapositiva 1

Roberto Polli, Solutions ArchitectPycon7 - Firenze, 16.04.2016

Tox as project descriptor: not only Continuous Integration

Who? What? Why?

Roberto Polli - Solutions Architect @ par-tec.it. Loves writing in C, Java and Python. Red Hat Certified Engineer and Virtualization Administrator.

Par-Tec Proud sponsor of this talk ;) provides expertise in IT Infrastructure & Services and Business Intelligence solutions + Vertical Applications for the financial market. Contributes to various FLOSS.

Use tox to describe python projects with code.

HI everybody! My name is Roberto Polli, solutions architect @par-tec. I love writing free software and I got a couple of RH certification.I'm very happy to be here - in a way or in an other ;) a concrete sign that pycon.it is more than a conference. It's community and friendship. Before beginning I'd like to thank the proud sponsor of this talk - par-tec - that is present here with 4 people.Par-Tec provides expertise in IT infrastructures, BI solutions and Applications for the financial market.

Agenda

Who? What? Why?

Continuous Integration: not only a java stuff

Project Description

Isolating projects: virtualenv, maven

Enters Tox a light project descriptor

Python 3, Flake, Packaging

Docker / Jenkins integration

Tox vs Travis.io

Today I'll speak about using tox to document projects with code.We'll start introducing CI and how it's addressed in the java world, which includes an overview on project description.Then we'll show how to isolate python projects with virtualenv and how to use tox for testing and project description: from python versions, to dependencies and code style.Finally a fast glance on using tox with Docker and Jenkins and an example of a travis.io file.

Continuous Integration

Merging all developer working copies several times a day

CI requires a project description and isolated builds.

Everything should be Reproducible

Test Automatically that integration works

CI is the practice of merging developer works several times a day.All this merge work requires runnable project descriptions, and a high degree of isolation, so that every step is reproducible and deterministic.We should test automatically that the merge works.

Continuous Integration

Merging all developer working copies several times a day

CI requires a project description and isolated builds.

Everything should be Reproducible

Test Automatically that integration works

CI is the practice of merging developer works several times a day.All this merge work requires runnable project descriptions, and a high degree of isolation, so that every step is reproducible and deterministic.We should test automatically that the merge works.

Continuous Integration

CI is not an exclusive of the "java/enterprise" world...

Python tools for project description, testing and CI

tox

CI has a solid ground in the java/enterprise world. Java developers are well acquainted with tools like jenkins - a CI server, arquillian - an ITTest library, openshift and so on.You can do CI with python too, thanks to a set of tools, including the TravisCI PaaS, Buildbod, tox & co. Jenkins - though written in java - can execute python tests.

Project Description

Describe carefully to Test continuously

maven Project Object Model: an established standard in the java world

POM describes and implements the whole software lifecycle:build, dependency downloads, Unit Test (java version, deps, ...)

packaging, local installation, Integration Test

docs, deploy on application server, release on stage

release on remote repositories (distribute)

To ensure that a software works nicely, we need to describe carefully every step. The java world has an established standard, the Project Object Model used by maven.maven POM describes and implements - via a DTD resulting in an XML file - every step of the software development lifecycle.

Project Description

MAVEN SUCKS but...

Though hated for the extremely complex description of the xml file and the slow build system, maven surely achieves a goal.

Project Descriptors

Maven enforces a standard workflow

One tool to build test deliver

Plugins for almost everything

Project descriptors with python

Python world is more fragmented:setup.py for distributable packages

pip to install packages (and easy_install?)

dependencies in requirements.txt and setup.py install_requires

setuptools to build|distribute (but distutils-only too)

egg, wheel, zip and source archives

pypi as a central repository

Where is my pom.xml ?

The python world is more fragmented, we have a wide variety of tools that changed in time - without considering python3 migration.

Distributable packages like libraries or projects on pypi, use the setup.py file to store project metadata;

Package installation happens via pip - that doesn't provide conflict resolution [todo].Pip read read dependencies from requirements.txt and the install_requires variable of setup.pyThe setup.py can then use the new setuptools library or its "subset" distutils (use setuptools!).Packages archives are distributed via egg (old, without metadata [todo]) or wheel (new) archives, but the wheel ones don't work on windows binaries. Then there are zip and source archives.A Java developer approaching python asks..."Where is my pom.xml"?

Isolating projects with virtualenv

Reproducible builds require isolation

Java doesn't have system libraries

Separate builds from system python

>>> print(sys.path)[ '','/usr/lib64/python2.7','/usr/lib64/python2.7/lib-dynload', '/usr/lib64/python2.7/site-packages' ]

# virtualenv -p python3 /tmp/py3
Running virtualenv with interpreter /usr/bin/python3New python executable in /tmp/py3/bin/python3Installing setuptools, pip, wheel...done.

Reproducibility requires isolation: while java packs all its dependencies together with the application, python has centralized archives for libraries. This is great for optimizing space, but not for isolation. Virtualenv is a tool that:installs a new python environment in a separate path [rpolli@rpolli tox-talk]$ virtualenv -p /usr/bin/python3 /tmp/py3 Running virtualenv with interpreter /usr/bin/python3Using base prefix '/usr'New python executable in /tmp/py3/bin/python3Also creating executable in /tmp/py3/bin/pythonInstalling setuptools, pip, wheel...done.Virtualenv

Describe projects with tox

tox is a virtualenv manager:create separate python environments

install dependencies via pip

may run setup.py

execute specific commands (py.test|nose)

Describe projects in tox.ini

Give java developers an entrypoint to your application

Tox is a virtualenv manager that:create one or more virtual env;install dependencies via pip, reading install_requires (setuptools) and requirements.txtinstall local packages when setup.py is providedrun specific commands - usually tests

As tox should be able to run test commands, we can use its configuration file as an entrypoint for project description.It's not as complete as pom.xml - we still need to read external files like setup.py - but it's a good starting point for python newcomers.

Enters tox

START WITH AN HOWTO

## tox.ini is the tox configuration file## Tox runs test in separate virtualenvs.## To download and install dependencies, # build software and run tests, execute:# #tox [-v]## Show the supported python versions via # #tox -l### To test only one python version, use -e# #tox -e py27#

Always start tox.ini with a comment with the run instruction. This is a file for a simple application (a library would have required a setup.py). In four lines we:specify the supported python versionreference dependencies and the source path how to run the tests.

Enters tox

[tox]envlist = py27# Supported python versions.skipsdist=True# skip source distribution # (aka setup.py)

[testenv]# Global settings.setenv =# The pythonpath PYTHONPATH = :.:./src

deps= # Install dependencies. nose flask

commands=# Finally run all the tests nosetests {posargs}# passing further args to nose# eg. via tox -- --verbose

This is a tox.ini for a simple application.We declare the supported python version (2.7) and - via the skipsdist - that we don't have a setup.py.A tox.ini for a library would have set skipsdist=False - the tox default.The application is entirely contained in the local src/ directory specified via the PYTHONPATH and depends onnose and flask - a simple web framework you probably know.Note the {posargs} token: contains all the arguments passed to tox after the -- (double-dash).

Enters tox

[tox]envlist= py27toxworkdir=/tmp/myprj/.tox# tmpfs is fast ;)

[testenv]...deps= nose# Add further deps -rrequirements.txt# here..

whitelist_external=# ..and os-depended /usr/bin/make# executables /usr/bin/gcc

commands=# Set default args nosetests {posargs:-w tests}

Put wheels under your feet!

SAVE TIME & CPU

PEP 427

setuptools >= 0.8

pip can populate wheel cache

Labels eg linux_x86_64 py3 Package retrieved from cache

Wheel is a binary packaging format described in PEPxxx.Instead of continuously recompile packages, we can just download them from a wheel repository. To do this we need to specialize the install_command, which defaults to "pip install" and add the --use-wheel option.If a package is not present in the wheel cache, pip will place it there after the build.Obviously to compile pandas we need (something more than) make and gcc.

Put wheels under your feet!

SAVE TIME & CPU

[tox]...

[testenv]...deps= # This is going to take pandas# a lot of time! nose

# Override default pip command and# use cached packages.install_command= pip install --use-wheel {packages}

commands=# Run tests. nosetests {posargs:-w tests}

Compatibility with py3

[tox]envlist = py27,py34 # Add another environment...skipsdist=True

[testenv]...deps= -rrequirements.txtcommands=# Next stanza does notnosetests# override the # 'commands' setting

[testenv:py34]# Override defaults# requirements for py3.deps= -rrequirements-3.txt

Ok, we have our - fast deployed - python2 project and want to be ready for python3.The first step is to *install* a python3 distribution on our operating system: tox doesn't install system packages, just use them to create a separate environment in another directory, set the PYTHONPATH and so on.Now we're ready to add python 3.4 to our environment list - see the second line. And we probably need to define a different set of dependencies. This is done creating a new stanza labeled with the environment name.Processing py34, tox will cascade then, using:deps from [testenv:py34]commands and other settings from the default [testenv]Clearly we could have moved "deps" from the default [testenv] to a [testenv:py27] stanza.

Compatibility matrix

[tox]# Generate 4 environments:# py27-pd17, py27-pd18,# py34-pd17, py34-pd18,

envlist = py{27,34}-pd{17,18}[testenv]basepython=# Now we *must* py27: python2.7# define the pyXX py34: python3.4# aliases

deps= # You can pin one numpy==1.10.0# dependency pd17: pandas>=0.17.0,