Advanced Django

Simon Willisonhttp://simonwillison.net/

PyCon UK, 8th September 2007

Unit testing

newforms

Ajax

And if we have time... OpenID

Today’s topics

Unit testing

Hard core TDD

Test Driven Development as taught by Kent Beck

“Write new code only if an automated test has failed”

Revolutionises the way you write code

Pretty hard to adopt

“”

I don’t do test driven development. I do stupidity

driven testing... I wait until I do something stupid, and then write

tests to avoid doing it again.

Titus Brown

What NOT to do

No matter how tough the deadline, don't let your test suite start breaking and say “I’ll fix it later”

This will hurt. A lot.

Testing in Django

Testing web apps is HARD, but Django helps out with a bunch of features:

Fixtures

Doctests

Test Client

E-mail capture

Doctests

Used by Django for both ORM testing and generated documentation

You are encouraged to add them to your own models.py

manage.py test to execute them

Great for regression tests - just copy and paste from an interpreter session

>>> p = Person(name="Bob", dob=date(1980, 1, 1))>>> p.age(date(1980, 1, 1))0>>> p.age(date(1979, 1, 1))-1>>> p.age(date(1981, 6, 1))1>>> p.age(date(2000, 1, 1))20>>> p.age(date(1999, 12, 31))19>>> p2 = Person(name="Barry", dob=date(1981, 5, 5))>>> p2.age(date(1981, 5, 5))0>>> p2.age(date(1982, 5, 4))0>>> p2.age(date(1982, 5, 5))1>>> p2.age(date(1982, 5, 6))1

class Person(models.Model): """ ... tests here ... """ name = models.CharField(maxlength=100) dob = models.DateField() def age(self, age=False): return 1

$ python manage.py test

...F======================================================================FAIL: Doctest: peopleage.models.Person----------------------------------------------------------------------Traceback (most recent call last): File "/usr/lib/python2.5/site-packages/django/test/_doctest.py", line 2161, in runTest raise self.failureException(self.format_failure(new.getvalue()))AssertionError: Failed doctest test for peopleage.models.Person File "/home/simon/Projects/Django/oscon07/peopleage/models.py", line 4, in Person

----------------------------------------------------------------------File "/home/simon/Projects/Django/oscon07/peopleage/models.py", line 7, in peopleage.models.PersonFailed example: p.age(date(1980, 1, 1))Expected: 0Got: 1

def age(self, age=False): if not age: age = date.today() delta = age - self.dob return int(math.floor(delta.days / float(365)))

File "/home/simon/Projects/Django/oscon07/peopleage/models.py", line 16, in peopleage.models.PersonFailed example: p.age(date(1999, 12, 31))Expected: 19Got: 20

def age(self, age=False): if not age: age = date.today() years = age.year - self.dob.year this_year_birthday = self.dob.replace(year=age.year) birthday_has_passed = age >= this_year_birthday if not birthday_has_passed: years = years - 1 return years

Fixtures

It’s useful to be able to clear and populate your test database in between tests

Fixtures let you do that (they also let you populate your database with real data when you deploy your application)

Denormalisation

“”

Normalised data is for sissies

Cal Henderson

A forum, where each thread can have one or more replies

Maintain a separate counter in the Forum table of number of replies, to speed up queries

class Thread(models.Model): subject = models.CharField(maxlength=100) num_replies = models.IntegerField(default=0)

class Reply(models.Model): thread = models.ForeignKey(Thread) message = models.TextField()

[{ "model": "forum.thread", "pk": "1", "fields": { "num_replies": 0, "subject": "First thread" } },{ "model": "forum.thread", "pk": "2", "fields": { "num_replies": 1, "subject": "Second thread" } },{ "model": "forum.reply", "pk": "1", "fields": { "thread": 2, "message": "First post!1" } }]

from django.test import TestCasefrom models import Thread, Reply

class ThreadCountTestCase(TestCase): fixtures = ['threadcount.json']

def test_add_reply(self): thread = Thread.objects.get(pk=2) self.assertEqual(thread.num_replies, 1) thread.reply_set.create(message="Another post") thread = Thread.objects.get(pk=2) self.assertEqual(thread.reply_set.count(), 2) self.assertEqual(thread.num_replies, 2)

def test_delete_reply(self): thread = Thread.objects.get(pk=2) self.assertEqual(thread.num_replies, 1) Reply.objects.get(pk=1).delete() thread = Thread.objects.get(pk=2) self.assertEqual(thread.reply_set.count(), 0) self.assertEqual(thread.num_replies, 0)

======================================================FAIL: test_add_reply (forum.tests.ThreadCountTestCase)----------------------------------------------------------------------Traceback (most recent call last): File "/home/simon/Projects/Django/oscon07/forum/tests.py", line 16, in test_add_reply self.assertEqual(thread.num_replies, 2)AssertionError: 1 != 2

======================================================FAIL: test_delete_reply (forum.tests.ThreadCountTestCase)----------------------------------------------------------------------Traceback (most recent call last): File "/home/simon/Projects/Django/oscon07/forum/tests.py", line 23, in test_delete_reply self.assertEqual(thread.num_replies, 0)AssertionError: 1 != 0

----------------------------------------------------------------------

class Reply(models.Model): ... def save(self): super(Reply, self).save() self.thread.num_replies = self.thread.reply_set.count() self.thread.save() def delete(self): super(Reply, self).delete() self.thread.num_replies = self.thread.reply_set.count() self.thread.save()

.......-----------------------------------------------------------Ran 7 tests in 0.372s

OK

Django’s TestClient lets you simulate a browser interacting with your site

It also provides hooks in to the underlying application, so you can test against the template and context that was used to generate a page

Testing views

from django.test import TestCase

class RegistrationTest(TestCase): def test_slash(self): response = self.client.get('/register') self.assertEqual(response.status_code, 301) response = self.client.get('/register/') self.assertEqual(response.status_code, 200) def test_register(self): response = self.client.get('/register/') self.assertEqual(response.template[0].name, 'register.html') self.assertTemplateUsed(response, 'register.html')

def test_signup_done_page(self): self.assertEqual(len(mail.outbox), 0) data = { 'email': 'simon@example.com', 'username': 'example', 'firstname': 'Example', 'lastname': 'User', 'password': 'password1', 'password2': 'password1', } response = self.client.post('/signup/', data) self.assertEquals(response.status_code, 302) self.assertEquals(response['Location'], '/welcome/') # Check that the confirmation e-mail was sent self.assertEqual(len(mail.outbox), 1) sent = mail.outbox[0] self.assertEqual(sent.subject, 'Welcome to example.com')

http://www.djangoproject.com/documentation/testing/

More on testing with Django:

newforms

Display a form

User fills it in and submits it

Validate their entered data

If errors, redisplay form with previously entered data and contextual error messages

Continue until their submission is valid

Convert submission to appropriate Python types

The perfect form

Manipulators

Manipulatorsnewforms

from django import newforms as forms

class UserProfileForm(forms.Form): name = forms.CharField(max_length=100) email = forms.EmailField() bio = forms.CharField(widget=forms.Textarea) dob = forms.DateField(required=False) receive_newsletter = forms.BooleanField(required=False)

Forms are declarative

A form instance can...

validate a set of data against itself

render itself (and its widgets)

re-render itself with errors

convert to Python types

def create_profile(request): if request.method == 'POST': form = UserProfileForm(request.POST) if form.is_valid(): # ... save the user’s profile return HttpResponseRedirect('/profile/saved/') else: form = UserProfileForm() return render_to_response('profile.html', {'form': form})

Simple form handling view

form = UserProfileForm( initial = { 'name': 'Simon Willison', 'email': 'simon@simonwillison.net', } )

Initial data

<style type="text/css">ul.errorlist { color: red; }</style>

...<form action="/profile/create/" method="POST">{{ form.as_p }}<input type="submit" value="Submit" /></form>...

Simple template

Output<form action="/profile/create/" method="POST"><p><label for="id_name">Name:</label> <input id="id_name" type="text" name="name" maxlength="100" /></p><p><label for="id_email">Email:</label> <input type="text" name="email" id="id_email" /></p><p><label for="id_bio">Bio:</label> <textarea id="id_bio" rows="10" cols="40" name="bio"></textarea></p><p><label for="id_dob">Dob:</label> <input type="text" name="dob" id="id_dob" /></p><p><label for="id_receive_newsletter">Receive newsletter:</label> <input type="checkbox" name="receive_newsletter" id="id_receive_newsletter" /></p><input type="submit" value="Submit" /></form>

Output<form action="/profile/create/" method="POST"><p><label for="id_name">Name:</label> <input id="id_name" type="text" name="name" maxlength="100" /></p><p><label for="id_email">Email:</label> <input type="text" name="email" id="id_email" /></p><p><label for="id_bio">Bio:</label> <textarea id="id_bio" rows="10" cols="40" name="bio"></textarea></p><p><label for="id_dob">Dob:</label> <input type="text" name="dob" id="id_dob" /></p><p><label for="id_receive_newsletter">Receive newsletter:</label> <input type="checkbox" name="receive_newsletter" id="id_receive_newsletter" /></p><input type="submit" value="Submit" /></form>

<form action="/profile/create/" method="POST"><p><label for="id_name">Name:</label> <input id="id_name" type="text" name="name" maxlength="100" /></p><p><label for="id_email">Email:</label> <input type="text" name="email" id="id_email" /></p><p><label for="id_bio">Bio:</label> <textarea id="id_bio" rows="10" cols="40" name="bio"></textarea></p><p><label for="id_dob">Dob:</label> <input type="text" name="dob" id="id_dob" /></p><p><label for="id_receive_newsletter">Receive newsletter:</label> <input type="checkbox" name="receive_newsletter" id="id_receive_newsletter" /></p><input type="submit" value="Submit" /></form>

Output

Custom validationfrom django import newforms as formsfrom django.newforms.util import ValidationError

class UserProfileForm(forms.Form): name = forms.CharField(max_length=100) email = forms.EmailField() bio = forms.CharField(widget=forms.Textarea) dob = forms.DateField(required=False) receive_newsletter = forms.BooleanField(required=False) def clean_email(self): if self.cleaned_data['email'].split('@')[1] == 'hotmail.com': raise ValidationError, "No hotmail.com emails, please."

Custom validationfrom django import newforms as formsfrom django.newforms.util import ValidationError

class UserProfileForm(forms.Form): name = forms.CharField(max_length=100) email = forms.EmailField() bio = forms.CharField(widget=forms.Textarea) dob = forms.DateField(required=False) receive_newsletter = forms.BooleanField(required=False) def clean_email(self): if self.cleaned_data['email'].split('@')[1] == 'hotmail.com': raise ValidationError, "No hotmail.com emails, please."

Custom rendering

<ol class="formItems longForm"> <li{% if form.email.errors %} class="errors"{% endif %}> <label for="id_email">Email: </label> {{ form.email }} {{ form.email.errors }} <p class="info">Your e-mail address.</p> </li> ...</ol>

Model shortcutsDRY: You’ve already declared your models; you shouldn’t have to repeat yourself in your forms

UserForm = form_for_model(User)

###############################

page = Page.objects.get(pk=1)PageForm = form_for_instance(page)

form = PageForm(request.POST)...if form.is_valid(): form.save()

http://www.djangoproject.com/documentation/newforms/

django/trunk/tests/regressiontests/forms/tests.py

Full documentation:

Ajax

First things first

If you're going to do Ajax, you need a JavaScript library

You could use Yet Another XMLHttpRequest abstraction... but the popular libraries offer fantastic convenience

Good libraries include YUI, Dojo, MochiKit and (controversial) Prototype...

.. and jQuery

I'm going to be using jQuery

Almost everything is done in terms of CSS selectors and chained methods

It looks like a gimmick, but it isn't

http://simonwillison.net/2007/Aug/15/jquery/

Ajax formats...

Django has great support for any and every Ajax format

HTML fragments

XML

JSON

Username available?

from django.contrib.auth.models import User

def check_username(request): reply = "" username = request.GET.get('username', '') if username: if User.objects.filter(username=username).count(): reply = 'Unavailable' else: reply = 'Available' return HttpResponse(reply)

jQuery('span#msg').load( '/check_username/?username=' + input.val() );

$('span#msg').load( '/check_username/?username=' + input.val() );

var input = $('input#id_username') input.keyup(function() { $('span#msg').load( '/check_username/?username=' + input.val() ); });

$(document).ready(function() { var input = $('input#id_username') input.keyup(function() { $('span#msg').load( '/check_username/?username=' + input.val() ); });});

$(function() { var input = $('input#id_username') input.keyup(function() { $('span#msg').load( '/check_username/?username=' + input.val() ); });});

Recycling server-side form validation

from django import newforms as forms

class ContactForm(forms.Form): subject = forms.CharField(max_length=100) message = forms.CharField(widget=forms.Textarea()) sender = forms.EmailField()

def validate_contact(request): "Validate post data, return errors as json" form = ContactForm(request.POST) if (request.GET.has_key('field')): # Validate a single field errors = form.errors[request.GET['field']] else: errors = form.errors return JsonResponse({ 'valid': not errors, 'errors': errors })

from django.utils import simplejson

class JsonResponse(HttpResponse): def __init__(self, data): HttpResponse.__init__( self, simplejson.dumps(data), mimetype='application/json' )

function validateInput(input) { $.post('/contact/validate/?field=' + input.attr('id').replace('id_', ''), $('form').formToArray(), function(data) { var json = eval('(' + data + ')'); showErrors(input, json.errors); } );}

$(function() { $(':input').blur(function() { validateInput($(this)); });});

function relatedErrorList(input) { var prevUL = $(input).parent().prev(); if (prevUL && prevUL.attr('class') == 'errorlist') { return prevUL; } var errorlist = $('<ul class="errorlist"></ul>'); input.parent().before(errorlist); return errorlist;}

function showErrors(input, errors) { var errorlist = relatedErrorList(input); errorlist.empty(); $.each(errors, function(i, error) { errorlist.append('<li>' + error + '</li>'); });}

Django often gets marked down in “framework comparisons” due to the lack of built in Ajax support

Personally I think that shipping without a recommended library is a feature, not a bug

Django philosophy

(bonus section)

What is OpenID?

OpenID is a decentralised mechanism for Single Sign On

An OpenID is a URL

http://simonwillison.myopenid.com/

http://simonwillison.net/

http://swillison.livejournal.com/

http://openid.aol.com/simonwillison

How it works

You enter your OpenID on a site (instead of the usual username and password)

It redirects you back to your OpenID provider

They authenticate you in some way

They redirect you back to the original site

Simple registration(an optional but useful extension)

Your preferred username

Your e-mail address

Your first and last name

Your date of birth

Your language, country and timezone

Consumers can also ask...

How do you use OpenID in a Django

application?

Use the JanRain OpenID library

Pretty much a reference implementation for the OpenID spec

Well written, well tested but takes a while to get the hang of

www.openidenabled.com/openid/libraries/python/

The hard way

The easy way

Use django-openid

A simple wrapper around JanRain

A middleware component, some models and a few pre-written views

http://code.google.com/p/django-openid/

Installation

Add 'django_openidconsumer' to your INSTALLED_APPS setting

manage.py syncdb

Add the OpenIDMiddleware to your MIDDLEWARE_CLASSES setting

Add the views to your URLconf

In urls.py

...

(r'^openid/$', 'django_openidconsumer.views.begin'),

(r'^openid/complete/$', 'django_openidconsumer.views.complete'),

(r'^openid/signout/$', 'django_openidconsumer.views.signout'),

...

request.openid

The middleware adds an openid property to the Django request object

If the user is not signed in, this will be None

Otherwise, it will be an OpenID object; the str() representation will be the OpenID (or use request.openid.openid)

def example_view(request): if request.openid: return HttpResponse("OpenID is %s" % escape(request.openid)) else: return HttpResponse("No OpenID")

The module supports users signing in with more than one OpenID at a time

request.openids provides a list of all authenticated OpenIDs

request.openid merely returns the most recent from this list

request.openids

(r'^openid/$', 'django_openidconsumer.views.begin', { 'sreg': 'email,nickname'}),

def example_sreg(request): if request.openid and request.openid.sreg.has_key('email'): return HttpResponse("Your e-mail address is: %s" % escape( request.openid.sreg['email'] )) else: return HttpResponse("No e-mail address")

For simple registration

django_openidauth, providing tools to associate OpenIDs with existing django.contrib.auth accounts

django_openidserver, to make it easy to provide OpenIDs for users of your Django application

Coming soon

More information

http://openid.net/

Also home to the OpenID mailing lists

http://www.openidenabled.com/

http://simonwillison.net/tags/openid/

http://code.google.com/p/django-openid/

Thank you!