Responsive Design: principles for effective technologicalsupport of collaborative education research

Paper Presented at the Conference of the American Educational Research Association

San Francisco, April 5th-11th, 2006

Richard Procter and Patrick Carmichael

Centre for Applied Research in Educational Technologies, Cambridge University

Abstract

The ‘Learning how to Learn’ project of the Economic and SocialResearch Council (ESRC) Teaching and Learning ResearchProgramme in the UK has, over its four-year lifespan, employed arange of electronic tools in support of its work developing formativeassessment practices in primary and secondary schools through aprogramme of development and research. These have been developedthrough an iterative process of ‘use-case analysis’ involvinguniversity-based researchers and teachers, and made possible by theinvolvement of software developers within the research team. This hasallowed the development of a number of specialized tools (for datacollection, analysis and user engagement in and with the work of theproject). These have reflected the design and needs of all projectmembers at different stages in its development, but have also beeninformed by a set of underlying design principles which have hadapplication to other projects and research contexts.

2

Introduction: The Learning How to Learn Project1

‘Learning How to Learn – in classrooms, schools and networks’ was a four year (2001-2005)development and research project within the ESRC Teaching and Learning ResearchProgramme (TLRP). It sought to develop understandings of the organizational conditions ofschools, and across networks, in which teachers are successful in developing pupils'knowledge and practices in learning and in learning how to learn through assessment forlearning. This reflected our aspiration to bring insights from classroom-level and school-levelresearch into closer alignment with one another. James, Pedder, Black, Wiliam andMcCormick (2003) argue that such alignment offers our best chance of furtheringunderstanding of effective learning, its nature, the teaching practices that promote it, and thekinds of professional learning and institutional conditions that help teachers to adopt newpractices. These avenues of investigation constitute an enormous range of inquiry and, sinceearlier reviews of research (Black and Wiliam, 1998) indicate a ‘poverty of practice’, therewas a perceived need to stimulate and support innovation through development. For thesereasons, it was necessary to bring together a large team of academic staff and researchers withdifferent areas of substantive and methodological expertise and willingness to take on avariety of roles.

Previous research on assessment for learning had been relatively small scale, so another areaof interest was to investigate the sustainability of innovations, which were developed inauthentic settings with modest levels of support, as they were adapted and spread acrossteachers and schools. This required the team to recruit a sizeable sample of schools and towork in partnership with head teachers, school coordinators, ‘focal’ teachers who wereidentified for detailed study, and Local Authority (LA) advisers. By the time of its conclusion,the project had involved 17 academics and researchers from five Higher EducationInstitutions and 40 infant, primary and secondary schools in five LAs and one 'VirtualEducation Action Zone' (VEAZ - a novel organizational form, based on the 'Education ActionZones' established in the UK but involving geographically distributed schools linked byelectronic networks). An advisory group, including primary and secondary teachers and LArepresentatives, also supported the project. This number and range of participants, and thescope and locations of project activities, presented the project team with a number ofchallenges, not least the need to develop and maintain internal and external communicationsfor various purposes

Not only was it important to ensure that team members were aware of project progress in allits dimensions, in order to avoid fragmentation, omission and duplication of effort, but thehighly integrated design required that different researchers in different sites for differentpurposes would need access to a centralised ‘data pool’. Alongside these internalrequirements there were also the usual demands for information about the project, and its

1 The 'Learning how to Learn' project (ref: L139 25 1020) was directed by Mary James (University ofCambridge until December 2004, then at the Institute of Education, University of London) and co-directed, from2002, by Robert McCormick (Open University). Other members of the research team were Patrick Carmichael,Mary-Jane Drummond, John MacBeath, David Pedder, Richard Procter and Sue Swaffield (University ofCambridge), Paul Black and Bethan Marshall (King’s College London), Leslie Honour (University of Reading)and Alison Fox (Open University). Past members of the team were Geoff Southworth, University of Reading(until March 2002), Colin Conner and David Frost, University of Cambridge (until April 2003 and April 2004respectively) and Dylan Wiliam and Joanna Swann, King’s College London (until August 2003 and January2005 respectively). Carmel Casey-Morley and Nichola Daily were project administrators. Further details areavailable at http://www.learntolearn.ac.uk.

3

outputs, from external audiences. The project team never underestimated the value of face-to-face contact in distributed teams in order to develop both intellectual and social capital somonthly whole team meetings, twice yearly advisory group meetings and regular sub-groupmeetings were the norm. As time progressed so dissemination talks to potential ‘user’ groupsalso increased. However, the sheer number of communications and the quantity of the datameant that this was a prime case for use of electronic tools. It is important to stress that, fromthe outset, members of the project team saw the role of electronic tools not only as a means ofsupporting the management of the research activities, but also as a means of establishing linkswith teachers in participating schools, LA representatives and other 'publics' interested in theconduct and findings of project.

The Project Schools: Some Important Early Findings

As we set about selecting and developing the electronic tools to support the project, we had totake into account a set of early findings from schools involved in the project, including thosethat were members of networks and of the VEAZ. On the whole, these schools were wellequipped with computers (the vast majority of which were connected to the internet) andteachers reported that they regularly used new technologies in their planning or teaching(nearly 80% of primary and secondary teachers reporting using the internet to find lessonresources and about half complete 'schemes of work'). However, in contrast, only about 20%reported using the World Wide Web to find information about development of teachingpractices: clearly an important finding since this was the focus of project activities!

In addition, there was little evidence of teachers contributing to networked resources (either atregional or local level) and few schools (even those in the Virtual Education Action Zone)reported establishing or using school-to-school electronic links. Another general pattern wasthat, even in those schools that had extensive electronic networks, electronic means ofcommunication were generally used to supplement or reinforce face-to-face meetings, papercommunications and telephone conversations. In one secondary school with nearly 650computers connected to a school network, the ratio of pupils to computers was less than 2 andall staff had wireless laptops for planning, administration and communication; despite this,staff still met for daily 'briefings' and announcements made by email were also disseminatedvia newsletters, memos and notice-boards (these findings will be reported in more detail in aforthcoming paper).

In the light of these findings, we realized that we could make no assumptions about:

• The ability or willingness of participating teachers to see an online resource such as aproject website as an appropriate source of information about the development ofteaching practices – the issue with which the project was most concerned!

• The likelihood of teachers contributing to a central 'resource bank', 'knowledgeexchange' or other central resource, even if they had found it a useful source ofinformation

• The likelihood of schools – even those in existing networks – sharing practice solelyvia electronic means

As such, we recognized the importance of employing a 'blended' approach to supporting andresearching development activities in schools. This would need to employ onlineenvironments and email alongside telephone and face-to-face meetings, and, moreimportantly, to be sufficiently responsive and flexible that it could adapt to different school

4

contexts, staff preferences and changing circumstances. To have built a significant element ofour research design or our plans for user engagement around a specific (and possibly a novel)technology might have 'raised the bar' for participation to the extent that individual teachers orwhole schools might be precluded from participation in the project as a whole.

So with this twin set of concerns (the needs of the project team and the need to engage withschool-based participants) in mind, we engaged in a series of ‘use case analysis’ activitiesinvolving both the project team and the project advisory group, with its teacher and LArepresentatives (see Carmichael, 2003, for a full account of this process). Our central roles inthe project, and our involvement in the full range of project activity (rather than, for example,being employed to develop software on a consultancy or contract basis) allowed a muchcloser association between the project design and its realization, on the one hand, and ourdevelopment of software to support it. Rather than working to develop software in response toa specification document or from a project plan, we were party to and frequently involved inthe discussions that generated them; when we encountered problems or decisions had to bemade, we were able to consult project members, either individually or collectively. As therole of electronic tools became more central to the project, they – and the process by whichthey had been developed - were also subject to the scrutiny of the project’s advisory group.

While participatory approaches are now becoming widespread in software design(Gottesdiener, 2002; Carmichael, 2003), this combination of iteration, flexibility andaccountability goes further than most participatory design processes. It allowed us to developa process of what we now characterize as ‘responsive design’ – responsive not only toimmediate needs of the ‘users’ but also to their often domain-specific concerns, approachesand theoretical frameworks. Wenger, in his survey of software supportive of community-oriented behaviour suggests that solutions generally arise through a process of 'combinationand convergence' (2001, 5); while we would not suggest that (even at the end of the project)we had reached a 'solution', the iterative and responsive process in which we were involvedprobably matched the needs of project participants more than any pre-determined strategymight have done.

Responsive Design: our Principles

The most obvious manifestation of the electronic tools we developed to support the project isits website (http://www.learntolearn.ac.uk). This has evolved over the life of the project tofulfil a number of roles including:

• the provision of information about the nature and scope of the project to bothparticipants and the wider public;

• provision of classroom and staff development materials for use by LA and schoolcoordinators involved in the project;

• data collection, storage and analysis• provision of project management and data analysis tools for use by the research team;• sharing exemplification of developing practice in classroom assessment.

Central to these roles is an innovative content management system that builds web pages andother types of outputs from data components held on a web server: a typical page viewed by auser might actually represent many small components embedded in a template. Thesecomponents represent the basic units of communication within the project; the largest arecharacteristically documents constructed in other applications (such as Microsoft Word

5

documents and PowerPoint presentations) with the smallest being single bibliographicalreferences, dates and times or email addresses.

Underpinning our development were a number of design principles that allowed us to respondto specific and in some cases very detailed demands and specifications from both researchersand participating teachers and LA representatives. We will outline these below: a moredetailed and rather more technical account is presented in Carmichael, Procter and James(2005). As the project developed, these principles allowed us to respond to changes inpatterns of engagement and to adapt to the 'use cases' that project participants presented,paralleling the iterative, participatory approaches employed across the project as a whole.Once again, this is a novel approach in that the software tools and resources were developedalongside other elements of the project, rather than being developed and deployed from theoutset. In this respect it closely resembles the relationship between 'technology' and'community' described by Verwijs, Mulder, Slagter & Moelaert (2002) in which they arguethat while initially at least “technology support should match the primary activities of thecommunity, its size, goals, and the preferences of community members”, the availability ofnew technologies “may inspire [community members] to explore new interaction styles, thusimproving the performance or efficiency of the community”(60-61).

Design Principle 1: Small is Beautiful

Small components (such as online diary entries – of which more shortly, bibliographicalreferences, personal contact information, descriptions of examples of classroom practice) canbe assembled in lots of interesting ways, and it is easier to aggregate content thandisaggregate it In many cases, however, a fine balance had to be struck between maintainingmeaningful units of analysis and allowing small components to be rearranged in new ways.With the information divided into its smallest descriptive components, we could developsoftware tools to organize, arrange, sort, search and display these components in a number ofdifferent ways.

Design Principle 2: Embedded Markup Considered Harmful2

Separating content from formatting allows varied data to be presented in different forms andformats. Had online diary entries been stored as Microsoft Word documents, at some pointwe would have had to extract the text from its formatting so that it could be imported intoqualitative data analysis software packages, for example. Similarly, rather than writing webpages within which content, structure and formatting information were combined, we storedall text content as fragments of XML (extensible markup language) which were thenincorporated into page 'templates'. So, for example, individual user diary entries could beincorporated into individual diaries, per-school reports or cross-project reviews. Whatappeared to be whole new sets of functions, or significant changes within the website, were,in fact, relatively quick and simple to achieve.

Design Principle 3: Reduce, Reuse, Recycle

The componentisation of the web content and project data more generally also allowed us tomaintain single, authoritative versions of content; they might be reused across the website inmultiple formats, but only existed in one 'place' online. This overcame the tendency forindividuals in collaborative groups all to have their own, slightly different, versions of

2 The title of an important paper by Nelson, T. (1997).

6

documents. We also applied this principle to the code we wrote to produce the web pages –wherever possible we reused program code and techniques, allowing us to build newtemplates and 'views' of data quickly in response to emerging user requirements.

Design Principle 4: Conformance with International Standards

Throughout our development work we have adhered to international standards for webresources, metadata, dates and times, and accessibility. This conformance means that not onlyare our applications capable of accessing other resources, but also that our website contentcan, where appropriate, be made accessible to a wider (even global) audience. This clearlyhas implications for research projects wishing to engage with the widest possible audience.

The ‘responsive’ approach and the principles outlined above allowed us to address (or at leastraise) a range of further issues relating not only to technical support but also to projectconcerns more generally. With the underpinning principles in place, we found it easier toaddress the 'use-cases' which the users of the website raised; they were not a substitute forinvolving project team members, but rather provided a basis upon which we could planresponses to (sometimes surprising) requests and suggestions. As project membersthemselves became more familiar with the principles underpinning our designs, they couchedtheir requests in terms of rearranging components or providing different 'views' of them.

Responsive Design in Action: Online Research Logs

One of the key expectations was the engagement of key staff in participating schools (‘schoolcoordinators’) both with and in the ideas and development activities of the project. To thisend, each school had an assigned member of the project team whose role was to provide'critical friendship' (Stenhouse, 1975; Costa and Kallick, 1993). This meant that we as aproject we needed to develop strategies to support a triangular relationship involving projectresearchers, critical friends and school coordinators. At the beginning of the project, theexpectation was that logs would take the form of structured Microsoft Word formsdownloaded from the website; these would be used by school coordinators to record projectactivities and any school-based developments. At the same time, the researchers and criticalfriends would keep diaries (paper or electronic) of their interaction with schools,observations, administrations of research instruments and other information.

School coordinators and 'critical friends' alike were concerned that this system was overlydependent either on collection of paper materials from schools and, more importantly, that itallowed little opportunity for dialogue between the different participants. An obvioussolution was to use email, but this would have necessitated careful management to ensure thatimportant research data was not lost; we also discovered that many of the school coordinatorsdid not have personal email addresses as LA email systems allowed only a limited number ofemail addresses per school and barred staff from using webmail systems such as 'Hotmail' inschool.

Our response to this was to develop a web-based ‘log’ tool to which school and LocalAuthority coordinators within the project and Project members based in Higher Educationinstitutions could contribute reports on the progress of the project in the settings in which theyworked. The design of the logging tool reflected our design principles; log entries werestored as small components for aggregation (principle 1) which were devoid of any formattinginformation (principle 2). The log entries could be viewed individually, as part of adeveloping school record, or to help project managers maintain an overview of project

7

progress as a whole. When data analysis began, the school logs could be output in formatssuitable for qualitative data analysis using packages such as Atlas/ti (principle 3). Log entrydata was stored using well-established standards such as vCalendar (for date information) andDublin Core (for broader descriptive information) and web pages produced were W3Ccompliant, in accordance with principle 4.

Our early version was little more than a web form replicating the original word documents,with drop-down lists allowing users to select their name, the school name and some broadcategories of activity (preliminary visits, in-service training, workshops, data collection,classroom observation etc), together with an area allowing 'free text' entry. By readingthrough the logs before any meeting with a school coordinator, the critical friend would beprepared and ‘brought up to speed’ about school-based initiatives or research activities in thatschool, allowing coordination of the development and research aspects of the project.

Figure 1: The 'Log' interface after revisions suggested by participants

But how did this online tool respond to developing project requirements and user comments?Almost immediately, participants requested a number of additions and adaptations, mostsignificantly the provision of separate spaces for 'reporting' activity and for more reflectiveaccounts and the development of dialogue between the various individuals concerned (seeFigure 1). Some participants used this to maintain links with critical friends and researchers:

“School Coordinator still to be decided. Handbook has not been received - could we haveone please?”(Log Entry by Head Teacher, School LDW, 12-07-2002)

Others used it to report on development activities and their impact, and to identifyorganisational factors and barriers to their progress:

8

“Meetings with Eng. Dept have resulted in discussion and review of Dept. Marking policyand, as a result, a revised marking policy, where pupils have greater control over themarking - peer marking implemented and trialed. The decision was made to not grade everypiece but to give formative comments instead. Finally… pupils were given marking policyand key to have displayed and to use in front of books and a target sheet, to record targetsset as a result of formative comment, to be kept in the back of the exercise books. Pupilsonly will be given a National Curriculum grade on a termly basis - this was a compromiseafter discussion with school.” (Log Entry by School Coordinator, School SEK, 18-10-2002)

[Workshop] was generally very successful. However, some … staff still need to be madefamiliar with the basics of Assessment for Learning. Questionnaires will be done herewithin the next fortnight, probably after [the school inspection] … (Log Entry by SchoolCoordinator, School WHE, 06-01-2003)

Others used the log effectively as a reflective diary to which they contributed regularly, eitherreporting on their experience of project activity or their own professional learning:

“I found describing the class quite difficult and wondered why this was. Perhaps I think ofthem too much as 'levels' of learning due to the focus on assessment. We have had thisdiscussion in the staff room. Are we thinking too much about children as a 1c or 2a? Alsothere are so many individuals within a class which you are focusing on as you teach. Do Iever think in 'whole class' terms? I'll keep a note of when I do.” (Log Entry by SchoolCoordinator, School KNG, 21-05-2003)

“So used to receiving feedback and discussing the lesson - found it strange just to give myopinion of how it went. I wanted to know how it went as far as [the researcher] wasconcerned and what [they were] focusing on. I wonder if children feel like this if we do notcomment on their work.” (Log Entry by School Coordinator, School KNG, 21-05-2003)

For school-based participants in particular, the increased flexibility allowed them to engage inmore depth with project activities, and 68% of all participant log entries comprised not onlydescriptive but reflective comments, although many reported organisational issues rather thanprofessional learning issues. In contrast, researcher log entries were more likely to bedescriptive accounts or reports of activities 'completed' (70%). Critical friends used the logsin a variety of ways and wrote both for a project audience and directly to school coordinators:

“I was acutely aware that this is an infants school and the examples I had to hand weredrawn from secondary practice. However, teachers were very ready to offer analogousexamples from their own pupils. For example misunderstanding that children experienceover the different meanings of words … The two examples of children's work brought byteachers …may be worth capturing as examples for the L2L website and to provideexamples from work with very young children in relation to this workshop.” (Log entry byCritical Friend, School KNG, 30-09-2002)

They also used the log to offer advice, support and references to sources of information. Thewide variation in the nature of the log entries in is keeping with the rather varied findingsreported from other studies as to whether contribution to online environments encouragesreflective thought – or even accounts of reflective practice. Maxwell et al. (2001, 10) reportthat while there was some evidence of university-based staff and other teachers developing‘person-orientated’ critical friend roles in an online discussion space, there was, however,‘little evidence of the critical analysis associated with the notion of “critical friend”’. In othercases, (for example, Hoel and Gudmundsdottir (1999)) where the purpose and parameters ofself-criticism and analysis were, perhaps, more clearly defined, online environments seem tohave been more effective at encouraging reflective writing.

We were aware that some school coordinators and some critical friends chose not to use thelogs, preferring to maintain paper diaries and communicate by telephone or email:

9

I’ve hopeless at filling in the document thing that we’re supposed to all log on. I’m terribleat doing that, I don’t know when I last filled that in. (School Coordinator Interview, GLI)

I prefer to phone [the project researcher]. But if I can’t phone [them], then I use the email… or I put the basic “I need to know about…. give me a ring” in an email. (SchoolCoordinator Network Interview, CCH)

We responded to this in two ways; we encouraged project team members to record telephonecontacts where possible, and added, in a later version of the log allowed individuals to 'cc'emails to or from schools to a special email address; a small server-side programmeperiodically transferred these emails to the school logs to which they related, based on athree-letter code included in the email 'subject line'.

Once the logs were developing, project members (particularly those with responsibility forseveral schools, and those involved in preliminary data analysis) requested that a set ofmanagement tools be built so that they could gain a quick overview of progress to date(including which parts of the project intervention had taken place, what data had beencollected and dates of future visits and meetings) across a group of schools, across a wholeLA or across the project as a whole. For a more complete account of the role of the loggingtool and the data gathered using it, see Carmichael, Procter and Honour (2003). What we alsodiscovered – both through the logs themselves and during school visits – was that thoseschool coordinators who had made regular use of the logs found them a useful 'runningrecord' of the progress of the project in their school:

When I did a summary of everything we had done to show our new co-ordinator, I wasamazed at the amount we have covered. It will need careful monitoring and support for thenew members of staff to ensure that they can take on board the key practice. (Log entry bySchool Coordinator, School KNG, 05-01-2004)

Responsive Design in Action: Other Emerging Issues for the Project

In retrospect, we were surprised by the extent to which the approaches we developed were notjust useful in terms of rapid software development (as in the case of the online logs) but alsobecame a focus for project-wide discussions that took place and threw some aspects of projectdesign and management into high relief. The following are some examples of these broaderissues to which we had to respond and where the associated discourse had wider implicationsacross the project.

Project Management and Informed Decision-Making

At the mid-point in the project, the project team needed to select approximately half of theschools for full data collection and as possible case-study sites. This selection processinvolved classifying schools according to the likelihood that we would be able to gather near-complete data sets from them, based both on their record of involvement up to that point, andthe views of researchers and critical friends. We were able to rapidly adapt the school ‘log’tool, but rather than using it to produce a web page per participating school, a largeorganizational chart (rather like a project-specific GANTT chart) was generated. This wasthen used – together with other data – to inform the selection process.

Reflecting Concerns about Privacy and Confidentiality

The issue of ‘who’ should see ‘what’ on the website was initially conceived in terms ofdifferential access for members of the project team, school-based participants and ‘the

10

public’. It soon became apparent, however, that this did not provide sufficiently ‘finegrained’ distinctions between users, nor did it allow complex patterns of access to beexpressed. A key example related to the level of access Local Authority representatives onthe project would have to individual school logs. Once this issue had been raised, a meetingof Local Authority staff and teacher members of the advisory group established aconfidentiality framework. Our component-based approach allowed us to define (in themetadata of each component) who should see not whole data sets, but specific elementswithin them. This allowed the confidentiality framework to be applied almost immediatelyacross the web site. In another development, a decision to publish a large set of projectresources to schools other than those participating in the project was reflected on the websitein a matter of minutes and without any large-scale reorganization.

Analysis and Archiving

As collection and analysis of data continued within the project, the issue of secure storage ofdata emerged as an important issue. Dialogue between researchers and developers continuedas how best to structure, store and describe datasets, and particular requirements (such as theability of researchers to retrieve data either by instrument or by research site) were addressedby our component-based approach. Our definition of ‘component’ was also extended toinclude transcripts, analytical accounts and summaries, images, quantitative data files andbibliographic references, as diverse data were uploaded to the website. This collection nowextends to over 1000 resources each of which is, where appropriate, accompanied by adescriptive metadata file.

With the funded period of project activity now at an end, a final phase of responsive activityhas centred on restructuring web content in such a way that project researchers are able tocontinue their analysis, collaborative writing tasks and dissemination activities, while at thesame time much of the project documentation, including presentations, workshops and reportshave now been placed into the public domain. Content has been reformatted and incorporatedinto a book aimed at teachers interested in developing formative assessment practice in theirown schools – illustrated with accounts, derived from online logs and interviews – of howparticipating project schools engaged in and with the project (James et al, 2006).

Reflections

On reflection, we can say with some confidence that the management of the project was madeeasier both by our capability to develop web resources and other software ‘in house’, and bythe use of our ‘responsive’ approaches. For some teachers participating in the project, theprovision of online support considerably enhanced their experience and provided them with‘critical friendship’ during what was sometimes a challenging development process. Theavailability of project resources on the website increased the reach and impact of the project,particularly when combined with face-to-face contact through meetings or professionaldevelopment activities. Viewed more critically, while some of the school coordinators hadengaged fully with the project and used the online log tool to enter into sustained discussionswith ‘critical friends’ or as reflective diaries, for others the project website was perceived assimply another available online resource – albeit one where the emphasis was on developingpractice rather than on provision of curriculum resources.

One aspect of practice that we were not able to develop to any extent was electronic school-to-school networking and sharing of practice, an issue attributed to prevailing schoolstructures (Cuban, Kirkpatrick and Peck, 2001; Wenger, 2001, 4). Participants in our project

11

also attributed this to competition between schools for resources, staff and examinationresults; to perceptions of schools being so dissimilar that even effective practice would not'translate' to other contexts; to poor and incompatible communication systems (especiallyvideo conferencing); and to a lack of recognition amongst staff of the value of theirprofessional knowledge, one head teacher suggesting:

Teachers are dreadful at recognising, you know, their own professional ability … if you aska teacher about their strengths and weaknesses, they always tell you the weaknesses straightoff. Making people sound confident that they’re skilled or aware of what their skills are isactually essential because they need to analyse something and say, ‘What do I do well?’which they probably do expand it or pass it on to other people. (L3 HT interview, SEK)

While we were aware of the development of professional knowledge and innovative practicein many project schools, this was generally communicated to other teachers at face-to-faceevents such as regular school coordinator meetings, after which some schools did, in fact,establish email communication and programmes of reciprocal visits. Alternatively, sharingdepended on researchers, critical friends or LA representatives acting as ‘brokers’ ofinnovative practice between those project schools with which they had contact. In fact, someLA representatives saw this as a key role for them more generally:

My role as an assessment advisor is to promote good assessment practice within the LEAwhether that is around data and summative assessment or good assessment practice in theclassroom … I am going into schools and picking up good practice that I can share withother schools (LA Coordinator Interview, JE)

“What we want to try and do is actually get those schools talking to each other moreelectronically … What I would like to do at that meeting is to set up … that would beginemail communication, so that they keep in closer contact with each other.” (LACoordinator Interview, TG)

So in one respect, schools participating in the project conformed to Hargreaves' model of the'knowledge creating school' in that they were taking part in a 'managed process of creatingnew professional knowledge' (2001) However, for a variety of reasons they failed todisseminate that knowledge systematically, or to have it validated by colleagues without theintervention of some kind of 'broker' (see Carmichael, Fox, McCormick, Procter and Honour,2006 for further discussion of the role of brokers). In the absence of such processes, thedominant model of dissemination remains, in Hargreaves' terms, 'outside-in' (2001, 35) withschools as net 'consumers' rather than 'producers' of professional knowledge.

Next Steps: Using the Sakai Virtual Collaboration Environment to Support the'Knowledge Creating School'

One outcome of our work within the Learning how to Learn project has been the extension ofsome of our online tools and our responsive approach to other projects within the Teachingand Learning Research Programme. A significant difference, however, is that we nowemploy an existing software platform to address many of the user needs we have identified –rather than developing 'home-grown' solutions as we did in Learning how to Learn. Ourchosen platform is Sakai (http://www.sakaiproject.org), an open source community projectthat allows the configuration of 'worksites' within which sets of 'tools' can be arranged andcombined (see Figure 2). The whole system is componentised with content being reusableacross tools and worksites; content and formatting are separated; and international standardsare applied – so this is an environment within which our commitment to responsiveness canbe maintained and developed further.

12

Figure 2: A Sakai 'Worksite' showing multiple collaboration tools.

We are currently evaluating Sakai as a platform for supporting social science research,including projects involving school-university collaboration, and have been able to apply theexperience gained in the course of the Learning how to Learn Project as we support, developand research the effectiveness of various online tools and configurations. With thedevelopment of 'wiki' and 'web log' tools for use within Sakai, the potential exists to supportthe kind of extended interaction between teachers, researchers, and critical friends weobserved in our own log tool. At the same time it is possible to configure Sakai as anenvironment within which email is the primary locus of interaction; an 'email archive' and'new contents' email alerting functions allowing participants to keep abreast of developmentsand contribute to discussion without any need to regularly log in to worksites. Amongst theprojects now participating in our evaluation of Sakai, there are some who use the platform touse it to build central resources to support dissemination; those who use it to provide private,access-controlled discussion spaces where the focus is on interaction rather than documents;and those that use it primarily to support widely distributed groups who communicate mainlyby email.

Sakai's fine-grained permissions system allows the control of access to resources, and whilethis is clearly useful for reflecting the kinds of relationships between schools and LA's wedescribe above, it also allows the phased release of data to progressively larger groups; thusan individual can work privately before inviting others to contribute, and to comment on andvalidate their activity. This support for 'graded publics' is identified by Altrichter (2005, 22)as a characteristic element of much educational research, and provides a means by which thevalidation and dissemination of professional knowledge can be supported within and betweenschools. Furthermore, the flexibility of the platform means that initially it can be configuredto replicate or reflect existing structures and working patterns, while offering opportunities tosupport new patterns of interaction and, potentially, new kinds of relationships betweenresearch participants.

13

Our experience of 'moving platforms' and encountering many of the same issues with Sakai aswith our own web tools raises an important final issue. It strongly suggests that, while 'usecase analysis' is important in identifying how groups of teachers might engage with newtechnologies, and our four principles are a helpful framework, the most important element ofall in the development of effective technological support of collaborative research (regardlessof technological platform) is the commitment to make the working practices and preferencesof participants themselves an object of systematic enquiry. If we had not identified the lowlevels of IT use by teachers in participating schools, we might have made assumptions whichmight have prevented many from taking part in the activities of the project. If we had notengaged in a continuous process of consultation, we might have failed to engage thoseteachers who relished the opportunity to reflect in detail upon their practice and who hadvaluable lessons to share. And perhaps most importantly, if we had not provided alternative,non-electronic channels of communication and sites for networking, many teachers wouldhave been denied the experience of participating in, validating and sharing their developingprofessional knowledge.

14

References

Altrichter, H. (2005) 'The Role of the Professional Community in Action Research' EducationAction Research 13(1) 11-26.

Black, P. & Wiliam, D. (1998) Inside the black box: raising standards through classroomassessment. London, King’s College.

Carmichael, P. (2003) 'Teachers as researchers and teachers as software developers: how use-case analysis helps build better educational software'. The Curriculum Journal 14(1) Spring2003, 105-122.

Carmichael, P., Procter, R. and Honour, L. (2003) 'Network Tools across a DistributedResearch Project: Reflections on the first two years of 'Learning how to Learn''. Paperpresented at the conference of the British Educational Research Association, Edinburgh,September 2003.

Carmichael, P., Procter, R. and James, M. (2005) 'Using electronic tools to support a largedevelopment and research project: a case study ' Paper presented at TLRP annual conference,November 2005, University of Warwick, Thematic Session on 'Managing Large Projects'.

Carmichael, P., Fox, A., McCormick, R., Procter, R. and Honour, L. (2006) 'Teachernetworks in and out of school' Research Papers in Education 21(2) 217-234.

Costa, A. and Kallick, B. (1993) ‘Through the Lens of a Critical Friend’, EducationalLeadership, 51(2), 49-51.

Cuban, L., Kirkpatrick, H. and Peck, C. (2001) High access and low use of technologies inhigh school classrooms: explaining an apparent paradox, American Educational ResearchJournal, 38 (4) 813-834.

Gottesdiener, E. (2002) Requirements by Collaboration. Boston, MA: Addison-Wesley.

Hargreaves, D. (2001) Creative Professionalism: the role of teachers in the knowledge society(London: Demos)

Hoel, T. and Gudmundsdottir, S. (1999) ‘The REFLECT Project in Norway: interactivepedagogy using email’ Journal of Information Technology for Teacher Education, 8(1); p.89-110.

James, M., Pedder, D., Black, P., Wiliam, D., & McCormick, R. (2003) ‘The Design of theESRC TLRP “Learning How to Learn” Project’. Paper presented at the 2003 AnnualConference of the British Educational Research Association. Edinburgh.

James, M., Black, P., Carmichael, P., Conner, C., Dudley, P., Fox, A., Frost, D., Honour, L.,MacBeath, J., McCormick, R., Marshall, B., Pedder, D., Procter, R., Swaffield, S. & Wiliam,D. (2006) Learning How to Learn: tools for schools. TLRP 'Improving Practice' Series.London, Routledge.

15

Maxwell, T.W., Reid, J. McLoughlin, C., Clarke, C and Nicholls, R. (2001) ‘Online supportfor Action Research in a Teacher Education Internship in rural Australia’ paper presented atthe Society for the Provision of Education in Rural Australia Annual Conference, WaggaWagga, 8-11 July, 2001.

Nelson, T. (1997) ‘Embedded Markup Considered Harmful’ World Wide Web Journal 2(4),129-136.

Stenhouse, L. (1975) An Introduction To Curriculum Research And Development. London:Heinemann Educational.

Verwijs, C., Mulder, I., Slagter, R., & Moelaert, F. (2002) ‘Leveraging communities withnew technologies’ In J. H. E. Andriessen, M. Soekijad, & H. J. Keasberry (Eds.) Support forknowledge sharing in communities (Delft, The Netherlands: Delft University Press), 57-70.

Wenger, E. (2001). 'Supporting communities of practice: A survey of community-orientedtechnologies'. Online at: http://www.ewenger.com/tech