The ChartEx Project: A Case Study in Interdisciplinary HCI

Christopher PowerAdvanced Topics in Interactive Technologies

January 31, 2013

Interdisciplinary Research

• Normally in HCI, we like to think of ourselves as interdisciplinary …

• … in our own group we have:– 1 Psychologist– 2 Computer Scientists– 1 Mathematician

• And yet we all think of ourselves as HCI researchers

Interdisciplinary Research (2)

• There are of course a huge number of fields that contribute to HCI:– Computer Science, Engineering, Psychology,

Sociology, Anthropology, Semiotics, Philosophy, Business and Management

• But is that what interdisciplinary really means?– That is all contained within our field – we are all

working in the same broad domain, often on the same problems

Interactive Systems for Science Research

• Interactive systems are everywhere and some fields get a lot of attention, notably scientific fields

Interactive Systems for Humanities?

• A huge amount of money and effort are changing physical resources into digital representations

• Where do we start in trying to design new interactive systems for these digital artefacts?

Digital Cultural Heritage• What is digital cultural heritage?

– Emerging discipline bringing together humanities researchers with purveyors of digital “stuff”

• Up until now, has been driven by digitization and preservation– Massive push in the EU over the last 10 years to digitize

collections in archives, museums and libraries– Full digitization: > 100 Billion Euros (Poole, 2010)– Massive investment already committed to an organization

called Europeana to archive and provide access to the millions of Euros of investment in digitization already spent

• So what is the problem?

Digital Cultural Heritage (2)• We can’t do anything with them!• The most advanced technology we have for interacting with them?

The equivalent of a file browsing search:– http://discovery.nationalarchives.gov.uk/SearchUI/– http://www.hrionline.ac.uk/causepapers/

• Sometimes small projects come up with gorgeous, enriching web interactions – however, these are often ad-hoc approaches and not reproducible – nor do we understand the design principles for them

• And what if we want to do more than just display documents? Surely people are doing things with the documents – why aren’t we supporting those tasks?

• But unlike home and office systems, we have no context or knowledge of what those tasks are!

A history lesson …• About 4 years ago we were approached by a

medieval historian with this great idea:– Create an online tool that would allow her to use her

domain knowledge to reason about documents and synthesize new knowledge about history – in particular about locations in space and time (e.g. where was the blacksmith shop on Goodram Gate?)

– What she really didn’t want was a computer just running off and coming up with a bunch of stuff automatically

– It was about helping her, the historian, do the thing she does best – tell stories about the past from the artefacts we have from history

A history lesson…(2)

• We immediately saw a parallel with an HCI idea from around 2000 from Beaudoin-Lafon: instrumental interaction

• Beaudoin-Lafon argued that many of the systems we build deskill people; we pursue systems that do everything automatically, when instead we should be finding ways to let the use contribute their knowledge to the system – where the system becomes an instrument (i.e. like a scalpel for a doctor)

A history lesson … (3)

• We liked the idea so much that we jumped in with both feet. It ticked all the boxes– Really hard problems that would stretch our HCI

theories and practices– Creation of new interesting interactive technologies– Fun! It isn’t boring like life insurance, or banking, or

scary like safety in aerospace or medicine – it is about enriching our lives and our culture

• So after about 3 years of trying, we got a grant to try to build this interactive system

ChartEx: The Charter Excavator

• Received money from the Digging into Data Challenge– Money to try to do something with all of this

digitized data – from the UK, Canada, US and Ned.• Our main purpose was to come up with an

interactive system that would allow people to interact with medieval charter documents and reason about the people and places in them

Wait … what’s a charter document?

• Charters (or charter deeds) are legal documents that describe the transfer of land holdings from one owner to another (oversimplified – the historians would scold me)

• They are some of the most important remaining legal documents that we have, especially in Europe, as they can help work out where people lived, or groups lived, or wealth distribution, or spheres of influence … you get the idea

• To work with these documents is a highly specialized skill• Some of the best holdings of charters in the world:

University of York and the University of Toronto

Charter Document• 408. Grant by Thomas son of Josce goldsmith and citizen of York to his

younger son Jeremy of half his land lying in length from Petergate at the churchyard of St. Peter to houses of the prebend of Ampleford and in breadth from Steyngate to land which mag. Simon de Evesham inhabited; paying Thomas and his heirs 1d. or [a pair of] white gloves worth 1 d. at Christmas. Warranty. Seal.

• Witnesses: Geoffrey Gunwar, William de Gerford[b]y,' chaplains,Robert de Farnham, Robert le Spicer, John le plastrer, Walter de Alna goldsmith, Nicholas Page, Thomas talliator, Hugh le bedel, John de Glouc', clerks, and others.

• January 1252 [1252/3].• SOURCE: VC 3/Vi 326 (161 mm. x 137 mm.)• ENDORSEMENT: Petergat', Donacio facta vicariis de domo que fuit Thome

aurifabri; Simonis Evesham.• SEAL: Slit.' Hole in MS.• NOTE: See 403.

ChartEx Tool Chain (1)

• In order to do anything with all of this data we have, we need to know what is in the documents – images aren’t going to work

• We need to process and mine some of the information from the documents to get at the key entities that historians use when doing their work– We’ll get back to those in a minute …

ChartEx Tool Chain (2)Charter Documents

Analysed individualdocuments

Language processing

Data Mining

Analysed integrateddocuments

Researcher’s workbench

Questions questions …

Classic problem in a different domain

Historian View of Problem Technologist View of Problem

Questions questions … (2)

• And there we learned our first rule of interdisciplinary work:– In true interdisciplinary work there is no common

vocabulary – everyone speaks a different language• For technology people the solution is to grab

their terminology (Linked Data is a favourite in this domain) – not useful for historians

• What do we do in HCI when we hit a problem like this?

Answers!• Participatory design

– This was the point of participatory design – getting domain experts working to design systems (think about Scenario Based Design)

• But we weren’t yet trying to design the interface – we were working at the level of the content of the documents

• Well … that’s kind of different – but maybe we can try some of our participatory stuff out on those documents to see if we can understand how people work with their contents

Participatory Design of a Content Model

• Proviso: This is one of the weirdest things I’ve ever done as an HCI practitioner.

• Historians work with documents in interesting ways – one thing they do is diplomatic markup – this is a process where the historian marks different entities and relationships in a document and gives them semantic meaning– Example: If I had the name John in a document, he is

(likely) a person. John could be the owner of some land, or he could be a witness, or he could be related to some other person. These things can be marked in the document.

Participatory Design of a Content Model

• We had a group markup session!– Had a moderator/scribe and 2-3 historians– Each historian brought their 3 favourite charters– We then chose a type of “thing” (entity) that we

wanted marked up: Actors (People, Institutions), Places, Sites, Events

– For each entity type, people pointed to parts of the document that were of that type

– For each entity, the historians indicated the relationships and roles that entity was playing in the document.

Identifying Entities• Initial examination charters for entities about actors,

sites and events………..

• 408. Grant by Thomas son of Josce goldsmith and citizen of York to his younger son Jeremy of half his land lying in length from Petergate at the churchyard of St. Peter to houses of the prebend of Ampleford and in breadth from Steyngate to land which mag. Simon de Evesham inhabited; paying Thomas and his heirs 1d. or [a pair of] white gloves worth 1 d. at Christmas. Warranty. Seal.

Identify Relationships between Entities

• Initial examination of charters for relationships between entities …

• 408. Grant by Thomas son of Josce goldsmith and citizen of York to his younger son Jeremy of half his land lying in length from Petergate at the churchyard of St. Peter to houses of the prebend of Ampleford and in breadth from Steyngate to land which mag. Simon de Evesham inhabited; paying Thomas and his heirs 1d. or [a pair of] white gloves worth 1 d. at Christmas. Warranty. Seal.

Participatory Design of a Content Model (2)

• We ended up with a collection of documents that had a robust collection of information– Gave a clue about the types of things that historians were using to

reason about documents• A content analysis on the documents gave us a high level set of

entities, an a set of relationships that can occur between relationships

• From that analysis– a content model (technical details omitted) for documents was produced– a set of markup guidelines were created for working with future

documents• More importantly – we now had a common understanding of what

we was interesting in these documents

Document processing

50documents

Trainingdocuments

Unmarked Documents

Newly markeddocuments

LanguageProcessing

Interactive System Design• We needed to know more than just content of the

documents, we needed to know what people did with them when doing research

• We immediately ran into an interdisciplinary problem:– Humanities people are trained VERY differently from

scientists – so we couldn’t even pose questions in a way that made sense to historians

• A further problem is: research is tacit knowledge; it is often very hard for researchers to describe how they synthesize new information

Reaching into the toolbox

• In your modules, you are learning a set of tools that allow you to overcome challenges like this.

• What elicitation methodology might help us overcome these problems?

Contextual Inquiry• To deeply understand the tasks

of historians, we undertook a contextual inquiry

• We asked the historians to find a problem they had recently solved and retrospectively walk us through the solution– One historian identified a set of

shops on a street in York– Another identified particular

trends in witness lists in Essex– Another traced the holdings and

influence of an Abbot in Catalonia

Contextual Inquiry (2)• We recorded the sessions and then did partial transcriptions of

the videos– Identified key actions and tasks undertaken– Utterances where people talked about things that were important in

either the documents or their notes• Distilled a set of basic requirements that fell into three broad

activities:– Searching for documents in collections– Interacting with individual documents to understand their contents– Relating information between documents

• Tied into all of this was ideas around their confidence in particular pieces of data (e.g. the Abbot in Document A is a Bishop in Document B)

Interaction Design

• We began by dividing the interface it the three activities

• Started with the easy one: searching for documents in collections– Built a prototype for an interface that was

centred on the metaphor of file boxes, sorting documents, carousels of documents

• Then we had our first co-design workshop to evaluate it with our historians …

Interaction Design FAIL• They really didn’t like it!• It emerged there were 3 key problems:

1. Despite repeatedly saying they wanted something new, they all wanted to go back to the file/folder metaphor of the desktop interface

2. The document search was nothing like the searches they were using in existing archive systems

3. They couldn’t get to “the document” easy enough• One interesting thing was that people had a hard

time expressing what the problems were on the fly during the demonstration

Interaction Design: Round 2

• Because of the feedback we had, we knew that we had to start getting down to working with the documents

• We moved to looking at the tasks of working with each document and relationships between documents more holistically

• We focused on making the document the key pivot point in the interface

Prototype Demonstration

• Low-fidelity prototype prepared with OmniGraffle

• Exported to web pages and later power point to demonstrate functionality

Evaluation of Round 2

• This time, we did evaluation in two phases– Demonstration of the interface functionality– Co-design workshop approximately 2 weeks

afterwards• This gave people some time to think

about what worked and didn’t work

Outcome of Evaluation• Overall, the historians liked what they saw

– Very positive about being able to investigate relationships between the documents

– Liked that they could keep their own copies and update those copies with their own knowledge (e.g. confidence)

• One really interesting outcome– Some historians wanted to be able to go from a high level view and

immediately see the relationships – then drive down to document level– This is something they couldn’t do previously, and shows how our

elicitation methods capture some things, but we need the evaluation steps!

• One funny outcome: – “The document takes up too much space – we shouldn’t have to have it

up.”

Future Work

• Next steps– High fidelity prototype of the workbench

integrating the feedback from the evaluations– Integration with large amounts of data (>

10000 charters)• (I am very skeptical about how our visualizations

scale!)– Move to a beach house surrounded by

enormous piles of money

Lessons Learned• About interdisciplinary work

– The gap between science and humanities is much larger than we expected

– The difference in training is important – there is very little shared language to build from

• About “technology think”– There is a major danger to move down to the implementation

level – for technology people it is comfortable and for the potential users it is arcane

– Technology people can often push users into solutions that are not appropriate

– Even within technology partnerships there remains gaps between interaction designers and “hard core” technologists

Lessons learned (2)

• About interaction design– Elicitation techniques like contextual inquiry

are good to start with – but this project emphasizes how important evaluation is to refining early prototypes

– A lot of what is done in the historians heads is about building relationships, and essentially building up mental models of the past – there isn’t a ton of design work doing that well

Lessons Learned (3)

• About HCI as a contributor to digital cultural heritage– We have a unique perspective that other technology

people don’t have – we know how and want to understand the users

– We have techniques that no one else has to understand these complex domains

– DCH is an almost untouched field in terms of good interfaces – we NEED to be there to avoid the types of problems we see in all other systems that are driven by technology and not the user

Questions?

Resources• Poole, Nick (2010). The Cost of Digitising Europe’s

Cultural Heritage, A Report for the Comité des Sages of the European Commission, Collections Trust. http://ec.europa.eu/information_society/activities/digital_libraries/doc/refgroup/annexes/digiti_report.pdf

• Beaudouin-Lafon, M. (2000, April). Instrumental interaction: an interaction model for designing post-WIMP user interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems (pp. 446-453). ACM.