A Lifecycle Approach to Evaluating MyArtSpace
description
Transcript of A Lifecycle Approach to Evaluating MyArtSpace
A Lifecycle Approach to Evaluating MyArtSpace
Giasemi Vavoula
withJulia Meek
Mike SharplesPeter LonsdalePaul Rudman
[email protected], [email protected] 2 / 16WMUTE 06
Outline
What is MyArtSpace What is Lifecycle Using Lifecycle to evaluate
MyArtSpace Results: Mobile learning for schools
in museums Conclusions
[email protected], [email protected] 3 / 16WMUTE 06
MyArtSpace – Situation of concern
School visits to museums Visits often isolated from classroom work
“Learning is a cumulative process involving connections and reinforcement among the variety of learning experiences people encounter in their lives: at home, during schooling, and out in the community and workplace” (Dierking et al. 2003)
“making the links between school and museum learning explicit, genuine, and continuous affords real opportunities for school students to have enjoyable learning experiences in both settings.” (Griffin 2004)
Making the visit personal and relevant Static displays, generic labels
Structuring enquiry learning. Children need specific support in:
planning appropriate investigations managing investigations interpreting results (de Jong et al., 2006)
Design linked pre, on, and post-museum visit learning experiences
Personalise interaction with exhibits
Design enquiry-led museum learning
[email protected], [email protected] 4 / 16WMUTE 06
MyArtSpace – What it is Service on mobile phones for enquiry-
led museum learning Aim to make school museum visits more
engaging and educational Learning through structured enquiry,
exploration, connection Combines
physical space (museum, classroom) virtual space (online store and gallery) personal space (mobile phones)
Museum test sites Urbis (Manchester) The D-Day Museum (Portsmouth) The Study Gallery of Modern Art (Poole)
About 3000 children during 2006
[email protected], [email protected] 5 / 16WMUTE 06
2. At the museum: Students are given Nokia 6680
multimedia mobile phones Use phones to ‘collect’
exhibits by typing a two-letter code
Prompted to type their reason for collecting
Document their experience (take pictures, record sounds, write comments)
All exhibits and recordings are sent automatically to a personal web space
MyArtSpace – How it works1. In the classroom:
Discuss ‘big question’ to explore, by collecting evidence from the museum visit
3. Back at school: Personal website shows their
notes, recordings, pictures, exhibits
They can view others’ collections, and items provided by the museum
They can create personal online galleries to show teachers and classmates, and to friends and family outside school
[email protected], [email protected] 6 / 16WMUTE 06
Lifecycle – What it is A methodology for the evaluation of
Educational Technologies Basic premises:
Start evaluation very early in the design process – and keep evaluating throughout
Undertake evaluation activities at key points addressing questions:
Who are the stakeholders and the users of this evaluation activity?
Why is this activity being undertaken (purpose/goals)? What evaluation methods/resources will be used When should the evaluation be undertaken? How will the results be communicated?
Feed evaluation outcomes into following stages, or re-iterate previous stages
Lifecycle Evaluation Toolkit
[email protected], [email protected] 7 / 16WMUTE 06
Lifecycle – What it is (2) Shares concepts and methods with software
engineering, educational and learning technology evaluation (User-Centred Systems Design, lifecycle approach to software engineering, etc.)
Emphasises the central role of Evaluation throughout learning system design, from system conception to final assessment.
Provides a framework for planning evaluation activities and using their outcomes.
Meek (2006), Meek & Sharples (2001)Contact: [email protected]
[email protected], [email protected] 8 / 16WMUTE 06
Lifecycle for MyArtSpace - Stakeholders
The Sea: MyArtSpace design and development team
Culture Online (UK Dept. for Culture, Media and Sport): funding body
Teachers: users Students: users Local Education Authority reps: expert
users / promoters Museum curators / education experts:
users / clients / test sites Education/accessibility consultants: experts
[email protected], [email protected] 9 / 16WMUTE 06
Lifecycle for MyArtSpace – Evaluation activities
Project Phase Activity Methods
Consultation Consultation workshop Round-table discussion
Consultation Requirements analysis Workshop
Design - Build Usability of designs Heuristic evaluation
Build First user trial Field study
Build Second user trial Field study
Deploy Usability of designs Heuristic evaluation
Deploy Third user trial Field study
Deploy Stakeholder consultation
Structured interviews
Deploy Teacher evaluation Questionnaire/interviews
[email protected], [email protected] 10 / 16WMUTE 06
Lifecycle for MyArtSpace – Evaluation activities (2)
Consultation workshop Development team, funding body, evaluation team/mobile learning
experts Improvement of initial concept
Requirements workshop Development team, funding body, teachers, LEA reps, evaluation
team, museum reps, accessibility expert Educational, user and accessibility requirements
Usability evaluations Heuristic evaluations of mobile phone, student/teacher web interface,
museum curators web interface; at critical points in interface design Field studies
Full scale trial with KS2 students at Urbis; main focus: system performance, usability
Full scale trial with KS3 students at D-Day; main focus: issues from previous trial, educational experience, usability
Final full scale trial with KS3 students at D-Day; main focus: educational and user experience, system impact
Stakeholder interviews After-pilot deployment interviews; focus: great expectations –
fulfilled? Teacher surveys
Questionnaires and semi-structured interviews with teachers; main focus: what works and what doesn’t, why, perceptions and attitudes, space for improvement
Jan 05
Dec 06
Jan 06
Aug 05
Jun 05
May 05
Conception
Consultation
Design
Build
Deploy
Market
Pro
ject T
imelin
e
[email protected], [email protected] 11 / 16WMUTE 06
Lifecycle for MyArtSpace – Use of evaluation output
MyArtSpace concept
MyArtSpace
designs
MyArtSpace system
Formulate
concept
Requirements
analysis
Usability evaluatio
nsField
Studies
Evaluation activity
System development stage
Teacher evaluati
ons
MyArtSpace
marketing
Stakeholder
Consultation
[email protected], [email protected] 12 / 16WMUTE 06
Lifecycle for MyArtSpace – A focused framework
Emergent evaluation framework for mobile learning systems and services
Expectations Reality Gaps
MicroIndividual activities performed during the learning experience – focus on usability/usefulness
How do stakeholders expect the system to work technically?
How does the system work technically for stakeholders?
What were the gaps between technical expectations and reality?
How does the Micro level affect the Meso level and vice versa?
MesoEducational aspect of the experience – focus on learning breakthroughs and breakdowns
How do stakeholders expect the system to work educationally?
How does the system work educationally for stakeholders?
What were the gaps between the educational expectations and reality?
How does the Meso level affect the Macro level and vice versa?
MacroEffect on educational practice – focus on educational transformations
What was the stakeholders’ vision for the system?
What impact / transformations does the system bring about?
What were the gaps between the expected system impact and the actual impact the system appears to have?
[email protected], [email protected] 13 / 16WMUTE 06
Lifecycle for MyArtSpace – Preliminary results
Mobile learning for schools in museums: Micro level (individual learning activities)
Students enthusiastic about creating their own content: taking dozens of pictures and recording lots of audio
Text comments not very popular – in recent trial, they override text comments and record audio instead
Meso level (the learning experience) “Made me look at artwork more” Students stop before exhibits and think – should I collect? Is
this relevant? Prompted to abstract main ideas of their galleries by e.g.
asking to provide a good title Durable visit output to take away
Macro level (the educational experience) Connect museum visit to school learning (“didn’t really use
museum visits for specific purposes (before)”, “give us an easier follow up lesson to make the value of the trip more worth while”)
Increase engagement
[email protected], [email protected] 14 / 16WMUTE 06
Lifecycle for MyArtSpace – Preliminary results (2)
Example inter-level influences (micro-meso): ‘Aggressive’ collecting: the more open and exploratory the
learning task is at the meso level, the more overuse is made of collection activities at the micro level…
… which makes harder the post-visit manipulation of collections at the micro level, which in turn hinders the learning experience at the meso level
learning task needs to be well designed and appropriately constrained
pre, on and post museum experiences need to be allocated enough time
No explicit link between pictures and comments at micro level results in difficulty in interpreting pictures post-museum at the meso level
Functionality needed: annotate picture with audio/text comment
[email protected], [email protected] 15 / 16WMUTE 06
Conclusions MyArtSpace:
Encourages students to stop and think about each exhibit Engaging, fun Consolidates formal and semi-formal learning experiences MyArtSpace is being re-branded as OOKL for commercial
release Lifecycle:
Focus on both usability and pedagogy early in the design Helped draw a full picture of the use of new technology to
be understood, assessed and improved Involved stakeholders in evaluation activities early on,
making it everyone’s business to provide input to evaluation activities and make use of the outcomes!
[email protected], [email protected] 16 / 16WMUTE 06
Partners - contacts Culture Online (Department of Culture, Media and
Sport) http://www.cultureonline.gov.uk/
The SEA – software development http://the-sea.com
University of Birmingham (CETADL) University of Nottingham (LSRI) Urbis, Manchester Study Gallery, Poole D-Day Museum, Portsmouth Lifecycle Ltd
Contact: Julia Meek ([email protected]) Partners: Julia Meek, Peter Lonsdale, Paul Rudman, Mike
Sharples, Giasemi Vavoula (MAS evaluation team leader), <other two>