Introduction

Participatory Assessment for Service Learning 1.0 and Civic Engagement 2.0

Daniel HickeyAssociate Professor

Learning Sciences ProgramIndiana University

Introduction Reframe Concepts Scaffold Participation Align DiscourseControl

AccountabilityAssess Reflection

Acknowledgements

• MacArthur Foundation’s Digital Media and Learning Initiative– 21st Century Assessment Project

• IU Office of the Vice President of Information Technologies– E-Portfolio Faculty Scholars Program

• US Department of Education Educational Technology Program– Monroe-Eastern Greene Network



Broad Goals for Improving Service Learning and Civic Engagement

• Resolve nagging tensions in education– Direct vs. constructivist instruction– Newer participatory approaches can resolve them

• Enhance diverse learning outcomes– Engaged communal participation– Individual conceptual understanding– Aggregated achievement

• Preserve validity of learning evidence



Research Context

• GenScope Assessment Project I (95-2000)– Core discursive formative assessment practice

• GenScope Assessment Project II (2000-05)– Alignment across three levels of outcome

• NASA COTF (03-05)– Streamlined model

• Recent/Current Projects– Quest Atlantis (05-10)– Secondary English & New Media (08-– Learning Sciences Graduate Courses (09--



Current Participatory Assessment Projects

Taiga Ecological Sciences in Quest Atlantis– 20 hour curriculum for

sixth graders– Five cycles of

refinement from 05-10

Fan Fiction Module– Classic texts– Microblogging as

characters– Characterization &

character analysis as fan practices

On-Line Learning Sciences Graduate Courses– P507, Assessment

in Education– P540 Cognition in

Education– Both taught three

times



Research Method

• Design Based Educational Research– Transcends fundamental vs. applied dichotomy– Build “local” in the context of iterative refinement– Pay careful attention to contextual framing

• Validate principles by testing them in context– Meta-principles (guiding ideas)– Design principles (what ideas mean in general)– Specific design principles (ideas in contexts)– Specific features (used to enact specific principles)

Introduction Activate Concepts Scaffold Participation Align DiscourseControl


• Learning is primarily communal participation in knowledgeable discourse– Individual learning is secondary– Dissolves learning versus engagement dichomomy

• Contexts give meaning to concepts– Contextual knowledge is fundamental– This is 21st Century knowledge.

• Activities, artifacts, assessments & accountability are all educational discourses– Represent learning at very different timescales

Participatory Learning Meta-Principles



Participatory Assessment Design Principles

1. Transform Concepts– Reframe academic knowledge as tools used in contexts.

2. Scaffold Participation– Guide worthwhile discourse around use of tools

3. Assess Reflections– Avoid grading student-produced artifacts directly

4. Control Accountability– Use assessments and tests carefully and narrowly

5. Align discourse– Iteratively refine activities & assessments to indirectly raise

achievement.



Specific Assessment Design Principles for Service/Civic Contexts

1. Transform Concepts– Reframe academic knowledge as tools for use in service/civic contexts.

2. Scaffold Participation– Provide informal guidance to help students enlist academic concepts in

service/civic contexts3. Assess Reflections

– Have students reflect on how their academic knowledge was transformed in service/civic contexts

4. Control Accountability– Assess individual service/civic learning carefully and discreetly– Measure service/civic programs in the aggregate

5. Align Discourse– Iteratively refine participation in service/civic activities to enhance reflections

and indirectly impact distal outcomes



• Contexts give concepts meaning– “Deep” understanding of concepts is relevance

across range of contexts.– 21st Century learning is about context

• Range of contexts in all educational settings– Across characters, units, students, etc.

• Reframe learning as a search for relevance– Learners need personally meaningful contexts– Compare value of contexts for learning concepts

#1. Transform Concepts: Reframe Academic Knowledge as Tools for Contexts



• E-Learning Courses– Each student defines own lesson context– Wikifolios identify most relevant and least relevant

implications and relevant specific• Quest Atlantis– Must use knowledge of inquiry and ecological

science to participate in game

Reframing Context Examples



E-Learning ClassesStudents First Define Problem



E-Learning ClassesReading Text as a Search for Relevance



E-Learning ClassSearch For Relevant Specifics



• Service and Civic Learning is about contexts– Void between school learning and contexts– Void is an opportunity rather than an obstacle

• Range of contexts civic/service learning– From school to service/civic context– Across students in same service/civic context– Across service/civic contexts

• Service/civic learning as search for relevance– How is your academic knowledge relevant?– How is this service/civic context transforming your knowledge?

Ideas for Reframing Academic Knowledge as Tools in Service/Civic Contexts



• Foster conversation about using conceptual tools in contexts– Different context reveal concepts differently

• Pay attention to initial participation– Must enlist tools initially before appropriately– Don’t grade or require participation– Encourage lurking

• Focus on the trajectory of the discourse– Don’t prematurely focus on accuracy

#2. Scaffold Participation: Guide Worthwhile Discourse Around Using Tools



• E-Learning Courses– Don’t use discussion forums– Students comment on each other’s wikifolios

• Quest Atlantis– Reframe Quest submissions as informal discourse– Ensure useful feedback is provided and used

Scaffolding Participation Examples



Example of Consequential Collaboration in Wiki Commenting

• Comments on wikifolios should lead to reflection and revision– Good use of instructor comments.

• Assume that physical and intellectual proximity supports engagement– Overcomes abstract and alienating online context



Juana’s Example• Chapter 2 Wiki on modal model of memory• Most relevant implication was information processing

is easier when to-be-learned information is distributed in working memory

• Linda’s comment suggested misunderstanding– “students often have so much information to manage, that

they may need to drop a lot of it in order to focus on the important points. That, to me, would fall more under the description of the 'bottleneck' implication than the 'distributed in working memory' implication, which I interpret to be more about presenting information in both a visual and auditory manner.”



Example Comment on Juana’s Wiki

• Linda’s Comment: “The way that I interpreted what you wrote for #1 is that students often have so much information to manage, that they may need to drop a lot of it in order to focus on the important points. That, to me, would fall more under the description of the 'bottleneck' implication than the 'distributed in working memory' implication, which I interpret to be more about presenting information in both a visual and auditory manner.” (emphasis added)



Juana’s Reflection• Consequential Reflection: “After reading everyone

comments it seems as though the implications selected are appropriate for my problem except for item number 1 that I have changed from working memory to Resource and Data limitations. (Resource and data limitations constrain learning) This is a much better fit for what my comments were and after rereading that section is clearly the right choice…



Jeraldine’s Case

• Example: Jeraldine’s Chapter 2 Wiki• Indicated “All students should be encouraged

to ‘manage their resources’” as the most relevant implication

• The comments redirected her thinking about relevance in different contexts.



Example Comment on Jeraldine’s Wiki

• Emerald’s comment: …almost all of your explanations you wrote about the importance of automaticity, making me wonder why "Automaticity facilitates learning by reducing resource limitations" wasn't your #1 implication. I could understand why you wouldn't choose this if your lesson included more challenging texts, but reading easy texts would require a lot of automaticity even for struggling readers. (emphasis added)



Jeraldine’s Reflection

• Collaborative Reflection: “Emerald commented on my use of automaticity throughout my wiki and wondered why the automaticity implication was not my first choice. I reread my reasoning for my first implication and realized that this is my goal for all my students. However, for my instructional problem, automaticity is the key. Hence, this implication should become my top choice. My second and third implications are almost subheadings of this top implication.”



• Anticipate different service/civic contexts in classes– Speculate about relevance early

• Articulate connections/voids between school and service/civic contexts

• Provide incentives for participation– Bring reflections in early

Ideas for Scaffolding Participation in Service/Civic Learning



#3. Assess Reflection: Avoid Grading Student-Produced Artifacts

• Grading artifacts undermines discourse– Requires detailed-but-shallow rubrics and exemplars

• Grade reflections as evidence of engagement– Grade infrequently– Aim to foster revision

• Focus reflections on engagement– Consequential, critical, and collaborative– Conceptual (i.e., understanding) “comes for free”



Assessing Reflection Examples

• E-Learning Courses– Wikifolios are never graded– Grade reflections every five weeks

• Quest Atlantis– Added reflections to quest submissions– “How does your report show you have learned?”



Reflect on Specific Types of Engagement

• Consequences of key concept in your each context– Also consequences of context for each concept

• Critique use of key concepts in each context– Critique context for learning concepts

• Collaboration associated with key concepts– Critique classes collaboration?



Ideas for Assessing Reflection in Service/Civic Learning

• Don’t have student report on what they did or what they learned– Have them report on how they engaged

• Articulate the types of engagement for reflection– Consequences of big ideas your context

• Consequences of context for big ideas– Critique use of big ideas in each context

• Critique context for learning big ideas– Collaboration associated with big ideas

• Critique classes collaboration?

• Do reflections publically via wikis or forums– Reward reflections that lead to action



#4. Control Accountability: Use Assessments and Tests Carefully

• Downplay importance of individual assessments and tests– Align with scaffolding and reflection– Engagement should ensure adequate

understanding• Assess and evaluate just a few outcomes• Don’t use program outcomes to assess

individuals– Compromises validity



Controlling Accountability Examples

• On-Line Learning Sciences Courses– Multiple choice midterm and exam from test bank– Worth only 15% each, use criterion for points

• Quest Atlantis– Open ended performance assessment used to

evaluate curriculum (not students)– Random sample of achievement items used to

evaluate gains over time



Ideas for Controlling Accountability in Service/Civic Learning

• Focus on proximal outcomes– Align close level activities to them– Use “distal” individual outcomes in the aggregate– Treat program outcomes as programmatic remote outcomes

• Be very focused with surveys– They are not particularly helpful for improvement

• Control accountability internally– Don’t let outsiders control it

• Assess and measure judiciously– Just because you can does not mean you should!



• Use design-based research methods– Refine features to validate specific principles.– Build local rather than fundamental theory

• Align participation to raise all outcomes– Preserve evidential validity by changing format

• Three or more increasingly formal cycles– Design (design activities and artifacts)– Implementation (select activities & artifacts)– Evaluation (compare achievement gains)

#5. Align DiscourseRefine Participation at all Levels



Three Levels Of Learning Outcomes

• Interpret participation in discourse (“close”)– Foster discourse in real time to increase understanding – Protect from corruption via assessment

• Assess individual understanding (“proximal”)– Refine design and choice of activities to raise

achievement– Protect from corruption by test

• Measure aggregated outcomes (“distal”)– Evaluate increased gains between or across courses– Secure test to preserve validity



• Learning Sciences Courses– Refining prompts to maximize quality discussion– Looking for improved reflections– Measure achievement gains from year to year

• Quest Atlantis– Continually improve questing discourse directly– Increased gains in understanding (Δ >2 SD)– Slowly raised gains in achievement (Δ >1 SD)

Aligning Discourse Examples



2006 Non-QA Comparison

(62)

2006 QA Baseline (54)

2007 QA Feedback

(94)

2008 QA Feedback

(50)

2008 QA Feedback + Incentive

(50)

2009 QA Reflective

Questions & FAQs (110)

0

0.5

1

1.5

2

Quest Atlantis Increased Gains in Problem Solving and Achievement across Years (in SD)

Curriculum-Oriented Problem Solving (Proximal)

Standards-Oriented Achievement (Distal)

• Focus on proximal outcomes like reflections on service– Align classes and other activities to proximal

assessments• Use program outcomes to evaluate programs

rather than individuals– Find valid measures for evaluating program

improvement

Ideas for Aligning Discourse in Civic/Service Learning



Introduction

Documents

Transcript of Introduction