Participatory Assessment for Service Learning 1.0 and Civic Engagement 2.0

Daniel HickeyAssociate Professor

Learning Sciences ProgramIndiana University

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Acknowledgements

• MacArthur Foundation’s Digital Media and Learning Initiative– 21st Century Assessment Project

• IU Office of the Vice President of Information Technologies– E-Portfolio Faculty Scholars Program

• US Department of Education Educational Technology Program– Monroe-Eastern Greene Network

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Broad Goals for Improving Service Learning and Civic Engagement

• Resolve nagging tensions in education– Direct vs. constructivist instruction– Newer participatory approaches can resolve them

• Enhance diverse learning outcomes– Engaged communal participation– Individual conceptual understanding– Aggregated achievement

• Preserve validity of learning evidence

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Research Context

• GenScope Assessment Project I (95-2000)– Core discursive formative assessment practice

• GenScope Assessment Project II (2000-05)– Alignment across three levels of outcome

• NASA COTF (03-05)– Streamlined model

• Recent/Current Projects– Quest Atlantis (05-10)– Secondary English & New Media (08-– Learning Sciences Graduate Courses (09--

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Current Participatory Assessment Projects

Taiga Ecological Sciences in Quest Atlantis– 20 hour curriculum for

sixth graders– Five cycles of

refinement from 05-10

Fan Fiction Module– Classic texts– Microblogging as

characters– Characterization &

character analysis as fan practices

On-Line Learning Sciences Graduate Courses– P507, Assessment

in Education– P540 Cognition in

Education– Both taught three

times

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Research Method

• Design Based Educational Research– Transcends fundamental vs. applied dichotomy– Build “local” in the context of iterative refinement– Pay careful attention to contextual framing

• Validate principles by testing them in context– Meta-principles (guiding ideas)– Design principles (what ideas mean in general)– Specific design principles (ideas in contexts)– Specific features (used to enact specific principles)

Introduction Activate ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Learning is primarily communal participation in knowledgeable discourse– Individual learning is secondary– Dissolves learning versus engagement dichomomy

• Contexts give meaning to concepts– Contextual knowledge is fundamental– This is 21st Century knowledge.

• Activities, artifacts, assessments & accountability are all educational discourses– Represent learning at very different timescales

Participatory Learning Meta-Principles

Introduction Activate ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Participatory Assessment Design Principles

1. Transform Concepts– Reframe academic knowledge as tools used in contexts.

2. Scaffold Participation– Guide worthwhile discourse around use of tools

3. Assess Reflections– Avoid grading student-produced artifacts directly

4. Control Accountability– Use assessments and tests carefully and narrowly

5. Align discourse– Iteratively refine activities & assessments to indirectly raise

achievement.

Introduction Activate ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Specific Assessment Design Principles for Service/Civic Contexts

1. Transform Concepts– Reframe academic knowledge as tools for use in service/civic contexts.

2. Scaffold Participation– Provide informal guidance to help students enlist academic concepts in

service/civic contexts

3. Assess Reflections– Have students reflect on how their academic knowledge was transformed in

service/civic contexts

4. Control Accountability– Assess individual service/civic learning carefully and discreetly– Measure service/civic programs in the aggregate

5. Align Discourse– Iteratively refine participation in service/civic activities to enhance reflections

and indirectly impact distal outcomes

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Contexts give concepts meaning– “Deep” understanding of concepts is relevance

across range of contexts.– 21st Century learning is about context

• Range of contexts in all educational settings– Across characters, units, students, etc.

• Reframe learning as a search for relevance– Learners need personally meaningful contexts– Compare value of contexts for learning concepts

#1. Transform Concepts: Reframe Academic Knowledge as Tools for Contexts

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• E-Learning Courses– Each student defines own lesson context– Wikifolios identify most relevant and least relevant

implications and relevant specific• Quest Atlantis– Must use knowledge of inquiry and ecological

science to participate in game

Reframing Context Examples

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

E-Learning ClassesStudents First Define Problem

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

E-Learning ClassesReading Text as a Search for Relevance

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

E-Learning ClassSearch For Relevant Specifics

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Service and Civic Learning is about contexts– Void between school learning and contexts– Void is an opportunity rather than an obstacle

• Range of contexts civic/service learning– From school to service/civic context– Across students in same service/civic context– Across service/civic contexts

• Service/civic learning as search for relevance– How is your academic knowledge relevant?– How is this service/civic context transforming your

knowledge?

Ideas for Reframing Academic Knowledge as Tools in Service/Civic Contexts

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Foster conversation about using conceptual tools in contexts– Different context reveal concepts differently

• Pay attention to initial participation– Must enlist tools initially before appropriately– Don’t grade or require participation– Encourage lurking

• Focus on the trajectory of the discourse– Don’t prematurely focus on accuracy

#2. Scaffold Participation: Guide Worthwhile Discourse Around Using Tools

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• E-Learning Courses– Don’t use discussion forums– Students comment on each other’s wikifolios

• Quest Atlantis– Reframe Quest submissions as informal discourse– Ensure useful feedback is provided and used

Scaffolding Participation Examples

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

wi

Example of Consequential Collaboration in Wiki Commenting

• Comments on wikifolios should lead to reflection and revision– Good use of instructor comments.

• Assume that physical and intellectual proximity supports engagement– Overcomes abstract and alienating online context

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Juana’s Example• Chapter 2 Wiki on modal model of memory• Most relevant implication was information

processing is easier when to-be-learned information is distributed in working memory

• Linda’s comment suggested misunderstanding– “students often have so much information to manage,

that they may need to drop a lot of it in order to focus on the important points. That, to me, would fall more under the description of the 'bottleneck' implication than the 'distributed in working memory' implication, which I interpret to be more about presenting information in both a visual and auditory manner.”

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Example Comment on Juana’s Wiki

• Linda’s Comment: “The way that I interpreted what you wrote for #1 is that students often have so much information to manage, that they may need to drop a lot of it in order to focus on the important points. That, to me, would fall more under the description of the 'bottleneck' implication than the 'distributed in working memory' implication, which I interpret to be more about presenting information in both a visual and auditory manner.” (emphasis added)

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Juana’s Reflection• Consequential Reflection: “After reading everyone

comments it seems as though the implications selected are appropriate for my problem except for item number 1 that I have changed from working memory to Resource and Data limitations. (Resource and data limitations constrain learning) This is a much better fit for what my comments were and after rereading that section is clearly the right choice…

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Jeraldine’s Case

• Example: Jeraldine’s Chapter 2 Wiki• Indicated “All students should be encouraged

to ‘manage their resources’” as the most relevant implication

• The comments redirected her thinking about relevance in different contexts.

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Example Comment on Jeraldine’s Wiki

• Emerald’s comment: …almost all of your explanations you wrote about the importance of automaticity, making me wonder why "Automaticity facilitates learning by reducing resource limitations" wasn't your #1 implication. I could understand why you wouldn't choose this if your lesson included more challenging texts, but reading easy texts would require a lot of automaticity even for struggling readers. (emphasis added)

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Jeraldine’s Reflection

• Collaborative Reflection: “Emerald commented on my use of automaticity throughout my wiki and wondered why the automaticity implication was not my first choice. I reread my reasoning for my first implication and realized that this is my goal for all my students. However, for my instructional problem, automaticity is the key. Hence, this implication should become my top choice. My second and third implications are almost subheadings of this top implication.”

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Anticipate different service/civic contexts in classes– Speculate about relevance early

• Articulate connections/voids between school and service/civic contexts

• Provide incentives for participation– Bring reflections in early

Ideas for Scaffolding Participation in Service/Civic Learning

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

#3. Assess Reflection: Avoid Grading Student-Produced Artifacts

• Grading artifacts undermines discourse– Requires detailed-but-shallow rubrics and

exemplars• Grade reflections as evidence of engagement– Grade infrequently– Aim to foster revision

• Focus reflections on engagement– Consequential, critical, and collaborative– Conceptual (i.e., understanding) “comes for free”

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Assessing Reflection Examples

• E-Learning Courses– Wikifolios are never graded– Grade reflections every five weeks

• Quest Atlantis– Added reflections to quest submissions– “How does your report show you have learned?”

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Reflect on Specific Types of Engagement

• Consequences of key concept in your each context– Also consequences of context for each concept

• Critique use of key concepts in each context– Critique context for learning concepts

• Collaboration associated with key concepts– Critique classes collaboration?

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Ideas for Assessing Reflection in Service/Civic Learning

• Don’t have student report on what they did or what they learned– Have them report on how they engaged

• Articulate the types of engagement for reflection– Consequences of big ideas your context

• Consequences of context for big ideas

– Critique use of big ideas in each context• Critique context for learning big ideas

– Collaboration associated with big ideas• Critique classes collaboration?

• Do reflections publically via wikis or forums– Reward reflections that lead to action

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Ideas for Assessing Reflection in Service/Civic Learning

• Don’t have student report on what they did or what they learned– Have them report on how they engaged

• Articulate the types of engagement for reflection– Consequences of big ideas your context

• Consequences of context for big ideas

– Critique use of big ideas in each context• Critique context for learning big ideas

– Collaboration associated with big ideas• Critique classes collaboration?

• Do reflections publically via wikis or forums– Reward reflections that lead to action

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

#4. Control Accountability: Use Assessments and Tests Carefully

• Downplay importance of individual assessments and tests– Align with scaffolding and reflection– Engagement should ensure adequate

understanding• Assess and evaluate just a few outcomes• Don’t use program outcomes to assess

individuals– Compromises validity

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Controlling Accountability Examples

• On-Line Learning Sciences Courses– Multiple choice midterm and exam from test bank– Worth only 15% each, use criterion for points

• Quest Atlantis– Open ended performance assessment used to

evaluate curriculum (not students)– Random sample of achievement items used to

evaluate gains over time

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Ideas for Controlling Accountability in Service/Civic Learning

• Focus on proximal outcomes– Align close level activities to them– Use “distal” individual outcomes in the aggregate– Treat program outcomes as programmatic remote outcomes

• Be very focused with surveys– They are not particularly helpful for improvement

• Control accountability internally– Don’t let outsiders control it

• Assess and measure judiciously– Just because you can does not mean you should!

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Use design-based research methods– Refine features to validate specific principles.– Build local rather than fundamental theory

• Align participation to raise all outcomes– Preserve evidential validity by changing format

• Three or more increasingly formal cycles– Design (design activities and artifacts)– Implementation (select activities & artifacts)– Evaluation (compare achievement gains)

#5. Align DiscourseRefine Participation at all Levels

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

Three Levels Of Learning Outcomes

• Interpret participation in discourse (“close”)– Foster discourse in real time to increase understanding – Protect from corruption via assessment

• Assess individual understanding (“proximal”)– Refine design and choice of activities to raise

achievement– Protect from corruption by test

• Measure aggregated outcomes (“distal”)– Evaluate increased gains between or across courses– Secure test to preserve validity

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

• Learning Sciences Courses– Refining prompts to maximize quality discussion– Looking for improved reflections– Measure achievement gains from year to year

• Quest Atlantis– Continually improve questing discourse directly– Increased gains in understanding (Δ >2 SD)– Slowly raised gains in achievement (Δ >1 SD)

Aligning Discourse Examples

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection

0

0.5

1

1.5

2

Quest Atlantis Increased Gains in Problem Solving and Achievement across Years (in SD)

Curriculum-Oriented Problem Solving (Proximal)

Standards-Oriented Achievement (Distal)

• Focus on proximal outcomes like reflections on service– Align classes and other activities to proximal

assessments• Use program outcomes to evaluate programs

rather than individuals– Find valid measures for evaluating program

improvement

Ideas for Aligning Discourse in Civic/Service Learning

Introduction Reframe ConceptsScaffold

ParticipationAlign Discourse

Control Accountability

Assess Reflection