Measuring Success & Impact: Challenges &

Opportunities for STEM Early Learning Programs

Tiffany R. LeeUniversity of Colorado Boulder

University of Washington

Introductions

• Name, organization, role• Short project description• Experience with measuring success and impact

(evaluation) – for this project or others

Lee 2014

Words (wordle?)

• Design• Implement• Explore• Evaluation• Assessment• Goals• Measure• Impact• Success• Refine

• Test• Scale• Learning• Reach• Data• Change• Outcomes• Feedback

• Others?Lee 2014

Others:Evidence

ScopeAnalysis

Time

What outcomes would indicate to you that your

program is achieving success?

First, let’s think about these questions..• What resources go into your program? (ex:

funding, staff, partners)• What activities does your program develop,

support, and implement?• What is produced from the activities and the

program’s efforts?

Lee 2014

What outcomes would indicate to you that your program is achieving success?• Development of early childhood STEM lessons /

activities?• Increased effectiveness of program facilitators?• Child learning of particular content or skills?• Change in parents’ / caregivers’ behavior?• Increased community reach?

Lee 2014

What are the changes or performances that result from

the program? Be specific.

Outcomes/Goals

Inputs

Activities

Outputs

What resources go into your program? (ex: funding, staff, partners)

What activities does your program develop, support, and implement?

What is produced from the activities and the program’s efforts?

What are the changes or performances that result from the program?

Prog

ram

Log

ic M

odel

Some Considerations for Measuring Success…

Some Questions to Consider

About Your Participants• Does your program have repeat vs. one-time interactions

over time with participants – children and/or caregivers?• Do you have a way to contact and follow up with your

program participants?

About Your Program• Do you have the same facilitators leading lessons /

activities throughout the duration of your program?• Does your program implement a lesson / activity once, or

do you refine it over multiple iterations?

Lee 2014

Your answers to these questions make a difference in what (and how) you can

measure success!

Questions About Your Participants Repeat vs. one-time interactions over time with children and/or caregivers• Can you measure changes in child / caregiver knowledge

and skills?• Alternatives: Changes in

group behavior, facilitation, work products (ex: child artwork)

Lee 2014

Questions About Your ParticipantsParticipant contact info• Can you collect information about what the child /

caregiver does that is related to a program experience?• Alternatives: Exit tickets,

repeat interactions?

Lee 2014

Questions About Your ProgramSame facilitators leading lessons / activity• Can you measure changes in individual’s facilitation

skills?• Alternatives: Differences between facilitators

Lee 2014

Questions About Your ProgramOne-time vs. multiple iterations of lesson / activity

• What is the rationale for making changes to a lesson / activity? Do they result in changes in participant engagement (as a group)?

• Alternatives: Design principles

Lee 2014

Methods for Measuring SuccessMethods• Descriptive statistics• Time sampling• Event sampling• Field notes• Exit tickets• Work samples• Photographs• Video / audio recordings• Conversations with caregivers• Team reflections

Lee 2014

Considerations• Time • Timing / frequency• Staffing• Permissions• Materials• Subjectivity / Objectivity

Refining Outcomes and Identifying Measures of

Success

Identifying Measures of Success

• Review Outcomes / Goals– Can you measure these given the constraints of your

program? (Think about your access to participants and program design)

– What does it look like when your program goals are met?

• Brainstorm possible measures– What resources will be needed? (additional staff member

on site, develop protocols)– What are some challenges and considerations?

Lee 2014

Indicators

Methods/Measures

Inputs

What methods/methods will be used to measure your outcomes/goals?

What resources will be needed to implement your methods/measures?

What does it look like when your program goals are met?

Mea

surin

g Su

cces

s

Challenges/Considerations

What are some anticipated challenges and considerations for implementing these measures?

Exit Ticket

1. What do you think will be the biggest challenge when implementing an evaluation plan?

2. What is most exciting to you about evaluation?

3. What is the most helpful thing you learned today?

Lee 2014

Thank you!

Tiffany Lee

tiffanyrlee@gmail.com