Evaluation and Attitude AssessmentBME GK-12 Fellows

12/1/10, 4-5pm 321 Weill Hall

Tom ArchibaldPhD Student, Adult & Extension Education

GRA, Cornell Office for Research on Evaluation

quick overview of evaluation

• merit and worth• program improvement• knowledge generation• very similar to research, with some differences

CORE’s “touchstone” statement on the goal of evaluation:

To obtain accurate, useful insights about the answers to evaluation questions in a manner that

is feasible, is credible to relevant stakeholders, makes strategic use of limited time and resources,

and contributes to our general knowledge, to future evaluations and to program evolution.

a systems approach for evaluation planning

• Programs are viewed (and modeled) as parts of larger systems dynamic and evolving related to other programs in the present connected to past and future programs being perceived differently by different stakeholders

and systems• Evaluation plan development takes this into

account and should help programs and systems evolve

4

5

6

Initiation Development Maturity Translation &Dissemination

RET

Program Phase

Evaluation Phase

REU

Module-Based Education

Step 2.05 - Program and Lifecycle Analysis

7

Step 2.07 - Pathway Modeling

The key elements of Evaluation Plan quality are

1. Consistency with a high-quality program model

2. Fitness of evaluation plan elements to the program and program context

3. Internal alignment of the evaluation plan elements

4. Holistic coherence

8

1. Consistency with a high-quality program model

– A “high-quality model” …• is grounded in knowledge of the program • incorporates perspectives of multiple stakeholders• shows causal pathways (program logic)• reflects careful thought about program “boundaries”• includes program assumptions and key elements of context• is connected to program evidence base (relevant research)

– “consistency with…” means• evaluation questions can be located in terms of model elements• evaluation “scope” makes sense

9

2. Fitness of evaluation questions and other evaluation plan elements to the program and program context

– Evaluation questions are “mined” from the model– Evaluation questions are appropriate for the program’s

maturity and stability, existing state of knowledge, and program needs

– Evaluation focus, methods, and tools meet needs of key stakeholders

– Evaluation plan makes efficient and strategic use of program and evaluation resources

10

3. Internal alignment of the evaluation plan elements

– Measures fits the constructs – Measure is the most strategic option among those that fit– Design is appropriate for lifecycle stage– Design can support claims implied in purpose statement– Sampling and Analysis plans can generate evidence

needed

11

4. Holistic coherence… the elusive element

– Evaluation planning requires myriad decisions about multi-faceted tradeoffs

– Making these decisions well requires a holistic comprehension of the program and the environment and systems that embed it

– The decisions may be invisible in the written plan so some of the resulting quality might be ascertainable, some of it will not be

12

Getting to Measures Worksheet 1. Starting Points: (a) Copy here what you want to know about your program

for this Evaluation Question. (b) Copy here your formal Evaluation Question for this

inquiry. 2. Clarify the “constructs” in your Evaluation Question

above – what exactly do you mean by the activity and/or outcome you are focusing on? Note: revisit your Evaluation Question above and see if it needs to be revised to capture this precision.

3. Using this sharper definition of what’s being evaluated, work

through the following sequence of questions: (a) What would this “look like” or consist of in practice? (List as

many options as you can. Be creative – think outside the “usual” options.)

(b) What might serve as evidence? For the most promising

candidates in (a), write down various kinds of evidence that would be informative about whether this has occurred.

(c) Review the strengths and weaknesses of the options, taking into

account several aspects that matter: “closeness” to the real thing; accuracy and reliability. Write down a short list of the most promising candidates from (b).

4. How could you gather this evidence? Again, think of as many options as you can and list them here. (Possibilities might include video-taping a demonstration, live observation during class sessions, directly asking participants, asking people who know the participants, testing the end-products, etc.) Identify the ways that seem best, taking into account accuracy, feasibility, fit with program context and target population. Indicate the top choices among those on your list.Once again, take a moment to revisit your Evaluation Question phrasing in 1(b) and see if it needs updating.

5. What types of measure would be needed? From your answer(s) to 4,

now ask yourself what type(s) of measure would allow you to gather evidence in this way. Here is a list of possible measure types from which to choose: case study, interview, observation, group assessment (e.g. focus group), expert or peer review, portfolio review, testimonial, test of knowledge or skill, photograph, slide or video, diary or journal, log, document analysis, action cards, simulation, problem story, creative expression, unobtrusive measures.

For more explanation of measure types, see:

Taylor-Powell, E., S. Steele (1996) Collecting Evaluation Data: An Overview of Sources and Methods, Cooperative Extension Publications, University of Wisconsin, Madison, WI. (http://learningstore.uwex.edu/Collecting-Evaluation-Data-An-Overview-of-Sources-and-Methods-P1025C237.aspx)

find v. writeSummary checklist (questions to help make the “find” versus “write” decision): 1.Is the evaluation question assessing an outcome? If yes, do you need

to make a strong claim of effectiveness? 2. Do you need to compare your evaluation results to those of others or

to some external norm? 3. Do stakeholders require a pre-validated measure? 4. Are staff resources available to draft measures in accordance with

best practices? To conduct a high quality search for measures? 5.After a quick scan, does it appear you are likely to be able to find

measure that fits your construct through peers, the Netway, your program system, program funders or professional groups related