Evaluating and Reporting Program Impacts: Contentious Public Issues

Loretta SingletaryProfessor & Extension Educator

University of Nevada Cooperative Extension

Common Assumptions Can’t measure impacts of “process”

programs because…Handing out evaluations is awkwardParticipants don’t like to complete forms I don’t need to measure the impacts because

I know when it is successful I don’t want to report results of little to no

impacts or negative impacts

Common Assumptions Can’t measure impacts of “process”

programs because…These programs are just too complexThere is too much controversy Some participants arrive angry, stay angry

and leave angryMy role is limited to that of facilitator onlyMy role is that of educator onlyMy role is fact-finder and researcher only

Evaluation DilemmaAll affected stakeholders do not

see collaboration as a better alternative

Stakeholders do not participate equally

Participants have different skills, commitment, motives

Evaluation DilemmaPlaying field is unevenStakeholders lack sufficient time and

receive varied “dosage” External forces are rushing the

processFunding and time constraints are

problems

Evaluation Required? LOGIC model is a systematic method for

program planning that requires (handout):1. Situational Analyses2. Priority Setting3. Action Plan4. Evaluation

Benefits of EvaluationServe to strengthen programs

over the short and long termEvaluations determine strategies

and methods that work best, under which conditions and for which purposes

Benefits of EvaluationEnable comparisons with

traditional decision-making processes to build public understanding of consensus-based processes

Evaluating Impacts of Programs that Address Contentious Public Issues WHAT? WHO? WHEN? HOW?

What to Evaluate? Learning objectives = desired changes Programs that address contentious

public issues try to accomplish some basics:

Educate Communicate Explore options Seek mutually satisfying agreements

What are Desired Changes? Contentious issue efficiently addressed Multiple options developed Multiple options considered Climate becomes less contentious Climate supports public discourse Conflict is resolved for mutual satisfaction

Primary Impacts (Gray, 1989)

Two objective criteria to measure impacts of a collaborative effort:1. Negotiated agreement is reached2. Agreement has been implemented

Secondary Impacts Subjective criteria show impacts of

collaborationAlter individuals’ skills to participate Alter perceptions of othersAlter perceptions of issueAlter attitudes Alter behavior Develop criteria to measure these changes

Secondary Impact Measures Improved…

Communication skillsNetworking skillsRelationship building skills

Increased hope for resolving the issue

Secondary Impact Measures Quality of Interaction

Volunteers learn how to share powerTreat one another fairly and with respect

Operational DetailsMeetings follow timelineMeetings accomplish goalsParticipants stay informed (Gray 1989)

Secondary Impacts… First order vs. second order effects Second order effects…

Increased knowledge about issues Increased awareness of issues Increased awareness of diverse views New personal and working relationships Stakeholders accept and understand scientific

analyses Respond to future conflict civilly (Innes, 1999)

Best when retrospectively measured

When to Evaluate? Midcourse – partway through to assess

progress and improvements End-of-process – at conclusion to

assess learning, satisfaction and first order effects

Retrospective – 12 to 18 months after process to identify value and stability of process and second order effects

Midcourse Evaluation After a few months of process

program to assess “how it is going”… Are process criteria being met? Are participants satisfied? Are representatives communicating with

their stakeholder groups? Do all participants feel well informed? Do all participants feel equally empowered? Are participants engaged and interested?

End-of-Process Evaluation Within month or two of final meeting Experience still “fresh” in everyone’s

mind Objectively assess accomplishments

Participants feel time well spent? Participants plan to stay involved with

implementing agreement? Participants would try this type of process

again?

Retrospective Evaluation Of single process takes place at least

one year after process completed Examines immediate and longer term

effects of program Traces over time program effects on

participants and their relationships to one another

Who to Evaluate? Program objectives determines WHO

Joint fact finding Policy dialogue Public-private partnerships Regulatory negotiation Advisory Committees Public Forums Seeking negotiated agreement

Who to Evaluate? Program objectives determines WHO Process participants (stakeholders—

those directly involved) Agencies not directly involved Citizens not directly involved Funding sources Program speakers, researchers,

facilitators

How to Evaluate?

Focus Groups (least expensive) Face-to-face interviews Postal surveys Internet surveys

How to Evaluate?Principles for Writing QuestionsPrinciples for Writing Questions

Simple wording, yet complete sentencesSimple wording, yet complete sentences Avoid vague quantifiersAvoid vague quantifiers Avoid specificity (how many)Avoid specificity (how many) Equal scalar (positive and negative)Equal scalar (positive and negative) Provide “Don’t Know” category for each question *Provide “Don’t Know” category for each question * State both ends of scale equally (Poor / Excellent)State both ends of scale equally (Poor / Excellent) Eliminate “check all that apply”Eliminate “check all that apply” Avoid open ended questionsAvoid open ended questions Offer ordered responses (scale)Offer ordered responses (scale)

How to Evaluate?

Close-ended questionsClose-ended questions with with ordered responsesordered responses provide provide participants with a response scale;participants with a response scale;Select one answer from a scale or Select one answer from a scale or fixed range of choices. fixed range of choices.

How to Evaluate?All citizens have a responsibility to All citizens have a responsibility to conserve water on a daily basis.conserve water on a daily basis. Strongly disagreeStrongly disagree Somewhat disagreeSomewhat disagree Neither agree nor disagreeNeither agree nor disagree Somewhat agreeSomewhat agree Strongly agreeStrongly agree

How to Evaluate?

My ability to listen to others’ views has:My ability to listen to others’ views has: Improved very littleImproved very little Improved littleImproved little Stayed about the sameStayed about the same Improved muchImproved much Improved very muchImproved very much

How to Evaluate?My understanding of the effect of My understanding of the effect of salinity on the Lahonton Cutthroat Trout salinity on the Lahonton Cutthroat Trout fishery has:fishery has: Improved very littleImproved very little Improved littleImproved little Stayed about the sameStayed about the same Improved muchImproved much Improved very muchImproved very much

How to Evaluate?

Close-ended questionsClose-ended questions with with unordered responsesunordered responses present present answers in no particular order and answers in no particular order and participants must pick the one best participants must pick the one best response. response.

How to Evaluate?Whose responsibility is it to see to it that Whose responsibility is it to see to it that

watershed water quality improves? (check watershed water quality improves? (check only one):only one):

Area farmers Area farmers Environmental Protection Agency Environmental Protection Agency My Regional Watershed Planning Group My Regional Watershed Planning Group Local elected officials Local elected officials Environmental interest groupsEnvironmental interest groups Every citizen living in the watershed area Every citizen living in the watershed area

How to Evaluate?An example of a An example of a partially closed-ended partially closed-ended questionquestion with with unordered responseunordered response is: is: How do you prefer to receive ABC Program How do you prefer to receive ABC Program information? (check all that apply)information? (check all that apply)

Printed material mailed to mePrinted material mailed to me Small group meetingsSmall group meetings Large public forumsLarge public forums Newspaper articlesNewspaper articles Other (please describe) Other (please describe) ________________________________________________________________

Reporting Quantitative Impacts Ranked means using Likert scale Percentages collapsing highs/low

Likert scale Regression analyses using participant

characteristics to find impact patterns Several other tests for relationships

Reporting Qualitative Impacts Frequency count by key words

VerbsNounsDirect objects

Categorize comments by classLiked mostLiked least

Reporting Impacts Citizen participants Participating agencies Your administrators Land-grant universities (academic body) Funding sources

Reporting Impacts 4-page fact sheets In-depth Extension publications Power point presentations to citizens Professional conferences (ANREP,

NACAA, NACDEP, ESP, Galaxy) Refereed journal articles (JOE) News articles Radio and television interviews

Bibliography Bickman, L., Rog, D.J. & Hedrick, T.E. 1998. Bickman, L., Rog, D.J. & Hedrick, T.E. 1998.

Applied Research Design: A Practical Approach Applied Research Design: A Practical Approach in L. Bickman and D.J. Rog (eds.) Handbook of in L. Bickman and D.J. Rog (eds.) Handbook of Applied Social Research Methods. Thousand Applied Social Research Methods. Thousand Oaks, CA: Sage Publications.Oaks, CA: Sage Publications.

Dillman, D. A. 2000. Mail and Internet Dillman, D. A. 2000. Mail and Internet Surveys: The Tailored Design Method, 2nd Surveys: The Tailored Design Method, 2nd edition. New York: John Wiley and Sons.edition. New York: John Wiley and Sons.

Dillman, D. A. 1978. Mail and Telephone Dillman, D. A. 1978. Mail and Telephone Surveys: The Total Design Method. New York, Surveys: The Total Design Method. New York, NY: John Wiley and Sons.NY: John Wiley and Sons.

Gray, B. 1989. Collaborating: Finding Gray, B. 1989. Collaborating: Finding Common Ground for Multiparty Problems Common Ground for Multiparty Problems

Bibliography Heron, J. 1996. Co-operative Inquiry: Research into Heron, J. 1996. Co-operative Inquiry: Research into

the Human Condition. Thousand Oaks, CA: Sage the Human Condition. Thousand Oaks, CA: Sage Publications.Publications.

Innes, J. E. 1999. Evaluating Consensus Building. Innes, J. E. 1999. Evaluating Consensus Building. Thousand Oaks, CA: Sage Publications.Thousand Oaks, CA: Sage Publications.

Patton, M.Q. 1978. Utilization-Focused Evaluation. Patton, M.Q. 1978. Utilization-Focused Evaluation. Beverly Hills, CA: Sage Publications.Beverly Hills, CA: Sage Publications.

Salant, P. & Dillman, D.A. 1994. How to Conduct Your Salant, P. & Dillman, D.A. 1994. How to Conduct Your Own Survey. New York: John Wiley and Sons.Own Survey. New York: John Wiley and Sons.

Singletary, L. 2004. Mail Surveys, in L. Singletary (Ed.) Singletary, L. 2004. Mail Surveys, in L. Singletary (Ed.) Conducting Community Situational Analysis: A Field Conducting Community Situational Analysis: A Field Guide to Dynamic Extension Programming. Reno, NV: Guide to Dynamic Extension Programming. Reno, NV: UNCE EB-04-02,available online at UNCE EB-04-02,available online at www.unce.unr.edu.

Loretta Singletary, Ph.D.Professor and Extension EducatorUniversity of Nevada Cooperative ExtensionPhone: (775) 463-6541 FAX: (775) 463-6545 Email: singletaryl@unce.unr.edu