Improving Local Evaluation through Training and Technical Assistance:

Wisconsin’s Strategy

Mary D. Michaud, MPP Ellen Taylor-Powell, PhD Bonita Westover, MSPH

University of Wisconsin-Extension

Acknowledgements

The authors wish to thank staff and members of the following organizations who are making Wisconsin's strategy for providing local program development and evaluation possible:

Members of Wisconsin's 77 Tobacco-Free CoalitionsWisconsin Tobacco Control BoardState of WI Department of Public HealthUW-Comprehensive Cancer CenterUW-Center for Health Policy and Program EvaluationUW-Center for Tobacco Research and InterventionUW-Cooperative Extension

What we will cover

Overview of our Wisconsin initiative Typical questions we receive Real examples of incorporating evaluation into

coalition activities Using a logic model to guide long-range

planning as prerequisite for useful evaluation

BackgroundWisconsin Tobacco Control Board Comprehensive program 5 year goals Commitment to evaluation

Monitoring and Evaluation Program (MEP)

MEP

MONITORINGUW-Comprehensive

Cancer Center

STATEWIDE PROGRAMEVALUATION

UW-Center for Health Policy and Program Evaluation

LOCAL EVALUATIONUW-Extension

Local Program Evaluation

GOAL:

Build capacity in program development and evaluation enabling local coalitions to effectively design, implement and assess tobacco control programs

What are you doing? What difference is it making for reducing tobacco use? How do you know?

How we define evaluation capacity

Evaluation capacity is having resources and ability to engage in evaluation that leads to learning, program improvement and enhanced accountability

Prerequisites: committed leadership, resources – technical and financial, attitude that values evaluation

How we do this: Operating principles

Empowerment/participatory approach Community members can learn and use planning

and evaluation concepts, techniques and findings to evaluate themselves and their programs to improve practice.

Coalitions conduct their own evaluation; our professional role is one of trainer, consultant, facilitator, coach.

Local advisory group of coalition members will help provide direction and feedback.

Operating principles….

Evaluation value Important learning occurs during the process of

‘doing’ evaluation that impacts those involved and leads to more effective programs and enhanced outcomes

Evaluation is more than measurement, findings and external reporting.

Value lies in learning and continuous improvement.

Operating principles…

Practical approach

Approaches and methods will be used that are practical, innovative and appropriate for cultural and low-resource contexts.

Participatory adult education principles will be applied.

Operating principles…

Mixed approach There are no “cookie cutter” approaches or

answers

Heterogeneity of coalitions and local contexts demands mixed approaches and mixed methods.

Innovation and creativity are key

Operating principles

Research base We will use research and best practices in program

planning and evaluation.

We hold ourselves to the same standards of accountability in learning and use of evaluation.

We will apply the evaluation standards: utility, feasibility, propriety, accuracy

UWEX staff

Grant $$

Research

Evaluatoin Advisory Group

Partners

Assess needs and assets

Develop tobacco specific planning and evaluation materials

Provide training and technical assistance

Facilitate cross site sharing

Local DH officer

Partners: DPH; CTRI; WTCB; MEP; Smokefree

Increased valuing of evaluation

Increased involvement in planning and evaluation

Increased resources committed to planning and evaluation

Increased knowledge and ability to collect and use data

Increased confidence and motivation to engage in evaluation

Increased number of coalitions that demonstrate actions of effective planning and evaluation

Integration of evaluation into coalition operations

Increased local evaluation capacity

More effective programs

Improved outcomes – WTCB goals

Coalitions

- facilitator

- members

Work with partners to create environment that understands and values evaluation

Our logic model

Our structure and processStructure Regional collaboration Statewide coordinator 5 regional specialists Local advisory group

Process Training: small-large group, face-to-face, distance TA: customized, individualized Resource development; distribution Partnering

Regional Collaborative Model

Wisconsin Regions

Content of our T and TA

Demystify evaluation Logic models Long-range planning: Focus on WTCB goals;

integrate evaluation Stakeholder engagement Writing outcomes (SMART objectives) Evaluation planning: process and outcomes Components of evaluation: Focus, data

collection, analysis and interpretation, use

What we do

Conduct workshops, 1meetingsDeliver servicesDevelop products, curriculum, resourcesTrainProvide counselingAssessFacilitatePartnerWork with media

Whom we reach

Participants

Clients

Agencies

Decision- makers

Customers

What the short term results are

Learning

Awareness

Knowledge

Attitudes

Skills

Opinions

Aspirations

Motivations

What the ultimateimpact(s) is

Conditions

Social

Economic

Civic

Environmental

What the medium term results are

Action

Behavior

Practice

Decision-

making

Policies

Social action

Assumptions External Factors

What we invest

Staff

Volunteers

Time

Money

Research base

Materials

Equipment

Technology

Partners

Outputs

Activities ParticipationInputsOutcomes – Impact

Short Term Medium Term Long Term

www.uwex.edu/ces/pdande

www.uwex.edu/ces/tobaccoeval/

Assessing strengths and barriers to evaluation capacity building

Attitudes Involvement Leadership Resources Knowledge-skills

Logic model Planning

Focus Data collection Analysis and interpretation Use

Multiple partners with different contexts and cultures require plenty of time for relationship building

Great variation in coalitions: type, functioning, resources, history, interest and abilities. Must start with where coalition is

Staffing: need for technical expertise, adult education, facilitation/relationship building/political savvy

Budget uncertainties created even greater need for constant communications and support

Lessons learned/learning

Lessons…

Annual deliverables resulted in an “evaluation frenzy”

Need for long-range planning What are tobacco “best practices”? Not all coalitions need or can use evaluation T

and TA; analysis may be contracted out General T and TA provides foundation but need

is for practical, “real” applications And the learning continues…

Typical Evaluation Technical Assistance Requests

“There is no such thing as ‘typical’.”

A Breadth of Examples of Evaluation Technical Assistance Requests

Q: How should I evaluate the effects of TATU on middle school kids?

A: “Let’s consider some possibilities…” Work with High School kids to identify learning

objectives. Develop evaluation questions based upon

learning objectives. Engage HS kids in development of pre/post

survey instrument and/or group interviews.

Q. Could you please review this survey and give me feedback for improvement?

A. “Could we start first with…

What is the purpose of the survey? What do you want to learn? Who will use the results- for what? Are you sure a survey is the right method?

“Let’s cover some tips for improving” surveys…

Use agency letterhead to improve credibility. Introduce survey – Who are you? Why are you

conducting survey – its importance? How will the information be used?

Keep it concise! If possible, provide incentives for responding. Make time for follow-ups

Improving Surveys…

Local program evaluation web sitehttp://www.uwex.edu/ces/tobaccoeval/

Search on “surveys”. Program Development & Evaluation web site

http://www.uwex.edu/ces/pdande Go to “Evaluation Publications” Go to “Quick Tips”

Q: We have conducted so many surveys in the past year. Are there some other evaluation things I can be doing?

A: “YES!” Consider using qualitative methods to gain

greater depth. Not everything needs to be evaluated

Frantic:

We have all these data & I don’t know what to do with them!

A. “I’ll walk you through what to do”First, what did you want to know/learn when

you collected this information?What do you hope to learn from these data?Are your data of good enough quality to merit

analysis? Who/how many responded? What was the sample?

I’ll walk you through what to do…

Are all the data together? In one place? Ready for analysis?

CodeEnter into a data management programCleanRun Frequencies and percentagesCall me as you have more questions

Q: Excel is making me crazy! How can I…?

A: Contact Jenny – she is our Excel guru and can

walk you through specific issues with the program.