Western Extension Risk Management Education Center 2019 … · 2019. 5. 3. · Navajo Ag & Natural...
Transcript of Western Extension Risk Management Education Center 2019 … · 2019. 5. 3. · Navajo Ag & Natural...
Western Extension Risk Management Education Center2019 Project Director Orientation
1:00PM Pacific Time – May 2, 2019
2019 Project Orientation
Documenting Producer Success through Performance MeasuresA Systems Evaluation Model
Shannon Neibergs, Director
Jo Ann Warner, Associate Director
Heather Schriver, Grants & Communications Coordinator
Webinar Logistics
• Type questions in chat box• Mute computer mic• Mute phone if calling in by phone• Recording and posted on website• Performance Measure/Evaluation resources on web
• Networking – emails
2019 Funded ProjectsTheresa Kiehn, [email protected], AgSafe, Labor Considerations for Scaling Up Production
Michael Patrick, [email protected], NMSU, Navajo Ag & Natural Resource Sustainability Project
Ryan Larsen, [email protected], USU, Risk & Profitability in Beef Production
Max Bennett, [email protected], OSU, Risk Mgmt Ed for Forest Producers in SW OR
Hernan Tejeda, [email protected], Regents UI, Wkshps to Improve RM & Fin. Deci-Mkg for Dairy Frmrs
Justin Oldfield, [email protected], CA Cattlemen's Assoc., Livestock Traceability
Megan Van Emon, [email protected], MSU, MT Master Cattle Producer's Program
Natalia Pinzon, [email protected], MESA, CA Farmers Build Fire Resilience
Tracy Mosley, [email protected], MSU Ext. Park Cty, Farm & Ranch Estate & Transition Planning
Ruby Ward, [email protected], USU, RME for Native American Farmers & Ranchers
Ruth Dahlquist-Willard, [email protected], UCANR, Bilingual Food Safety Trng for SE Asian & Latino Small-Scale Farms
Jean Brokish, [email protected], Oahu RC&D, Skill Up! Utilizing Agritourism to Grow Farm Income
Kirk Hanson, [email protected], NNRG, Reducing Prdcr Rsk in the San Juan Islands Thru Improved Forest Mgmt
Jay Olsen, [email protected], Snow College, Anlyzng Contributions of Entrprss Dvrsfctn to Frm Prftblty for
Chris Brown, [email protected], ALBA, Self-directed Mktg Approaches for Small Holder farms (SMASH)
What We Are Going to Cover
Alignment of ERME Goals with your Program Goals Using a Systems Evaluation Model
Risk Management OverviewSystems Evaluation ApproachReporting – Communicating Success with Evaluation
FindingsNew ERME Reporting Requirements to Establish a
National “Understanding” Benchmark MetricInvoicing and Reporting Logistics
USDA/NIFA Programs
Grounded in USDA Strategic GoalsRequire planning and accountability measuresRely upon streamlined data collectionReduces reporting burdenCreates a more transparent reporting process
Robust evidence based system Shared knowledge base
Alignment of your Project Goals with ERME Program Goals
Extension Risk Management Education (ERME)Mission:Educating America’s farmers and ranchers to manage the unique risks of producing food for the world’s table. Overarching Goal:Helping farm and ranch families improve their economic viability through targeted risk management education.
Extension Risk Management Education
Economic Viability:
Legislative Authority: To educate agricultural producers with training that improves their economic viabilityEconomic viability refers to the farm’s or ranch’s
ability to improve profitability and to increase opportunities for equity growth over time.Successful projects provide evidence of economic
gains for producers.
ERME Program Goals – Linking the Five Risk Areas to Improved Economic Viability
1. Production: Ensure profitability through increasing yields and reducing input costs
2. Market: Ensure increased revenues by increasing sales and execution of optimal market pricing
3. Financial: Ensure profitability by increasing cash flow and growing business equity
4. Legal: Ensure profitability by reducing legal risk exposure and increasing compliance
5. Human: Ensure profitability by developing labor and human capital for worker and family business success.
**Economic or profit indicator of reduced risk which is constant across all 5 risk areas
Performance Measures & Documenting Producer Success Systems Evaluation ModelIntegrates application content
1. Results – what you want participants to accomplish2. Rationale – Project Results Narrative3. Curriculum/Tools/Resources/Collaborators4. Activities (Trainings/workshops/field trips)
Establish a successful monitoring and performance measurement systemInput into the Results Verification System (RVS)
1. Quarterly Progress Reports2. Final Report
Article: Program Evaluation and Results Verification: https://westrme.wsu.edu/news/winter-2017-newsletter/program-evaluation/
A Systems Evaluation Approach:Creating a Road Map for your Project
Insert into a framework that:Aligns producer outcomes with ERME economic
program goalsProvides a rationale for how your project activities
will lead producers to the desired resultsMonitors project performanceMeasures results across the life of the project (short-
medium-long term)Uses different types of data (from
participants/industry and government) to measure progress towards improved profitability
End Goal for Producer Actions is Improved Profitability
Five levels of producer actions in order of increasing producer engagement:
1. Understand2. Analyze3. Develop4. Decide5. Implement
**
KnowledgeShort Term
Necessary Pre-ConditionsActions
Medium
ConditionsLong Term
A Systems Evaluation Approach –What You Will Measure
1. Short term learning – understanding (immediately after initial or introductory training)
2. Short term objectives – develop/analyze (changes in behavior through undertaking specific actions within 3-6 months)
3. Medium term strategies – decide (changes in practice that result in a measurable reduction of risk within 6-12 months)
4. Long term outcome measures – implement(achievement of improved economic viability with evidence of realized economic gains within 12-18 months)
A Systems Evaluation Approach –How You Will Verify
Conduct a series of evaluations that provides evidence of achieved outcomes:
a) Retrospective Evaluation for “Understanding” (Near Term)
b) Follow up Evaluations – within 3-6 months (Short - Medium Term)
c) Follow up Evaluations – within 6-18 months (Long Term)
A Systems Evaluation Approach –Questions that Measure Results
Develops detailed evaluation questions from your proposed results that answer the expectation of improved profitability for your participants.
Do they adequately describe and can they effectively measure the producer results identified in the proposal?
**Program Resources:Survey Design Techniques: https://s3.wp.wsu.edu/uploads/sites/2162/2017/03/Program-Evaluation-and-Survey-Design-Techniques.pdf
Measuring Understanding – A Short Term Learning Goal
Result: Livestock producers will learn how to develop a data-driven price strategy for direct-to-consumer markets.
Evaluation question that measures level of understanding: How do you rate your understanding of data-driven price strategies for direct to consumer markets after participation in this workshop? (Likert numbered scale)
Retrospective Evaluation for Measuring “Understanding”• An example of the retrospective evaluation
question:
• How do you rate your understanding of market risk management for developing a data driven price strategy for direct to consumer markets as a result of participating in this risk management workshop (series)?
No A A CompleteUnderstanding Little Neutral Lot Understanding
a. Before the meeting 1 2 3 4 5 b. After the meeting 1 2 3 4 5
Measuring Short Term Actions (Develop/Analyze)
Result: Livestock producers will initiate a marketing plan specific to their own operation.
Evaluation question that measures adoption of knowledge/skills, actual changes in behavior and practice. (Short term objectives)Please indicate your level of agreement with the following statement(s) (Likert rating scale): I adopted a data driven marketing strategy for interaction with specific meat processing outlets.
Measuring Medium Term Actions (Decide)
Result: Livestock producers will transition their herds from commodity to differentiated markets.
Evaluation question measures changes in practice that result in measurable reduction of risk. Please indicate your level of agreement with the following statement(s) (Likert rating scale): I followed a data driven marketing strategy to convert my herd to a certified management and marketing plan.
Measuring Long Term Outcomes (Implementation)
Result: Livestock producers will transition their herds from commodity to differentiated markets.Evaluation question measures achievement of improved economic viability or realized economic gainPlease estimate the economic gain from higher prices received for products in differentiated markets:$0-$2,000$2,000-$5,000$5,000-$10,000$10,000-$25,000$25,000-$50,000
Benefits Accrued – Long Term Economic Improvement for Producers
Evaluation question measures achievement of improved economic viability or realized economic gainWhat is the value to you of attending this workshop series; or the benefits you estimate will accrue to your operation from changes you have made as a result of attending these workshops?Less than $100$100-$500$500-$1000More than $1,000
Evaluation FindingsContribute to program developmentEvidence Based
Collection of data that is useful and validates change (measures what is really happening)
Links producer success stories (anecdotal and testimonials) with data for accountability.
Builds upon continuous collaboration with participants to earn their trust.
Produces a return on investment (ROI) for your target audience for improved economic viability. (Benefits accrued to Producers)
Fulfills ERME goal for economic viabilityProvides justification for ROI of NIFA fundsWorthwhile investment of taxpayer dollars
Systems Evaluation Model Within a Performance Measure Framework –
Your Final Report & Public Value Level Impacts
Demonstrates: Achievement of participant outcomes and program success in a public reporting system available to all stakeholder partners
USDA GovernmentalCongressionalPublic/Private Sector StakeholdersCommunitiesFunding AgenciesUpper Administration at Universities
NEW Reporting Requirements
1. ERME Benchmark Metric for “Understanding”
a) Each project will be asked to assess changes in understanding using a retrospective design.
b) A retrospective evaluation measures participants’ previous versus current knowledge on a risk topic area(s) at the end of a workshop or training series.
**All ERME projects are expected to conduct this retrospective evaluation at the end of their workshop or training series
ERME Retrospective Evaluation Across the Project Portfolio
• Prepare a retrospective evaluation for each area of risk that your project covered (from the five broad risk areas)Develop a separate evaluation question for each risk.
• Use a 5 point numbered (Likert) scale1 will indicate No Understanding to5 indicating Complete Understanding
• Administer to participating producers at the end of a workshop or training series.
Retrospective Evaluation for Measuring “Understanding”• The basic retrospective evaluation question for each risk area
is as follows:
• How do you rate your understanding of ___(insert risk area)___ risk management (including ___(insert relevant topics)___) as a result of participating in this risk management workshop (series)?
• The response for each evaluation question is:No Complete
Understanding Understandinga. Before the meeting 1 2 3 4 5 b. After the meeting 1 2 3 4 5
Retrospective Evaluation for Measuring “Understanding”• An example of the retrospective evaluation
question:
• How do you rate your understanding of market risk management for developing a data driven price strategy for direct to consumer markets as a result of participating in this risk management workshop (series)?
No CompleteUnderstanding Understanding
a. Before the meeting 1 2 3 4 5 b. After the meeting 1 2 3 4 5
Retrospective Evaluation for Measuring “Understanding”• An example of the retrospective evaluation
question:
• How do you rate your understanding of financial risk management for developing a business plan and enterprise budgets; and for using livestock decision aids as a result of participating in this risk management workshop (series)?
No CompleteUnderstanding Understanding
a. Before the meeting 1 2 3 4 5 b. After the meeting 1 2 3 4 5
NEW Reporting Requirements
2. Write at least one story about an individual producer in the following format:a) Describe the situationb) Include outcomes accomplished by that producerc) Describe any overarching impacts; i.e. to the
family/community/environment etc.i. Include evidence or indicators of improved profitability or
economic gaind) Include a testimonial from this individual producer or
family member
Story Format ExampleSituation: This family owned enterprise was running a fairly traditional cow calf operation, growing their own hay to feed their livestock. Constraints on their time was an ongoing issue…..Outcomes: Conducting a comprehensive evaluation of the their operation through the use of QB’s resulted in the following outcomes:
1. Significantly reduced feed costs…2. Acquired additional income through…3. Transitioned herd from….
Impacts: As a result of these changes, family relationships improved....and their average economic gain for the year was $....Producer Quote: “This program provided me with the tools I needed to convert my herd from a conventional operation to include differentiated markets….”Link to Producer Story: https://s3.wp.wsu.edu/uploads/sites/2162/2019/03/WCRME-Outstanding-Project-Success-Story-2019-1.pdf
RVS Under Development
Data input for Retrospective Evaluation
Story input for Individual Producer Story
Questions/Discussion?
Impact Reporting – Communicating your Success in the Results Verification System (RVS)
Reflect back on your Systems Evaluation Model – At the end of 18 months: What strategies did producers put into practice that resulted in
measurable reductions of risk? How many producers were able to reduce input costs/increase
sales/ensure profitability by growing business equity?a) Include indicators of profitability and;b) Evidence of realized economic gain
Impact:
Describe how your project led to improved economic resilience and viability for your producer participants.
Impact Reporting – Communicating your Success in the Results Verification System (RVS)
Summary of Evaluation Findings1. Project Overview – PAST Tense2. Proposed Results – ACTUAL Results and # of participants who
achieved
Reflect back on your Systems Evaluation Model1. Project Comments
a. Unexpected resultsb. Project Improvementsc. Keys to Success
Data collection methods and instruments that capture producer stories and indicate improved economic resilience and viability
a) Individual Story(s)/Quotes
Award Management1. http://westrme.wsu.edu/award-management/tools-
for-project-directors/2. http://westrme.wsu.edu/award-
management/reporting/education-projects/
**Bookmark these pages!
Award Management(Tools for Project Directors Tab)
• Results Verification – Documenting Risk Management Outcomes through Performance Measures
• Evaluation Resources• Funding Acknowledgement and Logo Requirements• InvoicingUniversityNon-UniversitySample Invoice
Award Management
• Acknowledgement of Support:• This material is based upon work supported by
USDA/NIFA under Award Number 2018-70027-28587.
• Logos – USDA/NIFA and Western ERME
Award Management
– Acknowledgment of Support• Proper Naming Convention
Western Extension Risk Management Education Center
OrWestern ERME Center
Invoicing
Invoicing – General
– Submit invoices monthly– Email to [email protected]– Invoice Certification Statement
• Include on your invoice– Sample Subcontract Invoice:
https://s3.wp.wsu.edu/uploads/sites/2162/2017/03/2016-9.-Attach-6-Sample-Invoice.pdf
Invoicing
• Invoicing Certification Statement:
• Include on your statement
InvoicingInvoicing Procedures
• Cost Reimbursable Subcontract• Supporting documentation
• Accounting of salaries and wages• Copies of receipts etc. from vendors• Travel detail
**ERME 2019 Subaward Attachment 4 – Non University or University
Reporting & Invoicing PolicyInvoice payment Current and satisfactory reporting
Progress Notes Section of Online Report No Program Delivery in a quarter? Write a short
note in the Progress Notes Section of report. Submitting invoices in the first 3 months of your
project?a) Check off the project steps that have been
completed or;b) Write a brief description of the program delivery
that has occurred to date in the Progress Notes Section of the report.
Questions?
Reporting Dates and Timeline – (18 Month Period of Performance)
Online Reporting
Online quarterly reports (2019/2020 Timeline)
• July 1, 2019• October 1, 2019• January 2, 2020• April 1, 2020• July 1, 2020• Final Report is due October 15, 2020.
October 15, 2019 Proposed Results, Project Steps, Overview
Auto populated in the online report system
**Additional progress reports may be submitted or report info entered at any time.
Reporting Resources for the RVS System
• http://westrme.wsu.edu/award-management/reporting/education-projects/
• http://westrme.wsu.edu/award-management/reporting/exploratory-grants/
• http://extensionrme.org/Resources.aspx#RVS
Reporting Tutorialhttp://extensionrme.org/pubs/ReportingTutorial.PDF
Entering Your Report
Important Information
Reporting Portal
Instructions Window
Submitting A Progress Report
Reporting Status
Final Report
Thank you!Questions?
Jo Ann Warner [email protected] Neibergs [email protected] Schriver [email protected]