FHWA Civil Rights: Data Collection and Program Evaluation …why should I care?
-
Upload
kory-webster -
Category
Documents
-
view
215 -
download
1
Transcript of FHWA Civil Rights: Data Collection and Program Evaluation …why should I care?
42510011 0010 1010 1101 0001 0100 1011
FHWA Civil Rights: Data Collection and Program
Evaluation
…why should I care?
4251
0011 0010 1010 1101 0001 0100 1011
Data Collection and Program Evaluation (Analysis)
…can be confusing
4251
0011 0010 1010 1101 0001 0100 1011
Objectives
• Identify the importance of data collection and program evaluation.
• Explain the basic structure of an evaluation plan or strategy.
• Design a specific program evaluation plan to– Identify specific data that must be collected,– Assess the implementation and effectiveness of
various civil rights programs, and– Ensure State DOT management is aware of
issues, needs and potential problems associated with various civil rights programs.
4251
0011 0010 1010 1101 0001 0100 1011
Why should I care?
• Reliable information about the impact of FHWA programs (program evaluation) is essential to ensuring non-discrimination.
• The relationship between FHWA programs and varying social factors have wide-ranging implications for any community, municipality, county, or state.
4251
0011 0010 1010 1101 0001 0100 1011
Why is it important?
• Data collection and program evaluation (analysis) can:– Help identify and prioritize problem areas,– Initiate and evaluate the effectiveness of
policies and programs to ensure non-discrimination,
– Assess the relationship between specific programs and beneficiaries to develop non-discrimination strategies.
4251
0011 0010 1010 1101 0001 0100 1011
Importance(continued)
– Draw agency attention and resources to resolve issues and problems.
– Provide justification for certain programs or illustrate a need for new programs.
– Help justify continued or additional funding.– Communicate the importance of the goals,
solutions, strategies, or programs.
4251
0011 0010 1010 1101 0001 0100 1011
“Program evaluation is like a bunch of feathers because it lifts you up
to get a bird’s eye view so you can see the whole picture” (unknown).
4251
0011 0010 1010 1101 0001 0100 1011
Current FHWA Data Collection
• How do you collect statistical data?• What do you do with it?• Project-based or Program-based?• Benefits vs. Burdens analysis?• “Letter of the law” or “Spirit of the law”?
Each State DOT must “develop procedures for the collection of statistical data (race, color, religion, sex, and national origin) of participants in, and beneficiaries of State highway programs, i.e., relocatees, impacted citizens and affected communities.” (23 CFR 200.9(b)(4))
4251
0011 0010 1010 1101 0001 0100 1011
Steps to look at the whole picture
• Step 1: Focus the Evaluation
• Step 2: Identify Barriers to Evaluation
• Step 3: Methods of Data Collection
• Step 4: Organizing, Analyzing, Interpreting, Summarizing and Reporting Results
• Step 5: Develop the Evaluation Plan
4251
0011 0010 1010 1101 0001 0100 1011
Step 1: Focus the Evaluation
• What will be evaluated?
• What is the purpose for this evaluation?
• Who will be affected by or involved in the evaluation?
• What are the critical questions the evaluation must address?
4251
0011 0010 1010 1101 0001 0100 1011
Step 1: Focus Example(Right-of-Way)
• What will be evaluated?– Real estate appraisals (project-based or program-based)
• What is the purpose of the evaluation?– To ensure real estate appraisals are equitable without
regard to race, color, or national origin.
• Who will be affected by or involved in the evaluation?– Certified Appraiser - Relocation Agents– Review Appraisers - Property Management– State DOT Appraisers - State DOT Leadership– Acquisition Agents - FHWA Division/Headquarters
4251
0011 0010 1010 1101 0001 0100 1011
• What are the critical questions the evaluation must address?– Was the resulting appraisal amount fair and
equitable?– Are the land unit values consistent throughout the
project area?– Were relocatees required to move to a similar
socio-demographic region?– Were relocatees provided relocation advisory
services to find replacement housing (i.e., transportation to/from available housing).
Step 1: Focus Example(Right-of-Way)
4251
0011 0010 1010 1101 0001 0100 1011
Step 2: Identify Barriers
• Factors that may influence the evaluation
• Implications for the evaluation• Strategies to overcome barriers
Evaluation is like an acorn because it must overcome adverse conditions to
grow into a tree” (unknown).
4251
0011 0010 1010 1101 0001 0100 1011
Step 2: Identify Barriers
• Organizational Politics:– Is there support for the evaluation?– Are there opponents to the evaluation?
• Project Leadership:– Who has control over the program?– How does the evaluation fit their goals?
• Professional Influences:– Are professional groups interested?– Are the supportive of an evaluation?
4251
0011 0010 1010 1101 0001 0100 1011
Step 2: Identify Barriers
• History:– What has been the tradition or history of
self-evaluation of programs?– Have evaluations been conducted before?
• Organizational Setting:– Does the program fit into a larger
organizational network? Where or how?– What kind of information could jeopardize
the program?
4251
0011 0010 1010 1101 0001 0100 1011
Step 2: Identify Barriers
• Economics:– Is fiscal support for the program and
evaluation secure?
• Interpersonal Patterns:– How much interagency or interpersonal
conflict is likely to occur as a result?– Could the evaluation be controversial to
internal or external audiences?
4251
0011 0010 1010 1101 0001 0100 1011
Step 2: Identify Barriers
• Legal Guidelines:– Will legal restrictions limit collection of
desired information/data?– Will the project be affected by pending
legislation?
• Resources:– Are resources (human, financial, time, etc.)
available to support the evaluation?
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Data Collection Methods
• What type of data should be collected?
• What methods should be used to collect data?
• How much information should you collect?
• Questionnaires
• Validity and Reliability
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Types of Data
• Descriptive information can include:– Characteristics of the program.– Demographic information on program
participants, beneficiaries, etc.– Actual benefits paid or realized by
beneficiaries.– Specific results of a program or service.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Types of Data
• Judgmental information can include:– Opinions from consultants.– Beneficiaries beliefs and values.– Agency personnel’s interpretation of laws.– Stakeholders perceived priorities.– Interpretations of policy, procedures and
guidelines.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Data Collection Methods
• Opinion Surveys: as assessment of how a person or group feels about a particular issue.
• Questionnaire: a group of questions that people respond to verbally or in writing.
• Case Studies: experiences and characteristics of selected persons involved with a program.
• Individual interviews: individual’s responses, opinions, or views.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Data Collection Methods
• Group Interviews: small groups’ responses, opinions, and views.
• Records: information from records, files, or receipts.
• Advisory, Advocate Teams: ideas and viewpoints of selected persons.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Choosing a Method
• Availability: information may already be available to you that can help answer some questions. Review information in prior records, reports and summaries.
• Interruption Potential: the more disruptive an evaluation is to the routine of the program, the more likely that it will be unreliable.
• Protocol Needs: you may need to obtain appropriate permission to collect information.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Choosing a Method
• Reactivity: you do not want “how” you ask something to alter the response you get.
• Bias: to be prejudiced in opinion or judgment.• Reliability: will people interpret your questions
the same way each time?• Validity: Will the collection methods produce
information that actually measures what you intend to be measured?
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: How Much Data?
• Sampling: a portion of subjects in order to learn something about the entire population without having to measure the whole group.– Random: each individual in the group has
an equal chance of being chosen for the sample.
– Purposive: a sample that will represent specific viewpoints.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: When Choosing a Method, Consider…
• Should you use a sample of a population or a census (an entire population, such as all people living in a project area)?
• Should you use a random or purposive sample?
• How large a sample size do you need?• Is your sample likely to be biased?
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Questionnaires
• Questionnaires are a widely used method of collecting information. They can be a cost effective way to reach a large number of people or a geographically diverse group.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Questionnaires
• Questionnaires should include the following key elements:– Cover letter: include the purpose of the
study, why and how participants were selected, etc.
– Introduction: a short recap of the information included in the cover letter.
– Instructions: clear and concise.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Questionnaires
• Questionnaires should include the following key elements:– Grouping questions: group questions with similar
topics together in a logical flow.– Demographic questions: age, gender, ethnicity,
etc.– Thank you: thank the respondent for completing
the questionnaire.– Disposition: what to do with the questionnaire
when complete.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Validity
• “Does the questionnaire measure what you want it to measure?” A questionnaire that is valid for a specific situation or audience may not be valid in a different situation or for a different audience.
• Have a panel of experts (professionals familiar with the program) review the questionnaire and provide feedback.
• Test the questionnaire on a similar population.
4251
0011 0010 1010 1101 0001 0100 1011
Step 3: Reliability
• Does the questionnaire consistently give the same results with the same group of people under the same condition?
• Can be shown when using a pilot sample with a questionnaire administered repeatedly over time.
• Are answers repeated over time?• Revise as needed.
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Organizing, Analyzing, Interpreting, Summarizing and Reporting
• Organize Evaluation Data
• Analyze
• Quantitative Data Analysis
• Qualitative Data Analysis
• Interpret (summarize)
• Report Results
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Organize Data
• Develop a system to organize your data (spreadsheet).
• As data are received, check to be sure it is complete.
• Set up protocol for who does/does not have access to the data.
• Track all data. Ensure data are not lost or overlooked as analysis and summarizing begin.
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Quantitative Data
• Quantitative Data Analysis: information that contains numerical data.– N: The total number of participants in the sample.– Mean: The average score of the sample.– Median: The score halfway between the high and
low score.– Mode: The response given the most.– Standard Deviation: The range from the mean in
which 68% of the responses are found.– Frequency: How often a particular response was
given.
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Quantitative Analysis Example
• On a scale of 1 to 5, where 1=poor and 5=excellent, how would you rate the overall quality of the workshop?– Answers (10 people): 4,5,2,4,3,4,3,3,5,4– Mean: 3.7 (the total divided by 10)– Median: 3.5 (halfway between the lowest score of
2 and the highest score of 5)– Mode: 4 (the score repeated the most)– Standard Deviation: 0.95 (68% of the answers
were between 2.75 and 4.65).– Frequency: 1=0, 2=1, 3=3, 4=4, 5=2.
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Qualitative Data
• Qualitative data analysis is the process of systematically searching and arranging the interview transcripts, field notes, and other materials to increase your own understanding and enable you to present what you have discovered. (Bodgan and Biklen(1992)).
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Qualitative Data
• Data resulting from interviews, focus groups, open-ended questions, or case studies.
• Develop common elements, themes, topics, sentiments, etc. from the qualitative data.
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Interpret
• Identify trends, commonalities and testimony that will help answer the critical evaluation questions that were generated in Step 1.
• To be useful, the data must be interpreted so that the stakeholders will understand the results and know how to use them.
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Interpret Example
• At the end of a public meeting to discuss a future construction project, persons were asked to rate their satisfaction with key components of the project on a scale of 1 to 5.
• 1=not satisfied and 5=very satisfied.
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Interpret Example
• A total of 75 people responded.– 40 Hispanic/Latino– 34 White (non-
Hispanic)– 29 African-American– 15 Disabled
Survey Responses (mean)
Project Location 3.15
Projected Cost 4.21
Length of Construction
2.80
Project Schedule 2.55
Detour Route 3.43
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Interpret Example
White Respondents (mean)
Project Location 4.85
Projected Cost 4.10
Length of Construction 2.90
Project Schedule 1.85
Detour Route 1.44
Hispanic Respondents (mean)
Project Location 2.61
Projected Cost 4.48
Length of Construction 3.12
Project Schedule 2.98
Detour Route 3.85
African-American Respondents (mean)
Project Location 1.98
Projected Cost 3.99
Length of Construction 2.49
Project Schedule 1.97
Detour Route 4.10
Disabled Respondents (mean)
Project Location 3.16
Projected Cost 4.25
Length of Construction 2.68
Project Schedule 3.41
Detour Route 4.34
What does it all mean?
4251
0011 0010 1010 1101 0001 0100 1011
Step 4: Reporting
• Serves as a basis for further program development and improvement.
• Provides support for continuing or expanding the program.
• Serves as a basis for public relations and promoting future programs.
• Identifies implications of the results, how they can be used, and recommendations for the overall program.
4251
0011 0010 1010 1101 0001 0100 1011
Step 5: Evaluation Plan
• Who should lead the evaluation?
• How should tasks and responsibilities be formalized, organized and scheduled?
• What problems are expected?
4251
0011 0010 1010 1101 0001 0100 1011
Step 5: Who Should Lead?
• One person or a team?• Leader’s responsibilities may include:
– Designing the evaluation– Constructing the instrument– Collecting data– Analyzing data– Writing/delivering reports– Managing and interacting with personnel
4251
0011 0010 1010 1101 0001 0100 1011
Step 5: Organizing Tasks
• When a project team, evaluation leader, and the stakeholders meet and ask “What must be done, when and by whom?” the evaluation management plan begins to take shape. These tasks should be put into some logical organizational structure.
4251
0011 0010 1010 1101 0001 0100 1011
Step 5: Organizing Tasks
• A management plan should include:– Evaluation design and a general plan specifying
what must be done.– When each activity needs to be conducted and
completed.– Responsible parties for each activity.– Available resources.– Ability to revise or refine the plan as time passes.– Contingency plans for problems that may arise.
4251
0011 0010 1010 1101 0001 0100 1011
Step 5: Expected Problems
• Murphy’s Law
• Monitor the evaluation’s progress– The earlier you detect a problem, the better
able you will be to resolve and continue.
• Develop a contingency plan.
4251
0011 0010 1010 1101 0001 0100 1011
Conclusion
• Program Evaluation including data collection and analysis can provide an overall focus to the civil rights programs.
• Requires a paradigm shift from daily implementation to program efficiency.
• An effective evaluation plan will help identify the varying types of data that need to be collected in the programs.
4251
0011 0010 1010 1101 0001 0100 1011
Thank you
• Questions?
• Contact information:Darren KaihlanenFHWA Oklahoma [email protected](405) 254-3312 (voice)(405) 254-3302 (fax)