Open Platforms & Data Smarts: How We Can Do Good Better
63
Open Platforms & Data Smarts: How We Can Do Good Better NAWB Forum, Washington DC
-
Upload
kristin-wolff -
Category
Government & Nonprofit
-
view
277 -
download
0
Transcript of Open Platforms & Data Smarts: How We Can Do Good Better
Open Platforms & Data Smarts: How We Can Do Good Better NAWB Forum, Washington DC March 13, 1016
Our Practice We help public servants, philanthropists, and world-changers find what works in solving their communities' toughest challenges.
Our approach to research, evaluation and hands-on-technical assistance is rooted in a commitment to equity and focus on learning – so we all grow smarter by working together.
(And we love data, obviously…)
WHY DATA? WHY NOW?
States Doing Good Better with Quality Data
Jenna Leventoff, Policy Analyst March 13, 2016
• Advocate for inclusive, aligned and market-relevant education and workforce data that can help our nation’s human capital policies meet the challenges of a changing economy.
• Promote federal and state reforms for data systems that provide useful information for policymakers, students and workers, business leaders and educators.
• State Blueprint with 13 key features of a high-quality data infrastructure
• Address federal legislation, funding and technical assistance
• Policy agenda developed by broad coalition of national organizations, state leaders and technical experts across education/workforce spectrum
WDQC Mission
Apollo Education Group
Bill & Melinda Gates Foundation
Joyce FoundationLaura & John Arnold Foundation
Lumina Foundation
National Partners Funders
• New laws (like WIOA) clearly emphasize the importance of evidence based policy.
• Why? Evidence based policy = better policy!
• Evidence is also useful to help secure funding!
The Importance of Evidence Based Policymaking
• Our 13-point state blueprint outlines the elements of a high quality data infrastructure
• They require a data system to be: ⎻ Well Governed ⎻ Sufficiently Funded⎻ Inclusive⎻ Used for Accessible Analysis
WDQC State Blueprint
State Progress
• Officials from 47 states and the District of Columbia submitted responses.
• Blueprint survey results reveal net improvement on almost all elements.
• Cross Agency Councils: Councils including members from K-12, Labor, Higher Education and CTE
• 43 states • Labor Market Information: Improving labor market information data
collection, analysis, and distribution• 43 states
• Know if Graduates Get Jobs: Linking employment and earnings data to see if graduates of workforce and education programs get jobs
• 39 states• Cross State Data Sharing: Sharing employment data across states,
typically through participation in WRIS-2• 36 states
Elements Where States Achieved the Most Success
• Success Securing Sustainable State Funding• Oregon • Alaska • Kansas
• New Data Sharing Agreements• New Jersey
• States Newly Recommending the Need for Better Data• California• New York
Promising State Practices
Greg WeeksForecasting Division Office of Financial ManagementNational Association of Workforce Boards ForumWashington, DC March 2016
Evaluation & Your Education Data Warehouse: What’s a Workforce Board to Think?
&EDUCATION RESEARCH
DATA CENTER
Why listen to me?
• Long career in evaluation research and workforce economic analysis
• 12 years as Washington state LMI Director• Private sector consulting experience• Currently research economist at Washington Education and Data
Research Center (ERDC)• Currently conducting a series of studies on the returns to
education for the ERDC.• And… I’m entertaining!
Outline
1. Overview and brief discussion of non-experimental evaluation research.
2. Description of Propensity Score Matching (PSM)3. Impacts of PSM4. STEM Results (with and without PSM)5. Relevance for a Workforce Board?6. Discussion/Next steps.But first, a word from our sponsor…
Washington StateEducation Research & Data Center
16
• ERDC created in 2007 to:• Act as objective broker for education and workforce data• Assemble, link and analyze education and workforce data• Provide research focusing on student transitions• Make data available to the education agencies and institutions
• Located in Governor’s budget agency (Office of Financial Management)
• Work closely with State Education Agency (OSPI)• Working on second SLDS and WDQI grants
• Focus on research and reporting projects• Broadening subject areas to human services, corrections and
data visualization• Continue to operate the ERDC data warehouse
Rigorous evaluation studies matter for programs that enrich human capital
• Often required for US Department of Labor grant funded programs
• Help define evidence-based approaches that work• Best practice• Efficiency• Help target audiences• Often analytically challenging
The problem
With random assignment (such as clinical trials and experimental evaluation designs):
1. The outcome of the treatment is conditionally independent from the treatment.• Chosen at random, the treatment and control groups are statistically identical• The only difference is one group has the treatment, the other does not.
2. In observational non-experimental studies this assumption is invalid, resulting in “selection bias.”• The treatment group may have better outcome measures even in the
absence of a treatment.• Measured outcomes reflect both the differences in the groups and the
differences attributable to the treatment.
Selection bias
• Selection bias occurs when observable or unobservable factors influence both the decision to participate in the treatment and the outcomes.
• For example, our Bachelor’s degree study assumes that college graduates differ from high school graduates in ways that affect both the likelihood of attending and completing college, and post-graduation earnings.
• Simple (unadjusted) comparisons of earnings by educational attainment lead to selection-biased (over-stated) estimates of the earnings premium associated with a college degree.
Propensity score matching (PSM)
• Propensity score matching is utilized to develop a closely matched comparison group and correct selection bias.
• A propensity score is the estimated probability that an individual from the treatment or comparison group will participate in the treatment.
• This single measure indexes all the variables in the characteristics vector and provides a selection corrected comparison of the outcomes between the two groups.
• Estimated propensity scores allow individual treatment group members to be matched with and compared to individual comparison group members.
PSM- the counterfactual
• “PSM uses information from a pool of units that do not participate in the intervention to identify what would have happened to participating units in the absence of the intervention”
• Heinrich, C., Maffioli, A. and Vazquez, G. “A Primer for Applying Propensity Score Matching”. Office of Strategic Planning and Development Effectiveness. Inter-American development Bank. 2010. Retrieved from: http://publications.iadb.org/bitstream/handle/11319/1681/A%20Primer%20for%20Applying%20Propensity-Score%20Matching.pdf?sequence=1
Requirements for PSM
• Comparison group roughly equivalent in size to treatment group.• Applicants for the training or educational program not accepted into the
program.• SLDS educational data warehouse – may be able to provide an anonymized
comparison group from same high school classes, or by gender or age.• Clearly defined treatment(s) – start date, end date, time for
follow up in UI wage record (often a six month lag).• Pre-treatment descriptive data – the SLDS educational data
warehouse may be able to help with this.• Clearly defined outcomes/effects – often UI wage data.
Basic PSM Process
Once data is assembled for both treatment and comparison groups:1. Use logistic regression using pre-treatment variables to predict the
probability (propensity score) of participating in the treatment (using both groups together)
2. Match comparison group members to treatment group members based on this propensity score.There are several matching approaches including with or without replacement, nearest neighbor, weighted, …)
3. The difference in outcome measures of the treatment group and the matched comparison group is the measure of program net impact or effect.
There is a substantial literature on PSM. I would recommend starting with: http://publications.iadb.org/bitstream/handle/11319/1681/A%20Primer%20for%20Applying%20Propensity-Score%20Matching.pdf?sequence=1
Some examples from our ERDC research
• Returns to a Bachelor’s degree by gender: (http://www.erdc.wa.gov/sites/default/files/publications/201403_0.pdf)
• Returns to STEM degrees by gender and race categories: (http://www.erdc.wa.gov/sites/default/files/publications/EarningsPremiums-STEMBachelorDegrees.pdf )
• Returns to an associate degree by gender: (http://www.erdc.wa.gov/sites/default/files/publications/201501.pdf)
Female and male earnings trajectory, bachelor’s degree and high school only, PSM, 2012 dollars, follow up years 1-7.
1 2 3 4 5 6 7$0
$5,000
$10,000
$15,000
$20,000
$25,000
$30,000
$35,000
$40,000
$45,000
Female Bachelor's degree Female HS only Male Bachelor's degree Male HS only
Follow up years since HS graduation
Dollars
Female bachelor’s degree earnings differentials, with and without PSM; current dollars
1 2 3 4 5 6 7
-$10,000
-$8,000
-$6,000
-$4,000
-$2,000
$0
$2,000
$4,000
$6,000
$8,000
$10,000
Female_PSM
Female_no_PSM
Male bachelor’s degree earnings differentials, with and without PSM; current dollars
1 2 3 4 5 6 7
-$15,000
-$10,000
-$5,000
$0
$5,000
$10,000
MalePSM Male_noPSM
Female and male STEM earnings premium in current dollars for years before and after graduation (year 0), no PSM adjustment
-$7 -$6 -$5 -$4 -$3 -$2 -$1 $0 $1 $2 $3 $4 $5 $6 $7
-$5,000
$0
$5,000
$10,000
$15,000
$20,000
$25,000
Male Female
Female and male STEM earnings premium, 2013 dollars, years after HS graduation (year 0), with PSM adjustment
1 2 3 4 5 6 7 8
-$5,000
$0
$5,000
$10,000
$15,000
$20,000
Male STEM earnings premium Female STEM earnings premium
Proportion of STEM graduates in occupations by gender
Median overall wage rates for top occupations of STEM graduates by gender
Relevance for a workforce board?
• These studies are methodologically comparable to workforce evaluation studies.
• We are assessing a program that increases human capital and looking at the net effects (impacts) on subsequent earnings.
• This is very similar to an evaluation of a training or job search assistance program.
Takeaways• Rigorous evaluations of job skills training and job
search assistance programs are more often possible than sometimes assumed.
• Selection bias should be taken into account whenever possible (PSM)
• A rigorous PSM study implies rigorous data requirements (treatment and comparison groups)
• Use your SLDS education data warehouse as a source of data and as a partner.
WHY SHARING DATAWITH THE PUBLIC IS AWESOME
& HOW TO DO IT
World-Changers2.5Q (Bytes/Day)Data used to be
scarce. No more.
Abundant DataThe Cloud. Software as service. Mobile. Enterprise grade
systems in the palm of your hand, for you
and Sergey Brin.
Accessible Tools
Why #CivicTech?(#OpenGov #OpenData #Gov20 #DigitalGov)
Millennials. Encore careerists. People
who want to be more than “customers.” They want to be
citizens.
The activities of a company associated with buying and selling a product.
It includes advertising, selling and delivering products to people. People who work in marketing departments try to get the attention
of target.
This is Way Hard*(*Hat-tip to @MattBailey)
Intelligence out
Datain
Legacy systems
“Soft” funding
No capacity (people)
Silo’d programs, d-bases,
governance
No interagenc
y agreement
Fear of results
No noisy constituenc
y
Data manageme
nt vs. research culture
Who→ People who are interested in data
and technology→ Hobbyists and professionals→ Students and retirees (and in-
between)→ Public and private sector
champions, sponsors, partners→ Community-based cheerleaders
What→ Community-powered nonprofit
organization building civic data projects that help all kinds of people do more public good
→ Runs Hack University → Located in Portland, OR→ Works statewide
About this word “hack…”
→There’s no deep, dark, or otherwise dangerous web involved.
→“It’s good, heartwarming and nifty.”
→It’s about citizens collaborating with government to extend capacity, solve problems, and improve our communities for everyone.
→It’s about learning how to adapt to the new world of abundant data together.
Here’s a hack…
Here are another 12 . . .
Caus
e &
(Ra
ise)
Eff
ect?
40/4
0/20
?
12 F
ilter
s
12
Filt
ers
10 Lessons: 1. The smartest person in the room is the room. (Hat-tip David Wenberger)2. Regular people can work miracles when they care about what they are working on. 3. Data literacy is a big deal.4. Experimentation/prototyping is also a big deal.5. Pictures are better than words.6. Sharing data begets questions, but they are much better questions.7. There’s no “done.” 8. Data is not evidence, but you need it to build evidence, which helps
you work smarter.9. Evidence-based policy making requires a cultural shift.10. See #1.
Government is better when we all work together.
Data Genius ( Kristin gave him this title) [email protected]
Greg Weeks, Ph.D.Policy [email protected], ext. 114
Jenna Leventoff
Director, Technical Assistance &Training [email protected]
Vinz Koller Adjunct, Senior [email protected]
Kristin Wolff
