Slide 1

2012 UI Integrity Summit March 13, 2012 Slide 2 2012 UI Integrity Summit Today's Panel Moderator: Brad Wiggins Integrity Project Coordinator, Office of Unemployment Insurance U.S. Department of Labor, Employment and Training Administration Panelist: Andrew Clarkwest Senior Researcher Mathematica Policy Research Slide 3 2012 UI Integrity Summit Our Agenda Review of DOLs Work Search Strategy Analysis of State UI Work Search Policies Mathematica Policy Research Group Discussion 3 Slide 4 2012 UI Integrity Summit UI Improper Payment Rate 4 Slide 5 2012 UI Integrity Summit Root Causes of UI Overpayments 5 Slide 6 2012 UI Integrity Summit 6 UI Integrity Strategic Plan DOL has been focused on prevention, detection and recovery in three major root cause areas: Claimants continuing to claim after returning to work (Benefit Year Earnings (BYE) Untimely/insufficient separation information from employers and Third Party Administrators Employment Service (ES) registration Focus in 2012 on New Strategies to Address Work Search Errors Slide 7 2012 UI Integrity Summit 7 Work Search Strategy Research: Analysis of State UI Work Search Laws, Policies, and Practices Pilot: Development of a Portable Web-based Tool to Capture, Organize and Share UI Claimant Work Search Records Working Group: Small Group of States (12) to Review Research and Frame National Strategies to Reduce Work Search Errors Slide 8 2012 UI Integrity Summit Analysis of State UI Work Search Policies Andrew Clarkwest Senior Researcher Mathematica Policy Research 8 Slide 9 2012 UI Integrity Summit 9 Study Background Slide 10 2012 UI Integrity Summit Violation of design and mission of UI program Temporary support for those to those with involuntary job loss Assumes claimants are trying to obtain new employment Fiscally costly Why Focus on Work Search Error? 10 Slide 11 2012 UI Integrity Summit Legal IPERA: Goal to reduce improper payments by $50 billion across federal programs UI identified as a program with unacceptably high rates of improper payments Political: Refocus UI to emphasize job search Why Focus on Work Search Error? 11 Slide 12 2012 UI Integrity Summit 1. How many benefits are being paid to claimants who do hold up their end of the bargain by actively searching for work? 2. How can that number be reduced? Two Fundamental Questions Two Fundamental Questions 12 Slide 13 2012 UI Integrity Summit Which state policies and practices are correlated with higher or lower rates of work search improper payments (IPs)? How can measured rates of work search IPs be improved to more accurately reflect actual rates of non-compliance? Research Questions 13 Slide 14 2012 UI Integrity Summit 14 Data Sources Slide 15 2012 UI Integrity Summit Quantitative dataset characterizing state policies on: Work search requirements Recording and reporting requirements Verification and enforcement State Policies and Practices Dataset 15 Slide 16 2012 UI Integrity Summit Work search error measures State practices observed in data Frequent ERP interviews Most common filing method Benefit certification frequency Work search exemption frequency Labor exchange registration requirement Staff job search assistance Use of warnings BAM Data 16 Slide 17 2012 UI Integrity Summit In-depth primary data collection from a subset of states Phone (7 of 9 responded): FL, IL, ME, NY, OH, TX, WI Selected for geographic variation, large dollar amount of work search improper payments Also a subset with very low work search error rates Written responses by OUI site visit staff (4 of 6 responded): AZ, CO, LA, VA Improper Payment High Priority States for fiscal year 2012 Work search a major cause of IPs in each state In-Depth Primary Data Collection 17 Slide 18 2012 UI Integrity Summit 18 Variation in State Policies Slide 19 2012 UI Integrity Summit Work Search Requirements 19 Required # of ActivitiesCount of Jurisdictions None specified16 1-223 3-412 5+1 Source:SPPD Slide 20 2012 UI Integrity Summit Documentation: Many states require claimants to keep a log of activities and provide it on request Reporting: Few states require claimants to regularly submit details of the log Required reporting of contacts appears to have decreased with technological advances Verification: Some states perform random audits of work search (other than BAM audits), but proportion of claimants audited tends to be very low Documentation and Reporting 20 Slide 21 2012 UI Integrity Summit 21 Variation in Work Search Error Rates Slide 22 2012 UI Integrity Summit Work Search Error Rates Across States 22 Slide 23 2012 UI Integrity Summit What factors explain variation in work search error rates? Challenge: Measured work search error rates may not be comparable across states How much of the observed variation in work search error rates reflects actual variation in work search compliance? Variation in Work Search Error Rate 23 Slide 24 2012 UI Integrity Summit Measuring Work Search Error Rates 24 Rate of Work Search Non- Compliance Claimant work search activities State work search requirements Slide 25 2012 UI Integrity Summit Measuring Work Search Error Rates 25 Measured Work Search Error Rate in BAM Claimant work search activities State work search requirements Leniency toward failure to meet requirements BAM implementation practices Consistency of state policies and BAM audits Slide 26 2012 UI Integrity Summit We only observe measured error rates BAM is the most comprehensive and reliable data source on error rates BAM data provide clues on differences between measured error rates and rates of claimant non- compliance Finding: much of the variation in measured error rates appears to be unrelated to actual rates Subsequent analyses must be interpreted with caution Issues in Measurement of Work Search Error Rates 26 Slide 27 2012 UI Integrity Summit Differences in treatment of nonresponse In 5 states, more than half of nonrespondents have work search error In 39 states, no nonrespondents have work search error Differences in formal warning rules 15 states determined that most work search errors were technically proper due to formal warning rule Issues in Measurement of Work Search Error Rates 27 Slide 28 2012 UI Integrity Summit Differences in exemptions from work search requirement Across states, 1 to 58 percent of claimants are exempt from work search requirement Conversations with states provide evidence of BAM practices that are at times inconsistent with state policies Independent assessment of claimant status in states with numeric work search requirement suggests higher error rates than reported Issues in Measurement of Work Search Error Rates 28 Slide 29 2012 UI Integrity Summit Reported vs. Independently Assessed Error Rates 29 Slide 30 2012 UI Integrity Summit 30 Analysis of Predictors of Work Search Errors Slide 31 2012 UI Integrity Summit Goal: Identify policies and practices that are associated with higher or lower work search error rates Methods: Compare error rates across states with different stringency of work search rules Conduct correlational analysis to identify significant predictors of work search error Overview of Analysis 31 Slide 32 2012 UI Integrity Summit Do states with more stringent policies have higher or lower work search error? Our approach: Construct a stringency measure based on state policies and practices Compare work search error rates across categories Findings: No significant differences across categories Error Rates by Stringency of Policies 32 Slide 33 2012 UI Integrity Summit Stringency of work search policies LowMediumHigh Number of states in each category182113 Mean work search error rate (percentage points) [standard deviation] 1.8 [2.5] 2.3 [3.0] 5.0 [9.1] Proportion of states with error rate less than 0.5 percent 0.500.520.54 Error Rates by Stringency of Policies 33 Note: Differences in work search error rates across categories are not statistically significant. Slide 34 2012 UI Integrity Summit Are specific policies and practices associated with high work search error rates? Our approach: Examine correlations between each policy and work search error rates Use a regression analysis to determine which policies were the best predictors of error rates Findings: A small number of policies predict work search error Results depend on the outcome considered Associations between policies and error rates do not prove a causal effect Predictors of Work Search Error 34 Slide 35 2012 UI Integrity Summit We considered several error rate outcomes: 1.Reported error rate 2.Reported error rate, excluding states with very low rates (less than 0.5 percent) 3.Indicator for rate less than 0.5 percent 4.Error rate treating formal warnings as errors 5.Assessment of error rate based on numeric work search requirement Work Search Error Outcomes 35 Slide 36 2012 UI Integrity Summit Policies Associated with Work Search Error 36 Policy (1) Primary (2) Exclude low error (3) Low error indicator (4) Include warnings Search must be customary for occupation * Exceptions for subsets of claimants for number of contacts * Claimants must log names of individuals contacted * Claimants can submit documents via the Internet ** Warning rule * At least 10 percent of claimants had ERP interview * * Denotes that the policy has a statistically significant correlation with the error rate or other outcome. Slide 37 2012 UI Integrity Summit States allowing claimants to submit work search information via the Internet have higher work search error rates When excluding states with very low error, states requiring claimants to log names of individuals contacted have higher error rates Possible that more stringent reporting requirement leads to lower burden of proof for SWA in detecting errors Predictors of Work Search Error 37 Slide 38 2012 UI Integrity Summit States requiring work search customary for occupation or having exemptions from requirement for some claimants are less likely to have very low work search error. When counting formal warnings as errors, states with formal warning policies have higher work search error rates. One explanation is that formal warning policies may reduce incentive to satisfy requirements. Predictors of Work Search Error 38 Slide 39 2012 UI Integrity Summit 39 Recommendations for Improving Data Consistency Slide 40 2012 UI Integrity Summit Understanding causes of work search error is difficult if data are not complete and consistent How can data be improved in order to better target work search error? Ensure that BAM audits apply state policies correctly Consider policies that promote more complete data Improving Data Consistency: Motivation 40 Slide 41 2012 UI Integrity Summit BAM data are sometimes inconsistent with state policy Some states requiring claimants to submit work search logs grant clemency to BAM nonrespondents Lack of employer contacts sometimes contrasts with determination of work search error Develop BAM procedures based on state policy If state policy requires work search log, code nonrespondents as having work search error Check employer contacts against state requirement Consistency of BAM Audits 41 Slide 42 2012 UI Integrity Summit Updating work search policies may enable better quality data collection Better data lowers the SWAs burden in verifying work search requirements Suggested policies: Require claimants to keep a log Require claimants to provide work search information as part of continuing claims process Clarify rules on activities that may substitute for employer contacts Eliminate formal warning rules Policies Promoting Higher Quality Data 42 Slide 43 2012 UI Integrity Summit 43 Work Search Compliance Recommendations: Ideas from Conversations with State Workforce Agencie Work Search Compliance Recommendations: Ideas from Conversations with State Workforce Agencies Slide 44 2012 UI Integrity Summit Enhance work search support through increased collaboration with the local One-Stop Career Centers Encourage or require more job search assistance Worker Profiling and Reemployment Services program is an example Collaborate on verification of work search activities Recommendations 44 Slide 45 2012 UI Integrity Summit Recommendations 45 Augment verification processes Require claimants to submit log of work search activities Implement a system that automatically checks completeness of reporting Provide increased funding for audits Impose harsher penalties for failure to meet work search requirements Disqualify claimants after a work search error Slide 46 2012 UI Integrity Summit Group Discussion Questions? 46 Slide 47 2012 UI Integrity Summit Group Discussion Identified challenge areas: BAM Training Automated work search verification BAM methodology Work search messaging Other challenges? Promising practices? 47 Slide 48 2012 UI Integrity Summit