Using longitudinal administrative data to evaluate area-based initiatives

Using longitudinal administrative data to evaluate area-based

initiatives

George Smith & David McLennan

Outline

Background Brief history Example 1: Evaluating the Kent Supporting

Independence Programme (SIP) Example 2: Evaluating the national Neighbourhood

Nurseries Initiative (NNI) Example 3: Evaluating the New Deal for

Communities (NDC) Future developments? (Not covering ethical, legal, contractual and access

issues)

Background: Using Administrative Data for Social Research

Administrative data: data primarily collected for administrative purposes, converted for use as a research tool

Such data traditionally released in an aggregated format but increasingly available in individual format

Increasingly linkable laterally (across different datasets) and over time (but still a very long way short of what is possible in Denmark and Sweden).

Key strengths: potentially virtual universal coverage of many of these datasets, and regular (sometimes continuous) updating; also may be virtually the only way to study some social phenomena

Weaknesses: may only contain data that is needed for administrative purposes and there may be ad hoc changes in criteria or definitions (through policy or admin changes).

Administrative data and survey data: very powerful combination?

Brief history – where are we coming from?

Phase 1: late 1980s: impact of national 1988 welfare benefit changes assessed by using an individual claimant extract from an LA housing benefit (HB) system taken before the reforms to estimate the post 1988 reform ‘gains and losses’.

Phase 2: mid 1990s: local longitudinal studies (e.g. Lone Mothers Moving On and Off Benefit), using linked HB extracts to plot benefit dynamics. Such data increasingly well postcoded (and linkable through LA specific ‘HBref’).

Phase 3: 1999 onwards: access by SDRC to national data systems at individual level (e.g. DSS/DWP) for the new Index of Multiple Deprivation (ID2000), led on to longitudinal linkage via encrypted NINOs e.g. for the SEU (e.g. Changing Fortunes , 2001, Growing Together or Growing Apart? 2002, which drew on individually linked welfare benefit data over 5 years, to give both national and local trends).

Phase 4: 2002 onwards: using such linked datasets to assess local programmes. Key strengths of such data are that it is already collected in a consistent form nationally. Is typically more than 99% postcoded and can be linked over time.

IMD work gave access to a very wide range of individual level admin data (welfare benefits, employment, education, health, crime). Strong focus on local areas but within a setting where we needed consistent information across all areas (a prime requirement for the IMD).

Previous experience with attempts to evaluate local area based programmes suggests that unless programmes are highly focussed and targeted, any effects are likely to be very small.

Example 1: Evaluating the Kent ‘Supporting Independence Programme’ (SIP)

Study commissioned by HM Treasury 2002-5 (and follow up to 2006/7) to evaluate impact of Kent’s SIP programme as part of the local Public Service Agreement (LPSA) - Target 12 ‘to reduce dependency and increase employment .. across the whole of Kent with an initial focus on particular districts’.

SIP was essentially an umbrella programme of local initiatives more or less targeted at Kent’s ‘priority wards’ (initially in East Kent) and later expanded to 33 wards in all 12 districts. These had a claimant population of about 18,000 in 2001.

The SIP programme had already begun by the time the evaluation was commissioned; it was a series of loosely targeted initiatives, in some cases with new facilities e.g. increased childcare; in other cases with advice centres or enhanced coordination between local and national agencies.

Some of the key outcomes could be directly measured using national welfare benefits data. So an opportunity to test the use of admin data in a local evaluation. Was it possible?

The Kent LPSA Evaluation

Stage 1: Baseline of ‘benefit spend’, based on data extract for April 2001, for all wards and districts in Kent. Numbers of cases and actual ‘spend’.

Stage 2: Repeated cross-sectional extracts at yearly intervals of major national welfare benefits (IS, JSA. IB/ SDA), allowing trends over time for the target and other areas.

Stage 3: Monitoring individual change in claiming patterns over time, and tracking geographical mobility on benefit.Key to uncovering how far area change is affected by

movement into or out of benefit by claimants. Also effects of geographical mobility on claiming patterns

Stage 4: Comparing the pattern of change by claimants in the Kent priority wards, with the pattern of change in the rest of the South East of England over the same periodModelling the pattern of exits for a cohort followed from 2001-2005 on IS, JSA, IB and SDA.

Some interim results

Limited number of explanatory variables in the admin data, or available from other sources at the local area level.

No clear cut overall effects, though possible improvement in 2003 as claimants in the most highly targeted wards had exit rates in line with other areas in the SE; also slightly higher than expected exit rates for some groups in these areas e.g. younger lone parents with one child under five.

Admin data comes up with numbers large enough to identify such groups, even at a local level. But problem of attributing this apparent change to the programme, without more evidence of a direct link. Issue may not be simply ‘does it work? But for whom, when and where?’

Further two years of data analysed to explore these patterns further. Report in preparation.

Evaluating the Neighbourhood Nursery Initiative (NNI)

NNI – national programme to increase childcare in the most disadvantaged areas (approximately 45,000 places in about 1400 centres by 2005)

Evaluation ( by groups at Oxford, Nat/Cen and IFS). covered implementation, quality of provision, neighbourhood change, and the impact of NNI provision on entry to employment,. Study made extensive use of admin data, including longitudinal DWP benefit data, WPLS and DfES PLASC data.

NNI did not serve tightly specified areas. So necessary to identify potential target areas.

NNI Rich and NNI Poor Areas

Identified all census output areas (OAs) in the 20% most disadvantaged Super Output Areas (SOAs) using the IDAC measure (itself generated from admin data).

Using individual level DfES PLASC data, analysed the distance travelled to primary school by children aged 5-7 in each area to establish the accessibility criterion for an NNI nursery.

Used this information to identify disadvantaged areas with reasonable access to NNI provision and places (‘NNI rich’) and those without (‘NNI poor’).

Propensity score matching techniques used to identify two matched groups of families with preschool children drawn from the child benefit system resident in NNI rich and NNI poor areas. Samples invited to take part in a screening survey to assess their intentions to (re)enter the labour market and make use of childcare.

Subsequent impact study (Nat/Cen and IFS) drew on the ‘work ready’ subset but was able to compare results from four different sources; those actually using NNI provision; their own self assessment of impact; ‘work ready’ families in NNI rich and poor areas, and the overall population of families with young children in these two areas. This included drawing on WPLS to check on the subsequent entry to labour market outcomes. The advantage is that the study includes both the effects on users and at the community level. One key point that emerged that even in the most highly targeted areas (NNI rich) the take up of NNI childcare was around 10% at best. So any effect on ‘users’ is significantly reduced when looking ‘community wide’. Report to be published shortly.

Tackling worklessness in NDC areas

Worklessness

Worklessness:“involuntary exclusion from the labour market and receipt of selected out-of-work benefits”

UnemploymentJob Seekers Allowance (JSA)

Work-limiting illnessIncapacity Benefit (IB)Severe Disablement Allowance (SDA)

Work and Pensions Longitudinal Study (DWP)

Tackling worklessness

Has the NDC programme reduced worklessness in the 39 partnership areas?

(1) Area level change:Reduction in area-level worklessness rates?

(2) Individual level change:Increased likelihood of NDC residents moving

out of worklessness and into employment?

Cross-Sectional Worklessness

24 of the 39 partnerships saw a reduction in the absolute percentage point ‘gap’ with local authority worklessness rate between 1999 and 2005

31 out of 39 saw a reduction in absolute gap with local authority on unemployment rate

4 out of 39 saw a reduction in absolute gap with local authority on work-limiting illness rate

So are 31 NDCs successfully tackling unemployment?

Unemployment example

Change in unemployment rate could be due to…

Improvement for area?

Improvement for unemployed people living in

area?

Unemployed people in the NDC area moving into jobs

Unemployed people in the NDC becoming unable to work due to illness

New people moving into the NDC area who are not unemployed

Unemployed people moving out of the area but remaining unemployed





area?


Yes








area?


Yes Yes








area?


Yes Yes


No







area?


Yes Yes


No No







area?


Yes Yes


No No


Yes






area?


Yes Yes


No No


Yes No






area?


Yes Yes


No No


Yes No


Yes





area?


Yes Yes


No No


Yes No


Yes No

Longitudinal analyses

Work and Pensions Longitudinal Study enables individuals to be tracked over time:

into, out of and between benefits and employment

geographically when claiming benefits

Enables analysis of the individual level dynamics driving area level changes

NDC Worklessness ‘inflows’ and ‘outflows’

Programme impact?

Has the NDC programme increased the likelihood of people leaving workless? And entering employment?

Need to track people over time and control for individual, household and area level factors to ‘isolate’ any programme effect

Propensity score matching and Difference-in-Difference modelling using WPLS

Previous results

NDC Evaluation Phase 1 results (June 2005):

Unemployed NDC residents were 1.1 times more likely to exit workless benefits compared to control group

NDC residents unable to work due to limiting illness were 1.6 times more likely to exit workless benefits compared to control group

But… Used GMS-ONE Unable to track people into employment Control group matched across whole of England

Current developments

NDC Evaluation Phase 2 analysis:

Using WPLSTracking people into employmentControl group matched to people in similarly deprived local

neighbourhoods

Results forthcoming: spring/summer 2007

Next Steps / Future Developments

Next Steps: WPLS

1. Add evaluations: E.g. Neighbourhood Renewal Fund

NRF areas

Non-NRF areas

Future possibilities: WPLS

2. Compare (people within) programmes:

NRF areas

EmploymentZones

NDC areas

Possible outcome measures include:

Likelihood of:1. Exiting to employment2. Exiting to sustained employment3. Returning to workless benefits

Future possibilities : WPLS

3. Find cumulative impact of people and place programmes

EmploymentZones NRF areas

New Deal25+

Future possibilities: WPLS

4. Find cumulative impact of combinations people and place programmes

No programme

People based programme Place based

programmeBoth

programmes

Future possibilities:Multi-Source Data Linkage

5. Assessing the impact of change in one outcome measure on change in another- Linking data on the same person

Access to Higher Education (UCAS)

Pupil Level Annual School Census &National Pupil Database (DfES)

Post 16 Qualifications, such as ILR, SERAP, NVQD (DfES)


6. Assessing the impact of change in one outcome measure on change in another- Linking data on the same person

Work and Pensions Longitudinal Study

Health data: e.g. GP Prescription Data?


7. Assessing the impact of change in one outcome measure on change in another- Linking data on different people

Work and Pensions Longitudinal Study(DWP)

Pupil Level Annual School Census & National Pupil Database (DfES)

Child Benefit records (HMRC)

Conclusion

Huge increase in interest and usage of longitudinal linkage of administrative data

Invaluable for unpicking individual-level dynamics driving area-level changes

Invaluable for small area analyses – surveys rarely robust at neighbourhood level

Great deal of untapped potential – especially for targeting, monitoring and evaluating interventions to tackle social disadvantage

Using longitudinal administrative data to evaluate area-based initiatives

Documents

Transcript of Using longitudinal administrative data to evaluate area-based initiatives