Paper by Tim Hawkes, presented by Emma Bentley
description
Transcript of Paper by Tim Hawkes, presented by Emma Bentley
Improving efficiency by introducing macro editing in
Statistics New Zealand business performance surveys
Paper by Tim Hawkes,
presented by Emma BentleyMay 2011
Background
Suite of business performance (BP) surveys, many started around 2005
Quality review recommendations: • Reduce manual micro edits, cut processing costs• More emphasis needed on macro editing• ‘Big picture’ perspective needed for analysts
New approach to editing developed for 2010 survey round
2
Previous editing approach
Micro editing – time consuming (2 FTE!)• Validity edits• Consistency edits• Statistical edits
Very few automatic edits, lots of manual intervention
Macro editing – limited• Pressure of publication deadlines• Not standard across the suite of surveys
3
Recommended new approach
BP analysts reviewed macro practices in-house and editing strategies internationally
Decided to improve:• Automatic micro editing
– Consistency edits– Validity edits
• Macro editing– Create more time for this– Wider range of strategies available to BP team
4
A routing question with:
Both Yes and No then the section is blank Auto edit to No
Both Yes and No then the section is answered Auto edit to Yes
No then the section is answered Auto edit to Yes
Yes then section is blank Auto edit to No or leave for imputation
Automatic micro editing
5
Sum of components not equal to given totals
Routing and validity edits
Macro editing
Calculating estimates during the data collection process• Produce initial estimates when 50-60% of target
response rate achieved• Early detection of influential observations for
validation• Problems with processing or estimation system can
be identified and resolved early in production process• Issue if insufficient responses for estimation purposes• Analysts gain better understanding of estimation
process6
Macro editing
Top-down editing• E.g. drill down to industry and stratum level estimates• Identify unusual components, drill down more• May provide evidence for estimate, or identify error
Use of sampling errors to identify suspicious estimates
• Compare with expectations and previous results• Unusual responses that have effect on sample errors,
may also effect estimates
7
Macro editing
Top contributor method• Ranked lists of top contributors e.g. for level
estimates or movements• Compare lists to previous years
Graphical analysis• Not widely used by BP surveys
Processing checks• Monitoring template developed containing these
checks
8
Implementation
Interactive training provided by Statistical Methods division
Analysts asked to identify potential questions suitable for automatic edits
Up to the lead analyst to implement the new approach and determine their macro editing techniques
BP team able to take more ownership of editing strategy now that the surveys are well established and topic knowledge developed
9
Review: Quality indicators
Quality indicators for business performance surveys
• No decline in quality detected
Quality Indicator
Business Operations
SurveyR&D Survey ICT Supply Energy Use
2009 2010 2008 2010 2008 2010 2009 2010
Average number of edit failures per unit 5.72 6.66 5.00* 4.23* 1.34 1.00
Clerical edits 4130 2497 2619 787 4000* 900* 1206 740
Resource usage (staff hours) – Micro 337.5 187.5 375* 150* 375* 100*
Resource usage (staff hours) - Macro 15 40.5 112.5* 150* 188** 375**
* Estimate** Estimate which also includes resource usage on imputation.
10
Review: Qualitative feedback
Generally positive
Reduction in amount of manual micro editing – popular with analysts!
Timing for incorporating changes a challenge• Set up of automatic editing, will be reusable in future• Saved time from micro editing taken up by
implementation of other methodology changes
11
Review: Qualitative feedback
More time allocated to macro editing
Better macro strategies helped analysts better understand the processes and data, this in turn helps subsequent analysis stage
Conservative approach for first cycle of new editing strategy, room for more efficiencies
Improved ability to calculate quality indicators would help assess efficiencies in future
Reviews are useful!
12