Data Quality – UK activities

Iain MacleayHead of Energy Balances, Prices and Publications

27 September 2013

1. Aspects of quality2. Standard errors3. Revisions4. Risk based quality reviews5. Quality training

Contents

DECC follow UK statistical practice:- Relevance- Accuracy- Timeliness & punctuality- Accessibility & clarity- Comparability- Coherence

Aspects of quality

DECC release data to pre-announced, year in advance, timetable – all releases at 9:30am. Energy Trends – Thursday 26 September;

Thursday 19 December Thursday 27 March Thursday 26 June

Dates set for coming year, by the DECC Chief Statistician – no political interference

If data not released at 9:30 – DECC need to report breech to UK National Statisticians Office

Data released as soon as available

Timeliness & punctuality

Difficult to measure, but …- Sample sizes of surveys published with

information on coverage- Where useful, standard errors published- Weighting to adjust for coverage- Administrative sources used where

appropriate- Check accuracy of recording by comparing

data sources (volume surveys, price surveys, company reports)

Accuracy

Sample sizes and standard errors for Quarterly Fuels Inquiry - published in industrial price methodology note

- As working in policy departments – regular liaison so data meet needs.

- Also try to anticipate their future needs.- Regularly survey of wider user

community to check meeting their needs, every 2 to 3 years – results published on web

- Review content of press notices and channels of communication (tweets etc)

Relevance

- Data presented in consistent format- Helpful commentary drawing users to key

points of interest (even if politically difficult), written independently by the statistics team

- Clear info on contact details of DECC statistical teams

- All info available for free on web- Metadata published – detailed method notes

on web- Some info on revisions published

Accessibility & clarity

Revisions – final consumption annual growth after one quarter

Number of observations (n) 32 First order of autocorrelation (a) of revisions -0.0654Mean of the revisions (m) -0.1074 Adjusted variance of the revisions 0.3531Variance of the revisions (s2) 0.4026 Number of independent observations (n*) 32t-statistics -0.9571 t-adjusted -1.0219t-critical(±) 2.0395 t-adjusted critical(±) 2.0369Test significant at 5% significance level? No Test significant at 5% significance level? No

Mean Revisions = 0.1074- Test used Standard Absolute mean revisions = 0.4842 Test significant? No Is test significant? No

Test for significance of mean revisions

Standard t-test for the revisions Adjusted t-stat for the revisions

-2.0

-1.5

-1.0

-0.5

0.0

0.5

1.0

1.5

Q1-2005 Q1-2006 Q1-2007 Q1-2008 Q1-2009 Q1-2010 Q1-2011 Q1-2012 Q1-2013

Perc

enta

ge p

oint

s

Revisions in the year-on-year percentage growth rates estimates in final consumption, after 1 quarter

- Monthly data consistent with quarterly, and annual data – revised in line with better more complete information

- Standard geographies used where possible

- Energy balance format used so supply and demand consistent

Coherence & comparability

- Data collections and publications should be reviewed on a regular basis

- Tricky in practice – time consuming activity

- Risk based approach being trialled- Methodically go through checklist- Most activities fairly low risk

Quality reviews

Risk based review template

Sources Methods Systems Processes Quality Users & reputation

People

Census Data acquisition/questionnaire design

System a Data collection & preparation process

Relevance User feedback

People

Admin Coverage of data

System b Results & analysis processes

Accuracy Future user needs

Survey Processing, edit & imputation

Timeliness & punctuality

Reputation

Analysis Accessibility & clarity

Disclosure Comparability

Coherence

Domestic fuel prices inquiry

Sources Methods Systems Processes Quality Users & reputation

People

Survey – 98% sample coverage

Complex detailed survey, many issues including change of tariff structure etc.

System redesigned in 2012

Data validation & editing

Produces bills based on standard consumption rather than actuals

Good feedback received

New person each year as data processed by sandwich student

Good geographical coverage

Spread sheet back-up available

Main system newly developed, but back-up used as double check

Release 12 weeks after end of quarter

Key policy area – so new data needs emerging

Large survey – company 100 tariffs in 14 regions - so much scope for problems.

Analysis

Information published is disclosive, but pre-agreed with former monopoly suppliers

Actions1. Meet companies to improve form filling2. Engage pro-actively with policy to find future

needs

3. Ensure good documentation4. Have sufficient staff trained to use system5. Check data with that from similar surveys6. Check data against firms published annual reports

- How do we ensure good quality statistics

- Well trained staff- Training sessions held focusing on

quality- All staff to attend – take through stages

of statistical value chain- In DECC two statisticians trained up to

train others

Quality training