The use and convergence of quality assurance frameworks for international and supranational...
-
Upload
merilyn-white -
Category
Documents
-
view
216 -
download
0
Transcript of The use and convergence of quality assurance frameworks for international and supranational...
The use and convergence of quality assurance frameworks for international and supranational
organisations compiling statistics
The European Conference on Quality in Official Statistics July 8-11 2008, Rome, Italy
Antonio Baigorri and Håkan LindenStatistical Governance, Quality and Evaluation
European Commission, Eurostat
2
The context of Total Quality Management
To have an encompassing approach with respect to quality work.
To implement the principles of institutional quality frameworks and in particular the principles related to statistical processes and outputs.
To improve the measurement, monitoring and management of data quality.
To coordinate ongoing quality initiatives (process descriptions, quality reports, evaluation activities etc.).
To build on existing quality work (standards, best practices etc.).
To promote a culture of systematic quality improvement work.
3
Institutional Quality Frameworks
User needs
Statistical
products
Production
processes
Institutionalenvironment
Management systems and
leadership
Support processes
Principles: 1.1, 1.3, 1.4, 1,5, 2.1, 2.2, 2.3, 3.1, 3.2,4.1, 5.3, 5.6, 6.1, 6.2, 7.1, 8.2, 9.1, 10.3, 10.4
Principles: 4.2, 5.1, 5.3, 5.4, 5.5, 8.3, 8.4, 9.2, 9.3, 9.4, 9.5, 10.1
Principles: 1.1, 1.2, 4.3, 4.4, 4.5, 4.6, 5.2, 6.2, 7.1, 7.2, 8.1, 10.2, 10.5
TQM
CCSA Principles
Institutional frameworks, like the Principles Governing Statistical Activities, the European Statistics Code of Practice and the IMF Data Quality Assessment Framework, can be seen as general superstructures forming the necessary basis for all other measures an International organisation needs for improving quality at statistical output and product level.
User needs
Statistical products
Productionprocesses
Institutionalenvironment
Management systems and leadership
Support processes
1 Professional independence 2 Mandate for data collection3 Adequacy of resources 4 Quality commitment 5 Statisticalconfidentiality 6 Impartiality and objectivity
7 Sound methodology 8 Appropriate statistical procedures9 Non-excessive burden on respondents 10 Cost effectiveness
11 Relevance 12 Accuracy and reliability 13 Timeliness andPunctuality 14 Coherence and comparability 15 AccessibilityAnd clarity
TQM
Code of Practice
Source: Eurostat (2007)
User needs
Statistical products
Productionprocesses
Institutionalenvironment
Management systems and
leadership
Support processes
0. Prerequistes of Quality (Legal and institutionalenvironment, Resources, Relevance, Other quality management 1. Assurance of Integrity (Professionalism, Transparency and Ethical Standards
2. Methodological Soundness 3.Accuracy and Reliability
4. Serviceability5. Accessibility
TQM
DQAF
4
Quality Assurance Frameworks Quality assurance frameworks (or frameworks for statistics production) have
the objective to establish, in a specific statistical organisation, a system of coordinated methods and tools guaranteeing the adherence to minimum requirements concerning the statistical processes and products. Similarly to institutional frameworks, this includes some kind of assessment.
Product/ output quality requirements are being explicitely documented.
Processes are defined and made known to all staff.
The correct implementation of the processes is monitored on a regular basis.
Users are being informed on the quality of the products and possible deficits.
A procedure is implemented that guarantees that the necessary improvement measures are being planned, implemented and evaluated.
5
Data quality aspects
1) The perception of the statistical product by the user.
2) The characteristics of the statistical product (or key statistical outputs)
3) The characteristics of the statistical production process.
6
Co
nce
ptu
al f
ram
ew
ork
(2
)
IT conditions (11) – Management, planning and legislation (12) – Staff, work conditions and competence (13)
User needs(3)
Data collection
(4)
Validation
Country level (5)International level (6)
Confidentiality (7)
Dissemination (9)
Documentation (8)
Follow-up (10)
RELEVANCE ACCURACY ACCESSIBILITY/CLARITY
TIMELINESS/ PUNCTUALITY
COMPARABILITY COHERENCE
Relationship between process and output quality
Source: Eurostat Process Quality Assessment Checklist [Eurostat, 2007]
7
Product/ Output Quality Components
OECD: relevance, accuracy, credibility, timeliness (and punctuality), accessibility, interpretability, coherence (within dataset, across datasets, over time, across countries)
Eurostat: relevance, accuracy, timeliness and punctuality, accessibility and clarity, coherence (within dataset, across dataset), comparability (over time, across countries)
ECB: accuracy/reliability, methodological soundness, timeliness, consistency
IMF: prerequisites of quality, accuracy and reliability, assurances of integrity, methodological soundness, serviceability (timeliness and periodicity), accessibility, serviceability (within dataset, across dataset, over time, across countries)
FAO: relevance (completeness), accuracy, timeliness, punctuality, accessibility, clarity (sound metadata), coherence, comparability
UNESCO: relevance, accuracy, interpretability, coherence
UNECE: relevance, accuracy (credibility), timeliness, punctuality, accessibility, clarity, comparability (across datasets, over time, across countries)
8
Product/ Output Quality Components
Relevance
Accuracy (and reliability)
Timeliness
Punctuality
Accessibility
Clarity/ interpretability
Coherence/ consistency
Comparability
9
Quality and metadata
Collection and sharing of metadata
- SDMX technical standards
- SDMX Content oriented guidelines (incl. Cross domains concepts)
Dissemination of metadata on quality
- Special Data Dissemination Standards (SDDS)
- Euro-SDMX Metadata Structure (ESMS)
Assessment and monitoring of metadata
Integrated information on quality assessment
10
Methods and tools for the assessment of statistics production
Production processes Statistical products
User perception
Process variables
Quality indicators
Quality reports
User satisfaction survey
Self assessments Quality reviews
Labelling
Institutional/ legal environment
User requirements Standards
III. Conformity
II. Evaluation
I. Documentation Measurement
Improvementactions
N.B. Figure derived from draft Handbook on Data Quality Assessment Methods and Tools (DatQAM), version 31.01.2007.
11
How to apply process oriented quality assessment tools
The office-wide management approach
Institutional preconditions (procedures and legislations)
Assessment methods already in use
Relevance – size and periodicity
Relevance – importance and specific legal frameworks
12
Quality Assessment Packages
Fundamentalpackage
Intermediatepackage
Advancedpackage
Process descriptions, product documentation, quality guidelines
Quality reports
Self assessments
Quality reviews
User satisfaction surveys
Quality indicators
Key process variables
Labelling
N.B. Figure derived from draft Handbook on Data Quality Assessment Methods and Tools (DatQAM), version 31.01.2007.
13
The assessment methods and tools
Documentation and measurement
- process descriptions
- quality reports (“Full Quality Report”, “Summary Quality Report”, and “Basic Quality Information”).
- user satisfaction surveys
Evaluation
- self assessments of all production processes (Quality Assessment Checklist)
- quality reviews for key statistical outputs
Conformity
- a process for labelling of key international statistics
14
Principles for implementation
Minimise burden for production domains
- test the approach in advance
- provide support
- flexibility
Build on existing information
- process analysis
- metadata on quality (quality reports etc.)
Profit from synergies with other horizontal activities
- evaluation function requirements
- cost/ benefit analysis
- input for management programming
15
Data quality assessment recommendations
Top management commitment
The role of middle managers
Data quality assessment is a long term project
Most methods should be implemented and fine-tuned in pilot projects
Standardise the use of the methods
Establish clear responsibilities and authorities
Sufficient resources allocated for supporting the assessments
16
Conclusions
Each international organisation should have a quality assurance framework in place.
The framework and the applied quality principles should be made explicit.
A quality assurance framework needs to be compatible with the general quality management model and office-wide procedures and rules.
It should be built into the organisational structure.
It contributes to increased awareness of quality concepts and promotes best practices.
It provides a mechanism for reengineering and quality improvements
It should always acknowledge performance/ cost.
Convergence of quality assurance frameworks by applying common concepts, standards, methods and tools (both content oriented and technical).
Development and sharing of “best practices” for statistics production between all stakeholders is maybe the most important for continuous quality improvement of a global statistical system.