Implementing SDMX exchange of national development indicators

SDMX Expert Group Meeting, Paris, 13 – 14 September 2012

UNSD-DFID Project

1. Introduction

2. Progress on country SDMX

3. Design

4. Key issues

5. Questions & discussion

Introduction: Objectives• Improve the coherence and clarity of dev’t indicators

- Improve coordination in the NSS- Collate development data in 1 place/database- Explain differences between intl. & nat. data

• Improve accessibility and visibility- Make access to national data easier- Draw attention to wider set of indicators- Reduce data request burden

• Enhance knowledge- Strengthen IT support- Use of latest IT software & practices- Training & study tours

Introduction: Background

• Implementation in 11 countries across Africa and Asia

• A pilot ran during 2008-2009, full phase started in late 2010

• All countries initiated now, 5 countries started implementing and 5 about to start.

• Project runs until Jul 2014

Progress on country SDMXCambodia : SDMX registry installed & mappings of 46 time series complete/ available (with further 175 time series identified for mapping)

Burundi : SDMX registry installed

Rwanda : Expanded SDMX registry

Lao & Uganda : Expanded SDMX registry & mapping tool

May 2011

Feb 2012

Jun 2012

Planned to Dec 2012

Design: Overview

• Most project countries are unfamiliar with SDMX– Many do not have experience with underlying

technologies such as XML, XSLT.

• Conflicting pressures on project solution:– Must be simple to use, to facilitate its

adoption– Must be powerful, to fulfill requirements– Must give the user a chance to familiarize

themselves with SDMX.

Design: SDMX solution

• Some of the ways in which project solution simplifies the implementation of SDMX:

– Automated mapping between DevInfo database structures and CountryData DSD

– Automated data export from DevInfo database into CountryData SDMX

– Automated publication of data and registration of datasets at a DevInfo registry.

Design: System

Line Ministries National Statistical Office United Nations

Line Ministry

Database

National Indicator Registry

National Repository DB

DevInfo

Upload

XLS

Scripts

Register files

Post

notification

Publish

SDMX-ML

Download

Mapping tool

Design: SDMX add-ons

• Project works with DevInfo to expand functionality offered by DevInfo registry and offer such new features as:

– User-defined dataflows, metadataflows, and provision agreements

– Constraints

– Metadata discovery and other Web service functionality.

Design: MDG DSD• Supports exchange of MDG Indicator data

between international agencies (UN, UNICEF, UNESCO, …)

• Developed by SDMX Task Team of Interagency and Expert Group on Development Indicators (IAEG)

• Implemented in SDMX 2.0

• CountryData DSD is based on the MDG DSD.

Design: CountryData DSD• Codelists amended to support non-MDG

development indicators published by countries

• To avoid confusion and clearly separate national and international data, a new DSD with its own ID was defined for the project, which reuses MDG artefacts as much as possible.

• Codelists are currently maintained by UNSD

– New indicators proposed by project countries are introduced with a lag

– Centralised codelist maintenance unsustainable in the long run as more countries join the project.

Design: CountryData MSD• Based on the MDG global database, agreed by

the IAEG (i.e. definition, method of computation etc.).

• Attachment at the indicator level

• Key to understanding differences between international estimates and country estimates of the same indicator

• Also benefits in consistency, transparency, reporting burden, …

Key issues: Codelist maintenance• Codelist maintenance must eventually be

taken over by the countries– Compatibility becomes an issue.

• Design under consideration envisages using constrained dataflows to maintain compatibility.

Key issues: Matching• Trying to match a country indicator with an

available indicator code

• At what point is a new code required

• i.e. For the indicator “Using improved drinking water / sanitation” does measurement on a household basis rather than population make it different indicators?

• Or using Gross instead of Net to calculate Gender Parity Indices make them a separate indicators?

Key issues: Mapping• Process of mapping the indicator time series

to other dimensions (i.e. Sex, Age group, Location etc.)

• Complicated by the fact that MDG and other development indicators vary greatly in their dimensionality

• Further guidance developed to ensure codelist items are applied properly

Key issues: Metadata exchange• Metadata is usually not comprehensive

enough to understand comparability.

• The importance of good metadata (and even what it is?) is little understood within some National Statistical Systems.

• Few examples of metadata exchange using SMDX

• Format issues: HTML vs. Plain Text, Presentation vs. security.

Key questions

• Project Sustainability− Sharing national development indicators in

SDMX− Commitment to maintaining good metadata

• Governance− Managing structural metadata among

multiple national and international organizations

• Expansion– System used for greater sharing between

different parties.

Thank you for your attention

http://unstats.un.org

Sources of further reference:

http://data.un.org/countrydata

http://nd.nis.gov.kh/registry/Cambodia SDMX

Registry

Rwanda SDMX Registry

http://sdmx.statistics.gov.rw/