SDMX Implementation in the European Statistical System

P. Everaers, A. Götzfried Eurostat

What is SDMX ?

• An information model delivering standardised data/metadata structure definitions;

• Statistical guidelines standardising statistical concepts and code lists, and establishing general governance;

• An IT infrastructure supporting the SDMX-based data and metadata exchanges.

SDMX = set of standards to be used for data/metadata exchange and dissemination

Mr. Bruno TISSOT

Ms. Mariana KOTZEVA

Mr. Louis Marc DUCHARME

Mr. Stefan SCHWEINFESTMs. Haishan FU Ms. Martine DURAND

Mr. Werner BIER

Costs/benefits of SDMX

The national/international data/metadata exchange processes are improved through the use of internationally agreed standards and the reuse of SDMX objects and IT infrastructure.

Costs: initial investment costs by statistical organisations; maintenance costs of the SDMX standards/infrastructure carried by the SDMX Sponsors.

Benefits: Reuse of the technical and statistical standards and of the IT infrastructure decreases the costs for data/metadata exchange and increases the data quality.

Main SDMX achievements in 2013/2014

•More and more internationally agreed Data Structure Definitions (DSDs) for global use (e.g. 4 Data Structure Definitions for National accounts); •Various guidelines for the creation/maintenance/governance of SDMX DSDs and other objects; •SDMX Global Registry, storing these DSDs and other objects opened in April 2014 (https://registry.sdmx.org); •New SDMX statistical guidelines/codelists published on sdmx.org; •SDMX Reference Infrastructure more and more used by countries as basic IT infrastructure.

SDMX deliverables planned for 2014/2015

•Internationally agreed validation language VTL as new SDMX standard;•Publication of the SDMX-JSON Data Exchange Format (easier to use) (currently under public review); •Redesign of the SDMX website; •New SDMX implementations in international statistical domains such as demography/population statistics; •SDMX Global Conference increasing training/capacity building actions (Bangkok, autumn 2015).

For data exchange

GNI INT TRADE * CENSUS AVIATIONGross National Income

WATER R&D* FISHERIES *

POPULATION * MARITIME

PESTICIDES USE EGR IS TICEurogroup register Trade in currency

ESSPROS TEC WASTE

NATIONAL A/Cs *STSShort-term Statistics BOP *

EDUCATION * FDI *

* Denotes that other International Organisations are involved.

Over 20 Eurostat domains are in various phases of SDMX implementation.This concerns about 25% of all eligible Eurostat processes.

Trade by Enterprise Characteristic

social protection statistics

ORCHARDS / PESTICIDES.

JVS / LCIJob Vacancy Statistics Labour Cost Index

Coordinated by Eurostat

Coordinated by IMFCoordinated by OECD

Coordinator not yet agreed upon

SDMX implementation in the ESS

For reference metadata exchange

SDMX-based reference Metadata Structure Definitions (MSDs):•Euro SDMX Metadata Structure (ESMS), and •ESS Structure for Standard Quality reports (ESQRS).

Both ESS standards are implemented for more and more ESS reference metadata flows; about 50 % of all national and Eurostat files are already standardised.

International exchange of reference metadata still to be put in place.

SDMX implementation in the ESS

SDMX implementation in the ESS

SDMX-based international data sharing

Objectives:

•reduction of the reporting burden on NSIs (data exchange only with one international organisation); •more efficient use of resources in the International Organisations (IOs) concerned; •improvement of the quality and consistency of official statistics disseminated by the IOs involved (identical data value disseminated by all IOs involved); •improvement of the global data exchange and dissemination processes thanks to the use of international standards.

International Data Sharing

The international data sharing model (1)

European statistics: From national to Eurostat

EU Member state

EU Member state

EU Member state

EU Member state

EU Member state

EU Member state

EU Member state

EU Member state

Data Validation Data Validation

EurostatEurostat

The international data sharing model (2)

EU countriesEU countries OECD countries(non-EU countries only)OECD countries(non-EU countries only)

Other countries(non-OECD countries only) Other countries

(non-OECD countries only)

Eurostat - ECBEurostat - ECB

OECDOECD

IMF, UN, WB, ILO, BIS, other IOs

IMF, UN, WB, ILO, BIS, other IOsU

SERS

USERS

Eurostat as international hub for European statistics

International Data Sharing

Necessary pre-conditions (1)

• Internationally agreed statistical standardsSNA 2008, ESA 2010, BPM6, UOE questionnaire, international

manuals, etc.

• SDMX data structure definitions for global use to be agreed upon

Integrated international data structure definitions need to be agreed between the International Organisation and national statistical organisations concerned (i.e. the different use needs from International Organisations need to be reconciled); a governance procedure is needed.

International Data Sharing

SDMX implementation in the ESS

Necessary pre-conditions (2)

• Internationally agreed data validation

Data validation rules and data validation standards need to be agreed upon; data validation should not be repeated by several international organisations; the upcoming SDMX standards should be used.

• Streamlined data exchange and dissemination processes

National and international organisations need to agree upon the frequency and timeliness for data exchange and data dissemination, data quality management etc.; SDMX IT infrastructure should be used.

• Formal agreements between international organisations necessary

EurostatEurostat

SDMX co-operation/capacity building provided by the ESS (1)

Creating and maintaining common SDMX data structures e.g. with

• The SDMX Sponsors (IMF, ECB, OECD etc.) • UN organisations such as UNSD, UNESCO, UNCTAD etc. • ………………

Capacity building (training services) provided for • The enlargement countries on SDMX in national accounts, July

2014, Skopje; • The ENP South countries on SDMX, June 2014, Luxembourg• The ENP East countries on SDMX, November 2014

EurostatEurostat

SDMX co-operation/capacity building provided by the ESS (2)

Capacity building (training services) provided for • ECLAC/South American Countries, in particular Chile; • The Statistical Centre for the Cooperation Council for the Arab

Countries of the Gulf (GCC- Stat)• Asian countries within the OECD Expert Group meeting in

09/2014; • Ecuador, Colombia on SDMX, autumn 2014

Attention: regular e-training provided by Eurostat which is accessible globally

EurostatEurostat

Useful links

Eurostat : http://epp.eurostat.ec.europa.eu/portal/page/portal/eurostat/home/

European Statistical System :http://epp.eurostat.ec.europa.eu/portal/page/portal/pgp_ess/ess/ess_news

SDMX Main Site :http://sdmx.org/

SDMX Global Registry : https://registry.sdmx.org/

EurostatEurostat

Conclusions

SDMX is the outcome of a fruitful and long-standing international co-operation delivering standards and an IT infrastructure for data/metadata exchange and dissemination.

The uptake of SDMX accelerated in the last years as more SDMX standards got available and as larger statistical domains will use these standards.

The implementation of SDMX if official statistics needs to progress to use its full potential.

EurostatEurostat

Questions

Are you aware of these data sharing activities in your organisation ?

What can be done from central data/metadata services or from central IT services to facilitate data sharing ?