Eurostat

7c. SDMX practical use case: National Accounts

Daniel Suranyi and Cristina BlanaruEurostat

SDMX Basics course, 27-29 October 2015

Eurostat

Content

• Projects around SDMX and National Accounts• Governance, maintenance, sharing

• Implementation in NA• Business process re-engineering• Structural and content validation• Interational data sharing

• National perspective• Working with the DSD Matric• Working with the SDMX-MES DSDs• Architecture & tools, validation demonstration

Eurostat

National processWhy? : The business case

I want to…

… structure my statistical datasets … exchange data with other organisations … align my definitions internationally using

standards … validate that data is correct

Eurostat

The big picture: ongoing activities

ESS.VIP Validation

ESS.VIP SERV

Eurostat NAPS-S

Service Oriented Architecture for

National Accounts

SDMX-VTLValidation &

Transformation Language

SDMX-MESOwnership Group

on SDMX for Macroeconomic

Statistics

TFIDCTask Force

International Data Sharing

Eurostat

The big picture: ongoing activities

ESS.VIP Validation

ESS.VIP SERV

Eurostat NAPS-S

Service Oriented Architecture for

National Accounts

SDMX-VTLValidation &

Transformation Language

SDMX-MESOwnership Group

on SDMX for Macroeconomic

Statistics

TFIDCTask Force

International Data Sharing

• SDMX for Macro-Economic Statistics

• Mandated by SDMX Sponsors in January 2014• As a single OG for NA & GFS, BOP and FDI

• Maintenance:NA/GFS (https://sdmx.org/?page_id=1498) and BOP/FDI (https://sdmx.org/?page_id=1747)

• New domains can be added (e.g. Prices)

• Representation of managementfrom content and SDMX side

• Oversees maintenance of SDMX artefacts• Mandated technical group(s)

SDMX-MES Ownership Group (OG)

National Accounts• SNA 2008• ESA 2010• Other NA related domains

FDI statistics

Other domains

Balance of PaymentsStatistics

Cross Domain Concepts and

Code ListsShared

Concepts and Code Lists

Concepts in SDMX-MES DSDs1 FREQ2 ADJUSTMENT3 REF_AREA4 COUNTERPART_AREA5 REF_SECTOR6 COUNTERPART_SECTOR7 CONSOLIDATION8 ACCOUNTING_ENTRY9 STO

10 INSTR_ASSET11 MATURITY12 ACTIVITY13 ACTIVITY_TO14 PRODUCT15 PRODUCT_TO16 EXPENDITURE17 PENSION_FUNDTYPE19 UNIT_MEASURE20 CURRENCY_DENOM21 VALUATION22 PRICES23 TRANSFORMATION24 CUST_BREAKDOWN

ConceptsCross Domain

SharedNA specific

Obs OBS_STATUSObs CONF_STATUSObs EMBARGO_DATESer DECIMALSSer TABLE_IDENTIFIERSer UNIT_MULTSer COMPILING_ORGSer REF_YEAR_PRICEDset COMMENT_DSET

… …

Coding in SDMX-MES DSDs1 FREQ2 ADJUSTMENT3 REF_AREA4 COUNTERPART_AREA5 REF_SECTOR6 COUNTERPART_SECTOR7 CONSOLIDATION8 ACCOUNTING_ENTRY9 STO

10 INSTR_ASSET11 MATURITY12 ACTIVITY13 ACTIVITY_TO14 PRODUCT15 PRODUCT_TO16 EXPENDITURE17 PENSION_FUNDTYPE19 UNIT_MEASURE20 CURRENCY_DENOM21 VALUATION22 PRICES23 TRANSFORMATION24 CUST_BREAKDOWN

Concept CL MaintenanceCross-Domain SDMX

Cross-Domain IMF/ESTATno CD code list

Cross-Domain IMF/ESTATcode list extended

Shared/Specific IMF/ESTATNot coded -

Obs OBS_STATUSObs CONF_STATUSObs EMBARGO_DATESer DECIMALSSer TABLE_IDENTIFIERSer UNIT_MULTSer COMPILING_ORGSer REF_YEAR_PRICEDset COMMENT_DSET

… …

Data Structure Definitions (DSDs)DSD IDNA_...

DSD Title Dataflow defintions

MAIN Main aggregates ESA / OECD: 1, 3, 5, 20, 22SEC Sector accounts

and government finance

ESA / OECD: 2, 6, 7, 8, 9, 11, 26, 27, 28ECB: MUFA, GFSEDP: 1, 2, 3, 4OECD: 119, 7HH, 7II, PSDG20: Recommendation 15

SU Supply / use and input / output

ESA : 15, 16, 17, 18, 19

REG Regional statistics ESA: 10, 12, 13PENS Pension schemes ESA: 29

Eurostat

The big picture: ongoing activities

ESS.VIP Validation

ESS.VIP SERV

Eurostat NAPS-S

Service Oriented Architecture for

National Accounts

SDMX-VTLValidation &

Transformation Language

SDMX-MESOwnership Group

on SDMX for Macroeconomic

Statistics

TFIDCTask Force

International Data Sharing

SDMX Validation Possibilities

“Technical” - Covered by SDMX today

- Format Check (SDMX-ML)

- Codes exist (SDMX DSD)

- Codes used correctly(SDMX Dataflow & Constraint)

“Statistical Domain” - Not yet covered by SDMX (VTL project)

- Value check

- Time series

- Revisions

- Validation expressions

13

What is VTL

A standard language Considers the validation as a kind of Transformation (calculation)

For defining validation and transformation rulesValidation (now)

Transformation (partially now, to be enriched at a later stage)

Main goals:Define and preserve validation and transformation rules

Exchange and share rules

Apply rules in industrialized processes

Apply to several standards (e.g. SDMX, DDI, GSIM) thanks to a generic information model

14

VTL features (4)

VTL 1.0 allows:• Persistent and temporary results• Operations on mono and multi measure data• Dealing with missing data• Dealing with Attributes and their propagation rules

https://sdmx.org/?page_id=5096

VTL 1.1 will introduce:• Other operators, mainly for validation purposes• Reusable rules• Bug fixing, fine tuning

Eurostat

15

Summary: Current ESA business process

National Production

Generate SDMX-ML

PUSH Transmissio

n

Structural Validation

Content Validation

Processing &

Calculation

Data Sharing &

Dissemination

Structural Validation

Data provider Eurostat

Eurostat

16

Shared / ReplicatedServices

For NAand others

Possible future architecture (NAPS-S)

SDMX Registry

Structural Validation

VTL Repository

Content Validation

Common Repository

Statistical Service A

Common Repository

Statistical Service B

Eurostat

SDMX Dataflow defintion

• 1 DSD covers multiple Dataflows• NA_MAIN DSD:

• Table 1: T0101_A, T0102_A, T0103_A …• Table 3: …• Table 5: …• …

• Each of those Dataflows gives a partial picture• "cutting the DSD": SDMX cube region• "listing each series": SDMX key set

Eurostat

RoleAttachement

levelDimensionDimensionDimensionDimensionDimensionDimensionDimensionMeasureAttribute series

Concept ID Concept Name

FREQ FrequencyREF_AREA Reference area ACTIVITY Activity classificationADJUSTMENT Adjustment indicatorPRICES PricesSTO Stocks, Transactions, Other FlowsTIME_PERIOD Reference periodOBS_VALUE Observation valueTABLE_IDENTIFIER Table identifier

Code list ID

CL_FREQCL_AREACL_ACTIVITYCL_ADJUSTMENTCL_NA_PRICESCL_NA_STO

CL_NA_TABLEID

SDMX Information Model - Summary

DSDConcept Scheme

Code listsReference Reference

Reference

Data Flow Constraint

Reference

Eurostat

SDMX Cube Region Constraint

• The DSD has an Activity code list with >170 items• In table 101 we only report A*10• SDMX cube region constraint (linked to dataflow):

• List of allowed codes by concept

?

Eurostat

SDMX Key Set Constraint

• Which series are allowed / forbidden?

Eurostat

The big picture: ongoing activities

ESS.VIP Validation

ESS.VIP SERV

Eurostat NAPS-S

Service Oriented Architecture for

National Accounts

SDMX-VTLValidation &

Transformation Language

SDMX-MESOwnership Group

on SDMX for Macroeconomic

Statistics

TFIDCTask Force

International Data Sharing

The necessary pre-conditions

• SDMX maintenance agreements Ownership and maintaining agency to be defined (normally between the international organisations)

• Internationally agreed data validation Data validation rules and SDMX data validation standards (beyond file structure validation)

• Streamlined data exchange processesFrequency and timeliness for data dissemination, data quality management etc.

SDMX Web Services

A possible data sharing model

EUOECD

other than EUWorld other than OECD

Eurostat - ECB

OECD

IMF, UN, WB, BIS, other IOs

USERS

Technical Integration

Nationalstatistical organisations

Internationalorganisations

National database

s(Mapping)

Source

Source

Source

SDMX Data Structure

Defintions & Data Flows

Nationaldata sources

Metadata driven process based on SDMX standards and tools Full automation of the data exchange possible

SDMXImplementation

Challenge

Eurostat

SDMX Basic Course

National perspective: National Accounts

29th October 2015

Daniel SURANYICristina BLANARU

Eurostat

National processWhy? : The business case

I want to…

… structure my statistical datasets … exchange data with other organisations … align my definitions internationally using

standards … validate that data is correct

Eurostat

Why? : The business case How we succeeded to… … structure my statistical datasets

using SDMX Data Structure Definitions … exchange data with other organisations

using Push, Pull or Hub infrastructure … align my definitions internationally using

standardswith SDMX cross-domain concepts, code lists

… validate that data is correctusing SDMX schemas and VTL

Eurostat

Overall process

Exchange Needs

Concept Scheme

Coding

DSD Matrix

Optimising

Eurostat

SDMX-NA DSD Matrix explaination- Overview Sheet(1)

Eurostat

SDMX-NA DSD Matrix explaination- Overview Sheet(2)

summarises all concepts and code lists

Define the concepts (Cross domain concepts, Domain specific concepts)

Adding code lists to the concepts

Eurostat

SDMX-NA DSD Matrix explainationMatrix Sheet(3)

Eurostat

SDMX-NA DSD Matrix explainationMatrix Sheet(4)

define the relationship between the datasets included in the scope of the transmission and the concepts

each concept has a hyperlink pointing to the corresponding code list sheet.

SDMX-NA DSD Matrix explainationMatrix Sheet(5)

DSD Matrix: data flows X conceptsData Flow FRE

QREF_ARE

ASTO

ACTIVITY

CONS

PRODUCT

Main Aggregates % # % #GDP and main indicators

% # %

Sector Accounts % # % #General Government % # % NSupply / Use A # % %

SDMX-NA DSD Matrix explainationMatrix Sheet(6)

• # concept fully used• % concept partially used• (code) one code of concept used• (blank) concept not used

SDMX-NA DSD Matrix explainationCode list Sheets(7)

Showing the contents of each of the code lists used:

CL_FREQ sheet CL_AREA sheet CL_ACCOUNT_ENTRY sheet CL_MATURITY sheet………….

Eurostat

Data Structure Definitions (DSDs)

• 5 DSDs covers multiple Dataflows• NA_MAIN DSD• NA_SEC• NA_REG• NA_SU• NA_PENS

• All structures (DSDs) stored on the SDMX Global Registry+ on SDMX website: www.sdmx.org

Data Structure Definitions on the SDMX Global Registry

Eurostat

Data Structure Definitions on the on SDMX website

DSD ID and Name Version SDMX DownloadExcel templates*last update

NA tables

NA_MAINMain Aggregates

1.5 2.0 / 2.1V1.4a 12/12/2014

ESA: 1, 3, 5, 20, 22

NA_SECSector Accounts & Government Finance

1.5 2.0 / 2.1

V1.4a 12/12/2014

ECB: MUFA, GFS, FAM

V1.4a 12/12/2014

ESA-FINA: 6, 7

V1.4c **01/06/2015

ESA-SA: 8, 26

V1.4a 12/12/2014 ESA-GFS: 2, 9, 11, (25), 27, 28

V1.4a 12/12/2014 OECD: 119, 7HH, 7II, PSD

NA_REGRegional Accounts

2.0 *** 2.0 / 2.1V1.4a *12/12/2014

ESA: 10, 12, 13

NA_SUSupply/Use & Input/Output Statistics

1.5 2.0 / 2.1V1.4b **19/01/2015

ESA: 15, 16, 17, 18, 19

NA_PENSPensions

1.5 2.0 / 2.1V1.4b **09/05/2015

ESA: 29

NA_MAIN93Main Aggregates, SNA93

1.5 2.0 / 2.1 N/A N/A

NA_SEC93Sector Accounts, SNA93

1.5 2.0 / 2.1 N/A N/A

IMF:GFSIMF Government Finance Statistics Manual 2014

1.0 2.0 / 2.1 N/A N/A

Data provider view:One goal – different possibilities

Webforms Online data collection tools SDMX-ML file

Database export

SDMX Converter SDMX-ML file

Database SDMX Reference Infrastructure

SDMX-ML file

SDMX Web Service

Same DSDSame Format

Excel sheets

SDMX Converter SDMX-ML file

Data provider view:One goal – different possibilities

Webforms Online data collection tools SDMX-ML file

Database export

SDMX Converter SDMX-ML file

Database SDMX Reference Infrastructure

SDMX-ML file

SDMX Web Service

Same DSDSame Format

SDMX Converter SDMX-ML fileExcel sheetsPHASE OUT FOR ESA IN 09/2016

 

SDMX Reference Infrastructurehttps://webgate.ec.europa.eu/fpfis/mwikis/sdmx/index.php/SDMX_Reference_Infrastructure_SDMX-RI

Data Provider Data Collector

Non-SDMXlocal database

SDMX data set

SDMX Registry

DSD

SDMX-RI

Mapping Assistant

Web Svc

Test Client

Web Client

NSIs: Local Database (1)

Create a DB for National Accounts (if you don't have one in your NSI)

One table per DSD (just 5 tables for the whole transmission program)

Avoid the proliferation of tables A simply DB structure Load data into table Use Mapping Assistant tool of SDMX-

RI package

NSIs: Local Database (2)

Create a link from NSI source DB to the DSD

Use Test Client of the SDMX –RI package (to export data from the table to an SDMX file)

Validate the table Send the table to EUROSTAT

Create a link from NSI source DB to the DSD Mapping Assistant: Database SDMX (3)

NSI Web Client

Web Client pivot table

Eurostat

Fusion Cloud SDMX Validation Servicehttps://data.sdmxcloud.org/

Validation against DSD Correct DSD used Correct concepts and identifiers Dimensions and mandatory attributes present Correct codes used

Validation against Data Flow Correct codes for specific ESA2010 table

Simple data viewer

Eurostat

Demo fusion cloud validation (1)

Eurostat

Demo fusion cloud validation (2)

Eurostat

Demo fusion cloud validation (3)

Eurostat

Summary

• Projects around SDMX and National Accounts• Governance, maintenance, sharing

• Implementation in NA• Business process re-engineering• Structural and content validation• Interational data sharing

• National perspective• Working with the DSD Matric• Working with the SDMX-MES DSDs• Architecture & tools, validation demonstration