Building a Data Portal with SDMX
description
Transcript of Building a Data Portal with SDMX
Restricted
Views expressed are those of the presenter and not necessarily those of the BIS1
Building a Data Portal with SDMXThe BIS SDMX Sandbox exercise
1
Gabriele Becker, Massimo BruschiBank for International Settlements
METIS, 7 May 2013
Restricted 2
The SDMX vision Users need …
• good quality data, up-to-date numbers, documentation• Single access point for data from different sources• User interfaces: GUI and also a Web service (for
automation) The SDMX vision …
• Data providers (originators) offer their data “in SDMX”• Dissemination = reporting = data sharing … from a single
storage• SDMX registries help users to find data via a GUI and an
SDMX Webservice The BIS SDMX Sandbox exercise
• How “real” is this SDMX vision?• How does it work with difference scenarios?
Restricted 3
The Sandbox setup “in the cloud” 14 participating central banks: 14 Sandboxes with
SDMX registry with user interface and SDMX web service
SDMX data base with data browser and SDMX web service
Space to host SDMX data files User interface for loading files, performing registrations
etc. Detailed tutorials
1 Sandbox for the BIS to experiment and participate “as central bank”
The Data Portal (Unified Data Catalogue or “UDC”) “central” SDMX registry operated by the BIS Data browser Synchronises with the central and CB registries Accesses registered data sources
SDMX Sandbox was developed by Metadata Technology
Restricted
SDMX Sandbox setup 14 Sandboxes for central banks, 1 for BIS as central agency, 1 Data portal (Unified Data Catalogue)
Another CB Sandbox
Another CB Sandbox
Another CB Sandbox
Another CB Sandbox
Another CB Sandbox
Another CB Sandbox
Another CB Sandbox
Data Portal: UDC
SDMX Sandbox Country A
CB SDMX Registy
Data browser
CB SDMXDatabase
BIS SDMXSandbox
Data browser
SDMX Registy
SDMXDatabase
UDC SDMX Registy
Unified Data CatalogueBrowser
4
Restricted
Scenarios1. Data reporting against a “global” SDMX DSD, eg
BOP2. Making internal data available via a “data portal” 3. Bringing external (public) data to your users via a
data portal Common aspects
data is available in SDMX files in a known DSD or …
SDMX web service offers data in a known DSD We just need to know where the data is and
what DSD is follows An SDMX Registry is the place for this
information A simple data browser can offer data selection
based on the known DSDs5
Restricted 6
ECB publishes a lot of data as SDMX files on its website
IMF and BIS offer the Joint Ext. Debt Hub (JEDH) contributions as SDMX files
Users may want to see this data via a common data portal
Organisations may wish to access this data via a SDMX 2.1 web service (that ECB, IMF and BIS currently do NOT offer) demo
Load SDMX DSD into registry demo
• In future: SDMX registries provide this as a service … Define Dataflow, Provision agreement and Categorisation Register the data files as sources for the UDC demo
Scenario 3: “making public data available in a data portal”
Restricted
Demo Movie
7
Restricted 8
Restricted 9
Conclusions I Building blocks are working in the SDMX Sandbox
SDMX registry and SDMX 2.1 webservice SDMX data base (with webservice) Data browser Unified Data Catalogue (data portal)
Connecting different data sources … SDMX files from public websites SDMX files from other (accessible) locations Different SDMX versions (1.0, 2.0, 2.1) SDMX databases (internal and external) SDMX Web service
First implementation of a federated SDMX registry
Restricted 10
Conclusions II
Harmonisation of data access techniques was achieved… User sees all data through one “data portal” (the
UDC) SDMX 2.1 web service for all data, also for those
offered as SDMX 1.0 files Web service supports automated data retrieval
Works best with harmonised data structures! BOP Sandbox example Key deliverables for SDMX: DSD for global use for
BOP and National Accounts … under way BOP Sandbox created by IMF based on this exercise
The SDMX Vision is real!
Restricted 11
Thank you !
Restricted
Background slides
Scenario 1 and 2
12
Restricted 13
Scenario I: “BOP” (Balance of payments) National agencies have to “report” their
data based on a commonly agreed data structure
This data needs to be available at international organisations and for the public
This is a real life scenario for BOP, National Accounts and other data domains. demo
SDMX BOP data structure “for global use” is a prerequisite• We invented a simplified one for the purpose of this
exercise …
Restricted 14
BIS SDMXSandbox
SDMX Sandbox : BOP example
Another CB Sandbox
Another CB Sandbox
Another CB Sandbox
Another CB Sandbox
Data Portal: UDC
Unified Data CatalogueUDC SDMX Registy
SDMX Sandbox Country A
Data browser
SDMXRegisty
Data browser
SDMXDatabase
SDMX-ML file with Central bank BOP
data
Register file as data source
UDC user can see BOP data from all participating central banks
Uses UDC
Retrieve data from DB or file
Registry sychronisation
Registry synchronisation
SDMX-ML file with Central bank BOP
data
CB data provider
Store on Web location
BOP DSD
BOP data flows, provision
agreements, data source, registrations
CB SDMX Registy
BOP DSD
CB SDMXDatabase
Central bank BOP
data
Import
Register asqueryable
data source
RetrieveDSD info
for browsing,registrations
for datalocation
BOP DSD
CB DBadmin
provider action user action
System interaction
· Data manager registers data file in UDC Registry
· SDMX data file is uploaded to a Web location and then registered as data source against the BOP provision agreement
· It’s immediately visible to a UDC user· Date manager imports data in the
Sandbox DB· It’s immediately visible to a user of the
Sandbox data browser
Restricted 15
Scenario 2: “dissemination database for BIS” BIS wishes to disseminate data to
internal users via a common data portal (example: BIS Banking Statistics)
BIS has/defines SDMX DSD for this data demo
BIS uses the SDMX database in the Sandbox as “dissemination database” and loads the data via SDMX files demo
Internal users use the data browser (similar to UDC) to access the “BIS SDMX database”. demo
For external dissemination the BIS registers its Sandbox database as a data source to the UDC demo
This also enables the UDC search service for this data demo
Restricted
UDC: Data Portal
BIS data provider
SDMX-ML file data for DSD
SDMX Sandbox : dissemination data base example
UDC user
BIS SDMX Sandbox
UDC Registryy
DSD, Data flows, Provision agreements
and data source registrations
Retrieves DSD info for browsing
and registrationsfor data location
SDMXDatabase
with SDMX Webservice
Register asqueryable
data source
Uses UDC
Load
Retrieves data from SDMX database
Registrysynchronisation
provider action user action
SDMX Registry
DSD, Data flows, Provision agreements
and data source registrations
System interaction
BIS data user
Data browser
Use data browser
Retrieves data from SDMX databaseCreate DSD,
data flow etc.
UDC Browser
SDMX-ML file data for DSD
Store on Web location
Register as queryable data source
· Data manager creates or loads a DSD into the registry
· defines data flow and provision agreement
· loads a data file in the SDMX Database. · It is immediately visible to a user of the
Data browser attached to the Sandbox.· Data manager registers SDMX DB as
data source for the UDC.· UDC users can see the data
16