Post on 22-Mar-2018
Big Data and Official Statistics The UN Global Working Group
Dr. Ronald Jansen
Chief, International Trade Statistics
United Nations Statistics Division
jansen1@un.org
Overview
What is Big Data?
What is Big Data for Official Statistics?
What are the Challenges?
UN Global Working Group on
Big Data for Official Statistics
What is Big Data?
Big Data characteristics
Highly distributed
Loosely structured
Large in volume
Big Data sources:
o Mobile Devices
o Digital Transactions
o Sensors
o Social Networking
Big Data benefits
Automatically
generated data
Timely data
Potentially relevant
What is
Big
Data?
Processing and Analyzing of Big Data sources (IT concerns):
o Parallel processing techniques (HADOOP)
o Non-relational data storage capabilities
o Advanced analytics and data visualization technology
Big Data for Development
UN Global Pulse
o Early warning
o Real-time awareness
o Real-time feedback
Big Data for
Development UN GLOBAL PULSE
o Research & Development
o Big Data Partnerships
o Pulse Lab Network
New York
Jakarta
Kampala
What is Big Data for Official Statistics?
Big Data sources / Statistics
• Mobile Phone Positioning Data / Tourism statistics
• Supermarket scanner data / Price statistics
• Vehicle tracking device / Transport statistics
• Twitter/ Consumer confidence statistics
• Satellite imagery / Agriculture statistics
What are the Challenges?
Big Data Challenges
• Methodology
• Privacy
• IT infrastructure
• Human skills
• Partnerships
Methodological Challenges
Methodology
• Representativeness ?
• Volatility (no time series?)
• Standardization ?
• Modelling
Big Data partnerships opportunities
Big Data providers
Data Sources
Research institutes and technology
providers
- Upstream processing
- IT infrastructure,
Academia and scientific communities
- Analytical capacity and modelling
National Statistical Offices
- Standards and methodology
Partnership
Challenges
Partnership arrangements
Financial commitments
Legislative issues
Privacy and confidentiality
Responsibilities and ownership
IT infrastructure/technical platforms
Three examples of partnership arrangements for Big Data based on mobile phone positioning data
Mobile Phone Positioning Data
Call Detail Record (CDR)
Data Detail Record (DDR) (Internet Protocol Detail Record)
Location Updates (LA)
Radio coverage updates (Abis data)
Example 1: Estonia
Data provider/ source
• Mobile phone operator
• Provides raw data to research institute
Research institute
• Provides IT infrastructure
• Preparation of data
• Ensuring privacy and confidentiality
Academia • Providing modeling
and analytics
National Statistical
Office
• Receives pre-processed and aggregated data
• Prepares and disseminates statistics
Shared responsibility for
data processing and
analysis
Example 2: Netherlands
Data source/ provider
• Vodafone
• Makes data available to research institute
Research institute
• Intermediate commercial group
• Provides IT platform for data filtering and privacy
National Statistical
Office
• Receives pre-processed Big Data
• Analyze, prepare and disseminate statistics
Responsibility for data
Example 3: Belgium
Data provider/source
• Belgacom
• Provides IT platform
• Processes data
National Statistical Office
• Prepares and disseminates statistics
Shared responsibility for preparation and
analysis of Big data
Example 4: Tanzania ??
Data provider/ source
• Orange?
• Provides IT platform ?
National Statistical
Office
• Prepares and disseminates statistics
Regional Office ?
• Processes data
UN Global Working Group on Big Data
Mandate
Based on Decision 45/110, the mandate of the working group is formulated as
follows:
1. Provide strategic vision, direction and coordination of a global programme
on Big Data for official statistics;
2. Promote practical use of sources of Big Data for official statistics, while
building on the existing precedents in Big Data and finding solutions for
i. Methodological issues, covering quality concerns and data analytics;
ii. Legal issues of access to data sources;
iii. Privacy issues, in particular those relevant to the use and reuse of
data, data linking and re-identification
iv. Security and management of data, including assessment of cloud
computing and storage;
v. IT/technological expertise, including cost benefit analysis
UN Global Working Group on Big Data
Mandate
(continue)
3. Promote capacity building, training and sharing of
experience;
4. Foster communication and advocacy of use of Big Data
for policy applications;
5. Build public trust in the use of private sector Big Data
for official statistics including issues of confidentiality
and privacy.
Composition of the group
Developed countries
Australia Denmark
Italy Mexico
Netherlands USA
Developing countries
Bangladesh China
Colombia Morocco
Philippines Tanzania
International Organizations
ITU OECD
UN Global Pulse UNECE
UNESCAP UNSD
World Bank
Deliverables
Guidelines / Handbooks
Methodology Privacy
IT infrastructure Human skills Partnerships
Big Data examples
* Mobile Phone Positioning Data
* Scanner data * Tracking device * Twitter * Satellite imagery
Pilot Projects