a long-standing cooperation to serve research
-
Upload
steel-joseph -
Category
Documents
-
view
25 -
download
5
description
Transcript of a long-standing cooperation to serve research
Data Archiving and Networked Services
&
Peter Doorn (DANS)Ruben Dood (CBS)
a long-standing cooperation to serve research
REGIONAL WORKSHOP16th & 17th October 2014, Athens
Contents
Past:• Long-standing collaboration dating back to 1960s
– Steinmetz Archives acquires punch-card collections– Scientific Statistical Agency founded in 1994– Digitization of historical censuses
Present: • Covenant governing the collaboration
– De-anonymized microdata sets of samples archived at DANS– Remote & On-Site Access: separate contracts with CBS
Future:• Linked data, big data, more data…
Problems of comparing and accessing the Housing Survey and the Labour Force Sample Survey of the CBS (1985)
One of my first publications
Tape of the Labour Force Sample Survey 1981
Digitization of historical censuses, 1996 - …
CBS-archive & library: analog data warehouseFormer CBS building in Voorburg
Archiving and digital restoration of the 1960 census
Program code by Rinus Deurloo merged with data records:
1115100421115 6302120001000995581111405057126086200 B"("N3=")"5ZD,10B 17601115100421110 1306363301000075-81111718035817732405 SC2+NSC3); 17701115100421116 1305352202000900521111205041728284204 ",/")"); 17801115100421119 4303430001000930521111203038829276500 B"("N3=")"5ZD,10B 1790
Punch errors or bitrot?121586010 3855413012060 3 52701322981010 106012158W000 3755113010010 2 52801322981010 0061121586001 3406713012050 00 0152701322981010 1860
Missing and double records:
Persons Missing persons
Double persons
Men 183.970 254.100Women 182.755 7.661Total 366.725 261.761
• 11 milion anonymized inhabitants on 11 million punch cards
• 2000 punch cards per box, 5500 boxes
• Stored in two locations in the late 60s, early seventies
• Punch cards read in 1973, stored on tape
• Further inventory of the census data in 1982
• Analysed and cleaned in 1994
• Re-analysis and digital restoration in 2004
7
datasets andadministrations
survey andadministations
Companies
(government) agencies
administrations
Internetsurvey
Callcenter
Interview
Statistics Netherlands process
Individuals
CBS=Statistics Netherlands (SN)
Linking ‘anonymous’ datasets
RIN
PersonCompanyHouseholdLocation
RINRIN
RIN
Connecting with previous years
RIN
20132012
20112010
Kinds of data
Secure Use Files• For SN employees only. Data at individual level.
Datasets can be linked.• Scientific research under very strict conditions.
Scientific Use Files• Also: Microdata Under Contract, Controlled
Circulation Files.• Data at individual level, but not as detailed as
secure use• Datasets can not be linked
Public Use Files• Can be freely distributed.
Risk management
The risk is that a single person, household or company can be recognized in the dataset.
With Secure Use Files this is unavoidable. Hence very strict conditions.
With Scientific Use Files this is still possible, therefore we provide extra security measurements.
Costs concerning types of data
Secure UseFiles
Scientific Use Files
Public Use Files
Production Covered by SN-budget
Publication
Costs concerning types of data
Secure UseFiles
Scientific Use Files
Public Use Files
Production Covered by SN-budget Not covered open data
Publication Not covered Not covered open data
Microdataservices
MicroDataServices (MiDaS)
Reserved for:• Institutions mentioned in the Statistics
Netherlands Act: universities, planning agencies and Eurostat.
• Institutions/projects admitted by the Central Committee for Statistics.
• no direct administrative authority,
• explicit research aim,
• publish for public use and
• a good reputation
Conditions for access tot secure use files
• Access is limited to the datasets that are required for the research at hand.
• Proof of employment for each researcher.• Confidentiality declaration by each
researcher.• IT-security: Citrix• Output screening by SN.• Additional bio-metric identification
Biometric identification
16
Current use
• Over 2000 well-documented datasets.• Over 100 customer organisations (mainly
universities, planning agencies and statistical research agencies).
• Over 500 registered researchers.• 8 On Site, 100 Remote Access work stations.
Over 15 of the RA-sites are not in the Netherlands.
• Over 10 new projects per month. • Over 280 active projects at this moment.
17
Scientific Use Files
Can be part of a contract between DANS and Statistics Netherlands. Can also be funded by a third party.
Are published in DANS EASY format and published on the DANS website.
Present: Covenant CBS-DANS• Period: 2011-2015• CBS makes available (micro-) data for scientific users entitled or
authorized by CBS, according to a yearly agreed list of data files. The files are delivered with sufficient documentation and a code book
• DANS archives these files in a durable way and makes them available to those entitled or authorized by CBS
• DANS pays CBS a yearly sum (116 Keuro since 2013)• CBS en DANS jointly propose better data access for research to
ministries
CBS Datasets in DANS EASY archive
of which 87 protected microdata files
Joint publications
[Including related projects:Historical GIS, HASH, HDNG]
Joint seminars & conferences
DANS at CBS Volleyball tournament
Present & future: Linked DataChallenge: link information from tables across time and to information in the outside world• 3 Types of Censuses [Population/Occupation/Housing]• 17 census years • 2288 census tables• 33283 annotations• 17 million characters
Tables
Linked data from tables (RDF)
Linked Open Data cloud
Harmonization Harmonization
T1 T2 T3
Census 1869 Province of Noord-Brabant
• 110,585,567 total triples• 10,272,862 marked cells triples• 389,132 hierarchical row headers• 7,960,911 data cells• 61,110 column headers• 3,609 row properties• 2,150 titles• 1,581,546 row headers• 274,404 metadata cells
http://www.cedar-project.nl/ http://lod.cedar-project.nl
original paper table
digital spreadsheet
Additional DANS wishes for future and further cooperation
1. More data (e.g. on health, economy, etc.)2. More data3. More data4. More user groups entitled/authorized to access CBS data
(and data produced in collaboration with CBS)5. Use of CBS register data for sample drawing for research
projects6. Joint project proposal on e.g. linked data and big data7. Archiving of historical CBS datasets by DANS8. CBS certification (as trustworthy digital repository)
Data Archiving and Networked Services
&Thank you for your attention
www.dans.knaw.nlwww.narcis.nlwww.cbs.nl
[email protected]: @pkdoorn @DANSKNAW