About opendata
-
Upload
lorenzino-vaccari -
Category
Science
-
view
218 -
download
0
Transcript of About opendata
Lorenzino Vaccari 22/06/2016
About Open Data(Please, interrupt me and make questions!)
1
Lorenzino Vaccari
Seminar at POLIMI@Lecco
Lorenzino Vaccari 22/06/2016
Who am I?
2
Lorenzino Vaccari 22/06/2016
Part 1: for data prosumers• What are Open Data useful for?• What is Open Data?• Why are Open Data useful? • How is Open Data related to Open
Government Data and Big Data? • The Open Data movement
3
Lorenzino Vaccari 22/06/20164http://15years.morizbuesing.com/
Austrian designer Moriz Büsing created this grim interactive map of migrant and refugee deaths on the way to Europe, or trying to stay in Europe; over 32,000 deaths in 15 years.
Lorenzino Vaccari 22/06/2016
Open Source & Open Data together to tackle with humanitarian projects and economic development
5
HOT: Humanitarian OpenStreetMap Team
http://hot.openstreetmap.org
Lorenzino Vaccari 22/06/20166http://www.webmapp.it/mappe/dolomiti/
Lorenzino Vaccari 22/06/2016
Map of the pianos
7https://github.com/brunetton/OpenPianosMap
Lorenzino Vaccari 22/06/20168http://content.stamen.com/files/cartography/index_watercolor.html
Lorenzino Vaccari 22/06/2016
Open Data Shoes
9http://in2.ccio.co/K2/LA/G/218143175671272930BwpG7bSyc.jpg
Lorenzino Vaccari 22/06/2016
What is Open Data?
10
Lorenzino Vaccari 22/06/2016
“is data that can be freely used, reused and redistributed by anyone – subject only, at most, to
the requirement to attribute and sharealike.” *
*(Source: )
http://opendatahandbook.org/guide/en/what-is-open-data/ 11
Lorenzino Vaccari 22/06/2016
• use• reuse• redistribution• commercial reuse• derivative works
BUT, may require:• attribution• share alike
J. Gray (OKF): http://www.slideshare.net/jwyg/open-government-data-what-why-how12
“open” =
Lorenzino Vaccari 22/06/2016
“Open” data
13
• Open License• Free• Open Access, e.g.:
• No registration
• No co-authorship
• Direct access (no services)
• ….
https://unsplash.com/@ryanmoreno
Lorenzino Vaccari 22/06/2016
Open License
14
• A license should be compatible with other open licenses.
• A license is open if its terms satisfy the following conditions...
https://unsplash.com/@rzunikoff
Lorenzino Vaccari 22/06/2016
Open license: Required Permissions
15
The license must irrevocably permit (or allow) the following:
Use: The license must allow free use of the licensed work.
Redistribution: The license must allow redistribution of the licensed work, including sale, whether on its own or as part of a collection made from works from different sources.
Modification: The license must allow the creation of derivatives of the licensed work and allow the distribution of such derivatives under the same terms of the original licensed work.
Separation: The license must allow any part of the work to be freely used, distributed, or modified separately from any other part of the work or from any collection of works in which it was originally distributed. All parties who receive any distribution of any part of a work within the terms of the original licenseshould have the same rights as those that are granted in conjunction with the original work.
Compilation: The license must allow the licensed work to be distributed along with other distinct works without placing restrictions on these other works.
Non-discrimination: The license must not discriminate against any person or group.
Propagation: The rights attached to the work must apply to all to whom it is redistributed without the need to agree to any additional legal terms.
Application to Any Purpose: The license must allow use, redistribution, modification, and compilation for any purpose. The license must not restrict anyone from making use of the work in a specific field of endeavor.
No Charge: The license must not impose any fee arrangement, royalty, or other compensation or monetary remuneration as part of its conditions.
http://opendefinition.org/od/2.1/en/
Lorenzino Vaccari 22/06/2016
Open license: Acceptable Conditions
16
The license must not limit, make uncertain, or otherwise diminish the permissions required in Section 2.1 except by the following allowable conditions:
Attribution: The license may require distributions of the work to include attribution of contributors, rights holders, sponsors, and creators as long as any such prescriptions are not onerous.
Integrity: The license may require that modified versions of a licensed work carry a different name or version number from the original work or otherwise indicate what changes have been made.
Share-alike: The license may require distributions of the work to remain under the same license or a similar license.
Notice: The license may require retention of copyright notices and identification of the license.
Source: The license may require that anyone distributing the work provide recipients with access to the preferred form for making modifications.
Technical Restriction Prohibition: The license may require that distributions of the work remain free of any technical measures that would restrict the exercise of otherwise allowed rights.
Non-aggression: The license may require modifiers to grant the public additional permissions (for example, patent licenses) as required for exercise of the rights allowed by the license. The license may also condition permissions on not aggressing against licensees with respect to exercising any allowed right (again, for example, patent litigation).
http://opendefinition.org/od/2.1/en/
Lorenzino Vaccari 22/06/2016
Open “Data”
17
Best practices:• Primary source• Timely• Open format • Updated and complete• Machine readable • ...
Lorenzino Vaccari 22/06/2016Maurizio Napolitano: http://www.youtube.com/watch?v=YlkjrVAW43Q
Primary source
18
Lorenzino Vaccari 22/06/2016
Open Format & Machine Readable
19http://5stardata.info/en/
Lorenzino Vaccari 22/06/201620https://upload.wikimedia.org/wikipedia/commons/7/79/14LaAc_periodic_table_IIb.jpg
“It’s great to have the data accessible on the Web under an open license (such as PDDL, ODC-by or CC0), however, the data is locked-up in a document. Other than writing a custom scraper, it’s hard to get the data out of the document.”
make your stuff available on the Web (whatever format) under an open license
Lorenzino Vaccari 22/06/201621
“Splendid! The data is accessible on the Web in a structured way (that is, machine-readable), however, the data is still locked-up in a document. To get the data out of the document you depend on proprietary software.”
make it available as structured data (e.g., Excel instead of image scan of a table)
Lorenzino Vaccari 22/06/201622
DateTime,MC2016-01-01 00:00:00.000,58.8083312016-01-01 00:10:00.000,59.3740012016-01-01 00:20:00.000,58.7208332016-01-01 00:30:00.000,57.982016-01-01 00:40:00.000,57.6060032016-01-01 00:50:00.000,56.7620012016-01-01 01:00:00.000,55.6591842016-01-01 01:10:00.000,54.942862016-01-01 01:20:00.000,54.2632682016-01-01 01:30:00.000,52.9224512016-01-01 01:40:00.000,53.1673472016-01-01 01:50:00.000,54.8079992016-01-01 02:00:00.000,57.0632632016-01-01 02:10:00.000,58.2571412016-01-01 02:20:00.000,58.0359992016-01-01 02:30:00.000,57.8612252016-01-01 02:40:00.000,57.071432016-01-01 02:50:00.000,56.3387762016-01-01 03:00:00.000,55.452….
http://data.jrc.ec.europa.eu/dataset/jrc-abcis-ap-pm10mc-2016
“Excellent! The data is not only available via the Web but now everyone can use the data easily. On the other hand, it’s still data on the Web and not data in the Web.”
make it available in a non-proprietary open format (e.g., CSV as well as of Excel)
Lorenzino Vaccari 22/06/201623https://data.europa.eu/euodp/en/data/dataset/jrc-names
“Wonderful! Now it’s data in the Web. The (most important) data items have a URI and can be shared on the Web. A native way to represent the data is using RDF, however other formats such as Atom can be converted/mapped, if required.”
use URIs to denote things, so that people can point at your stuff
Lorenzino Vaccari 22/06/201624
“Brilliant! Now it’s data,in the Web linked to other data. Both the consumer and the publisher benefit from the network effect.”
https://data.europa.eu/euodp/en/data/dataset/jrc-names
link your data to other data to provide context
Lorenzino Vaccari 22/06/2016
Why are Open Data useful?
25
Lorenzino Vaccari 22/06/2016
The value is in its use
26http://clicnews.ie/tag/lego/
Lorenzino Vaccari 22/06/2016
Open Data Benefits● The Open data are the knowledge base to:
● Improve the economic grow and the entrepreneurship based on the development of digital services reusing Public Sector Information
● Answer to social needs through the publication of innovative services and applications
● Aims at reducing the cost of the public administrative activities within Public – Private Partnerships (PPP)
● Improve the transparency of the activities of the public institutions and the participation of the citizens to these activities
27
Lorenzino Vaccari 22/06/201628
Economic Growth“Today, the cumulative value
of products and services derived from open access to weather data is estimated at $15 billion.”
http://www.accuweather.com/
http
://w
ww
.soc
rata
.com
/blo
g/ec
onom
ic-i
mpa
ct-o
pen-
data
/
Lorenzino Vaccari 22/06/2016
Potential value in Open Data ($billions)
29
Lorenzino Vaccari 22/06/2016
Innovation: new visualizations
http://wheredoesmymoneygo.org/ 30
Lorenzino Vaccari 22/06/2016
How are Open Data related to Open Government Data and Big Data?
31
Lorenzino Vaccari 22/06/2016
Open Government Data
32
“The three principles of transparency, participation, and collaboration form the cornerstone of an open government”
Barack Obama, 8/12/2009
https://www.whitehouse.gov/sites/default/files/omb/assets/memoranda_2010/m10-06.pdf
Lorenzino Vaccari 22/06/2016
Open Government Data
33
Lorenzino Vaccari 22/06/2016
Big Data & Open DataVariety
Volume Velocity
• Structured• Unstructured• Semi-structured• …
• Terabytes• Records• Transactions• Tables, Files
• Batch• Real Time• Streams• Near-time
3V’s
34
Open Data is often one of the sources for Big Data
Lorenzino Vaccari 22/06/2016
State of the art: the Open Data movementWhat is happening around us ? Some examples...
● Globally● Europe● Italy● Locally
35
Lorenzino Vaccari 22/06/2016
Open Data Charter - G8 (12/07/2013)The principles are:
● Open Data by Default
● Quality and Quantity
● Useable by All
● Releasing Data for
Improved
Governance
● Releasing Data for
Innovation
http://opensource.com/government/13/7/open-data-charter-g8
https://www.gov.uk/government/publications/open-data-charter/g8-open-data-charter-and-technical-annex
36
Lorenzino Vaccari 22/06/2016http://census.okfn.org/
OGD around the world
37
Lorenzino Vaccari 22/06/2016
The GEOSS portal
38
The GEOSS CORE data
principles
● Full and Open Exchange of
Data, recognizing Relevant
International Instruments
and National Policies
● Data and Products at
Minimum Time delay
● Free of Charge or minimal
Cost for Research and
Education
http://www.geoportal.org/web/guest/geo_home
Lorenzino Vaccari 22/06/2016
OpenStreetMap: OD & Crowdsourcing
39
OpenStreetMap is a free map of the world, created by someone like you
“OpenStreetMap project creates and provides geographical data, such as road
maps, freely available to anyone. Behind the establishment and growth of the project have been restrictions on use
or availability of map information across much of the world and the advent
of inexpensive portable satellite navigation devices”
https://www.openstreetmap.org
Lorenzino Vaccari 22/06/2016
An example: Lecco
40http://tools.geofabrik.de/
Lorenzino Vaccari 22/06/2016
OGD in Europe - Pan European
http://www.europeandataportal.eu/en/ 41
Connecting Europe Facility launches second call(16/05/2016)
The Connecting Europe Facility (CEF) in Telecom is an EU programme to facilitate cross-border interaction between public administrations, businesses and citizens, through the deployment of Digital Service Infrastructures. One of its aims is to support projects which contribute to the European ecosystem of the deployed interoperable and interconnected digital services.…
485,473 datasets found
Lorenzino Vaccari 22/06/2016
OGD in Europe - EU ODP● screenshots
http://open-data.europa.eu/ 42
Lorenzino Vaccari 22/06/2016
The INSPIRE geoportal
43http://inspire-geoportal.ec.europa.eu/discovery/
Lorenzino Vaccari 22/06/2016
OGD in Italy
http://www.dati.gov.it44
Lorenzino Vaccari 22/06/2016
OGD in Lombardy
45https://www.dati.lombardia.it/
Lorenzino Vaccari 22/06/2016
Open Data @Lecco?
46
● Search if Lecco has an official Open Data web site: ○ Which datasets (domains)?○ Which formats?
■ How many stars?○ Which licenses?
■ Is it clear the type of license for each dataset?
● Are there any other web sites in Lecco?○ Are there any Universities which share
Open Data?
Lorenzino Vaccari 22/06/2016
Your Open Data (data provider)
47
● Do you think you could be an Open Data provider? E.g. with the datasets of your thesis?
● Would you like to share them openly?
● If not, why?
Lorenzino Vaccari 22/06/2016
Your Open Data (data consumer)
48
● Which data are you working on?○ Where do you get them from?
● Which data would you like to find on Internet?○ Are the dataset you download fine with
you? If not, why?
Lorenzino Vaccari 22/06/2016
Questions?About Open Data
49
Next part: 2 - for men in the middle
Lorenzino Vaccari 22/06/2016
Part 2: for men in the middle● Open Data Issues● Two experiences:
○ Autonomous Province of Trento
■ The story started with GeoData…
■ Now “Open Data in Trentino”: http://dati.trentino.it
■ Community building
○ European Commission: Joint Research Centre
■ http://data.jrc.ec.europa.eu
● Want to learn more?
50
Lorenzino Vaccari 22/06/2016
“Yeahh!!!”
https://unsplash.com/@littleppl85
51
Lorenzino Vaccari 22/06/2016
LegalOrganizational TechnicalAdoptionBarriers
Contextual
52http://goo.gl/9dFm9v
“Ohoh!!!”
Lorenzino Vaccari, Juan Pane 22/06/201653
Organizational Barriers
● Not ready
● Lack of resources (IT, Human)
● Don’t want to be ready
http://montcomediation.org/images/MCMC_MyWayYourWay.jpg
Lorenzino Vaccari, Juan Pane 22/06/201654
Legal barriers
● Open the Data ○ All the data that was produced
using public money has to be
made publicly available (with
exceptions)
● vs Privacy○ You cannot open data that
could allow correlation of
private personal data
http://s177.photobucket.com/user/sealth2828/media/gavel.jpg.html
Lorenzino Vaccari, Juan Pane 22/06/201655
● Data is not contextualized
● Opening data is a complex task, opening
cleaned data is even more complex.
● Unclear licenses
Adoption barriers
http://www.thepadrino.com/2011/01/defendius-labyrinth-security-lock.html
Lorenzino Vaccari, Juan Pane 22/06/201656
Technical Barriers● Access to data:
○ Organizational○ Technical, Downtimes,
logins, ○ Payment fees
● Fragmentation, incomplete data, scattered
● Format● Cataloging,
indexing, search● Lack of explicit
semantics, metadata
● Conflicting standards, models, ontologies
Lorenzino Vaccari, Juan Pane 22/06/201657
● Privileged access to data● Transparency is bad for fraudulent business
Context Barriers
http://img.gawkerassets.com/img/182n8vzdlg1iojpg/original.jpg
Lorenzino Vaccari, Juan Pane 22/06/201658
● Zuiderwijk et al 2010
● Listed 118 socio-technical impediments for opening data in the literature such as:○ Findability○ Usability○ Understandablity○ Quality○ Linking○ Comparability and compatibility○ Metadata○ ….
Barriers
Lorenzino Vaccari, Juan Pane 22/06/201659
Congratulation for the presentation! I am curious about the data you used! Are these datasets freely available? Would you like to publish them as Open Data in the catalog we are creating at the JRC level? Here there is a draft version: http://data.jrc.ec.europa.eu/ . Cheers,Lorenzino
-------------------Hi Lorenzino,sorry but I am not allowed to publish my dataset.Cheers,Xyz
Meanwhile at the JRC… The Data are MINE !
Lorenzino Vaccari, Juan Pane 22/06/201660
Not Exactly...
Lorenzino Vaccari, Juan Pane 22/06/201661
Lorenzino Vaccari 22/06/201662
How to deal with Open Data issues?
Lorenzino Vaccari 22/06/2016
Autonomous Province of Trento● The story started with GeoData…● Now “Open Data in Trentino”: http://dati.
trentino.it ● Community building
63
Lorenzino Vaccari 22/06/201664
The story started with GeoData …
http://www.territorio.provincia.tn.it
Lorenzino Vaccari 22/06/201665
5 Stars Linked Geo Data Catalog
http://www.territorio.provincia.tn.it
Lorenzino Vaccari 22/06/2016
The “Open Data in Trentino” project
66
• The “Open Data in Trentino” project is a 3 years initiative finalized to develop an open data infrastructure to enhance Service Innovation for Trentino following the PAT strategy for services innovation enabled by ICT. The project will be developed within a partnership between Trento RISE and the Autonomous Province of Trento (PAT) according to the innovation PAT model
• Goals• Improved quality of life for citizens• Open Data and local businesses• Transparency• Improved efficiency and productivity
Adopted licences
08/10/2013Juan Pane, Lorenzino Vaccari67
CC0 CCBY
Permissions: share, create, adapt Permissions: share, create, adapt
Actual interoperability! Decent interoperability
Constraints: nothing! Constraints: attribution
Lorenzino Vaccari 22/06/201668
Catalogue
08/10/2013Juan Pane, Lorenzino Vaccari69
The Open Knowledge Foundation (OKF) is a non-profit organisation founded in 2004 and dedicated to promoting open data and open content in all their forms – including government data, publicly funded research and public domain cultural content.
http://okfn.org
Lorenzino Vaccari 22/06/201671
Lorenzino Vaccari 22/06/2016
… inspired from the guidelines of Lombardy!
72http://www.agendadigitale.regione.lombardia.it/
Lorenzino Vaccari 22/06/201673
Services & Apps (do you remeber the cake)? Services & Apps
http://dati.trentino.it/related
Lorenzino Vaccari, Juan Pane 22/06/2016
Create Community
74http://media.gettyimages.com/photos/members-of-the-colla-vella-de-valls-climb-up-as-they-construct-a-picture-id153610809
Lorenzino Vaccari 22/06/201675
Lorenzino Vaccari 22/06/2016
Trentino Open Data (TOD)
76https://www.facebook.com/groups/todgroup/
Lorenzino Vaccari 22/06/201677
Lorenzino Vaccari 22/06/201678
Lorenzino Vaccari 22/06/201679
Lorenzino Vaccari 22/06/201680 Lorenzino Vaccari
Lorenzino Vaccari 22/06/2016
European Commission: Joint Research Centre
● http://data.jrc.ec.europa.eu
81
Lorenzino Vaccari 22/06/2016
JRC and Open Access
82
● for scientific publications/data within Horizon 2020 and by other relevant initiatives (e.g. Research Data Alliance)
● overall trend for public move to open data (G8 charter, INSPIRE..)
As continuation of JRC's efforts to make available and transparent to the public the scientific knowledge produced, in 2014 the JRC will roll out its Open Access strategy for its publications
JRC Management Plan 2014Commission Decision on the reuse of Commission documents (2011/833/EU)
Open Access in EC and beyond
, Anders Friis-Christensen, Andrea Perego
Lorenzino Vaccari 22/06/2016
JRC Open Data project
83
JRC Data Policy- Open Data principles- Data acquisition principles- Data management principles- Implementation principles
JRC Data CatalogueContaining JRC datasets
related to, e.g., Soil, Water, Air quality, Marine,
Biodiversity, etc. http://data.jrc.ec.europa.eu
.
EU Open Data portal
A single access point to a growing range of data from the institutions and other bodies of the EU
https://open-data.europa.eu
Commission Decision on the reuse of Commission documents (2011/833/EU)
Lorenzino Vaccari, Anders Friis-Christensen 22/06/201684 Lorenzino Vaccari, Anders Friis-Christensen 22/06/2016
Lorenzino Vaccari, Anders Friis-Christensen 22/06/201685
A JRC data infrastructure
Lorenzino Vaccari, Anders Friis-Christensen 22/06/201686
Project Scope
JRC Data Policy
Data Policy Implementation Guidelines
Software components
(e.g. data dissemination)
Data
Open Data
Applies to
Lorenzino Vaccari 22/06/201687
http://dati.jrc.ec.europa.eu
Lorenzino Vaccari 22/06/201688
Lorenzino Vaccari 22/06/2016
Want to learn more?
89
Lorenzino Vaccari 22/06/201690http://opendatahandbook.org/pt_BR/
Lorenzino Vaccari 22/06/2016
The EU ODP Training
91http://www.europeandataportal.eu/elearning/
Lorenzino Vaccari 22/06/201692http://schoolofdata.org/
Lorenzino Vaccari 22/06/201694http://www.socrata.com/open-data-field-guide/
Lorenzino Vaccari 22/06/2016
Open Data event in Lecco (8/6/2016)
95http://www.comune.lecco.it/index.php/archivio-news/23-news-dal-comune/2437-convegno-open-data-e-sharing-economy
Lorenzino Vaccari 22/06/2016
Open Data & Smart Cities (EU ODP)
96
Analytical Report 4: Open Data in Cities
http://www.europeandataportal.eu/sites/default/files/edp_analytical_report_n4_-_open_data_in_cities_v1.0_final.pdf
Lorenzino Vaccari 22/06/2016
A question for you: is it difficult to use a data catalogue? Why?
97
From the user point of view (what I found):● I do not known about it● I cannot found what I need
○ “Spaghetti” catalogues■ many records■ not clear what is inside (no clear classification)■ too few datasets
● I do not receive updates○ On datasets I am interested in
● Even if I found it ○ I cannot access it
■ Broken links, access barriers (registrations,…)○ Is the dataset the last version?
Lorenzino Vaccari 22/06/2016
Questions?
98
Thanks For Your Attention!!!!
Acknowledgments: ● Anders Friis-Christensen● Maurizio Napolitano● Juan Pane● Andrea Perego
Lorenzino Vaccari [email protected]