In search of lost knowledge: joining the dots with Linked Data

Jon Blowerj.d.blower@reading.ac.uk

Department of MeteorologyReading e-Science Centre

National Centre for Earth ObservationUniversity of Reading

With thanks to all CHARMe project partners!

platform

instrument

algorithm

dataset

publicationscientists

How do you fill in the blanks?

• Hope the documentation contains pointers• … to the right versions of things• … and using consistent terminology

• In restricted communities, we can gather all information into a central location and harmonize it

• OR we hope that a Google search throws up the right results

Problems

1. Crossing communities is very hard

www.fooddeserts.org

Problems

1. Crossing communities is very hard2. Lots of useful stuff isn’t formally documented

1000 hypotheses

100 are true 900 are false

95 truepositives

5 falsenegatives

45 falsepositives

855 truenegatives

Test accuracy 95%

Lots of useful negative results, which aren’t publishedNearly 1/3 of positive results are false

Problems

1. Crossing communities is very hard2. Lots of useful stuff isn’t formally documented3. Unknown unknowns…

“There is a spurious decline in low-cloud fraction in the ISCCP cloud database due to the viewing angle.

It’s in the literature, but you might not find it if you’re not specifically looking for it. For example if you’re using the data for model validation you may not spot the problem.”

(Claire Barber, paraphrased)

Consequences

• We use the data that are most easily available, not necessarily the best

• Constant re-discovery of the same issues

• Very hard to share information outside communities

The “Web of Data”

platform

instrument

algorithm

dataset

publicationscientists

The Web is the“Web of Documents”

What is being cited?Why?

Fingers crossed!

• We hope that everything has a document written about it, or we have nothing to cite

• … and we hope someone writes a new document when “it” changes

• We hope that we are citing the right document (i.e. the most authoritative etc)

Photo by Susan Lesch, from Wikipedia

Sir Tim Berners-Lee“If … the Web made all the online documents

look like one huge book,

[the Semantic Web] will make all the data in the world look like one

huge database.”

Why is the Semantic Web different?

• Focuses on things, not documents-about-things

• Links things together in a meaningful way

• Information is readable by computers

• Here’s how it works…

1. Give things unique identifiers

Must be globally unique and persistent

http://www.reading.ac.uk/people/jon.blowermight represent me

http://nasa.gov/instruments/MODISmight represent the MODIS instrument

–(these are illustrative, not accurate!)

2. Express relationships between things

The key concept is the triple

subject predicate object

E.g. “MODIS is carried by the Terra satellite”

all three things need to be unique identifiers (we don’t use plain English!)

Examples of realistic triples(identifiers are illustrative, not necessarily correct!)

http://www.reading.ac.uk/people/jon.blowerhttp://xmlns.com/foaf/0.1/knowshttp://www.durham.ac.uk/staff/ed.llewellin

http://dx.doi.org/12345.678910 http://purl.org/spar/cito/citesAsDataSourcehttp://www.badc.ac.uk/datasets/sst

Aside: how to use the Semantic Web to ruin jokes

“A hundred kilopascals go into a bar”

“100000”^^unit:Pascalowl:sameAs“1”^^unit:Bar

(Using OWL and QUDT ontologies)

3. Publish your triples

• Can be as simple as putting a document in the right format on your website– (triples can be expressed in many formats)

• … or as complex as publishing a multi-billion-triple database– E.g. Ordnance Survey

• Either way, your data become part of the Web, not just on the web

4. Profit!• Everyone can publish their own “view of the world”

– Don’t need a single data structure “to rule them all”

• Use common identifiers and common vocabularies to link between communities

• Different vocabularies can be mapped to each other– E.g. map CF standard names to GRIB codes

• Gives a means to record provenance of information– “Direct line of sight from decisions to data” - NASA

• Reasoning engines can traverse the graph of links and discover new information

Example: SemantEco

Wildlife observationsEcosystem impactsAdministrative areasPollution data and policies

“Find me all the sites where chloride pollution exceeds the local policy limits”“Show me the species over time in this region”

So what’s “Linked Data” then?

• The Web is the “web of documents”• The Semantic Web is the “web of data”• “Linked Data” is really a set of principles for

using the Semantic Web– http://www.w3.org/DesignIssues/LinkedData.html

• (Many people use “Linked Data” and “Semantic Web” somewhat interchangeably)

carries

produced

describes / uses / queries …

produced

authored

The “Web of Data”

Linking Open Data cloud diagram, by Richard Cyganiak and Anja Jentzsch. http://lod-cloud.net/

CHARMe:Sharing climate knowledge through

Linked Data(Jan 2013 – Dec 2014)

Satellites

Climate Record

Creation and

Curation

ObservationsAnalysis

and applications

Climate DataRecords

Decision making

Reports Decisions,Policies

ClimateModelling

Model results(obs. sometimes

used for model initialization)

From observations to decisions

(Adapted from Dowell et al., 2013), “Strategy Towards an Architecture for Climate Monitoring from Space”)

Where can users go for help?• Scientific literature

– Huge, verbose and inaccessible to some communities– Not well linked to source data

• Technical reports and conference proceedings– Hard to find, scattered or inaccessible

• Data centres– increasingly strong at providing some important metadata, but don’t

usually include community feedback– Not all countries and communities have data centres!

• Websites and blogs– From CEOS Handbook to a scientist’s blog– Increasingly useful, but scattered

How can climate data users decide whether a dataset is fit for their purpose?

(N.B. We consider that “data quality” and “fitness for purpose” are the same thing)

Not specific to climate data!

“Commentary metadata”

37

Examples of commentary metadata• Post-fact annotations, e.g. citations, ad-hoc comments and notes;• Results of assessments, e.g. validation campaigns,

intercomparisons with models or other observations, reanalysis;• Provenance, e.g. dependencies on other datasets, processing

algorithms and chain, data source;• Properties of data distribution, e.g. data policy and licensing,

timeliness (is the data delivered in real time?), reliability;• External events that may affect the data, e.g. volcanic eruptions, El-

Nino index, satellite or instrument failure, operational changes to the orbit calculations.

General rule: information originates from users or external entities, not original data providers– However, sometimes information is not available from the data

provider!

How will this be done?

• CHARMe will create connected repositories of commentary information– Stored as triples in

“CHARMe nodes”,• Information can be read

and entered through websites or Web Services

• Using principles of Open Linked Data and the Semantic Web

CHARMe node CHARMe

node

CHARMe node

3rd party system

3rd party system

Data provider website

Open Annotation• We are using Open Annotation for

recording commentary– World Wide Web Consortium standard

• Associates a body with a target

• E.g. a publication could be the body, an EO dataset could be the target

• Can record the motivation behind an annotation– Bookmarking, classifying, commenting,

describing, editing, highlighting, questioning, replying… (lots more)

– Covers a lot of CHARMe use cases!

• An annotation can have multiple targets– A key requirement from users

Satellites

Climate Record

Creation and

Curation

ObservationsAnalysis

and applications

Analysis and

applications

Climate DataRecords

Decision making

Reports Decisions,Policies

ClimateModelling

(e.g. CMIP5)

Model results(obs. sometimes

used for model initialization)

Where does CHARMe fit in?

Supports analysts and scientists in production of information for

decision-makers

www.fooddeserts.org

Challenges• Ensuring community adoption:– We are “injecting” CHARMe capabilities into existing

websites used in the community– We will “seed” the CHARMe system with information

to attract users (e.g. links between publications and datasets)

• Ensuring quality of commentary metadata– Moderation is a strong user requirement– We will provide guides to creators of commentary –

what makes a comment helpful?

What CHARMe will enable(some examples)

Users: - “Find me all the documents that have been written about this dataset” - “… in both peer-reviewed journals and the grey literature” - “… and specifically about precipitation in Africa” - “… in both NEODC’s and Astrium’s archives”

- “What factors might affect the quality of this dataset?”e.g. upstream datasets, external events

- “What have other users already discovered about this dataset?”

- “I want to find information related to the dataset I’m looking at”

Data providers: - “Who is using my dataset and what are they saying about it?” - “Let me subscribe to new user comments and reply to them”

What CHARMe will not enable• “Give me the best dataset on sea surface temperature”

– The “best” dataset depends on the application

• CHARMe will not provide a new “quality stamp” for datasets– But will be able to link to such things if other people publish

them, e.g. Bates maturity index, QA4EO certification

• CHARMe will not provide access to actual data– (but will enable discovery of data)

• Not planning to create (another) “one-stop shop” for information– We want the information to appear where users are already

looking

Beware of 5-star ratings…

(words of wisdom from xkcd.com)

TERRIBLE

How will you use CHARMe?

• Search for datasets on existing data provider website• Use the “CHARMe button” to view or record

commentary on a particular dataset

“Significant events” viewer

• Google Finance (above) matches stock prices with news events• CHARMe tool will match climate reanalysis data with “significant events”

– Algorithm changes, instrument failures, new data sources• Will allow user annotations on the data and events• Will be available on the Web

Fine-grained commentary

• Will allow creation and discovery about specific parts of datasets

• E.g. variables, geographic locations, time ranges• cf http://maphub.github.io

• Using Linked Data to combine EO and socioeconomic data in 8 application areas• 16-partner EU FP7 project, just started!

– Coordinated from Reading

Jon BlowerDebbie CliffordTristan QuaifeSam Williams

Summary• Linked Data can join up information

from all over the Web

• Makes disparate data sources discoverable and processable

• CHARMe is using Linked Data techniques to help users of climate data to connect with all the experience in the community– Techniques are not specific to

climate data

• MELODIES project will combine environmental data and socioeconomic data to develop real-world services using Linked Data

Thank you!

j.d.blower@reading.ac.uk@Jon_Blower

http://www.charme.org.uk@MelodiesProject