Post on 23-Feb-2016
description
LOD 를 말하다 !
2014.6.27
김 우 주
연세대학교 정보산업공학과
LOD 를 말하다 . LOD 를 나누다 (Linked Data Party 5)
2
목차I. 빅데이터 시대와 정보의 홍수
II. 빅데이터 활용 사례
III. 빅데이터의 한계와 극복 방안
IV. Linked Data 의 구축과 활용
V. LOD 2 - 시맨틱 기술의 미래
LOD 를 말하다 !
An Instrumented Interconnected World 빅데이터 시대와정보의 홍수
2+ bil-lion
people on the
Web by end 2011
30 billion RFID tags today
(1.3B in 2005)
4.6 bil-lion
camera phones
world wide
100s of millions of GPS
enabled devices sold an-
nually
76 million smart meters in 2009… 200M by 2014
12+ TBs of tweet data
every day
25+ TBs of
log data every day
? TB
s of
data
eve
ry
day
5
Information Overflow on the Web
Growth of the Web The amount of information available on the Web grows so
fast. The February 2014 survey shows there exist at least
920,120,079 sites (http://news.netcraft.com/archives/category/web-server-survey/).
빅데이터 시대와정보의 홍수
6
Information Overflow on the Web
The Indexed Web contains at least 19.8 billion pages (Sunday, 02 March, 2014). http://www.worldwidewebsize.com/
빅데이터 시대와정보의 홍수
7
빅데이터란 ?
빅데이터란 ? (07/11/2013, European Commission) Every minute the world generates 1.7 million billion bytes
of data, equivalent to 360,000 standard DVDs. The big data sector is growing at a rate of 40% a year.
무엇이 빅데이터를 중요하게 하는가 ? Big data is already affecting all areas of the economy. Data-driven decision making leads to 5-6% efficiency
gains in the different sectors observed. Intelligent processing of data is also essential for
addressing societal challenges.
빅데이터 시대와정보의 홍수
8
IBM 의 예측 : 2014 년 6 대 빅데이터 트렌드 직감보다는 더 분석적인 경영 방식
Companies will grow increasingly data driven and willing to apply analytics-derived insights to key business operations.
빅데이터 프라이버시와 보안 문제
Organizations will make a greater effort to build security, privacy, and governance policies into their big data processes.
빅데이터에 대한 투자 확대
CDO(Chief Data Officer) 의 등장
More organizations will bring a chief data officer (CDO) on board.
보다 유용한 빅데이터 응용 시스템
외부 데이터에 대한 관심 증대
빅데이터 시대와정보의 홍수
10
구글의 독감 트렌드 ‘ 독감’ 관련 검색어 분석을 통한 독감 예보 가능성 확인
구글 검색 사이트에 사용자가 남긴 검색어의 빈도를 조사 , 독감 환자의 분포 및 확산
정보 제공
빅데이터 활용 사례
11
샌프란시스코 , 범죄 예방 시스템 과거 범죄 발생 지역과 시각 패턴 분석을 통한 경찰 인력 배치
과거 발생한 범죄 패턴을 분석하여 후속 범죄 가능성 예측
과거 데이터에서 범죄자 행동을 분석하여 사건 예방을 위한 해법 제시
빅데이터 활용 사례
12
미국 국세청 , 탈세 방지 시스템 빅데이터 분석을 통한 탈세 및 사기 범죄 예방 시스템 구축
사기 방지 솔루션 , 소셜 네트워크 분석 , 데이터 통합 및 마이닝 등 활용
세금 누락 및 불필요한 세금 환급 절감의 효과 발생
빅데이터 활용 사례
13
KT, 서울특별시 – 빅데이터 기반심야버스 노선 정책 지원 심야버스 노선 결정을 위한 유동인구 분석 및 노선 분석
서울시의 교통 환경 ( 정류장 / 전용차로 / 환승 ) 기반 지역별 최적 정류장
위치를 도출하고 KT 의 CDR 데이터 기반 심야시간 유동인구 및 목적지
통계를 융합하여 노선 검증
빅데이터 활용 사례
14
비씨카드 , 점포 평가 서비스 소상공인 창업 성공률 제고를 위한 상가데이터 및 신용카드거래데이터 기반의
빅데이터 분석
점포이력 , 상권분석 , 업종추천 등이 이루어지는 과거현황분석 , 추천 업종
또는 사용자 선택 업종 매출예측 , 수익예측 등의 서비스 제공
빅데이터 활용 사례
16
Information Overflow Problems
Problems How to cover all available information? - Recall How to find the relevant information? - Precision
빅데이터의 한계와극복 방안
Not data (search), but integration, analysis and insight, leading to deci-
sions and discovery
Information Silo Problem
Stove-piped Systems and Poor Content Aggregation
빅데이터의 한계와극복 방안
19
Semantic Interoperability
To cope with the problems mentioned in the preceding slide, we need Semantic Interoperability.
Semantics “The meaning or the interpretation of a word,
sentence, or other language form.”
What is Semantic Interoperability? “Processing or Integration of resources based on the
understanding what’s intended or expressed by other systems or parties.’’
빅데이터의 한계와극복 방안
What if I want to ...
Move my content from one place to another? RSS ? Not enough
Aggregate my data An open FriendFeed?
Re-use my Flickr friends on Twitter? Invite. Again and again ...
The Semantic Web and Ontology can help ! By providing a common framework to interlink data
from various providers in an open way.
21
빅데이터의 한계와극복 방안
22
How is it Possible?
Ontology: Agreement with Common Vocabulary & Domain Knowledge
Semantic Annotation: metadata (manual & automatic metadata extraction)
Reasoning: semantics enabled search, integration, analysis, mining, discovery
빅데이터의 한계와극복 방안
24
Three Technical Building Block
Basic Building Block URIs for unambiguous names for resources, RDF for common data model for expressing metadata, Ontology(OWL) for common vocabularies.
Semantic Web becomes: web of data/things/concepts
• What is a Thing/Concept? It can be anything in the world - a movie, a person, a disease, a location…
• Machines will be able to understand the concept behind a html page. • This page is talking about ‘Barack Obama’, He is a ‘Person’ and he is
the ‘President of USA’ ?
빅데이터의 한계와극복 방안
25
Who borrows this Idea?
Facebook Facebook Open Graph Protocol and Graph Search
Google Knowledge Graph
Twitter Real-time Semantic Web with Twitter Annotations
빅데이터의 한계와극복 방안
Linked Data
Building a “Web of Data” to enhance the current WebThe Linking Open Data (LOD) project:
http://linkeddata.org/ Translating existing datasets into RDF and linking them
together.• For example, DBpedia (Wikipedia) and GeoNames, Freebase, BBC
programmes, etc. Government data also available as Linked Data
• DATA.gov• DATA.gov.uk
27
Linked Data 의 구축과 활용
Web of Data (Statistics)
The size of the Web of Data The size of the Web of Data can be estimated based on
the data set statistics that are collected by the LOD community in the ESW wiki.
According to these statistics, the Web of Data currently consists of 31 billion RDF triples, which are interlinked by around 500 million RDF inter-links (09/19/2011).
31
Linked Data 의 구축과 활용
Semantic Search Engines
Top 7 Semantic Search Engines as An Alternative to Google Kngine Hakia Kosmix: now is part of @WalmartLabs DuckDuckGo Evri: specialized for iPad and iPhone Powerset: now is part of Bing Truevert: focus only on environmental concerns.
33
Linked Data 의 구축과 활용
35
LOD2 : What is LOD2?
LOD2(Linked Open Data) LOD2 is the large-scale integrating project co-funded by
the European Commission within the FP7 Information and Communication Technologies Work Programme.• Started in September 2010
Partners• 14 partners (11 European Country)
LOD 2 - 시맨틱 기술의 미래
36
LOD2 : Objectives of LOD2
LOD2 Project Objectives Achieving visualization, deployment, sharing, accessibility
for linked open data by software technology.• Increase visibility of Linked Data activities [Visualization]• Support deployment Linked Data components [Deployment]• Improve information sharing between Linked Data
components so that publishing Linked Data is eased. [Sharing]• Improve access to the content: the online Linked Open Data
[Accessibility]• Improve the software technology which support it [By software
technology]
LOD 2 - 시맨틱 기술의 미래
LOD2 Stack : Overview
LOD2 Stack LOD2 project provides LOD2
Stack for the sake of easy access to linked data software.
the LOD2 software stack is an integrated distribution of aligned tools supporting the life-cycle of Linked Data from extraction, authoring/creation over enrichment, interlinking, fusing to visualization and maintenance
37
LOD 2 - 시맨틱 기술의 미래
39
LOD2 Stack : The overview of tools
Apache Stanbol In the LOD2 Stack, Apache Stanbol can be used for
NLP services which rely on the stack internal knowledge bases, such as named entity recognition and text classification.
CubeViz CubeViz is a facetted browser for statistical data
utilizing the RDF Data Cube vocabulary which is the state-of-the-art in representing statistical data in RDF.
LOD 2 - 시맨틱 기술의 미래
40
LOD2 Stack : The overview of tools
Dbpedia Spotlight DBpedia Spotlight is a tool for automatically
annotating mentions of DBpedia resources in text, providing a solution for linking unstructured information sources to the Linked Open Data cloud through DBpedia.
D2RQ D2RQ is a system for accessing relational
databases(RDBMS) as virtual RDF graphs.
LOD 2 - 시맨틱 기술의 미래
41
LOD2 Stack : The overview of tools
DL-Learner The DL-Learner software learns concepts in
Description Logics (DLs) from user-provided examples. (Supervised-learning)
ORE The ORE (Ontology Repair and Enrichment) tool allows
for knowledge engineers to improve an OWL ontology by fixing inconsistencies and making suggestions for adding further axioms to it.
LOD 2 - 시맨틱 기술의 미래
42
LOD2 Stack : The overview of tools
Poolparty The PoolParty Extractor (PPX) offers an API
providing text mining algorithms based on semantic knowledge models.
LOD 2 - 시맨틱 기술의 미래
43
LOD2 Stack : The overview of tools
SemMap SemMap allows to visualize knowledge bases having a
spatial dimension.
Silk The Silk Link Discovery Framework supports data publishers
in accomplishing the second task. Using the declarative Silk - Link Specification Language (Silk-LSL), developers can specify which types of RDF links should be discovered between data sources as well as which conditions data items must fulfill in order to be interlinked.
LOD 2 - 시맨틱 기술의 미래
44
LOD2 Stack : The overview of tools
Sieve Sieve allows Web data to be filtered according to
different data quality assessment policies and provides for fusing Web data according to different conflict resolution methods.
LIMES LIMES is a link discovery framework for the Web of Data.
It implements time-efficient approaches for large-scale link discovery based on the characteristics of metric spaces.
LOD 2 - 시맨틱 기술의 미래
45
Silk : Link Discovery Framework
Interlinking and Fusion Stage Component of LOD2 Stack Can be used by data providers to generate RDF links
between data sets on the web of data• Especially, to set explicit RDF links between data items
within different data sources
“Data publishers can use Silk to set RDF links from their data sources to other data sources on the Web”
LOD 2 - 시맨틱 기술의 미래
46
Silk : Silk – Link Specification Language Example
Aggregation Example: Combines multiple confidence values into a single
value (average)
LOD 2 - 시맨틱 기술의 미래
Confidence value is the aver-age of two compared weight
Numeric differences between parameters
DL-Learner
Introduction The goal of DL-Learner is to provide a DL/OWL based
machine learning tool to solve supervised learning tasks.
The DL-Learner software learns concepts in Description Logics (DLs) from examples.
LOD 2 - 시맨틱 기술의 미래
DL-Learner : Features
Learning Problems Positive and Negative Examples (=previous example) Class Learning
• Find out Class Expression for given class• father
LOD 2 - 시맨틱 기술의 미래
50
SWCL - Sample Example LOD 2 - 시맨틱 기술의 미래
Country
Province
hasPart
positiveInteger
positiveInteger
PopulationValue
PopulationValue
?
51
Constraints Representation in SWCL
Target Constraint
Corresponding SWCL Code
LOD 2 - 시맨틱 기술의 미래
<swcl:Constraint rdf:ID=”numberOfPopulation"> <swcl:qualifier> <swcl:Variable rdf:id="y">
<swcl:bindingClass rdf:resource="#Country"/> </swcl:Variable> </swcl:qualifier>
<swcl:hasLHS> <swcl:TermBlock rdf:ID="termBlock_1"> <swcl:sign rdf:resource="&swcl;plus"/>
<swcl:aggregateOperator rdf:resource="&swcl;Sigma"/> <swcl:parameter> <swcl:Variable rdf:id="x">
<rdfs:subClassOf> <owl:Restriction> <owl:onProperty rdf:resource="#partOf"/>
<owl:hasValue rdf:resource="#y"/> </owl:Restriction> </rdfs:subClassOf>
</swcl:Variable> </swcl:parameter> <swcl:factor>
<swcl:FactorAtom> <swcl:bindingClass rdf:resource="#x"/> <swcl:bindingDatatypeProperty rdf:resource="#populationValue"/>
<swcl:FactorAtorm> </swcl:factor> </swcl:TermBlock>
</swcl:hasLHS> <swcl:hasOperator rdf:resource="&swcl;equal"/> <swcl:hasRHS>
<swcl:TermBlock rdf:ID="termBlock_2"> <swcl:sign rdf:resource="&swcl;plus"/> <swcl:factor>
<swcl:FactorAtom> <swcl:bindingClass rdf:resource="#y"/> <swcl:bindingDatatypeProperty rdf:resource="#populationValue"/>
</swcl:FactorAtom> </swcl:factor> </swcl:TermBlock>
</swcl:hasRHS> </swcl:Constraint>
Our Direction to the Future
Directions Open, Share your data, whenever and wherever you
want Semantic, Enhance your data, to make more sense of it
An example: LinkedGeoData.org We need an integrated framework to enhance
communication and information sharing in GeoData.
52
LOD 2 - 시맨틱 기술의 미래