UFMG, June 2002BYU Data Extraction Group Automating Schema Matching for Data Integration David W....

UFMG, June 2002BYU Data Extraction Group

Automating Schema Matchingfor Data Integration

David W. Embley

Brigham Young University

Funded by NSF

Information ExchangeSource Target

InformationExtraction

SchemaMatching

Leveragethis …

… to dothis

Presentation Outline

• Information Extraction

• Schema Matching for HTML Table

• Direct Schema Matching

• Indirect Schema Matching

• Conclusions and Future Work

Information Extraction

Extracting Pertinent Information from Documents

A Conceptual Modeling SolutionYear Price

Make Mileage

Feature

PhoneNr

Extension

hashas

has is for

1..* 1..*

0..1 0..10..1

Car-Ads OntologyCar [->object];Car [0..1] has Year [1..*];Car [0..1] has Make [1..*];Car [0...1] has Model [1..*];Car [0..1] has Mileage [1..*];Car [0..*] has Feature [1..*];Car [0..1] has Price [1..*];PhoneNr [1..*] is for Car [0..*];PhoneNr [0..1] has Extension [1..*];Year matches [4]

constant {extract “\d{2}”; context "([^\$\d]|^)[4-9]\d,[^\d]"; substitute "^" -> "19"; }, …End;

Recognition and Extraction

Car Year Make Model Mileage Price PhoneNr0001 1989 Subaru SW $1900 (363)835-85970002 1998 Elandra (336)526-54440003 1994 HONDA ACCORD EX 100K (336)526-1081

Car Feature0001 Auto0001 AC0002 Black0002 4 door0002 tinted windows0002 Auto0002 pb0002 ps0002 cruise0002 am/fm0002 cassette stero0002 a/c0003 Auto0003 jade green0003 gold

Schema Matching forHTML Tables

Table-Schema Matching(Basic Idea)

• Many tables on the Web• Ontology-Based Extraction:

– Works well for unstructured or semistructured data– What about structured data – tables?

• Method:– Form attribute-value pairs– Do extraction– Infer mappings from extraction patterns

Problem: Different Schemas

Target Database Schema{Car, Year, Make, Model, Mileage, Price, PhoneNr}, {PhoneNr, Extension}, {Car, Feature}

Different Source Table Schemas– {Run #, Yr, Make, Model, Tran, Color, Dr}– {Make, Model, Year, Colour, Price, Auto, Air Cond.,

AM/FM, CD}– {Vehicle, Distance, Price, Mileage}– {Year, Make, Model, Trim, Invoice/Retail, Engine,

Fuel Economy}

Problem: Attribute is Value

Problem: Attribute-Value is Value

Problem: Value is not Value

Problem: Implied Values

Problem: Missing Attributes

Problem: Compound Attributes

Problem: Merged Values

Problem: Values not of Interest

Problem: Factored Values

Problem: Split Values

Problem: Information Behind Links

Single-ColumnTable (formateda list)

Tableextendingover severalpages

Solution

• Form attribute-value pairs (adjust if necessary)

• Do extraction

• Infer mappings from extraction patterns

Solution: Remove Internal Factoring

Discover Nesting: Make, (Model, (Year, Colour, Price, Auto, Air Cond, AM/FM, CD)*)*

Unnest: (Model, Year, Colour, Price, Auto, Air Cond, AM/FM, CD)*(Year, Colour, Price, Auto, Air Cond, AM/FM, CD)*Table

Legend

Solution: Replace Boolean Values

Legend

CD Table

Yes,Yes,AutoAir CondAM/FM

Air Cond.

Solution: Form Attribute-Value Pairs

Legend

Air Cond.

<Make, Honda>, <Model, Civic EX>, <Year, 1995>, <Colour, White>, <Price, $6300>, <Auto, Auto>, <Air Cond., Air Cond.>, <AM/FM, AM/FM>

Solution: Adjust Attribute-Value Pairs

Legend

Air Cond.

<Make, Honda>, <Model, Civic EX>, <Year, 1995>, <Colour, White>, <Price, $6300>, <Auto>, <Air Cond>, <AM/FM>

Solution: Do Extraction

Legend

Air Cond.

<Make, Honda>, <Model, Civic EX>, <Year, 1995>, <Colour, White>, <Price, $6300>, <Auto>, <Air Cond>, <AM/FM>

Solution: Infer Mappings

Legend

Air Cond.

{Car, Year, Make, Model, Mileage, Price, PhoneNr}, {PhoneNr, Extension}, {Car, Feature}

Each row is a car. Model(Year, Colour, Price, Auto, Air Cond, AM/FM, CD)*TableMake(Model, Year, Colour, Price, Auto, Air Cond, AM/FM, CD)*(Year, Colour, Price, Auto, Air Cond, AM/FM, CD)*TableYearTable

Note: Mappings produce sets for attributes. Joining to form recordsis trivial because we have OIDs for table rows (e.g. for each Car).

Legend

Air Cond.

Model(Year, Colour, Price, Auto, Air Cond, AM/FM, CD)*Table

Legend

Air Cond.

PriceTable

Legend

Air Cond.

Yes,ColourFeatureColourTable AutoFeatureAutoAutoTable Air Cond.FeatureAir Cond.

Air Cond.Table AM/FMFeatureAM/FMAM/FMTable CDFeatureCDCDTableYes, Yes, Yes,

Experiment

• Tables from 60 sites• 10 “training” tables• 50 test tables• 357 mappings (from all 60 sites)

– 172 direct mappings (same attribute and meaning)– 185 indirect mappings (29 attribute synonyms, 5 “Yes/No”

columns, 68 unions over columns for Feature, 19 factored values, and 89 columns of merged values that needed to be split)

Results• 10 “training” tables

– 100% of the 57 mappings (no false mappings)– 94.6% of the values in linked pages (5.4% false declarations)

• 50 test tables– 94.7% of the 300 mappings (no false mappings)– On the bases of sampling 3,000 values in linked pages, we obtained 97%

recall and 86% precision

• 16 missed mappings– 4 partial (not all unions included)– 6 non-U.S. car-ads (unrecognized makes and models)– 2 U.S. unrecognized makes and models– 3 prices (missing $ or found MSRP instead)– 1 mileage (mileages less than 1,000)

Direct Schema Matching

Attribute Matchingfor Populated Schemas

• Central Idea: Exploit All Data & Metadata

• Matching Possibilities (Facets)– Attribute Names– Data-Value Characteristics– Expected Data Values– Data-Dictionary Information– Structural Properties

Approach

• Target Schema T

• Source Schema S

• Framework– Individual Facet Matching– Combining Facets– Best-First Match Iteration

Example

Source Schema S

has0:1

Modelhas

has0:1

Feature

0:*Cost

has0:1

Mileage

0:10:1

Modelhas

Target Schema T

has0:1

Make YearMake

ModelCar

Mileage Miles

Individual Facet Matching

• Attribute Names

• Data-Value Characteristics

• Expected Data Values

Attribute Names

• Target and Source Attributes – T : A – S : B

• WordNet• C4.5 Decision Tree: feature selection, trained on

schemas in DB books– f0: same word– f1: synonym– f2: sum of distances to a common hypernym root– f3: number of different common hypernym roots– f4: sum of the number of senses of A and B

WordNet Rule

The number

of different common

hypernym roots of A

The sum of distances of A and B to a

common hypernym

The sum of the

number of senses of A and B

Confidence Measures

Data-Value Characteristics

• C4.5 Decision Tree

• Features– Numeric data

(Mean, variation, standard deviation, …)

– Alphanumeric data(String length, numeric ratio, space ratio)

Confidence Measures

Expected Data Values

• Target Schema T and Source Schema S– Regular expression recognizer for attribute A in T

– Data instances for attribute B in S

• Hit Ratio = N’/N for (A, B) match– N’ : number of B data instances recognized by the

regular expressions of A

– N: number of B data instances

Confidence Measures

Combined Measures

Threshold: 0.5

000000

0 0 0 0 0 01

0 0 0 0

0 0 0 000000

0 010 00

Final Confidence Measures

Experimental Results

• This schema, plus 6 other schemas– 32 matched attributes– 376 unmatched attributes

• Matched: 100%

• Unmatched: 99.5%– “Feature” ---”Color”– “Feature” ---”Body Type”

F193.8%

F198.9%

F297.9%

F398.4%

F1: WordNetF2: Value CharacteristicsF3: Expected Values

Indirect Schema Matching

Schema Matching

Source

YearFeature

Target

MilesMileage

Make Make&

Body Type

Mapping Generation

• Direct Matches as described earlier:– Attribute Names based on WordNet– Value Characteristics based on value lengths, averages, …– Expected Values based on regular-expression recognizers

• Indirect Matches:– Direct matches– Structure Evaluation

• Union• Selection• Decomposition• Composition

Union and Selection

Source

YearFeature

Target

MilesMileage

Make Make&

Body Type

Decomposition and Composition

Source

YearFeature

Target

MilesMileage

Make Make&

Body Type

Structure

POShipTo POBillTo POLines

City Street City Street Item

Line Qty UoM

PurchaseOrder

DeliverToInvoiceTo

ItemItemCount

ItemNumber

Quantity UnitOfMeasure

City Street

Address

Target Source

Example Taken From [MBR, VLDB’01]

Structure(Nonlexical Matches)

POShipTo POBillTo POLines

Line Qty UoM

PurchaseOrder

DeliverToInvoiceTo

ItemCount

ItemNumber

City Street

Address

DeliverTo

Target Source

Structure(Join over FD Relationship Sets, …)

POBillTo POLines

Line Qty UoM

PurchaseOrder

InvoiceTo

ItemCount

ItemNumber

Street City

Street

POShipTo DeliverTo

Target Source

Structure(Lexical Matches)

POBillTo POLines

Line Qty UoM

PurchaseOrder

InvoiceTo

ItemCount

ItemNumber

Quantity

Street City

StreetCity

StreetStreet

Street

StreetCount

Line QtyQuantity UnitOfMeasure

POShipTo DeliverTo

Target Source

Experiments• Methodology• Measures

– Precision

– Recall

– F Measure

falseposcorrect

correctprecision

falsenegcorrect

correctrecall

recallprecsion

ResultsApplications

(Number of Schemes)

Precision

Recall

Correct False

Positive

Negative

Course Schedule (5) 98 93 96 119 2 9

Faculty Member (5) 100 100 100 140 0 0

Real Estate (5) 92 96 94 235 20 10

Data borrowed from Univ. of Washington [DDH, SIGMOD01]

Indirect Matches: 94% (precision, recall, F-measure)

Rough Comparison with U of W Results (Direct Matches only)

* Course Schedule – Accuracy: ~71%

* Faculty Members – Accuracy, ~92%

* Real Estate (2 tests) – Accuracy: ~75%

Conclusions and Future Work

Conclusions

• Table Mappings– Tables: 94.7% (Recall); 100% (Precision)– Linked Text: ~97% (Recall); ~86% (Precision)

• Direct Attribute Matching– Matched 32 of 32: 100% Recall– 2 False Positives: 94% Precision

• Direct and Indirect Attribute Matching– Matched 494 of 513: 96% Recall– 22 False Positives: 96% Precision

www.deg.byu.edu

Current & Future Work:Improve and Extend Indirect Matching

• Improve Object-Set Matching (e.g. Lex/non-Lex) • Add Relationship-Set Matching• Computations

Current & Future Work:Tables Behind Forms

• Crawling the Hidden Web

• Filling in Forms from Global Queries

Current & Future Work:Developing Extraction Ontologies

• Creation from Knowledge Sources and Sample Application Pages K Ontology + Data Frames, Lexicons, …– RDF Ontologies

• User Creation by Example

Current & Future Work:and Much More …

• Table Understanding• Microfilm Census Records• Generate Ontologies by Reading Tables• …

www.deg.byu.edu

UFMG, June 2002BYU Data Extraction Group Automating Schema Matching for Data Integration David W....

Documents

Transcript of UFMG, June 2002BYU Data Extraction Group Automating Schema Matching for Data Integration David W....

OntoSoar: Soar Finds Facts in Text Peter Lindes, Deryle Lonsdale, David Embley Brigham Young University 33 rd Soar Workshop, June 2013 pl 6/6/201333rd.

Justice XML Data Model Paul Embley 08 August 2003.

Ufmg 2013 Ufmg Engenheiro Civil Prova

Scott N. Woodfield David W. Embley Stephen W. Liddle Brigham Young University.

Copyright © 2006, Brigham S. Anderson Active Learning as Active Inference Brigham S. Anderson brigham brigham@cmu.edu School of Computer.

Semantic Understanding An Approach Based on Information-Extraction Ontologies David W. Embley Brigham Young University.

Resolving Under Constrained and Over Constrained Systems of Conjunctive Constraints for Service Requests Muhammed J. Al-Muhammed David W. Embley Brigham.

Soar and Construction Grammar Peter Lindes, Deryle Lonsdale, David Embley Brigham Young University 2014 Soar Workshop © 2014 Peter Lindes 6/19/2014PL 2014.

Producing XML Documents with Guaranteed “Good” Properties David W. Embley Brigham Young University Wai Y. Mok University of Alabama in Huntsville Sponsored.

DASFAA 2003BYU Data Extraction Group Discovering Direct and Indirect Matches for Schema Elements Li Xu and David W. Embley Brigham Young University Funded.

A Web of Knowledge for Historical Documents David W. Embley.

Boletim Informativo do Sistema de Bibliotecas da UFMG ... · Conexão.Biblioteca O, UFMG. Sistema de Bibliotecas da UFMG

BIG DATA Conceptual Modeling to the Rescue David W. Embley with special thanks to Stephen W. Liddle and the Data Extraction Research Group at Brigham Young.

Extracting and Structuring Web Data David W. Embley Department of Computer Science Brigham Young University D.M. Campbell, Y.S. Jiang, Y.-K. Ng, R.D. Smith.

David W. Embley , Stephen W. Liddle , & Deryle W. Lonsdale Brigham Young University, USA

Embley Park Historic Property

TANGO Table ANalysis for Generating Ontologies Yuri A. Tijerino*, David W. Embley*, Deryle W. Lonsdale* and George Nagy** * Brigham Young University **

David W. Embley Brigham Young University. ACM-L-2010 WORKSHOP DescriptionDescription / Problem / Topics / Capability / Application / Status / Deadlines.

RECOMMENDATION REPORT South of Embley Bauxite ......RECOMMENDATION REPORT South of Embley Bauxite Mine and Port Development Project, Cape York, QLD (EPBC 2010/5642) 1. Recommendation

Finding Genealogy Facts with Linguistic Analysis Peter Lindes, Deryle W. Lonsdale, David W. Embley Brigham Young University © 2014 Peter Lindes 3/19/2014PL.

TANGO Table ANalysis for Generating Ontologies Yuri A. Tijerino, David W. Embley, Deryle W. Lonsdale* and George Nagy** * Brigham Young University **