Summarization – CS 257 Chapter – 21 (Information Integration) Database Systems: The Complete...
-
Upload
rodger-davidson -
Category
Documents
-
view
218 -
download
1
Transcript of Summarization – CS 257 Chapter – 21 (Information Integration) Database Systems: The Complete...
Summarization – CS 257 Chapter – 21 (Information Integration)
Database Systems: The Complete Book
Submitted by:Deepti Kundu
Submitted to: Dr.T.Y.Lin
21.1 Introduction to Information 21.1 Introduction to Information IntegrationIntegration
Need for Information Integration All the data in the world could put in a single database
(ideal database system) In the real world (impossible for a single database):
databases are created independently hard to design a database to support future use
University DatabaseUniversity Database
Registrar: to record student and grade Bursar: to record tuition payments by students Human Resources Department: to record employees Other department….
InconvenientInconvenient
Record grades for students who pay tuition Want to swim in SJSU aquatic center for free in
summer vacation?
(all the cases above cannot achieve the function by a single database)
Solution: one database
How to integrateHow to integrate
Start over
build one database: contains all the legacy databases; rewrite all the applications
result: painful Build a layer of abstraction (middleware)
on top of all the legacy databases
this layer is often defined by a collection of classes
BUT…
Heterogeneity ProblemHeterogeneity Problem
What is Heterogeneity Problem
Aardvark Automobile Co.
1000 dealers has 1000 databases
to find a model at another dealer
can we use this command:
SELECT * FROM CARS
WHERE MODEL=“A6”;
Type of HeterogeneityType of Heterogeneity
Communication Heterogeneity Query-Language Heterogeneity Schema Heterogeneity Data type difference Value Heterogeneity Semantic Heterogeneity
21.2 Modes of Information Integration21.2 Modes of Information Integration
Federations The simplest architecture for integrating several DBs One to one connections between all pairs of DBs n DBs talk to each other, n(n-1) wrappers are needed Good when communications between DBs are limited
Wrapper a software translates incoming queries and outgoing
answers. In a result, it allows information sources to conform to some shared schema.
Federations DiagramFederations Diagram
DB2DB1
DB3 DB4
2 Wrappers
2 Wrappers
2 Wrappers
2 Wrappers
2 Wrappers 2 Wrappers
A federated collection of 4 DBs needs 12 components to translate queries from one to another.
Data WarehouseData Warehouse
Sources are translated from their local schema to a global schema and copied to a central DB.
User transparent: user uses Data Warehouse just like an ordinary DB
User is not allowed to update Data Warehouse
Warehouse DiagramWarehouse Diagram
Warehouse
Extractor Extractor
Source 1 Source 2
User query
result
Combiner
Construct Data WarehouseConstruct Data Warehouse There are mainly 3 ways to constructing the data in the warehouse:
1) Periodically reconstructed from the current data in the sources, once a night or at even longer intervals.Advantages:
simple algorithms. Disadvantages:
need to shut down the warehouse; data can become out of date.
2) Updated periodically based on the changes (i.e. each night) of the sources. Advantages:
involve smaller amounts of data. (important when warehouse is large and needs to be modified in a short period)
Disadvantages: the process to calculate changes to the warehouse is complex. data can become out of date.
3) Changed immediately, in response to each change or a small set of changes at one or more of the sources.
Advantages: data won’t become out of date.
Disadvantages: requires too much communication, therefore, it is generally too
expensive. (practical for warehouses whose underlying sources changes slowly.)
MediatorsMediators
Virtual warehouse, which supports a virtual view or a collection of views, that integrates several sources.
Mediator doesn’t store any data. Mediators’ tasks: 1)receive user’s query, 2)send queries to wrappers, 3)combine results from wrappers, 4)send the final result to user.
A Mediator diagramA Mediator diagram
Mediator
Wrapper Wrapper
Source 1 Source 2
User query
Query
Query
QueryQuery
Result
Result
Result
Result
Result
21.3 Wrappers in Mediator-Based 21.3 Wrappers in Mediator-Based SystemsSystems
Intro Templates for Query patterns Wrapper Generator Filter
Wrappers in Mediator-based SystemsWrappers in Mediator-based Systems
More complicated than that in most data warehouse system.
Able to accept a variety of queries from the mediator and translate them to the terms of the source.
Communicate the result to the mediator. How to design a wrapper?
Classify the possible queries that the mediator can ask into templates, which are queries with parameters that represent constants.
Wrapper GeneratorsWrapper Generators
Templates for Query Patterns: Use notation T=>S to express the idea that the template T
is turned by the wrapper into the source query S.
The wrapper generator creates a table holds the various query patterns contained in the templates.
The source queries that are associated with each. FilterFilter
Have a wrapper filter to supporting more queries.
A driver is used in each wrapper, the task of the driver is to:
Accept a query from the mediator. Search the table for a template that matches the
query. The source query is sent to the source, again using a
“plug-in” communication mechanism. The response is processed by the wrapper.
21.4 Capability Based Optimization21.4 Capability Based Optimization
IntroductionTypical DBMS estimates the cost of each query
plan and picks what it believes to be the bestMediator – has knowledge of how long its sources
will take to answerOptimization of mediator queries cannot rely on
cost measure alone to select a query planOptimization by mediator follows capability based
optimization
21.4.1 The Problem of Limited Source 21.4.1 The Problem of Limited Source Capabilities Capabilities
Many sources have only Web Based interfaces Web sources usually allow querying through a query
form E.g. Amazon.com interface allows us to query about
books in many different ways. But we cannot ask questions that are too general
E.g. Select * from books;
(con’t)(con’t) Reasons why a source may limit the ways in which
queries can be asked Earliest database did not use relational DBMS
that supports SQL queries Indexes on large database may make certain
queries feasible, while others are too expensive to execute
Security reasons
21.4.2 A Notation for Describing 21.4.2 A Notation for Describing Source Capabilities Source Capabilities
For relational data, the legal forms of queries are described by adornments
Adornments – Sequences of codes that represent the requirements for the attributes of the relation, in their standard order f(free) – attribute can be specified or not b(bound) – must specify a value for an attribute but any
value is allowed u(unspecified) – not permitted to specify a value for a
attribute
(cont’d)(cont’d)
c[S](choice from set S) means that a value must be specified and value must be from finite set S.
o[S](optional from set S) means either do not specify a value or we specify a value from finite set S
A prime (f’) specifies that an attribute is not a part of the output of the query
A capabilities specification is a set of adornments A query must match one of the adornments in its capabilities
specification
21.4.3 Capability-Based Query-Plan 21.4.3 Capability-Based Query-Plan Selection Selection
Given a query at the mediator, a capability based query optimizer first considers what queries it can ask at the sources to help answer the query
The process is repeated until: Enough queries are asked at the sources to resolve all the
conditions of the mediator query and therefore query is answered. Such a plan is called feasible.
We can construct no more valid forms of source queries, yet still cannot answer the mediator query. It has been an impossible query.
(cont’d)(cont’d)
The simplest form of mediator query where we need to apply the above strategy is join relations
E.g we have sources for dealer 2 Autos(serial, model, color) Options(serial, option)
Suppose that ubf is the sole adornment for Auto and Options have two adornments, bu and uc[autoTrans, navi]
Query is – find the serial numbers and colors of Gobi models with a navigation system
21.4.4 Adding Cost-Based 21.4.4 Adding Cost-Based OptimizationOptimization Mediator’s Query optimizer is not done when the capabilities
of the sources are examined Having found feasible plans, it must choose among them Making an intelligent, cost based query optimization requires
that the mediator knows a great deal about the costs of queries involved
Sources are independent of the mediator, so it is difficult to estimate the cost
21.5 Optimizing Mediator Queries21.5 Optimizing Mediator Queries
Chain algorithm – a greed algorithm that finds a way to answer the query by sending a sequence of requests to its sources. Will always find a solution assuming at least one solution
exists. The solution may not be optimal.
21.5.1 Simplified Adornment Notation21.5.1 Simplified Adornment Notation
A query at the mediator is limited to b (bound) and f (free) adornments.
We use the following convention for describing adornments: Nameadornments (attributes) where:
name is the name of the relation the number of adornments = the number of attributes
21.5.2 Obtaining Answers for 21.5.2 Obtaining Answers for Subgoals Subgoals
Rules for subgoals and sources: Suppose we have the following subgoal:
Rx1x2…xn(a1, a2, …, an),
and source adornments for R are: y1y2…yn. If yi is b or c[S], then xi = b. If xi = f, then yi is not output restricted.
The adornment on the subgoal matches the adornment at the source: If yi is f, u, or o[S] and xi is either b or f.
21.5.3 The Chain Algorithm21.5.3 The Chain Algorithm
Maintains 2 types of information: An adornment for each subgoal. A relation X that is the join of the relations for all the
subgoals that have been resolved. Initially, the adornment for a subgoal is b iff the mediator
query provides a constant binding for the corresponding argument of that subgoal.
Initially, X is a relation over no attributes, containing just an empty tuple.
(cont’d)(cont’d)
First, initialize adornments of subgoals and X. Then, repeatedly select a subgoal that can be resolved. Let
Rα(a1, a2, …, an) be the subgoal:
1. Wherever α has a b, we shall find the argument in R is a constant, or a variable in the schema of R. Project X onto its variables that appear in R.
(cont’d)(cont’d)2. For each tuple t in the project of X, issue a query to the
source as follows (β is a source adornment). If a component of β is b, then the corresponding
component of α is b, and we can use the corresponding component of t for source query.
If a component of β is c[S], and the corresponding component of t is in S, then the corresponding component of α is b, and we can use the corresponding component of t for the source query.
If a component of β is f, and the corresponding component of α is b, provide a constant value for source query.
(cont’d)(cont’d) If a component of β is u, then provide no binding for this
component in the source query. If a component of β is o[S], and the corresponding
component of α is f, then treat it as if it was a f. If a component of β is o[S], and the corresponding
component of α is b, then treat it as if it was c[S].
3. Every variable among a1, a2, …, an is now bound. For each remaining unresolved subgoal, change its adornment so any position holding one of these variables is b.
(cont’d)(cont’d)4. Replace X with X πs(R), where S is all of the variables
among: a1, a2, …, an.
5. Project out of X all components that correspond to variables that do not appear in the head or in any unresolved subgoal.
If every subgoal is resolved, then X is the answer. If every subgoal is not resolved, then the algorithm fails.
α
21.5.4 Incorporating Union Views at 21.5.4 Incorporating Union Views at the Mediator the Mediator
This implementation of the Chain Algorithm does not consider that several sources can contribute tuples to a relation.
If specific sources have tuples to contribute that other sources may not have, it adds complexity.
To resolve this, we can consult all sources, or make best efforts to return all the answers.
(cont’d)(cont’d)
Consulting All Sources We can only resolve a subgoal when each source for its
relation has an adornment matched by the current adornment of the subgoal.
Less practical because it makes queries harder to answer and impossible if any source is down.
Best Efforts We need only 1 source with a matching adornment to
resolve a subgoal. Need to modify chain algorithm to revisit each subgoal
when that subgoal has new bound requirements.
21.6 Local-as-View Mediators21.6 Local-as-View Mediators
GAV: Global as view mediators are like view, it doesn’t exist physically, but piece of it are constructed by the mediator by asking queries
LAV: Local as view mediators, defines the global predicates at the mediator, but we do not define these predicates as views of the source of data
Global expressions are defined for each source involving global predicates that describe the tuple that source is able to produce and queries are answered at mediator by discovering all possible ways to construct the query using the views provided by sources
Motivation for LAV MediatorsMotivation for LAV Mediators
LAV mediators help us to discover how and when to use that source in a given query
Example: Par(c,p)-> GAV of Par(c,p) gives information about the child and parent but does not give information of grandparents
LAV Par(c,p) will help to get information of chlid-parent and even grandparent
Terminology for LAV MediationTerminology for LAV Mediation
It is in form of logic that serves as the language for defining views.
Datalog is used which will remain common for the queries of mediator and source which is known as Conjunctive query.
LAV has global predicates which are the subgoals of mediator queries
Conjunctive queries defines the views which has unique view predicate and that view has Global predicates and associated with particular view.
Containment of Conjunctive QueriesContainment of Conjunctive Queries
Conjunctive query S be the solution to the mediator Q,
Expansion of S->E, produces same answers that Q produces, so, E subset Q.
A containment mapping from Q to E is function Γ(x) is the ith argument of the head E.
Add to Γ the rule that Γ(c) =c for any constant c. IF P(x1,x2,..xn) is a subgoal of Q, then P(Γ(x1), Γ(x2),.., Γ(xn)) is a subgoal of E.
Why Containment Mapping Test Works:Why Containment Mapping Test Works:
Questions:
1. If there is containment mapping, why must there be a containment of conjunctive queries?
2. If there is containment, why must there be a containment mapping?
Finding Solutions to a Mediator QueryFinding Solutions to a Mediator Query
Query Q, solutions S, Expansion E of S is contained in Q.
“If a query Q has n subgoals, then any answer produced by any solution is also produced by a solution that has at most n subgoals.
This is known by LMSS Theorem
Why the LMSS Theorem HoldsWhy the LMSS Theorem Holds
Query Q with n subgoals and S with n subgoals, E of S must be contained in query Q, E is expansion of Q.
S’ must be the solution got after removing all subgoals from S those are not the target of Q.
E subset or equal to Q and also E’ is the expansion of S’. So, S is subser of S’ : identity mapping. Thus there is no need for solution s among the solution S
among the solutions to query Q.
21.7 Entity Resolution21.7 Entity Resolution
Determining whether two records or tuples do or do not represent the same person, organization, place or other entity is called ENTITY RESOLUTION.
Deciding whether Records represent a Deciding whether Records represent a Common EntityCommon Entity
Two records represent the same individual if the two records have similar values for each of the fields associated with those records.
It is not sufficient that the values of corresponding fields be identical because of following reasons:
1. Misspellings2. Variant Names3. Misunderstanding of Names
4. Evolution of Values5. Abbreviations
Deciding Whether Records Represents Deciding Whether Records Represents a Common Entity - Edit Distancea Common Entity - Edit Distance First approach to measure the similarity of records is Edit
Distance.
Values that are strings can be compared by counting the number of insertions and deletions of characters it takes to turn one string into another.
So the records represent the same entity if their similarity measure is below a given threshold.
Deciding Whether Records Represents Deciding Whether Records Represents a Common Entity - Normalizationa Common Entity - Normalization
To normalize records by replacing certain substrings by others. For instance: we can use the table of abbreviations and replace abbreviations by what they normally stand for.
Once normalize we can use the edit distance to measure the difference between normalized values in the fields.
Merging Similar RecordsMerging Similar Records
Merging means replacing two records that are similar enough to merge and replace by one single record which contain information of both.
There are many merge rules:1. Set the field in which the records disagree to the
empty string.2. (i) Merge by taking the union of the values in each
field(ii) Declare two records similar if at least two of the three fields have a nonempty intersection.
Useful Properties of Similarity and Useful Properties of Similarity and Merge FunctionsMerge Functions
The following properties say that the merge operation is a semi lattice :
1. Idempotence : That is, the merge of a record with itself should surely be that record.
2. Commutativity : If we merge two records, the order in which we list them should not matter.
3. Associativity : The order in which we group records for a merger should not matter.
There are some other properties that we expect similarity relationship to have:
• Idempotence for similarity : A record is always similar to itself
• Commutativity of similarity : In deciding whether two records are similar it does not matter in which order we list them
• Representability : If r is similar to some other record s, but s is instead merged with some other record t, then r remains similar to the merger of s and t and can be merged with that record.
R-swoosh Algorithm for ICAR R-swoosh Algorithm for ICAR RecordsRecords Input: A set of records I, similarity function and a merge function. Output: A set of merged records O. Method:
O:= emptyset; WHILE I is not empty DO BEGIN
Let r be any record in I; Find, if possible, some record s in O that is similar to r; IF no record s exists THEN
move r from I to O ELSE BEGIN
delete r from I;
delete s from O;add the merger of r and s to
I; END; END;
Other Approaches to Entity ResolutionOther Approaches to Entity Resolution The other approaches to entity resolution are :
Non ICAR Datasets : We can define a dominance relation r<=s that means record s contains all the information contained in record r.If so, then we can eliminate record r from further consideration.
Clustering : Some time we group the records into clusters such that members of a cluster are in some sense similar to each other and members of different clusters are not similar.
Partitioning : We can group the records, perhaps several times, into groups that are likely to contain similar records and look only within each group for pairs of similar records.