Post on 17-Jan-2016
ArchIS: An Efficient Transaction-Time
Temporal Database System
Built on Relational Databases and XML
Fusheng Wang
University of California, Los Angeles
Motivation: Temporal Applications
Financial applications
Record-keeping applications
Scheduling applications
Scientific applications
Most database applications are temporal in nature:
Temporal Databases: the Reality
Over 40 temporal data models and query languages have been proposed in the past
A long struggle to get around the limitations of RDBMS
No DBMS vendors have moved aggressively to extend SQL with temporal support
What’s Needed?
Expressive temporal representations and data models with minimal or no extension
Powerful languages for temporal queries with minimal or no extension
Indexing, clustering and query optimization techniques for efficient query support
Architectures that bring these together
A temporal database system that provides:
Outline
Motivation
Viewing Relation History in XML
Temporal Queries with XQuery
The ArchIS System
Performance Study
Database Compression
Conclusion
Background: Publishing Relational Database as XML
Publishing relational DBs as XML
as actual XML documents: SQL/XML
as XML views: SilkRoute, XPeranto
Viewing Relation History in XML
Our proposal: view the history of relational DBs as XML documents:
Such history can be naturally represented in XML, without any extension to the data model
Temporal queries can be expressed in XQuery as is—without any extension to the language
Amenable for efficiently implementations
Temporal Grouping in XML
Temporal data models can be classified as:
Temporally ungrouped
Temporally grouped
Temporally grouped data models have more expressive power and are more natural for users
It is difficult to fit temporally grouped models into RDBMS
Temporally grouped data model can be represented well in XML
Example: Transaction-Time History of Tables
Timestamped tuple snapshots (temporally ungrouped)name empno salary title deptno DOB start end
Bob 10003 60000 Engineer d01 1945-04-09 1995-01-01 1995-05-31
Bob 10003 70000 Engineer d01 1945-04-09 1995-06-01 1995-09-30
Bob 10003 70000 Sr Engineer d02 1945-04-09 1995-10-01 1996-01-31
Bob 10003 70000 Tech Leader d02 1945-04-09 1996-02-01 1996-12-31
name empno salary title deptno DOB
Bob
1995-01-01:1996-12-31
10003
1995-01-01:1996-12-31
60000
1995-01-01:1996-05-31
Engineer1995-01-01:1995-09-30
d01
1995-01-01:1995-09-30
1945-04-09
1995-01-01:1996-12-3170000
1995-06-01:1996-12-31
Sr Engineer1995-10-01:1996-01-31
d02
1995-10-01:1996-12-31Tech Leader1996-02-01:1996-12-31
Temporally grouped history of employees
XML Representation of DB History
<employees tstart="1995-01-01" tend="1996-12-31"><employee tstart="1995-01-01" tend="1996-12-31">
<empno tstart="1995-01-01" tend="1996-12-31">10003</empno><name tstart="1995-01-01" tend="1996-12-31">Bob</name><salary tstart="1995-01-01" tend="1995-05-31">60000</salary><salary tstart="1995-06-01" tend="1996-12-31">70000</salary><title tstart="1995-01-01" tend="1995-09-30">Engineer</title><title tstart="1995-10-01" tend="1996-01-31">Sr Engineer</title><title tstart="1996-02-01" tend="1996-12-31">Tech Leader</title><deptno tstart="1995-01-01" tend="1995-09-30">d01</deptno><deptno tstart="1995-10-01" tend="1996-12-31">d02</deptno><DOB tstart="1995-01-01" tend="1996-12-31">1945-04-09</DOB>
</employee> <!-- … -->
</employees>
<employees tstart="1995-01-01" tend="1996-12-31"><employee tstart="1995-01-01" tend="1996-12-31">
<empno tstart="1995-01-01" tend="1996-12-31">10003</empno><name tstart="1995-01-01" tend="1996-12-31">Bob</name><salary tstart="1995-01-01" tend="1995-05-31">60000</salary><salary tstart="1995-06-01" tend="1996-12-31">70000</salary><title tstart="1995-01-01" tend="1995-09-30">Engineer</title><title tstart="1995-10-01" tend="1996-01-31">Sr Engineer</title><title tstart="1996-02-01" tend="1996-12-31">Tech Leader</title><deptno tstart="1995-01-01" tend="1995-09-30">d01</deptno><deptno tstart="1995-10-01" tend="1996-12-31">d02</deptno><DOB tstart="1995-01-01" tend="1996-12-31">1945-04-09</DOB>
</employee> <!-- … -->
</employees>
Advantages of XML Representations
The attribute value history is grouped, and can be queried directly
The H-document has a well-defined schema generated from the current table
The interval constraints are maintained in the updates
Outline
Motivation
Viewing Relation History in XML
Temporal Queries with XQuery
The ArchIS System
Performance Study
Database Compression
Conclusion
Temporal Queries with XQuery
XQuery: the coming standard query language for XML
With XQuery, we can specify temporal queries without any extension:
Temporal projection, snapshot queries, temporal joins, interval queries
Complex queries: A SINCE B, continuous periods, period containment
Temporal Queries with XQuery
Temporal projection: retrieve the salary history of “Bob”:
element salary_history { for $s in doc("employees.xml")/ employees/employee/[name=“Bob”]/salary return $s }
element salary_history { for $s in doc("employees.xml")/ employees/employee/[name=“Bob”]/salary return $s }
Snapshot queries: retrieve the departments on 1996-01-31:
for $d in doc("depts.xml")/depts/dept[tstart(.) <= "1996-01-31" and tend(.) >= "1996-01-31"]let $n := $d/name[tstart(.)<="1996-01-31" and tend(.)>="1996-01-31"]let $m := $d/manager[tstart(.)<="1996-01-31" and tend(.)>=
"1996-01-31"]return( element dept{$n,$m } )
for $d in doc("depts.xml")/depts/dept[tstart(.) <= "1996-01-31" and tend(.) >= "1996-01-31"]let $n := $d/name[tstart(.)<="1996-01-31" and tend(.)>="1996-01-31"]let $m := $d/manager[tstart(.)<="1996-01-31" and tend(.)>=
"1996-01-31"]return( element dept{$n,$m } )
Temporal Functions
Shield the user from the low-level details used in representing time, e.g., “now”
Eliminate the need for the user to write complex functions, e.g., coalescing
Predefined functions:Restructuring: coalese($l)Period comparison : toverlaps, tprecedes, tcontains, tequals,
tmeetsDuration and date/time: tstart($e), tend($e), timespan($e)
telement(Ts, Te): constructs an empty element element timestamped as tstart=Ts, tend=Te
Support for ‘now’
‘now’: no change until now Internally, “end of time” values are used to denote
‘now’, e.g., 9999-12-31 Intervals are only accessed through built-in
functions: tstart() returns the start of an interval, tend() returns the end or CURRENT_DATE if it’s different from 9999-12-31
In the output, tend value can be: “9999-12-31”CURRENT_DATE by using rtend($e) that recursively
replaces all the occurrence of 9999-12-31 with the current date,
“now”, using externalnow($e) that recursively replaces all the occurrence of \9999-12-31" with the string \now".
Outline
Motivation
Viewing Relation History in XML
Temporal Queries with XQuery
The ArchIS System
Performance Study
Database Compression
Conclusion
The ArchIS System
Two approaches are possible for storing and querying H-documents (H-views)
Native XML database approach: store H-documents directly into XML DB
XML-enabled RDBMS. Design issues include:
mapping (shredding) the XML views representing the H-documents into tables (H-tables)
translation of queries from the XML views to the H-tables
indexing, clustering and query mapping techniques
ArchIS: Archival Information System
The ArchIS System: Architecture
H-tables
Relational DataCurrent Database
Active Rules/update logs
Temporal XML Data
SQL Queries
Temporal XML Queries
H-views(H-documents)
AARRCCHHIISS
H-tables
AssumptionsEach entity or relation has a unique key ( or
composite keys) to identify it which will not change along the history. e.g., employee: empno
H-tables:attribute history table: store history of each attributekey table: built for the keyglobal relation table: record the history of relations
e.g.: current database:employee(empno, name, sex, DOB, deptno, salary,
title)
H-tables (cont’d)
current table
H-tables
employee global relation table
relations(relationname, tstart, tend)
empno key table employee_id(id, tstart, tend)
name attribute history
table
employee_name(id, name, tstart, tend)
… …
salary employee_salary(id, salary, tstart, tend)
title employee_title(id, title, tstart, tend)
H-tables (cont’d)
Sample contents of employee_salary:
ID SALARY TSTART TEND ======= ======= ========== ========== 100022 58805 02/04/1985 02/04/1986 100022 61118 02/05/1986 02/04/1987 100022 65103 02/05/1987 02/04/1988 100022 64712 02/05/1988 02/03/1989 100022 65245 02/04/1989 02/03/1990 100023 43162 07/13/1988 07/13/1989 ...
Updating Table Histories
Changes in the current database can be tracked with either update logs or triggers
DB2: triggers
ArchIS: update logs
Query Mapping
General purpose query mapping: XPeranto
In ArchIS, we have well-defined mapping between H-documents (or H-views) and H-tables
We map temporal XQuery queries into SQL, utilizing SQL/XML
SQL/XML is a new standard to map between RDBMS and XML
Both tag-binding and structure construction is pushed inside the relational engine, thus be very efficient
SQL/XML Publishing Functions
XMLElement and XMLAttribute
XMLAgg
select XMLElement (Name "dept", XMLAttributes (tstart as "tstart", tend as "tend"), deptname) from dept where deptname = ‘Sales’
<dept tstart = "02/04/1985" tend = "12/31/9999"> Sales </dept>
<dept tstart = "02/04/1985" tend = "12/31/9999"> Sales </dept>
select XMLElement (Name as "new_employees", XMLAttributes ("02/04/2003" as "Since") XMLAgg (XMLElement (Name as "employee", e.name))from employee_name as ewhere e.tstart >= ‘02/04/2003’ <new_employees Since =
"02/04/2003"> <employee>Bob</employee> <employee>Jack</employee></new_employees>
<new_employees Since = "02/04/2003"> <employee>Bob</employee> <employee>Jack</employee></new_employees>
XQuery Mapping to SQL with SQL/XML
select XMLElement (Name "salaryhistory", XMLAgg (XMLElement (Name as "salary", XMLAttributes (S.tstart as tstart, S.tend as "tend"), S.salary)))from employee_salary as S, employee_name as Nwhere N.id = S.id and N.name = 'Bob'group by N.id
select XMLElement (Name "salaryhistory", XMLAgg (XMLElement (Name as "salary", XMLAttributes (S.tstart as tstart, S.tend as "tend"), S.salary)))from employee_salary as S, employee_name as Nwhere N.id = S.id and N.name = 'Bob'group by N.id
Temporal projection: retrieve the salary history of “Bob”:
element salary_history { for $s in doc("employees.xml")/ employees/employee/[name=“Bob”]/salary return $s }
element salary_history { for $s in doc("employees.xml")/ employees/employee/[name=“Bob”]/salary return $s }
XQuery Mapping to SQL with SQL/XML: Steps
Identification of variable rangeMap variables in FOR/LET clause into underlying H-
tables
Generation of join conditionsThere is a join condition any pair of distinct tuple
variables: join them by ids
Translation of built in functionsMap built-in temporal functions in XQuery into
functions in ArchIS
Output generationuse XMLElement and XMLAgg constructs
Temporal Clustering and Indexing
Tuples in H-tables are stored in the order of updates, thus neither temporally clustered nor clustered by objects
Traditional indexes such as B+ Tree will not help on snapshot queries, and better temporal clustering is needed
For every segment, usefulness: U = Nlive/Nall
At the beginning, U =100%, and it decreases with updates
The minimum tolerable usefulness: Umin
Live
Segment-based Clustering Scheme
Segment 1
All
Segment 2
All
Live
Segment 3
All
segstart1 segend1 segstart2 segend2 segstart3 segend3
tstarttuple <= segendSEG
tendtuple >= segstartSEG
tstarttuple <= segendSEG
tendtuple >= segstartSEG
Segment-based Clustering Scheme
Initially all tuples for an attribute history table are archived in a live segment SEGlive with usefulness U =100%. With updates, when U drops below Umin:
1. A new segment is allocated;
2. The interval of this segment is recorded in the table segment(segno, segstart, segend);
3. All tuples in SEGlive are copied into a new segment Si sorted by id;
4. All live tuples in SEGlive are copied into a new live segment SEGlive', and the old live segment is dropped;
After that, the new segment SEGlive’ becomes the new starting segment for updates
Segment-based Clustering Scheme (cont’d)
Sample segments: Segment1 (01/01/1985 - 10/17/1991): ID SALARY TSTART TEND 100002 40000 02/20/1988 02/19/1989 100002 42010 02/20/1989 02/19/1990 100002 42525 02/20/1990 02/19/1991 100002 42727 02/20/1991 12/31/9999 ... Segment2 (10/18/1991 - 07/08/1995): ID SALARY TSTART TEND 100002 42727 02/20/1991 02/19/1992 100002 45237 02/20/1992 02/18/1993 100002 46465 02/19/1993 02/18/1994 100002 47418 02/19/1994 02/18/1995 100002 47273 02/19/1995 12/31/9999 ...
Advantages of Segment-based Clustering Scheme
The current live segment always has a high usefulness, assuring efficient updates;
Records are globally temporally clustered on segments;
For snapshot queries, only one segment is used; for interval queries, only segments involved are used;
Flexibility to control the number of redundant tuples in segments with Umin
Storage Usage of Segment-based Clustering
0.0 0.1 0.2 0.3 0.41.0
1.1
1.2
1.3
1.4
1.5
1.6
1.7
Sto
rage
Siz
e
Umin
1/(1-Umin
) Testing Data(U
min=0)
Relative storage size with different Umin
Nseg <= N0/(1-Umin)
NS
Query Performance on Temporal Data with Segment-based Clustering
Q1 Q2 Q3 Q4 Q5 Q6
1
2
3
4
5
Sec
onds
ArchIS without segment-based clustering ArchIS with segment-based clusteringQueries:
Point: Q1Snapshot: Q2Interval: Q5History: Q3, Q4, Q6
Outline
Motivation
Viewing Relation History in XML
Temporal Queries with XQuery
The ArchIS System
Performance Study
Database Compression
Conclusion
Performance Study: Experimental Setup
Systems: Tamino, DB2, and ArchISArchIS uses BerkeleyDB as its storage manager, and it
builds on top of it a SQL query engine
Temporal data set: the history of 300,024 employees over 17 yearsThe simulation models real world salary increases,
changes of titles, and changes of departmentsThe size of the XML data is 334MBThe single large XML document is cut into a collection
of 15,000 small XML documents with around 25KB each
Machine: Pentium IV 2.4GHz PC with RedHat 8.0
Performance Study: Query Performance
Q1 Q2 Q3 Q4 Q5 Q6
0.1
1
10
100
Sec
onds
DB2 ArchIS Tamino
snapshot query Q2 on ArchIS is 137 times faster than that on Tamino;interval query Q5 is 91 times faster; history Q6 is 25 times faster; Q4 4 times faster, and Q3 near 3 times faster.Tamino with clustering: snapshot Q2 is 3.3 times faster than without clustering ( still 41times slower than archIS); interval query Q5 is 2.9 times faster than without clustering( still 31 times slower than on ArchIS); history queries are much slower
DB2 and ArchIS: with clusteringTamino: without clustering
Storage Utilization
DB2 ArchIS Tamino (with compression)
0.0
0.5
1.0
1.5
Com
pres
sion
Rat
io
Outline
Motivation
Viewing Relation History in XML
Temporal Queries with XQuery
The ArchIS System
Performance Study
Database Compression
Conclusion
Database Compression
The disparity between CPU/memory and disk speeds is becoming larger and larger
Cost to read one IDE disk page: 14ms
Cost to uncompress one page: 1.1ms(500MHz CPU) 0.26ms(2.4GHz CPU)
Cost to retrieve one compressed page: 14ms + 0.26ms = 14.3ms
Cost to retrieve uncompressed pages (3.6 pages): 14ms x 3.6 = 50.4ms
Page-based Compression: PageZIP
Traditional data compression tools: compress a file as a whole
PageZIP: page-based compression and uncompression at the granularity of a page
Based on gzip library: zlib
Benefit: save space; point, snapshot or interval queries only retrieve a small fraction of the history, and can be efficient
PageZIP
Segment 1
Segment n
page 1 ID: 1001 - 1100page 2 ID: 1100 - 1203page 3 ID: 1203 - 1331…
…
Storage Utilization with Compression
For each attribute history table, we compress it as a sequence of pages and store each page as a BLOB in a RDBMSemployee_salary (sid, salary, tstart, tend) =>
employee_salary_blob(pageno, startsid, endsid, pageblob)
Tamino DB2 ArchIS
0.0
0.5
1.0
1.5 without compression
with compression
Com
pres
sion
rat
io
Query Performance with Compression
Q1 Q2 Q3 Q4 Q5 Q6
1
10
100
ArchIS without compression ArchIS with compression
DB2 without compression DB2 with compression TaminoSeconds
Update Performance
For RDBMS, only the current segment is used for updates. For Tamino, current data and historical data are clustered together
Update an employee’s salary:
DB2: 0.29 seconds; Tamino: 1.2 seconds
Assume that every employee gets updated once a year: about 1/260 of the total employee get updated every day on average
DB2: 1.52 seconds; Tamino: 15 seconds
In the worse case for segment-based archiving: 39 seconds for copying segments and 36 segments for compression: but only once
Summary
We built a transaction time temporal database on RDBMS and XML, with:XML to support temporally grouped (virtual)
representations of the database historyXQuery to express powerful temporal queries on such
views temporal clustering for managing the actual historical
data in a RDBMSSQL/XML for executing the queries on the XML views
as equivalent queries on the relational DBcompression as option for efficient storage
ArchIS provides a unified solution for a wide spectrum of temporal application problems
Future Work
Friendly temporal query interfaces based on temporally grouped models
Other clustering and indexing techniques to be investigated
Other efficient data compression techniques proposed for XML data to be investigated
Apply the approach to valid-time DB and bi-temporal DB
Apply the approach to OODBMS and semi-structured data model