Native XML Databases

Lior Schejter

Erez Hadad

Outline

Introduction: What? Why? How? An example XML DBMS: Natix Query processing

Natix query engine XML DBMS Storage

What is stored in an XML-DBMS? Storage Management: Natix Storage Storing documents and Indexes

Transaction Management Logging & Recovery Locking

Native XML DBMS – What?

A DBMS native to XML Providing programming interfaces to manage and query XML

data (1 or more documents) Using a full blown, consistent API

XML is the natural way of accessing data in this kind of DBMS Defining, Querying etc.

Providing all familiar features of DBMS Transactions Recovery Multi threading

Native XML Databases – Why?

XML structure provides information Complex data model

Ever tried representing an organizational hierarchy in a DBMS?

Flexible data model Both structured and semi structured data

Inherent XML “behavior” DBMS will be optimized for XML Think about querying \ storing \ updating thousands of

XML documents

Native XML Databases – How?

XML can be thought of as a: Text document (tags, simple text etc.) A data model (nodes, children, siblings)

Document centric – Storing and retrieving the entire document (or large parts of it) Fast document construction and storage Slow on queries, and retrieving data

Data centric – Expressing the entire data in an internal data structure Fast queries Slow document retrieval

A question of granularity

Natix

Natix – An XML DBMSDeveloped at Manheim University, Germany.

Designed from scratch for storing and accessing XML data

Supports XPath, XQuery Not designed to any specific language and

\ or environment

Natix - Architecture

Natix - Architecture

Storage layer – Manage all persistent data storage

Service Layer – Provides DBMS functionality

Binding Layer – Modules that map data and requests from other APIs to the Natix engine interface

Natix Query Execution Engine

Able to execute all queries in a typical XML query language (e.g. XQuery).

Expressive: Small number of powerful parameterized operators

2 main components:Natix Physical Algebra (NPA)

Algebraic operators, their composition etc.Natix Virtual Machine (NVM)

Plans used in the algebraic operators

NQE – The Rough Guide

NPA works on sequences of tuples Each tuple holds values which can be a

number, a string or a node handleXML node handles can point to any node type

NPA operators are implemented as iterators

NPA operators take programs for the NVM as parametersUsually passed at construction

Natix Virtual Machine

NVM commands operate on register sets An NVM command may access several register

sets Global (X) register set for global (but specific to

current plan of execution) data. Z, Y register sets for arguments (pass between

operators) Y register sets are used only for binary operators

A reference to register sets is passed between operators to avoid unnecessary copying

CMP_LEQ_SI4_ZCX 1 55 2

EXIT_F 2

X3 = Z1 + Z2PRINT X3

ARITH_ADD_A_SI4_ZZX 1 2 3

PRINT_SI4 3STOP

if (Z1 > 55) then exit

NVM program examples

XML NVM Commands

NVM has about 150 XML specific commandsCopying documents handles, comparing

fragments, traversing the XML document tree, printing etc.

Mainly commands which correspond to XPath axes (child, sibling, descendant etc.)

Example: UnnestMap operator

UnnestMapLogically: takes a set valued expression and

returns a single tuple for each element in the result set, flattening hierarchy 1 level deep

Physically: takes 3 programs and uses them as an iterator:

init – Initialize the first tuple to be returned step – compute the next tuple Fin – finalize, cleanup

UnnestMap Operator

init:

step:

fin: no finish program

XML_CHILD_ZZ 1 2

XML_VALID_ZX 2 3

EXIT_F 3

MV_XML_ZX 2 4

XML_SIBLING_NEXT_XX 4 4

XML_VALID_XX 4 3

EXIT_F 3

MV_XML_XZ 4 2

Natix Physical Algebra

Operators for selection and binding combination are borrowed from the relational and object databases contexts select, join, map, group etc.

The main concern: variable binding and result construction operators for XML

NPA – Query plans

Every plan of execution has a scan operation at the bottom of it: scanning a document and retrieving its root in a tuplee.g. Expression Scan

UnnestMap and PathScan are used for variable bindings as wellAn XPath expression can be translated into a

sequence of UnnestMap operationsOr a single PathScan operation, which also eliminates

duplicates, like in XPath

NPA Operators

e.g. SELECT

Any subplan

NPA Examples

Example DTD:

NPA Example 1

Query: <result>

{

FOR $c IN document(“bib.xml”)/bib/conference

WHERE $c/year > 1996

RETURN

<conference>

<title>{$c/title}</title>

<year>{$c/year}</year>

</conference>

}

</result>

NPA Example 1

The query plan: Function

Calls

Tuple

s

NPA Example 2

Query:<bib>

{

FOR $a IN document(“bib.xml”)//conference/article/author

RETURN

<author>

<name first={$a/@first} last={$a/@last} />

<articles>

{

FOR $b IN document(“bib.xml”)//conference/article,

$c IN $b/author

WHERE $c/@first=$a/@first AND $c/@last=$a/@last

RETURN

<article>{$b/title}</article>

}

</articles>

</author>

}

</bib>

NPA Example 2

The query plan:

Part II:Storage & Transactions inNative XML DBMS

What Is Stored In A Native XML-DB? XML Documents

The data itself The DBMS tries to maintain imported documents as

close to their original form as possible Data Definition Schemas

XML schemas, RelaxNG schemas, DTDs Used for:

Validating documents Organizing data on disk Validating and optimizing queries, constructing result sets Semantics: types, operations

What Is Stored In A Native XML-DB? Collections / Roots:

Bindings of XML documents into sets According to type or relevance A document may belong to more than one set

A collection may be related to a schemaCollections are valid instances of the data

model and can be processed through queriesfor $d in collection(“foo”)where $d/Book/Author/Lastname = “Dante”return $d

What Is Stored In A Native XML-DB? “Standard DB” components:

Indexes Speed up query execution

Stored functions / procedures / triggers Embed business logic in storage Server-local processing – reduce network traffic Create views – abstractions of data

Access control data Users, resources, groups, permissions

Storage Management

Usually, it is impossible to hold all the DB components in main memorySeveral orders of magnitude smaller than secondary

storage (disk) A common technique is to keep only a few fixed

objects in memory and load the rest on-demandAnalogous to virtual memory / disk caching

mechanisms of operating systems

Storage Management In Natix

Physical Storage

Partitions

Pages

Segments

Records(Slotted Page Segments)

BufferManager

PageInterpreter

PageInterpreter

Internal DatabaseStructure

How To Store XML Documents in a DB? Flat Stream:

Each XML document is a byte stream (e.g. a file or DBMS BLOB)

Fast handling of large sequential chunks or whole documents [document-centric]

Poor random access Requires parsing of XML

Example: Web server’s HTML file tree

How To Store XML Documents in a DB? Meta Modeling:

Separately store each element of the data model of an XML document (e.g. in a DBMS)

Analogy to RDBMS: Entities and relations (of an ERD model) stored in separate records in tables

Fast random access [data-centric]Slow processing of whole documents out of

(possibly) thousands of separate recordsMechanism required for “translating” between

data models, e.g. XML <=> Relational

How To Store XML Documents in a DB? Mix of FS and MM:

Redundant: Store each document both as a byte stream and as a collection of records

Read access is optimal: match the case Write access has high overhead: update both types of

storage in each operation

Hybrid: Define a “granularity threshold” A “small structure” object is stored as a flat stream inside a

single record of a database A “large structure” object is divided into several records Leverage time between whole-document and per-node

operations

Natix XML Storage

A hybrid approach:Each database record contains a single subtree

of an XML documentA dynamic granularity threshold, adapting to size

and structure of documents at runtimeA subtree can grow and split into several recordsSmall subtrees can be merged into a larger

subtree in a single record

Types of Stored Nodes in Natix

Aggregate nodes: inner nodes of the tree, containing their respective child nodes Helper aggregate nodes: “virtual” aggregate nodes

used for grouping subsets of children of an actual aggregate node into subtrees in records

Literal nodes: leaf nodes each containing an unparsed stream of bytes

Proxy nodes: “virtual” nodes that point to subtrees contained in other records

f1

h1

p1

f1

Natix Storage Example

XML File:

<f1>

<f2>..</f2> <f3>..</f3> <f4> <f7/> <f8>..</f8> </f4>

<f5/> <f6>..</f6> </f1>

Logical Tree: f1

f4

f8f7

f5 f6f3f2

Natix Storage Example

Physical Tree:

f1

f4

f8f7f3f2 f5 f6

h1 h2

p3p2p1

r1

r2 r3 r4

Modifying Documents In Natix

The physical tree is regarded as a B-Tree of records maintained balanced

If, when inserting a subtree, a record becomes too big

Split the record into a separator part, left part and a right part

Insert the separator into the parent record The algorithm may repeat in the parent

Similarly, a delete operation can result in record merger

Modifying Documents In Natix

f1

f6

f7

f8 f9

f11 f12

f13 f14f10f5

f2

f3 f4

S

RL

2. f7 is the Split Node

1. Add node f10 to a record containingthe following subtree:

3. Separator S is the pathfrom the root up to but notincluding f7

4. Right Forest R is inducedby the subtree of f7 and allthe descendants of S locatedright of f7

5. Left Forest L is inducedby the rest of the nodes

Modifying Documents In Natix

f12

f13 f14

f11f10f7

f8 f9

f5f2

f3 f4

f1

f6

h2h1

p1

p1p1

p1

1. In L & R, subtrees withsibling roots are groupedusing helper aggregates

S

RL

2. Each subtree is put intoa separate new partitionrecord

4. The separator eitherreplaces the proxy in theparent or forms a new root

3. The separator connectsto the partition records through proxie nodes

r4r3r1 r2

rparent

Indexes InNative XML-DBs Using indexes accelerate evaluation of queries by

quickly locating elements / values / text in the DB Indexes may be created

as fixed parts of the storage systems, or upon user request, or automatically due to repeated use of certain queries

Index granularity may vary: Point to each node that contains a specific key, or just to

the containing document A trade-off between size & construction speed of the

index vs. its effectiveness

Indexes InNative XML-DBs Common types of indexes:

Value indexes: list the locations of each typed value of a node:

E.g., locations of the integer value 1492.

Element indexes: list the locations of elements in documents, preserving hierarchy

Locate an element of a specific type (//footnote) or in a specific context (/appendix/footnote)

E.g., Tamino index structure, Natix XASR

Indexes In Native XML-DBs

Full Text Indexes: List the location of text within the content of elements

Common technique: inverted files “Location of word”: offset in file and / or in

document hierarchy Becomes more useful when the document is less

structured In the future, text index mechanism is expected to

resemble a modern search engine: Handle word equivalence (single/plural, synonyms) Ranked matching (degree of proximity instead of

true/false)

Natix eXtended AccessSupport Relations (XASR)

bioml

organism

organelle organelle

label label

“cytoskeleton” “mitochondrion”

112

211

36

710

45

89

dmin dmax eType docID parent

1 12 bioml 0 NULL

2 11 organism 0 1

3 6 organelle 0 2

4 5 label 0 3

7 10 organelle 0 2

8 9 label 0 7

1. The document is traversed in DFS order. Each node is assigned a value dmin upon entry and dmax upon exit. dmin is also the unique node id.

2. An XASR table is constructed as following:

3. During query evaluation, path connectors (‘/’ or ‘//’) are resolved through join operations on the XASR table:The join predicate for ‘/’ is: xi.docID = xi+1.docID and xi.dmin = xi+1.parentThe join predicate for ‘//’ is: xi.docID = xi+1.docID and xi.dmin < xi+1.dmin and xi.dmax > xi+1.dmax

Transaction Management

An XML DBMS provide support for transactions A sequence of operations on XML items that can be either

committed or rolled-back altogether Transaction execution follows ACID properties:

Atomicity: Each transaction should either complete or have no effect at all

Consistency: Each transaction should transform the DB from one consistent state to another

Isolation (serialization): Concurrently-executing transactions should behave as if they’re executing in some sequential order

Durability: Once committed, a transaction’s effect on the DB is permanent

Transaction Management

Consistency is achieved by properly defining the transaction boundaries Example: when moving money between bank

accounts, update both accounts in one transaction Responsibility of the application programmer

The other properties are provided by the DBMS Atomicity and durability are provided through logging

and recovery Isolation is maintained through locking

Logging And Recovery

Logging and recovery provide two important functions: undo and redoUndo of transactions that are aborted (will not

complete) Enables atomicity

Redo of committed transactions in case of DBMS failure before transaction results

were completely written to disk Enables durability

Logging And Recovery

The XML DBMS keeps a log of all operations affecting the DB

Each transaction operation that writes to a DB item generates a log recordWrite-ahead logging: log then write

All log records of the same transaction are linked chronologically forward (redo) and backward (undo)

Logging And Recovery In Natix

Optimizing L&R for XML hierarchies:Subsidiary Logging: log records are cached

and unified into more compact records before entering the log

One log record of adding a subtree instead of many added-single-node log records

Annihilator Undo: No need to perform undo operations that are covered by later undo operations

Skip remove-node / modify-node operations that are followed by matching remove-subtree operations

Locking Mechanism

In order to ensure that transactions are isolated from each other, each transaction locks its DB resources before operating on them Locking a resource prevents other transactions from

modifying/accessing it until unlocked Most common locking protocol: S2PL (Strict 2-Phase

Locking) Every resource is locked before its first access All resources are atomically unlocked together with

abort/commit

Types Of Locks

Classic Locks:S: Shared lock – for reading an item

Two transaction can share read access to an item Prevents item modification by other transactions

X: eXclusive lock – for modifying an item At most one transaction can hold X lock on an item Prevents other transactions from accessing item

S X

S + -

X - -

held lock

requestedlock

Compatibility Matrix:

Types Of Locks

Problem - While a transaction T operates on an element in an XML document:Other transactions should be allowed to operate

on other elements of the document Increase concurrency

However, no other transaction should be allowed to modify nodes that contain the element T is working on

Maintain the tree structure

Types Of Locks

Solution - Warning Locks:Lock all the nodes on the path from the root to the

designated node from being modified by other transactions, according to the intended operation on the designated node

IS: Intension Share – intend to read sub-elements of this node

IX: Intension eXclusive – intend to modify sub-elements of this node

Types Of Locks

SIX: Share Intension eXclusive – intend to read this node and modify sub-elements of it

Compatibility Matrix:

S X IS IX SIX

S + - + - -

X - - - - -

IS + - + + +

IX - - + + -

SIX - - + - -

held lock

requestedlock

Types Of Locks - Natix

What if a transaction accesses a node directly without traversing down from the root? E.g., when dereferencing an IDREF value Need to first find the root to begin locking Make sure the path to node is not modified until

locking completes In Natix, nodes are first locked in special Shared

Parent Pointer (SPP) mode on way up, then regular locks on way down Minimal lock, incompatible only with X

Lock Control In Natix

Lock Escalation: If a transaction holds too many locks, its locks are grouped and converted to bigger locks over entire documents (or even segments), if possible If the transaction still holds too many locks, it is

aborted

Deadlock Detection: If a transaction waits too long for a lock, the system starts looking for deadlocks involving the waiting transaction. If found, the transaction is aborted