SEQUENCE INDEXING SCHEMESRoman Čížek Erasmus 2687,

Nelly Vouzoukidou MET601

INTRODUCTION

Graph indexes precise Path, (twig only few methods)

Sequence indexing schemes Top-down or bottom-up XML document and XML queries in structure-encoded

sequences Path and twig

TOP-DOWN SEQUENCE INDEXES: VIST

VIST – VIRTUAL SUFFIX TREE

Top-down Sequence Indexes Represent XML documents and XML queries in

structure-encoded sequences Querying XML data is equivalent to finding subsequence

matching Avoid to expensive join operations Provides unified index on both content and structure Support dynamic index update B+Trees which are supported in DBMSs

DTD OF PURCHASE RECORDS

<!ELEMENT purchases (purchase*)><!ELEMENT purchase (seller, buyer)><!ATTRIST seller ID ID location CDATA name CDATA><!ELEMENT seller (item*)><!ATTRIST buyer ID ID location CDATA name CDATA><!ELEMENT item (item*)><!ATTRIST item name CDATA manufacturer CDATA>

A SINGLE PURCHASE RECORD

PREORDER SEQUENCE OF XML

Use capital letters to represent names of elements/attributes

Use hash function h(), to encode attribute values into integers

v1 = h(“dell”) v2=h(“ibm”)

Preorder sequence of XML purchase record example PSNv1IMv2Nv3IMv4INv5Lv6BLv7Nv8

Isomorphic trees may produce different preorder seq. DTD schema embodies linear order of all elements/attributes Without DTD – use lexicographical order

STRUCTURE-ENCODED SEQUENCE

Definition: A Structure-Encoded Sequence, derived from a prefix traversal of semi-structured XML document, is a sequence of (symbol, prefix) pairs:

D = (a1,p1), (a2,p2),…, (an,pn)

Where ai represents a node in the XML document tree, (of which a1, … ,an is the preorder sequence), and pi is the path from the root node to node ai.

STRUCTURE-ENCODED SEQUENCE

D= (P,ϵ),(S,P),(N,PS),(v1,PSN),(I,PS),(M,PSI),(v2,PSIM),(N,PSI),(v3,PSIN),(I,PSI),(M,PSII),(v4,PSIIM),(I,PS),(N,PSI),(v5,PSIN),

(L,PS),(v6,PSL),(B,P),(L,PB),(v7,PBL),(N,PB),(v8,PBN)

XML QUERIES IN GRAPH FORM

XML QUERIES IN PATH EXPRESSION AND SEQUENCE FORM

Query: Path Expression Structure-Encoded Sequence

Q1 : /Purchase/Seller/Item/Manufacturer (P, ϵ)(S,P)(I,PS)(M,PSI)

Q2 : /Purchase[Seller[Loc = v5]]/Buyer[Loc = v7] (P, ϵ)(S,P)(L,PS)(v5,PSL)(B,P)(L,PB)(v7,PBL)

Q3 : /Purchase/*[Loc = v5]

(P, ϵ)(L, P)(v5,P*L) Q4 : /Purchase//Item[Manufacturer = v3]

(P, ϵ)(I,P//)(M, P//I)(v3,P//IM)

QUERYING XML THROUGH STRUCTURE-ENCODED SEQUENCE MATCHING Querying XML is equivalent to finding (non-contiguous)

subsequence matches Most structural XML queries can be performed through direct

subsequence matching Exception: branch has multiple identical child nodes

Q5=/A[B/C]/B/D Two different sequences

(A, ϵ)(B,A)(C,AB)(B,A)(D,AB) (A, ϵ)(B,A)(D,AB)(B,A)(C,AB)

Find matches separately and union their result We may find false matches if the indexed documents contain

branches with identical child nodes, then we ask multiple queries and compute set difference on result

If the query contains a large number of same child nodes under the branch, we can choose disassemble the tree into multiple trees and use join operations to combine their results

ALGORITHMS

Naïve algorithm RIST – Relationships Indexed Suffix Tree ViST – Virtual Suffix Tree

NAÏVE ALGORITHM: SUFFIX-TREE-LIKE STRUCTURE

Doc1 : (P, ϵ)( S, P)(N, PS)(v1, PSN)(L, PS)(v2, PSL) Doc2 : (P, ϵ)(B, P)(L, PB)(v2, PBL) Q1 : (P, ϵ)(B, P)(L,PB)(v2, PBL) Q2 : (P, ϵ)(L, P*)(v2,P*L)

D-ANCESTORSHIP AND S-ANCESTORSHIP

D-Ancestorship Ancestor-descendant relationships in original XML tree Element (S,P) is a D-Ancestorship of (L,PS)

S-Ancestorship Ancestor-descendant relationships in suffix tree Element (v1, PSN) is an S-Ancestorship of (L, PS)

NAÏVE SEARCH :A NAÏVE ALGORITHM BASED ON SUFFIX TREES

RIST – INDEXING CONSTRUCTION

S-Ancestorship requires additional information Label each suffix tree node x by pair <nx, sizex>

nx prefix traversal order of x in suffix tree sizex is total number of descendants of x in suffix tree

x … <nx, sizex>, y …<ny, sizey> x is S-Ancestor of node y if ny ϵ (nx, nx + sizex]

Construct the B+Trees: Tree nodes into the D-Ancestorship B+Tree using (Symbol,

Prefix) as keys For all nodes x inserted with the same (Symbol, Prefix) we

index them by S-Ancestorship B+Tree, using the nx values of their labels as keys.

THE RIST INDEX STRUCTURE

SEARCH: NON-CONTIGUOUS SUBSEQUENCE MATCHINGUSING B+TREE

VIST – VIRTUAL SUFFIX TREE

Dynamic Virtual suffix tree labeling Semantic and statistical clues Dynamic scope allocation without clues

DYNAMIC SCOPE ALLOCATION

Number of child nodes of x is λ. We allocate 1/ λ of the remaining scope to x’s first child

Dynamic scope allocation with λ=2

DYNAMIC SCOPE OF A SUFFIX TREE NODE

SUBSCOPE(PARENT, E): CREATE A SUB SCOPEWITHIN THE PARENT SCOPE FOR E

INSERTION INDEX

Doc1 = (P, ϵ)(S,P)(N,PS)(v1,PSN)(L,PS)(v2,PSL) Doc2 = (P, ϵ)(S,P)(L,PS)(v2,PSL)

INDEX AN XML DOCUMENT

EXPERIMENTS - SAMPLE QUERIES

Path Expression DatasetQ1 /inproceedings/title DBLPQ2 /book/author[text=‘David’] DBLPQ3 /*/author[text= ‘David’] DBLPQ4 //author[text= ‘David’] DBLPQ5 /book[key=‘books/bc/MaierW88’]/author DBLPQ6 /site//item[location=‘US’]/mail/date[text=‘12/15/1999’]

XMARKQ7 /site//person/*/city[text=‘Pocatello’] XMARKQ8 //closed_auction[*[person=‘person1’]]/date[text=‘12/15/1999’]

XMARK

COMPARING INDEXING METHODS

time in seconds

INDEX STRUCTURE

DBLP (301 MB of data) XMARK (52MB of data)

CONCLUSION

structure-encoded sequences Sequence matching Avoid expensive join operations Top-down scope allocation method Index structure – B+Tree

PRIX:PRUFER SEQUENCES FOR INDEXING XML

PRIX: PRUFER SEQUENCES FOR INDEXING XML

Rao & Moon (2006) proposed a new method for indexing XML documents using sequences

It uses the same idea as in ViST index: The XML tree is transformed into a sequence and saved in the

database Each query is also transformed into a sequence The answer of the query is acquired by performing subsequence

matching

PRIX: PRUFER SEQUENCES FOR INDEXING XML

PRIX: PRUFER SEQUENCES FOR INDEXING XML

MOTIVATION: TWIG QUERIES AND WILDCARDS

Like in ViST, PRIX also tries to efficiently answer twig queries as well as queries containing wildcards (‘*’ any and ‘//’ self or descendant queries)

P

Q

T S

Twig queryXPath: P/Q[T]/S

Query with wildcardsXPath: P//Q/S

P

Q

S

MOTIVATION: PROBLEMS IN VIST INDEX

Memory requirements: In the worst case, ViST requires O(N2) space to index the

document

A

B

C

D

D = (A, ε), (B, A), (C, AB), (D, ABC), (E, ABCD)

EElements in height k

appear k times

<A> <B> <C> <D> <E> </E> </D> </C> </B></A>

MOTIVATION: PROBLEMS IN VIST INDEX

Memory requirements: In the worst case, ViST requires O(N2) space to index the

document False positives

In many cases, query processing in Vist results in false alarms

P

Q

T

R

TUS

Doc1 = (P, e) (Q, P) (T, PQ) (S, PQ) (R, P) (U, PR) (T, PR)

P

Q

T

Q

S

Doc2 = (P, e) (Q, P) (T, PQ) (Q, P) (S, PQ)

P

Q

T S

XPath: P/Q[T]/SQ = (P, e) (Q, P) (T, PQ) (S, PQ)

MOTIVATION: PROBLEMS IN VIST INDEX

Memory requirements: In the worst case, ViST requires O(N2) space to index the

document False positives

In many cases, query processing in Vist results in false alarms False negatives

Correctly answering a twig query depends on the order the branches are created

P

F

T

N

G

Doc = (P, e) (F, P) (T, PF) (N, P) (G, PN)

P

N F

Xpath: P[N]/FQ = (P, e) (N, P) (F, P) ???

MOTIVATION: PROBLEMS IN VIST INDEX

Memory requirements: In the worst case, ViST requires O(N2) space to index the

document False positives

In many cases, query processing in Vist results in false alarms False negatives

Correctly answering a twig query depends on the order the branches are created

PRIX: PRUFER SEQUENCES FOR INDEXING XML

PRIX ARCHITECTURE

INDEXING AND QUERYING IN PRIX

Indexing: The first step is to take as input an XML document and

convert it into a sequence This is achieved using Prufer Sequences

The sequence is saved in the database in a way equivalent to the one used in ViST It is a Virtual Trie implemented as B+ Trees

XML document

INDEXING AND QUERYING IN PRIX

Querying Queries are also transformed to trees and then to Prufer

Sequences

The query sequence looked up in the document sequence and all matching subsequences are retrieved

After this initial filtering, three refinement phases follow

XPath Query

PRIX: PRUFER SEQUENCES FOR INDEXING XML

INDEXING XML DOCUMENTS The first step is to transform the XML document to the

equivalent XML tree

Notice that both elements and text values are represented as nodes (the same stands for attributes)

The tree is not saved in the database

<A> <B></B> <B> <C> D </C> <C> <F/> <E/> </C> </B></A>

A

B B

CC

D F E

INDEXING XML DOCUMENTS

Then the Prufer Sequence is created from the XML tree A Prufer Sequence is a method proposed by Prufer

(1918) that constructs a one-to-one correspondence between a labeled tree and a sequence

8,A

1,B 7,B

6,C3,C

2,D 5,E4,F

8, 3, 7, 6, 6, 7, 8

INDEXING XML DOCUMENTS

Prufer Sequences can only be created from trees with numerical labeling, with each node having a unique number

Since the XML tree contains string labels (the names of elements etc.) we add an additional label to each node

We will use the post-order traversal to name the nodes The prufer sequence can be extracted for any labeling of the

tree, but using post-order numbering has some properties that makes the querying process easier

INDEXING XML DOCUMENTS

Initial labeling

A

B B

CC

D F E

8,A

1,B 7,B

6,C3,C

2,D 5,E4,F

INDEXING XML DOCUMENTS

Finding the Prufer Sequence The algorithm to find the Prufer sequence is the

following: Find the leaf with the smallest value and delete it. Add the label of its parent to the sequence Repeat until only one node is left

In PRIX index, two sequences are held: The actual Prufer Sequence holding the numbers of the

labels called Numbered Prufer Sequence: NPS The corresponding sequence holding the actual labels of the

nodes of the XML Tree called Labeled Prufer Sequence: LPS

INDEXING XML DOCUMENTS

Finding the Prufer Sequence The algorithm to find the Prufer sequence is the

following: Find the leaf with the smallest value and delete it. Add the label of its parent to the sequence Repeat until only one node is left

8,A

1,B 7,B

6,C3,C

2,D 5,E4,F

NPS : 8, LPS : A,

INDEXING XML DOCUMENTS

Finding the Prufer Sequence The algorithm to find the Prufer sequence is the

following: Find the leaf with the smallest value and delete it. Add the label of its parent to the sequence Repeat until only one node is left

8,A

7,B

6,C3,C

2,D 5,E4,F

NPS : 8, 3LPS : A, C

1,B

INDEXING XML DOCUMENTS

Finding the Prufer Sequence The algorithm to find the Prufer sequence is the

following: Find the leaf with the smallest value and delete it. Add the label of its parent to the sequence Repeat until only one node is left

8,A

7,B

6,C3,C

2,D 5,E4,F

NPS : 8, 3, 7, 6, 6, 7, 8LPS : A, C, B, C, C, B, A

1,B

INDEXING XML DOCUMENTS

Properties Both NPS and LPS have length N-1 (where N is the total

number of nodes Due to the fact that we delete one node at a time until only

one node is left

NPS : 8, 3, 7, 6, 6, 7, 8LPS : A, C, B, C, C, B, A

8,A

1,B 7,B

6,C3,C

2,D 5,E4,F

INDEXING XML DOCUMENTS

Properties Both NPS and LPS have length N-1 (where N is the total

number of nodes The i-th element deleted is always the node with label i

This helps us find the edges of the tree! (that is the mapping from the NPS to the tree)

NPS : 8, 3, 7, 6, 6, 7, 8LPS : A, C, B, C, C, B, A

8,A

1,B 7,B

6,C3,C

2,D 5,E4,F

Deleted node: 1, 2, 3, 4, 5, 6, 7

INDEXING XML DOCUMENTS

Properties Both NPS and LPS have length N-1 (where N is the total

number of nodes The i-th element deleted is always the node with label i LPS does not contain any leaves

NPS : 8, 3, 7, 6, 6, 7, 8LPS : A, C, B, C, C, B, A

8,A

1,B 7,B

6,C3,C

2,D 5,E4,F

Deleted node: 1, 2, 3, 4, 5, 6, 7

INDEXING XML DOCUMENTS

Indexes held in the database are The LPS (label prufer sequence) The NPS (numbered prufer sequence) The mapping between the number and the xml label of

the leaves of the tree

NPS : 8, 3, 7, 6, 6, 7, 8LPS : A, C, B, C, C, B, ALeaves mapping: 1 B, 2 D, 4 F, 5 E

8,A

1,B 7,B

6,C3,C

2,D 5,E4,F

PRIX: PRUFER SEQUENCES FOR INDEXING XML

QUERYING

When a query arrives it is also transformed to a prufer sequence

Then, an initial filtering is performed The results of the initial filtering are sorted out in order

to acquire the correct answer to the query after three more refinement phases.

XPath Query

QUERYING:TRANSFORMING A QUERY TO A PRUFER SEQUENCE The same process as in documents is followed For instance if we have the XPath query

A[B/C]/D/E/F The query tree is:

The NPS and LPS are: NPS(Q) = 2, 6, 4, 5, 6 LPS(Q) = B, A, E, D, A

QUERYING:FILTERING BY SEQUENCE MATCHING

Suppose we have the following XML tree (T) of the document:

NPS(T) = 15, 3, 7, 6, 6, 7, 15, 9, 15, 13, 13, 13, 14, 15 LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A

QUERYING:FILTERING BY SEQUENCE MATCHING

To find the correct results for the given query we find the subsequences of LPS(Q) inside LPS(T)

“A subsequence is any string that can be obtained by deleting zero or more symbols from a given string”

QUERYING:FILTERING BY SEQUENCE MATCHING To find the correct results for the given query we find the

subsequences of LPS(Q) inside LPS(T) Each subsequence represents a possible solution in the tree

LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A LPS(Q) = B, A, E, D, A

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the correct results for the given query we find the

subsequences of LPS(Q) inside LPS(T) Each subsequence represents a possible solution in the tree

LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A LPS(Q) = B, A, E, D, A

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the correct results for the given query we find the

subsequences of LPS(Q) inside LPS(T) Each subsequence represents a possible solution in the tree

LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A LPS(Q) = B, A, E, D, A

12 subsequences are found in total, while only 4 are correct

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the correct results for the given query we find the

subsequences of LPS(Q) inside LPS(T) Each subsequence represents a possible solution in the tree

LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A LPS(Q) = B, A, E, D, A

12 subsequences are found in total, while only 4 are correct

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the correct results for the given query we find the

subsequences of LPS(Q) inside LPS(T) Each subsequence represents a possible solution in the tree

LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A LPS(Q) = B, A, E, D, A

12 subsequences are found in total, while only 4 are correct

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the correct results for the given query we find the

subsequences of LPS(Q) inside LPS(T) Each subsequence represents a possible solution in the tree

LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A LPS(Q) = B, A, E, D, A

12 subsequences are found in total, while only 4 are correct

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the correct results for the given query we find the

subsequences of LPS(Q) inside LPS(T) Each subsequence represents a possible solution in the tree

LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A LPS(Q) = B, A, E, D, A

12 subsequences are found in total, while only 4 are correct

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the path in the tree that is represented by the

sequence found while filtering we use the NPS(T) Recall that the edges can be retrieved using the index in the

NPS(T)

NPS(T) = 15, 3, 7, 6, 6, 7, 15, 9, 15, 13, 13, 13, 14, 15 LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the path in the tree that is represented by the

sequence found while filtering we use the NPS(T) Recall that the edges can be retrieved using the index in the

NPS(T)

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 NPS(T) = 15, 3, 7, 6, 6, 7, 15, 9, 15, 13, 13, 13, 14, 15 LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the path in the tree that is represented by the

sequence found while filtering we use the NPS(T) Recall that the edges can be retrieved using the index in the

NPS(T)

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 NPS(T) = 15, 3, 7, 6, 6, 7, 15, 9, 15, 13, 13, 13, 14, 15 LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A

T Q

QUERYING:FILTERING BY SEQUENCE MATCHING To find the path in the tree that is represented by the

sequence found while filtering we use the NPS(T) Recall that the edges can be retrieved using the index in the

NPS(T)

1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14 NPS(T) = 15, 3, 7, 6, 6, 7, 15, 9, 15, 13, 13, 13, 14, 15 LPS(T) = A, C, B, C, C, B, A, C, A, E, E, E, D, A

T Q

QUERYING:REFINEMENT STEPS

Despite the filtering, some false positives are in the results.

To find these false positives we have 3 refinement steps, namely: Refinement by connectedness Refinement by structure Refinement by matching leaf nodes

QUERYING: FALSE NEGATIVES

A false negative can appear in the same case as in ViST index

The subsequence filtering relies on the assumption that the query branches come in the “correct” order

P

F

T

N

G

Document

P

N F

Query

QUERYING: FALSE NEGATIVES

The solution proposed by Rao and Moon is to test the query in all possible permutations of the branches and then return the union as the answer of the query N branches N! permutations

Their main argument is that queries usually have a small number of branches

QUERYING: FALSE NEGATIVES

The solution proposed by Rao and Moon is to test the query in all possible permutations of the branches and then return the union as the answer of the query N branches N! permutations

Their main argument is that queries usually have a small number of branches

P

N F D

S

P

N FD

S

P

NF D

S

… (three more permutations)

EXPERIMENTS

VIST VS PRIX: EXPERIMENTS

1.8GHz Pentium IV processor 512 MB RAM running Solaris 8 40GB EIDE disk drive (store data and indexes) Compiled by GNU g++ compiler version 2.95.3 Buffer pool size: 2000 pages of size 8K

VIST VS PRIX: EXPERIMENTS

VIST VS PRIX: EXPERIMENTS

VIST VS PRIX: EXPERIMENTS

DBLP dataset

VIST VS PRIX: EXPERIMENTS

SWISSPROT dataset

VIST VS PRIX: EXPERIMENTS

TREEBANK dataset

VIST VS PRIX

O(N2)

?

? ?

QUESTIONS?

THANK YOU!!