Augmenting Suffix Trees, with Applications

Augmenting Suffix Trees, with Augmenting Suffix Trees, with ApplicationsApplications

Yossi Matias, S. Muthukrishnan, Yossi Matias, S. Muthukrishnan, Suleyman Cenk Sahinalp, Jacob ZivSuleyman Cenk Sahinalp, Jacob Ziv

Presented by Genady GarberPresented by Genady Garber

AbstractAbstract

Theory of string algorithms play a fundamental Theory of string algorithms play a fundamental role inrole in• Information retrievalInformation retrieval• Data compressionData compression

This work consider one algorithmic problem from This work consider one algorithmic problem from each area.each area.

The algorithms rely on augmenting the The algorithms rely on augmenting the suffix suffix treetree (adding extra edges, resulting DAGs) (adding extra edges, resulting DAGs)

The algorithm construct these “suffix DAGs” and The algorithm construct these “suffix DAGs” and manipulate them to solve the problems.manipulate them to solve the problems.

IntroductionIntroduction

This paper presents two algorithmic problemsThis paper presents two algorithmic problems• Data CompressionData Compression• Information RetrievalInformation Retrieval

All these algorithms rely on the All these algorithms rely on the suffix treesuffix tree data data structurestructure

ST with suitably simple augmentations are very ST with suitably simple augmentations are very useful in string processing applicationsuseful in string processing applications

In described work the suffix tree was In described work the suffix tree was augmented with extra edges and additional augmented with extra edges and additional informationinformation

Problems and BackgroundProblems and Background

The Document Listing ProblemThe Document Listing Problem The HYZ Compression ProblemThe HYZ Compression Problem

The Document Listing ProblemThe Document Listing Problem Given a set of documents Given a set of documents T = {TT = {T11, . . . , T, . . . , Tkk}} Given a query pattern PGiven a query pattern P The problemThe problem

• To output a list of all documents containing P as a To output a list of all documents containing P as a substring substring (the standard problem can be solved in time, (the standard problem can be solved in time, proportional to number of occurrences of P in T. The goal is to proportional to number of occurrences of P in T. The goal is to solve the problem with a running time depending on the number solve the problem with a running time depending on the number of documents containing P)of documents containing P)

• To report the number of documents containing PTo report the number of documents containing P (existing (existing algorithm solves this problem in O(|P|) and is based on data algorithm solves this problem in O(|P|) and is based on data structures for computing Lowest Common Ancestor) structures for computing Lowest Common Ancestor)

May be used in “morbid “applications (discovering gene May be used in “morbid “applications (discovering gene homologies)homologies)

The (The ()-HYZ Compression Problem)-HYZ Compression Problem Given a binary string Given a binary string TT of length of length tt Need to replace disjoint Need to replace disjoint blocksblocks of size of size with desirably with desirably

shorter codewordsshorter codewords (to allow future perfect decompression)(to allow future perfect decompression)

Compression AlgorithmCompression Algorithm To compute the codeword cTo compute the codeword cjj for block for block jj we determine its context we determine its context

(the context of a block T[i : l] is the longest substring T[k : i - 1], k (the context of a block T[i : l] is the longest substring T[k : i - 1], k < i, of size at most < i, of size at most , T[k : l] occurs earlier in T) , T[k : l] occurs earlier in T)

The codeword cThe codeword cjj is the ordered pair { is the ordered pair {} where} where - length of the context of block - length of the context of block jj - rank of block - rank of block jj withwith respect to the context (according to respect to the context (according to

some predetermined order - lexicographic, etc.)some predetermined order - lexicographic, etc.) Intuition: similar symbols in data appear in similar contextsIntuition: similar symbols in data appear in similar contexts

The (The ()-HYZ Compression Problem )-HYZ Compression Problem (previous results)(previous results)

Case ofCase of = O(1) and = O(1) and is unboundedis unbounded• Average length of a codeword is shown to approach the Average length of a codeword is shown to approach the

conditional entropyconditional entropy for the block, within additive term of for the block, within additive term of cc11logH(C) + clogH(C) + c22 for constants c for constants c11 and cand c22, , provided that the input provided that the input is generated by a limited order Markovian source.is generated by a limited order Markovian source.

Case ofCase of >= loglog >= loglog tt and and = O(log = O(log tt))• This scheme also achieves the optimal This scheme also achieves the optimal

compression (in terms of CO)compression (in terms of CO)• Applies for all ergodic sourcesApplies for all ergodic sources

The problem stateThe problem state

Consider a set of document strings T = {TConsider a set of document strings T = {T11, T2, . . . T, T2, . . . Tkk}, }, of sizes tof sizes t11, t, t22, . . . , t, . . . , tkk..

The goal is to build a data structure, supporting the The goal is to build a data structure, supporting the following queries on an on-line pattern P of size p:following queries on an on-line pattern P of size p:

list query (the list of documents containing P)list query (the list of documents containing P) count query (the number of documents containing P)count query (the number of documents containing P)

Theorem 1Theorem 1 Given T and P, there is a data structure which responds Given T and P, there is a data structure which responds

to a count query in O(p) time, and a list query in to a count query in O(p) time, and a list query in O(pO(p loglog k + k + out)out) time, where time, where outout is the number of is the number of documents in T that contain P. documents in T that contain P.

The Suffix-DAG Data StructureThe Suffix-DAG Data Structure

Proof SketchProof Sketch Build the suffix-DAG of documents TBuild the suffix-DAG of documents T11, . . ., T, . . ., Tkk, in O(t) = O(, in O(t) = O(ttkk), ),

using O(t) spaceusing O(t) space The suffix-DAG of T, denoted by SD(T), contains the generalized The suffix-DAG of T, denoted by SD(T), contains the generalized

suffix tree, GST(T), of the set T at its core.suffix tree, GST(T), of the set T at its core. A GST(T) is defined to be the compact trie of all the suffixes of A GST(T) is defined to be the compact trie of all the suffixes of

each of the documents in T.each of the documents in T. Each leaf node Each leaf node ll in GST(T) is labeled with the list of documents in GST(T) is labeled with the list of documents

which have a suffix represented by the path from the root to which have a suffix represented by the path from the root to l.l. The substring represented by a path from the root to any given The substring represented by a path from the root to any given

node node nn denoted by P(n) denoted by P(n)

The Suffix-DAG Data Structure The Suffix-DAG Data Structure (cont)(cont)

The nodes of SD(T) are the nodes of GST(T)The nodes of SD(T) are the nodes of GST(T) The edges of SD(T) are of two types:The edges of SD(T) are of two types:

• the skeleton edges of SD(T) are the edges of GST(T)the skeleton edges of SD(T) are the edges of GST(T)• the supportive edges of SD(T) are defined as follows:the supportive edges of SD(T) are defined as follows:

for any nodes nfor any nodes n11 and nand n22 in SD(T) there is a pointer in SD(T) there is a pointer edge from nedge from n11 to nto n22 if and only if: if and only if:

– nn11 is an ancestor of n is an ancestor of n22 – among the suffix trees ST(Tamong the suffix trees ST(T11), ST(T), ST(T22), . . . , ST(T), . . . , ST(Tkk) there ) there

exists at least one , say ST(Texists at least one , say ST(Tii), which has two nodes , ), which has two nodes , nn1 , I 1 , I and nand n2 , I 2 , I such that such that

• P(nP(n11) = P(n) = P(n1 , i1 , i) ,P(n) ,P(n22) = P(n) = P(n2 , i2 , i))• nn1 , I 1 , I is the parent ofis the parent of nn2 , I2 , I

– such an edge is labeled with I for all relevant documents such an edge is labeled with I for all relevant documents in Tiin Ti

The Suffix-DAG Data Structure The Suffix-DAG Data Structure (cont.2)(cont.2)

In order to respond to the count and list queries, one In order to respond to the count and list queries, one of the standard data structures that support least of the standard data structures that support least common ancestor (LCA) queries on SD(T) in O(1) common ancestor (LCA) queries on SD(T) in O(1) time was built.time was built.

Also each internal node n of SD(T) containsAlso each internal node n of SD(T) contains• array that stores its supportive edges in pre-order array that stores its supportive edges in pre-order

fashionfashion• number of documents which include P(n) as substringnumber of documents which include P(n) as substring

ExampleExample

The independent suffix trees The independent suffix trees • T1: abcb ; T2: abca ; T3: ababT1: abcb ; T2: abca ; T3: abab

ST(T1)ST(T1)

aabb

ccbb

##

ccbb## ##

bb

ccbb

##

aabb

ccaa##

##

bbccaa##

ccaa

##

ST(T2)ST(T2)

aabb

aabb

####

aabb

####

ST(T3)ST(T3)

Example (cont.)Example (cont.)

The generalized suffix tree of the set of The generalized suffix tree of the set of documents {T1, T2, T3documents {T1, T2, T3}}

aa

bb

aabb

## bb##

cc##

aa##

##

bb cc

bb

##

cc

aa##

bb##

##aabb##

aa##

33 11 22 33 22 33 1,31,3 11 22 22 11

SK(T)SK(T)

Example (cont.2)Example (cont.2)

The suffix-DAG of the set of documents The suffix-DAG of the set of documents

aa

bb

aabb

## bb##

cc##

aa##

##

bb cc

bb

##

cc

aa##

bb##

##aabb##

aa##

33 11 22 33 22 33 1,31,3 11 22 22 11

SD(T)SD(T)

33

1,31,3

11

11

22

22

11

Example (cont.3)Example (cont.3)

The suffix-DAG of the set of documentsThe suffix-DAG of the set of documents

aa

bb

aabb

## bb##

cc##

aa##

##

bb cc

bb

##

cc

aa##

bb##

##aabb##

aa##

33 11 22 33 22 33 1,31,3 11 22 22 11

SD(T)SD(T)2,22,2

3,13,1

--

--

--

3,1,3,13,1,3,1

1,3,2,1,3,2,11,3,2,1,3,2,1

legend:legend:skeleton edgeskeleton edge

number ofnumber ofdocumentsdocumentsin the subtreein the subtreeand the and the pointer arraypointer array

33

33

22

33

33

22

22

Lemmas and ProofsLemmas and Proofs Lemma1Lemma1The suffix-DAG is sufficient to respond to the count queries in The suffix-DAG is sufficient to respond to the count queries in

O(p) time, and to list queries in O(p log k + out) time O(p) time, and to list queries in O(p log k + out) time Proof SketchProof Sketch

• To respond to count queries do as follows:To respond to count queries do as follows:– with P, trace down GST(T) until the highest level node n is with P, trace down GST(T) until the highest level node n is

reached for which P is a prefix of P(n)reached for which P is a prefix of P(n)– return the number of documents that contain P(n) as a return the number of documents that contain P(n) as a

substring substring • To respond to list queries do as followsTo respond to list queries do as follows

– locate n in SD(T) (defined above) and traverse SD(T) locate n in SD(T) (defined above) and traverse SD(T) backwards from n to the rootbackwards from n to the root

– at each node u on the path determine all supportive edges at each node u on the path determine all supportive edges out of u have their endpoints in the subtree rooted at nout of u have their endpoints in the subtree rooted at n

Lemmas and Proofs (cont.)Lemmas and Proofs (cont.) Complexity of list queriesComplexity of list queries

• the key observation is that all corresponding edges will form a the key observation is that all corresponding edges will form a consecutive segment.consecutive segment.

• the segment may be identified with two binary searches the segment may be identified with two binary searches (performing an LCA query, it takes O(1) time)(performing an LCA query, it takes O(1) time)

• maximum size of the array of supportive edges in any node is maximum size of the array of supportive edges in any node is at most k|at most k|| , where || , where || = O(1) ( size of alphabet), that means | = O(1) ( size of alphabet), that means this procedure takes O(log k) at each node uthis procedure takes O(log k) at each node u

• the output at all such segments may contain duplicate, but total the output at all such segments may contain duplicate, but total size of the output is O(out|size of the output is O(out||) = O(out), where out is the |) = O(out), where out is the number of occurrences of P in tnumber of occurrences of P in t

Lemmas and Proofs (cont.2)Lemmas and Proofs (cont.2) Lemma 2Lemma 2The suffix-DAG of the document set T can be constructed in The suffix-DAG of the document set T can be constructed in

O(t) time and O(t) spaceO(t) time and O(t) space Proof SketchProof Sketch

• The construction of GST(T) with all suffix links and data The construction of GST(T) with all suffix links and data structure are standardstructure are standard

• To complete SD(T) it is necessaryTo complete SD(T) it is necessary– to construct supportive edgesto construct supportive edges– to build supportive edge arrayto build supportive edge array– to explain, how the number of documents that include P(n) to explain, how the number of documents that include P(n)

is computedis computed

Lemmas and Proofs (cont.3)Lemmas and Proofs (cont.3) Proof Sketch (cont.)Proof Sketch (cont.)

• the supportive edge with label i can be built by emulating the supportive edge with label i can be built by emulating constriction for ST(Tconstriction for ST(Tii) ) (for each node is ST(T(for each node is ST(Tii), there is a ), there is a corresponding node in SD(T) with the appropriate suffix link)corresponding node in SD(T) with the appropriate suffix link)

• supporting edges buildingsupporting edges building– if there is supportive edge between nodes nif there is supportive edge between nodes n11 and n and n22 , then , then

either there is SE between neither there is SE between n11`̀( there is suffix link from n( there is suffix link from nii to n to nii`) `) and nand n22`, or there exist one intermediate node to which there is a `, or there exist one intermediate node to which there is a supportive edge from nsupportive edge from n11` ` (n(n22`̀).).

– The time to compute such nodes as length of string between The time to compute such nodes as length of string between nodes nnodes n11 and n and n22

• to compute the number of documents #(n), containing the substring to compute the number of documents #(n), containing the substring of n we need:of n we need:

– the number of supportive edges from n to its descendants the number of supportive edges from n to its descendants # # – the number of supportive edges to n from its ancestors the number of supportive edges to n from its ancestors ##

((n)n)((n)n)

Lemmas and Proofs (cont.4)Lemmas and Proofs (cont.4) Lemma 3Lemma 3For any node n, #(n) = SFor any node n, #(n) = Sn` n` E E children of n children of n #(n`)+#(n`)+

ProofProof• if a document Tif a document Tii includes the substring of more than one includes the substring of more than one

descendant of n, then there should exist a node in ST(Tdescendant of n, then there should exist a node in ST(Tii) ) whose substring is identical to that of nwhose substring is identical to that of n

• for any two supportive edges from n to nfor any two supportive edges from n to n11 and n and n22, , the path the path from n to nfrom n to n11 and the path from n to n and the path from n to n22 do not have any do not have any common edgescommon edges

)(# n ++)(# n

The Compression AlgorithmThe Compression Algorithm Compression scheme CCompression scheme C terms terms

• the input is a string T of size tthe input is a string T of size t• binary alphabetbinary alphabet

The schemeThe scheme• partition the T into contiguous substrings (blocks) of size partition the T into contiguous substrings (blocks) of size • replace each block by a corresponding codeword (function of replace each block by a corresponding codeword (function of

context)context)• the context of a block T[i : j] is the longest substring T[k : i-1] for the context of a block T[i : j] is the longest substring T[k : i-1] for

k < I, for which T[k :l] occurs earlier in T.k < I, for which T[k :l] occurs earlier in T.• if context exceeds if context exceeds it truncated for it truncated for rightmost characters rightmost characters• the codeword cthe codeword cj j is ordered pair is ordered pair where where

is the context sizeis the context size is the lexicographic order of block jis the lexicographic order of block j amongamong all possible substrings all possible substrings

of size of size immediately following earlier occurrences of context of immediately following earlier occurrences of context of block jblock j

Compression ExampleCompression Example T = 010011010T = 010011010 = 2 , = 2 , = 1 = 1 the context of block 9 is T[7 : 8] = 01the context of block 9 is T[7 : 8] = 01 the two substrings which follow earlier occurrences of the two substrings which follow earlier occurrences of

this context are T[3] = 0 and T[6] = 1 this context are T[3] = 0 and T[6] = 1 the lexicographic order of block 9 amount these the lexicographic order of block 9 amount these

substrings is 1substrings is 1

The Compression Algorithm(cont)The Compression Algorithm(cont) Theorem 2Theorem 2There is an algorithm to implement the compression There is an algorithm to implement the compression

scheme Cscheme Cwhich runs in O(twhich runs in O(t) time and requires ) time and requires O(tO(t) space, independent of ) space, independent of

ProofProofBuilding suffix tree we augment it as follows Building suffix tree we augment it as follows • for each node v we store an array of size for each node v we store an array of size in which for each in which for each

i = 1, . . ., i = 1, . . ., ::– store the number of distinct paths rooted at v of precisely i store the number of distinct paths rooted at v of precisely i

characters minus the number of such distinct paths of precisely characters minus the number of such distinct paths of precisely i-1 characters (the number may be negative)i-1 characters (the number may be negative)

The Compression The Compression Algorithm(cont.2)Algorithm(cont.2)

Lemma 4Lemma 4There is an algorithm to construct the augmented suffix There is an algorithm to construct the augmented suffix

tree of T in O(ttree of T in O(t) time) time ProofProofWhile inserting a new node into suffix tree, update the While inserting a new node into suffix tree, update the

subtree information of the ancestors of v which are at subtree information of the ancestors of v which are at most b characters higher than vmost b characters higher than v• number of such ancestors is at most number of such ancestors is at most • at most one of the at most one of the fields of information at any ancestor of v fields of information at any ancestor of v

needs to be updatedneeds to be updated


Lemma 5Lemma 5The augmented suffix tree is sufficient to compute the codeword The augmented suffix tree is sufficient to compute the codeword

for each block of input T in amortizedfor each block of input T in amortized O( )O( )

Proof SketchProof Sketch• The computation of The computation of can be performed by locating the node in can be performed by locating the node in

the suffix tree which represents the longest prefix of the context the suffix tree which represents the longest prefix of the context (can be achieved by using the suffix links in amortized O((can be achieved by using the suffix links in amortized O() ) time)time)

• The computation of The computation of – traverse the path between v and w (v represents the longest prefix traverse the path between v and w (v represents the longest prefix

of the context of block j, w, the descendant of v, represents the of the context of block j, w, the descendant of v, represents the longest prefix of the substring formed by concatenating of lock j longest prefix of the substring formed by concatenating of lock j and its context)and its context)

– during traversal, compute size of the relevant subtrees that during traversal, compute size of the relevant subtrees that smaler/greater then the substring represented by this pathsmaler/greater then the substring represented by this path

2


Theorem 3Theorem 3There is an algorithm to implement the compression method CThere is an algorithm to implement the compression method C for for

= log t and = log t and = log log t in O(t) time using O(t) space = log log t in O(t) time using O(t) space Proof SketchProof SketchFor any descendents wFor any descendents wii of v which are b chars apart from v, the of v which are b chars apart from v, the

DataStructure enables to compute lexicographic order of the path DataStructure enables to compute lexicographic order of the path between any wbetween any wii and v, allowing easy computation of codeword of and v, allowing easy computation of codeword of block of size brepresented by path between v and wblock of size brepresented by path between v and wii

The algorithm exploits the fact that the context size is bounded, and its The algorithm exploits the fact that the context size is bounded, and its seeks similarities between suffixes of the input up to a small size.seeks similarities between suffixes of the input up to a small size.


Lemma 6Lemma 6The augmented limited suffix tree of input T is sufficient to The augmented limited suffix tree of input T is sufficient to

compute the codeword for any input block j in O(b) timecompute the codeword for any input block j in O(b) time Proof SketchProof Sketch Given a block j of the input Given a block j of the input v represents its contextv represents its context w (descendant of v) represents substring: context(j) + block(j)w (descendant of v) represents substring: context(j) + block(j) b = logb = log loglog t => the maximum number of elements in the search t => the maximum number of elements in the search

data structure for v is = O(logdata structure for v is = O(log t)t) there is a simple data structure that maintains k elements and there is a simple data structure that maintains k elements and

compute the rank of any given element in O(log k) timecompute the rank of any given element in O(log k) time the lexicographic order of node w in only O(log log t) = O(the lexicographic order of node w in only O(log log t) = O() time ) time

may be computed (in future work).may be computed (in future work).

2


Lemma 7Lemma 7The augmented limited suffix tree of input T can be built The augmented limited suffix tree of input T can be built

and maintained in O(t) timeand maintained in O(t) time Proof SketchProof Sketch

• The depth of augmented limited suffix tree is bounded by log t => The depth of augmented limited suffix tree is bounded by log t => the total number of nodes in the tree is only O(t), allowing to adapt the total number of nodes in the tree is only O(t), allowing to adapt ST construction in O(t) time - without being penalized for building a ST construction in O(t) time - without being penalized for building a suffix trie rather than the suffix tree.suffix trie rather than the suffix tree.

• Because of each node v is inserted to the search data structure of at Because of each node v is inserted to the search data structure of at most one of its ancestors, it is possible to construct and maintain the most one of its ancestors, it is possible to construct and maintain the search data structure of all nodes in O(t) time, => the total number search data structure of all nodes in O(t) time, => the total number of elements maintained by all search data structures is O(t) of elements maintained by all search data structures is O(t)

• The insertion time of an element e to a search DS is O(1)The insertion time of an element e to a search DS is O(1)• As the total number of of nodes to be inserted is the DS is bounded As the total number of of nodes to be inserted is the DS is bounded

by O(t), it may be shown, that the total time for insertion of nodes in by O(t), it may be shown, that the total time for insertion of nodes in the search DS is O(t) the search DS is O(t)

Results and conclusionsResults and conclusions

Document listing problemDocument listing problem• Processing Processing kk documents in linear time ( O(documents in linear time ( O(ii ttii) ) ) )

and spaceand space• Time to answer a query with pattern P is Time to answer a query with pattern P is

O(O( |P||P| loglogkk + + outout))

The fastest known algoritm runs in time proportional The fastest known algoritm runs in time proportional to number of occurrences of the pattern in all to number of occurrences of the pattern in all documents. documents.

Results and conclusions (cont.)Results and conclusions (cont.)

((,,)-HYZ compression problem)-HYZ compression problem unbounded unbounded , , complexity O(tcomplexity O(t))

– gives linear time for gives linear time for = O(1)= O(1) The only previously known algorithm, where for The only previously known algorithm, where for

the author presents an O(tthe author presents an O(t) time algorithm, ) time algorithm, and for unbounded and for unbounded this running time is O( ) this running time is O( )

= O(log t), = O(log t), = log = log loglog t, complexity O(t)t, complexity O(t) There is no any previously known algorithmsThere is no any previously known algorithms

t2

Augmenting Suffix Trees, with Applications

Documents

Transcript of Augmenting Suffix Trees, with Applications