Efficient Approximation of Edit Distance

Efficient Approximation of Edit Distance

Robert Krauthgamer, Weizmann Institute of Science

SPIRE 2013


GenericSearchEngine

Given two strings xn, ym:

ed(x,y) = minimum number of character operations (insertion/deletion/substitution) that transform x to y.

Edit Distance (Levenshtein distance)

Applications:

• Computational Biology

• Text processing• Web search

Examples:ed( banana , ananas ) = 2

ed(00000, 1111) = 5

For simplicity: m = n.

2


Dynamic Programming Algorithm Compute ed(x,y) for input x,y n

O(n2) time by dynamic programming [WF’74] O(n2/log2 n) time when ||·O(1) [MP’80]

b a n a n a

a

n

a

n

a

s

2

56

1

1

1

1

11

222

2222

22

2

22

3

33

33

3 33 44

44

45

5

D(i,j)= min

D(i-1, j-1) , if x[i]=y[j]

D(i, j-1) + 1

D(i-1, j) + 1

D(i,j) = ed( x[1:i], y[1:j] )

Faster algorithms?

3

Focus of This Talk Approximating edit distance

Multiplicatively: ed(x,y) · output · A¢ed(x,y) Decision version: ed(x,y) · r or ed(x,y) > A¢r

Different computational models RAM, Sampling and query complexity, Sketching, (Streaming) Interactions (is it surprising?), Techniques

Variants of the problem

Efficient Approximation of Edit Distance 4

RAM Model: Sampling Idea 1: quickly estimate ed(x,y) by sampling a few positions Intuition:

If ed(x,y) is small, then “many” large blocks should “match” “Test” this by reading few (randomly chosen) blocks Apply this idea recursively (inside blocks)

Theorem [Batu-Ergun-Kilian-Magen-Raskhodinkova-Rubinfeld-Sami’03]: Factor nc “weak” approximation in sublinear time.

Obstacles: “Block match” means both “similar pattern” and “similar location” Argue that if and only if ed(x,y) is small then …

Can only distinguish ed(x,y)·n/(8A) from ed(x,y)>n/8.


Best approximation in (near) linear time?

5

Learn from Past Success Suppose x,y are permutations

Every symbol of appears exactly once Consider transpositions=block moves (“block edit distance”)

No Insert/delete (unreasonable), no substitution (not needed) Example: bed(0123456789, 0457689123)=2

An easy estimate (based on breakpoints) Compute Sx = {all length 2 substrings of x} = {x[i:i+1] | i=1,…,n-1}

Lemma: bed(x,y) · ½ |SxΔSy| · 3 bed(x,y) Proof idea: Fix x (wlog identity), let y=

Each block move “creates” at most 3 new breakpoints Break y at breakpoints, and move (rearrange) the blocks to get x

Can compute |SxΔSy| in linear time!! Best approximation known in poly-time is 1.375 [Elias-Hartman’06]


A B C D

Open: better approximation in linear-time?

6

Reduction to Hamming Distance |SxΔSy| = Hamming distance between their characteristic vectors

In fact, each vector has ||2=n2 coordinates, but only n-1 are non-zero

We thus obtain f : Permutations {0,1}n2 such that8x,y, bed(x,y) · ½ ||f(x)-f(y)||1 · 3 bed(x,y).

Such a reduction from one metric space (BED on permutations) to another (L1) is called an embedding. This one has distortion D=3. Known lower bound: distortion into L1 must be ¸ 4/3 [Polak-K.’12]


More benefits of “good” embeddings?

A sweet spot of fruitful interaction between Math/Geometry (“comparing” metric spaces using embeddings) and CS/Algorithms (solving new problems by “reducing” to old ones)

7

Sketching Model Idea: “summarize” each string separately, then estimate ed(x,y) only

from the short sketches s(x),s(y). Possible at all??

YES for Hamming distance, and even L1/L2 [Indyk-Motwani’98, Kushilevitz-Ostrosvky-Rabani’00] Approximation factor A=1+ε using sketch size O(ε-2) bits It’s essentially a “dimension reduction” [Johnson-Lindenstrauss’86] Achieved by projection on (inner product with) random direction in space

Consequently, YES also for block edit distance on permutations:

Applies whenever there is an embedding into L1 !!Efficient Approximation of Edit Distance

BED on perm. Hamming O(ε-2) bits sketchf s

distort. D=3 approx. 1+ε

8

Applications of Sketching Input: large database M, with |M| strings of length n each. Output: all pairwise distances or closest pair (BED on perm)

Naively: in time O(|M|2 n) Sketching [3+ε approx., decision version]: sketch each string, then

estimate all pairs in time O(|M|n + |M|2/ε2)

Practical viewpoint: filteration, i.e., fast pruning of “bad” pairs

Works similarly for Nearest Neighbor Search (NNS): Reduce NNS for permutations under BED, to NNS for Hamming (L1)


Q1. More embeddings?Q2. Sketching directly?Q3. Lower bounds?

9


Embedding ED on PermutationsTheorem [Charikar-K.’06]: Edit distance on permutations of length n

embeds into L1 with distortion O(log n).Proof. Define where

Lemma 1: ||f(P)-f(Q)||1 ≤ O(log n) ed(P,Q) Suppose Q is obtained from P by moving one symbol, say ‘s’

General case then follows by applying triangle inequality on P,P’,P’’,…,Q

Total contribution of coordinates s2{a,b} is 2k (1/k) ≤ O(log n) other coordinates is k k(1/k – 1/(k+1)) ≤ O(log n)

Intuition: sign(fa,b(P)) is indicator for “a appears before b” in P Thus, |fa,b(P)-fa,b(Q)| “measures” if {a,b} is an inversion in P vs. Q

10


Embedding ED on Permutations (2)Recall where

Lemma 1: ||f(P)-f(Q)||1 ≤ O(log n) ed(P,Q)

Lemma 2: ||f(P)-f(Q)||1 ¸ ½ ed(P,Q) Assume wlog that P=identity Edit Q into an increasing sequence (thus into P) using quicksort:

Choose a random pivot, Delete all characters inverted wrt to pivot Repeat recursively on left and right portions

Now argue ||f(P)-f(Q)||1 ¸ E[ #quicksort deletions ] ¸ ½ ed(P,Q)

QED

Surviving subsequence is increasing

ed(P,Q) ≤ 2 #deletions

For every inversion (a,b) in Q: Pr[a deleted “by” pivot b] ≤ 1/|Q-1[a]-Q-1[b]+1| ≤ 2 |fa,b(P) – fa,b(Q)|

11

Embedding Edit Distance Theorem [Ostrovsky-Rabani’05]: Edit distance on all strings (not

only permutations) embeds into L1 with distortion 2O(√log n).

Previously, distortion nc was known [BarYossef-Jayram-K.-Kumar’04, Batu-Ergun-Sahinalp’06]

Clever recursive method to match blocks much more accurately Penalizes both pattern and location errors

Not very fast (quadratic time), but influenced later work on near-linear time algorithms [Andoni-Onak’09, Andoni-Onak-K.’10]

Immediate consequences: NNS algorithms for edit distance Sketching


Lower Bounds Theorem [Khot-Naor’05, K.-Rabani’06]: Embedding edit distance

into L1 requires distortion Ω(log n) Main technique: Fourier analysis [Kahn-Kalai-Linial’88] L1 embedding $ sparsest-cuts $ Boolean functions f:{0,1}n {0,1}

Stronger assertion: O(1)-size sketches for edit distance require Ω(log n) approximation, even only for permutations [Andoni-K.’06] Actually tradeoff between approximation and sketch-size Techniques: communication complexity and Fourier analysis reduce the

problem to sketches that are linear functions (of their input x)


Q2’. Sketching vs embedding?

13

RAM Model: Asymmetric Sampling Idea 1’: Read all of y, and sampled positions of x

Motivations: Better chances to “obtain” information Which y’s are easier/harder? Sampling issues:

Focus on query complexity bounds (tight?) Adaptive vs non-adaptive queries Queries depend on y?

Use dynamic programming in time O(n1+ε)?


x

y

14


Asymmetric Sampling Results[Andoni-Onak-K.’10] Problem: Decide ed(x,y) ≥ n/10 vs ed(x,y) ≤ n/(10A) Complexity = #queries into x (unlimited access to y)

n1-ε A

Θ(log n)Θ(log2 n)

Θ(log3 n)

Θ(logt n)#queries

n1/2-ε n1/2n1/3n1/4n1/t-εn1/(t+1)

Approximation A: (log n)O(1/ε)

# Queries: O(nε)Ω(nε/loglog n)

[n1/(t+1), n1/t-ε]O(logt n)Ω(logt n)

15


Overview of Upper Bound Theorem 1: Can distinguish ed(x,y) ≥ n/10 vs ed(x,y) ≤ n/(10A) for

A=(log n)O(1/ε) approximation with nε queries into x (for any ε>0).

Proof structure:1. Characterize edit by “tree-distance” Txy

Parameter b≥2 (degree) Txy ≈ ed(x,y) up to 6b*log n factor

2. Prune the tree to subsample x

x1 x2 xn

b

sampled positions in x

16


Step 1: Tree Distance Partition x into b blocks, recursively, for h=logbn levels

x[1:n]

x[1:⅓n] x[⅔n:n]

…

x[1] x[2] x[3]

x[⅓n:⅔n]

y[1:n]

y[u:u+⅓n]

x[s:s+⅓n]

Ti(s,u) = tree-distance between x[s:s+ℓi] and y[u:u+ℓi] where ℓi is the block-length at level i

17


Tree Distance: Recursive Definition Recall Ti(s,u) = tree-distance between x[s:s+ℓi] and y[u:u+ℓi]

Base case: Th(s,u)=Hamming(x[s],y[u]) Output: Txy=T0(s=1,u=1) x[s:s+ℓi]

y[u:u+ℓi]

r0

x

y

18


Tree Approximates Edit Distance Lemma: Txy≈ed(x,y) up to 6b*logbn factor.

Hierarchical decomposition inspired by earlier approaches [BEKMRRS’03, OR’05] All had approximation recurrence of the type

A(n) = c*A(n/b) + bfor c≥2

Solves to A(n) ≥ 2√log n factor for every choice of b

Our characterization has no multiplicative loss (c=1):A(n) = A(n/b) + b

Analysis inspired by algorithms for smoothed instances [Andoni-K.’08]

19


Step 2: Compute the Tree Distance For b=2, tree-distance gives O(log n) approximation!

BUT know only how to compute T-distance in O(n2) time

Instead, for b=(log n)1/ε, can prune the tree to nO(ε) nodes, and approximate T-distance within factor 1+ε

Pruning: subsample (log n)O(1) children out of each node Works only when ed(x,y) ≥ (n) Generally, must subsample

the tree non-uniformly, using the Precision Sampling Lemma

b

sampled positions in x

20


Key tool: non-uniform sampling Goal:

For unknown a1, a2, …an[0,1] Estimate their sum, up to an additive constant error Using only “weak” estimates a1, a2, …an

Sum Estimator Adversary

0. fix distribution U1. Fix a1,a2,…an (unknown)

2. pick “precisions” ui

(our algorithm: ui~U[0,1] i.i.d.)3. provide a1,a2,…an

s.t. |ai-ai|<1/ui4. report S=S(a1,…,u1,…) with |S – ∑ai | < 1.

21


Precision Sampling Goal: estimate ∑ai from {ai} s.t. |ai-ai|<1/ui. Precision Sampling Lemma: Can achieve WHP

additive error 1 and multiplicative error 1.5 with expected precision Eu_i~U[ui]=O(log n).

Inspired by a technique from [IW’05] for streaming (Fk moments) In fact, PSL gives simple & improved algorithms for Fk moments,

cascaded (mixed) norms, ℓp-sampling problems [AKO’11]

Also distant relative of Priority Sampling [DLT’07]

22


Precision Sampling for Edit Distance Apply Precision Sampling to the tree from the characterization

recursively at each node If a node has very weak precision, can trim the entire sub-tree

23

Fast Approximation Algorithm Theorem [Andoni-Onak-K.’10]: Can approximate ed(x,y) within

factor (log n)O(1/ε) using nε queries to x and in time n1+ε (for any ε>0).

Exponential improvement over previous factor 2O(√log n) [Andoni-Onak’09] Asymmetric sampling approach, implemented faster by data structure

tricks Sampling is non-adaptive, independent of y


Smoothed Instances Smooth Instance (x,y) constructed by:

Start with arbitrary x*,y*2{0,1}n and their optimal alignment A* Replace each position w/probability p by random bit, but respect A*

Theorem [Andoni-K.’08]: Can approximate ed(x,y) within constant factor, in smoothed runtime that is (whp) near-linear n1+ε. Some extensions to sublinear time

Techniques: Match blocks of length L=O(1/p¢log n) that have edit distance ·εL.

A known heuristic technique (e.g. PatternHunter) To find block matches quickly, we use naive NNS algorithm

Because of smoothing, blocks are likely to be distinct (and even far), so modulo overlaps between blocks, we “effectively” have permutations


Open: Better time n¢polylog(n)? Approximation independent of p?

25

Variants of Edit Distance Edit distance with block operations

Admits O(log n¢log*n) approximation in near-linear time, via embedding into L1 [Muthukrishnan-Sahnialp’00,Cormode-Muthukrishnan’02]

Open: Distortion lower bounds? Better approximation in polytime?

Edit distance between trees (generalizes strings) Basic operations: insert/delete/relabel vertex Can be computed in O(n3) time [Demaine-Mozes-Rossman-Weimann’07] Open: Embedding?

Edit distance with “rich” alphabet Can model shape matching [Klein-Tirthapura-Sharvit-Kimia’00] Challenge: Cost of basic operation varies with symbols


Conclusion Having multiple computational models is fruitful

New ideas, techniques, viewpoints, applications can come full circle Lower bounds —in certain models — highlight limitations of methods Explore which instances are easy/hard

“Asymmetric algorithms” can work well for symmetric problems

Connections to other fields (sampling, embeddings, communication complexity, Fourier analysis) and computational problems (NNS)

Had much progress, but still many gaps, and much more to go


Thank You! 27

Efficient Approximation of Edit Distance

Documents

Transcript of Efficient Approximation of Edit Distance