Coherency Sensitive Hashing (CSH)Simon Korman and Shai AvidanDept. of Electrical Engineering Tel Aviv UniversityICCV2011 | 13th International Conference on Computer Vision

Outline• Introduction• Locality Sensitive Hashing for Finding Nearest Neighbors• Coherency Sensitive Hashing • Experiments• Conclusions

Introduction(1/2)Patch : k*k block

1920

1080

Find the closest patch

linear search(search every patch one by one)

query

Introduction(2/2)• (Streaming) Massive Data Sets => High Dimensional Vectors

• E.g. 8*8 patch => v = [ v1, v2, ..., vi , …, vN ] , dimension N = 64• Linear search = find nearest neighbor

• For very large databases of high-dimensional items• Time-consuming• Needs to find Approximate Nearest Neighbors (ANN) for each

patch in real time.• Curse of dimensionality

• Existing ANN methods include trees and hashes.• KD-trees• Locality-Sensitive Hashing(LSH)

• A collision occurs when two points hash to the same value

Hash tablebucket

Look up table

Projection = hash function

Repeat L times=>the closest point will collide most times

Random line

Locality Sensitive Hashing for Finding Nearest Neighbors(1/2)

Locality Sensitive Hashing for Finding Nearest Neighbors(2/2)

• Hash function:

• a is d-dimensional random vector• r is predefine integer

• Constant width of each quantization bin

• b is random value from range [0, r]• To balance quantization error

• v is the original vector

1. indexing

2. search

Coherency Sensitive Hashing (CSH)

s=1, 4x4 kernel

White:1, Black:-1

Coherency Sensitive Hashing (CSH)

4.2.1 Candidate Creation• Patch a, a1,a2 of image A; Patch b, b1,b2 of image B :

Each entry can store k patches from each image=>total k+2*(k+1)+k=4k+2 candidates

4.2.2 Candidate Ranking• Given the candidate set (of size 4k + 2), to find the

nearest one.• Main overall time consumer• Approximate the process, have a little impact, greatly reduce time.

• Use Walsh Hadamard (WH) projections• Already computed in the indexing stage.• Accumulating the projections of the differences of patches on the

WH kernels.

Experiments• We collected 133 pairs of images, taken from 1080p HD

Experiments• We computed the exact nearest neighbor match to serve as a ground

truth.• A novel algorithm PatchMatch [4] to compare :

• Not as accurate as LSH or KD-trees.• So fast. The key to this speedup is spatial coherent.

[4] C. Barnes, E. Shechtman, A. Finkelstein, and D. B. Goldman.PatchMatch: A randomized correspondence algorithm for structural image editing. In SIGGRAPH, 28(3), 2009.

Error : not the same match with ground truth

Image Reconstruction• Reconstruct image A, use image B• Such reconstructions are very common in many

applications.• image editing (e.g. retargetting, inpainting), image denoising and

super-resolution…• It simply replaces each pixel with the average of the

corresponding pixels.

Image A Image B

More results on web : http://www.eng.tau.ac.il/~simonk/CSH/index.html

Conclusions• We proposed an algorithm for computing ANN fields

termed Coherency Sensitivity Hashing.• Follows LSH search scheme • But combines image coherency cues

• It was shown to be faster and more accurate than PatchMatch.