2012 10-26 image-videoanalysis_lecture06
description
Transcript of 2012 10-26 image-videoanalysis_lecture06
![Page 1: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/1.jpg)
Анализ изображений и видео
Наталья Васильева [email protected] HP Labs Russia
26 октября 2012, Computer Science Center
Лекция 6: Поиск по подобию. Поиск нечетких дубликатов.
![Page 2: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/2.jpg)
2 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Поиск по визуальному подобию: описание
Задача: найти картинки, похожие на запрос визуально и семантически
Похожи Отдаленно Не похожи
![Page 3: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/3.jpg)
3 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Поиск нечетких дубликатов: описание
Задача: найти группы нечетких дублей в коллекции
Дубли Не дубли
![Page 4: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/4.jpg)
4 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Традиционная архитектура систем CBIR Поиск по содержанию, по визуальному подобию Content Based Image Retrieval (CBIR)
![Page 5: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/5.jpg)
5 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Индексирование изображений
Fig. credit: http://www.vlfeat.org/
Построение векторов
признаков
Помещение векторов в хранилище,
построение индекса
![Page 6: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/6.jpg)
6 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Поиск изображений
• Данные высокой размерности
− Среднее значение для каждого цветового канала: dim(x) = 3
− Гистограмма по одному каналу: dim(x) = 256 − ICA-based texture: dim(x) = 21*30 − SIFT: dim(x) = 128 * N
• Поиск по подобию, поиск ближайших соседей
Необходим способ так организовать хранение, чтобы можно было по запросу быстро и эффективно находить в хранилище наиболее похожие объекты
![Page 7: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/7.jpg)
7 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Каждый вектор – точка в пространстве высокой размерности (например, dim=128 для дескрипторов SIFT)
Slide credit: K. Grauman, B. Leibe
Индексирование
![Page 8: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/8.jpg)
8 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Соседние точки в пространстве признаков соответствуют близким векторам, которые соответствуют подобию изображений
Fig. credit: A. Zisserman
Индексирование
![Page 9: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/9.jpg)
9 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Поиск ближайших соседей
ε-Nearest Neighbor Search (ε - NNS)
Для запроса q найти подмножество всех точек p ∈ P (P – метрическое пространство), таких что:
Можно обобщить до k–nearest neighbor search ( K >1) (Top-K)
),(min)1(),( rqdistpqdistPr∈
+≤ ε
![Page 10: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/10.jpg)
10 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Поиск ближайших соседей
k–Nearest Neighbor Search (Top-k search)
Для запроса q найти подмножество (P – метрическое пространство): PA⊆
),(),(:\,|| 2121 pqdistpqdistAPpApkA ≤∈∀∈∀=
q
![Page 11: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/11.jpg)
11 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Для запроса q Вычислить dist(q,p) для всех точек p ∈ P
Наивный подход
O(N) для каждого запроса! (|P|=N)
![Page 12: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/12.jpg)
12 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Некоторые индексные структуры
Деревья –R-tree – подходит только для небольших размерностей (2D), пересечение узлов –k-D tree – неэффективны в случае ассиметричных данных высокой размерности –Vantage-Point tree (VP tree) (один из вариантов метрических деревьев)
Обратный индекс –Vocabulary tree
Хэширование
–LSH
![Page 13: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/13.jpg)
13 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
KD-Tree Construction
Pt X Y
1 0.00 0.00 2 1.00 4.31 3 0.13 2.85 … … …
We start with a list of n-dimensional points.
Slide credit: Brigham Anderson
![Page 14: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/14.jpg)
14 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
KD-Tree Construction
Pt X Y
1 0.00
0.00
3 0.13
2.85
… … …
We can split the points into 2 groups by choosing a dimension X and value V and separating the points into X > V and X <= V.
X>.5
Pt X Y 2 1.00 4.31 … … …
YES NO
Slide credit: Brigham Anderson
![Page 15: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/15.jpg)
15 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
KD-Tree Construction
Pt X Y
1 0.00
0.00
3 0.13
2.85
… … …
We can then consider each group separately and possibly split again (along same/different dimension).
X>.5
Pt X Y 2 1.00 4.31 … … …
YES NO
Slide credit: Brigham Anderson
![Page 16: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/16.jpg)
16 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
KD-Tree Construction
Pt X Y
3 0.13
2.85
… … …
We can then consider each group separately and possibly split again (along same/different dimension).
X>.5
Pt X Y 2 1.00 4.31 … … …
YES NO
Pt X Y
1 0.00
0.00
… … …
Y>.1 NO YES
Slide credit: Brigham Anderson
![Page 17: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/17.jpg)
17 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
KD-Tree Construction
We can keep splitting the points in each set to create a tree structure. Each node with no children (leaf node) contains a list of points.
Slide credit: Brigham Anderson
![Page 18: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/18.jpg)
18 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
KD-Tree Construction
We will keep around one additional piece of information at each node. The (tight) bounds of the points at or below this node.
Slide credit: Brigham Anderson
![Page 19: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/19.jpg)
19 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
KD-Tree Construction
Use heuristics to make splitting decisions: Which dimension do we split along? Widest Which value do we split at? Median of value of that split dimension for the points. When do we stop? When there are fewer then m points left OR the box has hit some minimum width.
Slide credit: Brigham Anderson
![Page 20: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/20.jpg)
20 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
We traverse the tree looking for the nearest neighbor of the query point.
Slide credit: Brigham Anderson
![Page 21: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/21.jpg)
21 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
Examine nearby points first: Explore the branch of the tree that is closest to the query point first.
Slide credit: Brigham Anderson
![Page 22: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/22.jpg)
22 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
Examine nearby points first: Explore the branch of the tree that is closest to the query point first.
Slide credit: Brigham Anderson
![Page 23: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/23.jpg)
23 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
When we reach a leaf node: compute the distance to each point in the node.
Slide credit: Brigham Anderson
![Page 24: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/24.jpg)
24 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
When we reach a leaf node: compute the distance to each point in the node.
Slide credit: Brigham Anderson
![Page 25: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/25.jpg)
25 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
Then we can backtrack and try the other branch at each node visited.
Slide credit: Brigham Anderson
![Page 26: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/26.jpg)
26 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
Each time a new closest node is found, we can update the distance bounds.
Slide credit: Brigham Anderson
![Page 27: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/27.jpg)
27 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
Using the distance bounds and the bounds of the data below each node, we can prune parts of the tree that could NOT include the nearest neighbor.
Slide credit: Brigham Anderson
![Page 28: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/28.jpg)
28 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
Using the distance bounds and the bounds of the data below each node, we can prune parts of the tree that could NOT include the nearest neighbor.
Slide credit: Brigham Anderson
![Page 29: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/29.jpg)
29 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Nearest Neighbor with KD Trees
Using the distance bounds and the bounds of the data below each node, we can prune parts of the tree that could NOT include the nearest neighbor.
Slide credit: Brigham Anderson
![Page 30: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/30.jpg)
30 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Spheres vs. Rectangles
• ↑↑⇒ ratiolitydimensiona1e)Volume(Cub
ere)Volume(Sphratio ≤= •
• relative distances
![Page 31: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/31.jpg)
31 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Метрические деревья
![Page 32: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/32.jpg)
32 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Vantage point method
σµ −≤),( qvd
σµ +>),( qvd
σσµµ =−−>−≥ |)(||),(),(|),( qvdpvdpqd
σµσµ =−+>−≥ |))(||),(),(|),( pvdqvdpqd
>∈ Sp
≤∈ Sp
![Page 33: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/33.jpg)
33 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Как выбрать опорные точки?
• Минимальная длина разделяющей дуги
• “Угловые” точки пространства
• Чтобы дерево было сбалансированным
• Максимум среднеквадратичного отклонения
![Page 34: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/34.jpg)
34 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Построение VP-tree
![Page 35: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/35.jpg)
35 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Обратный индекс
Для текстов: структура для эффективного поиска страниц, на которых упомянуто слово…
Наша цель: найти изображения, которым соответствуют заданные вектора признаков.
Чтобы использовать эту идею, нам надо построить из признаков словарь “визуальных слов”.
![Page 36: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/36.jpg)
36 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
«Визуальные слова»: основная идея
Извлечем локальные признаки из нескольких изображений…
Пространство локальных признаков (например, SIFT-дескрипторов)
Slide credit: D. Nister
![Page 37: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/37.jpg)
37 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
«Визуальные слова»: основная идея
Slide credit: D. Nister
Извлечем локальные признаки из нескольких изображений…
Пространство локальных признаков (например, SIFT-дескрипторов)
![Page 38: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/38.jpg)
38 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Slide credit: D. Nister
«Визуальные слова»: основная идея
Извлечем локальные признаки из нескольких изображений…
Пространство локальных признаков (например, SIFT-дескрипторов)
![Page 39: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/39.jpg)
39 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
«Визуальные слова»: основная идея
Извлечем локальные признаки из нескольких изображений…
Slide credit: D. Nister
Пространство локальных признаков (например, SIFT-дескрипторов)
![Page 40: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/40.jpg)
40 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
«Визуальные слова»: основная идея
Извлечем локальные признаки из нескольких изображений…
Slide credit: D. Nister
Пространство локальных признаков (например, SIFT-дескрипторов)
![Page 41: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/41.jpg)
41 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Slide credit: D. Nister
![Page 42: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/42.jpg)
42 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Slide credit: D. Nister
![Page 43: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/43.jpg)
43 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Отобразим дескрипторы в токены/слова путем квантования пространства признаков
Квантование при помощи кластеризации пространства Центры кластеров – «леммы» «визуальных слов»
Пространство признаков
«Визуальные слова»: основная идея
Slide credit: K. Grauman, B. Leibe
![Page 44: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/44.jpg)
44 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Отобразим дескрипторы в токены/слова путем квантования пространства признаков
Каждый вектор признаков отображаем в «слово» – находим ближайший центр кластера Описваем изображение через набор «визуальных слов» Пространство
признаков
«Визуальные слова»: основная идея
Slide credit: K. Grauman, B. Leibe
![Page 45: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/45.jpg)
45 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Фрагментам из одной группы соответствует одно и то же визуальное слово
Fig. credit: Sivic & Zisserman
Визуальные слова
![Page 46: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/46.jpg)
46 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Номер слова
Список номеров изображений
Изображение №5 Изображение №10
Slide credit: K. Grauman, B. Leibe
Обратный индекс изображений по визуальным словам
![Page 47: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/47.jpg)
47 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Bag of visual words «Мешок визуальных слов»
Fig. credit: Fei-Fei Li
![Page 48: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/48.jpg)
48 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Bag of visual words «Мешок визуальных слов»
Fig. credit: Fei-Fei Li
Словарь визуальных слов
![Page 49: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/49.jpg)
49 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Основные вопросы:
• Как выбрать фрагменты изображений? • Какой алгоритм квантования/кластеризации использовать? • С учителем / без учителя • Как выбрать множество изображений, чтобы получить универсальный словарь? • Размер словаря, число слов?
Как построить словарь?
Slide credit: K. Grauman, B. Leibe
![Page 50: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/50.jpg)
50 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Dense, uniformly Sparse, at interest points
Randomly
Multiple interest operators
• To find specific, textured objects, sparse sampling from interest points often more reliable.
• Multiple complementary interest operators offer more image coverage.
• For object categorization, dense sampling offers better coverage.
[See Nowak, Jurie & Triggs, ECCV 2006]
Выбор фрагментов
Slide credit: K. Grauman, B. Leibe
![Page 51: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/51.jpg)
51 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Методы квантования/кластеризации
Традиционные решения: • k-means, agglomerative clustering, mean-shift,… Использование иерархической кластеризации, организация словаря в виде дерева: • Vocabulary tree [Nister & Stewenius, CVPR 2006]
Более быстрая индексация и поиск при большем размере словаря!
![Page 52: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/52.jpg)
52 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Построение дерева
Slide credit: David Nister Slide credit: D. Nister
![Page 53: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/53.jpg)
53 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Индексирование
Slide credit: D. Nister
![Page 54: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/54.jpg)
54 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Индексирование
Slide credit: D. Nister
![Page 55: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/55.jpg)
55 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Индексирование
Img1, 1 Img2, 1 Img1, 2
Slide credit: T. Tommasi
• Каждый узел дерева – обратный индекс
• Каждый узел дерева хранит идентификаторы изображений и частоту соответствующего слова в изображении (или tf-idf)
![Page 56: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/56.jpg)
56 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Поиск
Slide credit: D. Nister
![Page 57: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/57.jpg)
57 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Vocabulary Tree: Performance
Evaluated on large databases • Indexing with up to 1M images Online recognition for database of 50,000 CD covers • Retrieval in ~1s
Find experimentally that large vocabularies can be beneficial for recognition
[Nister & Stewenius, CVPR’06]
Slide credit: K. Grauman, B. Leibe
![Page 58: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/58.jpg)
58 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Locality Sensitive Hashing (LSH): motivation
• Similarity Search over High-Dimensional Data − Image databases, document collections etc
• Curse of Dimensionality − All space partitioning techniques degrade to linear search for
high dimensions
• Exact vs. Approximate Answer − Approximate might be good-enough and much-faster − Time-quality trade-off
Need sub-linear dependence on the data-size for high-dimensional data!
![Page 59: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/59.jpg)
59 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
LSH: Key idea Hash the data-point using several LSH functions so that probability of collision is higher for closer objects
Indyk & Motwani, 1998
Fig. credit: P. Indyk, K. Grauman, B. Leibe
![Page 60: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/60.jpg)
60 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Slide credit: P. Indyk
![Page 61: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/61.jpg)
61 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
LSH (альтернативное определение)
Пусть P множество объектов (точек в пространстве выскокой размерности), sim – функция подобия объектов: Тогда схема LSH – это семейство хэш-функций H, имеющих распределение D, таких что при выборе функции h ∈ H согласно распределению D:
]1,0[: →×PPsim
PpqpqsimphqhHh ∈∀==∈ ,),()]()([Pr
![Page 62: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/62.jpg)
62 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
LSH: индексирование
Вход: • Set of N points { p1 , …….. pn } • L – длина хэша • M – число хэш-таблиц
Выход: • Хэш-таблицы Ti , i = 1 , 2, …. M
Алгоритм:
Foreach i = 1 , 2, …. M Initialize Ti with a random hash function gi(p)=<h1(p), h2(p),…hk(p)> Foreach i = 1 , 2, …. M Foreach j = 1 , 2, …. N Store point pj on bucket gi(pj) of hash table Ti
![Page 63: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/63.jpg)
63 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
LSH: индексирование
g1(pi) g2(pi) gM(pi)
TM T2 T1
pi
P
![Page 64: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/64.jpg)
64 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
LSH: поиск билижайшего соседа
Вход: • Запрос q
Выход: • Приблизительный ближайший сосед p для q
Алгоритм: Foreach i = 1 , 2, …. M Foreach { p: точа из ячейки gi(q) хэш-таблицы Ti } IF THEN return p crpq ≤− ||||
![Page 65: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/65.jpg)
65 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Использование LSH для поиска нечетких дубликатов
Шаг 1: Для каждого изображения коллекции вычислить набор локальных дескрипторов (например, набор SIFT-дескрипторов)
Шаг 2: Вычислить хэш-значение для каждого дескриптора
Шаг 3: Для каждой пары изображений вычислить колчиество совпадающих хэшей
Шаг 4: Дубликаты: пары изображений с числом совпадений > threshold
![Page 66: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/66.jpg)
66 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Оценка методов поиска
Быстродействие (efficiency) • Важно в связи с большими объемами данных
Качество (effectiveness) • Как померять?
Что оценивать?
![Page 67: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/67.jpg)
67 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Проблемы оценки
Необходимость наличия общей тестовой коллекции • Corel Photo CDs • Brodatz texture collection: http://www.ux.uis.no/~tranden/brodatz.html • CoPhIR: http://cophir.isti.cnr.it/whatis.html • Участие в ImageCLEF, TRECVID, imageEVAL, ROMIP
Как получить оценку релевантности? • Использование коллекций с выделенными группами (Corel collection) • Оценка пользователями
–Метод общего котла (Pooling) –Различные виды оценки (relevant – not relevant, ranking, …)
![Page 68: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/68.jpg)
68 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
“You can see, that our results are better”
Оценка качества
![Page 69: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/69.jpg)
69 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Численная оценка качестве
Точность (precision), полнота (recall)
–Average recall/precision –Recall at N, Precision at N –F-measure
![Page 70: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/70.jpg)
70 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Численная оценка
• Error rate
• Retrieval efficiency
![Page 71: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/71.jpg)
71 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Картиночные дорожки на
Поиск по визуальному подобию (с 2008) − Коллекция Flickr (20000 изображений)
− 750 tasks labeled
Поиск нечетких дубликатов (с 2008) − Коллекция: набор кадров из видео потока (37800 изображений) − ~1500 clusters
Построение текстовых меток (с 2010) − Коллекция Flickr (20000 изображений) − ~ 1000 labeled images
http://romip.ru
![Page 72: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/72.jpg)
72 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Поиск по визуальному подобию: наблюдения
Слабая согласованность оценок асессоров • Число слаборелевантных 377 (AND), 1086 (VOTE) и 3052 (OR)
Низкое качество результатов у всех участников
AND-weak
0
0,002
0,004
0,006
0,008
0,01
0,012
0,014
0,016
0,018
0,02
P(10) R-precision
Valu
e
OR-weak
0
0,01
0,02
0,03
0,04
0,05
0,06
0,07
0,08
P(10) R-precision
VOTE-weak
0
0,005
0,01
0,015
0,02
0,025
0,03
0,035
P(10) R-precision
Valu
e
![Page 73: 2012 10-26 image-videoanalysis_lecture06](https://reader031.fdocuments.net/reader031/viewer/2022020720/54900713b47959962d8b4d29/html5/thumbnails/73.jpg)
73 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Заключение
Поиск изображений: необходимость индексировать данные высокой размерности • Использование классических методов многомерного
индексирования (k-мерные деревья, метрические деревья) • Обратный индекс • «Мешок визуальных слов», vocabulary tree • Locality Sensitive Hashing
Оценка методов поиска • Единые тестовые коллекции, единые метрики оценки • Качество поиска пока все еще далеко от идеала • РОМИП – требуются добровольцы!