Palette Power: Enabling Visual Search through Colors

Aug 14, 2013

eBay Research Labs

http://labs.ebay.com

Changing Landscape of Search

2

Visual Search for Fashion

eBay

Inventory

Given an item image,

find similar eBay inventory

Query Image

Similar Items

3

Item Similarity

Fre

quency

Color Distributions

Dots Floral Checks

Patterns & Textures

Styles

4

Approach Overview

Large Image Data Speed Requirements

Take Advantage of Context

Our Approach – The Power of Color Distributions

Color Spaces

𝑑 = 𝑓(𝑖1, 𝑖2) Distance Functions

5

6

Challenges (1/3)

Low contrast between background and foreground

7

Challenges (2/3)

Background Clutter

8

Challenges (3/3)

Lighting Variation

9

Insights From Data

Object localization using spatial priors

Choosing the right color space

Why Object Localization?

10

Cluttered background degrades performance.

State-of-the-art segmentation too expensive.

Need a fast and reliable solution!

Spatial Prior to the rescue!

Understanding Spatial Prior

11

12

Choosing Best Color Space

13

Handling Color Confusion

14

Generating Color Histogram

Faster Lookup via k-center

15

Scaling via backend clustering/indexing.

Potential for semantic/intent diversification - e.g. query t-shirt image where you like style but not colors

Achieves 60x speedup close to 70% overlap!

Median speed-up Median %-overlap

16

Architecture

17

Experiment I – Fashion Dataset

Categories: Women’s Dresses, Tops & Blouses, Coats & Jackets,

Skirts, Sweaters and T-Shirts

Data Sets: 1600 Queries & 1 Million Inventory images, 15 users for

30 days

18

Results - Solid Queries

Results - Pattern Queries

19

Experiment II – Generic ecommerce Dataset

Categories: Toys, Sports, Camera

Data Sets: Query & Inventory sets for each category

Ground Truth: ~15 per query

20

Example Inventory Images

21

Toys

Sports

Camera

22

MAP Performance

23

Experiment III – INRIA Holidays Dataset

Categories: Personal Holidays Photos

Data Sets: 500 Queries (1 per group) & 1491 Inventory Images

Ground Truth: Human Annotations

24

MAP Performance

25

Computational Costs

Feature Extraction Time 10 ms

Retrieval Time 80 ms

Feature Vector Size 196 Bytes

Memory Required 190 MB

Machine Stats: 24 GB RAM, 2.53GHz

Index Size: 1M+

26

Summary

Color a fundamental cue

Spatial Prior can eliminate need for expensive

background removal

Future work to focus on efficient descriptors

27

Questions?