Visual Search: examining relationship between cropping and image size for

limited-screen devices

Paul Lin

Andres Odio

Idea

For browsers with limited screen space, how you display images matter

Browsing by small multiple thumbnails is a good start, but what happens when the screen space is extremely limited (such as on a PDA or mobile phone)?

How do you create effective visual search tools on these devices? What variables are involved? What are the relationships between the variables? How do you optimize these variables to create the fastest and most enjoyable browsing experience?

Cropping Automatic thumbnail cropping and its effectiveness (Suh, Ling,

Bederson, 2003) As long as the crop is “salient”

cropping is more effective regardless of technique and cropping size cropping is more effective regardless of content (18% for animal images,

24% for completely random Corbis images) So to create the most effective display method, we must optimize the

relationship between cropping, image size and the number of images per page

But... How does cropping relate to the size of the images, and the number of

images shown for each page? What is the relationship between cropped, resized and number of images on

the screen? How does cropping affect the user experience, and their level of enjoyability?

Hypothesis Rate of searching through items should be consistent for increasing

number of items However, we believe that for a large number of images, cropping is

more effective. As the number of images decreases, cropping becomes more effective, because you can use extraneous details considered “noise” for comparison

Thus, we expect results to be:

Number of Images

Per

form

ance

(ti

me)

Resized

Cropped

Experiment Design (1) Implementation

Created in C# Image search program 320 x 480, to simulate the fixed screen size of a typical Palm

PDA

Experiment Design (2)

Content domain consistency - women’s clothing only size consistency - upper body items (jackets,

shirts, sweaters) only image quality consistency - from Banana

Republic only Cropping mechanism

“salient” cropping 33% of original area crop

Experiment Design (3)

Experiment Design (4) 12 users 6 sets of searches/user

3x3 (9 images) per page, resized 3x3 (9 images) per page, cropped 5x5 (25 images) per page, resized 5x5 (25 images) per page, cropped 7x7 (49 images) per page, resized 7x7 (49 images) per page, cropped

15 searches/set 90 searches/user 190 searches/type of set 1080 data points total 148 images different used, with images randomized after each set Sequence of sets randomized for each user

Results

Analysed using SigmaPlot 9.0 Organized results by sets

Reject outliers by mean 2 standard deviations (95% confidence interval)

Plotted results to histogram to check for distribution, using SigmaPlot 9.0’s automatic binning

Distribution appears to be exponential decay (y=ax-bx) Best representation of the “mean” for exponential decay is

1/decay constant, or 1/b Lines were fitted to histograms to obtain equation, with all

equations satisfying > 90% confidence

0 20 40 60 800

10

20

30

40

50

0 10 20 30 400

5

10

15

20

25

30

0 10 20 30 400

10

20

30

40

0 20 40 600

10

20

30

40

50

60

0 20 40 600

10

20

30

40

50

0 20 40 600

10

20

30

40

3×3 Full

3×3 Cropped

5×5 Full

5×5 Cropped

7×7 Full

7×7 Cropped

21.93s

19.16s

17.24s

12.51s

23.53s

22.52s

Results(2)

5x5 is the best performing size, for both cropped and resized

Cropping is always faster and more efficient than resized

On average, cropping is 18.92% faster than resized images, which confirms Suh, Ling and Bederson’s results of 18-24% performance improvement

Number of Images VS Performance

10

12

14

16

18

20

22

24

26

0 20 40 60

Number of Images

Per

form

ance

(se

con

ds)

Full

Cropped

Quantitative Analysis

Questions: Why is the graph V shaped? Why is 3x3 almost as slow as 7x7, for both

cropped and resized? Why is 5x5 the fastest?

If it’s the number of images, then it 7x7 should be faster…

If it’s the size of the images, then 3x3 should be faster…

Quantitative Analysis(2)

New hypothesis: There is a third variable – number of pages For fixed screen size devices, with screen size

kept constant… number of pages is inversely proportional to number

of images number of pages is proportional to the size of image

The number of pages, and the time spent travelling through them, is as significant as the image size and number of images for overall result

Quantitative Analysis(3)

So if… overall result = f(g), number of page transitions = f(h), time constant for time spent on each page

transition = k, ideal graph where you have limitless screen area,

where page transition is not a factor = f(i), Then f(i) = f(g) – f(h).k

Quantitative Analysis(4)

Graphed the number of page transitions for each image search (forwards, backwards)

Again, organized results by sets, and rejected outliers by mean 2 standard deviations (95% confidence interval)

The graph is f(h), and for the constant k, we chose a k of 2 seconds

Number of Images VS Number of Page Transitions

0

1

2

3

4

5

6

7

8

9

0 20 40 60

Number of Images

Nu

mb

er o

f P

age

Tra

nsi

tio

ns

Full

Cropped

Number of Images VS Time spent on page transitions

02468

1012141618

0 20 40 60

Number of Images

Tim

e sp

ent

on

Pag

e T

ran

siti

on

s

Full

cropped

Quantitative Analysis(5) f(i) = f(g) – f(h).k Result = linear graph, where

difficulty of finding image increases with the number of images

Shows that if no page transitions are involved, then 3x3 should be a lot more efficient than 7x7

Similar to our hypothesis, except that cropped is always better than resized

Shows that page transitions has a huge effect on the efficiency of visual search

Ideal Number of Images VS Performance

y = 0.421x - 2.2031R2 = 0.9706

y = 0.3504x + 8.2133R2 = 0.9979

0

5

10

15

20

25

30

0 20 40 60

Number of Images

Per

form

ance

(se

con

ds)

Full

cropped

Linear (cropped)

Linear (Full)

Qualitative Analysis of User Experience

User survey conducted for each user Interesting results:

75% of users thought that they performed better for resized images, when numbers showed that cropped were almost always faster

Only 1 user thought that 5x5 cropped was the fastest

67% of the users enjoyed full sized images more than cropped

Conclusion Assuming that the design of the system doesn’t require page

transitions, then increasing number of images while decreasing in size will also increase difficulty

However, in real life, with screens being of fixed sizes, multiple pages are necessary for large numbers of images

Page transitions severely affects the overall image search times for larger images, and less for smaller images.

Thus, to calculate optimal display methods, one must find the ideal equation, then modify it with the predicted amount of time spent on page transitions. This amount of time will depend on the physical screen size.

Cropping is always faster than resized images, but it is also a less enjoyable experience.

Questions?