CrowdSearch: Exploiting Crowds for Accurate Real-Time Image Search on
Mobile PhonesOriginal work by Tingxin Yan, Vikas Kumar, Deepak Ganesan
Presented by Ashok Kumar Jonnalagadda
Roadmap Problem Description
What is “crowdsourcing”? System Architecture The Crowd Search Algorithms
Delay Prediction Validation Prediction
Experimental Evaluation Discussion/Criticism Questions
The Perceived Problem Text-based search is easy…
The Perceived Problem Mobile-based search will become more important in
the future. More than 70% of smart phone users perform searches.
Expected to be more mobile searches than non-mobile searches soon
Text-based mobile searches are easy as well… Issues:
Small form-factor and resource limitations. Typing on a phone is cumbersome Scrolling through multiple search results. multimedia searches requires significant
memory, storage, and computing resources. Mobile: GPS and voice for search is becoming
more commonplace.
Image Search from Mobile.? Image variations in
lighting Texture Type of features image quality and many other factors.
Even Google Goggle doesn’t work with all categories.
Automated image search has limitations in terms of
Humans are naturally good at distinguishing images
The Perceived Problem But how does a mobile phone user search for this?
No visible words/letters; too far away to know the address.
The Perceived Problem Ways to find out what that building is:
Ask random people on the street Travel to the building to see the address/sign Take a picture of the building with your mobile
device and send to a search engine… How easy is image searching on a mobile phone
though?
The Perceived Problem Image search is a non-trivial problem – have
to deal with variations in lighting, texture, image quality, etc.
Even when results are returned, scrolling through multiple pages on a mobile device is cumbersome. Search should be precise and return very few
erroneous results. Multimedia searches require significant
Memory Storage Computing resources
The Proposed Solution CrowdSearch – Attempts to provide an
accurate, image search system for mobile devices by combining… Automated image search and Real-time human validation of search results
Leverage crowdsourcing through Amazon Mechanical Turk (AMT)
The Proposed Solution Humans are good at comparing images
Could an automated search determine these two images are of the same building? Crowdsourcing increases search result accuracy.
System Architecture Three main components:
Mobile Device Initiates queries Displays responses Performs local image processing (maybe)
Remote Server Performs automated image search Triggers image validation tasks
Crowdsourcing System (AMT) Validates image search results
Apple iPhone Mobile Client
System Operation Overview
System Operation Overview
System Operation Overview
How do we minimize delay and cost while maximizing accuracy?
System Architecture
Balancing Tradeoffs Result delay
Should minimize delay or at least keep it within a user-provided bound
Result accuracy Strive for high (i.e., ≥ 95%) accuracy
Monetary cost Low cost is better than high cost
Energy Should consume minimal battery power
Accuracy Considerations How many validations are required for 95%
accuracy? Requiring at least
three validationsout of five achieves≥ 95% accuracy.
Optimizing Delay Utilize parallel posting
Post all candidate images to the crowdsourcing system at the same time.
But this approach increases cost!
5 cents
= 20 cents
5 cents
5 cents
5 cents
Optimizing Cost Utilize serial posting
Post top-ranked candidate first, wait for responses, then post next candidate if necessary.
This approach increases delay!
CrowdSearch Delay/Cost Optimization Combine elements of parallel and serial posting
Prediction requires delay and validation models Goal: want at least one verified result by the deadline.
CrowdSearch Delay/Cost Optimization
Delay Prediction Model The delay of a single response is the
combination of acceptance delay and submission delay. Both of these follow an exponential distribution
with an offset. Thus, overall delay is the convolution of these
delays.
Delay Prediction Model Performance
Validation Model Given a response set S, want to compute
probability of positive validation result. Use training data to set these probabilities If the probability of a positive
result is less than somethreshold, send the nextcandidate to validation.
In this example, if the threshold were set to < 76%, the server would post the next candidate image to AMT.
Power Considerations Should some image processing occur on the local
device or should it be outsourced to the server? It depends! Use remote
processing when WiFi is available.
Use local processingwhen only 3G is available
Extracting featuresfrom query Image
(Scale Invariant feature transform)
Experimental Results Any of the crowdsourcing schemes lead to
better results! Some types of images
are easier for automated searchesto handle than others
Experimental Results CrowdSearch leads to (given a long enough
deadline)… Behavior close to parallel posting for recall Behavior close to serial posting for search cost
Thoughts/Criticism The limited nature of the solution
Limitation to the four categories Buildings Books Flowers Faces
Only 1000 images in the backend database. Would increasing the number of automated search
images increase total task time in a significant way?
Thoughts/Criticism How useful is this anyway?
Are people willing to go through the trouble to set up a payment account and pay 5-20 cents for a search?
How much effort would it usually take for someone to find out what the object is through traditional means? Especially for books!
Privacy concerns People utilizing CrowdSearch must accept the fact that
random strangers know what they are looking at and searching for. Additionally, their GPS information might be provided to the
CrowdSearch servers. What about the privacy of the object of the search?
Undercover police officers
Questions?
Top Related