CUbRIK at SMILA Conference in Berlin

Human-enhanced Multimedia Processing

in CuBRIK with SMILA

Alessandro Bozzon, Ph.d.

Politecnico di Milano mail: [email protected]: aleboz

The CUbRIK project

36 month large-scale integrating project

partially funded by the European Commission’s 7th Framework ICT Programme for Research and Technological Development

www.cubrikproject.eu

04/13/2023 SMILA Themenkonferenz 3

http://www.cubrikproject.eu/


Objectives

The technical goal of CUbRIK is to build an open search platform grounded on four objectives: Advance the architecture of multimedia search Place humans in the loop Open the search box Start up a search business ecosystem


Objective: Advance the architecture of multimedia search

Multimedia search: coordinated result of three main processes: Content processing: acquisition, analysis,

indexing and knowledge extraction from multimedia content

Query processing: derivation of an information need from a user and production of a sensible response

Feedback processing: quality feedback on the appropriateness of search results


Objective: Advance the architecture of multimedia search

Objective: Content processing, query processing and

feedback processing phases will be implemented by means of independent components

Components are organized in pipelines Each application defines ad-hoc pipelines that

provide unique multimedia search capabilities in that scenario


CUbRIK architecture


6 March 2012 The CUbRIK Project is .... 8

SMILA is the backbone of CUbRIK

CUbRIK makes use of SMILA framework as a start-up service engine for supporting workflow definition and execution

Provides architectural extensions to SMILA for enhanced services:

Extensible content, query and feedback processing search workflow Multimodality, Orchestration of human and machine computation

tasks in all search processes Time and Space Awareness Support for social and human computation Persistency and Caching of content and metadata Support of federated configurations across a distributed architecture Different styles of User Interface for queries and presentation of

search results Includes tools and methods for application design

Objective: Humans in the loop

Problem: the uncertainty of analysis algorithms leads to low confidence results and conflicting opinions on automatically extracted features

Solution: humans have superior capacity for understanding the content of audiovisual material

State of the art: humans replace automatic feature extraction processes (human annotations)

Our contribution: integration of human judgment and algorithms

Goal: improve the performance of multimedia content processing


Example of CUbRIK Human-enhanced computation: Trademark Logo Detection

Problem statement: identifying occurrences of trademark logos in a video collection through keyword-based queries

Special case of the classic problem of object recognition

Use case: a professional user wants to retrieve all the occurrences of logos in a large collection of video clips

Applications: rating effectiveness of advertising, subliminal advertising detection, automatic annotation, trademark violation detection

1010

Human-powered trademark logo detection demo

Goal: integrate human and automatic computation to increase precision and recall w.r.t. fully automatic solutions


Problems in automatic logo detection: Object recognition is affected by the quality of the

input set of images

Uncertain matches, i.e., the ones with low matching score, could not contain the searched logo

Trademark Logo Detection: problems in automatic logo detection

12

Contribution in human computation Filter the input logos, eliminating the irrelevant

ones Segment the input logos

Validate the matching results

13

Trademark Logo Detection: contribution of human computation

14

Trademark Logo Detection: pipeline

14

The CrowdSearcher framework for HC task management

15

15

CrowdSearch framework in the Logo detection application

16

Types of tasks• Automatic tasks• Crowd tasks: tasks that are executed

by an open-ended community of performers

16

Community of Performers

17

The application is deployed as a Facebook application

Seed community Information Technology department of Politecnico di Milano

Task propagationEach user in the seed community can propagate tasks through the social networks

17

Design of “Validate Logo Images”

The “LIKE” task variant requires to choose relevant logos among a set of not filtered images

The “ADD”task variant requires to add new relevant image URLs

Please add new relevant logos

URL…

Send

18

People to task matching & Task Assignment

19

Execution criteriaConstraints of task execution

Time budget for the experiment

Content Affinity criteriaQuery on a representation of the users’ capacities• Current state: manual selection of users• Future work: Geocultural affinityQuestions are dispatched to the crowd according to the user experience in answering questions• Expert user: an user that has already

answered to three questions

New users answer to “LIKE” questions

Expert users answer to “LIKE”+“ADD” questions

19

Task propagation

Propagation over the Facebook graph: Platform: CrowdSearcher

Automatic task generation starting from a set of design criteria (e.g., question type, public/private…)

Seed community: Information Technology department of Politecnico di Milano

Each user in the seed community can propagate tasks through the social networks

Work in progress: Twitter/LinkedIn tasks Task assignment according to expertise, geocultural

information, past work history

04/13/2023 CUbRIK Pipelines 1 20

Task execution

21

“LIKE” task variant “ADD” task variant

21

Output aggregation

22

“LIKE” task variantsTop-5 rated logos are selected as relevant logos

“ADD” task variantsNew images are fed back to the LIKE tasks

22

Experimental evaluation

Three experimental settings: No human intervention Logo validation performed by two domain

experts Inclusion of the actual crowd knowledge

Crowd involvement 40 people involved 50 task instances generated 70 collected answers

23


24

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

No Crowd

ExpertsCrowd

No Crowd

Experts

CrowdNo Crowd

Experts

CrowdAleveChunkyShout

Precision

Reca

ll


25

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

No Crowd

ExpertsCrowd

No Crowd

Experts

CrowdNo Crowd

Experts


Precision

Reca

ll

Precision decreases

Reasons for the wrong inclusion• Geographical location of the

users• Expertise of the involved users


26

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

No Crowd

ExpertsCrowd

No Crowd

Experts

CrowdNo Crowd

Experts


Precision

Reca

ll

Precision decreases• Similarity between two

logos in the data set

Crowdsourced filtering of logos – Problem concept


Google Images

Filter Tasks

Filtered logos

Added logosAdd

Tasks

Integration in SMILA

The demo has been integrated into the SMILA architecture

Two main parts: Indexing part: made of asynchronous

components (in a SMILA sense) Indexing of videos Matching phase Interaction with the crowd

Search part: end users query the system by keyword-based queries


Integration in SMILA


Integration in SMILA: Indexing part overview


Reusable components


Crawling Google Images + Flickr crawler

Multimedia processing SIFT-based low level feature extraction Video segmentation component Key-frame extractor Robust low level feature matching component

Data storage “Data service” for referencing multimedia

resources

Integration in SMILA: Indexing part – Job1, Input images retrieval


Integration in SMILA: Indexing part – Job2, Logo collection indexing


Integration in SMILA: Indexing part – Job3, video collection indexing


Integration in SMILA: Indexing part – Job4, matching phase


Integration in SMILA: Indexing part – Job5, matches filtering


Demo: Search interface




Indexed logos thatmatch against the

video collection



Video preview



High confidence matches



Low confidence matches

6 March 2012 The CUbRIK Project is .... 43

CUbRIK Showcases

CUbRIK will showcase its technology with Demonstrators of examples of innovation in two domains: (Digital Libraries) History of Europe (Business Processes) CUbRIK search for SMEs,

Technical evaluation in real-world conditions including users will be based on these Demonstrators

Thanks for your attentionwww.cubrikproject.eu



CUbRIK at SMILA Conference in Berlin

Technology

Transcript of CUbRIK at SMILA Conference in Berlin