Multi-Task Learning for Boosting with Application to Web Search Ranking

Olivier Chapelle et al.

Presenter: Wei Cheng

Outline

• Motivation• Backgrounds• Algorithm– From svm to boosting using L1 regularization– Є-boosting for optimization– Overall algorithm

• Evaluation• Overall review and new research points

discussion

Motivation

Different/same search engine(s) for different

countries?

Domain specific engine is better!e.g. ‘gelivable’ (very useful)

Motivation

• Should we train ranking model separately?– Corps in some domains might be too small to train

a good model– Solution:

Multi-task learning

Backgrounds

Sinno Jialin Pan and Qiang Yang, TKDE’2010

Backgrounds

Sinno Jialin Pan and Qiang Yang, TKDE’2010

Backgrounds

• Why transfer learning works?

Sinno Jialin Pan et al. At WWW’10

Backgrounds

• Why transfer learning works?(continue)

Backgrounds

• Why transfer learning works?(continue)

Backgrounds

LearnerA

Input:

Target:

LearnerB

Dog/human Girl/boy

traditional learning

Backgrounds

Joint Learning Task

Input:

Target: Dog/human Girl/boy

Multi-task learning

Algorithm

• The algorithm aims at designing an algorithm based on gradient boosted decision trees

• Inspired by svm based multi-task solution and boosting-trick.

• Using Є-boosting for optimization

Algorithm

• From svm to boosting using L1 regularization• Previous svm based multi-task learning:

Algorithm

• Svm(kernel-trick)---boosting (boosting trick)

Pick set of non-linear functions(e.g., decision trees,

regression trees,….)

H|H|=J

Apply every single function to each data point

1( )

.( )

.

( )J

h x

x

h x

Xф(X)

Є-boosting for optimization

• Using L1 regulization

Using Є-boosting

Algorithm

Evaluation

• Datasets

Evaluation

Evaluation

Evaluation

Evaluation

Evaluation

Overall review and new research point discussion

• Contributions:– Propose a novel multi-task learning method based

on gradient boosted decision tree, which is useful for web-reranking applications. (e.g., personalized search).

– Have a thorough evaluation on we-scale datasets.• New research points:– Negative transfer:– Effective grouping: flexible domain adaptation

Q&A

Thanks!