EigenTransfer : A Unified Framework for Transfer Learning
description
Transcript of EigenTransfer : A Unified Framework for Transfer Learning
![Page 1: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/1.jpg)
EigenTransfer: A Unified Framework for Transfer
LearningWenyuan Dai, Ou Jin, Gui-Rong Xue, Qiang
Yang and Yong Yu
Shanghai Jiao Tong University & Hong Kong University of Science and Technology
![Page 2: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/2.jpg)
Motivation Problem Formulation Graph Construction Simple Review on Spectral Analysis Learning from Graph Spectra Experiments Result Conclusion
Outline
![Page 3: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/3.jpg)
Motivation
![Page 4: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/4.jpg)
A variety of transfer learning tasks have been investigated.
Motivation
Lifelong Learning (Thrun,
1996)
Multi-task Learning
(Caruana, 1997)
Cross-domain Learning (Wu et
al., 2004)
Cross-category Learning (Raina
et al., 2006)
Self-taught Learning (Raina
et al., 2007)
General
Framework
![Page 5: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/5.jpg)
Difference◦ Different tasks◦ Different approaches & algorithms
Common
Motivation
Auxiliary Data
Target Data (Training)
Target Data (Test)
Common parts or relation
![Page 6: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/6.jpg)
We can have a graph:
Motivation
Features
Auxiliary Data Training Data Test Data
Labels
New Representation
![Page 7: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/7.jpg)
We can get the new representation of Training Data and Test Data by Spectral Analysis.
Then we can use our traditional non-transfer learner again.
Motivation
![Page 8: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/8.jpg)
Problem Formulation
![Page 9: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/9.jpg)
Target Training Data: with labels Target Test Data: without labels Auxiliary Data:
Task◦ Cross-domain Learning◦ Cross-category Learning◦ Self-taught Learning
Problem Formulation
1{ }i nt t ix
1{ }i iutkx
1{ }i im
a ux
![Page 10: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/10.jpg)
Problem Formulation
![Page 11: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/11.jpg)
Graph Construction
![Page 12: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/12.jpg)
Graph Construction
Cross-domain Learning
-( )- -( )- -( )- -( 1 )- -( 1 )-
itx
jf,i jiux
jf,i jiax
jf,i j
itxiux
jCjC
![Page 13: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/13.jpg)
Graph Construction
Cross-category Learning
-( )- -( )- -( )- -( 1 )- -( 1 )-
itx
jf,i jiux
jf,i jiax
jf,i j
itxiux
jtCjaC
![Page 14: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/14.jpg)
Graph Construction
Self-taught Learning
-( )- -( )- -( )- -( 1 )-
itx
jf,i jiux
jf,i jiax
jf,i j
itx
jtC
![Page 15: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/15.jpg)
Graph Construction
Doc-Token Matrix Adjacency Matrix
Token Token …
Doc
Doc
…
Doc Feature
Label
Doc ?
Feature
? 0
Label 0 0
![Page 16: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/16.jpg)
Simple Review on Spectral Analysis
![Page 17: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/17.jpg)
G is an undirected weighted graph with weight matrix W, where .
D is a diagonal matrix, where
Unnormalized graph Laplacian matrix:
Normalized graph Laplacians:
Simple Review on Spectral Analysis
0ij jiWW
L D W
1/2 1/2 1/2 1/2sym D LD I D WDL
1 1rwL D L WI D
ii ijj
WD
![Page 18: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/18.jpg)
Calculate the first k eigenvectors The New representation:
Simple Review on Spectral Analysis
1 2, kv v v
v1 v2 v3
Node1
Node2
Node3
Node4
…
New Feature Vector of the
Node2
![Page 19: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/19.jpg)
Learning from Graph Spectra
![Page 20: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/20.jpg)
Graph G Adjacency matrix of G: Graph Laplacian of G: Solve the generalized eigenproblem:
The first k eigenvectors form a new feature representation.
Apply traditional learners such as NB, SVM
Learning from Graph Spectra
W
L D W
L Dv v
![Page 21: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/21.jpg)
DocFeatur
e Label
Doc
Feature
Label
Learning from Graph Spectra
DocFeatur
e Label
Doc
Feature
Label
v1 v2
Train
Test
Auxiliary
Feature
Label
Train
v1 v2
Test v1 v2
Classifier
W
L
![Page 22: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/22.jpg)
The only problem remain is the computation time.
Which is lucky:◦ Matrix L is sparse◦ There are fast algorithms for sparse matrix for
solving eigen-problem. (Lanczos) The final computational cost is linear to
Learning from Graph Spectra
( )nz L k
![Page 23: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/23.jpg)
Experiments
![Page 24: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/24.jpg)
Basic Progress
Experiments
Training Data
Test DataAuxiliary
Data
New Training
Data
New Test Data
15 Positive Instances &15 Negative Instances
Baseline
Result
Repeat 10 times
Calculate average
Sample
Classifier(NB/SVM/TSVM)
CV
![Page 25: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/25.jpg)
Cross-domain Learning Data
◦ SRAA◦ 20 Newsgroups (Lang, 1995)◦ Reuters-21578
Target data and auxiliary data share the same categories(top directories), but belong to different domains(sub-directories).
Experiments
![Page 26: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/26.jpg)
ExperimentsCross-domain result with NB
cdl-s
raa1
cdl-s
raa2
cdl-2
0ng1
cdl-2
0ng2
cdl-2
0ng3
cdl-2
0ng4
cdl-2
0ng5
cdl-2
0ng6
cdl-r
eute
rs1
cdl-r
eute
rs2
cdl-r
eute
rs3
aver
age
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Non-TransferSimple combineEigen Transfer
![Page 27: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/27.jpg)
ExperimentsCross-domain result with SVM
cdl-s
raa1
cdl-s
raa2
cdl-2
0ng1
cdl-2
0ng2
cdl-2
0ng3
cdl-2
0ng4
cdl-2
0ng5
cdl-2
0ng6
cdl-r
eute
rs1
cdl-r
eute
rs2
cdl-r
eute
rs3
aver
age
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
0.45
Non-TransferSimple combineEigen Transfer
![Page 28: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/28.jpg)
ExperimentsCross-domain result with TSVM
cdl-s
raa1
cdl-s
raa2
cdl-2
0ng1
cdl-2
0ng2
cdl-2
0ng3
cdl-2
0ng4
cdl-2
0ng5
cdl-2
0ng6
cdl-r
eute
rs1
cdl-r
eute
rs2
cdl-r
eute
rs3
aver
age
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
Non-TransferSimple combineEigen Transfer
![Page 29: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/29.jpg)
Cross-domain result on average
Experiments
Non-Transfer Simple Combine EigenTransfer
NB 0.250±0.036 0.239±0.000 0.134±0.031
SVM 0.190±0.039 0.213±0.000 0.095±0.018
TSVM 0.140±0.038 0.145±0.000 0.101±0.019
![Page 30: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/30.jpg)
Cross-category Learning Data
◦ 20 Newsgroups (Lang, 1995)◦ Ohscal data set from OHSUMED (Hersh et al.
1994) Random select two categories as target
data. Take the other categories as auxiliary labeled data.
Experiments
![Page 31: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/31.jpg)
ExperimentsCross-category result with NB
ccl-2
0ng1
ccl-2
0ng2
ccl-2
0ng3
ccl-2
0ng4
ccl-2
0ng5
ccl-o
hs1
ccl-o
hs2
ccl-o
hs3
ccl-o
hs4
ccl-o
hs5
aver
age
0
0.05
0.1
0.15
0.2
0.25
0.3
Non-TransferEigenTransfer
![Page 32: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/32.jpg)
ExperimentsCross-category result with SVM
ccl-2
0ng1
ccl-2
0ng2
ccl-2
0ng3
ccl-2
0ng4
ccl-2
0ng5
ccl-o
hs1
ccl-o
hs2
ccl-o
hs3
ccl-o
hs4
ccl-o
hs5
aver
age
0
0.05
0.1
0.15
0.2
0.25
Non-TransferEigenTransfer
![Page 33: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/33.jpg)
ExperimentsCross-category result with TSVM
ccl-2
0ng1
ccl-2
0ng2
ccl-2
0ng3
ccl-2
0ng4
ccl-2
0ng5
ccl-o
hs1
ccl-o
hs2
ccl-o
hs3
ccl-o
hs4
ccl-o
hs5
aver
age
0
0.05
0.1
0.15
0.2
0.25
Non-TransferEigenTransfer
![Page 34: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/34.jpg)
Cross-category result on average
Experiments
Non-Transfer EigenTransfer
NB 0.186±0.038 0.099±0.025
SVM 0.131±0.032 0.065±0.016
TSVM 0.104±0.010 0.091±0.013
![Page 35: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/35.jpg)
Self-taught Learning Data
◦ 20 Newsgroups (Lang, 1995)◦ Ohscal data set from OHSUMED (Hersh et al.
1994) Random select two categories as target
data. Take the other categories as auxiliary without labeled data.
Experiments
![Page 36: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/36.jpg)
ExperimentsSelf-taught result with NB
stl-2
0ng1
stl-2
0ng2
stl-2
0ng3
stl-2
0ng4
stl-2
0ng5
stl-o
hs1
stl-o
hs2
stl-o
hs3
stl-o
hs4
stl-o
hs5
aver
age
0
0.05
0.1
0.15
0.2
0.25
0.3
0.35
Non-TransferEigenTransfer
![Page 37: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/37.jpg)
ExperimentsSelf-taught result with SVM
stl-2
0ng1
stl-2
0ng2
stl-2
0ng3
stl-2
0ng4
stl-2
0ng5
stl-o
hs1
stl-o
hs2
stl-o
hs3
stl-o
hs4
stl-o
hs5
aver
age
0
0.05
0.1
0.15
0.2
0.25
Non-TransferEigenTransfer
![Page 38: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/38.jpg)
ExperimentsSelf-taught result with TSVM
stl-2
0ng1
stl-2
0ng2
stl-2
0ng3
stl-2
0ng4
stl-2
0ng5
stl-o
hs1
stl-o
hs2
stl-o
hs3
stl-o
hs4
stl-o
hs5
aver
age
0
0.05
0.1
0.15
0.2
0.25
Non-TransferEigenTransfer
![Page 39: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/39.jpg)
Self-taught result on average
Experiments
Non-Transfer EigenTransfer
NB 0.189±0.038 0.107±0.032
SVM 0.126±0.030 0.070±0.017
TSVM 0.106±0.011 0.098±0.024
![Page 40: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/40.jpg)
ExperimentsEffect of the number of Eigenvectors
![Page 41: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/41.jpg)
ExperimentsLabeled Target Data
![Page 42: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/42.jpg)
We proposed a general transfer learning framework.
It can model a variety of existing transfer learning problems and solutions.
Our experimental results show that it can greatly outperform non-transfer learners in many experiments.
Conclusion
![Page 43: EigenTransfer : A Unified Framework for Transfer Learning](https://reader035.fdocuments.net/reader035/viewer/2022062301/56813bf5550346895da53bea/html5/thumbnails/43.jpg)
Thank you!