Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming...
-
Upload
spencer-copeland -
Category
Documents
-
view
220 -
download
0
description
Transcript of Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming...
![Page 1: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/1.jpg)
Online Manifold Regularization: A New Learning Setting and Empirical Study
Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008).
Hu EnLiang Friday, April 17, 2009
![Page 2: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/2.jpg)
![Page 3: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/3.jpg)
Standard online learning VS. Online Manifold Regularization Both of them are long-life learning and learn
non-iid sequentially;
Standard online learning: traditionally assumes that every input point is fully labeled, it cannot take advantage of unlabeled data;
Online MR: it learns even when the input point is unlabeled.
![Page 4: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/4.jpg)
Online MR VS. batch MR (advantages) Online MR scales better than batch MR in time and
space;
Online MR achieves comparable performance to batch MR;
Online MR can handle concept drift;
Online MR is an “anytime classifier”, which continuously is being improved and its training is cheap.
![Page 5: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/5.jpg)
The principle of online MR
![Page 6: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/6.jpg)
![Page 7: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/7.jpg)
![Page 8: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/8.jpg)
The relationship of batch risk, instantaneous regularized risk and average instantaneous risk
![Page 9: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/9.jpg)
![Page 10: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/10.jpg)
![Page 11: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/11.jpg)
How to accelerate online MR
![Page 12: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/12.jpg)
![Page 13: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/13.jpg)
![Page 14: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/14.jpg)
![Page 15: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/15.jpg)
![Page 16: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/16.jpg)
![Page 17: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/17.jpg)
![Page 18: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/18.jpg)
![Page 19: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/19.jpg)
![Page 20: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/20.jpg)
Continue !!!
![Page 21: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/21.jpg)
A Brief Introduction to CBIR(Content-based Image
Retrieval)
Hu en liang
Tuesday, April 08, 2008
![Page 22: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/22.jpg)
Background:Content-based Image Retrieval
Properties: Querying image according to user’s semantic-co
ncepts. Querying images according to image’s contents,
such as: color, texture, shape, etc.
Hypothesis——similar contents means semantic affinity;
‘Semantic gap’——semantic affinity doesn't means similar contents.
![Page 23: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/23.jpg)
A prototype of feedback-based CBIR
![Page 24: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/24.jpg)
![Page 25: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/25.jpg)
![Page 26: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/26.jpg)
![Page 27: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/27.jpg)
Background: The Difficulty of ‘Semantic Gap’ Key problems:
1. How to extract user’s semantic-concept (intention)?2. How to bridge between content and semantic ?
Main methods:
1. Machine learning based RF (Relevance-Feedback); 2. The prior knowledge such as the historical logs.
![Page 28: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/28.jpg)
How to Connect CBIR to ML? (Semi-)supervised Metric Learning;
Manifold Learning, Dimension Reduction…
(Semi-)supervised Classification;
Active Learning; Co-training;
Assembling Classifier;
Ranking; …
![Page 29: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/29.jpg)
Some Individual Characteristics for feedback-based CBIR In contrast to typical ML, there are some special
characteristics for RF-CBIR :
The problem of the small size sample;
The problem of asymmetrical training sample;
The online algorithm with real-time requirement;
![Page 30: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/30.jpg)
Manifold Regularization (MR)
Manifold regularization: a geometric framework for learning from labeled and unlabeled examples. Mikhail Belkin, Partha Niyogi, Vikas Sindhwani. Journal of machine Learning Research 7, pp 2399-2434, 2006
![Page 31: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/31.jpg)
To Modify MR for CBIR There are some intrinsic characteristics for CBIR :
The problem of the small size sample; The problem of asymmetrical training sample; The online algorithm with real-time requirement;
The (1+x)-manifolds hypothesis There only single submanifold for positive clas
s, but multi-submanifolds for negative class!
![Page 32: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/32.jpg)
Negative manifold
positive manifold
The Problem of MR for the Multi-Submanifolds Case
![Page 33: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/33.jpg)
The Bias-MR Focusing on Single-Submanifold
![Page 34: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/34.jpg)
A review of LapSVM
![Page 35: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/35.jpg)
A review of LapSVM
O(l3) O(n3)O(n3)
![Page 36: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/36.jpg)
A higher efficiency in BLapSVM
O(q3
)
![Page 37: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/37.jpg)
The BLapSVM Algorithm for CBIR
![Page 38: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/38.jpg)
The ‘BEP’ Performance Chart
![Page 39: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/39.jpg)
The ‘Efficiency’ Performance Chart
![Page 40: Online Manifold Regularization: A New Learning Setting and Empirical Study Andrew B. Goldberg, Ming Li, and Xiaojin Zhu.(ECML PKDD, 2008). Hu EnLiang Friday,](https://reader035.fdocuments.net/reader035/viewer/2022062413/5a4d1b8a7f8b9ab0599be44f/html5/thumbnails/40.jpg)
Thanks for Your
Attention !