OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal Experiment Design (OED)for kernel ridge regression

and

the Minimum Volume Covering Ellipsoid (MVCE)

Tijl De Bie(K.U.Leuven)

Joint work with: Alexander Dolia John Shawe-TaylorMichael Titterington

Chris Harris

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 2

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

The next hour…

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 3

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design?OED?

Notation

Examples

Deliverables…

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 4

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design?OED?

Notation

Examples

Deliverables…

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 5

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design?OED?

Notation

Examples

Deliverables…

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 6

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

NotationOED?

Notation

Examples

Deliverables…

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 7

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

ExamplesOED?

Notation

Examples

Deliverables…

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 8

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

ExamplesOED?

Notation

Examples

Deliverables…

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 9

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Deliverables in this talk…OED?

Notation

Examples

Deliverables…

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 10

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Least squares regression (LS)

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 11

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Least squares regression (LS)

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 12

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design for LS

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 13

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design for LS

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 14

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design for LS

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 15

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design for LS

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 16

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Ridge regression (RR)

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 17

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Ridge regression (RR)

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 18

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Ridge regression (RR)

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 19

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design for RR

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 20

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design for RR

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 21

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Optimal experiment design for RR

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 22

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Kernel ridge regression (KRR)

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 23

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Kernel ridge regression (KRR)

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 24

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Kernel ridge regression (KRR)

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 25

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Kernel D-OED

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 26

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Kernel D-OED

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 27

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

OED: summaryD-OED MVCE

stan

dard

regu

lariz

edke

rnel

Least squares

Ridge regression

Kernel RR

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 28

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Now over to novelty detection!

Novelty detection

MVCE and duality

Regularized MVCE

Kernel MVCE

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 29

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Now over to novelty detection!

Novelty detection

MVCE and duality

Regularized MVCE

Kernel MVCE

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 30

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Regularized MVCE

Novelty detection

MVCE and duality

Regularized MVCE

Kernel MVCE

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 31

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Kernel MVCE

Novelty detection

MVCE and duality

Regularized MVCE

Kernel MVCE

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 32

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Kernel MVCE

Novelty detection

MVCE and duality

Regularized MVCE

Kernel MVCE

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 33

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Kernel MVCE

Novelty detection

MVCE and duality

Regularized MVCE

Kernel MVCE

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 34

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

MVCE: summary

Novelty detection

MVCE and duality

Regularized MVCE

Kernel MVCE

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 35

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

MVCE: summaryD-OED MVCE

stan

dard

regu

lariz

edke

rnel

Novelty detection

MVCE and duality

Regularized MVCE

Kernel MVCE

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 36

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

MVCE: dealing with outliers

Novelty detection

MVCE and duality

Regularized MVCE

Kernel MVCE

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 37

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Experiments – MVCE

Linear

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 38

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Experiments – MVCE

Linear, centered

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 39

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Experiments – MVCE

RBF-kernel

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 40

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Experiments – MVCE

RBF-kernel

Soft-margin

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 41

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Experiments – D-OED

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 42

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Experiments – D-OED

2-norm

infinity norm

1-norm

Costs:

Random vs

Uniform vs

OED (blue)

Eurandom - 20 Dec '05 Tijl De Bie - KULeuven 43

OED for KRR, and the MVCE

Introduction

OED

MVCE

Experiments

Conclusions