Linear Classification Lecture 3 - Artificial...

Fei-Fei Li & Andrej Karpathy Lecture 2 - 7 Jan 2015Fei-Fei Li & Andrej Karpathy Lecture 2 - 7 Jan 20151

Lecture 3:Linear Classification

Last time: Image Classification

assume given set of discrete labels{dog, cat, truck, plane, ...}

k-Nearest Neighbortraining set

test images

Linear Classification1. define a score function

class scores

Linear Classification1. define a score function

“weights” “bias vector”

data (image)

class scores“parameters”

Linear Classification1. define a score function(assume CIFAR-10 example so32 x 32 x 3 images, 10 classes)

weights bias vector

data (image)[3072 x 1]

class scores

Linear Classification1. define a score function(assume CIFAR-10 example so32 x 32 x 3 images, 10 classes)

weights[10 x 3072]

bias vector[10 x 1]

data (image)[3072 x 1]

class scores[10 x 1]

Linear Classification

Interpreting a Linear ClassifierQuestion:what can a linear classifier do?

Interpreting a Linear Classifier

Example training classifiers on CIFAR-10:

Interpreting a Linear Classifier

Bias trick

So far:We defined a (linear) score function:

2. Define a loss function (or cost function, or objective)

- scores, label loss.

Example:

Example: Question: if you were to assign a single number to how “unhappy” you are with these scores, what would you do?

2. Define a loss function (or cost function, or objective)One (of many ways) to do it: Multiclass SVM Loss

(One possible generalization of Binary Support Vector Machine to multiple classes)

loss due to example i sum over all

incorrect labelsdifference between the correct class score and incorrect class score

loss due to example i sum over all

incorrect labelsdifference between the correct class score and incorrect class score

Example:

e.g. 10

loss = ?

Example:

e.g. 10

There is a bug with the objective…

L2 Regularization

Regularization strength

L2 regularization: motivation

Do we have to cross-validate both and ?

So far…1. Score function

2. Loss function

Softmax Classifier score functionis the same

(extension of Logistic Regression to multiple classes)

i.e. we’re minimizing the negative log likelihood.

softmax function

i.e. we’re minimizing the negative log likelihood.

Softmax vs. SVM

- Interpreting the probabilities from the Softmax

Softmax vs. SVM

suppose the weights W were only half as large

(we use a higher regularization strength)

Softmax vs. SVM

suppose the weights W were only half as large:

Softmax vs. SVM- Interpreting the probabilities from the Softmax

suppose the weights W were only half as large:

What happens in the limit, as the regularization strength goes to infinity?

Softmax vs. SVM

scores:[10, -2, 3][10, 9, 9][10, -100, -100]

Softmax vs. SVM1

Interactive Web Demo time....

http://vision.stanford.edu/teaching/cs231n/linear-classify-demo/

Summary- We introduced a parametric approach to

image classification- We defined a score function (linear map)- We defined a loss function (SVM / Softmax)

One problem remains: How to find W,b ?

Next class: Optimization,

Backpropagation

Find the W,b that minimizes the loss function.

Linear Classification Lecture 3 - Artificial...

Documents

Transcript of Linear Classification Lecture 3 - Artificial...

ML with Tensorflow-new - GitHub Pageshunkim.github.io/ml/lec11.pdf · 2017-10-02 · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -68 27 Jan 2016 Case Study: AlexNet [Krizhevsky

Lecture 11 - Stanford Universitycs231n.stanford.edu/slides/2016/winter1516_lecture11.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 11 - 47 17 Feb 2016 The power of

Lecture’12:’Clustering’and’ Segmentaon’ - Artificial Intelligencevision.stanford.edu/teaching/cs131_fall1415/lectures/... · 2014-10-28 · Lecture 12 - !!! Fei-Fei Li!

Lecture 9: EpipolarGeometry - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1112/...Fei-Fei Li Lecture 9 - 9 24-Oct-11 Stereo-view geometry • Correspondence: Given

ML with Tensorflow-new - GitHub Pages · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 -68 27 Jan 2016 Case Study: AlexNet [Krizhevsky et al. 2012] Full (simplified) AlexNet

Lecture 1: Introduction - Artificial Intelligencevision.stanford.edu/teaching/cs231n/slides/2016/... · scene understanding • CS231n (this term, Prof. Fei-Fei Li & Andrej Karpathy

Lecture 7: Convolutional Neural Networks - Stanford …cs231n.stanford.edu/slides/2016/winter1516_lecture7.pdfLecture 7: Convolutional Neural Networks Fei-Fei Li & Andrej Karpathy

Lecture’8:’’ CameraCalibraon’ - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1213/lecture/...Lecture 8 - !!! Fei-Fei Li! Whatwe’will’learn’today?’

Lecture 5: Training Neural Networks, Part I - GitHub Pages · Widrow and Hoff, ~1960: Adaline/Madaline * Original slides borrowed from Andrej Karpathy and Li Fei-Fei, Stanford cs231n

Lecture’9:’’ Epipolar’Geometry’ - Artificial Intelligencevision.stanford.edu/.../lecture/lecture9_epipolar_geometry_cs231a.pdf · Fei-Fei Li! 9 197Oct12 ... Find’coordinates’of’3D’pointfrom’

Deep Visual-Semantic Alignments for Generating Image ... · Deep Visual-Semantic Alignments for Generating Image Descriptions Andrej Karpathy Li Fei-Fei Department of Computer Science,

Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 ...vision.stanford.edu/teaching/cs231n/slides/2016/... · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 14 - Lecture

Lecture 13 - vision.stanford.eduvision.stanford.edu/.../2016/winter1516_lecture13.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 13 - 24 Feb 2016 Learnable Upsampling:

Lecture 8: Camera Calibration - Artificial Intelligencevision.stanford.edu/teaching/cs231a_autumn1112/... · Fei-Fei Li Lecture 8 - 21 19-Oct-11 Affine cameras • Weak perspective

Lecture 29: Optimization and Neural Nets CS4670/5670: Computer Vision Kavita Bala Slides from Andrej Karpathy and Fei-Fei Li

ML with Tensorflow-new - GitHub Pageshunkim.github.io/ml/lec12.pdf · e.g. Machine Translation seq of words -> seq of words. Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture

Recurrent Neural Networks - Artificial Intelligencevision.stanford.edu/teaching/cs231n/slides/2016/... · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 10 - 38 8 Feb 2016

Lecture 7: Convolutional Neural Networksvision.stanford.edu/.../2016/winter1516_lecture7.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 7 - 50 27 Jan 2016 The brain/neuron

Lecture 12 - Stanford Universityvision.stanford.edu/.../cs231n/slides/2016/winter1516_lecture12.pdf · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 12 ... Fei-Fei Li & Andrej

Lecture 6: Training Neural Networks, Part 2vision.stanford.edu/teaching/cs231n/slides/2016/... · Fei-Fei Li & Andrej Karpathy & Justin Johnson Lecture 6 - 3 25 Jan 2016 Mini-batch