Decision making in episodic environments We have just looked at decision making in sequential...
-
date post
19-Dec-2015 -
Category
Documents
-
view
217 -
download
2
Transcript of Decision making in episodic environments We have just looked at decision making in sequential...
![Page 1: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/1.jpg)
Decision making in episodic environments
• We have just looked at decision making in sequential environments
• Now let’s consider the “easier” problem of episodic environments– The agent gets a series of unrelated problem
instances and has to make some decision or inference about each of them
– This is what most of “machine learning” is about
![Page 2: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/2.jpg)
Example: Image classification
apple
pear
tomato
cow
dog
horse
input desired output
![Page 3: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/3.jpg)
Example: Spam Filter
![Page 4: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/4.jpg)
Example: Seismic data
Body wave magnitude
Sur
face
wav
e m
agni
tude
Nuclear explosions
Earthquakes
![Page 5: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/5.jpg)
The basic classification framework
y = f(x)
• Learning: given a training set of labeled examples {(x1,y1), …, (xN,yN)}, estimate the parameters of the prediction function f
• Inference: apply f to a never before seen test example x and output the predicted value y = f(x)
output classification function
input
![Page 6: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/6.jpg)
Example: Training and testing
• Key challenge: generalization to unseen examples
Training set (labels known) Test set (labels unknown)
![Page 7: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/7.jpg)
Naïve Bayes classifier
ddy
y
y
yxPyP
yPyP
yPf
)|()(maxarg
)|()(maxarg
)|(maxarg)(
x
xx
A single dimension or attribute of x
![Page 8: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/8.jpg)
Decision tree classifier
Example problem: decide whether to wait for a table at a restaurant, based on the following attributes:1. Alternate: is there an alternative restaurant nearby?
2. Bar: is there a comfortable bar area to wait in?
3. Fri/Sat: is today Friday or Saturday?
4. Hungry: are we hungry?
5. Patrons: number of people in the restaurant (None, Some, Full)
6. Price: price range ($, $$, $$$)
7. Raining: is it raining outside?
8. Reservation: have we made a reservation?
9. Type: kind of restaurant (French, Italian, Thai, Burger)
10. WaitEstimate: estimated waiting time (0-10, 10-30, 30-60, >60)
![Page 9: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/9.jpg)
Decision tree classifier
![Page 10: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/10.jpg)
Decision tree classifier
![Page 11: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/11.jpg)
Nearest neighbor classifier
f(x) = label of the training example nearest to x
• All we need is a distance function for our inputs• No training required!
Test example
Training examples
from class 1
Training examples
from class 2
![Page 12: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/12.jpg)
Linear classifier
• Find a linear function to separate the classes
f(x) = sgn(w1x1 + w2x2 + … + wDxD) = sgn(w x)
![Page 13: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/13.jpg)
Perceptron
x1
x2
xD
w1
w2
w3
x3
wD
Input
Weights
.
.
.
Output: sgn(wx + b)
![Page 14: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/14.jpg)
Linear separability
![Page 15: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/15.jpg)
Multi-Layer Neural Network
• Can learn nonlinear functions• Training: find network weights to minimize the error between true and
estimated labels of training examples:
• Minimization can be done by gradient descent provided f is differentiable– This training method is called back-propagation
N
iii fyfE
1
2)()( x
![Page 16: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/16.jpg)
Differentiable perceptron
x1
x2
xd
w1
w2
w3
x3
wd
Sigmoid function:
Input
Weights
.
.
.te
t 1
1)(
Output: (wx + b)
![Page 17: Decision making in episodic environments We have just looked at decision making in sequential environments Now let’s consider the “easier” problem of episodic.](https://reader030.fdocuments.net/reader030/viewer/2022032800/56649d2e5503460f94a05395/html5/thumbnails/17.jpg)
Review: Types of classifiers
• Naïve Bayes • Decision tree• Nearest neighbor• Linear classifier• Nonlinear classifier