Learning First Order Markov Models for ControlPieter Abbeel and Andrew Y. Ng, Poster 48 Tuesday

Consider modeling an autonomous RC-car’s dynamics from a sequence of states and actions collected at 100Hz.

We have training data: (s1, a1, s2, a2, …).

We’d like to build a model of the MDP’s transition probabilities P(st+1|st, at). Slide #1

Learning First Order Markov Models for ControlPieter Abbeel and Andrew Y. Ng, Poster 48 Tuesday

• If we use maximum likelihood (ML) to fit the parameters of the MDP, then we are constrained to fit only the 1-step transitions:

max t p(st+1 | st, at)

• But in RL, our goal is to maximize the long-term rewards, so we aren’t really interested in the 1/100th-second dynamics.

• The dynamics on longer time-scales are often only poorly approximated (assuming the system isn’t really first-order).

• Algorithms for building models that better capture dynamics on longer time-scales.

• Experiments on autonomous RC car driving.

Slide #2

Learning First Order Markov Models for Control

Pieter Abbeel and Andrew Y. Ng

Stanford University

Autonomous RC Car

Motivation

• Consider modeling an RC-car’s dynamics from a sequence of states and actions collected at 100Hz.

• Maximum likelihood fitting of a first order Markov model constrains the model to fit only the 1-step transitions. However for control applications, we do not care only about the dynamics on the time-scale of 1/100 of a second, but also about longer time-scales.

Motivation

• If we use maximum likelihood (ML) to fit the parameters of a first-order Markov model, then we are constrained to fit only the 1-step transitions.

• The dynamics on longer time-scales are often only poorly approximated [unless the system dynamics are really first-order].

• However for control: interested in maximizing the long-term expected rewards.

Regardless of true model, ML will return

same model with .

Random Walk Example

• Random walk.

• Consider two cases

Increments i perfectly correlated: Var(ST) = T2.

Increments i independent: Var(ST) = T.

Examples of physical systems

• Influence of wind disturbances on helicopterVery small over one time step.Strong correlations lead to substantial effect over time.

• First order ML model may overestimate ability to control helicopter and car [thinking variance is O(T) rather than O(T2)]. This leads to danger of, e.g., flying too close to a building, or driving on too narrow a road.

• Systematic model errors can show up as correlated noise. E.g., oversteering or understeering of car.

Problem statement

• The learning problem: Given: state/action sequence data from a system. Goal: model the system for purposes of control (such as

to use with a RL algorithm).

• Even when dynamics are not governed by an MDP, we often would still like to model it as such (rather than as a POMDP), since MDPs are much easier to solve.

• How do we learn an accurate first order Markov model from data for control?

[Our ideas are also applicable to higher order, and/or more structured models such as dynamic Bayesian networks and mixed memory Markov models.]

Preliminaries and Notation

• Finite-state decision process (DP) S: set of states, A: set of actions, P: set of state transition probabilities

[not Markov!] : discount factor, D: initial state distribution, R: reward function, 8 s R(s) · Rmax .

• We will fit a model , with estimates of the transition probabilities .

• Value of state s0 in under policy

Where is the variational distance.

Parameter estimation when no actions

• Consider

• dvar is hard to optimize from samples, but can be upper-bounded by a function of KL-divergence.

• Minimizing KL-divergence is, in turn, identical to minimizing log-loss.

dvarKLlog-likelihood

[The last step reflects we are equally interested in every state

as possible starting state s0.]

The resulting lagged objective

• Given a training sequence s0:T, we propose to use

• Compare this to the maximum likelihood objective

S2S1

S2

S1

Lagged objective vs. ML

S0 S3S2S1

S0 S1

S2S1

S3S2

S0 S2

S3S1

S0 S3

• Consider a length four training sequence, which could have various dependencies.

• ML takes into account only the following transitions:

• Our lagged objective also takes into account:

[Yellow nodes are observed, white nodes are unobserved.]

• M-step: update such that

EM-algorithm to optimize lagged objective

• E-step: compute expected counts

and store in stats. I.e., 8 t, k, l, i, j

Computational Savings for E-step

• Inference for E-step can be done using standard forward and backward message passing. For every pair (t, t+k), the forward messages at position t+i depend on t only, not on k. So, computation of different terms in the inner-summation can share messages. Similarly for backward messages. This reduces the number of message computations by a factor T.

• Often only interested in some maximum horizon H. I.e., in the inner-summation of the objective only consider k=1,…,H.

Reduction from O(T3) to O(T H2).

• More substantial savings: (St=i, St+k=j) and (St’=i, St’+k=j) contribute same to stats( . , . )

Computing stats( . , . ) contribution for all such pairs only once.

Further reduction to O(|S|2 H2).

Incorporating actions

• If actions are incorporated, our objective becomes:

• The EM-algorithm is trivially extended by conditioning

on the actions during the E-step.

• Forward messages need to be computed only once for

every t, backward messages once for every t+k. [as before]

• Number of possibilities for at:t+k-1 is O(|A|k).

Use only a few deterministic exploration policies.

Can still obtain same computational savings as before.

Experiment 1: shortest vs. safest path

• Actions are 4 compass directions.

• Move in intended direction with probability 0.7, and a random direction with probability 0.3.

• The directions of the “random transitions” are dependent, and correlated over time. A parameter q controls the correlation between the directions of the random transitions on different time steps (uncorrelated if q=0, perfectly correlated if q=1).

• We will fit a first order Markov model to these dynamics (with each grid position being a state).

[Details: Noise process governed by a Markov process (not directly observable by the agent) with each of the 4 directions as states, with Prob(staying in same state) = q.]

Experiment 1: shortest vs. safest path

[Details: Learning was done using a 200,000 length state-action sequence. Reported results are averages over 5 independent trials. The exploration policy used independent random actions at each time step.]

If the noise is strongly correlated across time (large q), our model estimates the dynamics to have a higher “effective noise level.” As a consequence the more cautious policy (path B) is used.

(q)

Experiment 2: Queue

Actions: 3 service rates, with faster service rates being more expensive.

q0 = 0 reward = 0

q1 = p reward = -1

q2 = .75 reward = -10

Queue buffer length = 20; buffer overflow results in reward -1000.

Customers arrive over time to be served.

At every time, the arrival probability equals p.

Service rate = probability that the customer first in queue

gets serviced successfully in the current time step.

P( arrival | slow mode ) = 0.01

P( arrival | fast mode ) = 0.99

Steady state: P(slow mode)=0.8, P(fast mode)=0.2

Experiment 2: Queue

Underlying (unobserved!) arrival process has 2 different modes (fast arrivals and slow arrivals)

Additional parameter determines how rapidly system changes between fast and slow modes.

Fast switching

Slow switchingbetween modes

between modes

Experiment 2: Queue

Estimate/Learn first order Markov model with State = size of the queue, Actions = 3 service rates Exploration policy = repeatedly use same service rate for 25 time-steps. We used 8000 such trials.

15% better performance

at high correlation levels.

Same performance

at low correlation levels.

Experiment 3: RC-car

Consider the situation where the RC-car can choose between 2 paths

• A curvy path with high reward if successful in reaching the goal.

• An easier path with lower reward if successful in reaching the goal

We build a dynamics model of the car, and find a policy/controller in simulation for following each of the paths. The decision about which path to follow is then made based upon this simulation.

RC-car model

• : angular direction the RC-car is headed

• : angular velocity

• V : velocity of the RC-car (kept constant)

• ut : steering input to the car ( 2 [-1,1])

• C1, C2, C3 : parameters of the model, estimated using linear regression

• wt : noise term, zero-mean Gaussian with variance 2

.

Using the lagged objective, we re-estimate the variance 2, and compare its performance to the first-order estimate of 2.

Controller

• We use the following controller

desired steering angle = p1*(y-ydes) + p2*(-des);

u = f(desired steering angle);

We optimize over the parameters p1, p2 to follow the straight line y=0, for which we set ydes=0, des=0.

For the specific two trajectories, ydes(x), des(x) are optimized as a function of the current x position.

• For localization, we use an overhead camera.

Simulated performance on curvy trajectory

Plot shows 100 sample runs in simulation under the ML-model.

The ML-model predicts the RC-car can follow the curvy road >95% of the time.

Plot shows 10 sample runs in simulation under the lag-learned model.

The lag-learned model predicts the RC-car can follow the curvy road < 10% of the time.

Green lines: simulated trajectories, Black lines: road boundaries.

Simulated performance on easier trajectory

Plot shows 100 sample runs in simulation under the ML-model.

The ML-model predicts the RC-car can follow the easier road >99% of the time.

Plot shows 100 sample runs in simulation under the lag-learned model.

The lag-learned model predicts the RC-car can follow the curvy road > 70% of the time.

Green lines: simulated trajectories, Black lines: road boundaries.

ML would choose the curvy road if high reward along curvy road.

Actual performance on easier trajectory

[Movies available.]

The real RC-car succeeded on the easier road 20/20 times.

The real RC-car failed on the curvy road 19/20 times.

RC-car movie

Conclusions

• Maximum likelihood with a first order Markov model only tries to model the 1-step transition dynamics.

• For many control applications, we desire an accurate model of the dynamics on longer time-scales.

• We showed that, by using an objective that takes into account the longer time scales, in many cases a better dynamical model (and a better controller) is obtained.

Special thanks to Mark Woodward, Dave Dostal, Vikash Gilja and Sebastian Thrun.

Cut out slides follow

Lagged objective vs. ML

• Consider a length four training sequence, which could have various dependencies.

• ML takes into account only the following transitions.

• Our lagged objective also takes into account

[Shaded nodes are observed, white nodes are unobserved.]

Experiment 2: Queue [use this one or previous one?]

Queue size at time t Queue size at time t+1

s(t)

s(t+1) = s(t)+1

s(t+1) = s(t)

s(t+1) = s(t)-1

arrival

no arrival

unsuccessful servicing

unsuccessful servicing

successful servicing

successful servicing

Choice of actions between 3 service rates

q0 = 0 reward = 0

q1 = p reward = -1

q2 = .75 reward = -10

Buffer size = 20. Buffer overflow results in reward of -1000.

Arrival probability = p

Actual performance on curvy trajectory

[Movies available.]

Green lines: simulated trajectories, Black lines: road boundaries.

Real trajectories obtained as obtained on floor.

The actual RC-car fell off the curvy trajectory 19/20 times.

Alternative title slides follow

Learning First Order Markov Models for Control

Pieter Abbeel and Andrew Y. Ng

Stanford University

Learning First

Order Markov

Models for

Control

Pieter Abbeel and Andrew Y. Ng

Stanford University

Pieter Abbeel and Andrew Y. Ng

Stanford University