scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game...

21
scaling up RL with func1on approxima1on [email protected]

Transcript of scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game...

Page 1: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

scalingupRLwithfunc1onapproxima1on

[email protected]

Page 2: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

Human-levelcontrolthroughdeepreinforcementlearning,Mnihet.al.,Nature518,Feb2015hHp://www.nature.com/nature/journal/v518/n7540/full/nature14236.html

•pixelinput

•18joys-ck/bu3onposi-onsoutput

•changeingamescoreasfeedback

• convolu-onalnetrepresen-ngQ

•backpropaga-onfortraining!

humanlevelgamecontrol

http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html
Page 3: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

neuralnetwork

Page 4: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

convolu1on,weightsharing,andpooling

pixel

sharedfeaturedetector/kernel/filterw

pooled

featuremap

max(window)

fewerparametersduetosharingandpooling!

Page 5: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

reverseprojec-onsofneuronoutputsinpixelspace

whatdoesadeepneuralnetworkdo?

Page 6: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature
Page 7: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature
Page 8: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

backpropaga1on?Whatisthetargetagainstwhichtominimiseerror?

Page 9: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

prac1callyspeaking…minimiseMSEbySGD

Page 10: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

experiencereplay

savecurrenttransi-on(s,a,r,s’)inmemory

every1mestep

randomlysampleasetof(s,a,r,s’)frommemoryfortrainingQnetwork

(insteadoflearningfromcurrentstatetransi-on)

everystep=i.i.d+learnfromthepast

at

st

st+1

rt+1

Page 11: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

freezingtargetQmovingtarget=>oscilla-ons

stabiliselearningbyfixingtarget,movingiteverynowandthen

freeze

Page 12: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

doubleDQN

evalua1onoftargetac1on

selec1onoftargetac1on

maxQa’

DeepReinforcementLearningwithDoubleQ-learningvanHasseltet.al.,AAAI2016hHps://arxiv.org/pdf/1509.06461v3.pdf

https://arxiv.org/pdf/1509.06461v3.pdf
Page 13: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

priori1sedexperiencereplay

sample(s,a,r,s’)frommemory

basedonsurprise

Priori1sedExperienceReplaySchaulet.al.,ICLR2016hHps://arxiv.org/pdf/1511.05952v4.pdf

https://arxiv.org/pdf/1511.05952v4.pdf
Page 14: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

Combiningdecoupling(double),priori1sedreplay,andduellinghelps!

duellingarchitecture

Q(s,a)=V(s,u)+A(s,a,v)

u

v

Q

Q

DuelingNetworkArchitecturesforDeepRLWanget.al.,ICML2016hHps://arxiv.org/pdf/1511.06581v3.pdf

ac1ona’sadvantage

https://arxiv.org/pdf/1511.06581v3.pdf
Page 15: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

V~avgQQ1

Q2

Q3

Q4

V~avgQQ1

Q2

Q3

Q4

arrowsrepresentadvantageofanac1on

advantage?e.g.4ac1ons,

V=E[Q]~Qaveragedacrossac1ons

Page 16: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

howevertrainingis

SLOW

Page 17: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

makingdeepRLfasterandwilder(more

applicableintherealworld)!

Page 18: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

dataefficientexplora1on?

parallelism?

transferlearning?

makinguseofamodel?

Page 19: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

Q Q Q Q Q

Q Q Q Q Q

Q Qt

Qt Qt Qt Qt Qt

Qt Qt Qt Qt Qt

sharedparamsforQandtargetQ

parallellearnersgemngindividual

experiences

lock-freeparamupdates

AsynchronousMethodsforDeepReinforcementLearning,Mnihet.al.,ICML2016hHp://jmlr.org/proceedings/papers/v48/mniha16.pdf

http://jmlr.org/proceedings/papers/v48/mniha16.pdf
Page 20: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

codeforyoutoplaywith...

Telenor’sownimplementa1onofasynchronousdeepRL:hHps://github.com/traai/async-deep-rl

hHps://openrl.slack.comLet’skeeptheconversa1ongoing:

Joinusingyourstudent.matnat.uio.noemailaddress!

https://github.com/traai/async-deep-rl
https://gym.openai.com/
http://student.matnat.uio.no
Page 21: scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

intern with us in summer 2017 CV and motivation to:

[email protected] | [email protected] (2 positions)

topic still to be finalised, but it will be at the intersection of software

engineering and machine learning


MSS Pooling

MSS Pooling

Pixel por Pixel

Pixel por Pixel

Risk Pooling-Logistics

Risk Pooling-Logistics

Pooling Paper

Pooling Paper

The Pooling Effect

The Pooling Effect

Car Pooling

Car Pooling

Land Pooling

Land Pooling

Pooling and Spacing Eric R. King. 165:5-7-7: Pooling.

Pooling and Spacing Eric R. King. 165:5-7-7: Pooling.

POOLING Software

POOLING Software

Pooling Grua35K

Pooling Grua35K

Pipelayers RL 44 RL 54 RL 64 - Liebherr Group · RL 44 RL 54 RL 64. 2 RL 44 Litronic RL 54 Litronic RL 64 Litronic RL 44 RL 54 ... pulling heavy pipe-bending machines - the hydrostatic

Pipelayers RL 44 RL 54 RL 64 - Liebherr Group · RL 44 RL 54 RL 64. 2 RL 44 Litronic RL 54 Litronic RL 64 Litronic RL 44 RL 54 ... pulling heavy pipe-bending machines - the hydrostatic

Pooling and polling : creation of pooling in inventory … · Pooling and Polling Creation of Pooling in Inventory and Queueing Models PROEFSCHRIFT ter verkrijging van de graad van

Pooling and polling : creation of pooling in inventory … · Pooling and Polling Creation of Pooling in Inventory and Queueing Models PROEFSCHRIFT ter verkrijging van de graad van

Forced Pooling in Oklahoma - Oklahoma Corporation … Pooling Process in Oklahoma.pdfThe Pooling Process in Oklahoma Most OCC rules pertaining to forced pooling are found in Chapter

Forced Pooling in Oklahoma - Oklahoma Corporation … Pooling Process in Oklahoma.pdfThe Pooling Process in Oklahoma Most OCC rules pertaining to forced pooling are found in Chapter

Pooling Basics Spring 2017 - schd.wsschd.ws/hosted_files/2017governanceleadershipconferen/53/Pooling... · Pooling Basics Spring 2017 ... Each state has the authority to regulate

Pooling Basics Spring 2017 - schd.wsschd.ws/hosted_files/2017governanceleadershipconferen/53/Pooling... · Pooling Basics Spring 2017 ... Each state has the authority to regulate

Opinion pooling and portfolio optimization - Rmetrics · Introduction Black-Litterman:Theory Copula Opinion Pooling: Theory References Opinion pooling and portfolio optimization Francisco

Opinion pooling and portfolio optimization - Rmetrics · Introduction Black-Litterman:Theory Copula Opinion Pooling: Theory References Opinion pooling and portfolio optimization Francisco

CS6320 – Connection Pooling

CS6320 – Connection Pooling

Cash Pooling Multidivisas

Cash Pooling Multidivisas

SGSN Pooling - Cisco · Monitoring and Troubleshooting the SGSN Pooling feature SGSN Pooling Show Command(s) and/or Outputs ThissectionprovidesinformationregardingshowcommandsandtheiroutputsinsupportoftheSGSNPooling:

SGSN Pooling - Cisco · Monitoring and Troubleshooting the SGSN Pooling feature SGSN Pooling Show Command(s) and/or Outputs ThissectionprovidesinformationregardingshowcommandsandtheiroutputsinsupportoftheSGSNPooling:

RL 46 RL 56 RL 66

RL 46 RL 56 RL 66

Pixel Pixo - Poesia Pixel

Pixel Pixo - Poesia Pixel