scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving...

20
scaling up RL with func1on approxima1on

Transcript of scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving...

Page 1: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

scalingupRLwithfunc1onapproxima1on

Page 2: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

Human-levelcontrolthroughdeepreinforcementlearning,Mnihet.al.,Nature518,Feb2015hFp://www.nature.com/nature/journal/v518/n7540/full/nature14236.html

•pixelinput

•18joys-ck/bu3onposi-onsoutput

•changeingamescoreasfeedback

• convolu-onalnetrepresen-ngQ

•backpropaga-onfortraining!

humanlevelgamecontrol

http://www.nature.com/nature/journal/v518/n7540/full/nature14236.html
Page 3: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

neuralnetwork

Page 4: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

convolu1on,weightsharing,andpooling

pixel

sharedfeaturedetector/kernel/filterw

pooled

featuremap

max(window)

fewerparametersduetosharingandpooling!

Page 5: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

reverseprojec-onsofneuronoutputsinpixelspace

whatdoesadeepneuralnetworkdo?

Page 6: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)
Page 7: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)
Page 8: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

composi1onalfeaturescomposi1onalproblem

solvingmul1plica1on(circuitdesign)

<—composedofaddingnumbers

<—composedofaddingbits

output:x.y—————multiply————————addingnums———————addingbits————input:xandy

humanknowledgeorganisa1on

findrootsofalinearexpression

<—composedofseWngexpressiontozeroandsolvinglinearequa1ons

<—composedofrearrangingterms

output:x=-2—————(findrootsofx+2)————————————(setx+2=0)—(solve)——————————————————————(rearrange)—————input:x,+,2,=,0

deeplayersmakerepresenta1onofknowledgeandprocesseshappenwithfewerneurons!

Page 9: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

backpropaga1on?Whatisthetargetagainstwhichtominimiseerror?

Page 10: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

prac1callyspeaking…minimiseMSEbySGD

Page 11: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

experiencereplay

savecurrenttransi-on(s,a,r,s’)inmemory

every1mestep

randomlysampleasetof(s,a,r,s’)frommemoryfortrainingQnetwork

(insteadoflearningfromcurrentstatetransi-on)

everystep=i.i.d+learnfromthepast

at

st

st+1

rt+1

Page 12: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

freezingtargetQmovingtarget=>oscilla-ons

stabiliselearningbyfixingtarget,movingiteverynowandthen

freeze

Page 13: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

doubleDQN

evalua1onoftargetac1on

selec1onoftargetac1on

maxQa’

DeepReinforcementLearningwithDoubleQ-learningvanHasseltet.al.,AAAI2016hFps://arxiv.org/pdf/1509.06461v3.pdf

https://arxiv.org/pdf/1509.06461v3.pdf
Page 14: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

priori1sedexperiencereplay

sample(s,a,r,s’)frommemory

basedonsurprise

Priori1sedExperienceReplaySchaulet.al.,ICLR2016hFps://arxiv.org/pdf/1511.05952v4.pdf

https://arxiv.org/pdf/1511.05952v4.pdf
Page 15: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

Combiningdecoupling(double),priori1sedreplay,andduellinghelps!

duellingarchitecture

Q(s,a)=V(s,u)+A(s,a,v)

u

v

Q

Q

DuelingNetworkArchitecturesforDeepRLWanget.al.,ICML2016hFps://arxiv.org/pdf/1511.06581v3.pdf

https://arxiv.org/pdf/1511.06581v3.pdf
Page 16: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

howevertrainingis

SLOW

Page 17: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

makingdeepRLfasterandwilder(more

applicableintherealworld)!

Page 18: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

dataefficientexplora1on?

parallelism?

transferlearning?

makinguseofamodel?

Page 19: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

Q Q Q Q Q

Q Q Q Q Q

Q Qt

Qt Qt Qt Qt Qt

Qt Qt Qt Qt Qt

sharedparamsforQandtargetQ

parallellearnersgeWngindividual

experiences

lock-freeparamupdates

AsynchronousMethodsforDeepReinforcementLearning,Mnihet.al.,ICML2016hFp://jmlr.org/proceedings/papers/v48/mniha16.pdf

http://jmlr.org/proceedings/papers/v48/mniha16.pdf
Page 20: scaling up RL with func1on approxima1on · composi1onal features composi1onal problem solving mulplicaon (circuit design)

codeforyoutoplaywith...

Telenor’sownimplementa1onofasynchronousdeepRL:hFps://github.com/traai/async-deep-rl

hFps://openrl.slack.comLet’skeeptheconversa1ongoing:

https://github.com/traai/async-deep-rl
https://gym.openai.com/

RL 1, RL 2, RL 3, RL 4, RL 5, RL 7, RL 9, RL 10, RI 1, RI ...juliebernard.weebly.com/uploads/4/7/0/5/47054607/...Media Literacy pp. 67–69 RI 7 Legacy of the Era pp. 34–35 W 7 Quickwrite

RL 1, RL 2, RL 3, RL 4, RL 5, RL 7, RL 9, RL 10, RI 1, RI ...juliebernard.weebly.com/uploads/4/7/0/5/47054607/...Media Literacy pp. 67–69 RI 7 Legacy of the Era pp. 34–35 W 7 Quickwrite

Storage: Scaling Out > Scaling Up?

Storage: Scaling Out > Scaling Up?

NOMINATED RL LIST2201528586GUNPAL RL\827 2406502093RIYASAT ALI RL\828 2405010222JITENDRA KUMAR SHARMA RL\831 2405519571TILAK CHOUDHARY RL\832 2201007005BALWAN SINGH RL\834 2405500408TARACHAND

NOMINATED RL LIST2201528586GUNPAL RL\827 2406502093RIYASAT ALI RL\828 2405010222JITENDRA KUMAR SHARMA RL\831 2405519571TILAK CHOUDHARY RL\832 2201007005BALWAN SINGH RL\834 2405500408TARACHAND

Vance County Schools English II Pacing Guide 2016-2017 RL ...€¦ · RL. 9-10.1 RL. 9-10.2 RL. 9-10.3 RL. 9-10.4 RL. 9-10.6 Introduction to Course and the Following Topic: -Syllabus

Vance County Schools English II Pacing Guide 2016-2017 RL ...€¦ · RL. 9-10.1 RL. 9-10.2 RL. 9-10.3 RL. 9-10.4 RL. 9-10.6 Introduction to Course and the Following Topic: -Syllabus

FOR ISSUED - @NSWDepartmentofEducation · 2019-10-10 · learning learning assembly / courtyard street play pedestrian entry rl 50,150 rl 50,150 rl 50,500 rl 50,500 rl 50,500 rl 51,000

FOR ISSUED - @NSWDepartmentofEducation · 2019-10-10 · learning learning assembly / courtyard street play pedestrian entry rl 50,150 rl 50,150 rl 50,500 rl 50,500 rl 50,500 rl 51,000

YECUSEX CHUPAN Rl RUK'ATAJ Rl JESUCRISTOcenturypubl.com/wp-content/uploads/cakchiquel-lds-words-phrases-1.pdfRl TZIJ Rl YECUSEX CHUPAN Rl RUK'ATAJ \. Rl JESUCRISTO ,,, Cakch iquel

YECUSEX CHUPAN Rl RUK'ATAJ Rl JESUCRISTOcenturypubl.com/wp-content/uploads/cakchiquel-lds-words-phrases-1.pdfRl TZIJ Rl YECUSEX CHUPAN Rl RUK'ATAJ \. Rl JESUCRISTO ,,, Cakch iquel

Scaling of popular Sudoku solving algorithms - DiVA portal721634/FULLTEXT01.pdf · Scaling of popular Sudoku solving algorithms DD143X: Degree Project in Computer Science Authors

Scaling of popular Sudoku solving algorithms - DiVA portal721634/FULLTEXT01.pdf · Scaling of popular Sudoku solving algorithms DD143X: Degree Project in Computer Science Authors

Central Sydney Planning Committee (CSPC) - 23 June 2016 - Item … · 2016. 6. 23. · rl 24.48 rl 24.48 rl 24.35 rl 24.35 rl 24.35 rl 25.60 rl 25.60 rl 24.18 3a lobby 82.9 m² 3a

Central Sydney Planning Committee (CSPC) - 23 June 2016 - Item … · 2016. 6. 23. · rl 24.48 rl 24.48 rl 24.35 rl 24.35 rl 24.35 rl 25.60 rl 25.60 rl 24.18 3a lobby 82.9 m² 3a

 · 2019-12-18 · 3 € rl-4 rl-8 rl-2-307-rs rl-4-307-rs rl-8-307-rs rl-4 Высокопроизводительныйнасосслегкимвесом ...

 · 2019-12-18 · 3 € rl-4 rl-8 rl-2-307-rs rl-4-307-rs rl-8-307-rs rl-4 Высокопроизводительныйнасосслегкимвесом ...

La complémentaire santé collective recommandée par la ... · rg rl rg rl rg rl rg rl rg rl ... 290, 00 € 6,86 € 8,23 € 12,35 € 220,00 € 220,00 € 61,77 € 57,65 €

La complémentaire santé collective recommandée par la ... · rg rl rg rl rg rl rg rl rg rl ... 290, 00 € 6,86 € 8,23 € 12,35 € 220,00 € 220,00 € 61,77 € 57,65 €

RL 46 RL 56 RL 66

RL 46 RL 56 RL 66

APPROACH SURFACE HORIZONTAL SECTION RL 400…dsewebapps.dse.vic.gov.au/Shared/ATSAttachment1.nsf/(attachment... · rl 400 rl 400.52 rl 340 rl 350 rl 360 rl 370 rl 380 ... first section

APPROACH SURFACE HORIZONTAL SECTION RL 400…dsewebapps.dse.vic.gov.au/Shared/ATSAttachment1.nsf/(attachment... · rl 400 rl 400.52 rl 340 rl 350 rl 360 rl 370 rl 380 ... first section

RL-200 1S/2S 主な仕様 RL-200 1S RL-200 2S RL-200 1S/2S

RL-200 1S/2S 主な仕様 RL-200 1S RL-200 2S RL-200 1S/2S

scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

scaling up RL with func1on approxima1on€¦ · •backpropaga-on for training! human level game control. neural network. convolu1on, weight sharing, and pooling pixel shared feature

1 Scale: 1:200 · 2020. 8. 26. · SSD1.02/17-202.02/17-212.1 RL 58.38 RL 61.57 RL 65.275 RL 67.95 RL 54.99 RL 71.05 RL 54.65 PROPOSED RIDGE EXG + PROP 5TH FLOOR EXG + PROP 4TH FLOOR

1 Scale: 1:200 · 2020. 8. 26. · SSD1.02/17-202.02/17-212.1 RL 58.38 RL 61.57 RL 65.275 RL 67.95 RL 54.99 RL 71.05 RL 54.65 PROPOSED RIDGE EXG + PROP 5TH FLOOR EXG + PROP 4TH FLOOR

Solving Database Management, Migration, and Scaling Problems with DevOps Tools by Joshua Arvin Lat

Solving Database Management, Migration, and Scaling Problems with DevOps Tools by Joshua Arvin Lat

written permission of Rupert G.H. Milne Home, Keith Dunn ... · 7.1 RL 7.9 RL 7.5 RL 10.8 RL 9.9 RL 11.8 RL 10.9 RL 9.6 RL 8.4 RL 3.0 3.0 3.5 3.5 4.0 4.0 4.0 3.5 4.5 4.5 4.5 5.0 5.0

written permission of Rupert G.H. Milne Home, Keith Dunn ... · 7.1 RL 7.9 RL 7.5 RL 10.8 RL 9.9 RL 11.8 RL 10.9 RL 9.6 RL 8.4 RL 3.0 3.0 3.5 3.5 4.0 4.0 4.0 3.5 4.5 4.5 4.5 5.0 5.0

Scaling Energy Access in Africa: Solving the Battery Issue for Rural Clinics and Classrooms

Scaling Energy Access in Africa: Solving the Battery Issue for Rural Clinics and Classrooms

719 CITY PLANNING COMMITTEE MEETING 20 JULY 2016 … · 2019. 6. 23. · line basement line of basement rl 3.500 rl 3.650 rl 3.400 rl 3050 rl 2.800 rl 2.740 rl 2.720 rl 2.700 rl 2.460

719 CITY PLANNING COMMITTEE MEETING 20 JULY 2016 … · 2019. 6. 23. · line basement line of basement rl 3.500 rl 3.650 rl 3.400 rl 3050 rl 2.800 rl 2.740 rl 2.720 rl 2.700 rl 2.460

Map 9-1 DRAFT · RH/HR CO RMH RL RL RL RL MG MG MG ML RL RL RL MUC RML IN CN MP RL IN MUC RL RF 1:38,000 1. All plan designation boundaries are intended to follow property lines,

Map 9-1 DRAFT · RH/HR CO RMH RL RL RL RL MG MG MG ML RL RL RL MUC RML IN CN MP RL IN MUC RL RF 1:38,000 1. All plan designation boundaries are intended to follow property lines,