Adversarially Robust Policy Learning through

Active Construction of Physically-Plausible Perturbations

Ajay Mandlekar, Yuke Zhu, Animesh Garg, Li Fei-Fei, Silvio Savarese

1

Abstract

Policy search methods in reinforcement learn-ing have demonstrated success in scaling up tolarger problem sizes beyond toy examples. How-ever, deploying these methods on real robots re-mains challenging due to the large sample com-plexity required during learning and their vulner-ability to malicious intervention. We introduceAdversarially Robust Policy Learning (ARPL),an algorithm that leverages active computation ofphysically-plausible adversarial examples duringtraining to enable sample-efficient policy learn-ing in the source domain and robust performanceunder both random and adversarial input pertur-bations. We evaluate ARPL on four continu-ous control tasks and show superior resilience tochanges in physical environment dynamics pa-rameters and environment state as compared tostate-of-the-art robust policy learning methods.

1. Introduction

Renewed research focus on policy learning methods inreinforcement learning have enabled autonomy in manyproblems considered difficult until recently, such as videogames with visual input (Mnih et al., 2015), the determin-istic tree search problem in Go (Silver et al., 2016), roboticmanipulation skills (Levine et al., 2016b), and locomotiontasks (Lillicrap et al., 2015).

Imagine an autonomous robot making a parcel delivery. Al-though an all-out malicious attack can take down the robot,the robot might also be vulnerable to more subtle adversar-ial attacks. For example, a smart attacker could create aman-in-the-middle style attack which alters the policy be-havior only slightly so as to evade detection but get the par-cel delivered to himself at an unintended location resulting

1Department of Computer Science, Stanford University. Cor-respondence to: Ajay Mandlekar <amandlek@stanford.edu>.

Proceedings of the 34th International Conference on MachineLearning, Sydney, Australia, 2017. JMLR: W&CP. Copyright2017 by the author(s).

in endemic losses. While this is a hypothetical scenario,with increasingly pervasive autonomy in both personal andpublic spaces, it is a real threat.

As we move towards deploying learned controllers onphysical systems around us, robust performance is not onlya desired property but also a design requirement to ensurethe safety of both users and system itself. Machine learningresearch has shown that a spectrum of models, includingRL algorithms, are vulnerable to malicious attacks (Bar-reno et al., 2006; Biggio et al., 2013; Behzadan & Munir;Huang et al.). Recent works have studied the existence ofadversarial examples (Szegedy et al.; Goodfellow et al.;Papernot et al.); and showed that such instances are notonly easy to construct but are also effective against differ-ent models trained for the same task.

While successful, policy search algorithms, such as vari-ants of Q-Learning (Mnih et al., 2015) and Policy Gradi-ent methods (Lillicrap et al., 2015; Schulman et al., 2015),are highly data intensive. Furthermore, non-linear functionapproximators such as deep neural networks can worsenthe data dependence. Hence, a naıve approach that utilizesjoint training over an ensemble of domains to achieve ro-bustness can quickly become intractable.

This paper is a step towards addressing the problem of ro-bustness to malicious attacks while maintaining data effi-ciency in training. The key intuition of this study is that –adversarial examples can be used to actively choose per-turbations during training in the source domain. This pro-cedure exhibits robustness to both modeling errors and ad-versarial perturbations on the target domain without an in-crease in data-requirement. In particular, we explore train-ing in simulated continuous control tasks for evaluationacross varied simulated target domains. We analyze theeffect of perturbations on the performance via three kindsof modeling errors: model parameter uncertainty, processnoise, and observation noise.

Summary of Contributions:

1. We demonstrate that deep RL policies are susceptibleto both adversarial perturbations and model-mismatcherrors.

Adversarially Robust Policy Learning (ARPL)

Figure 1. A walker-2d agent trained on a fixed set of param-eters would be sensitive to noises in dynamics, process, and ob-servation. We propose a robust training method, ARPL, basedon adversarial examples to enhance the robustness of our modelagainst uncertainty.

2. We propose a method to synthesize physically-plausible adversarial perturbations for a white-boxmodel. Further, we present Adversarially Robust Pol-icy Learning method to use actively chosen adversar-ial perturbations for robust policy training.

3. We evaluate our method in 4 simulated environmentswith many policy training variations in each and ob-serve that training with ARPL results in improving cu-mulative reward.

2. Related Work

2.1. Deep Reinforcement Learning

Recent interest in reinforcement learning has resulted inthe resurgence of many classical algorithms that now useneural network function approximators, enabling problemswith larger state and action spaces (Mnih et al.; 2015; Lil-licrap et al., 2015). However, these methods are oftensample-inefficient and used mainly in simulated domains.Policy rollouts on target domain with real world robot con-trol are expensive and hence require sample efficient pol-icy learning. Despite recent results on robot reinforcementlearning (Levine et al., 2016b; Rusu et al., 2016b), the datarequirement for these studies has been very large, prompt-ing cynicism towards application to tasks with dynamicmotor skills.

A common approach to circumvent the sample com-plexity problem of model-free methods, is utilizingsample-efficient model-based reinforcement learning ap-proaches (Kober et al., 2013). Source simulated modelsapproximate the target real-world domain and provide a

computationally cheap way to learn policies. However, themain challenge in a source to target transfer is the system-atic discrepancy between the two domains (Taylor & Stone,2009).

2.2. Transfer in Reinforcement Learning

Transfer in RL algorithms to adapt across variations inmodeling errors is both important and an active area ofcurrent research. The transfer problem has been exam-ined from different perspectives, such as changing dynam-ics (Rajeswaran et al.), varying targets (Zhu et al., 2016),and multiple tasks (Devin et al., 2016; Rusu et al., 2016a).Taylor et al. provide an excellent treatise on the transferlearning problem (Taylor & Stone, 2009). A series of ap-proaches focused on reducing the number of rollouts per-formed on a physical robot by alternating between policyimprovement in simulation and physical rollouts (Abbeelet al., 2006; Levine et al., 2016a). After each physical roll-out, a time-dependent term is added to the dynamics to ac-count for unmodeled error. This approach, however, doesnot address robustness in the initial transfer, and the systemcould sustain or cause damage before the online learningmodel converges.

In contrast to previous work, we show that using activelychosen perturbations of the environment dynamics and ob-servations noise models can result in a more robust pol-icy in target domain during testing than randomly sampledsource domains.

2.3. Adversarial Examples in Supervised Learning

Concurrent with RL, another concern that is increasinglytaking center-stage is resilience and robustness of thelearned policies when deployed on critical systems. Ma-chine learning researchers have been exploring the effectof adversarial attacks in general machine learning mod-els (Barreno et al., 2006) and investigating both the ro-bustness and security of the models. Sequential decisionmaking is inherently vulnerable because of the ability of anadversary to intervene through both changing the underly-ing dynamics or the observations (Biggio et al., 2013). Forinstance, in an autonomous driving scenario, an adversarymay disrupt the controller by adding structured jitter to thecamera images or by changing the paint color on a STOPsign.

The notion of minimal perturbation of test-time input, im-perceptible to the human eye, to lead to misclassificationin Deep Neural Network (DNN) models was first shownfor computer vision applications in Szegedy et al. (Szegedyet al.). This has fueled an exciting direction in both de-tection and synthesis of adversarial examples (Goodfellowet al.; Kurakin et al.; Papernot et al.; Moosavi-Dezfooliet al.), and efforts to safeguard against such malicious at-

Adversarially Robust Policy Learning (ARPL)

tacks (Gao et al.; Grosse et al.; Moosavi-Dezfooli et al.,2016). It has since been discovered that adversarial inputsfor computer vision models can be computed with minimalcompute effort (Goodfellow et al.), can be applied to phys-ical print-outs of pictures (Kurakin et al.), and can be foundalmost universally for any given model (Moosavi-Dezfooliet al.). Furthermore, pre-designed adversarial attacks to re-liably quantify the robustness of these classifiers have beenexplored in (Papernot et al.) and (Moosavi-Dezfooli et al.,2016). And finally, researchers have also explored meth-ods to secure models against malicious attacks. Grosse etal. (Grosse et al.) show that adversarial examples are notdrawn from the same distribution than the original data, andcan thus be detected using statistical tests. Gao et al. (Gaoet al.) presents a method that limits the capacity an attackercan use generating adversarial samples and therefore in-crease the robustness against such inputs.

2.4. Robust and Adversarial Learning in Robotics

Robust control for modeling errors has been widely studiedin control theory. An excellent overview of methods is pro-vided in the book by Green & Limebeer (Green & Lime-beer, 2012). In the problem of robust transfer, we are inter-ested in parametric uncertainty between source and targetmodels. Nilim et al. (Nilim & El Ghaoui, 2005) and Mastinet al. (Mastin & Jaillet, 2012) have analyzed the bounds onthe performance of transfer in the presence of bounded dis-turbances in dynamics. Risk-sensitive and safe RL methodshave been proposed to handle uncertainty while enablingsafety, as reviewed in (Garcıa & Fernandez, 2015). Thesemethods model belief priors over the target domain andpreserve safety guarantees similar to robust control. How-ever scaling both robust control methods and risk-sensitiveRL methods beyond very simple examples has been a chal-lenge.

Recent studies on robust policy learning for transfer acrossdomains have adapted ideas from robust control and risk-sensitive RL to propose simplified sampling-based meth-ods for training. In particular, Rajeswaran et al. (Ra-jeswaran et al.) propose a method of sampling dynamicsparameters over a prior during training to improve policyrobustness to a similar but unseen target setting. Further,Yu et al. (Yu et al.) propose a similar robust policy learn-ing method through adding parameters to the system stateand with the additional option of performing an online es-timate of dynamics. Both of these methods use randomsampling methods which can be computationally challeng-ing for realistic domains where training a single policy ona stationary MDP is hard enough.

Additionally, it is worth noting that a majority of the studiesin adversarial perturbations have been for supervised learn-ing models. Recent works by Huang et al. (Huang et al.)

and Behzadan et al. (Behzadan & Munir) have illustratedthe existence and effectiveness of perturbations in RL. Thestudy in (Behzadan & Munir) shows that perturbations canbe constructed to prevent training convergence, and Huanget al. (Huang et al.) demonstrate the ability of an adversaryto interfere with the policy operation during test time.

This work is, to the best of our knowledge, one of thefirst studies to examine physically plausible perturbationsto not only observations but also to cause a systematic shiftin dynamics that result in a predictably worse policy per-formance. Furthermore, we also propose an algorithm toleverage adversarial perturbations to train policies that arerobust to a wide range of perturbations in dynamics andobservations.

3. Approach

3.1. Adversarial Perturbations in Deep Neural

Networks

Szegedy et al. (Szegedy et al.) discovered the insightful factthat deep learning models are highly vulnerable to adver-sarial examples. Furthermore, these adversarial examplesexhibit a remarkable generalization property - a variety ofmodels with different parametrization are often fooled bythe same adversarial example, even when these models aretrained on different subsets of the training data.

Methods of creating adversarial examples rely upon max-imizing the prediction error subject to a constraint on theperturbation size. In image classification, the objectiveis switching prediction classes, while minimizing the per-ceived image perturbation. In reinforcement learning, theobjective is to misguide the policy to output incorrect ac-tions, while minimizing the change in either the input state,the dynamics model, or the observation model. One of themost prevalent methods for generating adversarial exam-ples is the Fast Gradient Sign Method (FGSM) by Good-fellow et al. (Goodfellow et al.). FGSM offers computa-tionally efficiency at the cost of a slight decrease in attacksuccess rate. The FGSM method makes a linear approxi-mation of a Deep Neural Network and maximizes the ob-jective in a closed form.

FGSM focuses on adversarial perturbations of an image in-put where the change in each pixel is limited by e . Witha linear approximation of a DNN, p(s) = wTs, the optimalFGSM perturbation is defined as, d = e sign(w). Since wedefine d to be the perturbation, the output of the networkon the adversarial example s is p(s) = wTs+wTd . Nowassuming that the network output p(s;q) is instead a non-linear function parametrized by q , then a linearization ofthe loss function around the current input provides the fol-lowing perturbation

Adversarially Robust Policy Learning (ARPL)

d = e sign�—sh(pq (s))

�(FGSM) (1)

where h is a loss function over the policy p , which isparametrized by parameters q . Moreover, we note that akey assumption in FGSM is that the attacker has completeaccess to the target neural network – such as its architec-ture, weights, and other hyperparameters. This is a white-box setting. In contrast, in a black-box setting, the attackerdoes not have access to the parameters of the target net-work but only the output. Previous works have studied theblack-box setting where gradient computation can be donenumerically, and the attacker has unlimited query access tothe oracle. A common approach is to re-train a separatemodel for the same input and output space and leverage thetransferability of the adversarial examples to attack the tar-get policy. We restrict our analysis to a white-box settingbecause training the policy with numerical gradient esti-mates is no more efficient than computing the gradient onthe original policy.

3.2. Physically Plausible Threat Model

Consider a physical dynamical system:

xt+1 = f (xt ,ut ; µ)+n (Dynamics)

zt =g(xt)+w (Observation)(2)

where the Dynamics equation updates the state x withcontrol input u according to a function f , parametrizedby model parameters µ , and process noise n . TheObservation model maps the current state x to the ob-served state z with the observation function g and observa-tion noise w .

We use the full gradient method instead of the popularFGSM. FGSM is primarily designed for images where thestate is high dimensional and approx. IID. However, for dy-namical models, the state space is structured with differentdomains; hence a fixed unit step size can result in scalingissues.

We use an isometrically scaled version of the full gradient

d = e —sh(pq (s)) (ARPL) (3)

where h is a loss function over the policy p , which isparametrized by parameters q .

Type of Perturbation: A malicious adversary can changeeither of the three quantities µ,n , or w . A change in µ canbe equated to dynamics noise, i.e. uncertainty in physicalparameters such as mass, friction, and inertia, while n andw correspond to direct perturbations of state and observa-tion. Prior work in (Huang et al.; Behzadan & Munir) onlyexamines perturbations to the current state in image space,

i.e. n , which is often not physically plausible. We performperturbations to process noise n by adding gradient basedperturbation to the state of the system. Similarly, we addobservation noise to w by changing the observation whilepreserving the system state. Adversarial perturbation ondynamics noise through model parameters µ requires stateaugmentation s = [s,µ]T , and only the latter component ofthe gradient is used —s = [0,—µ ].

We maintain physical plausibility of all perturbationsthrough the projection of the perturbed state to its respec-tive domain, i.e. the state space for s and bounded variationin µ 2 [0.5µ0,1.5µ0], where µ0 is nominal (source) dynam-ics.

Modes of Perturbation: We build two threat modes: Ad-versarial and Random. For noise in states (n) and obser-vation (w), adversarial states are calculated using d , whilerandom perturbations are uniformly sampled from [�d ,d ].For dynamics noise, µ is set to be a uniform sample in[0.5µ0,1.5µ0] at each time iteration. For adversarial dy-namics noise, we first get a random sample as before, thenadd a gradient d evaluated at µadv ⇠U(0.5µ0,1.5µ0).

Frequency of Perturbation: The parameter f 2 [0,1] de-termines the frequency of applying adversarial (or random)updates. At each time step, an update is applied with prob.Bern(f). When f = 0, only the initial time step is per-turbed in each episode.

The three perturbation types described above combinedwith two modes of perturbation and three levels of pertur-bation frequency result in a total of 18 threat models. Eachmodel is also compared with the nominal source modelwith no changes as a baseline.

3.3. Robust Training with Adversarial Perturbations

Direct Policy Optimization methods utilize batch trajec-tory sampling for gradient estimates as in (Schulman et al.,2015). The core of the ARPL operates by modifying thetrajectory rollouts to include trajectory perturbation. In themost general setting, at each iteration, a trajectory is rolledout, and an adversarial perturbation is added to the modelwith probability f at each time step along the trajectory.The exact operation to compute the perturbation dependson the choice of threat model. A gradient update to the pol-icy parameters is then made after a rolling out a batch of ktrajectories.

ARPL achieves robustness by adding adversarial modelvariation in each rollout. However, training on adver-sarial examples is different from other data augmentationschemes. In supervised models, the data is augmented witha priori transformations such as translation and rotationwhich are expected to occur in the test set. By contrast, ad-

Adversarially Robust Policy Learning (ARPL)

versarial perturbations rely on the online generation of sce-narios that not only expose flaws in the ways that the modelconceptualizes its decision function, but also are likely tooccur naturally.

4. Experimental Results

We evaluated our proposed ARPL algorithm on fourcontinuous control tasks – inverted pendulum,half-cheetah, hopper, and walker-2d using theMuJoCo physics simulator (Todorov et al., 2012). Thesetasks involve complex non-linear dynamics and directtorque control on the actuated joints. Under-actuationand discontinuous contact render these tasks a challengingbenchmark for reinforcement learning methods. To under-stand our model’s robustness with physical plausibility, weuse a low-dimensional state representation that captures thejoint angles and velocities in these tasks. In such cases,perturbations on the state vectors naturally lead to a newstate that is physically realizable in the environments. Theobjective of the inverted pendulum task is to keepa pole upright by applying a force (control) to the baseof the pole. The agent keeps accumulating reward whilethe pole is upright and fails the task if the pole tilts by toomuch. The objective of the half-cheetah, hopper,and walker-2d tasks is to apply torque control to thejoints in order to move right as fast as possible until thebody falls over.

We use the state-of-the-art trust region policy optimiza-tion (TRPO) method (Schulman et al., 2015) to learn astochastic policy using neural networks. The policy isparametrized by a Gaussian distribution, where its mean ispredicted by a network that takes a state as input and con-sists of two hidden layers with 64 units per layer, and tanhas the non-linearity, and the standard deviation is an ad-ditional learned parameter that is independent of the state.For the loss function h that ARPLuses to generate adver-sarial perturbations, we use h(µq ) = kµqk2

2, where µq isthe output of the mean network with parameters q .

We used a curriculum learning approach to train our agentson increasing perturbation frequency (f ). All agents weretrained for 2000 iterations - the large number of iterationswas necessary to guarantee convergence for curriculumlearning. We define the curriculum by uniformly increasingf between 0 and fmax. For process noise perturbations, weset fmax = 0.1, and for dynamics noise perturbations we setfmax = 0.5. We update the curriculum every 200 iterations.Note that we omitted results on observation noise perturba-tions due to space constraints. It is likely that observationnoise will become much more important in the context ofreal world robot experiments.

Improving Model Robustness using Adversarial Train-

ing:

Here we evaluate the effectiveness of our robust trainingmethod proposed in Sec. 3.3. For every agent type, wetrained 15 agents and selected the agent with the best learn-ing curve. This is necessary since our method also tendsto produce agents with poor performance due to the highvariance of the training process. This is something wewould like to address in future work. Nominal agents weretrained with vanilla TRPO, random agents were trainedusing ARPL with random perturbations, and adversarialagents were trained using ARPL with adversarial pertur-bations. These results are for two sets of agents - one thatwere trained on process noise and another that were trainedon dynamics noise.

Figure 2. A comparison of agent performance with respect to ad-versarial process noise on the Inverted Pendulum task. Here,e = 0.01, and we evaluated agent performance as we increasedthe perturbation frequency. The baseline performance indicateshow each agent performs in an unperturbed environment. Noticethat the adversarial agent does the best in the region where is wastrained, but the random agent is more resistant in the higher fre-quency regime.

Fig. 2 and Fig. 3 show the effect of process noise in theInverted Pendulum task. As Fig. 2 demonstrates, the nom-inal agent is highly susceptible to process noise, but boththe random and adversarial agents are more robust. It isinteresting to note that the adversarial agent performs bet-ter in the region where it was trained, but the random agentseems to generalize outside of that region. However, Fig. 3shows that the adversarial agent is incredibly robust to ran-dom process noise, much more so than the random agent.This is a very promising result since random process noiseperturbations are much more likely to be encountered inpractice (for example, on a robot, with noise in sensor mea-surements).

Fig. 4 and Fig. 5 show the effect of dynamics noise in

Adversarially Robust Policy Learning (ARPL)

Figure 3. A comparison of agent performance with respect to ran-dom process noise on the Inverted Pendulum task. Here, e = 0.01,and we evaluated agent performance as we increased the pertur-bation frequency. The baseline performance indicates how eachagent performs in an unperturbed environment. Notice that the ad-versarial agent is robust across all perturbation frequencies whilethe random agent suffers with higher frequency noise.

Figure 4. A comparison of agent performance with respect to ad-versarial dynamics noise in the Inverted Pendulum task. Here,e = 10.0, and we evaluated agent performance as we increasedthe perturbation frequency. The baseline performance indicateshow each agent performs in an unperturbed environment. Noticethat the adversarial agent is robust across all perturbation frequen-cies.

the Inverted Pendulum task. We see that the adversarialagent is robust across both adversarial and random dynam-ics perturbations across all perturbation frequencies, whilethe random agent is significantly more robust than the nom-inal agent.

Fig. 6 and Fig. 7 show the effect of dynamics noise in theWalker task. We see that the adversarial and random agents

Figure 5. A comparison of agent performance with respect torandom dynamics noise in the Inverted Pendulum task. Here,e = 10.0, and we evaluated agent performance as we increasedthe perturbation frequency. The baseline performance indicateshow each agent performs in an unperturbed environment. Noticethat the adversarial agent is robust across all perturbation frequen-cies.

Figure 6. A comparison of agent performance with respect to ad-versarial dynamics noise in the Walker task. Here, e = 10.0, andwe evaluated agent performance as we increased the perturbationfrequency. The baseline performance indicates how each agentperforms in an unperturbed environment. Notice that the adver-sarial and random agents are robust across all perturbation fre-quencies.

are robust across both adversarial and random dynamicsperturbations across all perturbation frequencies.

Fig. 8, Fig. 9, Fig. 10, and Fig. 11 show the agents’ perfor-mance under different combinations of dynamics configu-rations. The center of the grid corresponds to the originalvalues of the dynamics parameters. We can see that nom-inal performance tends to suffer farther from the center of

Adversarially Robust Policy Learning (ARPL)

Figure 7. A comparison of agent performance with respect to ran-dom dynamics noise in the Walker task. Here, e = 10.0, and weevaluated agent performance as we increased the perturbation fre-quency. The baseline performance indicates how each agent per-forms in an unperturbed environment. Notice that the adversarialand random agents are robust across all perturbation frequencies.

the grid, as expected, while both random and adversarialagents are robust to the changed mass and friction values.In general, the adversarial agents tend to obtain higher re-ward, but the random agents are still much more robust thanthe nominal agents. It is not clear whether the use of ad-versarial training with respect to dynamics noise results insubstantial benefits. Additional investigation is necessary.

5. Conclusion

We motivated and presented ARPL, an algorithm for us-ing adversarial example during the training of RL agents tomake them more robust to changes in the environment. Wetrained and evaluated policies on 4 continuous control Mu-JoCo tasks, and showed that agents trained using vanillaTRPO are vulnerable to changes in the environment stateand dynamics. We demonstrated that using ARPL withboth random and adversarial dynamics noise leads to poli-cies that are robust with respect to the environment dynam-ics. We also demonstrated that using ARPL to train withrandom and adversarial process noise leads to agents thatare robust to noise in the environment state.

There are several aspects of this work that we will investi-gate in the future. First, we want to investigate the use ofperturbations with respect to different loss functions overthe policy network. This could include using modified pol-icy gradients with respect to the states and using numericalgradients to estimate the reward gradient with respect tothe state. We also want to investigate the effect of observa-tion noise further, as well as different forms of random pro-cess noise. We want to look into ways to compute dynam-

ics noise perturbations without state augmentation, sincethis information is not typically available in the real world.This might also lead to more differentiation between ran-dom and adversarial dynamics noise agents. Additionally,the agents trained by ARPL demonstrated incredibly highvariance in performance (hence why we compared resultsacross the best agents). We want to investigate methods ofvariance reduction. We would also like to develop a theo-retically sound justification for ARPL. Finally, we want totest an ARPL policy on a robot and investigate the robust-ness of the policy.

6. Team Member Contribution and

Acknowledgments

I originally started this work while working with my men-tors Yuke Zhu and Animesh Garg, but this quarter, I contin-ued to work independently on this project. Their guidanceand feedback was critical for the completion of this work,and I look forward to their continued help and support. Iwould also like to thank my advisors Silvio and Fei-Fei.

References

Abbeel, Pieter, Quigley, Morgan, and Ng, Andrew Y. Us-ing inaccurate models in reinforcement learning. In Pro-ceedings of the 23rd international conference on Ma-chine learning, pp. 1–8. ACM, 2006.

Barreno, Marco, Nelson, Blaine, Sears, Russell, Joseph,Anthony D, and Tygar, J Doug. Can machine learningbe secure? In Proceedings of the 2006 ACM Symposiumon Information, computer and communications security,pp. 16–25. ACM, 2006.

Behzadan, Vahid and Munir, Arslan. Vulnerability of deepreinforcement learning to policy induction attacks.

Biggio, Battista, Corona, Igino, Maiorca, Davide, Nelson,Blaine, Srndic, Nedim, Laskov, Pavel, Giacinto, Giorgio,and Roli, Fabio. Evasion attacks against machine learn-ing at test time. In Joint European Conference on Ma-chine Learning and Knowledge Discovery in Databases,pp. 387–402. Springer, 2013.

Devin, Coline, Gupta, Abhishek, Darrell, Trevor, Abbeel,Pieter, and Levine, Sergey. Learning modular neuralnetwork policies for multi-task and multi-robot transfer.arXiv preprint arXiv:1609.07088, 2016.

Gao, Ji, Wang, Beilun, and Qi, Yanjun. Deepmask: Mask-ing dnn models for robustness against adversarial sam-ples.

Garcıa, Javier and Fernandez, Fernando. A comprehensivesurvey on safe reinforcement learning. Journal of Ma-chine Learning Research, 16(1):1437–1480, 2015.

Adversarially Robust Policy Learning (ARPL)

Figure 8. Policy robustness of the inverted pendulum agents with respect to varying dynamics configurations.

Figure 9. Policy robustness of the half cheetah agents with respect to varying dynamics configurations.

Figure 10. Policy robustness of the hopper agents with respect to varying dynamics configurations.

Figure 11. Policy robustness of the walker agents with respect to varying dynamics configurations.

Goodfellow, Ian J., Shlens, Jonathon, and Szegedy, Chris-tian. Explaining and harnessing adversarial examples.

Green, Michael and Limebeer, David JN. Linear robustcontrol. Courier Corporation, 2012.

Grosse, Kathrin, Manoharan, Praveen, Papernot, Nicolas,

Backes, Michael, and McDaniel, Patrick. On the (statis-tical) detection of adversarial examples.

Huang, Sandy, Papernot, Nicolas, Goodfellow, Ian, Duan,Yan, and Abbeel, Pieter. Adversarial attacks on neuralnetwork policies.

Adversarially Robust Policy Learning (ARPL)

Kober, Jens, Bagnell, J Andrew, and Peters, Jan. Reinforce-ment learning in robotics: A survey. The InternationalJournal of Robotics Research, pp. 0278364913495721,2013.

Kurakin, Alexey, Goodfellow, Ian, and Bengio, Samy. Ad-versarial examples in the physical world.

Levine, Sergey, Finn, Chelsea, Darrell, Trevor, and Abbeel,Pieter. End-to-end Training of Deep Visuomotor Poli-cies. Journal of Machine Learning Research, 17(39):1–40, 2016a.

Levine, Sergey, Pastor, Peter, Krizhevsky, Alex, andQuillen, Deirdre. Learning hand-eye coordination forrobotic grasping with deep learning and large-scale datacollection. arXiv preprint arXiv:1603.02199, 2016b.

Lillicrap, Timothy P, Hunt, Jonathan J, Pritzel, Alexander,Heess, Nicolas, Erez, Tom, Tassa, Yuval, Silver, David,and Wierstra, Daan. Continuous control with deep re-inforcement learning. arXiv preprint arXiv:1509.02971,2015.

Mastin, Andrew and Jaillet, Patrick. Loss bounds for un-certain transition probabilities in markov decision pro-cesses. In Decision and Control (CDC), 2012 IEEE 51stAnnual Conference on, pp. 6708–6715. IEEE, 2012.

Mnih, Volodymyr, Badia, Adri Puigdomnech, Mirza,Mehdi, Graves, Alex, Lillicrap, Timothy P., Harley, Tim,Silver, David, and Kavukcuoglu, Koray. Asynchronousmethods for deep reinforcement learning.

Mnih, Volodymyr, Kavukcuoglu, Koray, Silver, David,Rusu, Andrei A, Veness, Joel, Bellemare, Marc G,Graves, Alex, Riedmiller, Martin, Fidjeland, Andreas K,Ostrovski, Georg, et al. Human-level control throughdeep reinforcement learning. Nature, 518(7540):529–533, 2015.

Moosavi-Dezfooli, Seyed-Mohsen, Fawzi, Alhussein,Fawzi, Omar, and Frossard, Pascal. Universal adversar-ial perturbations.

Moosavi-Dezfooli, Seyed-Mohsen, Fawzi, Alhussein, andFrossard, Pascal. Deepfool: a simple and accuratemethod to fool deep neural networks. In Proceedingsof the IEEE Conference on Computer Vision and PatternRecognition, 2016.

Nilim, Arnab and El Ghaoui, Laurent. Robust control ofmarkov decision processes with uncertain transition ma-trices. Operations Research, 53(5):780–798, 2005.

Papernot, Nicolas, McDaniel, Patrick, Goodfellow, Ian,Jha, Somesh, Celik, Z. Berkay, and Swami, Ananthram.Practical Black-Box Attacks against Deep Learning Sys-tems using Adversarial Examples.

Rajeswaran, Aravind, Ghotra, Sarvjeet, Ravindran, Balara-man, and Levine, Sergey. Epopt: Learning robust neuralnetwork policies using model ensembles.

Rusu, Andrei A, Rabinowitz, Neil C, Desjardins, Guil-laume, Soyer, Hubert, Kirkpatrick, James, Kavukcuoglu,Koray, Pascanu, Razvan, and Hadsell, Raia. Progres-sive neural networks. arXiv preprint arXiv:1606.04671,2016a.

Rusu, Andrei A, Vecerik, Matej, Rothorl, Thomas, Heess,Nicolas, Pascanu, Razvan, and Hadsell, Raia. Sim-to-real robot learning from pixels with progressive nets.arXiv preprint arXiv:1610.04286, 2016b.

Schulman, John, Levine, Sergey, Moritz, Philipp, Jordan,Michael, and Abbeel, Pieter. Trust region policy opti-mization. ICML, 2015.

Silver, David, Huang, Aja, Maddison, Chris J, Guez,Arthur, Sifre, Laurent, Van Den Driessche, George,Schrittwieser, Julian, Antonoglou, Ioannis, Panneershel-vam, Veda, Lanctot, Marc, et al. Mastering the game ofgo with deep neural networks and tree search. Nature,529(7587):484–489, 2016.

Szegedy, Christian, Zaremba, Wojciech, Sutskever, Ilya,Bruna, Joan, Erhan, Dumitru, Goodfellow, Ian, and Fer-gus, Rob. Intriguing properties of neural networks.

Taylor, Matthew E. and Stone, Peter. Transfer learning forreinforcement learning domains: A survey. Journal ofMachine Learning Research, 10:1633–1685, 2009.

Todorov, Emanuel, Erez, Tom, and Tassa, Yuval. Mujoco:A physics engine for model-based control. In Intelli-gent Robots and Systems (IROS), 2012 IEEE/RSJ Inter-national Conference on, pp. 5026–5033. IEEE, 2012.

Yu, Wenhao, Liu, C. Karen, and Turk, Greg. Preparing forthe unknown: Learning a universal policy with onlinesystem identification.

Zhu, Yuke, Mottaghi, Roozbeh, Kolve, Eric, Lim, Joseph J,Gupta, Abhinav, Fei-Fei, Li, and Farhadi, Ali. Target-driven visual navigation in indoor scenes using deep re-inforcement learning. arXiv preprint arXiv:1609.05143,2016.

Adversarially RobustPolicyLearningthroughActiveConstructionofPhysically-PlausiblePerturbations

AjayMandlekar,Yuke Zhu,Animesh Garg,LiFei-Fei,SilvioSavareseDepartmentofComputerScience,StanfordUniversity

Introduction

Demonstrated Robustness in Physical Dynamics Parameters

ARPL Algorithm

Physically-Plausible Threat Model

Experimental Setup

References

ARPL Agent Examples