Active Learning from Demonstration for Robust AutonomousNavigation

David Silver, J. Andrew Bagnell and Anthony StentzCarnegie Mellon University

Abstract— Building robust and reliable autonomous naviga-tion systems that generalize across environments and operatingscenarios remains a core challenge in robotics. Machine learninghas proven a significant aid in this task; in recent years learningfrom demonstration has become especially popular, leadingto improved systems while requiring less expert tuning andinteraction. However, these approaches still place a burdenon the expert, specifically to choose the best demonstrationsto provide. This work proposes two approaches for activelearning from demonstration, in which the learning system re-quests specific demonstrations from the expert. The approachesidentify examples for which expert demonstration is predictedto provide useful information on concepts which are eithernovel or uncertain to the current system. Experimental resultsdemonstrate both improved generalization performance andreduced expert interaction when using these approaches.

I. INTRODUCTION

Building dependable autonomous navigation systems re-mains a difficult challenge to the robotics community. Oneof the major obstacles to developing truly robust and re-liable systems is the necessity to determine proper valuesof the numerous parameters and preferences that residein every system. Manually tuning these settings is tediousand time consuming, and does not necessarily producehigh performing systems. As a result supervised learningbased approaches have gained acceptance in recent yearsfor automating the task of selecting system parameters. Arecent popular paradigm is the Learning from Demonstrationapproach, where parameters and preferences are automati-cally tuned to reproduce expert demonstrated behavior. Thisapproach has proven to both improve the performance ofautonomous navigation systems while reducing the time totheir deployment [1], [2], [3], [4].

In general, such approaches depend on the expert toprovide not only the actual demonstrations used to teach thesystem, but also to choose exactly what to demonstrate. Thisputs additional burden on the expert, and raises the bar forwho can qualify to train the system; the expert must now havesufficient knowledge of the actual system internals to knowwhich examples are most useful. In addition, the expert mustunderstand the structure of the example to fully comprehendhow useful or not it is. Having the expert choose what todemonstrate can also affect the quality of the final trainingset. It has been observed empirically that a good deal ofexpert demonstration proves to be redundant (see SectionV) simply repeating the same basic concepts. While thisdoes help to potentially fine tune such concepts, it can leaveimportant concepts undemonstrated. If the amount of expertinteraction is held constant, such inefficient use of time will

lower the overall effectiveness of the training set.Many of these issues are common to all forms of su-

pervised learning. As a result, the field of Active Learning,where the learner is at least partially responsible for choosingwhich training data to obtain, has received focus in orderto aid in the efficient selection of a good training set.Learning from demonstration could potentially be aided bythe incorporation of active learning techniques that requesteddemonstration in specific scenarios (e.g. how a robot shoulddrive in a specific type of terrain). Such techniques couldreduce the cognitive burden on an expert, as well as increas-ing the pool of potential experts. In addition, active learningcould help to ensure that each additional demonstration wasmaximally useful and not redundant.

This work investigates the application of active learningto learning from demonstration, specifically in the context ofautonomous navigation. The next section describes relatedwork in both fields. Sections III and IV then describetwo basic approaches to active learning in the context ofautonomous navigation. Experimental results are presentedin Section V with conclusions and in Section VI.

II. RELATED WORK

Autonomous navigation is the task of a robot traversingfrom some start location to a goal location. Along the way,it must maintain knowledge of its own position, observe thesurrounding environment, plan actions to achieve its goal,and finally execute these actions. Some of these subtasks,most notably perception and planning, are highly parameter-ized in that they make use of numerous heuristics, thresholds,and other settings that determine the actual behavior of therobot (e.g. the determination of traversability in a perceptionsystem, or specification of navigation rules for a planningsystem). Achieving truly autonomous behavior from a mobilerobot therefore requires not just robust and accurate percep-tion and planning algorithms, but also parameter settingsthat will produce the desired behavior. Towards this end,machine learning approaches are often applied, to both easethe burden on system engineers, as well as ensure a goodset of parameters are chosen. Learning from demonstrationhas gained recent popularity as the expert generated input isexactly the desired behavior, as opposed to a surrogate (e.g.performing supervised classification to identify dangerousterrain).

Learning from demonstration can be applied to robotcontrol and navigation in one of two basic ways. Using amodel free approach, expert demonstration can be used asthe training input to learn a specific policy for the robot to

0 10 20 30 40 50 601

1.5

2

2.5

3

3.5

4

Number of Training Paths

Val

idat

ion

Cos

t Rat

io

(a) Original Order

0 10 20 30 40 50 601

2

3

4

5

6

7

8

9

10

Number of Training Paths

Val

idat

ion

Cos

t Rat

io

(b) Random Subsets

Fig. 1. Validation performance for increasing numbers of example paths,in (a) their original order and (b) random permutations (error bars indicateone standard deviation). Over time, there are clear diminishing returns toadding additional (expert chosen) examples.

follow (that is, a direct mapping from robot percepts andstate to control outputs). For certain problems, this approachis highly effective. However, this approach requires all in-formation relevant to decision making to be encoded in thecurrent robot state. As a result, this approach performs poorlywhen long range decision making is required, or generalizingto previously unseen states. An alternate approach to learningfrom demonstration is model based. This approach usesexpert demonstration to learn a cost or reward function overrobot percepts and state, which can then be applied in novelscenarios to find the optimal policy for that scenario.

This latter technique mirrors one of the most popular ap-proaches to robot navigation: assign varies decisions (actionsand or locations) with some sort of cost, and then use anoptimal motion or path planner to identify the next sequenceof actions. In order to allow for generalization, the cost func-

tion usually takes as input a set of features that describe thecurrent state or have been extracted from the current percepts,rather than these raw inputs themselves. In this approach,the cost function essentially encodes the desired behavior ofthe robot. Therefore the proper parameterization and tuningof the cost function is of extreme importance to correctand reliable robot behavior. If the cost function is learnedfrom demonstration as described above, it can both reducethe amount of human parameter tuning necessary to propersystem function, as well as improving the performanceand generalization of the cost function. This generalizationoccurs for two primary reasons. First, machine learning bestpractices can be applied (e.g. to avoid overfitting) whichprovide certain theoretical guarantees. Second, learning fromdemonstration provides a very rich and varied training inputas it not only specifies what a robot should do, but it alsoimplies what a robot should not do. As a result, learning costor reward functions from demonstration has become popularin recent years [1], [2], [3], [4].

While these approaches have changed the responsibilityof a system expert from that of tedious parameter tuningto demonstrating desired behavior, there remain barriers toefficient deployment. In this paradigm, the demonstrator isresponsible for choosing exactly what to demonstrate. Thisallows for the possibility that potentially useful or importantconcepts could never be taught, or that the provided examplesare redundant. Redundancy in particular is a serious problem,as it has been empirically observed that increasing amountsof demonstration produce rapidly diminishing returns (seeFigure 1). It also raises the bar for who can demonstratebehaviors to the robot: the demonstrator must have at leasta basic intuition of the underlying perception and planningalgorithms in order to decide what exactly to demonstrate.Finally, for larger mobile systems actually demonstratingthe correct behavior can be expensive, especially if largetraining sets are required to properly capture complex desiredbehavior. In such scenarios it may only be possible to providea limited set of demonstrations, which ideally should bechosen to be as efficient as possible.

The field of Active Learning, where the learner is involvedin the selection of training data, has developed in responseto supervised learning in general facing this exact problem.Active learning with respect to certain forms of learning fromdemonstration has received some previous attention. The ap-proaches of [5], [6] each allow a robot to request instructionwhenever it is not sufficiently confident in a learned actionit is about to execute, based on its similarity to previouslyobserved actions. However, these approaches are primarilyconcerned with model free learning, as opposed to moremodel based and generalizable learning from demonstration.In contrast [7] investigates active learning in a model basedcontext; however, it assumes that there is an a priori rewardfunction, and demonstration is used to help discover theoptimal policy.

A common thread of these approaches is their use ofquery filtering: the learner encounters tasks one at a time,and must decide whether to request expert aid for eachtask. This requires the expert demonstrator to be ’on call’

to provide demonstration during robot operation. This modeof operation could potentially be quite useful for autonomousnavigation once an initial training set has been produced(and the robot was mostly functional). However, it wouldprove quite time consuming for the expert if used exclusivelyto train a system from scratch. Alternatively, pool basedapproaches (that request expert instruction over a smallsubset of a large, unlabeled data set) would minimize theinitial expert interaction necessary to train a robust system.Therefore, the techniques presented in the remainder of thiswork are primarily pool based approaches; however theycould also be applied for query filtering in an online settingwhen an expert is available.

III. ACTIVE LEARNING THROUGH NOVELTY REDUCTION

In an ideal setting, a set of training demonstrations pro-vided by an expert would come from the same distributionof examples as that which the robot would encounter whileoperating autonomously. This is quite unrealistic, especiallysince it is rarely the case that the training and operatingenvironments are identical. A more realistic desire would bethat the two distributions at least cover the same areas of theexample space: that is, that for every possible scenario therobot could find itself in, it has been trained on somethingsufficiently similar such that it can properly generalize.Unfortunately, this is also difficult to ensure, as the examplespace is hard to quantify. Instead of the example space, thefeature space (i.e. the input space to the cost function) canbe used as a proxy. Therefore, a potential goal of activelearning could be to ensure that for any type of decision therobot may encounter, it has at least seen something similarin the training set.

In order to understand what the robot has not been exposedto, it is necessary to model what the robot has been exposedto. This problem can be phrased as a form of densityestimation over the feature space of states and actions. Then,examples can be chosen from regions with low density.Such regions are often referred to as anomalous or novel,in that feature vectors that have not been frequently seenduring training represent something novel to the learner.Novelty detection in and of itself is a rich field of research[8], and has seen many applications to mobile robotics [9],[10]. [11], [12] specifically propose density estimation inthe context of active learning for robotic perception systems.Specifically, a large dataset consisting of a robot’s recordedperceptual history is analyzed to identify the most unlikelysingle percepts (given the entire history). These percepts arethen provided to human experts to be labeled for use intraining a terrain classification system. This process is thenrepeated to produce a full training set that properly coversthe entire space of percepts.

This same general approach could also be applied tolearning from demonstration. In this context, all terrain oraction features observed during previous training would beused to build a density model. Then, the entire (unlabeled) setof features observed at any time would be analyzed to detectnovel cases. Expert demonstration could then be requested

for novel structures or scenarios. Such a novelty based activelearning procedure would proceed in 3 phases

1) Initial Training: Perform learning from demonstrationon an existing training set, and build a data set of everyfeature vector encountered during learning.

2) Density Estimation: From the data set, build a densitymodel, such that for any test feature vector, a measureof how similar it is to already seen data can beproduced.

3) Example Selection: Once a density model has beencomputed, example problems are selected such thatthe resulting demonstration is likely to provide usefulinformation about novel areas.

This process would then be iterated to continually add themost novel examples.

An additional step that could be inserted to this processis that of dimensionality reduction before density estimation.Performing density estimation in a high dimensional space iscomputationally expensive, as well as prone to potential over-fitting. Dimensionality reduction can alleviate both of theseproblems, and is quite common as a precursor to densityestimation. Dimensionality reduction is itself another vastarea of research [13], with Principal Components Analysis(PCA) being perhaps the single most common approach.

As for performing density estimation in the (reduced)feature space, there are two main classes of approaches: para-metric and non-parametric. While [11] uses non-parametricdensity estimation, parametric approaches are generallymuch faster at test time, allowing for easier application inan online setting (to allow for query filtering as well as apool approach). Therefore, parametric density estimation ispreferable in this context. There are many different specificalgorithms that could be applied for both dimensionalityreduction and density estimation, each with their own ad-vantages and tradeoffs. In the end, any approach could beapplied that results in the ability to produce some sort ofnovelty measure for various test points.

Example selection is the most unique part of this proposedprocedure. In the context of autonomous navigation, thiswould consist of producing start and goal points such that theresulting example plan is likely to provide new informationabout novel terrain or maneuvers. The challenge is then totry and anticipate the demonstration that is likely to resultfrom a specific test problem. The key insight to solving thisproblem is that there already exists a seed set of expertdemonstration, and a cost function learned from the set thatseeks to reproduce the example behavior. Since the costfunction has been learned to try and imitate the expert, itcan also be used to try and predict their future behavior[14]. Therefore, between any two test waypoints the plannedtrajectory under the current cost function is a reasonableprediction for the expert’s future demonstration. Therefore,the aggregate novelty along the current planned path can beused as a measure of the overall novelty of the exampleproblem. Regardless of the expert’s actual demonstration,something will be learned about this novel example: if theexpert does produce the predicted example, then an exampleof preferring previously novel terrain or actions has been

added to the training set, and if the expert produces adifferent behavior then an example of not preferring novelterrain has been added.

Therefore, the following example selection heuristic isproposed. First, select a large set of random waypoint pairs,and plan behaviors under the current cost hypothesis. Next,evaluate the aggregate novelty along each plan. Finally,choose a random subset of the waypoint pairs to presentto the expert, with the likelihood of a pair’s selection relatedto its aggregate novelty. As the resulting demonstrationsare likely to provide information about previously novelregions of the feature space, they will help to ensure atleast minimum coverage of said space. An additional optionfor this heuristic is to evaluate aggregate novelty under twopossible plans: that under the current cost function, andunder the initial cost hypothesis (usually simply a constantcost over all states and actions). High cost regions of thefeature space are often novel simply because an expert neverdemonstrates behavior that contains them; also consideringthe initial cost hypothesis can produce waypoint pairs thatcontinually force the expert to demonstrate their aversion tosuch areas, thus lowering their novelty.

In addition to providing a useful metric for aiding in offlineactive learning, a novelty function learned through the aboveprocedure would also be useful in online active learning.Such a novelty function would allow for the identification ofterrain features or maneuvers on which the robot has not beensufficiently trained. As demonstrated in [9], [10], such nov-elty functions can be useful in the autonomous operation ofa mobile robot by preventing it from encountering scenariosfor which it is unprepared. Additionally, it could also be usedin a query filtering approach if an expert is available. Thatis, if the robot intended to drive through a low cost but novelsection of terrain, it could first request expert ’permission’ todo so before proceeding (and then learn from the resultingexpert response).

IV. ACTIVE LEARNING THROUGH UNCERTAINTYREDUCTION

As opposed to specifically seeking expert feedback onnovel examples, another approach is to request expert in-formation on examples for which the learner already hasinformation, but remains uncertain. There is of course arelationship between novelty and uncertainty, in that a learneris likely (but not guaranteed) to be uncertain about trulynovel examples. However, novelty is not the only source ofuncertainty; it can also come from similar examples withconflicting labels or demonstrated behavior.

When a learner explicitly models its own uncertainty, anactive learning technique known as Uncertainty Sampling[15] can be applied. However, explicit uncertainty samplingin the context of learning cost or reward functions wouldlimit the form of the functions to Gaussian Processes orBayesian linear regression. Gaussian Processes are not agood choice for learning a cost function when real time costproduction is required, as they are quite expensive to applywhen trained on large datasets. Bayesian linear regression isparametric; however this in turn limits the hypothesis space

Fig. 2. Density estimation for novelty detection in Figure 4. The fullfeature space is first reduced to 3 dimensions through PCA. Next, K-Meansis used to determine cluster centers to approximate the density of the originaldistribution. Active learning results in new examples that reside in previouslyuncovered regions of the space.

of cost functions. Therefore, explicit uncertainty sampling isnot a desirable option in this context.

However, there is another approach that can work withgeneral cost functions. The Query by Bagging approach [12]combines the idea of Query by Committee [16], [17] with theidea of Bagging [18] to measure the uncertainty still inherentin a training set for a particular class of learner. The idea isto train multiple learners on different random subsets (withreplacement) of the available training set, and see wherethey disagree. In the context of cost functions, the analogwould be to learn multiple functions from different subsetsof the available demonstration. Then, the uncertainty of aparticular example would be approximated as the varianceof the different cost functions over said example.

Therefore, an uncertainty based active learning approachcould be implemented that is nearly identical to the noveltybased approach. First, an uncertainty model would be con-structed based on existing expert demonstration. This model

(a) Satellite Imagery Summer 2004 (b) Satellite Imagery Fall 2007 (c) Shaded Relief Elevation Map

Fig. 3. The 9 km2 test site used for these experiments. Quickbird imagery courtesy of Digital Globe, Inc.

would then be used to evaluate plans (under both the currenthypothesis and the initial hypothesis) between random way-points. Waypoint pairs where the demonstration is predictedto traverse high uncertainty areas would then be presentedto the expert as a request for explicit demonstration, and theprocess repeated. Also as before, a final uncertainty modelcould be also used in an online setting for query filtering.As explored in [19] there are yet more additional uses forknowledge of uncertainty in a cost function.

Alternatively, another approach to example selection ispossible. Unlike with novelty based active learning, theuncertainty model provides a metric in the same units as cost.Therefore, the uncertainty model can be used to estimateupper and lower bounds on the ’true’ cost for any test point(the actual observed bounds could also be used). Rather thanchoosing waypoints that are likely to involve demonstrationthrough regions of high uncertainty, waypoints could bechosen such that the predicted plan under the upper boundof the cost hypothesis is significantly different than thepredicted plan under the lower bound hypothesis. Practicallyspeaking, this method of example selection would be morelikely to choose examples such that the cost uncertaintyactually affects the current plan.

V. EXPERIMENTAL RESULTS

In order to evaluate the effectiveness of both noveltyand uncertainty based active learning, experiments wereperformed in the context of learning from demonstration forinterpreting overhead data of an environment [1], [20]. Inthis context, an expert draws paths over satellite imagery ordigital elevation maps that indicate how the expert wouldprefer a robot to navigate said terrain. A cost function(mapping features of the raw data such as color or tex-ture to cost) is then learned over the entire environmentthat both reproduces the demonstrated behavior, as well asgeneralizing the concepts to untrained areas. As oppossedto experiments performed ona live autonomous system, theprimary advantage of experiments in this context is the easeof repeated experiments in a deterministic environment.

All experiments were performed on overhead data for alarge (9 km2) test site. The test site consists of several distinctterrain features such as short and tall grass, bushes, sand,water, different types of trees, and rolling hills. There are alsoman made features such as roads, buildings and areas usedfor controlled burns. The overhead data consists of two setsof satellite imagery and a digital elevation map (Figure 3).The metric used for performance evaluation is the Cost Ratio[20], which indicates how close the current cost function isto considering all expert demonstration optimal. Candidatecost functions are evaluated using the resulting cost ratio ona large validation set of 250 example paths.

Novelty based active learning was implemented as de-scribed in Section III. First, given an existing training set acost function is learned, and every feature vector encounteredduring training is recorded. The original feature space con-sisted of 30 dimensions, an unwieldy number for performingefficient density estimation. Therefore, PCA was used toreduce the data set to 3 dimensions. Once in this space,the K-Means algorithm [21] was used to identify clustersof points and their associated cluster centers. The numberof cluster centers K-Means is tasked with finding was setvery high (30-50 in these experiments) as the goal is notto actually identify distinct clusters, but simply regions ofthe feature space with high point density. For a test point inthe reduced space, the novelty is then defined as the squaredEuclidean distance to the nearest cluster center. Althougha very simplistic density estimation and novelty detectionapproach, it proves more than sufficient for identifyingregions of the feature space that have received little to nocoverage. Extension to more powerful techniques (such asusing the cluster centers to seed Gaussian Mixture Models)is straightforward.

Figures 2 and 4 show a single example iteration of thisprocess. Figure 4(a) shows 5 original example paths. Thesepaths only traverse or avoid roads, tan-colored grass, andtrees. As a result, the costmap learned from this training set(Figure 4(b)) must try to generalize to other terrain features.However, it does a very poor job of generalizing to wherethere was a recent controlled burn of the grass (the dark grey

(a) Original 5 Examples (b) Original Costmap (c) Novelty Map (d) Costmap after 5 new examples

Fig. 4. An example of density based novelty detection and active learning on overhead data.

regions). These areas are given an extremely high cost, whenin fact they are very preferable terrain to traverse through(due to the almost complete lack of vegetation).

After performing density estimation, the novelty map (thenovelty function applied to the features of the entire test site)is shown in Figure 4(c). As would be expected given theinitial training set roads, trees, and grass are low noveltywhile the burned areas, the river, and buildings are highnovelty. 5 new examples were then requested, by pickingrandom waypoints and evaluating the novelty of the currentplan. This results in examples that traverse the burned areas,helping to demonstrate their actual preferability. The costfunction learned as a result of the new training set (5 originalexamples and 5 new examples) is shown in Figure 4(d), witha more reasonable cost estimate over novel regions.

Figure 2 shows the distribution of observed feature vectors(in the PCA reduced space), over both the original andcombined training set. The original training set is confined toone region of the feature space (due both to the homogeneousnature of the original set, as well as the reduced space beingcomputed according to its distribution). The cluster centersidentified via K-Means are able to represent the distributionof this set with orders of magnitude fewer points. In contrast,the examples selected during active learning result in featuresthat occupy completely different regions of the space. Thisdemonstrates the ability of novelty based active learning toensure that new regions of the feature space (correspondingto the variety of observed terrain features) are covered viaexample demonstration.

Uncertainty based active learning was also implementedusing Query by Bagging to produce an uncertainty estimatein the cost function. Figure 5 provides an example of thisapproach in action. A training set of 20 example paths wereused to train the cost function shown in Figure 5(b). Thiscost function underestimates the cost on several types ofundesirable terrain, such as slopes, trees, and water. However,since the original training set did include some examples ofthese terrain features, they are not considered particularlynovel (Figure 5(a)). However, Query by Bagging is able toidentify these areas as regions of higher cost uncertainty((Figure 5(c)). That is, when training on some subsets ofthe full training set, these regions are given a high cost, andon others a lower cost. In contrast, regions that are alwaysgiven a similar cost (such as roads) are given a very lowuncertainty. 10 additional examples chosen via uncertainty

based active learning provide additional demonstration ofthe true preferences over the high uncertainty regions; theresulting cost function (Figure 5(d)) reflects this new infor-mation via increased cost on trees, water, and slopes. Thisnot only demonstrates the effectiveness of uncertainty basedactive learning, but also the way in which it can complementnovelty based active learning.

The novelty and uncertainty functions in Figures 4 and5 also demonstrate that, aside from just being useful foroffline (pool-based) active learning, there is potential foronline query filtering and vehicle safeguarding. What isespecially useful about these functions is that they have beenlearned with respect to the underlying cost function. Thatis, a perception system may be highly confident in its rawsensor data and generation of descriptive features, while thecost function is still uncertain about the desirability of thecombination of such features. This makes such functions anideal complement to novelty or uncertainty functions learnedfor a perception system, and could prove quite useful in theapplication of such techniques [10].

In addition to proof of concept experiments, larger activelearning experiments were also performed to produce quanti-tative results. The basis for comparison is a set of 60 expertproduced training paths. The performance of this trainingset is shown in Figure 1, both in its original order, and withrepeated random permutations. In order to compare activelearning based example selection to expert based exampleselection, 60 example paths were chosen using multipleiterations of novelty based active learning (in order to seedthe novelty function when the training set is empty, a randomsubset of the entire training area is used). An additional40 example paths were chosen using multiple iterations ofuncertainty based active learning, with the first 20 noveltychosen paths as a seed (uncertainty based active learningrequires an initial training set). The result is two (partiallyoverlapping) sets of training examples that can be comparedto the expert chosen set. Figure 6(a) shows the validationperformance of these three training sets when applied in theiroriginal order. The first fact of note is that for small trainingsets, novelty based active learning significantly outperformsexpert chosen examples, demonstrating the ability of activelearning to identify important novel examples. As moreexamples are added to each set, the final performance ofall 3 sets converges, although the active learning approachesstill outperform the expert chosen examples.

(a) Novelty Map (b) Original Costmap (c) Uncertainty Map (d) Costmap after new examples

Fig. 5. An example of uncertainty based active learning, and how it can complement novelty based active learning.

Figures 6(b) and 6(c) show the performance of all 3training sets when learning with multiple subsets in randomorder. Both active learning training sets consistently performbetter than the expert chosen set (significant at the 1%level). Also of note is that the uncertainty active learningset performs better than the novelty based set in a consistentmanner(significant at the 5% level up to 20 examples, and atthe 1% level subsequently). Since these sets share one thirdof their examples, this demonstrates how uncertainty basedactive learning complements novelty based active learning.Also of note is the variance in performance of each trainingset. The two active learning approaches achieve a (statisti-cally significant) lower variance than the expert chosen setat each subset size. However, the uncertainty based approachachieves a lower variance than the novelty based approach;this difference is statistically significant (at the 1% level) forsubsets of size 30 and greater. This concretely demonstratesthe advantage that uncertainty based active learning hasover novelty based active learning: it specifically choosesexamples that are likely to reduce this variance. Finally,it is informative to observe that the active learning basedapproaches achieve better performance with fewer examples;The novelty based set has better performance with 30 ormore examples than the expert chosen set with 50 examples,and the same is true with the uncertainty based set for20 or more examples. Overall, the take away result is thatchoosing a training set via active learning can achieve betterand more consistent performance, with equal or fewer expertdemonstrated examples.

In addition to choosing training sets from scratch, activelearning can also improve the performance of previouslyexisting training sets. To demonstrate this effect, activelearning was used to add additional training data to an expertchosen set. Starting from each different subset of 30 pathsof the expert chosen demonstration, two iterations of activelearning was performed, each adding 10 new examples. Thisprocess was repeated for both novelty and uncertainty basedactive learning.The results are shown in Figure 7, alongwith the effect of adding additional expert chosen examples.As described previously, for expert chosen examples thereare diminishing returns; the difference in performance (in apaired manner) is not even statistically significant between40 and 50 expert chosen examples. In contrast, both activelearning approaches result in validation performance that issignificantly better (at the 1% level) then the corresponding

expert chosen set. This demonstrates that even when effi-ciency is not an issue, active learning can identify importantand useful examples that improve performance.

VI. CONCLUSIONS

This work presents approaches for active learning in thecontext of model based learning from demonstration. Whilelearning from demonstration can improve the performance ofautonomous navigation systems while reducing the requiredexpert interaction, it places significant responsibility on theexpert to choose good examples to provide to the system.Two basic active learning heuristics are presented, one basedon reducing the novelty of potential examples and anotherbased on reducing their uncertainty. Experimental resultsdemonstrate the overall effectiveness of both approaches,showing improved performance with equivalently sized train-ing sets, or equivalent performance with smaller training sets.

Future work will primarily be focused on applying theseapproaches to dynamic settings, where demonstrations in-volving an actual robot are necessary. We will also explorethe potential applicability of these approaches in an onlinecontext for vehicle safeguarding, where a robot can requestremote assistance when faced with a novel or uncertainscenario, and then learn immediately from the feedback.Finally, the use of more advanced dimensionality reductionand density estimation approaches for novelty based activelearning will be explored.

REFERENCES

[1] D. Silver, J. A. Bagnell, and A. Stentz, “Learning from demonstrationfor autonomous navigation in complex unstructured terrain,” Interna-tional Journal of Robotics Research, vol. 29, pp. 1565–1592, October2010.

[2] J. Z. Kolter, P. Abbeel, and A. Y. Ng, “Hierarchical apprenticeshiplearning with application to quadruped locomotion,” in Neural Infor-mation Processing Systems, 2008.

[3] M. Kalakrishnan, J. Buchli, P. Pastor, and S. Schaal, “Learninglocomotion over rough terrain using terrain templates,” in IEEE/RSJInternational Conference on Intelligent Robots and Systems, pp. 167–172, 2009.

[4] M. Zucker, N. Ratliff, M. Stolle, J. Chestnutt, J. A. Bagnell, C. G.Atkeson, and J. Kuffner, “Optimization and learning for rough ter-rain legged locomotion,” International Journal of Robotics Research,vol. 30, pp. 175–191, February 2011.

[5] S. Chernova and M. Veloso, “Confidence-based policy learning fromdemonstration using gaussian mixture models,” in Proceedings ofthe 6th international joint conference on Autonomous agents andmultiagent systems, 2007.

[6] D. Grollman and O. Jenkins, “Dogged learning for robots,” in IEEEInternational Conference on Robotics and Automation, 2007.

0 10 20 30 40 50 601

1.5

2

2.5

3

3.5

4

Number of Training Paths

Val

idat

ion

Cos

t Rat

io

Original ExamplesNovelty Active LearningUncertainty Active Learninglowest possible validation error

(a) Original Order

0 10 20 30 40 50 601

2

3

4

5

6

7

8

9

10

Number of Training Paths

Val

idat

ion

Cos

t Rat

io

Original ExamplesNovelty Active LearningUncertainty Active Learninglowest possible validation error

(b) Random Subsets

0 5 10 15 20 25 30 35 40 45 50

1.4

1.6

1.8

2

2.2

2.4

Number of Training Paths

Val

idat

ion

Cos

t Rat

io

Original ExamplesNovelty Active LearningUncertainty Active Learninglowest possible validation error

(c) Random Subsets (Zoomed)

Fig. 6. Demonstration of how training sets chosen via active learning canperform superior to training sets chosen by an expert

30 40 50

1.3

1.35

1.4

1.45

1.5

1.55

1.6

1.65

1.7

1.75

Number of Training Paths

Val

idat

ion

Cos

t Rat

io

Original ExamplesNovelty Active LearningUncertainty Active Learninglowest possible validation error

Fig. 7. Even when a reasonably sized training set already exists, activelearning can identify useful and important new examples.

[7] A. P. Shon, D. Verma, and R. P. N. Rao, “Active imitation learning,”in AAAI, 2007.

[8] M. Markou and S. Singh, “Novelty detection: A review - part 1:Statistical approaches,” Signal Processing, vol. 83, no. 12, pp. 2481–2497, 2003.

[9] C. Brooks and K. Iagnemma, “Visual detection of novel terrain viatwo-class classification,” in ACM symposium of Applied Computing,pp. 1145–1150, 2009.

[10] B. Sofman, J. A. Bagnell, and A. Stentz, “Anytime online noveltydetection for vehicle safeguarding,” in IEEE International Conferenceon Robotics and Automation, May 2010.

[11] C. Dima, M. Hebert, and A. Stentz, “Enabling learning from largedatasets: Applying active learning to mobile robotics,” in InternationalConference on Robotics and Automation, vol. 1, pp. 108 – 114, April2004.

[12] C. Dima and M. Hebert, “Active learning for outdoor obstacle detec-tion,” in Proceedings of Robotics: Science and Systems, (Cambridge,USA), June 2005.

[13] I. Fodor, “A survey of dimension reduction techniques,” Tech. Rep.UCRL-ID-148494, Lawrence Livermore National Laboratory, May2002.

[14] B. Ziebart, N. Ratliff, G. Gallagher, C. Mertz, K. Peterson, J. A. Bag-nell, M. Hebert, A. Dey, and S. Srinivasa, “Planning-based predictionfor pedestrians,” in IROS, 2009.

[15] D. Lewis and W. Gale, “A sequential algorithm for training text clas-sifiers,” in Proceedings of the International ACM-SIGIR Conferenceon Research and Development in Information Retrieval, 1994.

[16] H. Seung, M. Opper, and H. Sompolinsky, “Query by committee,”in Proceedings of the Fifth Annual ACM Workshop on ComputationLearning Theory, pp. 287–294, 1992.

[17] Y. Freund, H. Seung, E. Shamir, and N. Tishby, “Selective samplingusing the query by committee algorithm,” Machine Learning, vol. 28,pp. 133–168, 1997.

[18] L. Breiman, “Bagging predictors,” Machine Learning, vol. 24,pp. 123–140, 1996.

[19] L. Murphy and P. Newman, “Planning most-likely paths from overheadimagery,” in International Conference on Robotics and Automation,2010.

[20] D. Silver, J. A. Bagnell, and A. Stentz, “High performance outdoornavigation from overhead data using imitation learning,” in Proceed-ings of Robotics Science and Systems, June 2008.

[21] J. MacQueen, “Some methods for classification and analysis ofmultivariate observations,” in Berkeley Symposium on MathematicalStatistics and Probability, pp. 281–297, 1967.