This is the pre-print of an article published in KI - Kunstliche Intelligenz, Vol. 30, Issue 3, 2016, pp. 267-77.The official published version is freely available at Springer Link:http://www.springerlink.com/openurl.asp?genre=article&id=doi:10.1007/s13218-016-0444-x

16 Years of RoboCup Rescue

Raymond Sheh · Soren Schwertfeger · Arnoud Visser

c© This article is distributed under the terms of the Creative Commons Attribution 4.0 LicenseThis license permits unrestricted distribution and reuse, provided that you give appropriate credit to the original authors, the source and the license.

Abstract The RoboCup Rescue competitions have been ini-tiated in 2000. To celebrate 16 years of research and devel-opment in this socially relevant initiative this article givesan overview of the experience gained during these competi-tions. This article provides an overview the state-of-the-artand the lessons learned from the RoboCup Rescue competi-tions.

1 Introduction

The urban search and rescue (USAR) scenario offers a greatpotential to inspire and drive research in multi-agent andmulti-robot systems. In this article we like to introduce theRoboCup Rescue leagues, which are respectively the Res-cue Robot League (RRL) and the Rescue Simulation League(RSL) [1, 2].

Disaster mitigation is an important social issue involv-ing large numbers of heterogeneous agents acting in hos-tile environments. The associated Urban Search and Res-cue (USAR) scenarios have great potential for inspiring anddriving research in both multi-agent and multi-robot sys-tems. Since the circumstances during real USAR missionsare extraordinarily challenging [3], benchmarks based onthem, such as the RoboCup Rescue competitions, are idealfor assessing the capabilities of intelligent robots. Robotsthat can navigate through affected areas after a disaster, mostlikely will also be capable of negotiating the very same en-vironment under normal circumstances. If robots are able to

Raymond ShehCurtin University, Australia

Soren SchwertfegerShanghaiTech University, China

Arnoud VisserUniversiteit van Amsterdam, The Netherlands

recognize humans entombed under piles of rubble of col-lapsed buildings, they will also be able to recognize themwithin their natural environment. The goal of the RoboCupRescue competitions is to compare the performance of dif-ferent algorithms for coordinating and controlling teams ofeither robots or agents performing disaster mitigation. Bytheir nature, the USAR scenarios demand solutions for thecoordination of large and distributed teams of heterogeneousrobots.

In the remainder of this paper first the developments inthe Rescue Robot League (section 2) are highlighted, fol-lowed by the development in the Rescue Simulation League(section 3).

2 Rescue Robot

The RoboCup Rescue Robot League (RRL) is a commu-nity of teams that make use of competitions, rescue campsand summer schools to advance the state of response robo-tics. Through the rescue competition the league is encourag-ing teams to work on robotic systems for USAR scenariosand providing opportunities to compare their solutions withother teams and get feedback from other experts and end-users. Figure 1 shows the teams, robots and administratorsthat participated in the latest RoboCup Rescue InternationalChampionship, held in 2015 in Hefei, China.

2.1 Competition Structure

The RoboCup Rescue Robot League is a points scoring com-petition. In general, this consists of a search task within anarena consisting of the standard test method apparatuses.The teams aim to reach and survey simulated victims withinthis arena and, in the process, overcome the various test

2 Raymond Sheh et al.

Fig. 1 Robots, Teams and Administrators at the RoboCup Rescue2015 World Championships in Hefei, China. These teams representthe best from regional opens around the world.

methods. Over time, additional tasks such as autonomy, map-ping and manipulation tasks have been added to the scoringmetric to reflect improving capabilities and new focus areas.

Such a structure has several benefits. It helps to fos-ter the spirit of collaboration, where all teams work on thecommon goal of the league, by specifically avoiding placingteams in an adversarial position. It contributes to fairness,since teams have the same challenges at their disposal. Theuse of standard test methods allows teams to analyze theirsuccesses and failures in a scientifically rigorous and repro-ducible manner, helping them to further improve their ca-pabilities. It also allows for direct comparisons between theperformance of robots within the competition and those inother settings.

One particular feature of the RoboCup Rescue RobotLeague is that it encourages the participation of not onlythe teams that are able to do well overall across the vari-ous tasks, but also teams that specialize in particular chal-lenges of the application. These include advanced mobility,autonomous operations, mapping, manipulation and confinedspace operations. The use of distinct test methods to buildthe arena, combined with the ability of teams to choose theirpath, allows teams to focus their missions on their areas ofstrength. For example, teams with advanced mobility capa-bilities can spend their efforts on points in the arena that aredominated by mobility challenges. In contrast, teams thatfocus on manipulation can stick to areas of the arena withless challenging terrain but where collecting points requiresdexterous robot arms to observe and manipulate objects.

Beyond the ability to demonstrate these specialized ca-pabilities, the League recognizes that teams with superiorcapabilities in these niche areas do not necessarily have theability to produce an entry that will perform well in the over-all competition. To encourage these teams to compete andshare their developments, the League tracks not only overallperformance but also points earned in these specific capa-

bilities. These points are combined with additional tests toevaluate these capabilities in isolation to determine the win-ners of the Best-in-Class awards. Past winners for the Best-in-Class awards appear in the RoboCup Rescue wiki1.

Other features of the competition structure include thesplitting of the competition into preliminary and final rounds.All teams are guaranteed a set number of opportunities torun in the preliminaries, allowing all teams to demonstratetheir capabilities in front of an audience of their peers. Thenumber of teams that progress to the finals depends on thescore distribution. The worst qualified team should be clearlybetter than the best team that failed to qualify. Teams thatspecialize tend to fail to qualify as their more specializedfocus places them at a disadvantage. Therefore, the Leagueencourages teams that did qualify, which tend to be moregeneral in nature, to incorporate a team that did not qualifybut who demonstrated superior performance in an area thatthey lack. The combined team progresses as one and anyawards are given to both teams. This mechanism promotescollaboration between teams, helping to disseminate Best-in-Class capabilities throughout the League. Salient exam-ples include mapping algorithms such as HectorSLAM [4].

The standard test methods, as defined by the DHS2-NIST3-ASTM4 International Standard Test Methods for Re-sponse Robots 5 [5], are used inside the RRL to balancesthe need to provide abstract, safe tasks that are conduciveto driving academic research, with operational relevance toensure that implementations that do well in the competitionalso represent capabilities that solve real-world challenges.It distills the real world, operational requirements of first re-sponders into elemental tasks. These tasks are a commonlanguage, which make it possible to create a benchmark forinnovation [6]. Through this language, the challenges of thefield are communicated to researchers, in a manner that isclear, easy to reproduce and where all robots can exhibitsome level of performance and yet few, if any robots, cansaturate. Similarly, the space of capabilities that exist in theresearch community can be communicated, via their perfor-mance in known tests, to first responders, robot manufactur-ers and government agencies.

2.2 The League and Community

The League extends its efforts to advance the state of re-sponse robotics beyond the aforementioned competition.Rescue camps and summer schools [7, 8] disseminate theBest-in-Class capabilities and implementations both within

1 wiki.robocup.org/wiki/Robot_League2 US Department of Homeland Security3 US National Institute of Standards and Technology4 Formerly the American Society for Testing and Materials5 ASTM International Committee on Homeland Security Applica-

tions; Operational Equipment; Robots (E54.08.01)

16 Years of RoboCup Rescue 3

and beyond the League. Participation as organizers and ascompetitors in other competitions ensure that the experienceinside the League spreads more widely.

Besides the world-championship (see Fig. 1 for a groupphoto) regional competitions are held in several countries,using the same scenarios and rules. Typically only the bestteams from the regional opens qualify for the main compe-titions. Thus the community spans over many more teamsthen the 20 to 30 teams in the world-championship. Big re-gional competitions that are open to teams from all regionsare regularly held in Germany, Iran, Japan, Thailand andChina.

Week-long teaching camps and summer schools, focusedon research level undergraduate students, PhD students andearly career researchers, have been hosted by the Leaguecommunity several times since 2004. The first Rescue Robo-tics Camp was held Italy [9] and was instrumental in notonly bringing together and disseminating the Best-in-Classcapabilities from the previous year but also to connect theLeague community more closely with the first respondercommunity.

This theme continued with subsequent events in Thai-land, Austria, Turkey and Australia. The 2012 Safety, Secu-rity and Rescue Robotics Summer School, held in Alanya,Turkey [10], was unique in that selected senior and retiredresponders from police bomb squads and fire and rescue ser-vices were embedded directly into the different groups forthe entire week. This allowed the students to gain a deeperappreciation for the challenges faced by responders in thefield. In addition it also allowed the first responders, typi-cally in management and advisory panels of their services,to better understand the current and future state of the art.

2.3 Lowing the barrier to enter the league

A major challenge is that robots for use in this field musthave a combination of mobility, sensing, communications,intelligence, user interface design and software engineer-ing. From its earliest years, the League has been seekinga standard robot platform or kit to lower the barrier for en-try to research in this field, especially for computer sciencebased teams who may lack the requisite mechanical engi-neering expertise to integrate a reliable, high mobility plat-form. Closing the loop for the first time, on a working robot,is often the greatest challenge. In the past, such teams haveresorted to inflexible and proprietary kit robots or toys thatlack durability and performance.

With the advent of 3D printing and low-cost smart ser-vos, highly capable embedded computers and other such re-sources, starting in 2014 the League has started the OpenAcademic Robot Kit6 [11]. This is an initiative to develop

6 www.oarkit.org

a family of robot designs where all mechanical parts are3D printable, All other parts are readily available off theshelf and all designs, instructions and source code are avail-able in easily editable form, online, under an open sourcelicense. Furthermore, the parts are, where possible, drawnfrom a common set of parts to maximize potential re-use.The aim is to generate a set of online resources that any-one can follow to create a working robot that they can thenimprove within their area of expertise. To complement thesmaller robots that tend to be constructed using 3D print-ing, the League is also launching the Rapidly ManufacturedRobot League, a competition designed for robots in confinedspaces, as described in Section 2.4.3. The first two robotsfrom this initiative are shown in Figure 2. These initial de-signs have also focused on being low in cost. At approxi-mately $500 USD, they are comparable to many moderatelyadvanced robotics construction kits and yet they are alreadycomplete with onboard cameras, computation via a Rasp-berry Pi embedded computer, and a user interface that canbe controlled from an Android device.

Fig. 2 The “Excessively Complex Six-Wheeled Robot” (top) and the“Emu Mini 2” (bottom), the first two robots from the Open AcademicRobot Kit.

Teams around the world, all working on similar opensource robots, can contribute improvements to a common

4 Raymond Sheh et al.

pool and thus form ad-hoc collaborations regardless of theirlocation or their stage of education. For example, high schoolstudents in Thailand might generate new wheel designs whilegraduate students in Germany could design vision algorithmsfor recognizing impassable terrain. A team from a maker-space in Australia might then design a new gripper whilean undergraduate team from the United States could build anew user interface. All of these improvements can be sharedand these groups connected via the kit, long before they maymeet at a competition or teaching event.

2.4 Additional Test Elements

2.4.1 Aerial Robots

Aerial vehicles are of tremendous use in response robot sce-narios and are widely used already today. But their applica-tion is mostly on wide-area surveying, mapping and search.But Unmanned Aerial Vehicles (UAV) also have a great po-tential for search and inspection close to or inside of build-ings. The RRL recognizes the opportunities and challengesof this use of UAVs. A number of standard test methods foraerial vehicles are installed in the aerial arena which is partof the overall arena. Safety features (e.g. low battery andcommunication loss behaviors) as well as specialized capa-bilities (e.g. building access through windows, station keep-ing) are tested for the Best in Class MicroAerial Robot.

2.4.2 Outdoor Robots

In 2016 for the first time an outdoor competition is organizedwhich is affiliated to the Robot Rescue League. In this Car-ryBot league cheap and simple, but yet capable, autonomousrobots for basic logistic purposes are tested. The goal is tosupport the response personnel by transporting material oreven victims over moderately difficult terrain along a pathpredefined with GPS coordinates.

2.4.3 Confined Space Robots

The scale of robots that compete in the RoboCup RescueRobot League are designed to enter spaces with a nominalclearance of 1.2 m (4 ft). The robots that enter these arenasare very capable; however they also tend to be very expen-sive and complex. Furthermore, there is a demand for robotsthat can operate in significantly smaller spaces, as are foundin collapsed buildings and other industrial, civil and domes-tic environments. To encourage the development of smallerrobots, and to allow cheaper robots such as those of the OpenAcademic Robot Kit to compete on their own terms, since2014 the League has developed a bracket of the competitionfor smaller robots. Named the Rapidly Manufactured RobotLeague (also referred to as the Mini Arena and formerly the

Fig. 3 The smaller scale Rapidly Manufactured Robot League arena,based on a 30 cm (1 ft) nominal clearance.

Confined Space Challenge), this arena is based on a 30 cm(1 ft) nominal clearance. This arena is shown in Figure 3.

Reducing the size of the arena and thus also reducing thecost of the robots required also allows the League to reachacross to the RoboCup Junior Rescue community. The ex-isting RoboCup Junior Rescue arenas are already based ona maze at a scale of 30 cm (1 ft). The Rapidly Manufac-tured Robot League provides a bridge competition that al-lows high school students to tackle research level problemsin mechanics, electronics, computer science, and user inter-faces, at a cost and level of required infrastructure that issimilar to their existing competitions.

2.5 Technological Developments and Lessons Learned

Recent years have seen several improvements in the technol-ogy employed by robotic rescue systems. Those improve-ments are then often met with more challenging tests in theRoboCup Rescue competition.

A development that can be easily overlooked is the grad-ually increasing difficulty in the terrain that the autonomousrobots have to face. Over the years we went from mostlyflat terrain to crossing ramps. We also introduced shortcutsinto the more difficult orange arena that require robots withsimple locomotion to detect that the terrain is impassible forthem. In 2015 obstacles that require 3D terrain classifica-tion were introduced as well as curtains made of light fabricthat require advanced sensing and reasoning skills from theautonomous robot.

Also the manipulation capabilities of the robots have im-proved. As a consequence we now have multiple doors inthe arena that can be opened in push as well as pull direc-tion. Further improvements are that two-way audio commu-nication is now required. On the operation side we are nowincluding the setup time of the operator station in the runtime and also restrict the size of the operator station. Thusthe teams are pushed to more ergonomic and easy to usehuman-robot interfaces.

16 Years of RoboCup Rescue 5

Another aspect is the adjudication of the league in itselfposes interesting scientific questions. The development ofthe standard test methods is one such area. Mapping and theevaluation of the generated maps is another research areathat is important for the league. The Fiducial method for2D grid map evaluation [12] has recently been extended to3D maps, using data from the RoboCup Rescue competition[13].

2.6 Influence outside the league

Members from the League community were extensively in-volved in the DARPA Robotics Challenge (DRC) Trials andFinals [14]. The challenges seen in the DRC Trials were de-veloped by members of the League organizing committeewhile one of the teams that qualified for the finals, Team Vi-GIR [15], consisted of many members of Team Hector, oneof the most successful teams in the League. Similar princi-ples were used to develop the challenges in the DRC Trialsas the test methods and the RoboCupRescue Robot League.These focus on test apparatuses that are easy to build, yieldstatistically significant results, exercise operationally rele-vant capabilities and that are easy for suitable robots to at-tempt and yet can challenge even the most capable robots.Early concepts for apparatuses such as valves and terrainswere evaluated in the RoboCupRescue Robot League priorto their final appearance at the DARPA Robotics Challenge.

In 2016, RoboCupRescue will welcome humanoid robotswith challenges that are an evolution of those that appearedin the DRC Finals. This crossover aims to showcase ad-vances in human technologies in disaster scenarios, providean evolving benchmark for disaster relief that requires moredexterity than standard wheeled robots, recruit new teamsand leverage the investment that the research communityhas made in the DRC efforts. The initial set of tasks for thisdemonstration would focus on using human tools in humanenvironments – analogous to the @Home league. The taskenvironment, however, would replicate aspects of the Res-cue league.

3 Rescue Simulation

The RoboCup Rescue Simulation League (RSL) aims to de-velop simulators that form the infrastructure of the simula-tion system and emulate realistic phenomena predominant indisasters and it aims to develop intelligent agents and robotsthat are given the capabilities of the main actors in a disasterresponse scenario.

The RoboCup Simulation League has two major com-petitions which will be described in the subsequent sections.The two competitions share the Infrastructure competition,which is intended to stimulate the further development of

the league with new challenges. Champions of the leagueare recognized at the League’s wiki7 and get the chance topublish their contribution [16, 17] in the Springer LectureNotes series.

A prequel of the DARPA Robotics Challenge Field Tri-als was the Virtual Robotics Challenge [18], with nearly 100teams participating. This humanoid challenge was based ona dedicated version of the Gazebo simulator, which in 2016also has become the basis of the RoboCup Rescue VirtualRobot competition [19].

3.1 Virtual Robot Competition

RoboCup Virtual Robot competitions are being held since2006 [20]. The intention of the competition was to create abridge between the RRL and RSL [21]. The competition at-tracts mainly academic teams from universities, some evenwith teams competing in both the RoboCup Rescue Robotand Simulation League. In 2016 the competition reachedacross and attracted high school teams with prior experiencein the RoboCup Junior Rescue community; performing pre-cisely the bridging function intended for the Rapidly Manu-factured Robot League.

The main challenge for the teams is the control of a largeteam of robots (typically eight) by a single operator. This isstill state-of-the art; the only real comparison is the cham-pion of the Magic competition [22], where 14 robots werecontrolled by two operators. In simulation it was demon-strated that a single operator is able to control a maximumof 24 robots [23].

The single operator has to use high-level commands (suchas the areas to be searched, routes to be followed, etc.) tobe able to control such large teams [24]. The operator’s at-tention is mostly needed to verify observations whether ornot one of the robots has detected a victim (based on colorand/or shape). Due to poor lighting and the number of oc-clusions, the conditions are generally not favorable for auto-matic victim detection, and manual conformation is alwaysneeded. The approach to a victim is quite critical (the robotshould come within the communication range (< 1m)), butis not allowed to touch the body or any of the limbs). Thismeans that the workload for the operator is quite high, pro-viding an advantage for the teams which are able to auto-mate the decision making within the robot team as far aspossible, and only involve the operator when needed.

The shared map generated by the robots during the com-petition has a central role in the coordination of such largerobot teams. The shared map is where the distributed sen-sor information is collected and registered, by each robotindependently. The information has to be sent via often un-reliable communication links [25], so the robot has selected

7 wiki.robocup.org/wiki/Rescue_Simulation_League

6 Raymond Sheh et al.

which information is to be broadcasted (the robots have aneed to know what could be of interest for its teammatesand the operator). The registration process is asynchronous;some information may arrive at the base-station even min-utes after the actual observation [26]. There is no guaran-tee that the operator has time to look at this informationdirectly, which implies that the map within the user inter-face has to be interactive and should allow the operator tocall back observations that were made at any point of in-terest (independent of when the observation was made andby which robot). At the same time the registration processshould keep the map clean (no false positives or wrong as-sociations), because it is the area where the coordination ofthe team behaviors is done.

Since the beginning of the competition [20], a numberof challenging disaster environments have been created. Al-ready at the RoboCup 2006 a quite large world was used,which had a street scenario, an office scenario and a hedgemaze in the garden. In later competitions a large disasterarea with a railway station at a waterfront was used. Theseenvironments were based on the Unreal Engine 2 (UT2004).

Fig. 4 Impressions of the Virtual Robot Competition in 2006 & 2008.

With the introduction of the Unreal Engine 3 (UDK)even larger and more detailed environments could be cre-ated. For instance, in the 2012 competition a world with verydynamic lighting with moving shadows was introduced. In2014 the outdoor worlds were already so large that onlyteams of combined air- and ground-robots could explore thedisaster site.

Fig. 5 Impressions of the Virtual Robot Competition in 2012 & 2014.

To be able to control explore these large environmentsnot only improvements of the user interface for the oper-ator were needed, but the teams also increased the auton-omy of the robots. For instance, several well-known meth-

ods [27, 28] were applied to be able to automatically recog-nize victims [29, 30].

To make pure teleoperation of robots based on visualfeedback more difficult, indoor environments were often filledwith smoke (which is realistic in disaster scenarios [31]). Tocounter this situation, the teams used methods [32] to in-crease the contrast in smoky and dark circumstances, whichis also valuable for first responders.

Many publications related to this competition were pub-lished, some with quite high impact [21, 33, 34]. The sub-jects were as diverse as walking robots, design of test arenasand mapping algorithms.

3.2 Agent Competition

The goal of the RoboCup Rescue Agent competition is tocompare the performance of different algorithms for coor-dinating and controlling a team of physical agents perform-ing disaster mitigation in a simulated city [35]. The goal ofteams participating in the competition is to provide a soft-ware system that reacts to a simulated disaster situation bycoordinating a group of agents. This goal leads to challengessuch as the exploration of large-scale environments in orderto localize fire-fronts and victims, as well as the schedulingof time-critical rescue missions. Agents have only a limitedamount of communication bandwidth they can use to coordi-nate with each other [36]. The problem cannot be addressedby a single entity, but has to be solved by a multi-agentsystem. Moreover, the simulated environment is highly dy-namic and only partially observable by a single agent. Agentshave to plan and decide their actions asynchronously in real-time.

Fig. 6 A simulation of the city of Kobe burning.

The agent competition consists of a simulation platformwhich resembles a city after an earthquake. Such a simula-tion of the city of Kobe is depicted by Figure 6. Into this

16 Years of RoboCup Rescue 7

environment, intelligent agents can be spawned for mitigat-ing the effects of simulated disaster events, such as floodand fire. For this purpose, agents may take on heterogeneousroles such as police force, fire brigade, and ambulance team,that all have different capabilities.

Several overview articles are written on the coordinationand task allocation research performed with the RoboCupRescue Agent simulator [37, 38]. As indicated by Ferreiraet al. [39], generic algorithms tend to be outperformed themethods applied by the winners of the RoboCup RescueAgent competition [40], which use various heuristics basedon a-priori knowledge on the domain.

Inspired by the influential paper by Murphy et al. [41]on physical rescue agents, several researchers have appliedtheir knowledge in real disaster situations [42, 43, 44]. Mostimportant, as implemented as task for the ambulance agentsin the Rescue Simulation Agent competition, is to reduce theamount of time a victim is entrapped.

Within the last years, there were several techniques formulti-agent strategy planning and team coordination intro-duced, such as decentralized communicating POMDPs [45],distributed constraint optimization [46], auction based meth-ods [47] and evolutionary learning [48, 49]]. Recently, thiswas extended with work on weighted synergy graphs [50],Tractable Higher Order Potentials constraints [51] and fluidteam allocations [52]. Furthermore, there has been substan-tial work on building information infrastructure and deci-sion support systems for enabling incident commanders toefficiently coordinate rescue teams in the field [53].

4 New Challenges

4.1 Rescue Simulation League

In 2013 the simulation league has initiated RMasBench, anew type of challenge having the goal to focus on the strate-gic decisions instead of the tactical decisions [54]. The ideais to extract from the entire problem addressed by the agentscertain aspects such as task allocation, team formation, androute planning, and to present these sub problems in an iso-lated manner as stand-alone problem scenarios with an ab-stract interface. As a consequence, participating teams aremore free to focus on their research without having to dealwith low-level issues. RMasBench introduced a generic APIfor distributed constraint optimization problem (DCOP) al-gorithms, including a library implementing state-of-the-artDCOP solvers, such as DSA and MaxSum as reference solu-tions. In 2016 the challenge is rephrased as Technical Chal-lenge, which the same intention to abstract away from thelow-level tactical decisions, but this time facilitated by anAgent Develop Framework [55].

The future of the Virtual Robot competition was rede-fined at the Future of Rescue Simulation workshop. One of

Fig. 7 Team Hector’s Centaur at the JVRC’s ’clear the road’ task.

the goals of the workshop was to define a roadmap for thedevelopment of the league for the coming years. In addi-tion a connection to the DARPA Virtual Robot Challengeand the Japanese Virtual Robot Challenge (JVRC) [56] wasmade. At the JVRC a centaur design was very successful;a tracked robot with a humanoid torso called MIDJAXON.The centaur design could also be a good combination of mo-bility and manipulation capabilities in the RoboCup VirtualRobot competition, as demonstrated at the workshop8 (SeeFig. 7).

As a result of the workshop the challenge of the VirtualRobot Competition is refined, which is reflected in a newrules document [57]. In this new rules document a clear tran-sition is made from the current Unreal/ROS based environ-ment [58] towards a ROS/Gazebo based environment[59].

4.2 Robot Rescue League

Starting in 2016 the RRL is adopting a new scheme for thecompetition. In the preliminary rounds in the first three daysbasic and specific capabilities of the robots are measuredin DHS-NIST-ASTM International Standard response robottest apparatuses, for which the testing procedures have beencustomized to the specific needs of the RoboCup Real Res-cue League. The test method apparatuses will be arrangedinto lanes and teams will be invited to run their robots mul-tiple times across the lanes. By running these tests in paral-lel, rigorous measurements of capabilities can be obtainedin isolation. This results in statistically significant testingin four areas: 1) basic sensing and MANeuvering capabil-ities, 2) advanced MOBility, 3) manipulation and inspec-tion DEXterity and 4) EXPloration, mapping and autonomy.Each of those four areas consists of five tests, which oftencorrespond to one of the standard ASTM test methods. Fig-ure 8 shows an overview of how the tests are laid out in thearena.

In the preliminaries the Best-in-Class winners in the ar-eas of Mobility, Dexterity and Exploration will be deter-mined as well as the overall best teams that will progressto the finals. For each area three Best-in-Class certificates

8 https://github.com/tu-darmstadt-ros-pkg/centaur_robot_tutorial/wiki

8 Raymond Sheh et al.

Fig. 8 Real Rescue arena plan for 2016.

are awarded: Best-in-Class Small Robot (for robots enteringthe tests through a 60cm square), Best-in-Class AutonomousRobots (for robots performing without operator interven-tion) and the general Best-in-Class certificates open to allteams.

In the finals the test elements will be combined such thattwo big arenas are formed. The finalists will then search inthere for simulated victims by traversing the various test el-ements within a single run.

Running the competition with this new scheme enablesus to conduct challenging and fair competitions that empha-size tasks that are of actual value for USAR applications.The RRL is now more closely resembling Response RobotExercises [60], which have been effective in communicatingand demonstrating functionality, reliability, operator profi-ciency, and autonomous/assistive capabilities of the systemsbetween robot manufacturers and responders.

As part of the emphasis on the dissemination and col-laborative development of technologies for response robo-tics, from 2016 on the Team Description Papers (TDP) havean updated template covering more technical aspects of therobotic solutions. The goal of this update is to better allowteams to express and share the novel aspects of their entries.The TDPs of all participating teams will be published on-line9 and thus be accessible to the general public.

The new rules have been tested and implemented at theRRL meeting in March 2016 in Koblenz, Germany and atthe Iran Open in April 2016. More details about the new waythe league is run can be found in the rules document [61].

5 Conclusions

In the last 16 years the RoboCup Rescue community hasproven that work on this grand challenge [2] is fruitful. Teams

9 http://www.robocuprescue.org/

from all over the world are now working on this sociallyrelevant application, evolving their initial hardware designsto very versatile robots. Also the perception, planning andcontrol of the robots have been substantial improved, whichmakes it possible to autonomously navigate through the dis-aster area and find victims in difficult circumstances. Therescue robots are no longer alone, but operate in heteroge-neous teams combining robots with different capabilities.Coordination inside the team of robots, so that they effi-ciently work on a joint goal, is extensively studied in sim-ulation and demonstrated with real robots.

One of the remaining challenges for the coming years isthe manipulation capabilities. The DARPA Robotics Chal-lenge has proven that humanoid robots could make use ofavailable tools (cars, drills, valves) in their rescue missions,but it is also clear that there could be a lot improved in ma-nipulation capabilities. Yet, in the future the rescue robotsshould not only be able to find the victims but also capableto free them carefully from their perilous situation.

References

1. S. Tadokoro, H. Kitano, T. Takahashi, I. Noda, H. Mat-subara, A. Shinjou, T. Koto, I. Takeuchi, H. Takahashi,F. Matsuno, M. Hatayama, J. Nobe, and S. Shimada.The RoboCup-Rescue project: A robotic approach tothe disaster mitigation problem. In Proc. of the IEEEInt. Conf. on Robotics and Automation (ICRA), 2000.

2. H. Kitano and S. Tadokoro. A grand challenge for mul-tiagent and intelligent systems. AI Magazine, 22:39–52,2001.

3. Robin R. Murphy, Satoshi Tadokoro, Daniele Nardi,Adam Jacoff, Paolo Fiorini, Howie Choset, and Ay-dan M. Erkmen. Springer Handbook of Robotics, chap-ter Search and Rescue Robotics, pages 1151–1173.Springer Berlin Heidelberg, Berlin, Heidelberg, 2008.ISBN 978-3-540-30301-5.

4. Stefan Kohlbrecher, Johannes Meyer, Thorsten Graber,Karen Petersen, Uwe Klingauf, and Oskar Stryk.RoboCup 2013: Robot World Cup XVII, chapter Hec-tor Open Source Modules for Autonomous Mappingand Navigation with Rescue Robots, pages 624–631.Springer Berlin Heidelberg, Berlin, Heidelberg, 2014.ISBN 978-3-662-44468-9.

5. Adam Jacoff, Raymond Sheh, Ann-Marie Virts, Tet-suya Kimura, Johannes Pellenz, Soren Schwertfeger,and Jackrit Suthakorn. Using competitions to advancethe development of standard test methods for responserobots. In Proceedings of the Workshop on PerformanceMetrics for Intelligent Systems, pages 182–189. ACM,2012.

6. F. Amigoni, E. Bastianelli, J. Berghofer, A. Bonarini,G. Fontana, N. Hochgeschwender, L. Iocchi, G. Kraet-

16 Years of RoboCup Rescue 9

zschmar, P. Lima, M. Matteucci, P. Miraldo, D. Nardi,and V. Schiaffonati. Competitions for benchmarking:Task and functionality scoring complete performanceassessment. IEEE Robotics Automation Magazine, 22(3):53–61, Sept 2015. ISSN 1070-9932.

7. Raymond Sheh and H Komsuoglu. The 2012 ieee robo-tics & automation society safety, security, and rescuerobotics summer school: An event for the dissemina-tion of the challenges and best-in-class capabilities inthe ssrr community [society news]. IEEE Robotics &Automation Magazine, 19(4):92–95, Dec 2012. ISSN1070-9932.

8. Raymond Sheh, Bill Collidge, Mihai Lazarescu, Hal-dun Komsuoglu, and Adam Jacoff. The response robo-tics summer school 2013: bringing responders and re-searchers together to advance response robotics. InIntelligent Robots and Systems (IROS 2014), 2014IEEE/RSJ International Conference on, pages 1862–1867. IEEE, 2014.

9. Jean Scholtz. Robot rescue camp. interactions, 12(2):79–80, 2005.

10. Raymond Sheh, Haldun Komsuoglu, Adam Jacoff, Tet-suya Kimura, Daniele Nardi, Johannes Pellenz, andGerald Steinbauer. The 2012 safety, security, and rescuerobotics summer school. In IEEE International Sympo-sium on Safety, Security, and Rescue Robotics (SSRR),2012, pages 1–2, 2012.

11. Raymond Sheh, Haldun Komsuoglu, and Adam Jacoff.The Open Academic Robot Kit: Lowering the barrierof entry for research into response robotics. In Pro-ceedings of the 2014 IEEE International Symposium onSafety, Security, and Rescue Robotics (SSSR), pages 1–6, Oct 2014.

12. Soren Schwertfeger, Adam Jacoff, Johannes Pellenz,and Andreas Birk. Using a fiducial map metric for as-sessing map quality in the context of robocup rescue. InInternational Workshop on Safety, Security, and RescueRobotics (SSRR), pages 208–214, Nov 2011.

13. Soren Schwertfeger and Andreas Birk. Using fiducialsin 3d map evaluation. In IEEE International Symposiumon Safety, Security, Rescue Robotics (SSRR), pages 1–7,Oct 2015.

14. E. Guizzo and E. Ackerman. The hard lessons ofdarpa’s robotics challenge [news]. IEEE Spectrum, 52(8):11–13, August 2015. ISSN 0018-9235.

15. Stefan Kohlbrecher, Alberto Romay, Alexander Stumpf,Anant Gupta, Oskar von Stryk, Felipe Bacim, Doug A.Bowman, Alex Goins, Ravi Balasubramanian, andDavid C. Conner. Human-robot teaming for rescue mis-sions: Team vigir’s approach to the 2013 darpa robo-tics challenge trials. Journal of Field Robotics, pages352–377, Jan-12-2014 2014.

16. Francesco Amigoni, Arnoud Visser, and MasatoshiTsushima. RoboCup 2012: Robot Soccer World CupXVI, chapter RoboCup 2012 Rescue Simulation LeagueWinners, pages 20–35. Springer Berlin Heidelberg,Berlin, Heidelberg, 2013. ISBN 978-3-642-39250-4.

17. Victor Spirin, Julian de Hoog, Arnoud Visser, andStephen Cameron. RoboCup 2014: Robot World CupXVIII, chapter MRESim, a Multi-robot ExplorationSimulator for the Rescue Simulation League, pages106–117. Cham, 2015. ISBN 978-3-319-18615-3.

18. Carlos E Aguero, Nate Koenig, Ian Chen, Hugo Boyer,Steven Peters, John Hsu, Brian Gerkey, Steffi Paepcke,Jose L Rivero, Justin Manzo, et al. Inside the virtualrobotics challenge: Simulating real-time robotic disas-ter response. IEEE Transactions on Automation Scienceand Engineering, 12(2):494–506, 2015.

19. Arnoud Visser Masaru Shimizu, Nate Koenig and To-moichi Takashi. RoboCup 2015: Robot World Cup XIX,chapter A Realistic RoboCup Rescue Simulation Basedon Gazebo, pages 331–338. 2016. ISBN 978-3-319-29338-7.

20. Steven Balakirsky, Chris Scrapper, Stefano Carpin, andMichael Lewis. Usarsim: providing a framework formulti-robot performance evaluation. In Proceedings ofPerMIS, 2006.

21. Stefano Carpin, Jijun Wang, Michael Lewis, AndreasBirk, and Adam Jacoff. High fidelity tools for rescuerobotics: results and perspectives. In RoboCup 2005:Robot Soccer World Cup IX, pages 301–311. Springer,2006.

22. Edwin Olson, Johannes Strom, Ryan Morton, An-drew Richardson, Pradeep Ranganathan, Robert Goed-del, Mihai Bulic, Jacob Crossman, and Bob Marinier.Progress toward multi-robot reconnaissance and themagic 2010 competition. Journal of Field Robotics, 29(5):762–792, 2012.

23. Huadong Wang, Shih Yi Chien, Michael Lewis,Prasanna Velagapudi, Paul Scerri, and Katia Sycara.Human teams for large scale multirobot control. InIEEE International Conference on Systems, Man andCybernetic (SMC), pages 1269–1274, 2009.

24. Alain Caltieri and Francesco Amigoni. High-level com-mands in human-robot interaction for search and rescue.In RoboCup 2013: Robot World Cup XVII, pages 480–491. Springer, 2013.

25. Jacopo Banfi, Alberto Quattrini Li, Nicola Basilico,and Francesco Amigoni. Communication-constrainedmultirobot exploration: Short taxonomy and compara-tive results. In Proceedings of the IROS Workshop onOn-Line Decision-Making in Multi-Robot Coordination(DEMUR2015), pages 1–8, October 2015.

26. Victor Spirin, Stephen Cameron, and Julian de Hoog.Time preference for information in multi-agent ex-

10 Raymond Sheh et al.

ploration with limited communication. In TowardsAutonomous Robotic Systems, pages 34–45. Springer,2013.

27. Paul Viola and Michael Jones. Rapid object detectionusing a boosted cascade of simple features. In ComputerVision and Pattern Recognition, 2001. CVPR 2001. Pro-ceedings of the 2001 IEEE Computer Society Confer-ence on, volume 1, pages I–511. IEEE, 2001.

28. Navneet Dalal and Bill Triggs. Histograms of orientedgradients for human detection. In Computer Vision andPattern Recognition, 2005. CVPR 2005. IEEE Com-puter Society Conference on, volume 1, pages 886–893.IEEE, 2005.

29. Helen Flynn, Julian De Hoog, and Stephen Cameron.Integrating automated object detection into mapping inusarsim. In Proceedings of the International Confer-ence on Intelligent Robots and Systems (IROS 2009),Workshop on Robots, Games, and Research: Successstories in USARSim, pages 29–34. Citeseer, 2009.

30. Y. Uzun, M. Balcilar, K. Mahmoodi, F. Davletov, M.F.Amasyali, and S. Yavuz. Usage of hog (histograms oforiented gradients) features for victim detection at dis-aster areas. In Electrical and Electronics Engineering(ELECO), 2013 8th International Conference on, pages535–538, Nov 2013.

31. Okke Formsma, Nick Dijkshoorn, Sander van Noort,and Arnoud Visser. Realistic simulation of laser rangefinder behavior in a smoky environment. In RoboCup2010: Robot Soccer World Cup XIV, pages 336–349.Springer, 2011.

32. Violeta Bogdanova. Image enhancement using retinexalgorithms and epitomic representation. BulgarianAcademy of Sciences Cybernetics, and InformationTechnologies, 10(3), 2010.

33. Marco Zaratti, Marco Fratarcangeli, and Luca Iocchi.A 3d simulator of multiple legged robots based on us-arsim. In Robocup 2006: Robot Soccer World Cup X,pages 13–24. Springer, 2006.

34. Max Pfingsthorn, Bayu Slamet, and Arnoud Visser. Ascalable hybrid multi-robot slam method for highly de-tailed maps. In RoboCup 2007: Robot Soccer WorldCup XI, pages 457–464. Springer, 2007.

35. H. Kitano, S. Tadokoro, I. Noda, H. Matsubara, T. Taka-hashi, A. Shinjou, and S. Shimada. RoboCup Rescue:Search and rescue in large-scale disasters as a domainfor autonomous agents research. In IEEE Conf. on Man,Systems, and Cybernetics(SMC-99), 1999.

36. Martijn N Rooker and Andreas Birk. Combining ex-ploration and ad-hoc networking in robocup rescue. InRoboCup 2004: Robot Soccer World Cup VIII, pages236–246. Springer, 2004.

37. Sarvapali D. Ramchurn, Alessandro Farinelli,Kathryn S. Macarthur, and Nicholas R. Jennings.

Decentralized coordination in robocup rescue. TheComputer Journal, 53(9):1447–1461, 2010.

38. Xiao Jia and Max Q-H Meng. A survey and analysisof task allocation algorithms in multi-robot systems. InRobotics and Biomimetics (ROBIO), 2013 IEEE Inter-national Conference on, pages 2280–2285. IEEE, 2013.

39. Paulo Roberto Ferreira Jr, Fernando Dos Santos,Ana LC Bazzan, Daniel Epstein, and Samuel J Waskow.Robocup rescue as multiagent task allocation amongteams: experiments with task interdependencies. Au-tonomous Agents and Multi-Agent Systems, 20(3):421–443, 2010.

40. Takeshi Morimoto, Kenji Kono, and Ikuo Takeuchi.Yabai the first rescue simulation league champion. InRoboCup 2001: Robot Soccer World Cup V, pages 49–59. Springer, 2001.

41. Robin R Murphy, Jenn Casper, and Mark Micire. Po-tential tasks and research issues for mobile robots inrobocup rescue. In RoboCup 2000: Robot Soccer WorldCup IV, pages 339–344. Springer, 2000.

42. Thorsten Linder, Viatcheslav Tretyakov, Sebastian Blu-menthal, Peter Molitor, Dirk Holz, Robin Murphy,Satoshi Tadokoro, and Hartmut Surmann. Rescuerobots at the collapse of the municipal archive ofcologne city: a field report. In IEEE International Work-shop on Safety Security and Rescue Robotics (SSRR),pages 1–6, 2010.

43. Geert-Jan M Kruijff, Viatcheslav Tretyakov, Tamas Lin-der, Fiora Pirri, Mario Gianni, Panagiotis Papadakis,Matia Pizzoli, Aloka Sinha, Emanuele Pianese, Salva-tore Corrao, et al. Rescue robots at earthquake-hit mi-randola, italy: a field report. In Safety, Security, andRescue Robotics (SSRR), 2012 IEEE International Sym-posium on, pages 1–8. IEEE, 2012.

44. Fumitoshi Matsuno, Noritaka Sato, Kazuyuki Kon, Hi-roki Igarashi, Tetsuya Kimura, and Robin Murphy. Uti-lization of robot systems in disaster sites of the greateastern japan earthquake. In Field and Service Robo-tics, pages 1–17. Springer, 2014.

45. Ranjit Nair, Milind Tambe, and Stacy Marsella. Teamformation for reformation in multiagent domains likerobocuprescue. In RoboCup 2002: Robot Soccer WorldCup VI, pages 150–161. Springer, 2002.

46. Paul Scerri, Alessandro Farinelli, Steven Okamoto, andMilind Tambe. Allocating tasks in extreme teams. InProceedings of the fourth international joint conferenceon Autonomous agents and multiagent systems, pages727–734. ACM, 2005.

47. Mohammad Nejad Sedaghat, Leila Pakravan Nejad,Sina Iravanian, and Ehsan Rafiee. Task allocation forthe police force agents in robocuprescue simulation.In RoboCup 2005: Robot Soccer World Cup IX, pages656–664. Springer, 2005.

16 Years of RoboCup Rescue 11

48. Ivette Martınez, David Ojeda, and Ezequiel Zamora.Ambulance decision support using evolutionary re-inforcement learning in robocup rescue simulationleague. RoboCup 2006: Robot Soccer World Cup X,pages 556–563, 2007.

49. Fernando Dos Santos and Ana LC Bazzan. Towards ef-ficient multiagent task allocation in the robocup rescue:a biologically-inspired approach. Autonomous Agentsand Multi-Agent Systems, 22(3):465–486, 2011.

50. Somchaya Liemhetcharat and Manuela Veloso.Weighted synergy graphs for effective team forma-tion with heterogeneous ad hoc agents. ArtificialIntelligence, 208:41–65, 2014.

51. Marc Pujol-Gonzalez, Jesus Cerquides, Alessan-dro Farinelli, Pedro Meseguer, and Juan AntonioRodriguez-Aguilar. Efficient inter-team task allocationin robocup rescue. In Proceedings of the InternationalConference on Autonomous Agents and MultiagentSystems, pages 413–421, 2015.

52. James Parker, Ernesto Nunes, Julio Godoy, and MariaGini. Exploiting spatial locality and heterogeneity ofagents for search and rescue teamwork. Journal of FieldRobotics, 2015.

53. N. Schurr, J. Marecki, J.P. Lewis, M. Tambe, andP. Scerri. Multi-Agent Programming: Languages, Plat-forms and Applications, chapter The Defacto System:Coordinating Human-Agent Teams for the Future ofDisaster Response, pages 197–215. Springer US,Boston, MA, 2005. ISBN 978-0-387-26350-2.

54. Alexander Kleiner, Alessandro Farinelli, SarvapaliRamchurn, Bing Shi, Fabio Maffioletti, and RiccardoReffato. Rmasbench: benchmarking dynamic multi-agent coordination in urban search and rescue. In Pro-ceedings of the 2013 international conference on Au-tonomous agents and multi-agent systems, pages 1195–1196. International Foundation for Autonomous Agentsand Multiagent Systems, 2013.

55. Kazuo Takayanag, Shunki Takami, Yoshiyuki Kozuka,and Nobuhiro Ito. Design and implementation of theagent framework for the robocup rescue simulation newentrants. In Proceedings of the he 29th Annual Confer-ence of the Japanese Society for Artificial Intelligence,2B5-NFC-02c-2, 2015.

56. M Okugawa, K Oogane, M Shimizu, Y Ohtsubo,T Kimura, T Takahashi, and S Tadokoro. Proposal ofinspection and rescue tasks for tunnel disasters??? taskdevelopment of japan virtual robotics challenge. In2015 IEEE International Symposium on Safety, Secu-rity, and Rescue Robotics (SSRR), pages 1–2, 2015.

57. RoboCup Technical Committee. RoboCup Rescue Sim-ulation League Virtual Robot Competition Rules Docu-ment Version 1.5, March 2016.

58. Zeid Kootbally, Stephen Balakirsky, and Arnoud Visser.Enabling codesharing in rescue simulation with usar-sim/ros. In RoboCup 2013: Robot World Cup XVII, vol-ume 8371 of Lecture Notes in Computer Science, pages592–599. Springer Berlin Heidelberg, 2014.

59. Masaru Shimizu, Nate Koenig, Arnoud Visser, and To-moichi Takashi. A realistic robocup rescue simulationbased on gazebo. In RoboCup Symposium 2015, Devel-opment Track, 2015.

60. Adam Jacoff, Elena Messina, Hui-Min Huang, AnnVirts, Anthony Downs, Richard Norcross, and Ray-mond Sheh. Guide for Evaluating, Purchasing, andTraining with Response Robots Using DHS-NIST-ASTM International Standard Test Methods. NationalInstitute of Standards and Technology report, 2014.

61. RoboCupRescue Organising Committee. RoboCupRescue Rulebook Version 1.0, March 2016.