RAVON - Platform for Outdoor

Field Robotics

Prof. Dr. Karsten Berns

Robotics Research Lab

Department of Computer Sciences

University of Kaiserslautern

Application Areas for Off-road Robots

• Service tasks

– agriculture, forestry, logistics,

environmental monitoring

• Assistance in disaster areas

– equipment transport, scouting

• Security tasks

– site / border patrol, demining

Off-road Robots for Different Applications

Timberjack

© John Deere

Workpartner

© Helsinki University of Technology

Telerob tEODor

© www.telerob.de

Remotec Wheelbarrow Revolution

Off-road Vehicles

Stanley (Stanford) RTS-Dora (Hannover)

Demo III (U.S. Army)

Navlab (CMU)

RoboScout (Base 10)

Sojourner (JPL) Amor

(Uni Siegen) MuCar

(UniBW München)

RAVON Project

Goal: Development of vehicle that can fully autonomously operate in

highly vegetated, rough terrain

Operational Environment

RAVON

• 4WD with separate motors

• Front and rear axes can be steered

independently

• Max. velocity: 10 km/h

• Max. slope: 100% at 7 km/h

• Energy source: 8 Lead batteries

• Runtime: about 4 hours

• LWH: 2.35x1.4x1.8 m

(highest point: GPS antenna)

• Weight: 750 kg

• Ground clearance: 0.3 m

Control System Architecture

Drive

Commands

Hardware Interface

Sensor Processing

Sector

Map

Creation

Mediation

Global Navigation

Short-term

Memory

Raw

Sensor Data

Se

cto

r M

ap

s

Status

Status Target

Poses

Target

Poses

Safety System

Point Access

Local Path Planning

Local Navigation

External Sensor Systems

Long Range Color Stereo System Large Scale Terrain Traversability Estimation

Short Range Color Stereo System Obstacle Detection

Rotating 2D Laser Scanner Obstacle Detection / 3D Local Memory

Planar 2D Laser Scanner Obstacle Detection / Safety System

Spring-Mounted Bumper Tactile Vegetation Discrimination

Instrumented Safety Bumper

Design and construction of the bumper

Tactile Mapping of the environment

Titel des Vortrags

3D Laser Scanner Concept

Sensor Processing

Main sensor: panning laser range finder

• Yields position, shape, and type of hazards in 3D space, e.g.:

Solid Object Penetrable Object

• Penetration technique for evaluating an obstacle’s rigidity

( Gras or rock?)

h

d h h

d

Positive Obstacle Negative Obstacle Overhanging Obstacle Water

Titel des Vortrags

beneath ground

ground

near ground

far ground

sky

shortest ground

distance

Y

X

• Extract ground representatives

(blue)

• From ground level classify:

• Pts beneath ground

• Associated ground pts (purple)

represent the load-bearing

surface

• Near ground pts

(surmountable)

• Far ground pts (positive or

overhanging obstacles)

• Sky points

Evaluation of Individual Scans

Titel des Vortrags

Solid Entities

Vegetation

Classification – Examples

Example of Content of Grid Map Cells

• General information: Properties

• Can also store height or organize cell space in 3D

Titel des Vortrags

Vision-Based Obstacle Detection

• Stereo-vision-based 3D scene analysis

• Tracking of obstacles over time

-> 4th Dimension

• Filter artefacts

• Building local maps

• Determination of colour

and texture

Water Detection based on Polarisation

Detection Process

AquaVisor C

(H&S Robotic Solutions, KL)

Compact design

90mm x 90mm x 130mm

Weight: 600g

Onboard sensor data

evaluation

Extension modules for

Vegetation detection

3D-enhanced Vision

Grid Map as Local Obstacle Memory

• Robot centric, orientation-fixed local grid map

• Scrolls in x- and y-direction

• Serves as local obstacle memory

– Robot “remembers” obstacles

– Collisions with obstacles not in direct sensor range can be

avoided

Sensor Data Representation

• Data of different sensor systems is entered into

grid maps

• Grid maps used as “local obstacle memory”

Important as sensor systems cannot

cover complete area around robot

• Sector maps as abstract and uniform

data representation

• Extracted from grid map

• Each sector contains

information about area

it covers Cartesian Sector Maps Polar Sector Map

Grid Maps

Extraction

Sector Maps around RAVON

Local Navigation (Pilot) and Drive Modes

• Task: drive robot to next target location while avoiding collisions with

obstacles

• Fast driving: requires long-range obstacle detection and the

evaluation of all detected obstacles

• Moderate driving:

– Uses penetration evaluation for vegetation density determination

– Maximum velocity reduced as only close-range area can be

monitored

• Tactile creep:

– Evaluates deflection of the bumper system

– Maximum velocity is limited to very low value

Pilot can adopt robot’s basic driving behaviour to situation

Behavior-based Control Approach

Approach: Decompose task into relatively simple independent

components

• Incremental implementation with subsequent evaluation

• Distributed representation of environment, no central world model

Problems to be solved:

• Find suitable decomposition

• Coordination of contradictory commands

• Validation of system behavior

Behavior-Based Architecture iB2C

• Uniform behavior modules

• Behavior B defined as B=(r,a,F) with

– Input vector e and output vector u sensor/control data

– Stimulation s in [0,1] gradually activating the behavior

– Inhibition i in [0,1] gradually deactivating the behavior

– Activation ι=s∙(1-i)

– Activity a in [0,1] indicating how much B is doing

– Target rating r in [0,1] assessing current situation

– Transfer function F(e,s,i)=u implementing the functionality

Basic behavior module

Behavioral Groups

Behavioural Group for

Fast Driving

Structure of the Pilot

Emergency Stop

Drive Commands

Point Access

Behavioural Group for

Moderate Driving

Behavioural Group for

Tactile Creep

State Switching and

Behavioural Group Stimulation

Stimulation

Drive Commands

Drive Commands

Fusion

Drive Commands

Human

Operator

Control Chains

Slow Down Behaviors

Safety Behaviors

Keep Distance Behaviors

Evasion Behaviors

Structure of the Guardian

Intermediate Layer - Passage Driving

• Use of passage realised by behaviour-based network

• Robot is drawn towards PEMP

=> does not drive into indentation

• Robot reaches PEMP with orientation of passage

=> is driving smoothly into the passage

Start

Passage Entry

Draw towards Passage

Target

Passage Orientation

Global Robot Navigation

• How can a robot navigate efficiently to

a distant location?

• Requires a world model (a map)

– quantify terrain traversability

→map cost measure

• Traversability estimation is hard

– ground properties

– flexible/rigid vegetation

?

Proposed Combined Approach

• Formulate a global (off-road) navigation strategy which

– scales to very large environments, but

– allows to reason about terrain traversability

• Solution strategy:

– use hybrid world representation

• global topological navigation layer

• local metrical pilot layer

– extend topological world model to handle traversability costs

– transfer local cost experience to global level

Abstract Views - Example

• Sensors

Physical

Short-Term Memory

Slowly Traversable Terrain Quickly Traversable Terrain

Sensors

Virtual

Sensors

Pilot

• Receives target from topological navigator

• Uses sensors to build metrical (grid) short-term memory

• Relevant memory content is transformed into standardized ‘abstract views’

• Behavior based control system uses abstract views for steering and collision avoidance

Local Traversability Maps (LTMs)

• Obstacle information is

relevant for later

topological map

extension

• Data is transferred from

abstract views into local

traversability maps

(LTMs)

• LTMs store abstracted

cost modifiers around a

topological node

Local Traversability Maps

A posteriori Cost Learning

• Observation of

behaviors yields cost

information after each

edge traversal

• Spatio-temporal

integration

• Transfers real-world

cost experience from

local to global scope

Topological Exploration

• Generate multiple new

path hypotheses

between two

unconnected nodes

• Predict likely costs

• Add cheapest route

• How to predict costs for

unknown edges ?

Edge Cost Calculation

• Three types of edge costs: risk, effort, and familiarity

• Risk and effort can be learnt during edge traversing by monitoring

distinct behaviours of pilot

No need to analyse the terrain extensively and construct a highly

detailed world model

• Different methods to predict costs of edges that have not been

traversed yet

Costs for driving in partly unknown terrain can be estimated

Vision Based Terrain Analysis

Image Feature Extraction

stereo camera system

color image pairs

texture area descriptors

HSV

VAR

• Highly parallelizable operators extract independent information

– Local Binary Pattern, VARiance, Hue Saturation Value

• Descriptors support efficient area union: →

contrast

color

LBP

BA

)( Adi

)()()( BdAdBAdiii

A

Unsupervised Segmentation

• Construct image regions with similar apperance → Fewer, more stable objects to classify

• Unsupervised Split & Merge segmentation – Jenson-Shannon divergence as pseudo-metric

– Threaded neighborhood lists for efficient merging

Self-Supervised Learning Procedure

1. Stereo image capture

2. Edge traversal

3. Experience projection

4. Region correlation

5. Prototype addition

Experiment: Self-Supervised Learning

Offroad Path Detection

• Not only look for free traversable space

• Find structures that help reaching the horizon

Long Range Stereo Reconstruction

Finding Ground and Horizon

• “Side-view“ Analysis

• Filter by density

• Apply RANSAC and least squares fitting

• Classify horizon and ground region

Obstacle Analysis

• Front-to-back Analysis

• Adopt obstruction to back

• Extract upper ground edge

• Fit polynomial to ground

• Use derivative for traversability

classification

• Take declination of vehicle into

account

.

.

.

Traversability Map

• Apply perspective back-transformation

• Extraction and filtering of traversable regions

Path Classification via Particle Filter

• Apply path primitives (polynomials of deg. 2)

• Primitives are managed as particles

• Overlapping of traversability map and primitive

form confidence

+ =

Results

Place Recognition

• Machine-readable description needed

• Descriptors must be comparable

• Common known fact: Single features do

no generally work on their own

• Redundancy compensates for single

feature failure

• What is a place?

– Relevance for navigation decisions

– Descriptor is prominent in local area

Point Features (SURF)

• Salient points in scale-space (similar to SIFT)

• Gradient in these point forms feature descriptor

• Set of features forms place descriptor

Comparison of Point Features

• Use RANSAC filter for affine transformed correspondence sets

• 500 random sets are feasible for shifted and scaled images

Area Features (LBP)

• Partition image and compute LBP histgram for each tile

• Image descriptor is the list of histograms

Comparison of Area Features

• Shift images and compare histograms

• Calculate normalized sum of absolute differences

• Example for not matching images

Comparison of Area Features (Matching)

• Minimal value for each pair relates to best shift

• Low value means high similarity

• Not robust against roll rotation

Results: Correct Decision due to SURF

• SURF (blue) outweighs uncertain and wrong LBP (orange)

• Image pairs showing the same place usually have a great number

(>>10) of SURF matches after RANSAC filtering

Results: Correct Decision due to LBP

• In some situations SURF failes

• LBP does not dominate in one place but can compensate

• Still correct with wron SURF (blue) and correct LBP (orange)

Control System Architecture

Travel through Palantine Forest

Thank you for your attention!

http://agrosy.informatik.uni-kl.de

References

• [Wünsche 07] LIDAR-based Perception for Offroad Navigation M.

Himmelsbach, F. Hundelshausen, H-J. Wünsche

• [Siegwart 07] Path Planning, Replanning, and Execution for

Autonomous Driving in Urban and Offroad Environments R.

Philippsen, S. Kolski, K. Macek, R. Siegwart

• [Peterson 05] Red Team Too, Darpa Grand Challenge 05, Technical

Paper, K. Peterson et. al.

• [Urmson 05] High Speed Navigation of Unrehearsed Terrain, C. Urmson

et. al.

• [Pfaff 08] Probabilistic Models for Autonomous Systems, P. Pfaff

References 2

• [Thrun 95] Map Learning and High Speed Navigation in RHINO,

S. Thrun et. al.

• [Vale 95] Mobile Robot Navigation in Outdoor Environments: A

Topological Approach, A. Vale

• [Tapus 05] A Cognitive Modeling of Space using Fingerprints of

Places for Mobile Robot Navigation, A. Tapus et. al.

• [Madrigal 99] Mobile robot path planning: a multicriteria

approach, J. Fernandeza-Madrigal et. al.

Related Work

• Most offroad robots use metrical grid maps

• SmartTer (ETH Zuerich) [Siegwart 07]

– global planning: multi-level-surface map, field D* algorithm

– local planning: grid map, driving arcs

• Red Team (Sandstorm, CMU) [Urmson 05]

– global: A* on high-resolution elevation map

– local: laser scan evaluation, path adjustment

• Red Team Too (CMU) [Peterson 05]

– global: A* on high-resolution elevation map

– local: obstacle + traversability cost map, A*

• MuCar (Uni BW München) [Wünsche 07]

– local: traversability cost map, driving ‚tentacles‘

Related Work

• Topological Maps are mostly used indoors or in urban areas

• RHINO (Uni Bonn) [Thrun 95]

– focus on fast path planning (dijkstra), abstraction (voronoi)

– distance based cost measure

• ATRV-Jr (Uni Lisbon) [Vale 05]

– Focus on localization (POMDP), map building (EM)

– constant cost measure

• Tapus (ETH Zürich) [Tapus 05]

– focus on localization (POMDP), map building (fingerprints)

– constant cost measure

• Ram-2 (Uni Malaga) [Madrigal 99]

– focus on multicriteria path planning

– hierarchical constraint satisfaction, pre-learned costs

LADAR based Classification