Towards Hierarchical Place Recognition for

Long-Term Autonomy

Kirk MacTavish and Timothy D. Barfoot

ICRA Workshop on Visual Place Recognition in Changing Environments

June 2014

•Why use Place Recognition?

•How can we deal with variable lighting?

•What about Long-term Autonomy?

Motivation | Overview

2

Motivation | GPS Denied Environments

3

Mines

Planetary ExplorationUrban Canyons

Motivation | What is place recognition?

Relative localization is sufficient for many

tasks.

Have we beenhere before?

Cummins, M. and Newman, P., “FAB-MAP: Probabilistic Localization and Mapping in the Space of Appearance,” The International Journal of Robotics Research, 27(6):647–665, 2008.

Perceptual Aliasing Scene & Perspective Change

4

•Why use Place Recognition?

•How can we deal with variable lighting?

•What about Long-term Autonomy?

Motivation | Overview

5

11 am 6 pm

Motivation | Images Change over Time

6

• We can get some degree of lighting invariance from color-space manipulation.

• Dealing with Shadows: Capturing Intrinsic Scene Appearance for

Image-based Outdoor Localisation (Corke et al. 2013)

• Shady Dealings: Robust, Long- Term Visual Localisation using

Illumination Invariance (McManus et al. 2014)

• Lidar intensity images are unaffected by lighting conditions over the full day-night period.

Motivation | Lighting Invariance

7

Motivation | 24 Hours of Images

Place recognition compares places after significant time has

passed. This makes lighting invariance extremely important.

8

Motivation | Discretization

Camera

Lidar

9

The discretization is less obvious

•Why use Place Recognition?

•How can we deal with variable lighting?

•What about Long-Term Autonomy?

Motivation | Overview

10

It would be interesting to design the algorithm for operation over a 10 year period.

• Algorithms should be able to run in a constant computational budget for 10 years.

• Algorithms should be able to learn and adapt to the changing environment over the 10 year period.

Motivation | Long-Term Autonomy

11

• “Highly Scalable Appearance-Only SLAM – FAB-MAP 2.0” (Cummins and Newman, 2009)

• Linear complexity but very fast

• Are We There Yet? Challenging SeqSLAM on a 3000 km Journey Across All Four Seasons (Sunderhauf, Neubert and Protzel, 2013)

• Performs well under seasonal change, but still linear complexity and is sensitive to

camera alignment

• “Appearance-Based Loop Closure Detection for Online Large-Scale and Long-Term Operation” (Labbé and Michaud, 2013)

• Constant-time, but does not consider the whole map

Motivation | Long-Term Autonomy

12

Methodology | Roadmap

Place Recognition Lighting Change

Computational

Complexity

Hierarchy

Place

Discretization

LIDAR

Two Problems

One Solution

13

•Place Hierarchy

•FAB-MAP with groups

Methodology | Overview

14

Methodology | Computational Complexity

15

•Place Hierarchy

•FAB-MAP with groups

Methodology | Overview

16

Methodology | FAB-MAP with groups

City Centre Dataset Groups of 128 images

• Used the OpenFABMAPimplementation

• Adapted Bag-of-Words (BoW) descriptor to use features from groups of images, rather than single images

• Have not yet done hierarchical expansion

17

Methodology | FAB-MAP with groups

City Centre Dataset

Groups of 128 images

18

Bag-of-Words is a histogram of discretized

visual features

Methodology | FAB-MAP with groups

City Centre Dataset

Groups of 128 images

Single Image

Larger Group

BoW descriptors became less sparse with larger groups. This invalidates training.

19

Methodology | FAB-MAP with groups

The number of times a word must be seen in a group to be counted

as present

20

Single Image

Larger Group

BoW descriptors became less sparse with larger groups. This invalidates training.

Methodology | FAB-MAP with groups

Single Image

Larger Group

Group BoW descriptors are now as sparse as single images.

Training is useful again.

The number of times a word must be seen in a group to be counted

as present

21

•Datasets

•Results on camera images

•Results on LIDAR intensity

Results | Overview

22

Results | Datasets

City Centre New College

Oxford Mobile Robotics Group

23

Results | Datasets

KITTI Vision Benchmark SuiteOdometry Dataset

24

Results | Datasets

ASRL Sudbury LIDAR Dataset

25

Available: http://asrl.utias.utoronto.ca/datasets/abl-sudbury/

Anderson A, McManus C, Dong H, Beerepoot E, and Barfoot T D.

“The Gravel Pit Lidar-Intensity Imagery Dataset”.

University of Toronto Technical Report ASRL-2012-ABL001

•Datasets

•Results on camera images

•Results on LIDAR intensity

Results | Overview

26

Results | Camera | Confusion Matrices

KITTI-06 New College City Centre

27

La

rger

Gro

up

s o

f Im

ag

es

Results | Camera | Performance

28

Single-image FABMAP

Single-image FABMAP

Results | Camera | City Centre with groups

29

•Datasets

•Results on camera images

•Results on LIDAR intensity

Results | Overview

30

Results | LIDAR | Sudbury Dataset

31

Single-image FABMAP

• Develop the hierarchical inferencing algorithm.

• Develop a deeper high-level descriptor that can describe large places better than Bag-of-Words.

• Adapt the algorithm to use an unstructured LIDAR descriptor for continuous scans.

Future Work

32

Questions?kirk.mactavish@mail.utoronto.ca

http://asrl.utias.utoronto.ca

Email:

Web:

33

Towards Hierarchical Place Recognition for Long-Term Autonomy