Accessible Aerial Autonomy?

Lilian de Greef, Brad Jensen, Kim Sheely

Nick Berezny, Cal Poly Pomona '12

Malen Sok, Cal Poly Pomona '13

special guest star! Steve Matsumoto, HMC '12

Accessible Aerial Autonomy?

Lilian de Greef, Brad Jensen, Kim Sheely

Nick Berezny, Cal Poly Pomona '12

Malen Sok, Cal Poly Pomona '13

special guest star! Steve Matsumoto, HMC '12

Motivation

Key lesson: Neither all drones programmers – are created equal

Neither all drones – nor all are created equal

Question

Does the ARDrone

effective

ARDrone make an

effective robot?

Question

Does the ARDrone

effective

Raw material:Raw material:

Goal: accomplish tasks with the drone, the

create, and computer vision

• closed hardware

• but an open, ASCII API

• two cameras

• internal sensing (gyro/accel.)

ARDrone make an

effective robot?

accomplish tasks with the drone, the

create, and computer vision

blind-enord.py

pydrone

ROS ~ Robot Operating System

libardroneARDronemsg

srv

arnetwork

too low level for our needs

enord.py

OpenCV

FLANN for nearest neighbors

Other ROS facilities

Robot Operating System

libardrone

arnetworkgeneric API/framework

too low level for our needs

Several tasks tried...

(0) Room/hallway flying

(1) Cooperating with the Create

(2) Navigating among landmarks

(3) Localization without landmarks

(4) Room/hallway flying(4) Room/hallway flying

(1) Cooperating with the Create

(2) Navigating among landmarks

(3) Localization without landmarks

i

Task 1: Follow that !

Image processing approach:(1) threshold image to find dark regions and contours

(2) circle? compare region with min. enclosing circle

(3) rectangle? compare region with min. enclosing rect.

(4) filter noise, find centers, and construct heading line

We put a ! on the Create to • help discern location

• help discern orientation

(1) threshold image to find dark regions and contours

compare region with min. enclosing circle

compare region with min. enclosing rect.

(4) filter noise, find centers, and construct heading line

! finding

GCER!

GCER cooperation demoGCER cooperation demo

Lessons learned

• The ! was far from a perfect landmark

• We wanted to use something more robust that

could give us more accurate pose estimation

• We decided to explore April Tags...• We decided to explore April Tags...

The ! was far from a perfect landmark

We wanted to use something more robust that

could give us more accurate pose estimation

April Tags...April Tags...

APRIL tags

Autonomy, Perception, Robotics,

Java-based landmark library from U.

We integrated it into ROS using Python's

an example tag in the center...

obotics, Interfaces, and Learning

landmark library from U. Michigan

using Python's os.system call...

provides full 6 DOF pose and scale

APRIL tags' scale range

an example tag in the center...

APRIL tags' scale range

provides full 6 DOF pose and scale

Task 2: The Hula-hoop hop

Getting from

A to B A B

hoop hop

Task 2: The Hula-hoop hop

getting from point A to point B

hoop hop

getting from point A to point B

Hula-hop's state machine

all transitions can also be made by the keyboard

hop's state machine

all transitions can also be made by the keyboard

Hula-hop demo

sliding-scale autonomy scale autonomy is crucial

Hula-hop challenges

Drone challenges:

• drift ~ not easily positionable

• connection ~ video freezes

• artifacts ~ image stream noise

APRIL tag challenges:

• too narrow a field of view: height/scale tradeoffs

• call to APRIL library is slow (.5 second/image)

• unmodifiable environments?

Could we do without

~ image stream noise

too narrow a field of view: height/scale tradeoffs

call to APRIL library is slow (.5 second/image)

without tags?

example encoding (?) artifact

Localization without tags?

SURF features

• locally unique image patches

• fast libraries for extraction

• each SURF feature is described

by a 64- or 128-dimensional by a 64- or 128-dimensional

vector that encodes size and

local edge orientations

• in general, similar descriptor

vectors are likely to be similar

(or identical) image features

Localization without tags?

locally unique image patches

each SURF feature is described

in general, similar descriptor

SURF features are great for

place-recognition training!

Localization plan?

new image map images + matches

Mapping (by hand)

• collect images and positions

• extract & store SURF features

Localization

• take a new image• take a new image

• extract SURF features

• match them against the map

• estimate a pose distribution

map images + matches

Image-based map...

four locations in the NW corner of Sprague

Locations with

stored images == stored images ==

nodes in a graph

four locations in the NW corner of Sprague

Image-based map...

beside the kitchen, facing roughly Wbeside the kitchen, facing roughly W

Image-based map...

beside the boardgames, facing roughly N, facing roughly N

Live localization

top three matches and their likelihood distribution plotted on the maptop three matches and their likelihood distribution plotted on the map

Demo...

Results

Simply counting the # of matches did not

yield good localization results:

< 25% of correct locations

< 10% of correct orientations< 10% of correct orientations

simple image

Simply counting the # of matches did not

yield good localization results:

< 25% of correct locations

< 10% of correct orientations< 10% of correct orientations

matches m

Σ

simple image-match score

1

Other scoring systems...

... and filters

weights features inversely

proportional to their SURF

distance

... and filters

bidirectional matches only

strong ratio-matches only

Other scoring systems...

matches m

Σ

distance-scaled score

1

ε + dist(m)

omits features whose nearest

neighbor is about as good a

match as its 2nd nearest neighbor

omits features whose best match

is not unique in both directions

Without bidirectional filtering

many different features can have the

Without bidirectional filtering

many different features can have the same best match

With bidirectional filtering

only mutually unique best matches are considered

With bidirectional filtering

only mutually unique best matches are considered

Comparative results

The AR Drone is a capable platform-- as long as precise positioning is not required

Options includeo research to improve localizationo tasks that do not require precision

Comparative results

o tasks that do not require precision

The ROS software scaffolding is excellent-- (though not always thoroughly documented)

The project's AR Drone drivers and supporting software generated interest at AAAI and will be released into the

community's (Willow Garage's) repository.

is a capable platformas long as precise positioning is not required

research to improve localizationtasks that do not require precisiontasks that do not require precision

software scaffolding is excellent(though not always thoroughly documented)

The project's AR Drone drivers and supporting software generated interest at AAAI and will be released into the

community's (Willow Garage's) repository.

Thoughts or comments?Thoughts or comments?

i

AR Drone Mechanics

• Drone itself has only 4 degreethrust)

• Commands give us control of 4 different degrees of freedom. (x, y, z and yaw)

Roll

Key lesson learned: The drone is

AR Drone Mechanics

rees of freedom. (roll, pitch, yaw,

Commands give us control of 4 different degrees of

Yaw

The drone is NOT precisely positionable.