Understanding Underwater Optical Image DatasetsOptical Modems: 1,000,000 bits/s at 100 meters [Farr...

1 PhD Thesis Defense - J.W. Kaeli, 2013

Computational Strategies for

Understanding Underwater

Optical Image Datasets

Jeffrey W. Kaeli

Advisor:

Hanumant Singh, WHOI

Committee:

John Leonard, MIT

Ramesh Raskar, MIT

Antonio Torralba, MIT


“latency of understanding” in underwater robotics


“latency of understanding” LOU 0 for ROVs


what do we lose by “cutting the cord?”


what can we gain by “cutting the cord?”


AUVs generate large volumes of data

O(1,000s) images

1 image

3 seconds

3600 s

1 hour

O(hours)

x

x

O(10,000,000,000) bits

O(10,000,000) bits

1 image

x

O(1-10) GB


underwater communication channels are limited

Optical Modems:

1,000,000 bits/s at 100 meters [Farr et al. 2010]

Acoustic Modems:

100 - 1,000 bits/s at kilometers

[Frietag et al. 2005]


wavelet-based compression coded with

Set Partitioning in Hierarchical Trees (SPIHT)

smaller packet size optimized for acoustics

fully embedded progressive encoding

throughput of 1 megapixel image / ~15 minutes

acoustic transmission of imagery

[Murphy 2012, PhD Thesis]


manual annotation is time consuming

[Ferrini et al., 2006] [Kaeli 2011] [Futrelle & York, 2012]


automatic classification aids quantitative analysis

habitats [Loomis 2011] [Pizarro et al., 2009]

[Rigby et al., 2010] [Steinberg et al., 2011]

fish [Loomis 2011] [Spampinato et al., 2010]

scallops [Dawkins et al., 2013]

starfish [Clement et al., 2002]

coral [Johnson et al., 2006] [Kaeli et al, 2006]

[Purser et al., 2009] [Soriano et al., 2001]

[Shihavuddin et al., 2013] [Stokes & Deane, 2009]


summary of latencies contributing to overall LOU

O(hours)

? O(days-weeks)

O(hours)


what is the smallest latency with which we can

begin to “understand” the survey environment?

RAW IMAGERY

PRE-PROCESSING

SELECTION

COMPRESSION

TELEMETRY

UNDERSTANDING

{ [Murphy 2012,

PhD Thesis]

Chap. 1,2

Chap. 2,3

Chap. 3


1. Image Correction

2. Computational Strategies

3. Understanding Underwater


outline of thesis chapters and contributions


1. Image Correction


review underwater image formation and broad

range of existing correction techniques

estimate water column properties and unknown

vehicle parameters using multi-sensor fusion

use estimated values to correct images




introduce novel multi-scale image processing

framework for efficient feature computations

discuss design of invariant features





propose lightweight keypoint detection and

description scheme for use in scene classification

modify surprise-based navigation summaries for

selecting images to transmit acoustically

demonstrate creation of low-bandwidth semantic

maps via cumulative image telemetry


1. Image Correction






underwater image formation


underwater robotic imaging platform assumptions

ignore gain

no natural light

assume white strobe

neglect spreading

no additive effects

ignore lens effects


log color channel means as a function of altitude


could additional sensors be useful for correction?

SeaBed - Autonomous

Underwater Vehicle

SeaSled – Towed

Camera System


use overlapping imagery to constrain equations

3 equations (RGB), 6 unknowns


use overlapping imagery to constrain equations

3 + 3 = 6 equations, same 6 unknowns!


attenuation coefficient estimates


beam pattern estimate in angular space


artifacts from artificial lighting and attenuation


comparison with our results

raw image

our method frame averaging

white balance (wb) homomorphic + wb

adapt. hist. eq.+ wb


sample corrected imagery


Demonstrated a multi-sensor fusion-based

approach to underwater image correction

Additional sensor information useful for correction,

but makes correction platform-dependent

[Rock et al., unpublished] [Kaeli et al., 2011] [Bryson et al., 2012]

Potential for historical water quality from past data

1. Image Correction

summary, conclusions, future work


1. Image Correction






Convolution is still a major bottleneck in many

multi-scale image processing frameworks such as

fast keypoint detection [Calonder et al., 2010]

Graphics Processing Units (GPUs) have increased

“brute force” computing power [Loomis 2011]

Can we exploit pixel grid geometries that allow us to

substitute adds and bit shifts for multiples while still

approximating a Gaussian? [Viola et al., 2001]

Applications on low-power robotic imaging platforms


the notion of “scale” is synonymous with blurring

Convolution can be thought as locally re-distributing

a signal based on some “kernel” or distribution

Place kernel 𝐾 𝑥 wherever 𝑆 𝑥 is, weight by 𝑆 𝑥 ,

them sum resulting functions

∗ = 𝑆(𝑥) ∗ 𝐾(𝑥) 𝐾(𝑥) 𝑆(𝑥)


the notion of a “scale-space” is a family of images

parameterized by a one-dimensional scale factor [Koenderink 1984] [Babaud et al., 1986]

continuous scale-space [Lindeberg 1994]

discrete pyramids [Burt & Adelson, 1983]


hierarchical discrete correlation (HDC) [Burt 1981]


octagonal pyramid


comparison of effective kernel distributions

𝜎 =1

2 𝜎 = 1 𝜎 = 2 𝜎 = 2 𝜎 = 8 𝜎 = 128


comparison of effective kernel distributions


octagonal pyramid decomposition

computed in 3P adds

and P bit shifts

traditional pyramids

with 5x5 separable

kernels use 3.3P

multiplies, 2.7 adds

for same 2 scale

resolution, 8 times

more operations


invariant features for underwater images




RAW IMAGERY

PRE-PROCESSING

SELECTION

COMPRESSION

TELEMETRY

UNDERSTANDING

{ [Murphy 2012,

PhD Thesis]

Chap. 1,2

Chap. 2,3

Chap. 3




introduce novel multi-scale image processing

framework for efficient feature computations

log opponent colors for illumination invariance,

partial correction of attenuation

future work in spatially-varying white balance

[Hsu et al., 2008]


1. Image Correction






create an image “fingerprint” based on histograms

of quantized keypoint descriptors [Sivic & Zisserman 2006]


create an image “fingerprint” based on histograms

of quantized keypoint descriptors [Lowe 2004]


keypoint detection using extrema of the difference

of Gaussian function [Lowe 2004]


keypoint detection using extrema of the difference

of Gaussian function


both methods detect similar keypoints


Quantized Accumulated Histogram of Oriented

Gradients (QuAHOG) around each keypoint


possible QuAHOG patterns

[Ojala et al., 2002]


keypoints detected and described with QuAHOGs




RAW IMAGERY

PRE-PROCESSING

SELECTION

COMPRESSION

TELEMETRY

UNDERSTANDING

{ [Murphy 2012,

PhD Thesis]

Chap. 1,2

Chap. 2,3

Chap. 3


surprise-based online navigation summaries

[Girdhard & Dudek, 2010, 2012]

[Itti & Baldi,2005]

Kullback-Leibler divergence


the threshold for “surprise” grows as the vehicle

experiences more of the world


modified navigation summaries for semantic maps

wait for surprise threshold to stabilize before

selecting first image to transmit (most represented)

once transmitted, a summary image should become

a static member of the summary set

non-summary images are attributed a summary

image to represent them

never remove summary images, merge with nearest

summary image, represent with more represented


selecting which summary image to transmit

semantic mapping via cumulative image telemetry


data collected by SeaBED AUV

in 2003 off Stellwagen Bank

can transmit approximately once

every 300 images

summary size set to 16,

approximately twice as many

images as will be sent


semantic mapping with heuristic merging


post-mission analysis of entire summary set


post-mission analysis of class membership




RAW IMAGERY

PRE-PROCESSING

SELECTION

COMPRESSION

TELEMETRY

UNDERSTANDING

{ [Murphy 2012,

PhD Thesis]

Chap. 1,2

Chap. 2,3

Chap. 3


nDeployments ≥ nRecoveries… LOU ∞

0

2

4

6

8

10

12

14

16

18

20

Deployments

Recoveries

[Kimball 2013]





propose lightweight keypoint detection and

description scheme for use in scene classification

modify surprise-based navigation summaries for

selecting images to transmit acoustically

demonstrate creation of low-bandwidth semantic

maps via cumulative image telemetry



we have proposed a realistic framework for

reducing the latency of understanding paradigm

in autonomous underwater robotics

future implementations in real AUV missions

more robust keypoint detectors/descriptors

“AUV Pandora” for adaptive mission planning


acknowledgements


questions?

Understanding Underwater Optical Image DatasetsOptical Modems: 1,000,000 bits/s at 100 meters [Farr...

Documents

Transcript of Understanding Underwater Optical Image DatasetsOptical Modems: 1,000,000 bits/s at 100 meters [Farr...