MASTER: Mobile Autonomous Scientist for...
Transcript of MASTER: Mobile Autonomous Scientist for...
![Page 1: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/1.jpg)
MASTER: Mobile Autonomous Scientist
for Terrestrial and Extra-terrestrial Research
Dr. Iain Wallace, Dr. Mark Woods
Autonomous & Intelligent Systems Group
Wednesday 13th May 2015
![Page 2: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/2.jpg)
2
What is MASTER?
Why do we need it?
![Page 3: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/3.jpg)
3
NASA/JPL
Missing the Obvious ?
Retracing its steps
![Page 4: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/4.jpg)
4
Why are SCISYS doing this work?
![Page 5: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/5.jpg)
5
Science Autonomy
Heritage
![Page 6: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/6.jpg)
6
![Page 7: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/7.jpg)
7
Neuro-Fuzzy, Computer Vision Based Science Classification
Used FL flight code
![Page 8: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/8.jpg)
8
Scene Understanding
Autonomously detect
what’s interesting to
scientists
Autonomously and
safely work out how
to gather more data
We have a basis for
an autonomous
scientific – robotic
“apprentice”
+ Tactical Planning = Closed loop science
Fusing Technologies
![Page 9: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/9.jpg)
9
STOP-PRESS: ProviScout win in FP 7
CREST – Autonomous Robotic Scientist Showed that autonomous, science data acquisition/investigation was feasible
![Page 10: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/10.jpg)
10
WP0
WP1
WP2
WP4Out Crop
Unusual
Material
Unusual
MaterialInteresting Stop
WP3
Full Autonomy Trials in Etna, Iceland, or Tenerife
![Page 11: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/11.jpg)
11
Step Back Need for fundamental approach and review
![Page 12: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/12.jpg)
12
Objectives • Conceive a generic novelty detection system architecture
• Test with Planetary Science and EO examples
• But this is an open research problem
• Currently several algorithmic choices for detection pipeline
steps but which ones work best?
• MASTER is GSP Early TRL so we:
• built the prototype architecture implementation
• and a methodology to evaluate the algorithmic choices –
• System allows you to use alternate algs. for different steps
and therefore compare
![Page 13: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/13.jpg)
13
MASTER Team • Prime Contractor, Req’s Definition, System Design
Development and Test – SCISYS
• State of the Art Literature Review – MRG Uni. of
Oxford
• Planetary Science Dataset – Uni. of Leicester
• Earth Observation Dataset – Pixalytics
![Page 14: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/14.jpg)
14
Autonomy and Robotics Group Scope
Robotic Platform + low-level software
Payload
Insts.
Operators
Autonomous Navigation
Scene Analysis
Mission Planning
Autonomy Framework
Rob
otic A
RM
Mission Planning
3D Data Understanding
Data
Remote Operations
![Page 15: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/15.jpg)
The MASTER Problem
• Can you spot what’s unusual in this image?
1 image
~10cm
traverse
~2.5s
operations
![Page 16: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/16.jpg)
The MASTER Problem
• Can you spot it now?
• …and you know what you’re looking for.
9 images
~1m traverse
~25s
operations
![Page 17: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/17.jpg)
The MASTER Problem
• Can you spot it now?
25 images
~2.5m
traverse
~1 min
operations
![Page 18: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/18.jpg)
The MASTER Problem
• Can you spot it now?
100 images
~10m
traverse
~5 mins
operations
![Page 19: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/19.jpg)
The MASTER Problem
• Miss anything else interesting?
• 250 images, ~25m traverse, ~15 mins operations
• How many can be saved/returned?
• How do you choose?
![Page 20: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/20.jpg)
• A domain independent software architecture for
detecting scientific events in images.
» This was narrowed to detecting (and defining) novelty.
• Capturing expert scientist’s knowledge.
• Two test domains
» Earth Observation
» Planetary Science
• System output is location and extent of regions of
interest.
• Project output is a detailed analysis of the problem
and properties of the presented solution.
MASTER Objectives
![Page 21: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/21.jpg)
• Expected – This case corresponds to easily classified phenomena seen in training.
• Unusual – Phenomena that can be classified, but an expert has somehow indicated would be unusual.
• Unexpected – The phenomena primarily applied to different contexts in training.
• Classified Novelty – Reasonably probable the classifier has identified the phenomena, but it is a statistical outlier.
• Unclassified Novelty – Where the output is above the threshold to create a new class, this is “unclassified novelty”.
Classes of Novelty N
ovel
Not
Novel
![Page 22: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/22.jpg)
• NASA/JPL Photojournal Planetary Image archive
Test Datasets - PS
![Page 23: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/23.jpg)
• Collected during ESA SAFER trial.
Test Datasets – PS SAFER
![Page 24: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/24.jpg)
• Rover NAVCAM from ESA Seeker trial
Test Datasets - Seeker
![Page 25: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/25.jpg)
• Processed MERIS Level 1 data
Test Datasets - EO
![Page 26: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/26.jpg)
System Overview
![Page 27: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/27.jpg)
Saliency
Example Saliency Maps
Left-to-right: Original image, scale-space saliency map, image-signature saliency map.
• What’s outstanding in this image?
» Looking for local interest.
• Many algorithms evaluated
» Complete system can select from a library.
![Page 28: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/28.jpg)
• Performance varies by domain » Varies by phenomena of interest too.
• SAFER Dataset performance appears poor, but this is a relative measure not absolute.
» Reflects the relatively sparse labelling.
• Seeker data also exhibited very tight error bounds – due to very homogenous data.
• Operating point selection important, as saliency represents a bound on performance.
Saliency Evaluation Conclusions
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
SK Pssafer PS EO
AUC
![Page 29: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/29.jpg)
• Well-proven HOG/Visual Words/Spatial histogram based features » Others also tested, ICF, ACF…
• SVM Classifiers » libSVM implementation
» One classifier per class
» Trained “one versus the rest”
• Parameter selection, then training on complete training set.
Classifiers
0.8
classifier
Feature Vector
![Page 30: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/30.jpg)
“On average, they work most of the time”
Summary of Classifier Performance
![Page 31: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/31.jpg)
• Performance on the whole good.
» Generality exceeded our expectations.
• Best performance with homogeneity between test
and training.
» As seen in SAFER and Seeker data.
• More training examples improve performance.
» 25 samples a sensible lower bound in homogenous data.
• Classes should be well defined.
» Poor relative performance of “roi” and “flare” classes.
Classification Conclusions
![Page 32: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/32.jpg)
• Simple Novelty
» If the most confident classifier is above a threshold, assign
a not-novel class according to training frequency.
Otherwise Unclassified Novelty.
• Group Novelty Classification
» Similar to the above, but considers also many confident
classes (confusion) to be unclassified novelty, and defines
classified novelty as a large margin to an uncertain class
or many unsure classes.
Novelty Detection
![Page 33: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/33.jpg)
• What is novelty, in relation to the training data?
• How important is the detection of novelty in an image?
• How important is identifying the location and extent of
any novelty?
• How desirable is it to classify regions of interest that are
not novel?
• What is the appropriate trade-off between detecting
novelty, and the cost of false positives?
• Is the problem one of selection – “is this image novel?”
• Is the problem one of ranking – “which is the most novel
image?”
System Evaluation
![Page 34: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/34.jpg)
Varying Thresholds
0
0.2
0.4
0.6
0.8
1
0 0.5 1
Recall
False-Discovery Rate
Varying Novelty
0
0.2
0.4
0.6
0.8
1
0 0.5 1
Recall
False-Discovery Rate
Varying Saliency
![Page 35: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/35.jpg)
Labelling – Future Possibilities
![Page 36: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/36.jpg)
Carnot Discovery - Saliency
![Page 37: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/37.jpg)
UKSA Chameleon - Classifiers
![Page 38: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/38.jpg)
UKSA Chameleon - Performance
Data product or Test
average time from
command to capture
(seconds) std. dev
CPU
consumption
(watts) data product size (Mb)
sensor power
consumption
(hardware)
memory usage
(component
specified, Mb) notes
energy cost over
idle overseer
(joules, rough)
idle system na na 6.15 0 - n/a
running overseer, no
sensors or data capture,
average over 10 minutes na na 11.39 13.1Mb in 253s -
generating map xtion m0 0.047 10.8 -0.02773
imu 0.00009 0.00004 11.66 0.00008 42.96875 20Hz 0.0000243
SSC Mode 1 rectified
images PAIR 0.0088 12.91 0.3 - 86.91894531 15Hz 0.013376
hokuyo to PC 0.12 0.014 11.69 0.009 - 43.45703125 20Hz 0.036
XB3 rectified images 0.012 0.021 20.48 0.81 - 166.1591797 15Hz commanded 0.10908
map to occupancy grid
(includes dtm) 0.007 0.005 28.05 0.05 n/a 87.43457031 SSC 1 pose 5Hz, PC 0.5hz ROR, Mapping 0.5hz 1.5 dem refresh, 3.3m drive0.11662
SSC Mode 7 rectified images 0.022 17.08 1.12 - 15Hz 0.12518
Xtion QVGA@60Hz PC 0.017 30.22 0.69 - 50.02734375 mode 2, 60hz, SOR filtered 0.32011
XB3 PC 0.12 14.71 2.6 - 196.0117188 1Hz 0.3984
Xtion VGA@30Hz PC 0.031 25.65 4.6 - 70.484375 30hz, mode 0, SOR filtered 0.44206
SSC Mode 1 VO estimate 0.059 26.05 0.0003 - 92.69140625
15Hz
still camera on tripod 0.86494
teval 0.069 0.0029 24.08 0.0001 85.43066406 SSC mode1, 15Hz 0.87561
XB3 VO estimate 0.057 0.0056 27.94 0.0003 - 176.2138672 15Hz commanded 0.94335
SSC Mode 1 PC no filtering 0.3 15.77 7 - 121.6806641 1Hz (commanded) 1.314
SSC Mode 7 VO estimate 0.13 29.1 0.0003 - 142.0976563 15Hz 2.3023
mapping eval 0.16 29.57 0.0002 148.4599609 eval 1Hz, SSC mode1 for PC 1Hz no filter 2.9088
generating map SSC1 0.55 24.3 0.175 SSC 1Hz PC no filter, Mapping 1Hz 7.1005
generating map SSC7 1.11 35.3 0.14 SSC 1Hz PC no filter, Mapping 1Hz 26.5401
XB3 PC SOR filter 2 27.01 2.3 - 200.6005859 1Hz 31.24
XB3 PC ROR filter 3.1 27.33 2.6 - 199.5898438 1Hz 49.414
SSC Mode 7 PC 3.2 28.77 26 - 228.1054688 1Hz 55.616
SSC Mode 1 PC SOR 5.6 26.73 6.4 - 156.6689453 1Hz 85.904
SSC Mode 1 PC ROR 8.6 26.78 7 - 157.4609375 1Hz 132.354
SSC Mode 7 PC SOR filter 24.2 27.42 26 - 391.0273438 1Hz 387.926
SSC Mode 7 PC ROR filter 142 27.75 26 - 412.1826172 1Hz 2323.12
![Page 39: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/39.jpg)
• General classifiers show good performance
• Visual saliency shows good utility.
• Evaluation of novelty detection is not simple.
• Labelling and annotation is a big deal.
» Several good datasets created.
• Future application performance baselines have
implications:
» Where all images are inspected – e.g. labelling
» Selecting a few images for downlink
» No images are inspected – e.g. navcams.
• Evaluation is slow, execution is fast.
General Conclusions
![Page 40: MASTER: Mobile Autonomous Scientist for …robotics.estec.esa.int/ASTRA/Astra2015/Presentations/Session 7B...for Terrestrial and Extra-terrestrial Research ... • System allows you](https://reader033.fdocuments.net/reader033/viewer/2022051723/5ab6f4767f8b9adc638ea06c/html5/thumbnails/40.jpg)
Thank you [email protected] www.scisys.co.uk
See our rovers in the exhibition area this morning! We are recruiting for autonomy R&D engineers.