Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D....
-
date post
20-Dec-2015 -
Category
Documents
-
view
226 -
download
2
Transcript of Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D....
![Page 1: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/1.jpg)
Closing the Loop for ISPusing Performance Prediction
Dec-05
Greg Arnold, [email protected]
Sensors Directorate
Air Force Research LaboratoryAFRL/SNAT, Bldg 620; 2241 Avionics Circle
WPAFB OH 45433-7321; (937) 255-1115x4388
![Page 2: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/2.jpg)
2
Trilogy of Thoughts / Goals
• Playground- Urban SASO (Security & Stability Ops)• ISP- context is UAV swarms & S-S fusion
– Need multiple sensors– Confirmatory Sensing and Interrogation– Anomaly detection & backtracking– Understand the problem
• Active Vision- manipulate the sensor to improve perf– Offline: ATR-driven sensing– Online: Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!– Reasoning in 3D (requires metrics)– Images are samples from world– General => Specific for robustness
Uninhabited Air Vehicle
![Page 3: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/3.jpg)
3
Target RecognitionLevels of Discrimination
• Detection: the level at which targets are distinguished from non-targets, i.e., clutter objects such as trees, rocks, or image processing artifacts.
• Classification: the level at which target class is resolved, e.g., building, vehicle, or aircraft.
• Recognition: the level at which target subclass is determined, e.g., for a tracked vehicle, tank, APC, or ADU.
• Identification: the level at which the model/make of a target is resolved, e.g., for a tank: M60, M1, or T72.
• Fingerprint: the serial number of a particular instance of a target, i.e. Vince’s Caravan vs. Lori’s Caravan.
Coarse
Fine
![Page 4: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/4.jpg)
4
Target RecognitionLevels of Automation
• Interactive decision aid: – Human and machine work interactively
• Automatic decision aid: – Machine is autonomous from input of data to
output to human– Human makes final decision (Human-in-the-loop)
• Autonomous system: – Machine makes the final decision– Human is NOT in the loop
![Page 5: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/5.jpg)
5
What Does ATR Mean?
Community Function Execution Speed
Intelligence Assist a Human Interpret Data*
Non Real-Time (weeks – years) (hours)
Surveillance Assist a Human Recognize Objects
Non Real-Time (hours-days)
Targeting/Fire Control
Assist a Human Recognize Target
Non Real-Time (seconds – minutes)
Autonomous Guided Munition
Autonomously Acquire Target
Real-Time (milliseconds)
Battle Damage Assessment
Assist a Human Evaluating Effects
Sooner the Better (milliseconds-days)
![Page 6: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/6.jpg)
6
Found Something
Go Get It
ISR Goals
ISR Goals• No Sanctuary• Persistent (PISR)• All Weather• Day / Night• All Terrain (city, country,
forest, desert, ocean)
• Moving & Stationary• Safety !!!
Intelligence, Surveillance, and Reconnaissance
ShotKill Chain
Hit It!
![Page 7: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/7.jpg)
7
Automated Target Recognition (ATR) Insights
• Information Limited: Believe current performance is information limited– Human (Data >> Information)
• Pixels/Pupils ratio• Better SNR, resolution, modalities
– Machine• False Alarms (Google Search)• Finer Discrim./Obscuration (>> higher resolution)
• 3-D: Intuitively understand geometric (3-D) information
• UAVs: UAVs transform the CID problem!
![Page 8: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/8.jpg)
8
Sensors Directorate Structure
• SN Directorate– SNA: Sensor ATR Technology– SNJ: Light (EO) Sensors– SNR: Radio (RF) Sensors– SNZ: Applications
• SNA Division: Mike Bryant, Lori Westerkamp (Ed Zelnio)– SNAA: Evaluation– SNAR: Applications– SNAS: Modeling and Signatures– SNAT: Innovative Algorithms
• SNAT Branch: Dale Nelson, Rob Williams– Generation After Next Technologies & Algorithms (Greg
Arnold)– Tracking and Registration (Devert Wicker)– Vigilance (Kevin Priddy)
![Page 9: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/9.jpg)
9
ATR Thrust Scope
GEOLOCATE
FIND
TRACK
ID M60
FU
NC
TIO
NS
SENSORS
MATURATION PROCESSAlgorith
ms
Signatures
Assessment0 100
1
PD
FAR
![Page 10: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/10.jpg)
10
Challenge ProblemsStandard MetricsATR Theory
Characterized PerformanceHigh Performance ComputingOperational Databases
ASSESSMENT& FOUNDATION
Sensor Data Management System (SDMS)
ATR Thrust Approach - Subthrusts
INNOVATIVE ALGORITHMS
FIND FIX TRACK & ID
Phenomenology ExplorationEM ModelingSynthetic Data
Operational Target Models/Databases
SIGNATURES & MODELING
SignatureCenter
5
10
15
20
10
20
30
40
02.5
5
7.5
10
5
10
15
20
![Page 11: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/11.jpg)
11
![Page 12: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/12.jpg)
12
3-D ATR− 3-D Imaging for RF Floodlight− 3-D for urban context− ATR Theory challenge problem
ATRC
ap
abili
ty /
Dif
fic
ult
y
Near Mid Far
Spiral D
evelopment
We need a generalized pattern recognition capability that will classify things previously unseen, actively manage assets, and predict the intent and actions of combatants.
Adaptive ATR − On-the-fly modeling / reacquisition− Reasoning with uncertainty− Adaptive metrics derived from user
ATR for Anticipatory ISR − Multi-X fusion for PISR/TST− Dynamic GIG Sensor Management− ATR Theory for Anticipation
Goals
![Page 13: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/13.jpg)
13
Assumptions / BeliefsBackground / Framework
• Must Define Problem & EOC’s– Whether or not applying model-based vision– Necessary for testing algorithm capabilities
• Model-Based Vision– More than just CAD models– Characterization of the data
and the system at some level– “If I can’t model it, I don’t understand it”
• Physics-Based Vision– What can we do before appealing to statistics
Real World Dimensionality
DataModel
s
![Page 14: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/14.jpg)
14
Operating Conditions (OCs)
TargetsTargetsSensorsSensors
InteractionsInteractions
EnvironmentEnvironment
OCs: Everything that changes the sensor response. Most OCs have infinite variation
![Page 15: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/15.jpg)
15
Real world variability:Extended Operating Conditions (EOCs)
. . .
20 Target Types
6 DOFPose
Squint & Depression
AngleArticulation
x
y
zObscuration
Variants
Billions20 6524
23
22
61 QQQQQQN Billions20 65
24
23
22
61 QQQQQQN
Configuration
![Page 16: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/16.jpg)
16
Discrimination vs. Robustness
Data Models
Robustness
Disc
rimin
atio
n
Seria
l #
Sam
e Se
nsor
Sens
or T
ype
Synt
hetic
Shape
Reflectivity
Tuni
ng
MSEMSE
Quantized
Quantized
Metrics
Metrics
Binary
Binary
Shape
Shape
Points
Points
![Page 17: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/17.jpg)
17
Using Information More Effectively
More Information
ATR Driven Sensing
Multisensor Approaches
Bio-Inspired Adaptive ATR
Challenge Space
1717
![Page 18: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/18.jpg)
18
Trilogy of Thoughts / Goals
• Playground- Urban SASO• ISP- context is UAV swarms & S-S fusion
– Need multiple sensors– Confirmatory Sensing and Interrogation– Anomaly detection & backtracking– Understand the problem
• Active Vision- manipulate the sensor to improve perf– Offline: ATR-driven sensing– Online: Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!– Reasoning in 3D (requires metrics)– Images are samples from world– General => Specific for robustness
![Page 19: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/19.jpg)
19
Vertically Integrated Sensor Exploitation for Generalized Recce & Instant Prosecution
(VISEGRIP)
![Page 20: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/20.jpg)
20
TheoryTheory
Goal: Quantify the accuracy, completeness, & relevance of information with demonstrable authority. Challenge problem: support counter-WMD
Objective: Theory & algorithm research to incorporate ATR Theory principles into Sensor Mgmt infrastructure modified to implement confirmatory sensing & interrogation
Payoff: Pattern Recognition discipline that is more expressive to assure users that source is authoritative and information is “actionable”
Confirmatory Sensing & Interrogation
AlgorithmsAlgorithms
BackgroundBackground
ATR Theory•Aims to design and predict performance of sensor data exploitation systems•Includes all forms of sensor data exploitation i.e. target detection, tracking, recognition, and fusion
Information TheoryStudies the collection and manipulation of information
Query Generation What question to ask
Query Processing When, How, & Who to ask
Data Fusion Align redundant informationAssess unique or contradictory informationAssimilate valuable information
Evidence Assessment Quantify accuracy and completeness of assertionsPredict a window of opportunity
![Page 21: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/21.jpg)
21
![Page 22: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/22.jpg)
22
• Playground- Urban SASO• ISP- context is UAV swarms & S-S fusion
– Need multiple sensors– Confirmatory Sensing and Interrogation– Anomaly detection & backtracking– Understand the problem
• Active Vision- manipulate the sensor to improve perf– Offline: ATR-driven sensing– Online: Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!– Reasoning in 3D (requires metrics)– Images are samples from world– General => Specific for robustness
Trilogy of Thoughts / Goals
![Page 23: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/23.jpg)
23
ATR-Driven Sensing
Cueing, Prioritization for the Human
~1’10’ ~1’
Short Circuits
Resonant Cross Slots Cavity
Low Dielectric/Low Loss Face Sheet
![Page 24: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/24.jpg)
24
• Playground- Urban SASO• ISP- context is UAV swarms & S-S fusion
– Need multiple sensors– Confirmatory Sensing and Interrogation– Anomaly detection & backtracking– Understand the problem
• Active Vision- manipulate the sensor to improve perf– Offline: ATR-driven sensing– Online: Time-reversal. Active filter. Gotcha,
• ATR Theory- performance prediction is the key!– Reasoning in 3D (requires metrics)– Images are samples from world– General => Specific for robustness
Trilogy of Thoughts / Goals
Uninhabited Air Vehicle
![Page 25: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/25.jpg)
25
What is your Objective Function?
• L-p (L-1, L-2, L-infinity)• Diffusion Distance• Hausdorff• Chamfer• Ali-Silvey• Earth Movers Distance• Chi Squared• Entropy• Kullback-Liebler• Mutual Information• Maximum Likelihood• Renyi
• Pd (Prob. of Detection)• Pcc (correct classification)• Pe (Prob. of Error)• Pfa (False Alarm)• Confusion Matrix• Precision• Recall• ROC Curve
![Page 26: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/26.jpg)
26
‘Clear Box’ View of ATR
0 100
1
PD
FAR
Environment
DetectDetectTrackTrack
GeolocateGeolocateIDID
SensorTargetATR
DecisionsHuman
Decisions
TrainedFeatures Templates Models
Target Knowledge
FeatureExtractor
Discriminator
DecisionRule
Target Knowledge
Target Models &Database
![Page 27: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/27.jpg)
27
ATR/Fusion Processes
Target Models &DatabaseSensor
Model
Environment
DetectDetectTrackTrack
GeolocateGeolocateIDID
Sensor(s)TargetATR
DecisionsHuman
Decisions
Sensor Management
Registration
Environment Model
Performance Model
Adaptation
Behavior Models
Anticipate
![Page 28: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/28.jpg)
28
Performance Model is the Lynchpin
• ATR System is dependent on the Performance Model• Need performance prediction
– Determine where / when to use sensors– Estimate effectiveness of sensors for given task
– Sensor Management– Registration– Learning– …
![Page 29: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/29.jpg)
29
If Somebody Asks…
• Typical DARPA question– “Is it physically possible to do X?”– We’ve invested $K and achieved P% performance, is it
worth investing more?
• Examples– How likely are we to detect a dismount with an HSI
system with 1m spatial resolution? 1ft? 1in?– We spent $40M and achieved 80% of perfection.
Have we reached the knee in the performance curve?– Organization X says it can build a system to do Y.
Does this violate physics?
![Page 30: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/30.jpg)
30
Aspects of ATR Theory Objectives
• Measure the information content of sensor imagery
• Given a set of data and a MOP, determine attainable performance range
• What are the critical design constraints to achieve a desired outcome, using this data?
• Estimate exploitation level of available information
• Establish “feedback loop” between ATR designers and sensor developers
• What are the critical design constraints to achieve a desired outcome at a particular level of confidence?
• Information gain from using models and data adaptively (learning)
• Determine theoretical upper bound on performance of given ATR
• Given an ATR system and a set of data, determine how much information can be exploited
• Determine how close a given system comes to achieving the optimal bound
• What are the critical design constraints to achieve a desired outcome, using this sensor and algorithm?
• What was the benefit of adding ‘this’ (additional data/processing)?
Data Assessment Design System Evaluation
![Page 31: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/31.jpg)
31
Problem Simplification
• Having said all that, let’s examine a problem for which we have some intuition
• 4 or 5 points undergoing rotation, translation, and maybe scale and skew
• 1-D, 2-D, and 3-D• Understand the projection from world to sensor
![Page 32: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/32.jpg)
32
What is Shape?
• Pose and scale invariant, coordinate independent characterization of an arrangement of features.
• Residual geometric relationships that remain between features after “mod-ing out” the transformation group action.
• Captured by a “shape space” where each distinct configuration of features (up to transformation) is represented by a single point.
![Page 33: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/33.jpg)
33
Beyond Invariants
Invariants +
Projection
Object-Image Relations
![Page 34: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/34.jpg)
34
Generalized Weak Perspective
• Projection model applicable to optical images(pinhole camera)
• Approximates full perspective for objects in ‘far field’
• Affine transformations on 3-space, and in the image plane (2-space)
• Denoted GWP
![Page 35: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/35.jpg)
35
Affine Transformations
• In 3D
110003333231
2232221
1131211
z
y
x
taaa
taaa
taaa
(Rotate, Scale, Skew | Translate) (3-D Point)
![Page 36: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/36.jpg)
36
GWP Projection3D to 2D
11...111
...
...
1321
1321
nn
nn
vvvvv
uuuuu
11...111
...
...
...
1000 1321
1321
1321
4
4
3
3
2
2
1
1
nn
nn
nn
zzzzz
yyyyy
xxxxx
b
a
b
a
b
a
b
a
Image Projection Object
![Page 37: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/37.jpg)
37
Object-Image Relation Motivation
Object 1
Object 2Image 1
Image 2
• Image 1 is not equivalent to Image 2 (in 2-D)
• Object 1 is not equivalent to Object 2 (in 3-D)
![Page 38: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/38.jpg)
38
Object - Image Relations Concept
“The relation between objects and imagesexpressed independent of the camera parameters
and transformation group”
(1) Write out the camera equations (geo or photo)
(2) Eliminate the group & camera parameters(3) Recognize the result as a relation between
the object and image invariants.
But pure elimination is VERY difficult even for polynomials.
![Page 39: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/39.jpg)
39
Weak PerspectiveObject - Image Relations
(Generalized)
• Parallel things remain parallel• The object size is 1/10 the distance from the camera• (Standard Position Method)
![Page 40: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/40.jpg)
40
Weak Perspective Camera
3-D Model Pi={xi,yi,zi} N-points (3N DOF)Rotate,Translate,Scale,Shear (12 Constraints)
3N-12 Absolute Invariants
2-D Image qi={ui,vi} N-points (2N DOF)Rotate,Translate,Scale,Shear (6 Constraints)
2N-6 Absolute Invariants
Camera ModelN-points (2N DOF)Union 2-D & 3-D (8 Constraints)
2N-8 relations
Need 5 corresponded points (minimum)q1 q2
q3
q4 q5
P1 P2
P3
P4 P5
![Page 41: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/41.jpg)
41
||
||,
||
||,
||
||
4321
5432
4321
5431
4321
5321
PPPP
PPPP
PPPP
PPPP
PPPP
PPPP
3-D Invariants
P1 P2
P3
P4 P5 3-D Model Pi={xi,yi,zi,1} 5-pointsGL3+Translation (12 Constraints)
3N-12 Absolute Invariants
Invariant is a function of the Ratio of Determinants:
A useful standard position is:
11111100001000010
53
52
51
III
![Page 42: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/42.jpg)
42
||
||,
||
||,
||
||,
||
||
321
532
321
432
321
521
321
421
qqq
qqq
qqq
qqq
qqq
qqq
qqq
qqq
11111100010
5242
5141
iiii
2-D Invariants
2-D Image qi={ui,vi,1} 5-pointsGL2+Translation (6 Constraints)
2N-6 Absolute Invariants
Invariant is a function of the Ratio of Determinants:
A useful standard position is:
q1 q2
q3
q4 q5
![Page 43: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/43.jpg)
43
11111100001000010
1000010001
11111100010
53
52
51
42
41
5242
5141
III
ii
iiii
Object - Image RelationGeneralized Weak Perspective Camera
(2-D Standard Position) = (Camera Transform) (3-D Standard Position)
The camera transforms the first 4 object point to image points,the remaining points satisfy the object - image relation iff:
5342525253415151 ; IiIiIiIi
11111
1000
0100
0010
100011111
100
010
53
52
51
0232221
0131211
5242
5141
I
I
I
vttt
uttt
ii
ii
Eliminate camera transform parameters:
![Page 44: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/44.jpg)
44
Object-Image Relation Abstraction
All objects that could have
produced the image.
All images of the object.
x
uObject-Image
Relations
GX n /
Space ShapeObject 3 GU n ˆ/
Space Shape Image2
![Page 45: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/45.jpg)
45
GWP Shape Spaces
• The shape spaces in the GWP case are Grassmann manifolds
• In 3D– Gr(n-4,H) or dually the Schubert cycle of 4-planes in
Gr(4,n) which contain (1,….,1)– Manifold has dimension 3n-12
• In 2D– Gr(n-3,H) or dually the Schubert cycle of 3-planes in
Gr(3,n) which contain (1,….,1)– Manifold has dimension 2n-6
H is the subspace of n-space orthogonal to the vector (1,…,1)
![Page 46: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/46.jpg)
46
Why
• We associate to our object data, viewed as a linear transformation from n-space to 4-space, its null space K of dimension n-4.
• Likewise to our image data in 2D we associate the null space L of dimension n-3.
11...111
...
...
...
1321
1321
1321
nn
nn
nn
zzzzz
yyyyy
xxxxx
![Page 47: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/47.jpg)
47
Global Shape Coordinates
• Better than local invariants
• Come from an isometric embedding of the shape space in either Euclidean space or projective space.
• Matching expressed in these coordinates will gracefully degrade
![Page 48: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/48.jpg)
48
Example in GWP
• 3D, n = 5 feature points
• Global shape coordinates are the Plucker coordinates (or dual Plucker coordinates) of the 4xn object data matrix or the 3xn image data matrix.
1111
det],,,[,,,lkji
lkji
lkji
lkji zzzz
yyyy
xxxx
lkjiM
111
det],,[,, tsr
tsr
tsr vvv
uuu
tsrN
![Page 49: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/49.jpg)
49
Global Object-Image Relations
• General– If and only If conditions– Overdetermined set of equations
• GWP– To match, K must be contained in L (iff)– This incidence condition can be expressed in terms of the
global shape coordinates– For n=5, 10 (non-independent) relations that look like:
[1234][125]-[1235][124]+[1245][123]
Locally only 2 of the 10 are independent, because the locus V of matching pairs (object shape, image shape) in the 7 dimensional product space XxY has dimension 5, codimension 2.
![Page 50: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/50.jpg)
50
Beyond Object-Image Relations
Object-Image Relations
+
Matching
Object-Image Metrics
![Page 51: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/51.jpg)
51
• We intuitively know that if we want to measure something we need a metric… ATR is no different.
• How far apart are these points?
1. The triangle inequality provides efficient match searching
2. Reliable & predictable Unknowns rejection3. Theoretical performance prediction
Why Metrics?
![Page 52: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/52.jpg)
52
The Triangle Inequality Advantage
u: image, x*: prototype object, xk: object from group
Shape Space
u
Measure the distance from the image to each shape object?
Search the group iff the distance to the prototype is less than the sum of the max intragroup distance and noise threshold.
X*
X*
X*
u
Measure the distance from the image to each shape prototype!
Group 1
Group 2
Group 3
![Page 53: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/53.jpg)
53
Using the Triangle Inequality
*
*
noise*
Offline,
*
tMeasuremen
*
**
**
Threshold],[
],[max],[],[
],[],[],[
],[],[],[
x
d
jKj
k
kk
kk
dxud
xxdxudxud
xxdxudxud
xxdxudxud
x
Equivalent
Grouping
Decision
u: image, x*: prototype object, xk: object from group
Search the group iff the distance to the prototype is less than the sum of
the max intragroup distance and noise threshold.
XkX*
uShape Space
Reject beyond Thresholdnoise
![Page 54: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/54.jpg)
54
What are Shape and Distance?
• Shape: What is left after translation & rotation are removed (more generally, the group)
• This is the (Partial) Procrustes definition of distance– R represents rotation and T represents translation– Procrustes normalizes the size of the objects
2222111
,,,21 )ˆ()ˆ(inf),(
2211
TORTOROOdTRTR
p
![Page 55: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/55.jpg)
55
New Metrics?
Any ol’ metric just won’t do…
1. Invariant to translation & rotation of 3-D object (+ more)
2. Invariant to the camera projection (+ discretization)
• This leads to the concept of Object-Image Relations (O-IR’s)
• Incomplete
– O-IR’s are only surrogate metrics
– =0 iff the object and image features are consistent
• Object-Image Metrics satisfy all the metric properties
• Shape Space is NOT Euclidean!• There is some evidence that human similarity perception is not always
metric
![Page 56: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/56.jpg)
56
Metrics on the Shape Spaces
• How to compare objects to images!
• We want a natural shape matching metric– Invariant to transformations of the 3D or 2D data,– e.g. Rotations, translations, or scale of the object or image
• Generalize Weak Perspective– We use the natural Riemannian metric on the Grassmannian
to measure distances between object shapes and image shapes
– This involves the so called principal angles between subspaces and is easily computed from the original data matrices via QR decomposition and SVD.
![Page 57: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/57.jpg)
57
Object-Image Metrics
Two ways to compute an “object to image” distance
1. Object SpaceCompute the minimum distance in object space from the given object to the set of all objects capable of producing the given image
2. Image SpaceCompute the minimum distance from the given image to the set of all images produced by that object
![Page 58: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/58.jpg)
58
Object-Image Metrics & Duality
],[inf],[Obj xxdxud uEuxu
],[inf],[Img xExu
uudxudx
Duality Theorem: ],[],[ ImgObj xudxud
xu: all objects that could have produced the
image.
ux: all images of the object.
x
uObject-Image
Relations
Matching can (in principle)Matching can (in principle) be performed in either object be performed in either object or image space without loss of performance !or image space without loss of performance !
GX n /3 GU n ˆ/2Object Shape Space Image Shape Space
![Page 59: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/59.jpg)
59
Duality
• Theorem - with suitable normalization
These metrics are the same!
In the GWP case this distance turns out to
be the distance between two subspaces of
different dimension defined again by using
principal angles.
![Page 60: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/60.jpg)
60
Image Geodesics
•2 random images
•Geodesic between them
•Not Linear
•Not the projection of a line
•Not even coplanar•Geodesics on the this cone have the same length as the calculated image distance!
![Page 61: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/61.jpg)
61
Orthographic Shape Space3 Points in 1-D & 2-D
• 3 points modulo translation, rotation, reflection yields…
• 1-D: Surface of a 30o cone w/ axis along {1,1,1}
• 2-D: Interior of the cone
• {0,0,0} object @ origin• {a,a,b} objects partition cone• Scale: lines through origin
• Geodesics on the this cone have the same length as the calculated image distance!
-2
0
2
-2
0
2
-4
-3
-2
-1
0
-2
0
2
-2
0
2
Rotation on the ‘wrong’ side aboutthe centroid rotates the cone (isotropy condition).
![Page 62: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/62.jpg)
62
Object-Image Relations (1)
•Fix an Object
•Set of Images it can produce
•Always circumscribe the cone
•Not conic sections!
•Equilateral triangle produces
a slightly smaller circle
•‘Image’ produces line to origin
![Page 63: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/63.jpg)
63
Object-Image Relations (2)
• Fix an Image• Set of objects that could
produce the given image• “Bent over cone”• Touches along line through
origin and image• Eventually converges to the
cone surface– Large objects must be
nearly collinear to produce the image
![Page 64: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/64.jpg)
64
Epsilon Balls & Noise Analysis
• The set of all shapes of distance 1 from the given shape (image)
• The image + Gaussian IID noise added to each image point location(std. dev. 0.5)
• Still working the analytic model of noise in the shape space
![Page 65: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/65.jpg)
65
Intrinsic Separability
• How many different shapes can I hope to identify?– Shape space as a unit volume– Epsilon balls defined by metric– Noise balls generated by Gaussian noise
• 5 points in 3D (Generalized Weak Perspective)– Epsilon in [0, 0.73] (max radius of ball)– P(Random Shape in Epsilon Ball)=1.37*Epsilon
• Example: Epsilon=0.01– P(Random Shape in Epsilon Ball)=0.014– Requires as least 73 balls to cover shape space– (Could be more or less efficient coverings)– Separability on a gross level
![Page 66: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/66.jpg)
66
Summary
• Integration of Sensing and Processing• Active Vision• ATR Theory
• These are all connected & overlapping areas• Provides a rich field of problems and applications
ATR Theory
Active VisionISP
![Page 67: Closing the Loop for ISP using Performance Prediction Dec-05 Greg Arnold, Ph.D. Gregory.Arnold@wpafb.af.mil Sensors Directorate Air Force Research Laboratory.](https://reader036.fdocuments.net/reader036/viewer/2022062421/56649d4d5503460f94a2c4e3/html5/thumbnails/67.jpg)
67
Open Problems in Inf. Exploitation
• Open Problems: ATR Theory– Classification before Recognition before ID– Representation
• Modeling free forms spanning discrete-continuous• Uncertainty in models (i.e. target variability)?• Cross-sensor phenomenology (registration / fusion)• Correspondence
– Unmixing: automated methods for separating foreground / background in various scenarios (ID before segmentation or pose estimation)
– Intrinsic Separability• How objects separate in quotient space as a function of sensor• Confusers• Unknowns: How to separate knowns from unknowns
– (Object-Image) Metrics• Efficient Search of Large Databases • How to choose the metrics based on the expected noise model & the type of
quotient space derived from the choice of metric• Long Poles:
– Probabilities for Fusion / Reasoning with Uncertainty – Recognition By Components (including construction/decomposition) – Non-Gaussian, Nonlinear Analysis (i.e. most models and algorithms assume
these two properties) – Adaptive systems– How to modulate the prior probabilities on-the-fly