פברואר 1968, בסיום מעלה האיסיים
-
Upload
anika-combs -
Category
Documents
-
view
39 -
download
7
description
Transcript of פברואר 1968, בסיום מעלה האיסיים
, בסיום מעלה האיסיים1968פברואר , בסיום מסע למצדה1968פברואר
, עובדת )מסע גדנ"ע(1969פברואר , אודיטוריום פיסיקה, 1969יוני בית-בירם
Image Registration of Remotely Sensed Data
Nathan S. Netanyahu
Dept. of Computer Science, Bar-Ilan Universityand Center for Automation Research, University of Maryland
Collaborators:
Jacqueline Le Moigne NASA / Goddard Space Flight CenterDavid M. Mount University of Maryland Arlene A. Cole-Rhodes, Kisha L. Johnson Morgan State University, MarylandRoger D. Eastman Loyola College of MarylandArdeshir Goshtasby Wright State University, OhioJeffrey G. Masek, Jeffrey Morisette NASA / Goddard Space Flight CenterAntonio Plaza University of Extremadura, SpainHarold Stone NEC Research Institute )Ret.(Ilya Zavorin CACI International, MarylandShirley Barda, Boris Sherman Applied Materials, Inc., IsraelYair Kapach Bar-Ilan University
Akavia Memorial Lecture, Haifa Univ., 11.01.20073
What is Image Registration / Alignment / Matching?
The above image is rotated and shifted with respect to the left image.
Akavia Memorial Lecture, Haifa Univ., 11.01.20074
Motivation
• A crucial, fundamental step in image analysis tasks, where final information is obtained by the combination / integration of multiple data sources.
Akavia Memorial Lecture, Haifa Univ., 11.01.20075
Motivation / Applications
• Computer Vision )target localization, quality control, stereo matching(
• Medical Imaging )combining CT and MRI data, tumor growth monitoring, treatment verification(
• Remote Sensing )classification, environmental monitoring, change detection, image mosaicing, weather forecasting, integration into GIS(
Akavia Memorial Lecture, Haifa Univ., 11.01.20076
Application Examples
• Global Wafer Alignment
Akavia Memorial Lecture, Haifa Univ., 11.01.20077
Wafer Alignment (cont’d)
Courtesy: Shirley Barda and Boris Sherman, Applied Materials, Inc.
Akavia Memorial Lecture, Haifa Univ., 11.01.20078
Application Examples (cont’d)
• Integration of medical images
Registration of MR and PET images of the same person (courtesy: A. Goshtasby)
Akavia Memorial Lecture, Haifa Univ., 11.01.20079
Application Examples (cont’d)
• Change Detection
1975 2000
Satellite images of Dead Sea, United Nations Environment Programme (UNEP) website
Akavia Memorial Lecture, Haifa Univ., 11.01.200710
Change Detection (cont’d)
1990 2005
Satellite images of Amona hilltop, Peace Now website
Akavia Memorial Lecture, Haifa Univ., 11.01.200711
Change Detection (cont’d)
IKONOS images of Iran’s Bushehr nuclear plant, GlobalSecurity.org
Akavia Memorial Lecture, Haifa Univ., 11.01.200712
What is the “Big Deal”?
How do humans solve this?By matching control points, e.g., corners, high-curvature points.
Zitova and Flusser, IVC 2003
Akavia Memorial Lecture, Haifa Univ., 11.01.200713
Automatic Image Registration
• Books:– Medical Image Registration, J. Hajnal, D.J. Hawkes, and D. Hill )Eds.(, CRC 2001– Numerical Methods for Image Registration, J. Modersitzki, Oxford University Press 2004– 2-D and 3-D Image Registration, A. Goshtasby, Wiley 2005– Image Registration for Remote Sensing, J. LeMoigne, N.S. Netanyahu, and R.D. Eastman )Eds.(, Cambridge
University Press, in preparation.
• Surveys:– A Survey of Image Registration Techniques, ACM Comp. Surveys, L.G. Brown, 1992– A Survey of Medical Image Registration, Medical Image Analysis, J.B.A. Maintz and M.A. Viergever, 1998– Image Registration Methods: A Survey, Image and Vision Computing, B. Zitova and J. Flusser, 2003
• Sample Papers:– Image Sequence Enhancement Using Sub-pixel Displacements, CVPR, D. Keren, S. Peleg, and R. Brada, 1988– Improving Resolution by Image Registration, CVGIP, M. Irani and S. Peleg, 1991– Computing Correspondence Based on Regions and Invariants without Feature Extraction and Segmentation,
CVPR, C. Lee, D. Cooper, and D. Keren, 1993– Robust Multi-Image Sensor Image Alignment, ICCV, M. Irani and P. Anandan, 1998– Fast Block Motion Estimation Using Gray Code Kernels, Israel CV Workshop, Y. Moshe and H. Hel-Or, 2006– Image Matching Using Photometric Information, Israel CV Workshop, M. Kolomenkin and I. Shimshoni, 2006
Akavia Memorial Lecture, Haifa Univ., 11.01.200714
Automatic Image Registration Components
0. Preprocessing
– Image enhancement, cloud detection, region of interest masking
1. Feature extraction )control points(
– Corners, edges, wavelet coefficients, segments, regions, contours
2. Feature matching
– Spatial transformation )a priori knowledge(
– Similarity metric )correlation, mutual information, Hausdorff distance(
– Search strategy )global vs. local, multiresolution, optimization(
3. Resampling
I1
I2Tp
Akavia Memorial Lecture, Haifa Univ., 11.01.200715
Example of Image Registration Steps
Feature extraction
Feature matching
Resampling Registered images after transformation
Zitova and Flusser, IVC 2003
Akavia Memorial Lecture, Haifa Univ., 11.01.200716
Automatic Image Registrationfor Remote Sensing
• Sensor webs, constellation, and exploration
• Selected NASA Earth science missions
• Domain-dependent characteristics
Akavia Memorial Lecture, Haifa Univ., 11.01.200717
Automatic Multiple Source
IntegrationSatellite/Orbiter,and In-Situ Data
Planning andScheduling
Sensor Webs, Constellation, and Exploration
Intelligent Navigationand Decision Making
Akavia Memorial Lecture, Haifa Univ., 11.01.200718
0.1 0.4 0.5 0.6 1.0 1.3 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 15.0
Instrument Number of(Spat. Resol.) Channels
AVHRR (D) 5 Channels(1.1 km)
TRMM/VIRS 5 Channels(2 km)
Landsat4-MSS 4 Channels(80 m)
Landsat5&7-TM&ETM+(30 m) 7 Channels
Landsat7-Panchromatic(15m)
IRS-1 4 ChannelsLISS-I (73m) - LISS-2 (36.5m)
JERS-1 8 Channels(Ch1-4:18m; Ch5-8:24m)
SPOT-HRV Panchromatic(10m) 1 ChannelSpot-HRV Multispectral(20 m) 3 ChannelsMODIS 36 Channels(Ch1-2:250 m;3-7:500m;8-36:1km)
EO/1
ALI-MultiSpectr. 9 Channels (30m)
ALI-Panchrom. 1 Channel (10m)
Hyperion 220 Channels
(30m)
LAC 256 Channels
(250m)
IKONOS-Panchromatic
(1m) 1 Channel
IKONOS-MS 4 Channels (4m)
ASTER 14 Channels
(Ch1-3:15m;4-9:30m;10-14:90m)
CZCS 6 Channels(1 km)
SeaWiFS (D) 8 Channels(1.1 km)
TOVS-HIRS2 (D) 20 Channels
(15 km)
GOES 5 Channels(1 km:1, 4km:2,4&5, 8km:3)METEOSAT 3 Channels(V:2.5km,WV&IR:5km)
.
13.0 14.00.7 10.0 11.0 12.0
Near-IR Mid-IR Thermal-IR
1 2 4 5
2 3 4 5
1 2 3 4 5 7 6
1
1 2 3
21 3 4 5
1
3
1 2 3 4 5 6 7 8
20 19 18
17 to 13
12 11 10 9 8 7 to 1
UltraViolet
1 2 3 4 5
Visible IRWaterVapor
Visible
1 2 3 4
1 23
&465 7 8
1,13,14
2,16-19
3,8-10
11,4,12
5 6 7 20-2515 26 27 28 29 30 31 32 33-36
1 2 3 4
1
1 to 220
5'1' 1 2 3 4 5 7
1
1 to 256
1 2 3 4 5-9 10,11 12 13,14
1
1 2 3 4
Selected NASA Earth Science Missions
Akavia Memorial Lecture, Haifa Univ., 11.01.200719
MODIS Satellite System
From the NASA MODIS website
Akavia Memorial Lecture, Haifa Univ., 11.01.200720
MODIS Satellite Specifications
Orbit: 705 km, 10:30 a.m. descending node )Terra( or 1:30 p.m. ascending node )Aqua(, sun-synchronous, near-polar, circular
Scan Rate: 20.3 rpm, cross track
Swath Dimensions: 2330 km )cross track( by 10 km )along track at nadir(
Telescope: 17.78 cm diam. off-axis, afocal )collimated(, with intermediate field stop
Size: 1.0 x 1.6 x 1.0 m
Weight: 228.7 kg
Power: 162.5 W )single orbit average(
Data Rate: 10.6 Mbps )peak daytime(; 6.1 Mbps )orbital average(
Quantization: 12 bits
Spatial Resolution: 250 m )bands 1-2(500 m )bands 3-7(1000 m )bands 8-36(
Design Life: 6 years
Akavia Memorial Lecture, Haifa Univ., 11.01.200721
Landsat 7 Satellite System
New Orleans, before and after Katrina 2005 (from the USGS Landsat website)
Akavia Memorial Lecture, Haifa Univ., 11.01.200722
Landsat 7 Satellite Specifications
Launch
Date April 15, 1999
Vehicle Delta II
Site Vandenberg AFB
Orbit Characteristics
Reference system WRS-2
Type Sun-synchronous, near-polar
Altitude 705 km )438 mi(
Inclination 98.2°
Repeat cycle 16 days
Swath width 185 km )115 mi(
Equatorial crossing time 10:00 AM +/- 15 minutes
Akavia Memorial Lecture, Haifa Univ., 11.01.200723
IKONOS Satellite System
Akavia Memorial Lecture, Haifa Univ., 11.01.200724
IKONOS Satellite Specifications
Launch Date24 September 1999
Vandenberg Air Force Base, California, USA
Operational Life Over 7 years
Orbit 98.1 degree, sun synchronous
Speed on Orbit 7.5 kilometers per second
Speed Over the Ground 6.8 kilometers per second
Number of Revolutions Around the Earth 14.7 every 24 hours
Orbit Time Around the Earth 98 minutes
Altitude 681 kilometers
Resolution
Nadir:0.82 meters panchromatic3.2 meters multispectral26° Off-Nadir1.0 meter panchromatic4.0 meters multispectral
Image Swath11.3 kilometers at nadir
13.8 kilometers at 26° off-nadir
Equator Crossing Time Nominally 10:30 a.m. solar time
Revisit Time Approximately 3 days at 40° latitude
Dynamic Range 11-bits per pixel
Image Bands Panchromatic, blue, green, red, near IR
Akavia Memorial Lecture, Haifa Univ., 11.01.200725
Domain-Dependent Characteristics
• Very large images )~ 6200 x 5700 of typical Landsat 7 scene(
• Practically “flat”, 2D images
• Rigid/similar transformations
• A priori knowledge )e.g., small rotation and scale(
Akavia Memorial Lecture, Haifa Univ., 11.01.200726
Challenges in Processing of Remotely Sensed Data
• Multisource data• Multi-temporal data• Various spatial resolutions• Various spectral resolutions
• Subpixel accuracy• 1 pixel misregistration ≥ 50% error in NDVI classification
• Computational efficiency• Fast procedures for very large data sets
• Accuracy assessment• Synthetic data• “Ground Truth" )manual registration?(• Consistency )"circular" registrations( studies
Akavia Memorial Lecture, Haifa Univ., 11.01.200727
Fusion of Multi-temporal Images
Improvement of NDVI classification accuracy due to fusion of multi-temporal SAR and Landsat TM over farmland in The Netherlands (source: The Remote Sensing Tutorial by N.M. Short, Sr.)
Akavia Memorial Lecture, Haifa Univ., 11.01.200728
Integration of Multiresolution Sensors
Registration of Landsat ETM+ and IKONOS images over coastal VA and agricultural Konsa site
(source: LeMoigne et al., IGARSS 2003)
Akavia Memorial Lecture, Haifa Univ., 11.01.200729
Feature Extraction
Top 10% of wavelet coefficients )due to Simoncelli( of Landsat image over Washington, D.C. (source: N.S. Netanyahu, J. LeMoigne, and J.G. Masek, IEEE-TGRS, 2004)
Gray levels BPF wavelet coefficients Binary feature map
Akavia Memorial Lecture, Haifa Univ., 11.01.200730
Feature Extraction (cont’d)
Image features )extracted from two overlapping scenes over D.C.( to be matched
Akavia Memorial Lecture, Haifa Univ., 11.01.200731
Feature Matching / Transformations
• Given a reference image, I1(x, y), and a sensed image I2(x, y), find the
mapping )Tp, g( which “best” transforms I1 into I2, i.e.,
where Tp denotes spatial mapping and g denotes radiometric mapping.
• Spatial transformations:Translation, rigid, affine, projective, perspective, polynomial
• Radiometric transformations )resampling(:Nearest neighbor, bilinear, cubic convolution, spline
2 1( , ) ( ( ( , ), ( , ))),p pI x y g I T x y T x y
Akavia Memorial Lecture, Haifa Univ., 11.01.200732
Transformations (cont’d)
cos sin
sin cos
0 0 1
x
p y
s s t
T s s t
' cos sin
' sin cosx
y
x s x s y t
y s x s y t
Objective: Find parameters of a transformation Tp (consisting of a translation,
a rotation, and an isometric scale) that maximize similarity measure.
Akavia Memorial Lecture, Haifa Univ., 11.01.200733
Similarity Measures (cont’d)
• L2 norm:
Minimize sum of squared errors over overlapping subimage
• Normalized cross correlation (NCC):Maximize normalized correlation between the images
1 1 2 2
1 22 2
1 1 2 2
( , ) ( , )
( , )
( , ) ( , )
x y
x y x y
I x y I I x y I
NCC I I
I x y I I x y I
2
2 1pI T I
Akavia Memorial Lecture, Haifa Univ., 11.01.200734
Similarity Measures (cont’d)
• Mutual information (MI):Maximize the degree of dependence between the images
or using histograms, maximize
1 2
1 2
1 2 1 2
, 1 21 2 , 1 2
1 2
,, , log ,I I
I Ig g I I
p g gMI I I p g g
p g p g
1 2
1 2
1 2 1 2
, 1 21 2 , 1 2
1 2
,1, , log I I
I Ig g I I
Mh g gMI I I h g g
M h g h g
Akavia Memorial Lecture, Haifa Univ., 11.01.200735
Similarity Measures (cont’d), An Example
MI vs. L2-norm and NCC applied to Landsat 5 images
(source: Chen, Varshney, and Arora, IEEE-TGRS, 2003)
Akavia Memorial Lecture, Haifa Univ., 11.01.200736
Similarity Measures (cont’d), an MI Example
Source: Cole-Rhodes et al., IEEE-TIP, 2003
Akavia Memorial Lecture, Haifa Univ., 11.01.200737
Similarity Measures (cont’d)
• (Partial) Hausdorff distance (PHD):
where
1 1 2 21 2 1 2, min dist , ,th
K p I p IH I I K p p
11 K I
1K
2K
1K I
Akavia Memorial Lecture, Haifa Univ., 11.01.200738
Similarity Measures (cont’d), PHD Example
PHD-based matching of Landsat images over D.C. (source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004)
Akavia Memorial Lecture, Haifa Univ., 11.01.200739
Feature Matching / Search Strategy
• Exhaustive search
• Fast Fourier transform )FFT(
• Optimization )e.g., gradient descent; Thévenaz, Ruttimann, and Unser )TRU(, 1998; Spall, 1992(
• Robust feature matching )e.g., efficient subdivision and pruning of transformation space(
Akavia Memorial Lecture, Haifa Univ., 11.01.200740
Computational Efficiency
• Extraction of corresponding regions of interest )ROI(
• Hierarchical, pyramid-like approach
• Efficient search strategy
Akavia Memorial Lecture, Haifa Univ., 11.01.200741
Computational Efficiency (cont’d), ROI Extraction
UTM of 4 scene corners known from systematic correction
Input Scene
1. Extract reference chips and corresponding input windows using mathematical morphology
2. Register each )chip-window( pair and record pairs of registered chip corners )refinement step(
3. Compute global registration from multiple local ones
4. Compute correct UTM of 4 scene corners of input scene
Reference Scene
Advantages:• Eliminates need for chip database• Cloud detection can easily be included in process• Process any size images• Initial registration closer to optimal registration => reduces computation time and increases accuracy.
Source: Plaza, LeMoigne, and Netanyahu, MultiTemp, 2005
Akavia Memorial Lecture, Haifa Univ., 11.01.200742
Computational Efficiency (cont’d),An Example of a Pyramid-Like Approach
0
1
2
3
32 x 32
64 x 64
128 x 128
256 x 256
2
2x x
y y
s s
t t
t t
Akavia Memorial Lecture, Haifa Univ., 11.01.200743
IR Example Using Partial Hausdorff Distance
64 x 64
128 x 128
256 x 256
Akavia Memorial Lecture, Haifa Univ., 11.01.200744
IR Example Using PHD (cont’d)
Source: Netanyahu, LeMoigne, and Masek, IEEE-TGRS, 2004
Akavia Memorial Lecture, Haifa Univ., 11.01.200745
IR Components (Revisited)
Gray Levels EdgesFeatures
SimilarityMeasure
Strategy
Correlation L2-Norm Mutual Information Hausdorff Distance
Fast Fourier TransformGradient Descent
Spall’sOptimization
RobustFeature
Matching
Thevenaz,Ruttimann,
Unser Optimization
Wavelets or Wavelet-Like
Akavia Memorial Lecture, Haifa Univ., 11.01.200746
IR Components (Revisited)
Features
SimilarityMeasure
Strategy
Correlation L2-Norm MI Hausdorff Distance
FFT
RobustFeature
MatchingGradient Descent
Spall’sOptimization
Thevenaz,Ruttimann,
Unser Optimization
Gray Levels Spline or SimoncelliLPF
Simoncelli BPF
L2-Norm MI
Gradient Descent
Spall’sOptimization
Thevenaz,Ruttimann,
Unser Optimization
Akavia Memorial Lecture, Haifa Univ., 11.01.200747
Goals of a Modular Image Registration Framework
• Testing framework to:– Assess various combinations of components – Assess a new registration component
• Web-based registration tool would allow user to “schedule” combination of components, as a function of:– Application– Available computational resources– Required registration accuracy
• Prototype of web-based registration toolbox:– Several modules based on wavelet decomposition– Java implementation; JNI-wrapped functions
Akavia Memorial Lecture, Haifa Univ., 11.01.200748
Web-Based Image Registration ToolboxTARA (“Toolbox for Automated Registration & Analysis”)
Akavia Memorial Lecture, Haifa Univ., 11.01.200749
Web-Based Image Registration ToolboxTARA (“Toolbox for Automated Registration & Analysis”)
Akavia Memorial Lecture, Haifa Univ., 11.01.200750
Current and Future Work
• Conclude component evaluation – Sensitivity to noise, radiometric transformations, initial
conditions, and computational requirements– Integration of digital elevation map )DEM( information
• Build operational registration framework/toolbox– Web-based– Applications:
• EOS validation core sites• Other EOS satellites )e.g., Hyperion vs. ALI registration( and beyond• Image fusion, change detection
View of the moon, as seen from Apollo 11 )1969)
“Icarus fell because he flew too close to the sun. Columbia - and the whole American manned space program today - fell because it flies too close to the Earth” – Charles Krauthammer (Feb. 2003)
Back to the Moon