The Joint Agency Commercial Imagery Evaluation (JACIE) Team

and Product Characterization Approach

Vicki ZanoniNASA Earth Science Applications Directorate

John C. Stennis Space Center

ISPRS Commission I/WG2International Workshop on

Radiometric and Geometric CalibrationDecember 1-5, 2003

Co-authors

• Lockheed Martin Stennis Operations– Mary Pagnutti– Robert Ryan

• South Dakota State University– Dennis Helder

• U.S. Geological Survey– Greg Snyder

• Booz-Allen Hamilton, NIMA Commercial Imagery Program– William Lehman– Spencer Roylance

• Several U.S. Government agencies have purchased commercial remote sensing data products to support research and applications

• The Government’s use of commercial remote sensing products requires thorough knowledge of data quality

Joint Agency Commercial Imagery Evaluation (JACIE) Team

• The Joint Agency Commercial Imagery Evaluation (JACIE) team was formed to leverage capabilities for the characterization of commercial remote sensing products

• Each agency brings unique characterization expertise– National Aeronautics and Space Administration (NASA) - Systems

Characterization (radiometric, spatial, geometric)– National Imagery and Mapping Agency (NIMA)- Photogrammetry

and Image Interpretability– U.S. Geological Survey (USGS) - Cartographic Assessments

SCIENCE USERS

JACIE TEAM

EOS Validation

Teams U of Maryland U of Arizona

S. Dakota State U

JACIE Characterization Team

Product Characterization

• For both research and applications, commercial products must be well-characterized for accuracy and repeatability.

• Since commercial systems are built and operated with no government insight or oversight, the JACIE team provides an independent product characterization of delivered image and image-derived end products.

• End product characterization differs from the systems calibration approach that is typically used with government systems, where detailed system design information is available.

• The product characterization approach addresses three primary areas of product performance: geopositional accuracy, image quality, and radiometric accuracy.

Geopositional Accuracy Assessment Method

• Utilize sites containing several “image-identifiable” targets and compare their known locations with those defined by the commercial image product.

• Follow Federal Geographic Data Committee (FGDC) standards on target number and distribution

• Compute accuracy statistics. – Root Mean Square Errors in X and Y directions (RMSEx and RMSEy)– Radial (net) RMSE (RMSE r)– Circular error at a 90% confidence level (CE90)

• commonly used to specify geopositional accuracy– Circular error at a 95% confidence level (CE95)

Geopositional Accuracy Equations

n

xx contimg 2 n

yy contimg 2

22 RMSEyRMSEx

CE90 = 1.5175 • RMSEr

CE95 = 1.7308 • RMSEr

RMSEx = RMSEy =

RMSEr =

Where Ximg and Yimg are the image-derived positions of the targetXcont and Ycont are the actual, measured locations of the target

n is the number of points used in the analysis

Above equations assume no systematic bias.

Image-identifiable targets are GPS-surveyed to within sub-meter accuracy, typically within a few centimeters

Geopositional Accuracy Targets

Road intersectionsBrookings, South Dakota

2.44-meter geodetic targetat NASA Stennis Space Center

Spatial Response and Image Quality

• The effective spatial resolution of a system or data product is driven by several parameters, including ground sample distance (GSD), point spread function (PSF), and signal-to-noise ratio (SNR).

• System performance is often specified in terms of modulation transfer function (MTF) at the Nyquist

frequency. • The JACIE team employs 2 approaches to

characterize spatial response– use of edge targets to estimate edge response

• Relative edge response is easier to measure and is mathematically related to PSF and MTF.

– use of pulse targets to estimate MTF

Edge Response MethodSystem Point Spread Function

x *

Edge Target

Edge Response

Slope ~ 1/x

SpatialDomain

Steepness of edge response effects spatial resolution

Targets should allow for both cross-track and along track measurements.Size and placement of target are critical to ensuring proper sampling along edges.

Edge Targets

QuickBird image of 20-meter concrete edge target located at NASA Stennis

Space Center

20-meter X 20-meterreflectance tarps used in edge response characterization at

NASA Stennis Space Center.

Example Edge Responses and Line Spread Functions

Northing directionFeb. 17, 2002

Easting directionJan. 15, 2002

IKONOS (MTFC-Off) panchromaticedge responses, derived from 20 X 20 metertarp edge targets

Numerical differentiation of the edge response yields the line spread function (LSF).

The full width half maximum (FWHM) value of the LSF is often compared to the systems reported GSD.

edge response

line spread function

Pulse Target Method

• For larger GSD products, it is difficult to find edge targets large enough to estimate edge response.

• Thus, for the commercial multispectral products, the pulse target method provides a better estimate of image quality.

• In the pulse method, long “strip” or pulse targets are employed.

Includes material © Space Imaging, L.P.

Pulse tarp orientation with respect to true north. Orientation of both pulse and edge targets must allow for proper pixel sampling across the edge transitions.

Pulse Target

IKONOS image of 12 meter X 60 meter pulse target tarp used to characterize multispectral MTF. The size of the tarp is critical to the effectiveness of the pulse method.

Illustration of Pulse Method

Pulse target input and output response for QuickBird blue band, acquired Aug.25, 2002

The Fourier transform of the input and output responses.

The output transform divided by the input transform provides the MTF. The value of MTF at the Nyquist frequency is often used to specify spatial performance.

Image Interpretability

• Imagery is also characterized using the National Imagery Interpretability Rating Scale (NIIRS) and Essential Elements of Information (EEIs) , a means of quantifying the ability to identify certain targets (e.g., railcars, airplanes) within an image product.

• NIIRS is a 10-level rating scale (0 - 9) that defines the ability to identify certain features or targets within an image.– Detect trains or strings of standard rolling stock on railroad

tracks (not individual cars) - NIIRS 3

– Detect individual spikes in railroad ties.- NIIRS 9

• EEIs are certain features and targets (e.g. railcars, aircraft) that correspond to the various NIIRS levels.

NIIRS Assessment Method

• Several image chips are extracted from images acquired over a given time period and over multiple locations.

• The image chips are evaluated by a group of NIMA-certified image analysts.

• The analysts each evaluate the same set of images under the same conditions. i.e. using the same computer and amount of image magnification, no additional image processing or enhancement.

• The analysts assign NIIRS ratings and confidence ratings associated with identification of EEIs.

• Statistical analyses are performed on analysts’ results to understand the consistency and reliability of the different analysts, and to identify any outlier image chips used in the assessment.

• Good correlation among the analysts provides confidence in the average NIIRS assigned.

General Imagery Quality Equation (GIQE)

• The GIQE mathematically relates NIIRS to several parameters as a means of quantifying image quality

SNR

GHRERbGSDaNIIRS GMGMGM

344.0656.0loglog251.10 1010

where

GSDGM is the geometric mean of the ground sampled distance,

RERGM is the geometric mean of the relative edge response,

HGM is the geometric mean-height overshoot caused by MTFC (Leachtenauer et al., 1997), and

G is the noise gain associated with MTFC. In the current form of the GIQE,

SNR is estimated for differential radiance levels from Lambertian scenes with reflectances of 7% and 15% with the noise estimated from photon, detector, and uniformity noise terms.

If the RER exceeds 0.9, then a equals 3.32 and b equals 1.559; otherwise, a equals 3.16 and b equals 2.817.

Radiometric Characterization

• Reflectance-based vicarious calibration approach– Characterize reflectance of large, uniform target at time of

satellite overpass• Measurements taken of target area and a 99% reflectance

spectralon panel (Jackson BRDF model)

– Characterize atmosphere at time of satellite overpass– Use radiative transport code to predict at-sensor radiance– Compare predicted at-sensor radiance to actual radiance

acquired by sensor

• Combine results of multiple independent teams. Each team has slightly different measurement techniques and data processing methods.

Radiometric Characterization Sites

QuickBird image of NASA Stennis Space Center

radiometric characterization site

Grass field in Brookings, South Dakota

Lunar Lake playa, Nevada

In-situ Data Acquisition for Radiometric Characterization

Spectroradiometer measurementsof target albedo

Spectroradiometer measurementsof Spectralon 99% reflectance panel

Solar radiometer measures optical depth

Multi-filter rotating shadowband radiometer measures direct and diffuse solar irradiation

Full sky imager documents cloud cover conditions

Radiosonde measures pressure, temperature, and humidity profiles

At Sensor Radiance Prediction Method

Atmospheric Parameters

H2O, O3, Pressure

Site Geometry

Altitude, Latitude, Longitude, Time

Aerosol Model

Size and Number

Surface CharacteristicsASD Measured Target Surface Albedo

Surrounding Surface AlbedoTarget BRDF

Solar Radiometer

MODTRAN

Visibility and Asymmetry adjusted for best fit

transmittance and diffuse to global ratio

MeasuredAtmosphericTransmittance

PredictedAtmosphericTransmittance

Solar Geometry

Sensor Characteristics

Viewing Geometry

Spectral Response

ASD/Spectralon Geometry

MODTRAN

Verification of input parameters by comparison to calibrated ASD radiance

measurements

MODTRANMODTRAN

ASD/Spectralon Radiance

ASD MeasuredRadiance

PredictedRadiance

At Sensor Radiance Prediction

Example of JACIE-derived blue band radiometric calibration curve and coefficient compared to original and revised QuickBird calibration curves/coefficients. The JACIE-derived curve was the result of several vicarious calibration activities conducted by JACIE team members during the 2002 acquisition season.

Example Radiometric Characterization Results

Conclusions

• JACIE product characterizations provide thorough estimates of data quality for products provided to end users.

• The team has performed spatial, radiometric, and geopositional characterization of IKONOS and QuickBird products

• Several improvements in IKONOS and QuickBird-2 data quality have resulted

• Characterization results presented at annual JACIE High Spatial Resolution Commercial Imagery Workshops

• The JACIE team will soon focus on characterization of OrbView-3.

Back up

Pre-launch Data Characterizations

• For commercially-developed systems, the government and science end users do not have direct insight into pre-launch calibrations

• Some critical parameters can only be accurately measured in the laboratory

• JACIE reviews pre-launch calibration results that enable thorough V&V of data specification such as– Spectral filter response curves across sensor FOV– Radiometric calibration coefficients– SNR estimates– Dynamic range/Linearity measurement– Polarization sensitivity measurement– Modulation Transfer Function (MTF) measurement– Geometric calibration coefficients– Band-to-band registration– Bad pixel maps

FWHM

numericallydifferentiate

FWHM

edge response line spread function

line spread function

Fouriertransform