Validated data and removal of bias through

Traceability to SI units

Nigel Fox

Centre for Optical and Analytical Measurement

Dec 03

Resolution adopted by CEOS Plenary 14 (Nov 2000)

1/ All EO measurement systems should be verified traceable to SI units for all appropriate measurands.

2/ Pre-launch calibration should be performed using equipment and techniques that can be demonstrably traceable to and consistent with the SI system of units, and traceability should be maintained throughout the lifetime of the mission.

Traceability – Property of the result of a measurement or the value of a standard whereby it can be related to stated references, usually through an

unbroken chain of comparisons all having stated uncertainties

Vocabulary for International Metrology (VIM)

SI units – The coherent system of units adopted and recommended by the General Conference of Weights and Measures (CGPM).

Reproducibility of results of measurements – Closeness of the agreement between the results of measurements of the same measurand carried out under changed conditions of measurement.

Accuracy of measurement – Closeness of the agreement between the result of a measurement and a true value of the measurand.

Precision – No metrological definition except to state that it should never be used in the context of “accuracy” and, because of possible confusion its use, should normally be avoided in metrological applications.

Repeatability of results of measurements – closeness of the agreement between the results of successive measurements of the same measurand carried out under the same conditions of measurement.

Uncertainty of measurement – Parameter, associated with the result of a measurement, that characterises the dispersion of the values that could reasonably be attributed to the measurand.

Error of measurement – Result of a measurement minus a true value of the measurand Stability – Ability of a measuring

instrument to maintain constant its metrological characteristics with time.

Traceability – Property of the result of a measurement or the value of a standard whereby it can be related to stated references, usually through an unbroken chain of comparisons all having stated uncertainties.

Precision – No metrological definition except to state that it should never be used in the context of “accuracy” and, because of possible confusion its use, should normally be avoided in metrological applications.

Uncertainty of measurement – Parameter, associated with the result of a measurement, that characterises the dispersion of the values that could reasonably be attributed to the measurand.

Convention of the Meter – established 1875

RM ORegional M etro logy Organisation

BIPMBureau de In ternational

des Poids et M esures

CCPRConsultative Com m ittee

for Photom etry and Radiom etry

CCTConsultative Com m ittee

for Therm om etry

CCMC onsultative C om m ittee for

M ass and R elated Q uantitiesCC...

C IPMCom ite In ternational des Poids et M easures

CPGMConference Generale des Poids et M esures

66 m em ber & associate states

SI Traceability: The Mututal Recognition Arrangement (MRA)

SIM

CCPR Key comparisons

• Spectral Irradiance

• Spectral Responsivity

• Luminous intensity

• Luminous Flux

• Spectral transmittance

• Spectral diffuse reflectance

(total hemispherical)

CCPR

Monitoring and interpreting the Earth’ systems

Solar Reflected RadiationAtmosphere - Aerosol (size & distribn)

- Clouds - Pollution (impact on health)

Water - Pollution (originator)- Algae plumes

Land - Useage / condition - Type/quantity of vegetation

- Minerals - Carbon & hydrological cycles

Governments – Treaties, Tax, Planning

Spatial variability requires good stability and SNR (signal to noise ratio) from a single sensor - but long term studies “climate change” need accuracy and consistency

Engineering specification of SNR should equate to accuracy

Thermal Emitted RadiationAtmosphere – Atmospheric chemistry Water – Temperature Land – Fires, Volcanoes, Pollution,

Incoming Solar RadiationDrives all the processes of the Earth

System and potentially damaging (UV) to Biosphere (Human health)

Need for improved Quality Assurance

Requirement - baseline for climate studies

- improve models- prediction of weather systems- identify crops from weeds

- global warming - Man or Nature?

- detection of change

- monitoring the treaties - auditing carbon sinks - efficiency of carbon sinks

- QA of operational services (GMES)

- instrument synergy

Difficulties - bias between sensors

- instruments change on launch and degrade in-orbit (gain and spectral)

To strengthen the evidence

Need for improved Quality Assurance

- application of correction for atmosphere loss

- lack of cohesion between networks and “ground truth” validation data (atmosphere an exception)

- models inadequate

- no consistent statements of uncertainty or

confidence.

Anomalies in NOAA/AVHRR data

N-6 N-11N-7 N-9 N-14

Normalised Difference Vegetation Index (NDVI) over “stable” desert as measured by AVHRR

–Demonstrates both in-flight “ageing” and initial calibration biases

Temporal change difficult to identify even using “identical” instrumentation without normalisation

Total solar Irradiance (Solar constant) –only normalisation allows a long term record to be established (Biases are 10 X larger than necessary to detect impact on climate change)

LOS 1998 IEEE Trans G & RS

Traceability chain for optical radiation measurement

Optical power =Po

Electrical Heater Power = PEAbsorbing black coating

Copper disk

When thermometer temperature T=To=TE then Po=PE

Electrical Substitution Radiometry a 100 yr old technology

Mechanical cryogenic cooler “Fridge” (T = 20 K)

Principle of Cryogenic radiometry

Optical power =Po

Absorbing cavity (~ 0.99999)Electrical Heater Power = PE

Cooling improves sensitivity by 1000 X

Thermal shroud When T =To=TE then Po=PE

Shutter

25 yrs of cryogenic radiometry at NPL

Primary standard lamp

Working standard lamp

Cal Lab Primary lamp

Cal Lab working std Lamp

User Cal Lamp

User Instrument

Spectroradiometer

Spectroradiometer

Spectroradiometer

Spectroradiometer

Fundamental constants (SI)

Primary standard cryogenic radiometer

LaserCal interval ~100nm

LaserCal interval ~0.1 nm

Photodiode (spectral responsivity

Filter Radiometer

Radiance Temperature

Ultra High Temperature Black Body (3500 K)

Radiance continuum (Planck)

Spectroradiometer (multi-band filter radiometer

Spectral Radiance/Irradiance calibrations LAND OCEAN ATMOSPHERE

Satellite Pre-flight Calibration

Traceability ??

Satellite In-flightCalibration

Data products

Atmosphere/Model

Vicarious

Lamp Solar illuminated Diffuser

Main uncertainty components: k = 1 confidence level

Uncertainty sources with an effect on irradiance less than 0.03%- Blackbody temperature:

- Size of source effect- Mathematical approximations- Geometric Factor- Electronics- Filter radiometer relative shape

-Blackbody absorption (apart from ~380 nm)-Blackbody-integrating sphere geometry-SRIPS linearity-Monochromator wavelength error-Monochromator bandwidth and subsequent mathematical approximations-Lamp current control

Lamp alignment

SRIPS repeatability

Blackbody Emissivity

Lens Transmission

FR absolute responsivity

Blackbody Stability

Blackbody Uniformity

0.06%0.06%0.06%

0.07%0.08%0.26%

0.02%0.03%0.05%0.089 K

0.02%0.03%0.05%0.103 K

0.03%0.03%0.06%0.115 K

0.06%0.07%0.13%0.258 K

0.07%0.09%0.17%0.327 K

T

E

M

P.

700 nm550 nm300 nmIndividual

uncertainty

Effective uncertainty, 3050 K BB

Uncertainty component

S R IP S p r im a r y s c a le u n c e r ta in ty : O v e r a l l a t 9 5 % c o n f id e n c e le v e l

0 .0 %

0 .5 %

1 .0 %

1 .5 %

2 .0 %

2 .5 %

3 .0 %

3 .5 %

4 .0 %

4 .5 %

2 5 0 5 0 0 7 5 0 1 0 0 0 1 2 5 0 1 5 0 0 1 7 5 0 2 0 0 0 2 2 5 0 2 5 0 0W a v e le n g th (n m )

Radi

ance

unc

erta

inty

(%)

P r im a ry S c a le

B B ra d ia n c e

S R IP S R e p e a ta b il i ty

U V F W m o d e ll in g

L a m p p e r f o r m a n c e : F E L s ( a p p r o x 1 in 3 s h o w a d e v ia t io n )

- 1 4 .0 0 %

-1 2 .0 0 %

-1 0 .0 0 %

-8 .0 0 %

-6 .0 0 %

-4 .0 0 %

-2 .0 0 %

0 .0 0 %

2 .0 0 %

4 .0 0 %

2 5 0 5 0 0 7 5 0 1 0 0 0 1 2 5 0 1 5 0 0 1 7 5 0 2 0 0 0 2 2 5 0 2 5 0 0

W a v e le n g th

Dif

fere

nc

e

Traceability for in-flight / in-situ / vicarious calibration

Spectral Radiance

- lamp illuminated spheres

- Filter radiometers (spectroradiometers)

- Irradiance source + diffuser

Lamp + spectralon or ….

Sun + spectralon or ….

Sun + Moon

Spectral Reflectance

- in-situ absolute ratio (using radiometers)

- Ratio to “standard” reflector/diffuser

Via models / atmosphere correction

to satellite for cal/val

(radiances)

To bio/geophysical quantities (refelectances)

The NPL diffuse reflectance scale is derived goniometrically for the spectral region 300 to 2500 nm

Uncertainty of <0.2 % in the visible and shown equivalence with NIST

Diffuse reflectance (BRDF)

SampleDetectorSpectralon 400 nm

0.65

0.75

0.85

0.95

1.05

-90 -60 -30 0 30 60 90

radi

ance

fact

or

Beta-NPL (400 nm), Tile 1

Beta-PTB (400 nm), Tile 1

Beta-NPL (400 nm), Tile 2

Beta-NIST (400 nm), Tile 2

Validated data products require all processing steps and data to be QA – Accredited?

Pre-flight- User specification- Instrument build compliance- Calibration?

Post-launch- In-flight checks- Ground “Truth” comparison- Inter-sensor cross

calibration Processed data released

- “validated”- Uncertainty statement?

Rare for all these activities to have been independently reviewed and/or audited

Global Monitoring Environment and Security (GMES)Joint initiative of ESA and EU

Aim: to establish “operational services” for Earth Observation data to meet needs of key stakeholders , public services, private industry,

academia and the citizen with a view to financial self-sufficiency.

Robust evidence requires robust QA

Success requires: - Combination of data from many sources, (satellites, in-situ, aircraft) - Efficient production of cost effective, reliable, data products / maps - Data must provide the evidence to allow decisions to be taken with confidence. - Innovation in measurement and analysis

Reliability: Implies Quality assurance and statements of confidence associated with data (not only for end users but also

“operational service” providers

Users generally assume QA

Infrastructure for innovation in measurement, validation and QA of EO data

Public sector

Private Industry

Academia

QA

Advice

Calibration

TraceabilityAudit Validation

• Transfer standards

• Comparisons

• Innovation on techniques

• Measurement & test protocols

• International link

• Independence

In-situ

Pre-flight

airbornePost-launch

Modelling & Data processing

NPL ++

NIST ++

Summary Primary scales, transfer standards and techniques now allow high

accuracy to be achieved for both pre-flight and vicarious calibration (particularly for radiance)

All aspects/steps of producing EO data products needs validation and traceability (instrument calibration and algorithms/models)

Consistent presentation and breakdown of uncertainty budgets

Flexibility to allow innovation

Regular comparisons to evaluate biases

Establish well characterised ground targets as “reference standards”

Develop improved “in-flight” calibration methods e.g TRUTHS (Fox et al Proc. SPIE 4881, p395 2003)

For Earth Observation to provide the evidence to support policy

requires the industry and its data to be as robust as traditional industries