Gap-filling workshop, Jena 09/2006 Markus Reichstein

Gap-filling: What, why, how? -

an Introduction

Gap-filling Comparison Workshop, September 18-20, 2006Max Planck Institute for Biogeochemistry

BEAM-DIG

MPI-BGC

BEAM-DIG

MPI-BGC

Biogeochemical

Model-Data Integration Group

M. Reichstein

(Biogeochemical Model-Data Integration Group, Max-Planck Intstitute for Biogeochemistry, Jena)

Gap-filling workshop, Jena 09/2006 Markus Reichstein

Why are we here – a short historical perspective ?

2001: Falge et al.

FLUXNET (AMERIFLUX, EUROFLUX) 2002: MDS online

gap-filling tool

MIND 2003: Boost of gap-filling methods

FLUXNET2004: Eddy QC/QA/GF/FP workshop

CARBOEUROPE

Today: Comp.of 15 methods + spin-offs from gap-filling

Gap-filling workshop, Jena 09/2006 Markus Reichstein

What is a gap ?

• “Gap is a synonym for any hole or opening; a chasm. Many uses of the word are either literally or figuratively based on this meaning.” (wikipedia.org)

• “A gap is a series of missing data of eddy-covariance flux data (and/or meteorology) caused by instruments failure, unfavorable measurement conditions or removal of data point during the quality control.”– Univariate gaps (e.g. only NEE)– Multivariate flux gaps (e.g. all fluxes missing sonic failure)– Flux and meteo gaps (e.g. all missing storm or Xmas)– Length of a gap?

Gap-percentage varies

Falge et al. 2001

CE database ~30%(without ½ year gaps)

Gap-filling workshop, Jena 09/2006 Markus Reichstein

Gaps abundance

0 50 100 150 200-4

-2

0

2

4

6

Length of gap [days]

Fre

qu

en

cy [

log

(ye

ar-1)]

0.0 0.5 1.0 1.5 2.0 2.5-4

-2

0

2

4

6

Fre

qu

en

cy [

log

(ye

ar-1)]

Length of gap [log(day)]

Gap-filling workshop, Jena 09/2006 Markus Reichstein

Days affected by gaps

0 50 100 150 2000

2

4

6

Length of gap [days]

To

tal

day

s af

fect

ed [

day

s/ye

ar]

0 50 100 150 2000

5

10

15

20

25

Length of gap [days]

Cu

mu

lati

ve p

erce

nta

ge

affe

cted

Gap-filling workshop, Jena 09/2006 Markus Reichstein

Why ?

‘Annual sums’

Model validation at daily to monthly scale

Syntheses at monthly time scale

Model parameterizationat hourly to daily scale

Statisticaltime-series analysis

Uncertainty estimation

Modellers

Data analysts

Gap-filling workshop, Jena 09/2006 Markus Reichstein

Available data at daily and monthly scale before and after gap-filling

0%

20%

40%

60%

80%

100%

120%

Fully Gapy fqcOK>0.95 fqcOK<0.95

Daily

Monthly

Gap-filling workshop, Jena 09/2006 Markus Reichstein

How?

• Gap-filling requirements– Conservation of annual sums– Conservation of fluxes at other time integrals– Minimum of a-priori theoretical assumptions– Usage of a much as possible information from

data– Applicability with available data– Conversation of statistical time-series properties– Availability of conditional error estimate

Development of gap-filling methods

Method Relies on … / Exploits …

Mean diurnal variation (Falge et al. 2001) Temporal Autocorrelation, diurnal variation

Non-linear regression (Falge et al. 2001) Functional dependence on meteo conditions

Look-up table (Falge et al. 2001) Dependence on meteo conidtions

Neural networks (Papale, Valentini, 2003) Functional dependence on meteo conditions and time of the year

Advanced statistical filtering techniques (Multiple imputation (Hui et al. 2004); State dependent parameter estimation, Jarvis et al. 2004)

Functional dependence on meteo conditions, temporal autocorrelation; statistical assumptions (normalty of data etc.)

Gap-filling workshop, Jena 09/2006 Markus Reichstein

Classification of gap-filling methods

• With vs. without meteorological drivers• Data-oriented versus process-oriented

approaches• Incorporation vs. ignorance of autocorrelation• Smooth versus non-smooth methods• Look-up tables vs. regressions vs. neural

networks

Gap-filling workshop, Jena 09/2006 Markus Reichstein

Gap-filling methods characterized

Meteo info Auto-correlation

Theo. assumption

Conservation of error

MDV - x - -

LUT x - - -

MDS, MLUT x x - (x)

NLIN, DA x - X -

SDP x x (x) x

ANN x (x) - -

Gap-filling workshop, Jena 09/2006 Markus Reichstein

Conclusions

• Gap-filling is important from different perspectives

• Annual NEE is not the only target

• Existence of vast majority of methods with different characteristics need for a characterization and cross-

comparison

Gap-filling workshop, Jena 09/2006 Markus Reichstein

Conclusions II

Open questions• Can we transfer methods established for NEE also to

energy fluxes and meteorological data ?• How can discontinuous systems be gap-filled ?• How critically do gap-filling methods affect the

statistical properties of the time-series?• How can gap-filling be used for uncertainty estimation

of flux data ?• And for flux-partitioning?