Post on 21-Sep-2020
ANMN IMOS-ToolboxUpdate on QC tests
Guillaume Galibert, 30th of August 2012
PP QC
Context
Convert data files from ANMN sensors to IMOS NetCDF QC’d files
Data file
Local deployment
database
Data fileData files
IMOS - eMIIdata center
IMOS NetCDF
file
Data fileData file
Data files
Basic QC policy-Qualitative QC.
-Every data point is checked procedure byprocedure.
-For each procedure, only a higher flagcan/must override the current flag of a point.
-As an overall result, only the number of pointswhich failed is recorded for each test.
-For tests involving neighbours, only pointswith flags lower or equal to 3 are considered.
IOC quality flag scale
Procedures for CTD / WQM on mooringsQC procedure Variables checked
(* can be tuned by user)Parameterized
Pass flag
Fail flag
Impossible date TIME Yes Good Bad potentially correctable
Impossible location LATITUDE, LONGITUDE Yes, and per site
Good Bad potentially correctable
In/out water All No Non QC’d
Bad potentially correctable
Global range TEMP, PSAL, PRES/DEPTH, DO, CHL-A * Yes Good Bad
Regional range TEMP, PSAL, DO * Yes, and per site
Good Bad
Impossible depth PRES/DEPTH Yes Good Bad
Spike TEMP, PSAL, PRES/DEPTH, DO, CHL-A * Yes Good Bad
Gradient / Rate of change
TEMP, PSAL, PRES/DEPTH, DO, CHL-A * Yes Good Bad potentially correctable
Stationarity TEMP, PSAL, PRES/DEPTH, DO, CHL-A * No Good Bad
Climatology TEMP, PSAL, DO (when possible?) Yes Good Bad
1 - Impossible dateThe recorded time in a file takes place after a minimum valid date and after a maximum valid date. TIME is flagged, not the PARAMETERS. Default values are :
01 Jan. 2007 <= TIME <= today
2 - Impossible LocationActual position of a mooring is in a box or circle centred on the site nominal location :
Site Nominal longitude
Nominal latitude
Box +/- in Longitude
Box +/- in Latitude
Circle radius (km)
NRSMAI 148.2333 -42.59667 0.2 0.25 2.5
NRSNSI 153.58 -27.389 0.2 0.25 2.5
NRSKAI 136.4473 -35.83217 0.2 0.25 2.5
NRSESP 121.85 -33.93333 0.2 0.25 2.5
NRSROT 115.4167 -32 0.2 0.25 2.5
NRSNIN 113.947 -21.86652 0.2 0.25 2.5
NRSDAR 130.7681 -12.4 0.2 0.25 2.5
NRSYON 147.6184 -19.30847 0.2 0.25 2.5
NRSPHB 151.25 -34.08333 0.2 0.25 2.5
... ... ... ... ... ...
3 – In/out water testAnything outside a specified time period fails the test.
In water date/time is taken from the ddb by order of preference :1. Time first in position2. Time first wet3. Time switched on
Out water date/time is taken from the ddb by order of preference :1. Time last in position2. Time on deck3. Time switched off
4 – Global rangeEach listed parameter has its values compared to global upper and lower valid values :
IMOS parameter
CF title Unit Valid min
Valid max
Source
TEMP Sea water temperature Celsius -2.5 40 ARGO Quality control manual v2.7 (Jan. 2012)
PSAL Sea water salinity PSU 2.0 41 ARGO Quality control manual v2.7 (Jan. 2012)
PRES Sea water pressure dbar 5 12 010 ARGO Quality control manual v2.7 (Jan. 2012)
PRES_REL Sea water relative pressure
dbar -5 12 000 ARGO Quality control manual v2.7 (Jan. 2012)
DEPTH Depth m -5 12 000 ARGO Quality control manual v2.7 (Jan. 2012)
DOX1 Mole concentration of dissolved molecular oxygen in sea water
umol/l 0 900 000 MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)
DOX2 Moles of oxygen per unit mass in sea water
umol/kg 0 880 000 MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)
CPHL, CHLU, CHLF
Mass concentration of chlorophyll in sea water
mg/m3 0 100 MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)
... ... ... ... ...
5 – Regional rangeEach listed Site and parameter has its values compared to regional upper and lower valid values :
Site IMOS parameter
CF title Unit Valid min
Valid max
Source
NRSPHB TEMP Sea water temperature Celsius 10 28 Port Hacking historical water samples.
NRSPHB PSAL Sea water salinity PSU 31 37 Port Hacking historical water samples.
NRSPHB DOX2 Moles of oxygen per unit mass in sea water
umol/kg 50 400 Port Hacking historical water samples.
NRSMAI TEMP Sea water temperature Celsius 8 21 Maria Island historical water samples.
NRSMAI PSAL Sea water salinity PSU 33 37 Maria Island historical water samples.
NRSMAI DOX2 Moles of oxygen per unit mass in sea water
umol/kg 100 400 Maria Island historical water samples.
... ... ... ... ... ... ...
6 – Impossible depthCompares the actual depth of the instruments to its nominal depth.
Acceptable ranges are derived from this formula :
upperBound = instNomZ + upperCoeff * (siteNomZ / instNomZ)
lowerBound = instNomZ - lowerCoeff * (siteNomZ / instNomZ)
Default value for upperCoeff = 3.Default value for lowerCoeff = 7.
7 – Spike testEach listed parameter has its own threshold value to be used in the ARGO spike test :
|Vn-(Vn+1 + Vn-1)/2| - |(Vn+1 - Vn-1)/2| <= threshold
PABIM white book v1.3 (Feb. 2010) threshold :
|median(Vn-2,Vn-1,Vn,Vn+1,Vn+2)| + |stDev(Vn-2,Vn-1,Vn,Vn+1,Vn+2)|
IMOS parameter
CF title Unit Threshold
Source
TEMP Sea water temperature Celsius 6 ARGO Quality control manual v2.7 (Jan. 2012)
PSAL Sea water salinity PSU 0.9 ARGO Quality control manual v2.7 (Jan. 2012)
PRES, PRES_REL, DEPTH
Sea water pressure, Sea water relative pressure, Depth
dbar, m 3
DOX1, DOX2
Mole concentration of dissolved molecular oxygen in sea water, Moles of oxygen per unit mass in sea water
umol/l, umol/kg
PABIM MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)
CPHL, CHLU, CHLF
Mass concentration of chlorophyll in sea water
mg/m3 PABIM MyOcean - Real Time Quality Control of biogeochemical measurements v1.0 (Jan. 2011)
... ... ... ...
8 – Gradient Vs Rate of changeGradientEach listed parameter has its ownthreshold value to be used in the ARGOgradient test :
|Vn-(Vn+1 + Vn-1)/2| <= threshold
IMOS parameter
CF title Unit Gradientthreshold
Rate of changethreshold
TEMP Sea water temperature
Celsius 9 2*stDev(first month)
PSAL Sea water salinity PSU 1.5 2*stDev(first month)
PRES, PRES_REL, DEPTH
Sea water pressure, Sea water relative pressure, Depth
dbar, m
3 2*stDev(first month)
DOX1, DOX2
Mole concentration of dissolved molecular oxygen in sea water, Moles of oxygen per unit mass in sea water
umol/l, umol/kg
90 2*stDev(first month)
CPHL, CHLU, CHLF
Mass concentration of chlorophyll in sea water
mg/m3 3 2*stDev(first month)
... ... ... ... ...
Rate of changeEach listed parameter has its ownthreshold value to be used in theMyOcean rate of change test :
|Vn – Vn-1| + |Vn – Vn+1| <= 2*threshold
-Points with more than 1h apartare not considered asneighbours and current point isflagged as “Non QC’d”.
-Tests only flag current point Vn.
8 – Gradient Vs Rate of changeGradient
|Vn-(Vn+1 + Vn-1)/2| <= constant
Rate of change
|Vn – Vn-1| + |Vn – Vn+1| <=2* 2*stDev(first month)
9 - StationarityThis IOC test finds and flags any consecutive equal values of the below listedparameters when :
number of consecutive points of equal values > 24*(60/delta_t)
with delta_t the sampling interval in minutes.
IMOS parameter CF title Unit
TEMP Sea water temperature Celsius
PSAL Sea water salinity PSU
PRES Sea water pressure dbar
PRES_REL Sea water relative pressure dbar
DEPTH Depth m
DOX1 Mole concentration of dissolved molecular oxygen in sea water umol/l
DOX2 Moles of oxygen per unit mass in sea water umol/kg
CPHL, CHLU, CHLF Mass concentration of chlorophyll in sea water mg/m3
... ... ...
10 – Climatology testIf a climatology exist for the studied site then the deployment data set will be checkedagainst it using pre-defined minimum and maximum valid values.
The default range is defined as follow :
validMin = min(mean - 6*stdDev, mini-2*stdDev)
validMax = max(mean + 6*stdDev, maxi+2*stdDev)
Procedures for ADCP on mooringsQC procedure Variables checked
(* can be tuned by user)Parameterized
Pass flag
Fail flag
Impossible date TIME Yes Good Bad potentially correctable
Impossible location LATITUDE, LONGITUDE Yes, and per site
Good Bad potentially correctable
In/out water All No Non QC’d
Bad potentially correctable
Global range TEMP, PSAL, PRES/DEPTH, UCUR, VCUR, WCUR ? *
Yes Good Bad
Regional range TEMP, PSAL * Yes, and per site
Good Bad
Impossible depth PRES/DEPTH Yes Good Bad
Spike TEMP, PSAL, PRES/DEPTH * Yes Good Bad
Rate of change TEMP, PSAL, PRES/DEPTH * Yes Good Bad potentially correctable
Stationarity TEMP, PSAL, PRES/DEPTH * No Good Bad
Climatology TEMP, PSAL Yes Good Bad
Teledyne UCUR, VCUR, WCUR Yes Good Bad
Side lobe contamination
UCUR, VCUR, WCUR No Good Bad
11 – Teledyne-For Workhorse ADCPs manufactured by Teledyne RD Instruments.
12 – Side lobe contaminationFlags all the bins above the last one which measurements are contaminated by the sidelobes of the antenna diagram :
contaminated depth = depth*(1-cos(beam angle))
Future work / ideas-Choose Gradient or Rate of Change test with best threshold values.-Define and describe final strategy for climatology building.-Define best valid ranges for climatology test.-Introduce overall QC flag per deployment like ARGO.-Introduce visual QC to validate automated QC.
-Introduce quantitative QC (Daniel Smith and Monique Breslin presentations).
Thank you+ special thanks to QC working groupAnd Esmee Van Wijk (ARGO DM QC)