GENERAL PAPER

Uncertainty-based measurement quality control

Hening Huang

Received: 8 August 2013 / Accepted: 24 December 2013

� Springer-Verlag Berlin Heidelberg 2014

Abstract According to a simple acceptance decision rule

for measurement quality control, a measured value will be

accepted if the expanded uncertainty of the measurements

is not greater than a preset maximum permissible uncer-

tainty. Otherwise, the measured value will be rejected. The

expanded uncertainty may be calculated as the z-based

uncertainty (the half-width of the z-interval) when the

measurement population standard deviation r is known or

the sample size is large (30 or greater), or by a sample-

based uncertainty estimator when r is unknown and the

sample size is small. The decision made based on the

z-based uncertainty will be deterministic and may be

assumed to be correct. However, the decision made based

on a sample-based uncertainty estimator will be uncertain.

This paper develops the mathematical formulations for

computing the probability of acceptance for two sample-

based uncertainty estimators: the t-based uncertainty (the

half-width of the t-interval) and an unbiased uncertainty

estimator. The risk of incorrect decision-making, in terms

of the false acceptance probability and false rejection

probability, is derived from the probability of acceptance.

The theoretical analyses indicate that the t-based uncer-

tainty may result in significantly high false rejection

probability when the sample size is very small (especially

for samples of size 2). For some applications, the unbiased

uncertainty estimator may be superior to the t-based

uncertainty for measurement quality control. Several

examples from acoustic Doppler current profiler stream-

flow measurements are presented to demonstrate the

performance of the t-based uncertainty and the unbiased

uncertainty estimator.

Keywords Measurement uncertainty �Maximum permissible uncertainty � Uncertainty estimator �Samples of size 2

Introduction

The problem of interest is that an unknown constant

quantity is measured n times by a measurement system to

give the observations of x1,…, xn. Assume the observations

only have random errors and follow a normal distribution;

the mean, denoted as l, of the observation population is the

true value of the quantity, and the sample mean �X is taken

as an estimate of l. That is, the sample mean is taken as the

measured value. The error of the measured value, though

may be unknown, is e ¼ �X � l. The standard uncertainty

and expanded uncertainty associated with the measurement

may be estimated based on the principles described in the

Guide to the expression of uncertainty in measurement

(GUM) [1]. The uncertainty, defined as the half-width of an

interval having a stated level of confidence (see p. 2 in [1])

(i.e., the coverage probability), is an indicator of the quality

of the measurement. The uncertainty may be attributed to

instrument noise and/or environmental factors (e.g., tur-

bulence). This paper deals with the statistical quality

control of measurements based on the uncertainty associ-

ated with the measurement, regardless of whatever

instrument or system is used in the measurement.

Having measured a quantity, one might decide to (a)

accept the measured value or (b) reject the measured

value, based on an acceptance criterion (i.e., a decision

rule) for measurement quality control. According to a

H. Huang (&)

Teledyne RD Instruments, 14020 Stowe Drive, Poway,

CA 92064, USA

e-mail: hhuang@teledyne.com

123

Accred Qual Assur

DOI 10.1007/s00769-013-1032-5

simple acceptance decision rule, a measured value will be

accepted if the expanded uncertainty U associated with

the measurement at a specified coverage probability 1-a(usually 95 %) is not greater than a preset maximum

permissible uncertainty (MPU)

U�MPU ð1Þ

When the population standard deviation r is known or the

sample size is large (30 or greater), the expanded uncer-

tainty U is often calculated as the z-based uncertainty (i.e.,

the half-width of the z-interval). The decision made based

on the z-based uncertainty will be deterministic and may be

assumed to be correct.

However, in complicated measurements, such as

acoustic Doppler current profiler (ADCP) streamflow

measurement, the population standard deviation is often

unknown and the sample size is often small; the z-based

uncertainty is not available. In this situation, the expanded

uncertainty U may be estimated by a sample-based

uncertainty estimator. However, the decision made based

on a sample-based uncertainty estimator will be uncertain;

it may be correct or incorrect. That is, there is a risk in the

decision-making. The risk may be measured by the false

acceptance probability or the false rejection probability,

which depends on the sample-based uncertainty estimator

used.

Two sample-based uncertainty estimators are available

in the literature. One is the well-known t-based uncertainty

(i.e., the half-width of the t-interval) and the other is an

unbiased uncertainty estimator (or called the Craig model

[2, 3]). Although the concept of false acceptance or false

rejection has been well known in statistics and has been

used in conformity assessment (e.g., [4, 5, 6]), to the

author’s knowledge, the formulations of the false accep-

tance probability and the false rejection probability

associated with the t-based uncertainty and the unbiased

uncertainty estimator in measurement quality control have

not been found in the literature.

The objective of this paper is to develop the mathe-

matical formulations for computing the probability of

acceptance associated with the t-based uncertainty and that

associated with the unbiased uncertainty estimator. The

false acceptance probability and the false rejection proba-

bility can be derived from the probability of acceptance.

Several examples from ADCP streamflow measurements

are presented to examine the performance of the t-based

uncertainty and the unbiased uncertainty estimator.

When r is known or sample size is large

When the population standard deviation r is known or the

sample size is large (30 or greater), the expanded

uncertainty U is calculated as the z-based uncertainty

(denoted as Uz). Note that, in practice, it is widely accepted

that Uz may be calculated with a sample of size equal or

greater than 30. Accordingly, the uncertainty-based

acceptance criterion is

Uz ¼ za=2

rffiffiffi

np �MPU ð2Þ

where za/2 is the z-score. Equation (2) applies for n = 1, 2,

3, …, including the single observation.

Maximum permissible uncertainty is an uncertainty-

based tolerance limit for the measurement in consideration.

Note that MPU is a constant and independent from the

sample size. Since the uncertainty Uz decreases with

increasing sample size, Eq. (2) can always be met with a

larger sample.

As long as Eq. (2) is met, the measurement error will not

be greater than MPU with a probability of 1-a or greater.

That is,

Pðjej �MPUÞ� 1� a ð3Þ

Let ±MPU denote (-MPU, ?MPU), which is a coverage

interval for the measurement error e. The coverage

probability of ±MPU depends on the ratio between MPU

and Uz. The ratio is defined as the measurement quality

index (MQI)

MQI ¼ MPU

Uz

ð4Þ

Note that MQI defined by Eq. (4) is different from the

measurement capability index Cm defined in conformity

assessment (e.g., [4, 5]) in which the numerator is maxi-

mum permissible error (MPE). Cm measures the

measurement quality (Uz) relative to a measurement

instrument error limit (MPE); it is an indicator of the

instrument’s measurement capability. MQI measures the

measurement quality (Uz) relative to a permissible mea-

surement uncertainty limit (MPU); it is an indicator of the

quality of measurement results. The higher the MQI, the

better the measurement quality. This can be seen by

examining the coverage probability of ±MPU as a function

of MQI. When MQI = 1, the coverage probability of

±MPU is 1-a, the same as the coverage probability of the

z-interval. When MQI [ 1, the coverage probability of

±MPU will be greater than 1-a. For example, when

MQI = 1.3143 and za/2 = 1.96 (thus 1-a = 95 %), the

coverage probability of ±MPU is 99 %. On the other hand,

if MQI \ 1, the coverage probability of ±MPU will be

smaller than 1-a; the measurement result does not meet

the desired precision requirement (MPU).

It is important to note that, according to Eq. (2),

whether a sample (i.e., a measured value) is acceptable

solely depends on the population standard deviation r and

Accred Qual Assur

123

the sample size n. That is, if Eq. (2) is met, the whole

population of observations will satisfy Eq. (3) and any

sample drawn from the population is acceptable. This can

be readily understood by considering a simple case that

water temperature is measured by a temperature sensor

and MPU is set at 0.01 �C. Assume there are no random

error sources in the water temperature measurement other

than the sensor’s noise. If the temperature sensor is cal-

ibrated with the expanded uncertainty Uz = za/2rB 0.01 �C at n = 1, any reading from the sensor’s single

measurement is acceptable. However, if the sensor is

calibrated with the expanded uncertainty Uz = za/2r B

0.02 �C at n = 1, four observations (n = 4) must be

made to meet MPU = 0.01 �C.

When r is unknown and sample size is small

When the population standard deviation r is unknown and

the sample size is small (less than 30), the expanded

uncertainty U may be estimated by a sample-based

uncertainty estimator. Two sample-based uncertainty esti-

mators are considered in this paper: the t-based uncertainty

and an unbiased uncertainty estimator.

The t-based uncertainty, denoted as Ut, is written as

Ut ¼ ta=2 s=ffiffiffi

np

, where s is the sample standard deviation

and ta/2 is the t-score. When Ut is used, the uncertainty-

based acceptance criterion becomes

Ut ¼ ta=2

sffiffiffi

np �MPU ð5Þ

It should be pointed out that the t-based uncertainty is not

an unbiased estimate of the z-based uncertainty. It exhibits

significantly high positive bias error and precision error

when the sample size is very small [2, 3, 7].

The unbiased uncertainty estimator is written as

za=2 s= ðc4

ffiffiffi

npÞ, where c4 is the bias correction factor for

s (e.g., [8]). The author proposed the unbiased uncertainty

estimator for estimating the uncertainty of ADCP stream-

flow measurements in 2006 [9]. Two years later, the author

discovered through Internet search that the unbiased

uncertainty estimator is exactly the same as the first-order

approximation of the Craig’s approach to the probable

error of a mean [10]. Detailed discussion on the unbiased

uncertainty estimator can be found in [2, 3]. The unbiased

uncertainty estimator is denoted as Uz/c4 and is called the

z/c4-based uncertainty hereafter. In addition, Jenkins [7]

developed an empirical, unbiased uncertainty estimator

that is nearly the same as Uz/c4.

When Uz/c4 is used for measurement quality control, the

uncertainty-based acceptance criterion becomes

Uz=c4 ¼ za=2

s

c4

ffiffiffi

np �MPU ð6Þ

Note that Eqs. (5) or (6) applies for n = 2, 3, …. They do

not apply for the single observation.

According to Eq. (5), a sample with its s [ MPU �ffiffiffi

np

=ta=2 will be rejected. According to Eq. (6), a sample

with its s [ MPU � c4

ffiffiffi

np

=za=2 will be rejected. Since

ta/2 [ za/2/c4, the samples rejected based on the t-based

uncertainty will always be more than the samples rejected

based on the z/c4-based uncertainty.

To visualize how the samples are accepted or rejected

according to Eqs. (5) or (6), we employed the Monte Carlo

simulation to generate 5000 pairs of error e and sample

standard deviation s. The Monte Carlo simulation involved

randomly drawing samples of size n from a normally dis-

tributed population with l = 50 (any unit) and r = 5

(same unit as l). Two sample sizes, n = 2 and n = 4, were

considered because our discussions focused on very small

samples. The simulation was implemented using an Excel

spread sheet to generate 2 9 5000 random numbers for

n = 2 and 4 9 5000 random numbers for n = 4. The error

e and the sample standard deviation s were calculated for

each sample.

Figure 1a, b shows the scatter plots of |e| and s (nor-

malized by r = 5) from the Monte Carlo simulation for

n = 2 and n = 4, respectively. The MPU is assumed to

be 1.3143Uz at the 95 % coverage probability (za/2 =

1.96). Thus, the normalized MPU is 1.822 for n = 2 and

1.288 for n = 4. The indications of the samples rejected

(or accepted) according to Eqs. (5) and (6) are shown in

the figures. Note that the measurement quality index

MQI = 1.3143 in this situation so that, due to MQI [ 1,

all of the samples from the Monte Carlo simulation are

acceptable according to Eq. (2). However, as can be seen

in Fig. 1, many samples are rejected according to Eqs. (5)

or (6), which, apparently, are false rejections. Therefore,

there is an uncertainty in the quality control when using

Eqs. (5) or (6). In order to examine the uncertainty, we

define the probability that the sample satisfies either Eqs.

(5) or (6) as the probability of acceptance, Pa. The

probability of acceptance Pa associated with the t-based

uncertainty is

Pa ¼ Pta=2ffiffiffi

np s�MPU

� �

ð7Þ

Substituting MPU ¼ MQI � za=2r=ffiffiffi

np

into Eq. (7) yields

Pa ¼ Ps

r�MQI �

za=2

ta=2

Þ ¼ Pðffiffiffiffiffiffiffiffiffiffiffi

n� 1p s

r�MQI �

ffiffiffiffiffiffiffiffiffiffiffi

n� 1p za=2

ta=2

� �

ð8Þ

Denote a random variable x ¼ffiffiffiffiffiffiffiffiffiffiffi

n� 1p

s=r that follows the

chi distribution. Thus, the probability of acceptance Pa is

the same as the cumulative probability function of x. It is

calculated as

Accred Qual Assur

123

Pa ¼cðk=2; x2=2Þ

Cðk=2Þ ð9Þ

where k = n-1 is the degree of freedom and c (.) is the

lower incomplete gamma function.The probability of

acceptance Pa associated with the z/c4-based uncertainty is

Pa ¼ Pza=2

c4

ffiffiffi

np s � MPU

� �

ð10Þ

Substituting MPU ¼ MQI � za=2r=ffiffiffi

np

into Eq. (10) yields

Pa ¼ Ps

r�MQI � c4

� �

¼ Pffiffiffiffiffiffiffiffiffiffiffi

n� 1p s

r�MQI �

ffiffiffiffiffiffiffiffiffiffiffi

n� 1p

c4

� �

ð11Þ

When MQI C 1, all of the samples drawn from the mea-

surement population are supposed to be accepted according

to Eq. (2), and the probability of acceptance Pa should be

100 %. However, when using Eqs. (5) or (6), the proba-

bility of acceptance Pa will be less than 100 %. In this

situation, 1-Pa is the false rejection probability or the

probability of the Type I error. On the other hand, when

MQI \ 1, all of the samples are supposed to be rejected

according to Eq. (2), and the probability of acceptance Pa

should be zero. However, when using Eqs. (5) or (6), the

probability of acceptance Pa will not be zero. In this situ-

ation, Pa becomes the false acceptance probability or the

probability of the Type II error.

We first examine a case in which MQI [ 1. Figure 2

shows the probabilities of acceptance associated with the

t-based uncertainty and the z/c4-based uncertainty for

MQI = 1.3143 with the nominal coverage probability

1-a = 95 %.

In this case (MQI = 1.3143), the probability of accep-

tance should be 100 %. However, it can be seen from

Fig. 2 that the probability of acceptance associated with the

t-based uncertainty is very small for small samples, only

16.1 % at n = 2. It increases with increasing sample size

and the increase is rapid. The probability of acceptance is

76.8 % at n = 10 and approaches 100 % when n [ 60. On

the other hand, the probability of acceptance associated

with the z/c4-based uncertainty is 70.6 % at n = 2 and

90.1 % at n = 10; it approaches 100 % when n [ 60. Note

that in this case, 1-Pa is the false rejection probability. For

the samples of size 2, the t-based uncertainty would result

in 84 false rejections out of 100 samples; the z/c4-based

uncertainty would result in 30 false rejections out of 100

samples. The results indicate that the z/c4-based uncertainty

is conservative, whereas the t-based uncertainty is overly

conservative and even misleading when the sample size is

very small.

Next, we examine a case in which MQI \ 1. Figure 3

shows the probabilities of acceptance associated with the

t-based uncertainty and the z/c4-based uncertainty for

0

1

2

3

4

5N

orm

aliz

ed e

rror

||/

Normalized sample standard deviation s/

Rejected according to Eq. (6)

Rejected according to Eq. (5)

(a)

0

1

2

3

4

5

0 1 2 3 4

0 1 2 3 4

Nor

mal

ized

err

or |

|/

Normalized sample standard deviation s/

(b)

Rejected according to Eq. (5)

Rejected according to Eq. (6)

Fig. 1 Samples rejected (or accepted) according to Eqs. (5) and (6)

with MPU = 1.3143Uz at the 95 % coverage probability: (a) n = 2;

(b) n = 4. The dashed line is |e| = MPU, normalized by r

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100

Pro

babi

lity

of a

ccep

tanc

e P a

(%)

Sample size n

t-based uncertainty

z/c4-based uncertainty

Fig. 2 Probability of acceptance Pa for MQI = 1.3143. In this case,

1-Pa is the false rejection probability

Accred Qual Assur

123

MQI = 0.5 with the nominal coverage probability

1-a = 95 %.

In this case (MQI = 0.5), the probability of acceptance

should be zero. However, it can be seen from Fig. 3 that Pa is

not zero for small samples. It is 6.2 % at n = 2 and less than

1 % when n [ 7 for the t-based uncertainty. It is 31 % at

n = 2, 7.3 % at n = 10, and less than 1 % when n [ 10 for

the z/c4-based uncertainty. Note that in this case, Pa becomes

the false acceptance probability, and 1-Pa is the rejection

probability. For the samples of size 2, the t-based uncertainty

would result in 6 false acceptances out of 100 samples; the

z/c4-based uncertainty would result in 31 false acceptances

out of 100 samples. The results are expected because the

quality control based on the t-based uncertainty will be

always more conservative (overly conservative for very

small samples) than that based on the z/c4-based uncertainty.

Finally, we examine the case in which MQI = 1. Fig-

ure 4 shows the probabilities of acceptance associated with

the t-based uncertainty and the z/c4-based uncertainty for

MQI = 1 with the nominal coverage probability

1-a = 95 %.

In this case (MQI = 1), the probability of acceptance

should be 100 %. However, it can be seen from Fig. 4 that

the probability of acceptance associated with the t-based

uncertainty is only 12.3 % at n = 2. It increases with

increasing sample size, but the increase is slow. The

probability of acceptance is only 33.8 % at n = 10 and

45.1 % at n = 100. It approaches 50 % when the sample

size n goes to infinity. On the other hand, the probability of

acceptance associated with the z/c4-based uncertainty is

57.5 % at n = 2; it approaches 50 % as the sample size

increases. This is expected because the z/c4-based uncer-

tainty is an unbiased estimate of the z-based uncertainty.

When the sample size is large, the estimated uncertainty

will approach the z-based uncertainty and fluctuate about it.

That is, half of the total samples will have an estimated

uncertainty, that is, slightly greater than the z-based

uncertainty and the other half slightly smaller than the z-

based uncertainty. As a result of the fluctuation, the prob-

ability of acceptance becomes 50 %. Note again that, in

this case, 1-Pa is the false rejection probability. For the

samples of size 2, the t-based uncertainty would result in

88 false rejections out of 100 samples; the z/c4-based

uncertainty would result in 43 false rejections out of 100

samples.

However, if MQI is just slightly less than unity, the

discussions on the MQI = 1 case should be reversed. Pa

becomes the false acceptance probability, and 1-Pa

becomes the rejection probability. Thus, in the situation

that MQI is near unity, the false rejection probability is

significantly higher than the false acceptance probability

when using the t-based uncertainty for very small samples.

It becomes less significantly high for large samples. On the

other hand, the false rejection probability and the false

acceptance probability are about the same (i.e., about

50/50 %) when using the z/c4-based uncertainty regardless

of the sample size (except for very small samples).

We further examine the performance of the two sample-

based uncertainty estimators with fixed sample sizes.

Figure 5a, b, and c shows Pa as a function of MQI at

1-a = 95 % for n = 2, 4, and 32, respectively. Note again

that 1-Pa is the false rejection probability for MQI C 1,

and Pa is the false acceptance probability for MQI \ 1. It

can be seen from Fig. 5a that, for sample size 2, using the t-

based uncertainty will result in much higher false rejection

probabilities than the z/c4-based uncertainty. The false

rejection probability associated with the t-based uncer-

tainty is very high, as high as 64.4 %, even at MQI = 3.

The situation improves for sample size 4 (Fig. 5b). When

0

10

20

30

40

50

60

70

80

90

100

0 20 40 60 80 100

Pro

babi

lity

of a

ccep

tanc

e P a

(%)

Sample size n

t-based uncertainty

z/c4-based uncertainty

Fig. 4 Probability of acceptance Pa for MQI = 1. In this case, 1-Pa

is the false rejection probability

0

10

20

30

40

50

0 10 20 30 40 50

Pro

babi

lity

of a

ccep

tanc

e P a

(%)

Sample size n

t-based uncertainty

z/c4-based uncertainty

Fig. 3 Probability of acceptance Pa for MQI = 0.5. In this case, Pa

becomes the false acceptance probability and 1-Pa is the rejection

probability

Accred Qual Assur

123

the sample size is large, say, n = 32 (Fig. 5c), the risk in

making an incorrect decision associated with the t-based

uncertainty and the z/c4-based uncertainty gets closer.

In addition, it should be pointed out that the coverage

probability of ±MPU defined by Eq. (3) is the ‘‘true’’

confidence level an operator may have for a measured

value. Note that the coverage probability of ±MPU is

independent from a sample-based uncertainty estimator.

That is, whatever sample-based uncertainty estimator, with

any nominal coverage probability, is used, the coverage

probability of ±MPU is the same. Although the t-based

uncertainty has a greater coverage probability than the z/c4-

based uncertainty (assume the coverage probabilities are

estimated by the same random-interval procedure), it may

not warrant a higher level of confidence for making correct

decisions in measurement quality control. It should also be

pointed out that, in the z/c4-based uncertainty approach, the

reduction of false rejection probability (due to reduction of

coverage probability) is achieved on account of increasing

the risk of a false acceptance.

In practice, the preference of the t-based uncertainty or

the z/c4-based uncertainty may depend on a specific prob-

lem or application. It may be resolved based on an analysis

of expected losses resulting from false decisions (false

rejection or false acceptance). In ADCP streamflow mea-

surement quality control, there is no specific ‘‘loss’’

associated with a less accurate discharge data that are

falsely accepted. Moreover, our experiences indicate that,

in most situations, a false acceptance is unlikely because

the measurement quality index MQI associated with ADCP

streamflow measurements is often greater than unity. On

the other hand, if a measured discharge that is supposedly

accepted was falsely rejected, more repeated measurements

would be required, resulting in unnecessarily additional

costs of labor, time, and energy. Therefore, the z/c4-based

uncertainty is preferred in the quality control of ADCP

streamflow measurement.

Examples

This section presents several examples of quality evalua-

tion and control for ADCP streamflow measurements. First,

we analyze a large data set. Then, we analyze a number of

small data sets.

A large data set

A large data set for the ADCP streamflow measurements in

the Mississippi River on January 30, 1992, was available

from Gordon [11]. The data set contains 30 discharge

observations under a steady flow condition. The average

discharge of the 30 observations, 14240 m3/s, is assumed

to be the true discharge, and the standard deviation with a

bias correction, 223 m3/s, is assumed to be the population

standard deviation.

It should be pointed out that this large data set was from

an exceptional experiment. The ADCP streamflow mea-

surements under a steady flow condition usually involve

four observations [12, 13]. This large data set offers an

opportunity to evaluate the performance of the t-based

0

10

20

30

40

50

60

70

80

90

100P

roba

bilit

y of

acc

epta

nce

Pa

(%)

Measurement quality index (MQI )

t-based uncertainty

z/c4-based uncertainty

(a)

0

10

20

30

40

50

60

70

80

90

100

Pro

babi

lityo

f acc

epta

nce

P a(%

)

Measurement quality index (MQI )

t-based uncertainty

z/c4-based uncertainty

(b)

0

10

20

30

40

50

60

70

80

90

100

0 0.5 1 1.5 2 2.5 3

0 0.5 1 1.5 2 2.5 3

0 0.5 1 1.5 2 2.5 3

Pro

babi

lityo

f acc

epta

nce

P a(%

)

Measurement quality index (MQI)

(c)

t-based uncertainty

z/c4-based uncertainty

Fig. 5 Probability of acceptance Pa as a function of MQI at

1-a = 95 %: (a) n = 1, (b) n = 4, and (c) n = 32

Accred Qual Assur

123

uncertainty and the z/c4-based uncertainty for small sam-

ples as compared to the z-based uncertainty that is

available. We consider two and four observations (n = 2

and 4) only. We group the data in sequence into samples of

size 2 and 4 (n = 2 and n = 4), obtaining 29 and 27

samples, respectively. Note that the way that the data is

grouped does not produce independent samples. However,

the correlation between the samples is not a concern for the

measurement quality evaluation discussed here.

Since the population standard deviation is known, the

z-based uncertainty for each sample size can be calculated

and Eq. (2) can be used to evaluate the measurement

quality. The relative MPU for ADCP streamflow mea-

surement is 4.3 % [14]. The z-based uncertainty (relative to

the assumed true discharge) at the 95 % coverage proba-

bility is calculated as 3.06 % for n = 1, 2.17 % for n = 2

and 1.53 % for n = 4, all of which are smaller than the

relative MPU 4.3 %. Thus, any measured discharge (i.e.,

any sample, regardless of the number of observations)

meets Eq. (2) and is acceptable.

The measurement quality index is calculated as

MQI = 1.4052, 1.9816, and 2.8105, for n = 1, 2, and 4,

respectively. Accordingly, the coverage probabilities of the

relative ±MPU (4.3 %) are 99.4116, 99.9887, and

99.999998 %. Thus, as expected, the quality of the mea-

surement results get higher as the number of observations

increases.

We then pretend that the population standard deviation

is unknown and use Eqs. (5) and (6) to evaluate the mea-

surement quality. The t-based uncertainty and the z/c4-

based uncertainty (also relative to the assumed true dis-

charge) at the nominal coverage probability 95 % for

n = 2 and n = 4 are calculated. Figure 6a and b shows the

results. The relative z-based uncertainty at the coverage

probability 1-a = 95 % and the relative MPU 4.3 % are

also shown in the figures.

It can be seen from Fig. 6a that, for n = 2, there are 21

samples whose relative t-based uncertainties are greater

than the relative MPU 4.3 %. Therefore, using the t-based

uncertainty for the measurement quality control will result

in 21 false rejections. On the other hand, there are three

samples whose relative z/c4-based uncertainties are greater

than the relative MPU 4.3 %. Therefore, using the

z/c4-based uncertainty will result in 3 false rejections. The

results indicate that the z/c4-based uncertainty is conser-

vative, whereas the t-based uncertainty is overly

0

5

10

15

20

25

30

35

40R

elat

ive

unce

rtai

nty

(%)

Sample (n=2)

t-based uncertaintyz/c4-based uncertaintyz-based uncertainty: 2.17 %MPU: 4.3 %

(a)

0

1

2

3

4

5

6

7

8

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

Rel

ativ

e un

cert

aint

y (%

)

Sample (n=4)

t-based uncertaintyz/c4-based uncertaintyz-based uncertainty: 1.53 %MPU: 4.3 %

(b)

0 2 4 6 8 10 12 14 16 18 20 22 24 26 28 30 32

Fig. 6 Measurement quality evaluation based on uncertainty analysis

for Mississippi River ADCP discharge measurements: (a) n = 2 and

(b) n = 4

Table 1 Data sets of ADCP streamflow measurements at 7 sites

Site

#

River site Observations

(discharge in m3/s)

#1 #2 #3 #4

1a Yangtze River at Yichang,

China (300 kHz ADCP)

11234 11582 11485 11476

1b Yangtze River at Yichang,

China (150 kHz ADCP)

11092 10980 11367 11319

2a Aksu River, Antalya, Turkey

(600 kHz ADCP)

38.88 40.30 38.23 39.78

2b Aksu River, Antalya, Turkey

(2 MHz ADCP)

37.96 38.78 38.85 39.05

3 Manavgat River, Antalya,

Turkey (600 kHz ADCP)

228.7 225.9 233.1 228.4

4 Kopru River, Antalya, Turkey

(600 kHz ADCP)

96.60 96.53 93.90 91.79

5 Irrigation Canal at Imperial

Irrigation District, California

(2 MHz ADCP)

3.99 4.02 4.05 4.14

6 Irrigation Canal at Angoori

Barrage, India (2 MHz

ADCP)

22.33 22.07 22.3 22.19

7 River Elbe near Hamburg,

Germany (1200 kHz ADCP)

4588 4576 4631 4517

The author involved in the data collection at Sites 1, 5, and 6. The

data sets for Sites 2, 3, and 4 are from Michel [15]; the data sets for

Site 7 are from Gordon [16]

Accred Qual Assur

123

conservative and even misleading because it results in so

many false rejections. This is consistent with the theoreti-

cal analysis for the probability of acceptance.

It can be seen from Fig. 6b that, for n = 4, all of the

relative t-based or z/c4-based uncertainties are smaller than

the relative MPU 4.3 %, yielding 100 % acceptance. The

results indicate that both of the t-based uncertainty and the

z/c4-based uncertainty become less conservative for n = 4

than n = 2. This is also consistent with the theoretical

analysis for the probability of acceptance.

Small data sets

Table 1 shows 9 data sets (each contains four observations)

of the ADCP streamflow measurements at 7 sites. Since the

population standard deviation is unknown at these sites,

Eq. (2) cannot be used for the measurement quality eval-

uation and control. Instead, Eqs. (5) and (6) are used.

Table 2 shows the sample mean �X and standard deviation

s, the z/c4-based uncertainty Uz/c4, and the t-based uncer-

tainty Ut for the first two observations (n = 2) and those for

the total four observations (n = 4). The uncertainties are

shown in percentage, relative to the sample means.

It can be seen from Table 2 that, for four observations

(n = 4), the z/c4-based uncertainties and the t-based

uncertainties are all smaller than the relative MPU 4.3 %;

therefore, all samples are acceptable. However, for two

observations (n = 2), one out of the 9 z/c4-based uncer-

tainties and 7 out of the 9 t-based uncertainties are greater

than the relative MPU 4.3 %. These are apparently the

false rejections.

The above examples suggest that the z/c4-based uncer-

tainty is in general conservative for measurement quality

control. It is appropriate even if the sample size is only two.

The t-based uncertainty is overly conservative and may be

misleading for very small samples (e.g., many false

rejections for the two observations).

The above examples also suggest that two observations

may be enough to meet the relative MPU 4.3 % criterion

for the ADCP streamflow measurements, and it is unnec-

essary to make four observations. Considering the fact that

hydrologists around the world may conduct hundreds of

thousands of ADCP streamflow measurements each year, it

will lead to significant savings in labor and time, and in gas

consumption (many measurements are conducted using gas

engine boats) if the number of observations is reduced to

two.

Conclusions

The uncertainty-based measurement quality control cannot

be deterministic unless the z-based uncertainty is available.

The quality control will be uncertain when a sample-based

uncertainty estimator is used. The false rejection proba-

bility and the false acceptance probability depend on the

uncertainty estimator used. A good estimator should pro-

vide a balance between the false rejection and false

acceptance, i.e., the balance between making the Type I

and Type II errors.

Both of the theoretical analyses and examples suggest

that the measurement quality control based on the t-based

uncertainty is overly conservative and may be misleading

when the sample size is very small. This is because the

t-based uncertainty is not an unbiased estimate of the

z-based uncertainty and exhibits significantly high positive

bias error and precision error when the sample size is very

small. Therefore, the t-based uncertainty is inappropriate

for measurement quality control for small samples. On the

other hand, the measurement quality control based on the

z/c4-based uncertainty is, in general, conservative. For some

applications, the z/c4-based uncertainty may be superior to

the t-based uncertainty. The reason why the z/c4-based

uncertainty is appropriate, even for very small samples, is

Table 2 Sample mean and standard deviation, and estimated uncertainties (at the nominal coverage probability 95 %) for the data sets shown in

Table 1

Site # Two observations (n = 2) Four observations (n = 4)

�X (m3/s) s (m3/s) Uz/c4 (%) Ut (%) �X (m3/s) s (m3/s) Uz/c4 (%) Ut (%)

1a 11408 246 3.75 19.38 11444 148 1.38 2.06

1b 11036 79.5 1.25 6.47 11189 184.2 1.75 2.62

2a 39.6 1.00 4.40 22.78 39.3 0.92 2.50 3.73

2b 38.4 0.58 2.62 13.53 38.7 0.48 1.32 1.98

3 227.3 1.92 1.47 7.60 229.0 2.97 1.38 2.07

4 96.6 0.05 0.09 0.48 94.7 2.31 2.60 3.89

5 4.01 0.021 0.92 4.76 4.05 0.065 1.70 2.55

6 22.2 0.18 1.44 7.44 22.2 0.12 0.57 0.85

7 4582 8.49 0.32 1.66 4578 47.02 1.09 1.63

Accred Qual Assur

123

that the z/c4-based uncertainty is an unbiased estimate of

the z-based uncertainty.

Very small samples, even the samples of size 2, are still

useful when the z/c4-based uncertainty is employed for

measurement quality control. In contrast, the t-based

uncertainty basically rules out the usefulness of very small

samples, especially the samples of size 2. The use of the

z/c4-based uncertainty for measurement quality control

makes it possible to make reasonable, less incorrect deci-

sions based on a small number of observations. This may

lead to a potential reduction in the number of observations

required for measurement quality assurance and control,

which may have a great significance for time and labor

consuming or costly measurements or experiments.

Acknowledgments The author would like to thank the anonymous

reviewers for their valuable comments that helped improve the paper.

The author also would like to thank Mark Kuster of Pantex Metrology

for making the copy of the Jenkins paper available to the author and

for stimulating discussions.

References

1. Joint Committee for Guides in Metrology (JCGM) (2008) Eval-

uation of measurement data—Guide to the expression of

uncertainty in measurement (GUM 1995 with minor corrections).

Sevres, France. http://www.bipm.org/utils/common/documents/

jcgm/JCGM_100_2008_E.pdf

2. Huang H (2010) A paradox in measurement uncertainty analysis.

Presented at the 2010 Measurement Science Conference in Pas-

adena, CA, Mar 25–26. In: Global Measurement: Economy and

Technology 1970–2010 Proceedings (DVD)

3. Huang H (2012) Comparison of uncertainty calculation models.

Cal Lab Int J Metrol 19(1):24–29

4. Joint Committee for Guides in Metrology (JCGM) (2012) Evalu-

ation of measurement data—The role of measurement uncertainty

in conformity assessment. Sevres, France. http://www.bipm.org/

utils/common/documents/jcgm/JCGM_106_2012_E.pdf

5. Pendrill LR (2007) Optimized measurement uncertainty and

decision-making in conformity assessment. NCSLI Meas

2(2):76–86

6. Pendrill LR (2008) Operating ‘cost’ characteristics in sampling

by variable and attribute. Accred Qual Assur 13:619–631

7. Jenkins JD (2007) The Student’s t-distribution uncovered. Mea-

surement Science Conference Proceedings, Long Beach

8. Wadsworth HM Jr (1989) Summarization and interpretation of

data. In: Wadsworth H M Jr (ed) Handbook of statistical methods

for Engineers and Scientists, 2.1–2.21. McGraw-Hill, New York

9. Huang H (2006) Analysis of random error in ADCP discharge

measurement I: model for predicting random uncertainty.

J Hydraul Eng 37(5):619–624 (in Chinese)

10. Craig CC (1927) Remarks on the probable error of a mean. Am

Math Mon 34(9):472–476

11. Gordon RL (1992) Mississippi river discharge. RD Instruments

Report, San Diego, California

12. Mueller DS, Wagner CR (2002) Measuring discharge with

acoustic Doppler current profilers from a moving boat. USA.

Geological Survey Techniques and Methods, 3A-22, 72

13. Oberg KA, Morlock SE, Caldwell WS (2005) Quality-assurance

plan for discharge measurements using acoustic Doppler current

profilers. Scientific investigations rep. 2005-5183, U.S. Geolog-

ical Survey (USGS), Reston, VA

14. Huang H (2008) Estimating precision of moving boat ADCP dis-

charge measurement. In: Proceedings (DVD) of 14th Australian

Hydrographers Association Conference: Water Initiatives 2008

and Beyond, Australia Hydrographers Association, Northcote

15. Michel L (2011) Ekoton, Turkey Visit Report. Teledyne RD

Instruments Europe, La Gaude, France

16. Gordon RL (1989) Acoustic measurement of river discharge.

J Hydraul Eng ASCE 115(7):925–936

Accred Qual Assur

123