How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.
-
Upload
maria-russell -
Category
Documents
-
view
217 -
download
1
Transcript of How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.
![Page 1: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/1.jpg)
How to use the best of your QA data
Dubrovnik Course in Zagreb 2015
Gunnar Nordin
![Page 2: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/2.jpg)
The topics
Evaluate your QC results! Be interested. Discuss them at the lab and with friends.
Why are my, and not others, results out of limits?
“The EQA material is not commutable”“The assigned target value is not correct”“The EQA material is not stable”
There are too many ways to express “accuracy”
![Page 3: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/3.jpg)
Standard for the evaluation of a participant in an EQA scheme
![Page 4: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/4.jpg)
Standard for assessment of IVD procedures
![Page 5: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/5.jpg)
A flowshart for deviating EQA-results
![Page 6: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/6.jpg)
A flowshart for deviating EQA-results
![Page 7: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/7.jpg)
A flowshart for deviating EQA-results
![Page 8: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/8.jpg)
EQA material should be commutable
Zhang et al, Commutability of Possible External Quality AssessmentMaterials for Cardiac Troponin Measurement, 2014
![Page 9: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/9.jpg)
EQA material should be stable
![Page 10: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/10.jpg)
ISO 13528:2015
41
42
43
44
45
46
day 1 day 3
resu
lt, g
/L
test day
Stability tested by repeated measurments at one site at the day of distribution and at the last day for the round.
� Standard error
<0.3σPT
![Page 11: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/11.jpg)
The target value for a EQA material should be a true value
1. Target value known by formulation
2. Target value from by reference measurement procedure
3. Target value by ’expert laboratory methods’
4. Target value by consensus mean
![Page 12: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/12.jpg)
The consensus value is often not a true value
Consensus value
![Page 13: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/13.jpg)
”Mean of method mean” (MOMM)
Mean of the major and homogenous method group mean values
Deviating method groups are excluded
Small method groups are excluded
![Page 14: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/14.jpg)
14
”Mean of method mean” (MOMM)
50 55 60 65 70 75 80 85 90 95 100 105 110 115 120
P-Creatin ine, µmol/L
Num
ber
of o
bser
vatio
ns
5M AD
5% of observations
![Page 15: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/15.jpg)
Do you think MOMM is a more realiable target value than the consensus mean?
Yes ?
No ?
15
”Mean of method mean” (MOMM)
![Page 16: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/16.jpg)
Uncertainty is clearly defined
![Page 17: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/17.jpg)
What is accuracy?
”more” or ”less” are results on an ordinal scale
![Page 18: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/18.jpg)
Is it possible to measure accuracy?
Yes
No
![Page 19: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/19.jpg)
One type of measure of accuracy
Fraction of results within quality specifications
A pragmatic approach for POCT HbA1c in Sweden:
Max 2 of 10 results outside the limits
![Page 20: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/20.jpg)
Accuracy and bias reported to participants in Equalis HbA1c scheme
![Page 21: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/21.jpg)
Accuracy as ”fraction of values within” denoted ”P30” for accuracy of eGFR
Stevens, 2008
![Page 22: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/22.jpg)
Equalis accuracy graphs
![Page 23: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/23.jpg)
Accuracy described as ”total error”the simple model TEa = bias(%) + 1.65 x CV
0
2
4
6
8
10
12
-20 -10 0 10 20
Rand
om e
rror
(CV%
)
bias (%)
Total error; 10, 15 and 20% TEa
Contour NEXT USB (n=6)
Contour XT (n=5)
MedCore Care (n=21)
fig 2
![Page 24: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/24.jpg)
Accuracy described on the ”six sigma” scale
0
2
4
6
8
10
12
-20 -10 0 10 20
Rand
om e
rror
(CV%
)
bias (%)
Total error 15 % and sigma scale
Contour NEXT USB (n=6)
Contour XT (n=5)
MedCore Care (n=21)
sigma 2
sigma 3
sigma 4
![Page 25: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/25.jpg)
Accuracy (or total error?) with the root mean square concept (RiliBÄK)
![Page 26: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/26.jpg)
Accuracy described as ”long term uncertainty measurement” (LTUM)
Matar et al, 2015
![Page 27: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/27.jpg)
mean absolute relative deviation (MARD) – a new accuracy metric
Much used to describe performance of continous glucose measurment (CGM)
In case of no bias MARD ≈ 0,8 x CV
No additional informationcompared with “p15”
0%
2%
4%
6%
8%
10%
12%
14%
60% 65% 70% 75% 80% 85% 90% 95% 100%
MA
RD
p15
bias 0
bias 0.5
bias 1
![Page 28: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/28.jpg)
Alternative expressions for accuracy
1. Something that can only be good or bad?
2. A fraction of results within performance specification?
3. The “total error”
4. The root mean square of bias and precision
5. The long term uncertainty measurement (LTUM)
6. The mean absolute relative deviation (MARD)
![Page 29: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/29.jpg)
Thank you for listening
![Page 30: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/30.jpg)
30
![Page 31: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/31.jpg)
Do you think MOMM is a more realiable target value than the consensus mean?
A. Yes
B. No
YesNo
0%0%
![Page 32: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/32.jpg)
Is it possible to measure accuracy?
A. Yes
B. No
YesNo
50%50%
![Page 33: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/33.jpg)
Which expression for accuracy do you prefer?
A. Something that can only be good or bad?
B. A fraction of results within performance specification?
C. The “total error”
D. The root mean square of bias and precision
E. The long term uncertainty measurement (LTUM)
F. The mean absolute relative deviation (MARD)
G. The question is wrong, beacuse these terms do cover different concepts and should be given different names
A. B. C. D. E. F. G.
14% 14% 14% 14%14%14%14%
![Page 34: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/34.jpg)
How should target values be assigned to commutable EQA materials?
A. Reference method values
B. Value transfer from certified reference materials
C. Consensus mean from all participants
D. Mean of method mean (MOMM)
E. Method group mean values should be used
A. B. C. D. E.
20% 20% 20%20%20%
![Page 35: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/35.jpg)
Some words on results that are not quantitative
![Page 36: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/36.jpg)
36
![Page 37: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/37.jpg)
1. Semi quantitative test
2. Classification test
3. Binary test
4. Test with results ”yes/no”
5. Test with results on an ordinal scale
37
What is a ”qualitative test” ?
![Page 38: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/38.jpg)
38
Performance characterstics for pregnancy tests
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
0 1 10 100 1 000 10 000 100 000
frac
tion
of p
ositi
v res
ults
hCG(IU/L)
Performance in EQA for two pregnancy tests
Test A 20 IU/L
Test B 40 IU/L
c5
c95
c50
Unreliability zone
![Page 39: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/39.jpg)
39
How to draw the line….
Hyltoft Petersen, P., et al. (2008) Scand J Clin Lab Invest 68(4): 298-311.
![Page 40: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/40.jpg)
40
How to draw the line….
-3
-2
-1
0
1
2
3
0,1 1 10 100 1000 10000 100000
Rank
it (z
)
hCG(IU/L)
Performance in EQA for two pregnancy tests
Test A 20 IU/L
Test B 40 IU/L
![Page 41: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/41.jpg)
41
Proposal 2: c5, c50 an c95
For ordinal binary test with a quantitative back ground scale assays should be characterized
with the three quantities:
“c5”, ”c50” and ”c95”.
The manufacturer should declare the c50 value for the assay, and describe the metrological traceability
![Page 42: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/42.jpg)
<c5 c5 – c95 >c95
False pos - True pos
True neg - False neg
42
Internal quality assessment for ordinal tests
![Page 43: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/43.jpg)
<c5 c5 – c95 >c95
False pos True pos ? True pos
True neg True neg ? False neg
43
External quality assessment for ordnal tests
![Page 44: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/44.jpg)
44
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 000 10 000 100 000 1 000 000 10 000 000 100 000 000
And
el p
ositi
va re
sulta
t
Koncentration grupp A streptokocker (CFU/mL)
QuickVue Dipstick Strep ATest A
Empirical, ”state of the art” limits for
performance
![Page 45: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/45.jpg)
45
How do we report EQA results on ordinal scale
![Page 46: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/46.jpg)
46
Tests with deviating performance can be revealed
0%
10%
20%
30%
40%
50%
60%
70%
80%
90%
100%
1 000 10 000 100 000 1 000 000 10 000 000 100 000 000
And
el p
ositi
va re
sulta
t
Koncentration grupp A streptokocker (CFU/mL)
Analyz Strep ATest B
![Page 47: How to use the best of your QA data Dubrovnik Course in Zagreb 2015 Gunnar Nordin.](https://reader036.fdocuments.net/reader036/viewer/2022062519/5697bff11a28abf838cbb720/html5/thumbnails/47.jpg)
Take a home message
All QC results must be evaluated and discussed.
If deviating QC results are not related the performance of the method, it might be due to properties of the material used. Some examples are:
Non commutable materialIncorrect target value for the materialUnstable material
The measure of method performance varies. Unfortunately there are today many different measures for “accuracy”. We need to agree on a common use of the concept accuracy.