The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia;...
-
Upload
monica-baldwin -
Category
Documents
-
view
219 -
download
1
Transcript of The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia;...
![Page 1: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/1.jpg)
The Best Method of Noise Filtering
Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia;
Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,
Moscow, Russia
1
![Page 2: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/2.jpg)
History: Adaptive peak approximation
2
![Page 3: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/3.jpg)
Rough slope width estimate
• Evaluate baseline using default gap (minimum peak width Integration parameter)
• Evaluate peak height using default gap
• Count all points from peak apex to slope end with height bigger than half-height of the peak. Count obtained is an estimate of the slope width.
3
![Page 4: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/4.jpg)
Properties of adaptive peak approximation
4
• Good noise suppression at each slope• Minimal peak shape disturbances• All peak parameters are resistant to oversampling • Baseline approximation may be poor – either noisy
(small gap) or disturbed (large gap).• No approximation outside of peaks• Does not improve formal signal/noise ratio• Baseline position is one of the most important
sources of error
![Page 5: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/5.jpg)
Improvement 1: Non-central approximation
5
*x
2G
1G
![Page 6: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/6.jpg)
Confidence intervals
6
Hei
ght
Concentration
0 1 2 3 4 5 6 7 8 9 100
2
4
6
8
123
456
789
![Page 7: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/7.jpg)
Confidence interval estimate
7
where
n - number of data points used for polynomial approximation (gap of the filter);
p - power of the polynomial; X - matrix of x power values on independent axis (time);Y - vector of detector response values;
- Student’s coefficient for confidence probability (1-δ) and m degrees of freedom
x* - position at which smoothed (approximated) value is estimated.
*
)2/1( uStC pnY
pnS
)ˆ()ˆ(2 βXYβXY
*1
* xX)X(x *u
YXXXβ 1)(ˆ mt
},...,,1{ **pxx*x
![Page 8: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/8.jpg)
Approximation using confidence intervals
8
*x
2G
*x
2G confidence interval
1 G
![Page 9: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/9.jpg)
Algorithm of simple Confidence filter approximation
9
• Evaluate points and confidence intervals for new (shifted) window
![Page 10: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/10.jpg)
Algorithm of simple Confidence filter approximation
10
• Evaluate points and confidence intervals for new (shifted) window
• Compare new confidence interval with that for previously evaluated point. If the new one is smaller than previous, replace approximated point and its confidence interval.
![Page 11: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/11.jpg)
Algorithm of simple Confidence filter approximation
11
• Evaluate points and confidence intervals for new (shifted) window
• Compare new confidence interval with that for previously evaluated point. If the new one is smaller than previous, replace approximated point and its confidence interval.
• Computational complexity of Confidence filter is comparable to that of simple convolution, (e.g. Savitzky-Golay) and linearly depends on the product gap (degree of the polynomial)∙ .
![Page 12: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/12.jpg)
Bonus #1: Correct handling of baseline steps and array boundaries
12
dotted – raw data; thick line – Confidence Filter; thin line – Savitzky-Golay filter
340 350 360 370 380 390 400 410 420 430 440 450 460 470 480 490 500 Nmeas
0
500
1000
1500
2000
mv
OriginalSGASG
![Page 13: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/13.jpg)
Confidence filter algorithm improvement: Adaptive gap of the polynomial
13
• Repeat confidential filter algorithm for approximations with different windows (gaps)
• Computational complexity: degree gap (gap-1)/2∙ ∙
• Logarithmic step: next gap is k times smaller, than previous, e.g. gap2 = gap1/k, k>1; Computational complexity: degree gap k/(k-∙ ∙1)
![Page 14: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/14.jpg)
Confidence interval estimate
14
where
n - number of data points used for polynomial approximation (gap of the filter);
p - power of the polynomial; X - matrix of x power values on independent axis (time);Y - vector of detector response values;
- Student’s coefficient for confidence probability (1-δ) and m degrees of freedom
x* - position at which smoothed (approximated) value is estimated.
*
)2/1( uStC pnY
pnS
)ˆ()ˆ(2 βXYβXY
*1
* xX)X(x *u
YXXXβ 1)(ˆ mt
},...,,1{ **pxx*x
![Page 15: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/15.jpg)
t(df) for confidence probability 0.975
15
![Page 16: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/16.jpg)
Confidence interval profiles for different slits (degree = 3)
16
-15 -14 -13 -12 -11 -10 -9 -8 -7 -6 -5 -4 -3 -2 -1 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 150.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1
![Page 17: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/17.jpg)
Confidence Interval profiles, 31 points, 0…5 degrees
17
![Page 18: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/18.jpg)
σ evaluation problems:
18
• Small gaps: accidental perfect fit• Large gaps: treating small peaks as a noise due to
large number of degrees of freedom• Is pump pulsation a noise or a signal? • Small gaps: confidence interval depends on
confidence level
σ evaluation solutions:• Evaluate in advance using the whole data array• Use the estimate for evaluation of confidence
intervals
![Page 19: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/19.jpg)
Handling σ estimate
19
*)2/1( uStC pnY
*½ utC pnY
)(,
)(,22
22
bFormulaS
aFormulaSCY
),(222 pnSR
![Page 20: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/20.jpg)
20
Noise Filtering: How it works 1
20 21 22 23 24 25 min
-20
0
20
Shift21
22
23 24 25
26
27
20 21 22 23 24 25 min0
10
20
Gap21
22
23
24
25
26 27
20 21 22 23 24 25 min
-0.005
-0.004
AU
280nm
21
22
23
2526 27
Smoo280
![Page 21: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/21.jpg)
21
Noise Filtering: How it works 2
![Page 22: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/22.jpg)
Automatic selection of degree and gap of
approximating polynomial
22
Raw
0…3
1…3
2...3
3
3…10
33 34 35 36 37 38 39 40 41 min
0.0784 mV
ch1
33 34 35 36 37 38 39 40 41 min
0.0733 mV
ch1
33 34 35 36 37 38 39 40 41 min
0.075 m V
ch1
33 34 35 36 37 38 39 40 41 min
0.0773 mV
ch1
33 34 35 36 37 38 39 40 41 min
0.0784 mV
ch1
33 34 35 36 37 38 39 40 41 m in
0.0784 m V
ch1
![Page 23: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/23.jpg)
Is pump pulsation a noise or a signal?
23
![Page 24: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/24.jpg)
Conclusions:
24
• Confidence filter introduces a measure of approximation quality
• Confidence filter helps to select the best set of functions that approximate the data set
• Confidence filter is metrologically the best noise filtering method and can be used in the fight with legal metrology
Patent pending
![Page 25: The Best Method of Noise Filtering Yuri Kalambet, Sergey Maltsev, Ampersand Ltd., Moscow, Russia; Yuri Kozmin, Shemyakin Institute of Bioorganic Chemistry,](https://reader033.fdocuments.net/reader033/viewer/2022052603/56649f395503460f94c5604f/html5/thumbnails/25.jpg)
Thank you!
25