Sensors, Measurement systems, Signal processing and...
Transcript of Sensors, Measurement systems, Signal processing and...
![Page 1: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/1.jpg)
.
Sensors, Measurement systemsSignal processing and Inverse problems
Ali Mohammad-DjafariLaboratoire des Signaux et Systemes,
UMR8506 CNRS-SUPELEC-UNIV PARIS SUD 11SUPELEC, 91192 Gif-sur-Yvette, France
http://lss.supelec.free.fr
Email: [email protected]://djafari.free.fr
Files: http://djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014_01.pdf
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 1/118
![Page 2: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/2.jpg)
Contents
1. Sensors and Measurement systems Direct and indirect measurement sensors Primary sensor characteristics
2. Basic sensors design and their mathematical models R, L, C models and equations Forward model and simulation
3. Basic signal and image processing of the sensors output Averaging, Convolution Fourier Transform, Filtering
4. Indirect measurement and inverse problems Deconvolution X ray Computed Tomography Ultrasound, Microwave and Eddy current NDT
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 2/118
![Page 3: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/3.jpg)
Contents
5. Inversion methods Analytical methods Algebraic methods Regularization
6. Bayesian estimation approach Basics of Bayesian estimation Bayesian inversion
. Multivariate data analysis Principal Component Analysis (PCA) Independent Component Analysis (ICA)
8. Blind sources separation Classical methods Bayesian approach
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 3/118
![Page 4: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/4.jpg)
1. Sensors and measurement systems
Direct and indirect measurement
Direct measurement: Length, Time, Frequency
Indirect measurement: All the other quantities
Temperature Sound Vibration Position and Displacement Pressure Force ... Resistivity, Permeability, Permittivity, Magnetic inductance Surface, Volume, Speed, Acceleration ...
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 4/118
![Page 5: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/5.jpg)
Sensors and measurement systemsDifferent sensors:
Fluid Property Sensors
Force Sensors
Humidity Sensors
Mass Air Flow Sensors
Photo Optic Sensors
Piezo Film Sensors
Position Sensors
Pressure Sensors
Scanners and Systems
Temperature Sensors
Torque Sensors
Traffic Sensors
Vibration Sensors
Water Resources Monitoring
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 5/118
![Page 6: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/6.jpg)
Sensors and measurement systems
Main Glossary and Nomenclature:
Sensor:Primary sensing element(example: thermistor which translates changes in temperatureto changes to resistance)
Transducer:Changes one instrument signal value to another instrumentsignal value(example: resistance to volts through an electrical circuit)
Transmitter:Contains the transducer and produces an amplified,standardized instrument signal(example: A/D conversion and transmission)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 6/118
![Page 7: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/7.jpg)
Primary sensor characteristics
Range:The extreme (min and max) values over which the sensors canmake correct measurement over controlled variable.
Response time:The amount of time required for a sensor to completelyrespond to a change in its input.
Accuracy:Closeness of the sensor output to indicating the actual valueof the measured variable.
Precision:The consistency of the sensor output in measuring the samevalue under the same operating conditions over a period oftime.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 7/118
![Page 8: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/8.jpg)
Primary sensor characteristics
Sensitivity:The minimum small change in the controlled variable that thesensor can measure.
Dead band:The minimum amount of a change to the process which isrequired before the sensor responds to the change.
Costs:Not simply the purchase cost, but also the installed/operatingcosts?
Installation problems:Special installation problems, e.g., corrosive fluids, explosivemixtures, size and shape constraints, remote transmissionquestions, etc.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 8/118
![Page 9: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/9.jpg)
Signal transmission
Pneumatic:Pneumatic signals are normally 3-15 pounds per square inch(psi).
Electronic:Electronic signals are normally 4-20 milliamp (mA).
Optic:Optical signals are also used with fiber optic systems or whena direct line of sight exists.
Hydraulic
Radio
Glossary:http://lorien.ncl.ac.uk/ming/procmeas/glossary.htm
http://www.sensorland.com/GlossaryPage001.htmlhttp://www.sensorland.com/
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 9/118
![Page 10: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/10.jpg)
2. Basic sensors designs and their mathematical models
We can easily measures electrical quantities: Resistance: U = RI or u(t) = Ri(t) Capacitance: ∂u(t)
∂t= 1
Ci(t) or i(t) = C ∂u(t)
∂t
Inductance: u(t) = L∂i(t)∂t
Sensors and transducers are used to convert many physicalquantities to changes in R, C or L.
Resistance: Resistive Temperature Detectors (Thermistors) Strain Gauges (Pressure to resistance)
Capacitance: Capacitive Pressure Sensor
Inductance: Inductive Displacement Sensor
Thermoelectric Effects: Temperature Measurement
Hall Effect: Electric Power Meter
Photoelectric Effect: Optical Flux-meter
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 10/118
![Page 11: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/11.jpg)
Resistivity/Conductivity
Resistance R : R = ρ l/s (Ohm) ρ: Resistivity ohm/meter 1/ρ: conductivity Siemens/meter l: length meter s: section surface meter2
Dipole model:u(t) = R i(t)
Impedance
U(ω) = RI(ω) −→ Z(ω) =U(ω)
I(ω)= R
Power dissipation
P (t) = R i2(t) = u2(t)/R
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 11/118
![Page 12: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/12.jpg)
Capacity C
Capacitance: C = QU = ε0
ΦU (Farads)
Q Electric charge (coulombs) U Potential (volts) ε0 Electrical permittivity Φ Electric charge flux (weber)
Dipole model:
u(t) =1
C
∫ t
0i(t′) dt′
∂u(t)
∂t=
1
Ci(t) or i(t) = C
∂u(t)
∂t
I(ω) = jωC U(ω)
Impedance
Z(ω) =1
jωC
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 12/118
![Page 13: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/13.jpg)
Inductance L
Inductance: L = ΦI (Henri)
Φ Magnetic flux (Weber) I Current (Amp)
Dipole model (Faraday) :
u(t) = L∂i(t)
∂t
U(ω) = jωL I(ω)
Impedance
U(ω) = jωL I(ω) −→ Z(ω) = jω L
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 13/118
![Page 14: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/14.jpg)
Measuring R, C and L
Measuring R: Simple voltage divider
Bridge measurement systems Single-Point Bridge Two-Point Bridge (Wheatstone Bridge) Four-Point Bridge
Measuring C and L AC voltage dividers and Bridges
(Maxwell Bridge) Resonant circuits
(R L C circuits)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 14/118
![Page 15: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/15.jpg)
Measuring R
Wheatstone bridge:
At the point of balance:
R2
R1=
Rx
R3⇒ Rx =
R2
R1·R3
VG =
(Rx
R3 +Rx− R2
R1 +R2
)Vs
See Demo here:http://www.magnet.fsu.edu/education/tutorials/java/wheatstonebridge/index.html
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 15/118
![Page 16: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/16.jpg)
Measuring R
The Wien bridge:
At some frequency, the reactanceof the series R2C2 arm will bean exact multiple of the shuntRxCx arm. If the two R3 and R4
arms are adjusted to the same ratio,then the bridge is balanced.
ω2 =1
RxR2CxC2and
Cx
C2=
R4
R3− R2
Rx.
The equations simplify if one chooses R2 = Rx and C2 = Cx;the result is R4 = 2R3.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 16/118
![Page 17: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/17.jpg)
Measuring C
Maxwell Bridge:
R1 and R4 are known fixed entities. R2 and C2 are adjusteduntil the bridge is balanced.
R3 =R1 ·R4
R2−→ L3 = R1 ·R4 · C2
To avoid the difficulties associated with determining theprecise value of a variable capacitance, sometimes afixed-value capacitor will be installed and more than oneresistor will be made variable.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 17/118
![Page 18: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/18.jpg)
Forward modeling and simulation of circuits
Input-Output model
R-C and R-L circuits
L-C and R-L-C circuits
Transfert function and impulse response
General linear systems
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 18/118
![Page 19: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/19.jpg)
Forward modeling and simulation of circuitsInput-Output model: Examples of R− C circuits:
−−−
f(t)
−−−
−−R−−−−−|C|
− − − −−−−−
−−−
g(t)
−−−
∂g(t)
∂t=
1
Ci(t), i(t) =
(f(t)− g(t))
R
∂g(t)
∂t=
1
RC(f(t)− g(t)) −→ g(t) +RC
∂g(t)
∂t= f(t)
G(ω) +RCjωG(ω) = F (ω) −→ H(ω) =G(ω)
F (ω)=
1
1 + jRCω
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 19/118
![Page 20: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/20.jpg)
Forward modeling and simulation of circuitsInput-Output model: Examples of R− L circuits
−−−
f(t)
−−−
−−R−−−−−|L|
− − − −−−−−
−−−
g(t)
−−−
L∂i(t)
∂t= g(t), i(t) =
(f(t)− g(t))
R
L
R
∂(f(t)− g(t))
∂t= g(t) −→ g(t) +
L
R
∂g(t)
∂t=
L
Rjωf(t)
G(ω) +L
RjωG(ω) =
L
RF (ω) −→ H(ω) =
G(ω)
F (ω)=
1 + jωL/R
L/R
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 20/118
![Page 21: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/21.jpg)
Forward modeling and simulation of circuitsL− C circuits
−−−
f(t)
−−−
−− L−−−−−−|C|
− −− −−−−−−
−−−
g(t)
−−−
∂g(t)
∂t=
1
Ci(t), L
∂i(t)
∂t= (f(t)− g(t))
LC∂2g(t)
∂t2= (f(t)− g(t)) −→ g(t) + LC
∂2g(t)
∂t2= f(t)
G(ω)− LCω2G(ω) = F (ω) −→ H(ω) =G(ω)
F (ω)=
1
1− LCω2
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 21/118
![Page 22: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/22.jpg)
Forward modeling and simulation of circuitsR − L−C circuits
−−−
f(t)
−−−
−R−−L−−−−−|C|
− − − −−− −−−
−−−
g(t)
−−−
∂g(t)
∂t=
1
Ci(t), Ri(t) + L
∂i(t)
∂t= (f(t)− g(t))
RC∂g(t)
∂t+ LC
∂2g(t)
∂t2= (f(t)− g(t))
g(t) +RC∂g(t)
∂t+ LC
∂2g(t)
∂t2= f(t)
G(ω)+RCjωG(ω)−LCω2G(ω) = F (ω) −→ H(ω) =G(ω)
F (ω)=
1
1 + jRCω − LCω2
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 22/118
![Page 23: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/23.jpg)
Resonant circuits
The resonant pulsation is:
ω0 =
√1
LC
which gives:
f0 =ω0
2π=
1
2π√LC
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 23/118
![Page 24: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/24.jpg)
Forward modeling and simulation of circuitsGeneral linear systems:
f(t) −→ H(ω) −→ g(t)
H(ω) is called Transfert function of the system.
h(t) = IFTH(ω) is the impulse response of the system.
Given f(t) and h(t) or H(ω) we can compute g(t).
G(ω) = H(ω)F (ω) −→ g(t) = h(t) ∗ f(t)
f(t) = δ(t) −→ g(t) = h(t)
f(t) = u(t) =
0 t < 01 t ≥ 0
−→ g(t) =∫ t0 h(t) dt
f(t) = sin(ωt) −→ g(t) = |H(ω)| sin (ωt+∠H(ω))
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 24/118
![Page 25: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/25.jpg)
3- Data and signal processing of sensors output Some measurement systems defaults:
Full-scale error: Calibration Offset error: Offset elimination Drift: changes with temperature Non-linearity
In the following, we do not deal with these points. But, weare going to do:
Dealing with noise: Averaging, Filtering Considering linear measurement systems, to model their
input-output and then do inversion.
Fixed averaging Moving Average (MA) filtering Autoregressive (AR) filtering Moving Average Autoregressive (ARMA) filtering
Errors, noise and uncertainties −→ Background on Probabilitytheory
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 25/118
![Page 26: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/26.jpg)
Background on Probability theory
Why we need Probability theory ?
What is the significance of probability ?
What means a random variable ?
Discrete variables x1, · · · , xnProbability distribution: p1, · · · , pn with
∑pn = 1
Continuous variables x ∈ R or x ∈ R+ or x ∈ [a, b]Probability density function p(x) with
∫ +∞−∞ p(x) dx = 1,
Partition function: F (x) = P (X ≤ x) =∫ x
∞
p(x) dx
Expected value: E X =∫x p(x) dx
Variance value: Var X =∫(x− E X)2 p(x) dx
Mode value Mode = argmaxx p(x)
Normal distribution N (x|m, v)
Gamma distribution G(x|α, β)A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 26/118
![Page 27: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/27.jpg)
Discrete random variables
X takes values xi with probabilities pi, i = 1, · · · , n. P (X = xi) = pi, i = 1, · · · , n is probability distribution (pd).
If we sort xi in such a way that x1 ≤ x2 ≤ · · · ≤ xn, then wecan define the ”probability cumulative distribution (pcd)”:
F (x) = P (X ≤ x) =∑
i:xi≤x
P (X = xi)
P (a < X ≤ b) =∑
i:a<xi≤b
P (X = xi)
x1 x2 xi xn
p1p2 pi pn
X
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 27/118
![Page 28: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/28.jpg)
Discrete events
Probabilities Distribution
x1 x2 xi xn
p1p2 pi pn
X
Cumulative Probablity Distribution
x1 x2 xi xn X0
1
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 28/118
![Page 29: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/29.jpg)
Discrete events
Expected value
E X =< X >=∑
i
pi xi
Variance
Var X =∑
i
pi (xi − E X)2 =∑
i
pi (xi− < X >)2
Entropy
H(X) = −∑
i
pi ln pi
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 29/118
![Page 30: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/30.jpg)
Discrete variables probability distributions
Bernouilli distribution: A variable with two outcomes onlyX = 0, 1, P (X = 1) = p, P (X = 0) = q = 1− p
0 1
qp
X
Bernoulli trial B(n, p): n independent trials of an experimentwith two outcomes only 0010001100000010
p probability of success q = 1− p probability of failure
Binomial distribution Bin(.|n, p) :The probability of k successes in n trials:
P (X = k) =
(n
k
)pk (1− p)n−k
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 30/118
![Page 31: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/31.jpg)
Binomial distribution Bin(.|n, p)
The probability of k successes in n trials:
P (X = k) =
(n
k
)pk (1− p)n−k, k = 0, 1, · · · , n
E X = n p, Var X = n p q = n p (1− p)
0 1 2 3 4 5 6 7 8 9 100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
binopdf(k,n,p)
p = 0.2; n = 10; k = 0:n
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 31/118
![Page 32: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/32.jpg)
Poisson distribution
The Poisson distribution can be derived as a limiting case tothe binomial distribution as the number of trials goes toinfinity and the expected number of successes remains fixed
X ∼ Bin(n, p) limn 7→∞,np 7→λ
X ∼ P(λ)
P (X = k|λ) = λk exp [−λ]
k!
E X = λ, Var X = λ
If Xn ∼ Bin(n, λ/n) and Y ∼ P(λ) then for each fixed k,limn→∞ P (Xn = k) = P (Y = k).
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 32/118
![Page 33: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/33.jpg)
Poisson distribution
0 5 10 15 20 25 30 35 40 45 500
0.02
0.04
0.06
0.08
0.1
0.12
0.14
0.16
0.18
poisspdf(x,5)
poisspdf(x,10)
poisspdf(x,25)
normpdf(x,25,5)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 33/118
![Page 34: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/34.jpg)
Continuous case
Cumulative Distribution Function (cdf): F (x) = P (X < x) Measure theory
P (a ≤ X < b) = F (b)− F (a)
P (x ≤ X < x+ dx) = F (x+ dx)− F (x) = dF (x)
If F (x) is a continuous function
p(x) =∂F (x)
∂x
p(x) probability density function (pdf)
P (a < X ≤ b) =
∫ b
ap(x) dx
Cumulative distribution function (cdf)
F (x) =
∫ x
−∞p(x) dx
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 34/118
![Page 35: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/35.jpg)
Continuous case
Expected value
E X =
∫x p(x) dx =< X >
Variance
Var X =
∫(x− E X)2 p(x) dx =
⟨(x− E X)2
⟩
Entropy
H(X) =
∫− ln p(x) p(x) dx = 〈− ln p(X)〉
Mode: Mode(X) = argmaxx p(x) Median Med(X):
∫ Med(X)
−∞p(x) dx =
∫ +∞
Med(X)p(x) dx
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 35/118
![Page 36: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/36.jpg)
Uniform and Beta distributions
Uniform:
X ∼ U(.|a, b) −→ p(x) =1
b− a, x ∈ [a, b]
E X =a+ b
2, Var X =
(b− a)2
12
Beta:
X ∼ Beta(.|α, β) −→ p(x) =1
B(α, β)xα−1(1−x)β−1, x ∈ [0, 1]
E X =α
α+ β, Var X =
αβ
(α+ β)2(α+ β + 1)
Beta(.|1, 1) = U(.|0, 1)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 36/118
![Page 37: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/37.jpg)
Uniform and Beta distributions
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10.5
1
1.5
2
2.5
3
3.5
4
4.5
5
betapdf(x,.4,.6) betapdf(x,.6,.4)
betapdf(x,1,1)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 37/118
![Page 38: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/38.jpg)
Gaussian distributions
Different notations:
classical one with mean and variance:
X ∼ N (.|µ, σ2) −→ p(x) =1√2πσ2
exp
[− 1
2σ2(x− µ)2
]
E X = µ, Var X = σ2
mean and precision parameters:
X ∼ N (.|µ, λ) −→ p(x) =λ√2π
exp
[−λ
2(x− µ)2
]
E X = µ, Var X = σ2 =1
λ
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 38/118
![Page 39: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/39.jpg)
Generalized Gaussian distributions
Gaussian:
X ∼ N (.|µ, σ2) −→ p(x) =1√2πσ2
exp
[−1
2
((x− µ)
σ
)2]
Generalized Gaussian:
X ∼ GG(.|α, β) −→ p(x) =β
2αΓ(1/β)exp
[−( |x− µ|
α
)β]
E X = µ, Var X =α2Γ(3/β)
γ(1/β)
β > 0, β = 1 Laplace, β = 2: Gaussian, β 7→ ∞: Uniform
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 39/118
![Page 40: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/40.jpg)
Gaussian and Generalized Gaussian distributions
−3 −2 −1 0 1 2 30
0.1
0.2
0.3
0.4
0.5
0.6
0.7
beta=1
beta=2
beta=5
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 40/118
![Page 41: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/41.jpg)
Gamma distributions
Forme 1:
p(x|α, β) = βα
Γ(α)xα−1e−βx for x ≥ 0
E X =α
β, Var X =
α
β2, Mod(X) =
α− 1
α+ β − 2
Forme 2: θ = 1/β
p(x|α, θ) = θ−α
Γ(α)xα−1e−x/β for x ≥ 0
α = 1: Exponential,
0 < α < 1 decreasing,
α > 1 Mode=α−1β
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 41/118
![Page 42: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/42.jpg)
Gamma distributions
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 50
0.2
0.4
0.6
0.8
1
1.2
1.4
1.6
1.8
alpha=.5
alpha=1
alpha=2
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 42/118
![Page 43: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/43.jpg)
Student-t and Cauchy distributions
Student’s t-distribution has the probability density function:
p(x|ν) = Γ(ν+12 )√
νπ Γ(ν2 )
(1 +
x2
ν
)− ν+12
=1√
ν B(12 ,
ν2
)(1 +
x2
ν
)− ν+12
where ν is the number of degrees of freedom, Γ is the Gamma function and B is the Beta function.
ν = 1 gives Cauchy distribution.
p(x) =π
1 + x2
Cauchy distribution:
p(t|µ) = π
1 + (x− µ)2
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 43/118
![Page 44: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/44.jpg)
Student-t and Cauchy distributions
Three parameters location (µ) / scale (λ) / degree of freedom(ν) version
p(x|µ, λ, ν) = Γ(ν+12 )
Γ(ν2 )
(λ
πν
) 12[1 +
λ(x− µ)2
ν
]− ν+12
E X = µ for ν > 1,Var X = 1
λν
ν−2 for ν > 2,
mode(X) = µ.
Interesting relation between Student-t, Normal and Gammadistributions:
S(x|µ, 1, ν) =∫
N (x|µ, 1/λ)G(λ|ν/2, ν/2) dλ
S(x|0, 1, ν) =∫
N (x|0, 1/λ)G(λ|ν/2, ν/2) dλ
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 44/118
![Page 45: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/45.jpg)
Student and Cauchy
p(x|ν) ∝(1 +
x2
ν
)− ν+12
−5 −4 −3 −2 −1 0 1 2 3 4 50
0.05
0.1
0.15
0.2
0.25
0.3
0.35
0.4
tpdf(x,2)
tpdf(x,1)
normpdf(x,0,1)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 45/118
![Page 46: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/46.jpg)
Vector variables
Vector variables: X = [X1,X2, · · · ,Xn]′
p(x) probability density function (pdf)
Expected value
E X =
∫∫x p(x) dx =< X >
Covariance
cov[X] =
∫(X − E X)(X − E X)′ p(x) dx
=⟨(X − E X)(X − E X)′
⟩
Entropy
E(X) =
∫− ln p(x) p(x) dx = 〈ln p(X)〉
Mode: Mode(p(x)) = argmaxx p(x)A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 46/118
![Page 47: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/47.jpg)
Vector variables
Case of a vector with 2 variables: X = [X1,X2]′
p(x) = p(x1, x2) joint probability density function (pdf)
Marginals
p(x1) =
∫p(x1, x2) dx2
p(x2) =
∫p(x1, x2) dx1
Conditionals
p(x1|x2) =p(x1, x2)
p(x2)
p(x2|x1) =p(x1, x2)
p(x1)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 47/118
![Page 48: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/48.jpg)
Multivariate Gaussian
Different notations:
mean and covariance matrix (classical): X ∼ N (.|µ,Σ)
p(x) = (2π)−n/2|Σ|−1/2 exp
[−1
2(x− µ)′Σ−1(x− µ)
]
E X = µ, cov[X] = Σ
mean and precision matrix: X ∼ N (.|µ,Λ)
p(x) = (2π)−n/2|Λ|1/2 exp[−1
2(x− µ)′Λ(x− µ)
]
E X = µ, cov[X] = Λ−1
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 48/118
![Page 49: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/49.jpg)
Multivariate normal distributions
−3 −2 −1 0 1 2 3−3
−2
−1
0
1
2
3
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 49/118
![Page 50: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/50.jpg)
Multivariate Student-t
p(x|µ,Σ, ν) ∝ |Σ|−1/2
[1 +
1
ν(x− µ)′Σ−1(x− µ)
](ν+p)/2
p = 1
f(t) =Γ((ν + 1)/2)
Γ(ν/2)√νπ
(1 + t2/ν)−(ν+1)
2
p = 2, Σ−1 = A
f(t1, t2) =Γ((ν + p)/2)
Γ(ν/2)√νpπp
|A|1/22π
1 +
p∑
i=1
p∑
j=1
Aijti tj/ν
−(ν+2)2
p = 2, Σ = A = I
f(t1, t2) =1
2π(1 + (t21 + t21)/ν)
−(ν+2)2
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 50/118
![Page 51: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/51.jpg)
Multivariate Student-t distributions
−3 −2 −1 0 1 2 3−3
−2
−1
0
1
2
3
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 51/118
![Page 52: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/52.jpg)
Multivariate normal distributions
−3 −2 −1 0 1 2 3−3
−2
−1
0
1
2
3
−3 −2 −1 0 1 2 3−3
−2
−1
0
1
2
3
Normal Student-t
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 52/118
![Page 53: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/53.jpg)
Dealing with noise, errors and uncertainties
Sample averaging: mean and standard deviation
x =1
n
N∑
n=1
xn
S =
√√√√ 1
n− 1
N∑
n=1
(xn − x)2
Recursive computation: moving average
xk =1
n
k∑
i=k−n+1
xi, xk−1 =1
n
k−1∑
i=k−n
xi
xk = xk−1 +1
n(xk − xk−n)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 53/118
![Page 54: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/54.jpg)
Dealing with noise
Exponential moving average
xk =1
n
k∑
i=k−n+1
xi, xk+1 =1
n+ 1
k+1∑
i=k−n+1
xi
xk+1 =n
n+ 1xk +
1
n+ 1xk+1
xk =n
n+ 1xk−1 +
1
n+ 1xk = αxk−1 + (1− α)xk
The Exponentially Weighted Moving Average filter placesmore importance to more recent data by discounting olderdata in an exponential manner
xk = αxk−1 + (1− α)xk = α[αxk−2 + (1− α)xk−1](1− α)xk
xk = αxk−1 + (1− α)xk = α2xk−2 + α(1− α)xk−1(1− α)xk
xk = α3xk−3 + α2(1− α)xk−2 + α(1 − α)xk−1 + (1− α)xk
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 54/118
![Page 55: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/55.jpg)
Dealing with noise
Exponential moving average
xk = αxk−1 + (1− α)xk
xk = α2xk−2 + α(1− α)xk−1(1− α)xk
xk = α3xk−3 + α2(1− α)xk−2 + α(1 − α)xk−1 + (1− α)xk
The Exponentially Weighted Moving Average filter is identicalto the discrete first-order low-pass filter:
Consider the Laplace transform function of a first-orderlow-pass filter, with time constant τ :
x(s)
x(s)=
1
1 + τs−→ τ
∂x(t)
∂t+ x(t) = x(t)
∂x(t)
∂t=
xk − xk−1
Ts−→ xk =
(τ
τ + Ts
)xk−1+
(Ts
τ + Ts
)xk
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 55/118
![Page 56: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/56.jpg)
Other Filters
First order filter:
x(s)
x(s)= H(s) =
1
(1 + τs)
Second order filter:
H(s) =1
(1 + τs)2
Third order filter:
H(s) =1
(1 + τs)3
Bode diagram of the filter transfer function as a function of τand as a function of the order of the filter.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 56/118
![Page 57: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/57.jpg)
Background on linear invariant systems
A linear and invariant system: Time representation
f(t) −→ h(t) −→ g(t)
A linear and invariant system: Fourier Transformrepresentation
F (ω) −→ H(ω) −→ G(ω)
A linear and invariant system: Laplace Transformrepresentation
F (s) −→ H(s) −→ G(s)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 57/118
![Page 58: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/58.jpg)
Sampling theorem and digital linear invariant systems
Link between the FTs of a continuous signal and its sampledversion
Sampling theorem: If a Band limited signal(|F (ω)| = 0,∀ω > Ω0) is sampled with a sampling frequencyfs =
1Ts
two times greater than its maximum frequency(2πfs ≥ 2Ω0), its can be reconstructed without error from itssamples by an ideal low pass filtering.
Z-Transform is used in place of Laplace Transform to handlewith digital signals
A numerical or digital linear and invariant system:
f(n) −→ h(n) −→ g(n)
F (z) −→ H(z) −→ G(z)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 58/118
![Page 59: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/59.jpg)
Moving Average (MA)
f(t) −→ Filter −→ g(t)
Convolution Continuous
g(t) = h(t) ∗ f(t) =∫
h(τ)f(t − τ) dτ
Discrete
g(n) =
q∑
k=0
h(k)f(n− k), ∀n
Filter transfer function
f(n)−→ H(z) =
q∑
k=0
h(k)z−k −→g(n)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 59/118
![Page 60: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/60.jpg)
Autoregressive (AR)
Continuous
g(t) =
p∑
k=1
a(k) g(t − k∆t) + f(t)
Discrete
g(n) =
p∑
k=1
a(k) g(n − k) + f(n), ∀n
Filter transfer function
f(n)−→ H(z) =1
A(z)=
1
1 +∑p
k=1 a(k) z−k
−→g(n)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 60/118
![Page 61: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/61.jpg)
Autoregressive Moving Average (ARMA)
Continuous
g(t) =
p∑
k=1
a(k) g(t − k∆t) +
q∑
l=0
b(l) f(t− l∆t) dt)
Discrete
g(n) =
p∑
k=1
a(k) g(n − k) +
q∑
l=0
b(l) f(n− l)
ǫ(n)−→ H(z) =B(z)
A(z)=
∑qk=0 b(k)z
−k
1 +∑p
k=1 a(k) z−k
−→f(n)
Filter transfer function
ǫ(n)−→ Bq(z) −→ 1
Ap(z)−→f(n)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 61/118
![Page 62: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/62.jpg)
Parameter estimation
We observe n samples x = x1, · · · , xn of a quantity X whosepdf depends on certain parameters θ: p(x|θ).The question is to determine θ.
Moments method:
Exk
=
∫xkp(x|θ) dx ≈ 1
n
n∑
i=1
xki , k = 1, · · · ,K
Maximum Likelihood
L(θ) =n∏
i=1
p(xi|θ) or lnL(θ) =n∑
i=1
ln p(xi|θ)
θ = argmaxθ
L(θ) = argminθ
− lnL(θ)
Bayesian approach
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 62/118
![Page 63: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/63.jpg)
Bayesian Parameter estimation
Likelihood
p(x|θ) =n∏
i=1
p(xi|θ)
A priorip(θ)
A posteriorip(θ|x) ∝ p(x|θ)p(θ)
Infer on θ using p(θ|x).For example:
Maximum A Posteriori (MAP)
θ = argmaxθ
p(θ|x)
Posterior Mean
θ =
∫∫θp(θ|x) dθ
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 63/118
![Page 64: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/64.jpg)
Parameter estimation: Normal distribution
p(x|µ, σ2) =1√2πσ2
exp
(−(x− µ)2
2σ2
)
p(µ, σ2|x) = p(µ, σ2)
p(x)
N∏
i=1
p(xi|µ, σ)
p(µ, σ2|x) = p(µ, σ2)
p(x)
1
(2πσ2)N/2exp
[−
N∑
i=1
(xi − µ)2
2σ2
]
x =1
N
N∑
i=1
xi and s2 =1
N
N∑
i=1
(xi − x)2
p(µ, σ2|x) = p(µ, σ2)
p(x)
1
(2πσ2)N/2exp
[−(µ− x)2 + s2
2σ2/N
]
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 64/118
![Page 65: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/65.jpg)
Parameter estimation: Normal distribution: σ known σ2 known: p(µ, σ2) = p(µ) δ(σ − σ0)
p(µ|x) = p(µ)
p(x)
1(2πσ2
0
)N/2exp
[−
N∑
i=1
(xi − µ)2
2σ20
]
=p(µ)
p(x)
1(2πσ2
0
)N/2exp
[−(µ− x)2 + s2
2σ20/N
]
∝ p(µ) exp
[−(µ− x)2
2σ20/N
]
p(µ) = c −→ p(µ|x) = N (µ|x, σ20/N)
µ = x± σ0√N
p(µ) = N (µ0, v0) −→ p(µ|x) = N (µ|µ, v)
µ =v0
v0 + σ20
x+σ20
v0 + σ20
µ0, v =v0 + σ2
0
v0σ20
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 65/118
![Page 66: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/66.jpg)
Parameter estimation
We observe n samples x = x1, · · · , xn of a quantity X whosepdf depends on certain parameters θ: p(x|θ).The question is to determine θ.
Moments method:
Exk
=
∫xkp(x|θ) dx ≈ 1
n
n∑
i=1
xki , k = 1, · · · ,K
Maximum Likelihood
L(θ) =n∏
i=1
p(xi|θ) or lnL(θ) =n∑
i=1
ln p(xi|θ)
θ = argmaxθ
L(θ) = argminθ
− lnL(θ)
Bayesian approach
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 66/118
![Page 67: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/67.jpg)
Bayesian Parameter estimation
Likelihood
p(x|θ) =n∏
i=1
p(xi|θ)
A priorip(θ)
A posteriorip(θ|x) ∝ p(x|θ)p(θ)
Infer on θ using p(θ|x).For example:
Maximum A Posteriori (MAP)
θ = argmaxθ
p(θ|x)
Posterior Mean
θ =
∫∫θp(θ|x) dθ
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 67/118
![Page 68: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/68.jpg)
Parameter estimation: Normal distribution
p(x|µ, σ2) =1√2πσ2
exp
(−(x− µ)2
2σ2
)
p(µ, σ2|x) = p(µ, σ2)
p(x)
N∏
i=1
p(xi|µ, σ)
p(µ, σ2|x) = p(µ, σ2)
p(x)
1
(2πσ2)N/2exp
[−
N∑
i=1
(xi − µ)2
2σ2
]
x =1
N
N∑
i=1
xi and s2 =1
N
N∑
i=1
(xi − x)2
p(µ, σ2|x) = p(µ, σ2)
p(x)
1
(2πσ2)N/2exp
[−(µ− x)2 + s2
2σ2/N
]
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 68/118
![Page 69: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/69.jpg)
Parameter estimation: Normal distribution: σ known
σ2 known: p(µ, σ2) = p(µ) δ(σ − σ0)
p(µ|x) = p(µ)
p(x)
1(2πσ2
0
)N/2exp
[−
N∑
i=1
(xi − µ)2
2σ20
]
=p(µ)
p(x)
1(2πσ2
0
)N/2exp
[−(µ− x)2 + s2
2σ20/N
]
∝ p(µ) exp
[−(µ− x)2
2σ20/N
]
p(µ) = N (µ0, v0) −→ p(µ|x) = N (µ|µ, v)
µ =v0
v0 + σ20
x+σ20
v0 + σ20
µ0, v =v0 + σ2
0
v0σ20
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 69/118
![Page 70: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/70.jpg)
Conjugate priors
Observation lawp(x|θ)
Prior lawp(θ|τ )
Posterior lawp(θ|x, τ ) ∝ p(θ|τ )p(x|θ)
BinomialBin(x|n, θ)
BetaBet(θ|α, β)
BetaBet(θ|α+ x, β + n− x)
Negative BinomialNegBin(x|n, θ)
BetaBet(θ|α, β)
BetaBet(θ|α+ n, β + x)
MultinomialMk(x|θ1, · · · , θk)
DirichletDik(θ|α1, · · · , αk)
DirichletDik(θ|α1 + x1, · · · , αk + xk)
PoissonPn(x|θ)
GammaGam(θ|α, β)
GammaGam(θ|α+ x, β + 1)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 70/118
![Page 71: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/71.jpg)
Conjugate priors
Observation lawp(x|θ)
Prior lawp(θ|τ )
Posterior lawp(θ|x, τ ) ∝ p(θ|τ )p(x|θ)
GammaGam(x|ν, θ)
GammaGam(θ|α, β)
GammaGam(θ|α+ ν, β + x)
BetaBet(x|α, θ)
ExponentialEx(θ|λ)
ExponentialEx(θ|λ− log(1− x))
NormalN(x|θ, σ2)
NormalN(θ|µ, τ2)
Normal
N(µ|µσ2+τ2x
σ2+τ2, σ2 τ2
σ2+τ2
)
NormalN(x|µ, 1/θ)
GammaGam(θ|α, β)
GammaGam
(θ|α+ 1
2 , β + 12(µ −
NormalN(x|θ, θ2)
Generalized inverse NormalINg(θ|α, µ, σ) ∝|θ|−α exp
[− 1
2σ2
(1θ − µ
)2]Generalized inverse NormalINg(θ|αn, µn, σn)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 71/118
![Page 72: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/72.jpg)
3- Inverse problems : 3 main examples
Example 1:Measuring variation of temperature with a thermometer
f(t) variation of temperature over time g(t) variation of length of the liquid in thermometer
Example 2: Seeing outside of a body: Making an image usinga camera, a microscope or a telescope
f(x, y) real scene g(x, y) observed image
Example 3: Seeing inside of a body: Computed Tomographyusing X rays, US, Microwave, etc.
f(x, y) a section of a real 3D body f(x, y, z) gφ(r) a line of observed radiography gφ(r, z)
Example 1: Deconvolution
Example 2: Image restoration
Example 3: Image reconstruction
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 72/118
![Page 73: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/73.jpg)
Measuring variation of temperature with a thermometer
f(t) variation of temperature over time
g(t) variation of length of the liquid in thermometer
Forward model: Convolution
g(t) =
∫f(t′)h(t− t′) dt′ + ǫ(t)
h(t): impulse response of the measurement system
Inverse problem: Deconvolution
Given the forward model H (impulse response h(t)))and a set of data g(ti), i = 1, · · · ,Mfind f(t)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 73/118
![Page 74: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/74.jpg)
Measuring variation of temperature with a thermometer
Forward model: Convolution
g(t) =
∫f(t′)h(t− t′) dt′ + ǫ(t)
0 10 20 30 40 50 60−0.2
0
0.2
0.4
0.6
0.8
t
f(t)−→Thermometer
h(t) −→
0 10 20 30 40 50 60−0.2
0
0.2
0.4
0.6
0.8
t
g(t)
Inversion: Deconvolution
0 10 20 30 40 50 60−0.2
0
0.2
0.4
0.6
0.8
t
f(t) g(t)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 74/118
![Page 75: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/75.jpg)
Seeing outside of a body: Making an image with a camera,
a microscope or a telescope
f(x, y) real scene
g(x, y) observed image
Forward model: Convolution
g(x, y) =
∫∫f(x′, y′)h(x− x′, y − y′) dx′ dy′ + ǫ(x, y)
h(x, y): Point Spread Function (PSF) of the imaging system
Inverse problem: Image restoration
Given the forward model H (PSF h(x, y)))and a set of data g(xi, yi), i = 1, · · · ,Mfind f(x, y)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 75/118
![Page 76: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/76.jpg)
Making an image with an unfocused cameraForward model: 2D Convolution
g(x, y) =
∫∫f(x′, y′)h(x− x′, y − y′) dx′ dy′ + ǫ(x, y)
f(x, y) h(x, y) +
ǫ(x, y)
g(x, y)
Inversion: Image Deconvolution or Restoration
?⇐=
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 76/118
![Page 77: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/77.jpg)
Seeing inside of a body: Computed Tomography
f(x, y) a section of a real 3D body f(x, y, z)
gφ(r) a line of observed radiography gφ(r, z)
Forward model:Line integrals or Radon Transform
gφ(r) =
∫
Lr,φ
f(x, y) dl + ǫφ(r)
=
∫∫f(x, y) δ(r − x cosφ− y sinφ) dx dy + ǫφ(r)
Inverse problem: Image reconstruction
Given the forward model H (Radon Transform) anda set of data gφi
(r), i = 1, · · · ,Mfind f(x, y)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 77/118
![Page 78: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/78.jpg)
2D and 3D Computed Tomography
3D 2D
−80 −60 −40 −20 0 20 40 60 80
−80
−60
−40
−20
0
20
40
60
80
f(x,y)
x
y
Projections
gφ(r1, r2) =
∫
Lr1,r2,φ
f(x, y, z) dl gφ(r) =
∫
Lr,φ
f(x, y) dl
Forward problem: f(x, y) or f(x, y, z) −→ gφ(r) or gφ(r1, r2)Inverse problem: gφ(r) or gφ(r1, r2) −→ f(x, y) or f(x, y, z)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 78/118
![Page 79: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/79.jpg)
Inverse problems: Discretization
g(si) =
∫∫h(si, r) f(r) dr + ǫ(si), i = 1, · · · ,M
f(r) is assumed to be well approximated by
f(r) ≃N∑
j=1
fj bj(r)
with bj(r) a basis or any other set of known functions
g(si) = gi ≃N∑
j=1
fj
∫∫h(si, r) bj(r) dr, i = 1, · · · ,M
g = Hf + ǫ with Hij =
∫∫h(si, r) bj(r) dr
H is huge dimensional
LS solution : f = argminf Q(f) withQ(f) =
∑i |gi − [Hf ]i|2 = ‖g −Hf‖2
does not give satisfactory result.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 79/118
![Page 80: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/80.jpg)
Inverse problems: Deterministic methodsData matching
Observation modelgi = hi(f) + ǫi, i = 1, . . . ,M −→ g = H(f ) + ǫ
Mismatch between data and output of the model ∆(g,H(f ))
f = argminf
∆(g,H(f )) Examples:
– LS ∆(g,H(f )) = ‖g −H(f )‖2 =∑
i
|gi − hi(f)|2
– Lp ∆(g,H(f )) = ‖g −H(f )‖p =∑
i
|gi − hi(f)|p , 1 < p < 2
– KL ∆(g,H(f )) =∑
i
gi lngi
hi(f)
In general, does not give satisfactory results for inverseproblems.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 80/118
![Page 81: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/81.jpg)
Inverse problems: Regularization theory
Inverse problems = Ill posed problems−→ Need for prior information
Functional space (Tikhonov):
g = H(f) + ǫ −→ J(f) = ||g −H(f)||22 + λ||Df ||22
Finite dimensional space (Philips & Towmey): g = H(f ) + ǫ
• Minimum norm LS (MNLS): J(f) = ||g −H(f )||2 + λ||f ||2• Classical regularization: J(f) = ||g −H(f )||2 + λ||Df ||2• More general regularization:
J(f) = Q(g −H(f)) + λΩ(Df)or
J(f) = ∆1(g,H(f )) + λ∆2(f ,f∞)Limitations:
• Errors are considered implicitly white and Gaussian• Limited prior information on the solution• Lack of tools for the determination of the hyper-parameters
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 81/118
![Page 82: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/82.jpg)
Bayesian inference for inverse problems
M : g = Hf + ǫ
Observation model M + Hypothesis on the noise ǫ −→p(g|f ;M) = pǫ(g −Hf)
A priori information p(f |M)
Bayes : p(f |g;M) =p(g|f ;M) p(f |M)
p(g|M)
Link with regularization :
Maximum A Posteriori (MAP) :
f = argmaxf
p(f |g) = argmaxf
p(g|f) p(f)
= argminf
− ln p(g|f)− ln p(f)
with Q(g,Hf ) = − ln p(g|f) and λΩ(f) = − ln p(f)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 82/118
![Page 83: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/83.jpg)
Bayesian inference for inverse problems
Linear Inverse problems: g = Hf + ǫ f H ♥+ǫ
g
Bayesian inference:
p(f |g,θ) = p(g|f ,θ1) p(f |θ2)
p(g|θ)
with θ = (θ1,θ2) θ2
p(f |θ2)
Prior
⋄
θ1
p(g|f ,θ1)
Likelihood
−→ p(f |g,θ)Posterior
−→ f
Point estimators: Maximum A Posteriori (MAP): f = argmaxf p(f |g, θ) Posterior Mean (PM): f = Ep(f |g,θ) f =
∫∫f p(f |g, θ) df
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 83/118
![Page 84: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/84.jpg)
Bayesian Estimation: Two simple priors
Example 1: Linear Gaussian case:
p(g|f , θ1) = N (Hf , θ1I)p(f |θ2) = N (0, θ2I)
−→ p(f |g,θ) = N (f , P )
with P = (H ′H + λI)−1, λ = θ1
θ2
f = PH ′g
f = argminf
J(f) with J(f) = ‖g −Hf‖22 + λ‖f‖22
Example 2: Double Exponential prior & MAP:
f = argminf
J(f) with J(f) = ‖g −Hf‖22 + λ‖f‖1
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 84/118
![Page 85: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/85.jpg)
Full Bayesian approachM : g = Hf + ǫ
Forward & errors model: −→ p(g|f ,θ1;M)
Prior models −→ p(f |θ2;M)
Hyper-parameters θ = (θ1,θ2) −→ p(θ|M)
Bayes: −→ p(f ,θ|g;M) = p(g|f ,θ;M) p(f |θ;M) p(θ|M)p(g|M)
Joint MAP: (f , θ) = argmax(f ,θ)
p(f ,θ|g;M)
Marginalization:
p(f |g;M) =
∫∫p(f ,θ|g;M) dθ
p(θ|g;M) =∫∫p(f ,θ|g;M) df
Posterior means:
f =
∫ ∫f p(f ,θ|g;M) dθ df
θ =∫ ∫
θ p(f ,θ|g;M) df dθ
Evidence of the model:
p(g|M) =
∫∫p(g|f ,θ;M)p(f |θ;M)p(θ|M) df dθ
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 85/118
![Page 86: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/86.jpg)
Full Bayesian: Marginal MAP and PM estimates
Marginal MAP: θ = argmaxθ p(θ|g) where
p(θ|g) =∫∫
p(f ,θ|g) df ∝ p(g|θ) p(θ)
and then f = argmaxf
p(f |θ,g)
or
Posterior Mean: f =
∫∫f p(f |θ,g) df
Needs the expression of the Likelihood:
p(g|θ) =∫∫
p(g|f ,θ1) p(f |θ2) df
Not always analytically available −→ EM, SEM and GEMalgorithms
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 86/118
![Page 87: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/87.jpg)
Full Bayesian Model and Hyper-parameter Estimation
↓ α,β
Hyper prior model p(θ|α,β)
θ2
p(f |θ2)
Prior
⋄
θ1
p(g|f ,θ1)
Likelihood
−→p(f ,θ|g,α,β)
Joint Posterior
−→ f
−→ θ
Full Bayesian Model and Hyper-parameter Estimation scheme
p(f ,θ|g)Joint Posterior
−→ p(θ|g)Marginalize over f
−→ θ −→ p(f |θ,g) −→ f
Marginalization for Hyper-parameter Estimation
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 87/118
![Page 88: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/88.jpg)
Full Bayesian: EM and GEM algorithms
EM and GEM Algorithms: f as hidden variable,g as incomplete data, (g,f ) as complete dataln p(g|θ) incomplete data log-likelihoodln p(g,f |θ) complete data log-likelihood
Iterative algorithm:
E-step: Q(θ, θ(k)
) = Ep(f |g,θ
(k))ln p(g,f |θ)
M-step: θ(k)
= argmaxθ
Q(θ, θ
(k−1))
GEM (Bayesian) algorithm:
E-step: Q(θ, θ(k)
) = Ep(f |g,θ
(k))ln p(g,f |θ) + ln p(θ)
M-step: θ(k)
= argmaxθ
Q(θ, θ
(k−1))
p(f ,θ|g) −→ EM, GEM −→ θ −→ p(f |θ,g) −→ f
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 88/118
![Page 89: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/89.jpg)
Two main steps in the Bayesian approach Prior modeling
Separable:Gaussian, Gamma,Sparsity enforcing: Generalized Gaussian, mixture ofGaussians, mixture of Gammas, ...
Markovian:Gauss-Markov, GGM, ...
Markovian with hidden variables(contours, region labels)
Choice of the estimator and computational aspects MAP, Posterior mean, Marginal MAP MAP needs optimization algorithms Posterior mean needs integration methods Marginal MAP and Hyper-parameter estimation need
integration and optimization Approximations:
Gaussian approximation (Laplace) Numerical exploration MCMC Variational Bayes (Separable approximation)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 89/118
![Page 90: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/90.jpg)
Different prior models for signals and images: Separable
Gaussian Generalized Gaussianp(fj) ∝ exp
[−α|fj |2
]p(fj) ∝ exp [−α|fj|p] , 1 ≤ p ≤ 2
Gamma Betap(fj) ∝ fα
j exp [−βfj] p(fj) ∝ fαj (1− fj)
β
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 90/118
![Page 91: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/91.jpg)
Sparsity enforcing prior models Sparse signals: Direct sparsity
0 20 40 60 80 100 120 140 160 180 200−3
−2
−1
0
1
2
3
0 20 40 60 80 100 120 140 160 180 2000
0.5
1
1.5
2
2.5
3
Sparse signals: Sparsity in a Transform domain
0 20 40 60 80 100 120 140 160 180 2000
0.5
1
1.5
2
2.5
3
0 20 40 60 80 100 120 140 160 180 200−6
−4
−2
0
2
4
6
0 20 40 60 80 100 120 140 160 180 200−1
−0.8
−0.6
−0.4
−0.2
0
0.2
0.4
0.6
0.8
1
0 20 40 60 80 100 120 140 160 180 200−3
−2
−1
0
1
2
3
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10
20
40
60
80
100
120
140
160
180
200
20 40 60 80 100 120 140 160 180 200
1
2
3
4
5
6
7
8
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 91/118
![Page 92: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/92.jpg)
Sparsity enforcing prior models
Simple heavy tailed models: Generalized Gaussian, Double Exponential Symmetric Weibull, Symmetric Rayleigh Student-t, Cauchy Generalized hyperbolic Elastic net
Hierarchical mixture models: Mixture of Gaussians Bernoulli-Gaussian Mixture of Gammas Bernoulli-Gamma Mixture of Dirichlet Bernoulli-Multinomial
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 92/118
![Page 93: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/93.jpg)
Simple heavy tailed models• Generalized Gaussian, Double Exponential
p(f |γ, β) =∏
j
GG(fj |γ, β) ∝ exp
−γ
∑
j
|fj |β
β = 1 Double exponential or Laplace.0 < β ≤ 1 are of great interest for sparsity enforcing.
−10 −8 −6 −4 −2 0 2 4 6 8 100
0.01
0.02
0.03
0.04
0.05
0.06
p ∝ exp(−γ*|x|β)
β=2.0, γ=1β=1.5, γ=1β=1.0, γ=1β=0.5, γ=1
−2 −1.5 −1 −0.5 0 0.5 1 1.5 22.5
3
3.5
4
4.5
5
5.5
6
6.5
7
p ∝ exp(−γ*|x|β)
β=2.0, γ=1β=1.5, γ=1β=1.0, γ=1β=0.5, γ=1
Generalized Gaussian family
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 93/118
![Page 94: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/94.jpg)
Simple heavy tailed models• Symmetric Weibull
p(f |γ, β) =∏
j
W(fj |γ, β) ∝ exp
−γ
∑
j
|fj|β + (β − 1) log |fj|
β = 2 is the Symmetric Rayleigh distribution.β = 1 is the Double exponential and0 < β ≤ 1 are of great interest for sparsity enforcing.
−10 −8 −6 −4 −2 0 2 4 6 8 100
0.05
0.1
0.15
0.2
0.25
0.3
0.35
GW
−4 −3 −2 −1 0 1 2 3 40
5
10
15
20
25
GW
Symmetric Weibull familyA. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 94/118
![Page 95: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/95.jpg)
Simple heavy tailed models• Student-t and Cauchy models
p(f |ν) =∏
j
St(fj|ν) ∝ exp
−ν + 1
2
∑
j
log(1 + f2
j /ν)
Cauchy model is obtained when ν = 1.
−10 −8 −6 −4 −2 0 2 4 6 8 100
0.005
0.01
0.015
0.02
0.025
0.03
0.035
GC
−4 −3 −2 −1 0 1 2 3 43
3.5
4
4.5
5
5.5
6
6.5
7
7.5
8
GC
Student-t and Cauchy families
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 95/118
![Page 96: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/96.jpg)
Simple heavy tailed models
• Elastic net prior model
p(f |ν) =∏
j
EN (fj|ν) ∝ exp
−
∑
j
(γ1|fj |+ γ2f2j )
−10 −8 −6 −4 −2 0 2 4 6 8 100
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.08
0.09
0.1
GEN
−4 −3 −2 −1 0 1 2 3 40
5
10
15
20
25
GEN
Elastic Net family
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 96/118
![Page 97: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/97.jpg)
Simple heavy tailed models
• Generalized hyperbolic (GH) models
p(f |δ, ν, β) =∏
j
(δ2 + f2j )
(ν−1/2)/2 exp [βx)]Kν−1/2(α√
δ2 + f2j )
−10 −8 −6 −4 −2 0 2 4 6 8 100
0.005
0.01
0.015
0.02
0.025
0.03
GGH
−4 −3 −2 −1 0 1 2 3 43.5
4
4.5
5
5.5
6
6.5
7
7.5
8
GGH
Generalized hyperbolic family
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 97/118
![Page 98: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/98.jpg)
Mixture models• Mixture of two Gaussians (MoG2) model
p(f |α, v1, v0) =∏
j
[αN (fj |0, v1) + (1− α)N (fj |0, v0)]
• Bernoulli-Gaussian (BG) model
p(f |α, v) =∏
j
p(fj) =∏
j
[αN (fj |0, v) + (1− α)δ(fj)]
−10 −8 −6 −4 −2 0 2 4 6 8 100
0.005
0.01
0.015
0.02
0.025
0.03
GMoG2
−4 −3 −2 −1 0 1 2 3 43.5
4
4.5
5
5.5
6
6.5
7
7.5
8
GMoG2
Mixture of 2 Gaussians familiesA. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 98/118
![Page 99: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/99.jpg)
• Mixture of Gammas
p(f |λ, v1, v0) =∏
j
[λG(fj |α1, β1) + (1− λ)G(fj |α2, β2)]
• Bernoulli-Gamma model
p(f |λ, α, β) =∏
j
[λG(fj |α, β) + (1− λ)δ(fj)]
• Mixture of Dirichlets model
p(f |λ,H1,α1,H2,α2) =∏
j
[λD(fj|H1,α1) + (1− λ)D(fj|H2,α2)]
D(fj|H,α) =
K∏
k=1
Γ(α)
Γ(α0)Γ(αK)aαk−1k , αk ≥ 0, ak ≥ 0
where H = a1, · · · , aK and α = α1, · · · , αKwith
∑k αk = α and
∑k ak = 1.
• Bernoulli-Multinomial (BMultinomial) model
p(f |λ,H,α) =∏
j
[λδ(fj) + (1− λ)Mult(fj|H,α)]
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 99/118
![Page 100: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/100.jpg)
Hierarchical models and hidden variables
All the mixture models and some of simple models can bemodeled via hidden variables z.
p(f) =
K∑
k=1
αkpk(f) −→
p(f |z = k) = pk(f),P (z = k) = αk,
∑k αk = 1
Example 1: MoG model: pk(f) = N (f |mk, vk)2 Gaussians: p0 = N (0, v0), p1 = N (0, v1), α0 = λ, α1 = 1−λ
p(fj|λ, v1, v0) = λN (fj|0, v1) + (1− λ)N (fj |0, v0)
p(fj|zj = 0, v0) = N (fj|0, v0),p(fj|zj = 1, v1) = N (fj|0, v1), and
P (zj = 0) = λ,P (zj = 1) = 1− λ
p(f |z) = ∏j p(fj|zj) =
∏j N
(fj|0, vzj
)∝ exp
[−1
2
∑j
f2j
vzj
]
P (zj = 1) = λ, P (zj = 0) = 1− λ
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 100/118
![Page 101: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/101.jpg)
Hierarchical models and hidden variables
Example 2: Student-t model
St(f |ν) ∝ exp
[−ν + 1
2log
(1 + f2/ν
)]
Infinite mixture
St(f |ν) ∝=
∫ ∞
0N (f |, 0, 1/z)G(z|α, β) dz, with α = β = ν/2
p(f |z) =∏
j p(fj|zj) =∏
j N (fj|0, 1/zj) ∝ exp[−1
2
∑j zjf
2j
]
p(z|α, β) =∏
j G(zj |α, β) ∝∏
j zj(α−1) exp [−βzj ]
∝ exp[∑
j(α− 1) ln zj − βzj
]
p(f ,z|α, β) ∝ exp[−1
2
∑j zjf
2j + (α− 1) ln zj − βzj
]
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 101/118
![Page 102: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/102.jpg)
Hierarchical models and hidden variables
Example 3: Laplace (Double Exponential) model
DE(f |a) = a
2exp [−a|f |] =
∫ ∞
0N (f |, 0, z) E(z|a2/2) dz, a > 0
p(f |z) =∏
j p(fj|zj) =∏
j N (fj|0, zj) ∝ exp[−1
2
∑j f
2j /zj
]
p(z|a22 ) =∏
j E(zj |a2
2 ) ∝ exp[∑
ja2
2 zj
]
p(f ,z|a22 ) ∝ exp[−1
2
∑j f
2j /zj +
a2
2 zj
]
With these models we have:
p(f ,z,θ|g) ∝ p(g|f ,θ1) p(f |z,θ2) p(z|θ3) p(θ)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 102/118
![Page 103: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/103.jpg)
Bayesian Computation and Algorithms
Often, the expression of p(f ,z,θ|g) is complex.
Its optimization (for Joint MAP) orits marginalization or integration (for Marginal MAP or PM)is not easy
Two main techniques:MCMC and Variational Bayesian Approximation (VBA)
MCMC:Needs the expressions of the conditionalsp(f |z,θ,g), p(z|f ,θ,g), and p(θ|f ,z,g)
VBA: Approximate p(f ,z,θ|g) by a separable one
q(f ,z,θ|g) = q1(f) q2(z) q3(θ)
and do any computations with these separable ones.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 103/118
![Page 104: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/104.jpg)
MCMC based algorithm
p(f ,z,θ|g) ∝ p(g|f ,z,θ) p(f |z,θ) p(z) p(θ)General scheme:
f ∼ p(f |z, θ,g) −→ z ∼ p(z|f , θ,g) −→ θ ∼ (θ|f , z,g)
Estimate f using p(f |z, θ,g) ∝ p(g|f ,θ) p(f |z, θ)When Gaussian, can be done via optimization of a quadraticcriterion.
Estimate z using p(z|f , θ,g) ∝ p(g|f , z, θ) p(z)Often needs sampling (hidden discrete variable)
Estimate θ usingp(θ|f , z,g) ∝ p(g|f , σ2
ǫ I) p(f |z, (mk, vk)) p(θ)Use of Conjugate priors −→ analytical expressions.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 104/118
![Page 105: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/105.jpg)
Variational Bayesian Approximation
Approximate p(f ,θ|g) by q(f ,θ|g) = q1(f |g) q2(θ|g)and then continue computations.
Criterion KL(q(f ,θ|g) : p(f ,θ|g)) KL(q : p) =
∫ ∫q ln q/p =
∫ ∫q1q2 ln
q1q2p =∫
q1 ln q1+∫q2 ln q2−
∫ ∫q ln p = −H(q1)−H(q2)− < ln p >q
Iterative algorithm q1 −→ q2 −→ q1 −→ q2, · · ·
q1(f) ∝ exp[〈ln p(g,f ,θ;M)〉q2(θ)
]
q2(θ) ∝ exp[〈ln p(g,f ,θ;M)〉q1(f)
]
p(f ,θ|g) −→VariationalBayesian
Approximation
−→ q1(f) −→ f
−→ q2(θ) −→ θ
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 105/118
![Page 106: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/106.jpg)
Summary of Bayesian estimation 1
Simple Bayesian Model and Estimation
θ2
p(f |θ2)
Prior
⋄
θ1
p(g|f ,θ1)
Likelihood
−→ p(f |g,θ)Posterior
−→ f
Full Bayesian Model and Hyper-parameter Estimation
↓ α,β
Hyper prior model p(θ|α,β)
θ2
p(f |θ2)
Prior
⋄
θ1
p(g|f ,θ1)
Likelihood
−→p(f ,θ|g,α,β)
Joint Posterior
−→ f
−→ θ
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 106/118
![Page 107: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/107.jpg)
Summary of Bayesian estimation 2
Marginalization for Hyper-parameter Estimation
p(f ,θ|g)Joint Posterior
−→ p(θ|g)Marginalize over f
−→ θ −→ p(f |θ,g) −→ f
Full Bayesian Model with a Hierarchical Prior Model
θ3
p(z|θ3)
Hidden variable
⋄
θ2
p(f |z,θ2)
Prior
⋄
θ1
p(g|f ,θ1)
Likelihood
−→ p(f ,z|g,θ)Joint Posterior
−→ f
−→ z
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 107/118
![Page 108: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/108.jpg)
Summary of Bayesian estimation 3• Full Bayesian Hierarchical Model with Hyper-parameter Estimation
↓ α,β,γ
Hyper prior model p(θ|α,β,γ)
θ3
p(z|θ3)
Hidden variable
⋄
θ2
p(f |z,θ2)
Prior
⋄
θ1
p(g|f ,θ1)
Likelihood
−→ p(f ,z,θ|g)Joint Posterior
−→ f
−→ z
−→ θ
• Full Bayesian Hierarchical Model and Variational Approximation
↓ α,β,γ
Hyper prior model p(θ|α,β,γ)
θ3
p(z|θ3)
Hidden variable
⋄
θ2
p(f |z, θ2)
Prior
⋄
θ1
p(g|f , θ1)
Likelihood
−→ p(f , z, θ|g)
Joint Posterior
−→
VBA
q1(f)q2(z)q3(θ)
SeparableApproximation
−→ f
−→ z
−→ θ
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 108/118
![Page 109: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/109.jpg)
Which images I am looking for?
50 100 150 200 250 300
50
100
150
200
250
300
350
400
450
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 109/118
![Page 110: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/110.jpg)
Which image I am looking for?
Gauss-Markov Generalized GM
Piecewize Gaussian Mixture of GM
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 110/118
![Page 111: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/111.jpg)
Gauss-Markov-Potts prior models for images
f(r) z(r) c(r) = 1− δ(z(r)− z(r′))
p(f(r)|z(r) = k,mk, vk) = N (mk, vk)
p(f(r)) =∑
k
P (z(r) = k)N (mk, vk) Mixture of Gaussians
Separable iid hidden variables: p(z) =∏
r p(z(r)) Markovian hidden variables: p(z) Potts-Markov:
p(z(r)|z(r′), r′ ∈ V(r)) ∝ exp
γ
∑
r′∈V(r)
δ(z(r) − z(r′))
p(z) ∝ exp
γ
∑
r∈R
∑
r′∈V(r)
δ(z(r) − z(r′))
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 111/118
![Page 112: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/112.jpg)
Four different cases
To each pixel of the image is associated 2 variables f(r) and z(r)
f |z Gaussian iid, z iid :Mixture of Gaussians
f |z Gauss-Markov, z iid :Mixture of Gauss-Markov
f |z Gaussian iid, z Potts-Markov :Mixture of Independent Gaussians(MIG with Hidden Potts)
f |z Markov, z Potts-Markov :Mixture of Gauss-Markov(MGM with hidden Potts)
f(r)
z(r)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 112/118
![Page 113: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/113.jpg)
Application of CT in NDTReconstruction from only 2 projections
g1(x) =
∫f(x, y) dy, g2(y) =
∫f(x, y) dx
Given the marginals g1(x) and g2(y) find the joint distributionf(x, y).
Infinite number of solutions : f(x, y) = g1(x) g2(y)Ω(x, y)Ω(x, y) is a Copula:
∫Ω(x, y) dx = 1 and
∫Ω(x, y) dy = 1
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 113/118
![Page 114: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/114.jpg)
Application in CT
20 40 60 80 100 120
20
40
60
80
100
120
g|f f |z z c
g = Hf + ǫ iid Gaussian iid c(r) ∈ 0, 1g|f ∼ N (Hf , σ2
ǫ I) or or 1− δ(z(r) − z(r′))Gaussian Gauss-Markov Potts binary
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 114/118
![Page 115: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/115.jpg)
Proposed algorithm
p(f ,z,θ|g) ∝ p(g|f ,z,θ) p(f |z,θ) p(θ)General scheme:
f ∼ p(f |z, θ,g) −→ z ∼ p(z|f , θ,g) −→ θ ∼ (θ|f , z,g)
Iterative algorithm:
Estimate f using p(f |z, θ,g) ∝ p(g|f ,θ) p(f |z, θ)Needs optimization of a quadratic criterion.
Estimate z using p(z|f , θ,g) ∝ p(g|f , z, θ) p(z)Needs sampling of a Potts Markov field.
Estimate θ usingp(θ|f , z,g) ∝ p(g|f , σ2
ǫ I) p(f |z, (mk, vk)) p(θ)Conjugate priors −→ analytical expressions.
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 115/118
![Page 116: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/116.jpg)
Results
Original Backprojection Filtered BP LS
Gauss-Markov+pos GM+Line process GM+Label process
20 40 60 80 100 120
20
40
60
80
100
120
c 20 40 60 80 100 120
20
40
60
80
100
120
z 20 40 60 80 100 120
20
40
60
80
100
120
c
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 116/118
![Page 117: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/117.jpg)
Application in Microwave imaging
g(ω) =
∫f(r) exp [−j(ω.r)] dr + ǫ(ω)
g(u, v) =
∫∫f(x, y) exp [−j(ux+ vy)] dx dy + ǫ(u, v)
g = Hf + ǫ
20 40 60 80 100 120
20
40
60
80
100
120
20 40 60 80 100 120
20
40
60
80
100
120
20 40 60 80 100 120
20
40
60
80
100
120
20 40 60 80 100 120
20
40
60
80
100
120
f(x, y) g(u, v) f IFT f Proposed method
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 117/118
![Page 118: Sensors, Measurement systems, Signal processing and ...djafari.free.fr/Cours/Master_MNE/Cours/Cours_MNE_2014.pdf · Sensors, Measurement systems Signal processing and Inverse problems](https://reader031.fdocuments.net/reader031/viewer/2022022016/5b5efc0d7f8b9a553d8daa2f/html5/thumbnails/118.jpg)
Conclusions
Bayesian Inference for inverse problems
Different prior modeling for signals and images:Separable, Markovian, without and with hidden variables
Sparsity enforcing priors
Gauss-Markov-Potts models for images incorporating hiddenregions and contours
Two main Bayesian computation tools: MCMC and VBA
Application in different CT (X ray, Microwaves, PET, SPECT)
Current Projects and Perspectives :
Efficient implementation in 2D and 3D cases
Evaluation of performances and comparison between MCMCand VBA methods
Application to other linear and non linear inverse problems:(PET, SPECT or ultrasound and microwave imaging)
A. Mohammad-Djafari, Sensors, Measurement systems, Signal processing and Inverse problems, Master MNE 2014, 118/118