Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning...
Transcript of Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning...
![Page 1: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/1.jpg)
The world leader in serving scienceProprietary & Confidential
Pavel Potocek, Thermo Fisher Scientific, Eindhoven, Nederlands
2017-09-07
Matlab + Compressive Sensing + Scanning Electron Microscopy
![Page 2: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/2.jpg)
2 Proprietary & Confidential
• Life Science, Electron Microscopy
• Matlab Integration: Image Acquisition Stability
• Compressive Sensing Basics
• Image Acquisition with CS
• Matlab: Reconstruction Algorithms
• Matlab Integration: Integration to Acquire Big Data
Overview – How we use Matlab for Big Data Acquisition
![Page 3: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/3.jpg)
3 Proprietary & Confidential
• Life Science, Electron Microscopy
• Matlab Integration: Image Acquisition Stability
• Compressive Sensing Basics
• Image Acquisition with CS
• Matlab: Reconstruction Algorithms
• Matlab Integration: Integration to Acquire Big Data
Overview – How we use Matlab for Big Data Acquisition
![Page 4: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/4.jpg)
4 Proprietary & Confidential
Electron Microscopy for Life Science
• Increased interest to understand the brain functionality
From Connections to Cognition
Connectome
Human Connectome Project
(USA launched 2010 to map
wiring human brain)
EU: 2013
Sum of all brain’s connections
![Page 5: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/5.jpg)
5 Proprietary & Confidential
Brain investigation example
Neurons and their connections - synapses
Volume size ~ 300-500um
Details ~ 10nmReality
Neurons
Synapses
Expected constraints
![Page 6: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/6.jpg)
6 Proprietary & Confidential
Which tool for Imaging?
TEM 1931 – Knoll, Ruska
SEM 1937 – von Ardenne
![Page 7: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/7.jpg)
7 Proprietary & Confidential
Very Short introduction to Scanning Electron Microscopy
Beam-specimen interactions
10um 1um 1um
![Page 8: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/8.jpg)
8 Proprietary & Confidential
Brain Reverse Engineering
• Reverse engineering is one of the most important techniques
Knife-edge, SEM
300-500nm slice
full mouse brain about 100 hours
Knife
1mm slice
Blade
70um slice
Diamond knife, SEM
25-50nm slice
![Page 9: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/9.jpg)
9 Proprietary & Confidential
Data acquisition workflow
VolumeScope
Teneo SEM Microtome MAPSMulti-Energy
ReconstructionAMIRA
Hardware Software
![Page 10: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/10.jpg)
10 Proprietary & Confidential
Data acquisition workflow overview - movies
VolumeScope data acquisition
Neuron reconstruction(from National Geographic, Lichtman)
![Page 11: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/11.jpg)
11 Proprietary & Confidential
Acquisition timing example
Volume size ~ 300-500um
Details ~ 10nm
• No cutting
• No tiling involved – No stage movements
• No auto-functions - No processing
• Single 16bits data instance
• No failure
Acquisition Dwell time 100ns 1us
Single section image Pixel resolution 32-64k 32-64k
Data size 2 – 8 GB 2 – 8 GB
Acquisition time 2-8 m 20-80 m
Full 3D volume Data size 64 - 512 TB 64 - 512 TB
Acquisition time 38 – 304 d 380 – 3040 d
MPI:
Denk, Helmstaedter:
13kx50kx50k = 33TP
8 weeks acquisition;
2years to reconstruct
![Page 12: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/12.jpg)
12 Proprietary & Confidential
Overview – How we use Matlab for Big Data Acquisition
• Life Science, Electron Microscopy
• Matlab Integration: Image Acquisition Stability
• Compressive Sensing Basics
• Image Acquisition with CS
• Matlab: Reconstruction Algorithms
• Matlab Integration: Integration to Acquire Big Data
![Page 13: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/13.jpg)
13 Proprietary & Confidential
Why do we need auto-functions?
Without any adaptation Compensation after auto-functions
Reference
Different imaging conditions (beam energy, working distance, …)
Reference
![Page 14: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/14.jpg)
14 Proprietary & Confidential
Matlab to test different focusing algorithms
Acquisition + Focus & Processing
control in Python
Sharpness algorithms
Matlab
In focus
De-focusing
![Page 15: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/15.jpg)
15 Proprietary & Confidential
Matlab inline in Python notebook with other languages
Code Output
![Page 16: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/16.jpg)
16 Proprietary & Confidential
Overview – How we use Matlab for Big Data Acquisition
• Life Science, Electron Microscopy
• Matlab Integration: Image Acquisition Stability
• Compressive Sensing Basics
• Image Acquisition with CS
• Matlab: Reconstruction Algorithms
• Matlab Integration: Integration to Acquire Big Data
![Page 17: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/17.jpg)
17 Proprietary & Confidential
Compressible vs Sparse Signals
Compressed images – Compressible images – Sparse images
Cropped spectrums
Original
DCT basis
![Page 18: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/18.jpg)
18 Proprietary & Confidential
Compressive sensing – motivation from JPEG
Statistics for ~1000images
Sparsity=relative # DCT coefficients for >99.75% energy
2012 Sparse imaging for fast electron microscopy
(S.Anderson)
75% of all images
Sparsity
• How sparse is the microscopy image?
![Page 19: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/19.jpg)
19 Proprietary & Confidential
Signal reconstruction for Compressive sensing
• Signal reconstruction from sampling measurements
• Nyquist-Shannon theorem
# samples depends on the signal’s frequencies
• Candes-Tao-Donoho (2004-2006)
# samples depends on the signal’s sparsity
= x
![Page 20: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/20.jpg)
20 Proprietary & Confidential
Compressive sensing principle
Sparse spectrum y
Transformation
Ψ
xx=
Sensing matrix ΦMeasurement m Spectrum y’
Reconstructed
image i
Sample
Sparse undetermined system
Il-posed problem
(uniqueness is lacking)
Sensing matrix Φ
=x
Measurement process Reconstruction process
Sparse recovery:
- Non-linear optimization that promotes sparsity
- Find the sparsest solution that is consistent with measured data
y’ = (Φ Ψ)-1 m
Iterative searching for…
min 𝑦 0
such that
m = Φ Ψ y
i = Ψ y
![Page 21: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/21.jpg)
21 Proprietary & Confidential
Overview – How we use Matlab for Big Data Acquisition
• Life Science, Electron Microscopy
• Matlab Integration: Image Acquisition Stability
• Compressive Sensing Basics
• Image Acquisition with CS
• Matlab: Reconstruction Algorithms
• Matlab Integration: Integration to Acquire Big Data
![Page 22: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/22.jpg)
22 Proprietary & Confidential
What we need for the Compressive Sensing for EM?
• Sparse scanning
on SEM/STEM
• Reconstruction
algorithms
ReconstructionReference Simulated
acquisition
SEM scan
control
Reconstruction
Promise of high speed and low dose imaging.
![Page 23: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/23.jpg)
23 Proprietary & Confidential
Scanning strategies
Visiting of random positions
Minimum
path scan
1 μs 300 ns
SEM Elstar; HFW:6um; Sparsity 20%
Pseudo-raster
Optimize global settling time
![Page 24: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/24.jpg)
24 Proprietary & Confidential
Overview – How we use Matlab for Big Data Acquisition
• Life Science, Electron Microscopy
• Matlab Integration: Image Acquisition Stability
• Compressive Sensing Basics
• Image Acquisition with CS
• Matlab: Reconstruction Algorithms
• Matlab Integration: Integration to Acquire Big Data
![Page 25: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/25.jpg)
25 Proprietary & Confidential
Reconstruction algorithms
Too many in the air …
Signal characteristic(s) Sparsifying transform Ψ
Locally periodic Discrete cosine (DCT)
Periodic Fourier (DFT)
Piecewise smooth Wavelets
Piecewise constant Finite differences (TV)
Specific Custom dictionaries
Tested
Selected
Reference
GOAL 20%
BPFA 20%
GO
AL –
Ge
Om
etr
icA
na
lysis
op
era
tor
Le
arn
ing
BP
FA
–B
eta
Pro
ce
ss F
acto
r A
na
lysis
![Page 26: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/26.jpg)
26 Proprietary & Confidential
Reconstruction algorithm comparison at 50%
GSR_SBI TCSVT_IRJSM BPFA
ALSBRCoSBP + DCT
Reference
DT=1us
Raster 500ns
![Page 27: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/27.jpg)
27 Proprietary & Confidential
Reconstruction algorithm comparison at 20%
Reference
DT=1us
Raster 200ns
GSR_SBI TCSVT_IRJSM BPFA
ALSBRCoSBP + DCT
![Page 28: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/28.jpg)
28 Proprietary & Confidential
Dose-Sparsity Image Quality Analysis
Helios datasets
Conditions: BSE mode; pixel size: 5nm; beam
current: 400pA; Field of view: 2.56µm.
Reconstruction using the GOAL algorithm
Reconstruction Fidelity Metrics
![Page 29: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/29.jpg)
29 Proprietary & Confidential
Dose-sparsity image quality analysis
10nm/pixel
5nm/pixel
![Page 30: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/30.jpg)
30 Proprietary & Confidential
Overview – How we use Matlab for Big Data Acquisition
• Life Science, Electron Microscopy
• Matlab Integration: Image Acquisition Stability
• Compressive Sensing Basics
• Image Acquisition with CS
• Matlab: Reconstruction Algorithms
• Matlab Integration: Integration to Acquire Big Data
![Page 31: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/31.jpg)
31 Proprietary & Confidential
How Matlab helps to evaluate Compressive Sensing?
Cutting, CS Acquisition & Processing
control in Python
CS algorithms
Matlab
Shared data-storage
Processing systems
![Page 32: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/32.jpg)
32 Proprietary & Confidential
Big Dataset acquisition
Volume: 122 x 110 x 80 um (Voxel: 10x10x30nm)
Section count: 2,683
Total data size: ~ 1.6TB, ~400 Ga Voxels
Helios, UHR, 3.5mm, 200pA, 1.8kV, 1us
3x3 tiles, stitched image size: 12200 x 11000
11 days continuous acquisition
Simultaneous acquisition for:
- Full frame raster images
- 20% compressive sensing with reconstruction
- Overview image
Cycle time: 315 secs
![Page 33: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/33.jpg)
33 Proprietary & Confidential
Large Dataset on Helios
Sparse Volume
Reconstruction
Full Grid Volume
Side by side
Comparison
Differential
View
![Page 34: Matlab + Compressive Sensing + Scanning Electron Microscopy · Very Short introduction to Scanning Electron Microscopy Beam-specimen interactions 10um 1um. 8 Proprietary & Confidential](https://reader033.fdocuments.net/reader033/viewer/2022050310/5f728b1cad57be2dcb52e4cd/html5/thumbnails/34.jpg)
34 Proprietary & Confidential
To take away
• Matlab is a nice computational tool
• It is even nicer when integrated in full workflow
• It is very good to evaluate algorithms
(image sharpness, compressive sensing, …)