Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep...
Transcript of Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep...
![Page 1: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/1.jpg)
Big Data, Machine Learning and Remote Sensing
Deep Learning for semantic segmentation of hyperspectral and Multispectral data
Amina Ben Hamida Alexandre Benoit, Patrick Lambert, Chokri Ben Amar
![Page 2: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/2.jpg)
●●
○○
![Page 3: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/3.jpg)
●●
○○
![Page 4: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/4.jpg)
Use case: Sentinel 2generates daily 1.6 TBytes of compressed raw image data from the two-satellite constellation.
● Self taught
● Unsupervised
● Supervised
Playing major roles in different domain:
● Agriculture
● Forecasting
● ….
![Page 5: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/5.jpg)
●●
○○
![Page 6: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/6.jpg)
● Bag of Visual Words
● Deep Learning Fine
tuningUC merced Database :
● 256*256 pixels, RGB color, 1 foot resolution
● 21 classes with 100 images for each one (2100 total)
![Page 7: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/7.jpg)
![Page 8: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/8.jpg)
Sparse gray BOVW 79%(SIFT) /86%(SURF)
Spatial Pyramid 78%(SIFT)
Sparse RGB BOVW 78%(SIFT) /85%(SURF)
Dense gray BOVW 87%(SIFT) / 91%(SURF)
Dense RGB BOVW 86%(SIFT) /90%(SURF)
Deep Learning fc7+linSVM 96%
Deep Learning fc7+newfc8 94%
[1] A. Ben Hamida, A. Benoit,P. Lambert C.Ben Amar .“Could multimedia approaches help remote sensinganalysis?,” in Conference Image Information Mining:Earth Observation meets Multimedia. IIM, 2015.
![Page 9: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/9.jpg)
●●
○○
![Page 10: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/10.jpg)
![Page 11: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/11.jpg)
![Page 12: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/12.jpg)
Temporal components are not taken into account
Taking into account the spatial and spectral components
Separately processing
the spectral and spatial
components using SAE
Combining the spatial and
spectral information at
early phases
Only taking into account
the spectral information
Explodes the
number of
training
parameters and
requires large
amount of data
Disregard the
spatial
component
that can hold
important
information
Perfectly fits the
concept of
hyperspectral /
Multispectral data
classification
![Page 13: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/13.jpg)
●●
○○
![Page 14: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/14.jpg)
The Pavia Center is a 102-band dataset that presents oneimage of size 1096×1096 pixels
The Pavia University is a 103-band dataset that presents oneimage of size 610×340 pixels
![Page 15: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/15.jpg)
![Page 16: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/16.jpg)
* architecture : 6 layers with 5× 5× 3 sized filters
![Page 17: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/17.jpg)
![Page 18: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/18.jpg)
![Page 19: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/19.jpg)
Squeezing the net on Pavia University[2]
![Page 20: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/20.jpg)
Squeezing the net on Pavia Center[2]
![Page 21: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/21.jpg)
8layers: 3*3 8layers: 5*5
8layers: 3*3 8layers: 5*5 [4]
[4][3]
University of Pavia Pavia Center
* [3] accuracy of 98% from 9% of data for tranining with 20000 params* [2] accuracy of 99% from 9% of data for tranining with 7000 params
![Page 22: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/22.jpg)
At coarse reference resolutionAt high level resolution
Two main strategies for Land Cover estimation
![Page 23: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/23.jpg)
●●
○○
![Page 24: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/24.jpg)
fine grained pixel level land cover estimation
*Relying on light DenseNet and 3D DenseNet architectures
e4 e5 b7 d5 d4
g16
![Page 25: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/25.jpg)
Relying on light and generic architectures (DenseNet)
![Page 26: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/26.jpg)
Semantic segmentation on the ISPRS Vaihingen dataset
a: near infrared, red and green bands, b:DSM and c: semantic labels
full reference :9cm/pixel
![Page 27: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/27.jpg)
Input image full reference coarse reference inferred semantic map(NIR, red and green bands)
Full reference : 9cm/px
Coarse reference : 135cm/px
![Page 28: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/28.jpg)
Full reference : 9cm/px
Coarse reference : 135cm/px
train/test with full reference
train/test with coarse reference
Training on high resolution/confident reference improves results
![Page 29: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/29.jpg)
![Page 30: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/30.jpg)
2016 Sentinel-2 images and 2009 CORINE land cover dataset
Rescaled Sentinel-2 images : 20m/pixel
CORINE landcover reference : 300m/pixel
22 classes of
May-october images
with no cloud included
23 classes of
June-August images
with cloud included
![Page 31: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/31.jpg)
Input image CORINE reference inferred segmentation
T2
T1
* Model : e4_5_b7_g16
Limitations: Temporal consistency
![Page 32: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/32.jpg)
Only confident on large stable area but fail in rich heterogenous areas
Estimation at the coarse reference level using SegNets
The Multispectral approach performs better with less required images
![Page 33: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/33.jpg)
●●
○○
![Page 34: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/34.jpg)
The semantic segmentation in the absence of ground truth:● The resorting to generative models to learn data
distribution: fits the used sensors● Fine tuning over some annotated regions for more specific
applications ( semantic segmentation, classification…)
Finalize the current work on poorly annotated databases:● enriching the currently used models ( dense, ResNet..) with
larger datasets● Predictions stability study over same zones within a small
period of time
![Page 35: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/35.jpg)
● “Could multimedia approaches help remote sensing analysis?,” in Conference Image Information Mining: Earth Observation meets Multimedia. IIM, 2015.
● “Deep learning approach for remote sensing image analysis,” in Conference on Big Data from Space. BIDS, 2016.
● “Deep learning for semantic segmentation of remote sensing images with rich spectral content”, IGARS 2017
● “ Three dimensional Deep Learning approach for remote sensing image classification, TGARS
![Page 36: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/36.jpg)
[1] A. Ben Hamida, A. Benoit, P. Lambert C.Ben Amar .“Could multimedia approaches help remote sensing analysis?,” in Conference Image Information Mining:Earth Observation meets Multimedia. IIM, 2015.
[2]A. Ben Hamida, A. Benoit, P. Lambert, and C.BenAmar, “Deep learning approach for remote sensing image analysis,” in Conference on Big Data from Space. BIDS, 2016.
[3]Xiaorui Ma, Jie Geng, and Hongyu Wang, “Hyperspectral image classification via contextual deep learning,”EURASIP Journal on Image and Video Processing, vol. 2015, no. 1, pp. 1–12, 2015.
[4] S. Lefevre, L. Chapel, and F. Merciol, “Hyperspectral image classification from multiscale description with constrained connectivity and metric learning,” in 6th International Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing. Lausanne, Switzerland, 2014.
![Page 37: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/37.jpg)
![Page 38: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/38.jpg)
![Page 39: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/39.jpg)
“Rachel”
![Page 40: Remote Sensing - projects.listic.univ-smb.fr · Big Data, Machine Learning and Remote Sensing Deep Learning for semantic segmentation of hyperspectral and Multispectral data Amina](https://reader036.fdocuments.net/reader036/viewer/2022081409/607724f87b7b765dd148ae7f/html5/thumbnails/40.jpg)
Represent each image as a histogram of different visual words
Unsupervised Visual words
clustering
SIFT/SURF
Supervised Tiles
classificationBag of Visual Words
(BoVW)
Training tiles+ labels
Features extraction &Visual word
matching
Test phase
Supervised classifier(SVM, KNN, etc.)
Label:"Freeway"
Training with annotations
Test tile