1. 1. INNOVATIVE METHODS FOR THE RECONSTRUCTION OF NEW GENERATION SATELLITE REMOTE SENSING IMAGES November 29th, 2012 PhD Student: Luca Lorenzi lorenzi@disi.unitn.it Ph.D. thesis defense Advisors: Farid Melgani Grgoire Mercier
2. 2. Introduction General Problem Missing data in VHR optical image; Mainly due to acquisition conditions, e.g., the presence of: clouds: partially or completely missing data; shadows: partially missing data. 2 MODIS: Black see GeoEye-1: Doha Stadium, Qatar
3. 3. Introduction General Solution A common solution approach: 1. Pre-process the image (co-registration, calibration); 2. Detect the location of the contaminated regions; 3. Attempt to restore the missing areas. 3 Missing Area Detection Missing Area Reconstruction Original Image Image Pre- Processing Restored Image
4. 4. Objective Propose new methodologies for the reconstruction of missing areas for new generation satellite remote sensing images. Missing areas due to the presence of: Clouds Shadows 4
5. 5. Cloud-Contaminated Images Contribution 1: different solutions based on the inpainting approach. Three strategies: local image properties; isometric transformations; multiresolution processing scheme. Contribution 2: to improve the reconstruction process by integrating both radiometric and spatial information, through a specific kernel. Contribution 3: new methods based on the Compressive Sensing theory. Three strategies: Orthogonal Matching Pursuit (OMP); Basis Pursuit (BP); An alternative solution based on Genetic Algorithms (GAs). 5
6. 6. Shadow-Contaminated Images Contribution 4: a novel approach to solve both problems of detection and reconstruction. Shadow detection is performed through a hierarchical supervised classification scheme, while the proposed reconstruction relies on a linear prediction function, which exploits information returned by the classification. Contribution 5: to try to answer to the following question: Is it possible to know a priori if a shadow area can be well recovered? Eight different criteria. A fuzzy logic combination is explored. 6
7. 7. two contributions in detail In the next Slides
8. 8. MISSING AREA RECONSTRUCTION IN MULTISPECTRAL IMAGES UNDER A COMPRESSIVE SENSING PERSPECTIVE PUBLISHED IN THE IEEE TRANSACTIONS ON GEOSCIENCE AND REMOTE SENSING, VOL. 51, IN PRESS, 2013 L. Lorenzi, F. Melgani, and G. Mercier
9. 9. Problem formulation In I(1), any pixel can be expressed as: We have to evaluate: From I(1) : In I(2) : 9 2,1,)()()( iI iii )1()1( x )1()1()1( ,, lkx )1()1( , xf )2()2( x f ? source area missing area
10. 10. Compressive sensing (CS) Compressive sensing theory [1] Idea: exploit redundancy in signals Signals like images are sparse many coefficients close to zero To enforce the sparsity constraint, CS finds a vector which minimizes: D is a dictionary with a predefined number of atoms; x is the original pixel, expressed by a linear combination of atoms; Eq. (1) represents a NP-hard problem computationally infeasible to solve. Cands and Tao [2], reduce the Eq. (1) in a relatively easy linear programming solution: under some reasonable assumptions: 10 Dxsubject tomin 0 0:#0 ii (1) [1] D. L. Donoho, Compressed Sensing, IEEE Trans. Inf. Theory, vol. 52, no. 4, pp. 1289-1306, Apr 2006. [2] E. J. Cands and T. Tao, Decoding by Linear Programming, IEEE Trans. Inform. Theory, vol. 51, no. 12, pp. 4203-4215, Dec. 2005. 01 minmin
11. 11. Orthogonal Matching Pursuit (OMP) 12 [3] Y. C. Pati, R. Rezaiifar and P. S. Krishnaprasad, Orthogonal Matching Pursuit: Recursive Function Approximation with Applications to Wavelet Decompositions, in Proc. 27th Asilomar Conf. on Sig., Sys. and Comp., Nov. 1-3, 1993. Orthogonal Matching Pursuit (OMP) [3] finds the atoms which has the highest correlation with the signal: where dictionary D is a collection of atom vectors and R(m) is a residual. It updates the coefficients of the selected atoms at each iteration (adopting a least-squares step), so that the resulting residual vector R is orthogonal to the subspace spanned by the selected atoms. )( 1 m m i dd Dd dd Rx ii Ddd
12. 12. Orthogonal Matching Pursuit (OMP) 13 OMP pseudo code i=0: x(0)=0 , R(0)=x and D(c0)={} i=k: i=m: ,)( 1 )( m m i dd m Rx ii )(m RR Step 1: find which ; add to the set of selected variables; update . Step 2: let denote the projection onto the linear space spanned by the elements of update Step 3: compute s.t. Step 4: if ||R(k)||
13. 13. Basis Pursuit (BP) To solve Eq. (1), we may adopt the Basis Pursuit principle [4]: convexification from L0 to L1; Thanks to that, it becomes a support minimization problem; Eq. (2) can be reformulated as a linear programming (LP) problem, and solved using the Simplex methods. Given that, it is possible to rewrite L1 norm in Eq. (2) as: where If we substitute it in Eq. (2), it allows to perform a linear minimization problem. 14 [4] S. S. Chen, D. L. Donoho and M. A. Saunders, Atomic Decomposition by Basis Pursuit, SIAM J. on Sci. Comp., vol. 20, pp. 33-61, 1999. Dxsubject tomin 1 (2) i iii i vu1 00, 00, iiii iiii ifuv ifvu
14. 14. Genetic Algorithm (GA) To cope with complex optimization problems, there exist metaheuristic techniques, like the evolutionary algorithms. Genetic Algorithms (GAs) [5] are: inspired by evolutionary biology; general purpose randomized optimization techniques; based on simple rules. Their basic idea is to evolve iteratively a population of chromosomes, each representing a candidate solution to the considered problem. In complex problems requiring the simultaneous optimization of multiple objectives, GAs are particularly indicated for they deal simultaneously with a population of solutions. Advantages: little information about the problem required; robust to local optima; optimization of real, integer and binary problems. 15 [5] L. Chambers, The Practical Handbook of Genetic Algorithms. New York: Champan & Hall, 2001.
15. 15. GA Principal Steps 1. Initial population of M chromosomes is generated; 2. The goodness of each chromosome is evaluated according to predefined fitness functions; 3. Successively, the GA favors the selection of the best chromosomes and removes the others; 4. In the next step a new population is represented by adopting genetic operators, e.g., crossover and mutation; 5. All these steps are iterated until a predefined condition is satisfied. 16 Condition satisfied? Nextiteration Y N
16. 16. GA Setup Idea: exploit the GA capabilities for solving the L0 norm problem. The NP-hard problem is: Chromosome structure: The number of chromosomes M must be fixed; i is a chromosome which contains w genes with real values; The length w of the chromosome is thus equal to the one of the dictionary D. 17 Gene 1 Gene 2 Gene w i1 i2 iw Chromosome i Fitness functions: Multiple objective optimization [6] 01 min f 2 22 min xDf NSGA-2 Dxsubject tomin 0 (1) [6] K. Deb, Multi-Objective Optimization Using Evolutionary Algorithms. Chichester, U.K.: Wiley, 2001.
17. 17. Experimental Dataset Aim: to compare the results obtained by the CS reconstructions with: Multiresolution Inpainting (MRI) [7] Contextual Multiple Linear Prediction (CMLP) [8] In order to quantify the reconstruction accuracy: 1. consider I(1) cloud-free 2. simulate a presence of clouds by obscuring partly I(2) 3. compare the reconstructed I(2) with its original one Evaluate the sensitivity to two aspects: Test 1: kind of ground cover obscured Test 2: size of the contaminated area 20 [7] L. Lorenzi, F. Melgani and G. Mercier, Inpainting Strategies for Reconstruction of Missing Data in VHR Images, IEEE Geosci. Remote Sens. Letters, vol. 8, no. 5, pp. 914-918, Sep. 2011. [8] F. Melgani, Contextual Reconstruction of Cloud-Contaminated Multitemporal Multispectral Images, IEEE Trans. Geosci. Remote Sens., vol. 44, no. 2, pp. 442455, Feb. 2006.
18. 18. Multiresolution Inpainting (MRI) Based on the Region-based inpainting (RBI) from Criminisi et al [9] 21 [7] L. Lorenzi, F. Melgani and G. Mercier, Inpainting Strategies for Reconstruction of Missing Data in VHR Images, IEEE Geosci. Remote Sens. Letters, vol. 8, no. 5, pp. 914-918, Sep. 2011. [9] A. Criminisi, P. Perez and K. Toyama, Region filling and object removal by exemplar-based image inpainting, IEEE Trans. on Image Process., vol. 13, no. 9, pp. 1-14, Sep 2004. In [7], we proposed a processing scheme which recursively injects multiresolution information for a better reconstruction of the missing area.
19. 19. Contextual Multiple Linear Prediction Contextual Multiple Linear Prediction (CMLP) [8] : Training: Prediction: 22 A simple solution is based on the minimum square error pseudoinverse technique. [8] F. Melgani, Contextual Reconstruction of Cloud-Contaminated Multitemporal Multispectral Images, IEEE Trans. Geosci. Remote Sens., vol. 44, no. 2, pp. 442455, Feb. 2006.
20. 20. Test 1 Contamination of Different Ground Cover We suppose multiple land cover contamination, namely different areas of the image are missing Each mask is composed by ~2,000 pixels 23
21. 21. Test 1 Contamination of Different Ground Cover Dictionary D is composed by pixels regularly sub- sampled from region: 316 for DS1 402 for DS2 24 D Sub-sampling
22. 22. Test 1 Datasets 1 & 2 26 Mask A Mask B Mask C PSNR Complexity Time [s] PSNR Complexity Time [s] PSNR Complexity Time [s] MRI 22.54 - 2856 16.05 - 2517 33.77 - 2898 CMLP 20.99 1 1 20.11 1 1 24.05 1 1 OMP 23.96 3 4 20.60 3 4 31.97 3 4 BP 22.22 294 66 24.74 168 59 30.67 301 60 GA 23.78 148 68621 23.15 95 26312 32.01 138 43193 Mask A Mask B PSNR Complexity Time [s] PSNR Complexity Time [s] MRI 24.27 - 2995 29.54 - 3614 CMLP 24.61 1 1 27.69 1 1 OMP 26.36 3 5 30.43