Unsupervisedreal …We have developed constrained linear discriminant anal-ysis (CLDA) algorithm for...

Pattern Recognition 40 (2007) 1510–1519www.elsevier.com/locate/pr

Unsupervised real-time constrained linear discriminant analysis tohyperspectral image classification

Qian Du∗

Department of Electrical and Computer Engineering, Mississippi State University, MS 39762, USA

Received 2 November 2005; received in revised form 13 June 2006; accepted 14 August 2006

Abstract

We have proposed a constrained linear discriminant analysis (CLDA) approach for classifying the remotely sensed hyperspectral images.Its basic idea is to design an optimal linear transformation operator which can maximize the ratio of inter-class to intra-class distancewhile satisfying the constraint that the different class centers after transformation are aligned along different directions. Its major advantageover the traditional Fisher’s linear discriminant analysis is that the classification can be achieved simultaneously with the transformation.The CLDA is a supervised approach, i.e., the class spectral signatures need to be known a priori. But, in practice, these informations maybe difficult or even impossible to obtain. So in this paper we will extend the CLDA algorithm into an unsupervised version, where theclass spectral signatures are to be directly generated from an unknown image scene. Computer simulation is used to evaluate how wellthe algorithm performs in terms of finding the pure signatures. We will also discuss how to implement the unsupervised CLDA algorithmin real-time for resolving the critical situations when the immediate data analysis results are required.� 2006 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved.

Keywords: Hyperspectral imagery; Classification; Constrained linear discriminant analysis; Unsupervised constrained linear discriminant analysis;Real-time processing

1. Introduction

Because of high spectral resolution and resultant con-tiguous spectral signatures, hyperspectral image data arecapable of providing more accurate identification of surfacematerials than multispectral data, and are particularly usefulin national defense related applications. In some cases, suchas national disaster assessment, law enforcement activities,and military applications, real-time data processing is in-evitable to process data online and provide the informationfor immediate response.

We have developed constrained linear discriminant anal-ysis (CLDA) algorithm for hyperspectral image classifica-tion [1]. In CLDA, the original high-dimensional data isprojected onto a low-dimensional space as done by Fisher’s

∗ Tel.: +10 662 325 2035.E-mail address: [email protected].

0031-3203/$30.00 � 2006 Pattern Recognition Society. Published by Elsevier Ltd. All rights reserved.doi:10.1016/j.patcog.2006.08.006

LDA, but different classes are forced to be along differentdirections in this low-dimensional space. Thus, all classesare expected to be better separated, and the classificationis achieved simultaneously with the CLDA transform. Thetransformation matrix in CLDA maximizes the ratio of inter-class distance to intra-class distance while satisfying theconstraint that the means of different classes are alignedwith different directions. The experimental results in Ref. [1]demonstrated that the CLDA algorithm could provide moreaccurate classification results than other popular methods inhyperspectral image analysis.

In order to implement the CLDA algorithm in real-time,we modified the formula and adaptively adjusted �−1, theinverse sample covariance matrix, in our recent papers[2,3]. In our research, we assume that an image is acquiredfrom left to right and from top to bottom. Two real-timeprocessing fashions were discussed. One was pixel-by-pixelprocessing, and the other was line-by-line processing. In thepixel-by-pixel fashion, a pixel was processed right after it

http://www.elsevier.com/locate/pr

mailto:[email protected]

Q. Du / Pattern Recognition 40 (2007) 1510–1519 1511

was received and the analysis result was generated withinan acceptable delay. While in the line-by-line fashion, a lineof pixels was processed after the entire line was received.The experimental results in Refs. [2,3] demonstrated theeffectiveness of the real-time CLDA algorithm.

The CLDA is a supervised approach, i.e., the class spectralsignatures need to be known a priori. But in practice, theseinformation may be difficult or even impossible to obtain, inparticular, when dealing with remote sensing images. Thisis due to the facts that: (1) any atmospheric, background,and environmental factors may have impact on a spectralsignature, which makes the in-field spectral signature of amaterial or object not be well correlated to the one definedin a spectral library; (2) a hyperspectral sensor may extractmany unknown signal sources because of its very high spec-tral resolution, whose spectral signatures are difficult to bepre-determined; (3) an airborne or spaceborne hyperspectralsensor can take images from anywhere, whose prior back-ground information may be unknown and difficult to obtain.So in this paper we will extend the CLDA algorithm intoan unsupervised version without assuming any prior knowl-edge. We will investigate an unsupervised signature genera-tion algorithm. The generated signatures are assumed to bethe most distinctive in the image scene so that they can beclaimed as different classes. The real-time implementationof the proposed unsupervised CLDA algorithm will also beinvestigated.

2. Unsupervised CLDA algorithm

2.1. CLDA algorithm and its real-time implementation

Let S denote the entire class signature matrix, i.e., c classmeans. It was proved in Ref. [2] that the CLDA-based clas-sifier can be equivalently expressed as

PTk = [0 · · · 0 1︸︷︷︸

k

0 · · · 0](ST�−1S)−1ST�−1 (1)

for classifying the kth class in S, where � is the samplecovariance matrix.

The major advantage of using Eq. (1) is the capabilityof real-time implementation since the data whitening pro-cess is avoided. The key is the adaptation of �−1. Detailsabout the strategy for its real-time adaptation can be found inRef. [3].

In real applications, target and background signatures inS may be unknown. In our research, these signatures aregenerated directly from the image scene in an unsupervisedfashion. We will introduce the constrained linear unmixingproblem, then present an unsupervised class signature gen-eration algorithm based on constrained least squares linearunmixing error and quadratic programming (QP). After theclass signatures in S are determined, Eq. (1) can be applieddirectly for classification.

2.2. Constrained linear unmixing

Because of the rough spatial resolution, it is generallyassumed that the reflectance of a pixel in a remotely sensedimage is the linear mixture of reflectances of all the materialsin the area covered by this pixel. Let x denote a hyperspectralpixel vector of size L × 1 with L spectral bands. Accordingto the linear mixture model, x can be represented as

x = S� + n, (2)

where S=�s1, s2, . . . , sp� is an L×p signature matrix withp linearly independent endmembers (including desired tar-gets, undesired targets, and background objects) and si isthe ith endmember signature; � = (�1�2 . . . �p)T is a p × 1abundance fraction vector, where the ith element �i repre-sents the abundance fraction of si present in that pixel; n isan L × 1 vector which can be interpreted as noise term ormodel error. Abundances of all the endmembers in a pixelare related as

∑pi=1 �i = 1 and 0��i �1 for any i, which

are referred to as sum-to-one and non-negativity constraints,respectively.

The task is to estimate � with these two constraints beingsatisfied for each pixel, which is referred to as fully con-strained linear unmixing [4]. When S is known, there arep unknown variables to be estimated with L equations andL?p. This means the problem is over-determined, and nosolution exists. However, we can formulate a least squaresproblem to estimate the optimal �̂ that minimizes the esti-mation error defined as

e = ‖x − x̂‖2=‖x − S�̂‖2=xTx − 2�̂TSTx + �̂TSTS�̂. (3)

When the two constraints are to be relaxed simultaneously,there is no closed form solution. Fortunately, if S is known,this constrained optimization problem can be formulated intoa typical QP problem:

Minimize f (�̂) = xTx − 2�̂TSTx + �̂TSTS�̂

Subject to �̂1 + �̂2 + · · · + �̂p = 1, 0� �̂i �1

for 1� i�p. (4)

QP refers to an optimization problem with a quadratic ob-jective function and linear constraints (including equalityand inequality constraints). It can be solved using nonlin-ear optimization techniques. But we prefer to use linearoptimization-based techniques in our research since they aresimpler and faster [5]. With the help of MATLAB optimiza-tion toolbox, QP can be easily applied to solve Eq. (4) whenS is known.

2.3. Unsupervised signature generation

When S is unknown, endmembers can be generated us-ing the following algorithm based on linear unmixing errorand QP. Initially, a pixel vector is selected as an initial sig-nature denoted by s0. Then it is assumed that all other pixelvectors in the image scene are made up of s0 with 100%

1512 Q. Du / Pattern Recognition 40 (2007) 1510–1519

Fig. 1. The spectra of AVIRIS five signatures.

abundance. This assumption certainly creates estimation er-ror. The pixel vector that has the largest least squares error(LSE) between itself and s0 is selected as a first endmem-ber signature denoted by s1. Because the LSE between s0and s1 is the largest, it can be expected that s1 is most dis-tinct from s0. The signature matrix S=[s0s1] is then formedto estimate the abundance fractions for s0 and s1, denotedby �̂0(x) and �̂1(x) for pixel x, respectively, by using theQP-based constrained linear unmixing technique in Section2.2. Now the optimal constrained linear mixture of s0 ands1, x̂ = �̂0(x)s0 + �̂1(x)s1, is used to approximate the x. TheLSE between x and x̂ is calculated for all pixel vectors. Onceagain a pixel vector that yields the largest LSE between it-self and its estimated linear mixture will be selected to be asecond endmember signature s2. As expected, the pixel thatyields the largest LSE is the most dissimilar to s0 and s1,and most likely to be an endmember pixel yet to be found.The same procedure with S = [s0s1s2] isrepeated until theresulting LSE is below a prescribed error threshold �.

In order to investigate the performance of this QP-basedsignature generation algorithm, computer simulations wereconducted. The objectives are to assess its capability of find-ing the pure pixels in an image scene and the sensitivity tothe initial signature s0. The used data set was five AVIRIS re-flectance signatures, blackbrush, creosote leaves, dry grass,red soil, and sagebrush, as shown in Fig. 1. Five hundredpixels were generated via linear combination of these fivesignatures. White Gaussian noise with zero mean was addedto each pixel to achieve 30:1 SNR. Red soil and dry grasswere used as background signatures, and 480 backgroundpixels were made from these two signatures by generatingtwo random variables (r.v.) within [0.1, 0.9] whose sum wasequal to 1. As shown in Table 1, 10 pixels contain objects.Among them only five were pure pixels located at 100, 200,300, 400 and 500. Another five pixels were well mixed, con-taining some portions of the three foreground signatures.

Table 1Abundance fractions of 500 simulated pixels

Index Blackbrush Creosote Sagebrush Redsoil Drygrass

100 1 0 0 0 0200 0 1 0 0 0300 0 0 1 0 0400 0 0 0 1 0500 0 0 0 0 1

50 0.1 0.3 0.5 0.05 0.05150 0.5 0.3 0.1 0.05 0.05250 0.9 0 0 0.05 0.05350 0 0.9 0 0.05 0.05450 0 0 0.9 0.05 0.05

Others 0 0 0 r.v. r.v

Five cases were simulated:Case 1: Partial knowledge about endmember signatures

was known, which was correct (i.e., pure pixel). This purepixel (e.g., pixel 100) was used to initiate the algorithm. Thenall the other four pure pixels were picked up as endmemberssubsequently.

Case 2: Partial knowledge about endmember signatureswas known, which was incorrect (i.e., a mixed pixel).A mixed pixel (e.g., pixel 250) was used to initiate thealgorithm. Then all the five pure pixels were picked up asendmembers subsequently.

Case 3: No knowledge about endmember signatures wasknown, and the algorithm was initiated by the pixel withthe maximum length (i.e., 300), which was pure. Then theother four pure pixels were picked up as endmembers sub-sequently.

Case 4: No knowledge about endmember signatures wasknown, and the algorithm was initiated by the pixel withthe minimum length (i.e., pixel 100), which was pure. Thenthe other four pure pixels were picked up as endmemberssubsequently.

Case 5: No knowledge about endmember signatures wasknown, and the algorithm was initiated by a randomly pickedpixel. A mixed pixel (e.g., pixel 1) was randomly selectedas initial. Then all the other five pure pixels were picked upas endmembers subsequently.

This computer simulation demonstrates the capability ofthe unsupervised signature generation algorithm: (1) whenthe data SNR is moderate and pure pixels are present, itwill extract the pure pixels first because they are the mostdistinct pixels in the data; (2) when no knowledge aboutpure pixels is available, using the pixels with maximum orminimum norm seem to be a good choice (or both as thefirst two signatures), because these pixels are most likelyto be pure; (3) it is always a good idea to remove the firstinitial signature s0 from S particularly when it is randomlyselected, because if it is mixed, it should be removed; but ifit is pure, it will be eventually extracted in a late step.


3. Unsupervised real-time CLDA algorithm

In the real-time processing, a pixel (pixel-by-pixel fash-ion) or a line of pixels (line-by-line fashion) is processedright after it is received. Obviously, the pixel-by-pixel fash-ion is not feasible in the unsupervised case, because theunsupervised signature generation algorithm in Section 2cannot be performed on a single pixel basis. In the line-by-line fashion, unsupervised signature estimation algorithmshould be performed to find the class signatures after eachline is received. However, since this algorithm is iterativeand the chance for new classes to exist in each line is small,time may be wasted if the algorithm is performed for eachline. In order to reduce the processing delay, the real-timefashion in our research was changed to “after several lines ofpixels are received, signatures of new classes will be gener-ated and CLDA algorithm will be applied for classification”.

It should be noted that the line-by-line processing hasdifficulty in practical implementation [3], because it requiresa large matrix inversion module. Therefore, we will have adata buffer to hold the lately received lines of pixels for classsignature generation. But after the matrix S is updated, theselines of pixels will still be classified pixel-by-pixel since thepixel-by-pixel processing does not require a matrix inversionmodule.

Another problem is about when to terminate the unsuper-vised signature generation algorithm. One way is to stop thealgorithm when the maximum least squares estimation erroramong all the pixels currently used is less than a threshold�. The � is easy to determine at the first stage (i.e., whenprocessing the first several lines of pixels of an image). Butit may be varied at the following stages and difficult to ad-just. Fortunately, we found out that after the unsupervisedsignature generation algorithm has extracted enough differ-ent class signatures, one more class signature to be extractedwould be linearly dependent to one of signatures that havebeen extracted. In other words, if there is no more distinctsignature present in the image scene but the algorithm isforced to execute continuously, the new extracted signaturewill not be independent of the signatures in S. Therefore,whether or not the algorithm should be stopped can be de-termined by checking the linear independence between thelately extracted signature and those extracted signatures al-ready in S. This can be easily accomplished by temporarilyincluding the lately extracted signature into S and evaluat-ing the rank of S: if the rank is one less than the numberof signatures, this means the lately added signature is lin-early dependent with some of old signatures, and it shouldbe removed.

In summary, the unsupervised real-time CLDA algorithmcan be described as follows.

(1) After receiving the first q lines of pixels at time t1, usingthe unsupervised signature generation algorithm to findthe distinct pixels for constructing the signature matrixS. The algorithm was initiated using the pixel s0 with

Fig. 2. The HYDICE image scene used in the experiment.

the maximum or minimum norm. The constructed S isdenoted as S1 = [s1, s2, . . . , sq1 ], where q1 denotes thenumber of classes present in the first q lines of pixels.

(2) The CLDA classifier is constructed using Eq. (1) withS being substituted by S1, and applied to the q lines ofpixels one after another.

(3) After receiving the second q lines of pixels at time t2,using the unsupervised signature generation algorithmto find the new distinct pixels for updating the signaturematrix S. Now the algorithm was initiated using S1, andthe resulting S2 has q1 + q2 signatures, where q2 is thenumber of new classes extracted within the second qlines of pixels.

(4) Go to step 2 to update the CLDA classifier using S2, andclassify the second q lines of pixels one by one. Repeatthe same procedure for the mth q lines of pixels at timetm until the entire image scene is received and classified.

Several comments are noteworthy.

(1) It is possible that there is no new class present in thelately received q lines of pixels. Then these pixels willbe classified using the previous S matrix.

(2) The independence among the extracted signatures ischecked in the signature generation algorithm. But dueto the spectral variation and noise presence, two pixelsbelonging to the same classes may be treated as the pix-els for different classes, and both of them are extractedinto the S matrix. So the final number of classes may belarger than the actual. But this may not be a problem ifthe objective of image analysis is to find the anomalousor suspicious target pixels in surveillance applications.

(3) As shown by the computer simulation in Section 2.3,the unsupervised signature generation algorithm can ex-tract the pure pixels if any. In real-time processing, if nopure pixel exists at first, it will extract the most distinctmixed pixels. Then if a pure pixel for the same class isreceived later, the pure pixel will be extracted as well.


Table 2Classified images at different states in the unsupervised real-time processing


Table 2continued


Table 2Continued


As a result, a single class will also be split. When thesituation allows for a post-processing and class mergingis really necessary, classes can be merged by evaluat-ing the similarity of their corresponding signatures in Susing a similarity measure, such as spectral angle map-per (SAM). When the similarity between two class sig-natures is above a threshold, these two signatures canbe combined using their averaged signature. Then theS matrix in Eq. (1) will be updated accordingly for re-classification.

4. Experiment

The same image scene in Ref. [1] was used in the ex-periment, which is shown in Fig. 2. It contains five rows ofpanels, and each row of panels belongs to the same class,which are denoted as P1, P2, P3, P4, and P5, respectively.

The classification sequence is shown in Table 2, wherethe classified images of the same class at different stageswere displayed in a column, and the classified images ofall the classes at the same stage were in a row. Only thosestages finding new classes were presented here. As a whole,30 classes were extracted within 16 stages. At stage 1, theunsupervised signature generation algorithm was executedafter the first four lines of pixels were received. The algo-rithm was initiated by finding the pixel with the maximumnorm. Here, six class signatures were found and classifiedusing the CLDA algorithm. At stage 2, lines 5–8 were re-ceived and processed. The existing six signatures were usedto initiate the algorithm and another three new class signa-tures were found. Then the total nine classes were classi-fied using the CLDA algorithm. At stage 3, lines 9–12 werereceived and processed. The existing nine signatures wereused to initiate the algorithm and another three new classsignatures were found and classified. At stage 4 for lines13–16, three new class signatures were found; at stage 5 forlines 17–20, five new class signatures were found; at stage6 for lines 21–24, three new class signatures were found;at stage 8 for lines 29–32, three new class signatures werefound; at stages 9, 12, 15 and 16, one new class signatureswere found; at Stages 7, 10, 11, 13, and 14, no new classwas found. So totally there were 30 classes being extractedand classified. We can see that all the panel classes, P1, P2,P3, P4, and P5, were correctly classified as Class 8 at Stage2, Class 17 at Stage 5, Class 24 at Stage 8, Class 28 at Stage12, and Class 29 at Stage 15, respectively.

The pixel-level ground truth about the five panel classeswas used for the quantitative comparison between the su-pervised real-time and unsupervised real-time classificationresults. The number of pixels in each panel class is 3, 4,4, 4, and 4, respectively. NC , the number of correctly clas-sified pixels, and NF , number of false alarm pixels, werecounted and listed in Table 3 by using the same approachin Ref. [3]. We can see that unsupervised real-time classifi-cation provided the same result as the supervised real-timeclassification.

Table 3The performance of supervised and unsupervised real-time classificationresults in the HYDICE experiment (NC : number of correctly classifiedpixels; NF : number of false alarm pixels)

NP Supervised real-time Unsupervised real-time

NC NF NC NF

P1 3 2 0 2 0P2 4 3 0 3 0P3 4 3 0 3 0P4 4 3 0 3 0P5 4 3 0 3 0

Total 19 14 0 14 0

5. Conclusion

In this paper, we extend the CLDA into an unsupervisedversion for hyperspectral image classification. This is to meetthe need in practical applications of remote sensing imageanalysis when the class spectral signatures have in-field vari-ation and difficult or even impossible to be pre-determined.An unsupervised signature generation algorithm is investi-gated which is based on constrained least squares linear un-mixing in conjunction with QP. It is easy to implement andvery efficient to extract distinct signatures in an image scenewith moderate SNR. In order to implement the unsupervisedCLDA algorithm in real-time so as to provide the immedi-ate data analysis results for resolving critical situations, sev-eral practical real-time implementation issues are discussed.The HYDICE experiment demonstrates that the proposedunsupervised real-time algorithm can correctly classify thetargets with similar spectral signatures without prior infor-mation.

Acknowledgment

This research was supported by National Geospatial-intelligence Agency (NGA) through Grant NMA401-02-1-2015.

References

[1] Q. Du, C. -I Chang, Linear constrained distance-based discriminantanalysis for hyperspectral image classification, Pattern Recognition 34(2) (2001) 361–373.

[2] Q. Du, H. Ren, Real-time constrained linear discriminant analysis totarget detection and classification in hyperspectral imagery, PatternRecognition 36 (1) (2003) 1–12.

[3] Q. Du, R. Nekovei, Implementation of real-time constrained lineardiscriminant analysis to remote sensing image classification, PatternRecognition 38 (4) (2005) 459–471.

[4] D. Heinz, C.-I Chang, Fully constrained least squares linear mixtureanalysis for material quantification in hyperspectral imagery, IEEETrans. on Geoscience and Remote Sensing 39 (3) (2001) 529–545.

[5] P. Venkataraman, Applied Optimization with MATLAB Programming,Wiley-Interscience, New York, 2002.


About the Author—QIAN DU received her Ph.D. degree in electrical engineering from University of Maryland Baltimore County in 2000. She was anassistant professor in the Department of Electrical Engineering and Computer Science at Texas A&M University-Kingsville from 2000–2004. Since Fall2004, she has been with the Department of Electrical and Computer Engineering at Mississippi State University as an assistant Professor. Dr. Du hasbeen working on Remote Sensing Image Analysis for many years with the expertise on Hyperspectral Imaging.She is a senior member of IEEE, and member of SPIE, ASPRS, and ASEE.

Unsupervisedreal …We have developed constrained linear discriminant anal-ysis (CLDA) algorithm for...

Documents

Transcript of Unsupervisedreal …We have developed constrained linear discriminant anal-ysis (CLDA) algorithm for...