Blind Signal Separation using Principal Component Analysis...

Blind Signal Separation using Principal Component

Analysis

ECE 539

Project Report

Alok Ahuja 9020863693

[email protected]

Introduction to Artificial Neural Networks and Fuzzy Systems

Blind Signal Separation using Principal Component Analysis 2

CONTENTS

Introduction 3

Motivation 4

Problem Formulation 5

Solution Proposal 7

PCA Algorithm 9

Simulation Results 11

Discussion 15

References 16

Appendix A 17



Introduction In signal separation, multiple streams of information are extracted from linear mixtures of these signal streams. This process is blind if examples of the source signals, along with their corresponding mixtures are unavailable for training. Blind Signal Separation (BSS) is sometimes used interchangeably with independent component analysis (ICA), although technically, BSS and ICA are different tasks. BSS is most appropriate in situations where a linear mixture model is plausible. Interest in blind signal separation has recently developed for three reasons: (1) the development of statistical frameworks for understanding the BSS task, (2) a corresponding development of several BSS methods, of which one is proposed in this project, and (3) the identification of many potential applications of BSS. Some of these applications are outlined below. Array processing in wireless communications – In modern-day wireless communication networks, several wireless devices attempt to communicate with a base station. The base station employs multiple antennas to better receive and isolate the various users’ signals within the spectral bandwidth of the communication channel. When the devices’ transmitted signals overlap in time and frequency, BSS can be applied to the antenna measurements to separate and enhance the various transmitted signals without knowing the devices’ positions relative to the base station and without knowing the exact forms of the transmitted signals [1]. Signal enhancement in medicine – Many noninvasive medical sensing technologies (e.g., EEG and MRI) characterize bodily processes through multi-channel recordings. Due to the complicated propagation properties of human body tissue, these multi-channel recordings can be difficult to decipher. BSS offers the potential of extracting coherent and identifiable signal features that can be more easily tied to specific bodily functions or ailments. [2]. Speech separation in acoustics – Speech signals that are collected by distant microphones in room environments can be hard to understand, particularly if multiple conversations are ongoing. BSS can be used to



separate individual speech signals from the microphones’ signals, thereby making them more intelligible and listenable [3]. The above three applications differ in the type of mixing generated by the measurement process. Generally, two types of mixing conditions are possible: (1) instantaneous or spatial mixing, in which the mixtures are weighted sums of the individual source signals without dispersion or time delay, and (2) convolutive or spatio-temporal mixing, in which the mixtures contain filtered, delayed, and reverberated versions of the individual signals due to multipath effects. In the examples described above, narrowband array processing in wireless communications involves spatial mixing conditions, whereby multi-microphone speech separation involves spatio-temporal mixing conditions.

Motivation I have tried to investigate the use of neural models for BSS. The application that I am aiming at is array processing, and this is reflected in the simulation runs that have been performed. There is a close equivalence between methods employed for BSS and blind beamforming, in that they utilize higher order statistics (HOS). The importance of blind beamforming can be understood from the fact that, in land mobile radio systems, it has been shown that spatial reuse of allocated frequency slots by employing an antenna array with multiple beams increases the system capacity substantially. The essential part in such a scheme is array beamforming [4]. Neural models for BSS are traditionally based on higher order moments, as already mentioned. For example, models based on PCA extensions, such as nonlinear PCA, perform analysis of signals into independent components. However, second order models have also used for BSS under the assumptions of non-correlation (as opposed to independence) and of the different spectral coloring of the sources. Yet these models were either based on second order optimization or on linear extensions of PCA (such as Asymmetric PCA). In this project, I have tried to implement a standard PCA neural model to perform BSS.



Problem Formulation

The above figure shows the structure of BSS for instantaneously mixed sources [5]. At the left is the unknown source signal vector at a particular instant of time, given by

]...,[ 21 qxxxx =

where xi is the ith source signal. These q source signals are linearly mixed by the (pXq) unknown mixing matrix M, yielding the p-dimensional measured or observed signal vector y as

nMxy +=

where n is the p-dimensional noise vector sequence that is unrelated to the source signal. In typical BSS applications, the source signals contain useful but unknown information, and the mixing model describes the undesirable



smoothing and propagation effects inherent in some physical measurement process. The inclusion of noise within the model accounts for any sensor noise at the measuring devices. An important feature of the BSS model is the relative values of q and p. We can distinguish three important cases of the type as outlined below:

1. When q=p, the number of sources is equal to the number of measurement sensors.

2. When q<p, the number of sources is fewer than the number of measurement sensors.

3. When q>p, the number of sources is more than the number of sensors. The case where q<p is more desirable in practice, as then all of the sources in x can be extracted from y if rank [M] = q using a linear system of the form

yMs)

=

in the noise free case. In other words, when the mixing conditions are linear and q<=p, a linear demixing system is sufficient to isolate each source signal in a separate output.



Solution Proposal The stress is on the use of standard PCA algorithms such as APEX [6] to perform BSS. Utilizing the approach of Diamantaras [7], it can be shown that BSS is PCA preceded by temporal filtering. This result is proved theoretically here and then the implementation verifies the result to a great degree.

Defining the observation and source vectors T

n kxkxkx )](},...{[)( 1= and

[ ]Tn ksksks )()...()( 1= we write ,

)()( ksAkx o= (1) The assumptions are as follows A1. The sources are at least wide sense stationary with zero mean and unit variance; furthermore, they are pair-wise uncorrelated:

{ } Is(k)s(k)E)(R Ts ==0 (2)

Although [A1] constrains the sources magnitude their sign is still ambiguous, while obviously, it does not affect the ordering ambiguity. It follows that if A is a solution to the BSS problem, then so is AP, for any signed permutation matrix P. The non-correlation between the sources is the second-order counterpart of the independence assumption employed in higher order methods. A2. There are 1≥M distinct, positive time lags, such that the lagged covariances )( ms lR are non zero:

{ } 0)()()( ≠=−= diaglksksElR Tmms (3)

A3. The source colors are distinct:

)()(:0 lrlrl jjii ≠≠∀ (4)



A common preprocessing step for many second and higher order BSS methods is the whitening of the observations. This is done by a transformation of the form )()0()( 5.0 kxRkx x

−= such that the correlation matrix becomes the identity. Consider now a temporally filtered version of :)(kx

∑=

−+=M

mmm lkxhkxky

1

)()()( (5)

where the lags satisfy the assumptions [A2]. The 0-lag covariance matrix is given by

∑=

−=M

qpqpxqpy llRhhR

0,

)()0( (6)

According to (1) it follows that

Tmsmx AlARlR )()( = (7)

Hence we get,

Ty ADAR =)0( (8)

with D given by ∑=

−=M

qpqps llRD

0,

)( (9)

Under assumption [A2], the matrix D is diagonal. In addition A is orthogonal since ( ) ( ) IRAARAA x

TS

T === 00 . Therefore, Equation (8) represents the Eigen value decomposition of )0(yR . Furthermore, the Eigen values are given by D. If the Eigen values are distinct, then the PCA of y leads to an estimate of A. Let U be the Eigen matrix of )0(yR then U=AP where P is a signed permutation matrix. The sources are estimated by

( ) )()(' ksPkxUks TT == (10)



PCA Algorithm I have used the Adaptive Principal Component Extraction (APEX) algorithm [6] to extract the Eigen values of the matrix D.

The above figure depicts the APEX model. There are n inputs { }nxx ....1 , connected to m outputs { }myy .....1 through the feed-forward weights { }ijw . Additionally, there exist lateral weights jc , forming vector c, that connect all the first m-1 units with the mth one. These connections play a very important role in the model: they work toward the orthogonalization of the synaptic weights of the mth neuron versus the extracted principal components stored in the weights of the previous m-1 neurons.



Eventually, all the weights in the network will span an orthonormal basis of the m-dimensional principal component subspace, while the lateral weights tend to zero as orthogonalization is achieved. The input vector ...,2,1,0, =kxk is a stationary stochastic process with positive-definite autocorrelation matrix XR . We shall arrange the normal Eigen vectors neee ,...,, 21 of XR so that the corresponding Eigen value sequence in the decreasing order: nλλλ ≥≥≥ ...21 . The activation of each neuron is a linear function of its inputs

Wxy =

ycxwy TTm −=

Only w and c are trained (W remains fixed). The kth iteration of the algorithm is

)( 21 kmkkmkkkk wyxyww −+=+ β

)( 21 kmkkmkkkk cyyycc −+=+ β

where â is a positive sequence of step-size parameters. Hence we have both Hebbian and anti-Hebbian learning rules incorporated in these equations. It

can be shown that with probability 1, )( mmk eorew −→ and ,0→kc as

∞→k .



Simulation Results

I have simulated a communications environment for testing the above theoretical results. Hence a random digital bit stream is generated which is sampled to get its sampled version. Next I have performed Pulse Amplitude Modulation with 5 amplitude levels symmetric around zero, to get the desired source signals with length N=200 each. There are n=4 source signals that have been used.

0 50 100 150 200-2

-1

0

1

2Observa t ion x1

0 50 100 150 200-5

0

5

10Observa t ion x2

0 50 100 150 200-4

-2

0

2

4Observa t ion x3

0 50 100 150 200-4

-2

0

2

4Observa t ion x4

The above figure shows the observation vectors after the hidden sources have been mixed using a mixing matrix that just has to satisfy the orthonormal property after its multiplication with the prewhitening matrix. But before this, the sources are passed through 4 random FIR filters of



length 20 to randomly color them. After this, the resulting mixture is spatially whitened, using the operator mentioned in the theory, i.e. the inverse square root of its covariance matrix.

The estimation of the covariance matrix was tricky. There are many iterative methods in literature, such as those of Douglas and Cichocki [8] , to calculate the prewhitening matrix. But in this case, I chose to divide the data into blocks of 4 samples each, and calculate the ensemble average correlation matrix of the 200 ensembles. This has served its necessary purpose as can be seen in the results. Next the filtered sequence is fed into the APEX model with 4 output neurons. The training is stopped after 60 sweeps of the entire data, when the model has obtained the steady-state. I could not figure a variable stopping criteria for the training, and hence experimented with the number of sweeps required for the results to be acceptable. Also, theory states that any filter sequence can be used to put into the model. Hence instead of using the time lagged version y(k)=x(k)+x(k-1), I just used y(k)=x(k) to test this hypothesis, with considerably decent reconstruction of the original hidden signals. The following page displays the results of the reconstruction of the hidden sources superimposed on the original hidden sources. The noteworthy point is that due to the random initialization of the sources, the results are local to each run.



0 20 40 60 80 100 120 140 160 180 200-15

-10

-5

0

5

10

15source1

Samples

Am

plut

ude

or ig inal sourcees t imated source

0 20 40 60 80 100 120 140 160 180 200-6

-4

-2

0

2

4

6source2

Samples

Am

plut

ude




0 20 40 60 80 100 120 140 160 180 200-5

-4

-3

-2

-1

0

1

2

3

4

5source3

Samples

Am

plut

ude


0 20 40 60 80 100 120 140 160 180 200-8

-6

-4

-2

0

2

4

6source4

Samples

Am

plut

ude




Discussion

The results are pretty decent judging by the sum square errors that have been obtained for the original and estimated sources, as also the shape of the reconstructed signals.

These results go on to prove the theoretical foundations that have been

established. Thus, under suitable assumptions, the blind signal separation problem can be solved using standard PCA neural models and temporal prefiltering. The advantage to prefiltering is that it practically imposes no constraints, since almost any filter will work. But the question of the optimal filter is still open. Also, now the BSS problem can be more efficiently addressed using second order methods such as PCA used in this project, since PCA is a well studied problem with a number of fast adaptive/neural algorithms such as GHA [9], APEX[6], etc.

It must be noted that the second order methods do not replace higher

order ones since each approach is based on different assumptions. Hence they are applicable to different situations also. Also, I haven’t found a second order method in literature which actually addresses a real world problem where BSS is applicable. Hence I believe that more study needs to go in the suitability of these algorithms under the case of perturbations as also sensitivity issues need to be addressed.



References [1] A.J. Paulraj and C.B. Papadias, Space-time processing for wireless communications, IEEE Signal Processing Magazine, vol. 14, no. 6, pp. 49-83, 1997. [2] M.J. McKeown, S.Makeig, G.G. Brown, T.P. Jung, S.S. Kindermann, A.J. Bell and T.J. Sejnowski, Analysis of fMRI data by blind separation into independent spatial components, Human Brain Mapping, vol. 6, pp. 160-188, 1998. [3] K. Torkkola, Blind separation of delayed and convolved sources, in Unsupervised Adaptive Filtering, Vol. I: Blind Signal Separation, S.Haykin, Ed., (New York: John Wiley & Sons, 2000), pp. 321-375. [4] Q. Wu and K.M. Wong, Blind adaptive beamforming for cyclostationary signals, IEEE Transactions on Signal Processing, vol. 44, no. 11, pp. 2757-2767, 1996. [5] S.C. Douglas, Blind signal separation and blind deconvolution, in Handbook of Neural Network Signal Processing, Y.H. Hu and J.N. Hwang, Ed., (Florida CRC Press, 2002), pp. 7-1 – 7-31. [6] S.Y. Kung and K.I. Diamantaras, Adaptive principal component extraction (APEX) and applications, IEEE Transactions on Signal Processing, vol. 42, no. 5, pp. 1202-1217, 1994. [7] K.I. Diamantaras, PCA neural models and blind signal separation, Proceedings of IJCNN 2001, International Joint Conference on Neural Networks, vol. 4, pp. 2997-3002, 2001. [8] S.C. Douglas and A. Cichocki, Neural networks for blind decorrelation of signals, IEEE Transactions on Signal Processing, vol. 45, pp. 2829-2842, 1997. [9] T.D. Sanger, Optimal unsupervised learning in a single-layer linear feedforward neural network, Neural Networks, vol. 2, no. 6, pp. 459-473, 1989.



Appendix A Source Code function [] = apexalgo() % apexalgo.m % Main program that generates the data % and implements the APEX algorithm % The variables are saved in the specified mat file % at the end of the program % The results shown are local to each simulation run % because of the random initialisation of the source vectors each time % Created 12/15/01 % Copyright Alok Ahuja clear all; close all; echo off; clc; disp('Starting the main Program...') disp(' ') % Mixing matrix A= [-0.7283,-0.5760,0.8071,0.8784; -0.8719,-0.6425,0.7825,0.9195; -0.6235,0.8974,-0.5981,0.5525; -0.7057,0.6310,-0.5692,-0.6003]; M = 5; % Number of symbols in alphabet len = 50; % Number of symbols in the original message Fd = 1; % Assume the original message is sampled % at a rate of 1 sample per second. Fs = 4; % The modulated signal will be sampled % at a rate of 3 samples per second. b=1; % Constant used in filter design ni=4; % Number of input patterns ntp=240; % Number of training passes neig=4; % Number of eigen vectors snr=10; % Signal to noise ratio ulen=4; % Length of observed vector utilised lr=0.01; % Learning Rate % -------------------------------------------------- % Create the random source signals



% -------------------------------------------------- for i=1:4, signal(:,i) = randint(len,1,M); % Random digital message % Creating a PAM signal with 5 levels modsignal(:,i) = dmodce(signal(:,i),Fd,Fs,'ask',M); end % Coloring the source signals % Generate the 20 length FIR coefficients g1=randn(1,20); g2=randn(1,20); g3=randn(1,20); g4=randn(1,20); % Displaying the frequency response of these filters disp('Press a key to display the frequency response of the 4 filters...') disp(' ') pause [h,w]=freqz(g1,b); figure(1) clf reset freqzplot(h,w); [h2,w2]=freqz(b,g2); figure(2) clf reset freqzplot(h2,w2); [h3,w3]=freqz(b,g3); figure(3) clf reset freqzplot(h3,w3); [h4,w4]=freqz(b,g4); figure(4) clf reset freqzplot(h4,w4); disp('Press any key to continue') disp(' ') pause; close all disp('Now creating the sources...') disp(' ') % Create the sources s1=filter(g1,b,modsignal(:,1)); s2=filter(g2,b,modsignal(:,2)); s3=filter(g3,b,modsignal(:,3)); s4=filter(g4,b,modsignal(:,4)); ln=200; % length of signal figure(1) clf reset



subplot 221,plot(1:ln,s1); title('Source 1') subplot 222,plot(1:ln,s2); title('Source 2') subplot 223,plot(1:ln,s3); title('Source 3') subplot 224,plot(1:ln,s4); title('Source 4') s=[s1,s2,s3,s4]; % ---------------------------------------------------- % Create the mixed signals % ---------------------------------------------------- xhat=A*s'; % Now to estimate the covariance matrix, % Consider the 200 ensembles by using 4 eigen values % and the length of the data vector to be 4 sum=0; for k=1:4:197, sum=sum+(xhat(1,k:k+3)'*xhat(1,k:k+3)); sum=sum+(xhat(2,k:k+3)'*xhat(2,k:k+3)); sum=sum+(xhat(3,k:k+3)'*xhat(3,k:k+3)); sum=sum+(xhat(4,k:k+3)'*xhat(4,k:k+3)); end R=sum/200; % Prewhitening of the observations x=R^(-0.5)*xhat; [v,d]=eig(R); % Displaying the observations disp('Now displaying the observations...') figure(2) subplot 221,plot(1:ln,x(1,:),'r'); title('Observation x1') subplot 222,plot(1:ln,x(2,:),'r'); title('Observation x2') subplot 223,plot(1:ln,x(3,:),'r'); title('Observation x3') subplot 224,plot(1:ln,x(4,:),'r'); title('Observation x4') disp('Press any key to continue') disp(' ') pause W=0.01*rand(neig,ulen); %initial random weights c=tril(0.01*rand(neig,neig),-1); %initial inhibitory weights not_converged=1; dataser=[x(1,:),x(2,:),x(3,:),x(4,:)]; % serialise the data index=0; % index into the data



pwr=[]; % Output power sweep=1; % Initialise the sweep variable ( 1 sweep = 1 pass through the data ) data=x; t0=clock; % ------------------------------------------------------------- % APEX Implementation starts here within this loop % ------------------------------------------------------------- disp('Starting the main APEX algorithm...') while sweep < 61, %for iter=1:ntp*ni, index=index+1; % Check if one sweep has occured if index>797, index=1; sweep=sweep+1; sum=0; for it=1:4:797, t1=W*dataser(1,it:it+3)'; t2=c*t1; sum=sum+t1-t2; end pwr=[pwr,sum./200]; end %noise=(rand(1,ulen)-0.5); %simple additive noise sig=dataser(1,index:index+3); %x=sig'+noise'; %add in noise x=sig'; ye=W*x; %compute excitatory values yi=c*ye; %compute inhibitory values y=ye-yi; %compute net values for i=1:neig %adjust the weights yp=y(i); dw1=x'; %first order hebbian adjustment dw2=y(i)*W(i,:); %second order hebbian adjustment W(i,:)=W(i,:)+lr*yp*(dw1-dw2); %adjust weights end for i=2:neig %adjust the inhibitory connections yp=y(i); da1=y(1:i-1)'; %first order adjustment da2=yp*c(i,1:i-1); %second order adjustment c(i,1:i-1)=c(i,1:i-1)+lr*yp*(da1-da2); end end pwr2=pwr.^2; % Actual output power is square of y



% save everything in a mat file save apexdata % display the time elapsed for computation tme=etime(clock,t0); disp (['The time required is ' int2str(floor(tme/60)) ' minutes and ' int2str(mod(tme,60))... ' seconds']) disp(' ') disp('Press any key to display the results ...') pause close all %Now calculate the permutation matrix %call program autoapex.m autoapex('apexdata'); %--------------------------------------------------------- % end of main program % -------------------------------------------------------- function [] = autoapex(filename) % autoapex.m % To calculate the permutation % applied to the weight matrix manually % By trying out all possible orders of the eigen vectors % and the sign reversal also % Created 12/15/01 % Copyright Alok Ahuja % load the data from the saved mat file disp(' ') disp('Now calculating the estimated sources... ') load (filename) Wfinal=zeros(4,4); sest_final=zeros(200,4); W1=W; % calculate the permutations of the specific eigen vectors for src=1:4, disp(' '); disp(['Source No. ' int2str(src) ]) error=[]; % the sumsqr error scale_err=[]; % the scaling error V1=perms(W1(src,:)); % main loop for the 16 possible permutations of the vector % It also incorporates the sign reversal



for count=1:16, W1=W; W1(src,:)=V1(count,:); for snchange=-1:2:1, W1(src,:)=-W1(src,:); sest=(W1*data)'; err=sumsqr(s(:,src)-sest(:,src)); error=[error;err]; %disp(['count = ' int2str(count) ' , Error = ' int2str(err)... % ' , Sign = ',int2str(snchange)]); %figure(1) %plot(1:200,s(:,src),'r',1:200,sest(:,src),'b'); %title(['source' int2str(src)]); %pause; end end % Now calculate the appropriate signal [finerr,index]=min(error); perm_number=ceil(index/2); sign=mod(index,2); W1=W; W1(src,:)=V1(perm_number,:); if sign==1, W1(src,:)=-W1(src,:); end sest=(W1*data)'; % Now try to scale the source % and see if the error reduces for scale = 0.2:0.1:6, err=sumsqr(s(:,src)-sest(:,src).*scale); scale_err=[scale_err;err]; %figure(1) %plot(1:200,s(:,src),'r',1:200,sest(:,src).*scale,'b'); %title(['source' int2str(src)]); %disp(['scale = ' num2str(scale) ' , Error = ' int2str(err)]); end [finerr1,index1]=min(scale_err); scale_range=0.2:0.1:6; sest(:,src)=sest(:,src).*scale_range(index1); figure(src) clf reset plot(1:200,s(:,src),'r',1:200,sest(:,src),'b'); title(['source' int2str(src)]); legend('original source','estimated source') xlabel('Samples'),ylabel('Amplutude') Wfinal(src,:)=W1(src,:); sest_final(:,src)=sest(:,src);



disp (['The Sum Square Error is ' num2str(finerr1) ]); disp ('Press any key to continue') pause end disp(' ') disp('The final demixing matrix is ...') disp(' ') disp(num2str(Wfinal)) % --------------------------------------------------------------------- % End of program % ---------------------------------------------------------------------

Blind Signal Separation using Principal Component Analysis...

Documents

Transcript of Blind Signal Separation using Principal Component Analysis...