CS229 MACHINE LEARNING, STANFORD UNIVERSITY, DECEMBER 2016 1

Models of Neuron Coding in Retinal GanglionCells and Clustering by Receptive Field

Kevin Fegelis, SUID: 005996192, Claire Hebert, SUID: 006122438, and Brett Larsen, SUID: 006048301

Abstract—One of the fundamental questions in neurosciencetoday is understanding the functional relationship between stim-uli and the firing of neurons, often referred to as the neuroncoding problem. [1] In this report, we consider both biologically-inspired Maximum-Likelihood estimators and feedforward neu-ral networks in order to learn models of the retina and itsresponse to visual stimuli. Data is taken from recordings ofretinal ganglion cells taken by Steve Baccus’s lab at Stanford[2]. A comparison of these models is presented based on thePearson Correlation Coefficient between the true test responseof a neuron, and the models’ predicted outputs. Using thismetric, we show that the deep Artificial Neural Network usedhas superior generalization to unseen data, as compared with theprobabilistic models. Using parameters fit to a Linear-Nonlinear-Poisson model, we also present a hierarchical clustering oflearned space-time receptive fields of all 28 recorded retinalneurons resulting in six primary clusters.

I. INTRODUCTION AND RELATED WORK

THE retina in general consists of three layers: photore-ceptors which transduce photons into biological signals,

an intermediate network of neurons consisting of what arecalled bipolar cells and retina amacrine cells, and finally retinalganglion cells which transfer the visual information to thebrain as shown in Figure 1. [3] The aim of this projectis to learn a model p(y|x) of how spike trains y from theganglion cells will respond to various stimuli x shown to thephotoreceptors and then use this model to predict future spiketrains for new stimuli. We are unconcerned about modelingthe explicit structure of the network of neurons between thephotoreceptors and the ganglion cells; rather, we seek to modelthe phenomenological neural response.

For this project, there are two different types of relatedwork. The first is biological research about the behavior ofneurons in the retina which we wish to incorporate into ourmodel. Previous research has shown that each ganglion cellhas a receptive field, or region in space that it responds tostimuli over time. [3] We will treat the receptive field ofthe neuron as a linear filter that acts on the stimuli beforepassing it to our model of the cell. Furthermore, becauseseveral of the models used in this project learn the neuron’sspace-time receptive field, we will also seek to cluster neuronsby receptive field type. Secondly, neurons are also knownto exhibit refractoriness, a property meaning that the cell isless likely to respond to a stimulus immediately after firing.Thus, we will also integrate this dependence on the spike trainhistory into our model. [1]

The other type of related work consists of statistical modelsused commonly in the field of computational neuroscience todescribe neuron behavior. The linear-nonlinear Poisson model

Fig. 1: Diagram of the structure of retinal neurons, includingphotoreceptors (P), bipolar cells (B), horizontal cells (H),retinal amacrine cells (A), and retinal ganglion cells (G)(Redrawn from [3]).

and the generalized linear model described in Section IIIare two biologically-inspired models of neurons as Poissonprocesses that incorporate the receptive field and refractorinessproperties of the neuron into the intensity λ. [1] [4] [5]Artificial neural networks, which themselves are inspired bythe brain but do not seek to provide biological plausibility,have also proven powerful tools for learning relationshipsbetween stimuli and output. In particular, researchers in SuryaGanguli’s lab at Stanford have used convolutional neuralnetworks (CNNs) to model responses in the retina with greatsuccess. [6]

II. DATASET AND FEATURES

The data used for this project was acquired in the Baccuslab by surgically removing a salamander retina and stimulatingit with two different types of signals: high-contrast Gaussiannoise (white noise) and natural scenes. [2] Samples of thesestimuli are shown in Figure 2. While the photoreceptorsreceive these signals, the output of the retinal ganglion cellsare simultaneously recorded. When working with recordedspike trains, our first step is to bin the spikes into some smalltime interval which we call τ , in this case 10 ms. Predictingspike trains then reduces to predicting the number of spikesthat occur in each bin. For the white noise 30,000 bins wererecorded, and for the natural scenes 3,000 bins were recorded.Because we assume the system is causal, our features willbe the tensor of movie images (2 space dimensions, 1 timedimension) in a time window before the point we want to

CS229 MACHINE LEARNING, STANFORD UNIVERSITY, DECEMBER 2016 2

(a) Gaussian Noise (b) Natural Scene

Fig. 2: Sample stimuli [2].

predict say 350 ms. In addition, we also sometimes use thespike trains in this window proceeding the bin as a feature.

III. METHODS FOR MODELLING NEURONAL RESPONSE

We will split the probabilistic relationship between thestimuli and the retinal responses p(y|x) into two parts: aprobabilistic model of neuron spiking which is typicallydescribed using a Poisson process (an exponential familydistribution) and a functional relationship between all thefactors that we think affect neuron spiking (stimuli, spiketrain history, properties such refractoriness) and the intensityλ in the Poisson process. Together these lead to two commonmodels for p(y|x) in neurons: the Linear-Nonlinear Poisson(LNP) and the generalized linear model (GLM). [4]

A. Poisson Process Model of Neurons

As mentioned in the previous section, our first step is tobin the spikes into 10 ms segments. Due to the small sizeof the interval, the number of spikes per bin is small andthus naturally leads to modelling using a Poison distributiondescribed by the following equation [4]:

p(y|λ) = (λτ)y

y!exp(−λτ) (1)

where λ is the intensity and thus λτ is the expected numberof spikes in a given bin. If we bin our spike train at resolutionτ into a vector of spike counts y = {yt} indexed by timeand assume each bin is independent, then the likelihood thata time-varying Poisson process generated a given spike trainis given by:

p(y|θ) =∏t

(λtτ)y

y!exp(−λtτ) (2)

where we have allowed the intensity λt to vary over time andused θ = {λt} to refer to the vector of intensities. [4] Takingthe log-likelihood and grouping terms independent of θ into aconstant c yields the following:

`(θ) =∑t

yt log λt − τ∑t

λt + c (3)

(a) Linear-Nonlinear Poisson (LNP)

(b) Generalized Linear Model (GLM)

Fig. 3: Diagram of the two models for neural spiking (Redrawnfrom [1]).

B. Linear-Nonlinear Poisson Model

Next, we need a model of how the intensities in our Poissonmodel relate to the stimuli observed by the photoreceptors.We assume the system is causal, meaning that the spike trainis only affected by proceeding stimuli, and thus consider awindow of a certain length beforehand, say 350 ms. We willcall η the number of bins in this window which equals 350ms/τ = 350 ms/10 ms = 35. Note that this means with ourmodel we are not able to describe the spike train for the firstη bins because we do not have the full window of stimuli. Wedenote xt = (xt−η, . . . , xt−1) to be the vector of stimuli inthe η bins proceeding time t. [4]

In the LNP model we assume that the receptive field of theneuron acts linearly on the stimuli vector before being passedthrough a static nonlinearity f to give us the spiking rate:

λt = f(k · xt) (4)

For now we will use the exponential function as our nonlinear-ity. Plugging this into our distribution for the Poisson process,we obtain the log-likelihood for our parameter vector θ:

`(θ) =∑t

ytk · xt − τ∑t

exp(k · xt) + c (5)

This expression is concave in the receptive field k and thuscan be maximized using gradient ascent. The overall model isdepicted in Figure 3a. [4]

C. Generalized Linear Model

The generalized linear model (GLM) we construct is iden-tical to the LNP model described in the previous sectionexcept that we incorporate the idea that spiking rates arealso affected by the spiking history of the neuron due toproperties like refactoriness. [4] Similar to xt we defineyt = {yt−η, . . . , yt−1} to be the vector of spike counts in the350 ms window proceeding t. We pass these through a learnedfilter h (called the post-spike filter) in the same manner as thereceptive field yielding the following relationship for the time-varying intensity λt:

CS229 MACHINE LEARNING, STANFORD UNIVERSITY, DECEMBER 2016 3

λt = f(k · xt + h · yt) (6)

Again, we will use the exponential as our nonlinearity andthus obtain the following log-likelihood:

`(θ) =∑t

yt(k ·xt+h ·yt)−τ∑t

exp(k ·xt+h ·yt))+c (7)

This expression is concave in the receptive field k and post-spike filter h and thus can be maximized using gradient ascent.The overall model is depicted in Figure 3b. [4]

D. Artificial Neural Network

Another approach to predicting spike trains commonlyused in computational neuroscience is to learn relationshipsbetween spike trains and stimuli using deep learning. Inparticular, there has been recent success in using ConvolutionalNeural Networks (CNNs) to predict responses through thePrimary Visual Cortex [6]. As their name implies, CNNs em-ploy a convolution operation through tiling overlapping filtersacross an image. This allows the network to to detect low-levelfeatures in the image, which are stored as weights in the hiddenlayers. Almost all standard deep-learning frameworks supporttwo-dimensional convolutions; however, the stimuli vectorsin our problem are three-dimensional movies (again, 2 spacedimensions, 1 time dimension). Three-dimensional CNNs havein general been less prevalent in the literature than their two-dimensional counterparts and are very resource-intensive dueto the three-dimensional convolution operator.

As such, we develop a deep feedforward Artificial NeuralNetwork (ANN) as an alternative means of prediction. TheTensorFlow deep-learning library is chosen as a developmentenvironment for our model. [7] To ensure that we are ableto find an analogous set of low-level features as a shallowCNN would find, we extend the depth of the model to havesix hidden layers, an architecture design shown in Figure 4.

All hidden units of the model are Rectified Linear units withactivations of the form hi = max(0,W ihi−1 + bi), whereW i, bi are weights and biases corresponding to the hiddenlayer of index i. Dropout is applied between all but the last twohidden layers, with a training drop percentage of 50%. Thispromotes a biologically plausible sparsity in the model, whereeach unit is encouraged to produce useful features independentof other units. Minibatches of 25 examples are fed into thenetwork between every iteration of an ADAM optimizer. [8]

IV. REGRESSION RESULTS

In this section, we present our implementation in Pythonof the LNP, GLM and ANN for predicting firing rates ofa neuron exposed both to high-contrast Gaussian noise andnatural scenes. Each retinal neuron’s response is averagedover 28 time trials for the same stimulus to counteract thenatural variability of the retina. To perform regression, werandomly sample both training and testing subsets from thisaverage neural response: 70% of the trial was used for trainingand 30% was used for testing. Mini-batch stochastic gradientdescent with a batch size of 5 was used for training the

Fig. 4: Deep network architecture designed in TensorFlow.Weights initialized to 10−3; optimized using ADAM with alearning rate of 10−4

Fig. 5: Pearson correlation coefficient across all trials. Notethat the LNP and GLM results were calculated across all 28neurons while the ANN results are only from 5 neurons dueto long training time and project time constraints.

LNP and GLM (this was the number of stimuli tensors thatconveniently fit into the RAM on a laptop); ADAM was usedas the optimization algorithm for the ANN. This process wasrepeated for the average retinal neuron response of both typesof stimuli for all 28 neurons recorded (with the exception ofthe ANN results which are only from 5 neurons due to longtraining time and project time constraints).

Figure 6 show sample predicted values from each of ourthree models on a sample neuron (the testing data is selectedrandomly, however, so the prediction is made on differenttime segments for each). One can see visually that the testpredictions are fairly accurate and do not seem to be overfittingthe data. Generally, the models are able to predict the locationof the spikes but often somewhat over or underestimate theactual firing rate. In addition, the ANN seems to fit a baselinenoise not present in the original signal.

To give a more quantitative idea of the overall performanceof the model, we have compiled the average Pearson corre-lation coefficient across all trials and neurons in Figure 5.The Pearson correlation coefficient is a typical measure inneuroscience of similarity between time series and is defined

CS229 MACHINE LEARNING, STANFORD UNIVERSITY, DECEMBER 2016 4

Fig. 6: Sample predictions for the LNP, GLM, and ANN.Predictions are for the same neuron but the sections of thedata used for testing and training were randomized; as a result,predictions for each model are made on different segments ofthe spike train.

as follows for two time-series X and Y :

RX,Y =Cov[X,Y ]√Var[X]Var[Y ]

(8)

The first column is for models trained and tested on the whitenoise and the second column is for models tested and trainedon natural scenes.

TABLE I: White noise

GLM LNP ANNPearson Correlation 0.512 0.726 0.866Standard Deviation 0.130 0.0530 0.0255

TABLE II: Natural scenes

GLM LNP ANNPearson Correlation 0.686 0.660 0.897Standard Deviation 0.0897 0.148 0.0251

Fig. 7: Clustering of the neurons by learned space-time recep-tive field using hierarchical clustering. Samples of the learnedreceptive field over time show the different types of neuronalbehavior observed in each cluster. .

While the Pearson correlation coefficient is the standardmetric for comparing time series in neuroscience, and interest-ing extension of the project would be to consider alternativemetrics for measuring the success of the predicted spike trains.Furthermore, the regression could be potentially improved bybinning the predicted spiking rate as well rather than predictinga continuous value.

V. CLUSTERING RESULTS

After training the LNP model for each of the 28 neuronsto convergence, we extract the fitted k receptive field fil-ters. Given large amounts of retinal data, a useful tool forneuroscientists would be to quickly determine what type ofreceptive field a neuron has, without needing to physicallyexamine each. Clustering methods provide exactly this capa-bility, but are typically limited by a need to pre-determinethe amount of clusters. For stationary (not time-dependent)receptive fields, there are typically four patterns of sensitivity:

CS229 MACHINE LEARNING, STANFORD UNIVERSITY, DECEMBER 2016 5

Highly responsive cells sensitive to bright patches on a darkbackground or black patches on bright backgrounds, and cellswhich are weakly or non-sensitive with receptive fields ofuniform brightness or darkness, respectively. [9]

However, the receptive fields are known to change theirorientation and are generally time-dependent. [9] To capturethis, we use a hierarchical clustering algorithm whose resultingdendrogram can be seen in Figure . Before passing the recep-tive field to the clustering algorithm, the fields were smoothedwith a one-pixel wide Gaussian filter. These algorithms makeno assumptions on the number of clusters, and instead separatethe data in a decision tree like manner. Examples of neuronsbelonging to each cluster are plotted below the dendrogram.

From the figure, we can see that the algorithm is pickingup on six main groupings of the receptive field. Cluster 1is atypical and comprised of only a single neuron whosereceptive field switches in time from white-on-black to black-on-white, and back to white-on-black. All other neurons makeat most one ”polarity” switch during the time delay studied(and are all inside ”super-cluster” number 2). Cluster 3 hasneurons which are insensitive to the stimulus and will onlyfire in a stochastic and uncorrelated manner. Clusters 4 and 5are neurons whose sensitivites are strong in the few momentsproceeding a spiking event, and are only different in the factthat neurons of cluster 5 seem to develop structure sooner.Clusters 6 and 7 are those where the strongest sensitivities arewell proceeding a spiking event, where cluster 6 neurons haveon average weaker receptive fields.

VI. CONCLUSIONS

In order to model the phenomenological response of neuronsin the retina to various stimuli, we fit three models to datataken by the Baccus lab recording the response of salamanderretina ganglion cells to two types of stimuli: high-contrastGaussian noise and natural scenes. The data was first pre-processed by binning the spikes in 10 ms intervals. For eachneuron, the stimuli were averaged across 28 trials (to reducethe effects of the natural variability of the retina) and 70% ofthe data was randomly selected for training and the remaining30% was used for testing. Because we assume the system iscausal, the 350 ms of stimuli proceeding a bin was used asthe main feature to predict the spiking rate in that bin. In theGLM, the spike train during this interval was also used as afeature.

All three models were able to generalize fairly well tounseen, i.e. were able to predict spike trains in response to thesame stimuli they are trained on. Overall, the ANN performedthis task the best, although it also seemed to be fitting a smallbaseline noise when the spiking rate should be 0; this is mostlikely the ANN adjusting its bias to be close to the meanfiring rate. The GLM on the other hand performed worse thanthe LNP model when trained and tested on natural scenes.McIntosh et al. obtained a similar result in [6]; likely theadditional features in the GLM are causing some form ofoverfitting.

Finally, the receptive fields learned for each neuron in fittingthe LNP model were used to cluster the neurons. The space-time receptive field in our model is a tensor that shows over the

previous 350 ms before the spike what part of the stimuli theneuron was receptive to. Receptive fields are known to changetheir orientation during this time before the spike, and in fourof the observed clusters the ”polarity” of the receptive fieldswitched once during the 350 ms interval, e.g. from black-on-white to white-on-black. One additional cluster was foundwhere the neuron switched its polarity multiple times duringthe interval while our neurons were insensitive to the stimulusall together. This type of clustering could potentially helpscientists further understand retinal ganglion cells and evenidentify new subtypes of cells.

VII. FUTURE WORK

The immediate next step for the project would be tofurther examine the idea of using neural networks to predictspike trains. Because of their natural application to imageprocessing, we plan to examine the performance of ConvNetson predicting spike trains based on exciting results from workin Surya Gangulis lab at Stanford [6]. As part of the project,we coded a basic CNN using TensorFlow, but it was toocomputationally intensive to run in the time remaining in theproject and often such networks require significant fine-tuning.In addition, it would be interesting to see how the learnedfilters generalize from one type of stimuli to another, i.e.how the learned receptive filters from the white noise wouldperform on predicting spike trains from natural scenes.

ACKNOWLEDGMENT

The authors would like to thank Steve Baccus’s lab forthe use of the retinal ganlion cell data as well as NiruMaheswaranathan and Lane McIntosh in Surya Gangulis labfor many useful discussions and advice.

REFERENCES

[1] K. Doya, S. Ishii, A. Pouget, and R. P. N. Rao, Bayesian brain:Probabilistic approaches to neural coding. Cambridge, MA: MIT Press,2007.

[2] S. Baccus, “Baccus lab: Department of neurobiology,”https://sites.stanford.edu/baccuslab/.

[3] S. A. Baccus, “Timing and computation in inner retinal circuitry,” Annu.Rev. Physiol., vol. 69, pp. 271–290, 2007.

[4] J. Shlens, “Notes on generalized linear models of neurons,” arXiv preprintarXiv:1404.1999, 2014.

[5] T. Gollisch and M. Meister, “Eye smarter than scientists believed: neuralcomputations in circuits of the retina,” Neuron, vol. 65, no. 2, pp. 150–164, 2010.

[6] L. McIntosh, N. Maheswaranathan, A. Nayebi, S. Ganguli, and S. Baccus,“Deep learning models of the retinal response to natural scenes,” NIPS,2016.

[7] M. A. et al., “TensorFlow: Large-scale machine learning on heterogeneoussystems,” 2015, software available from tensorflow.org. [Online].Available: http://tensorflow.org/

[8] D. P. Kingma and J. Ba, “Adam: A method for stochasticoptimization,” CoRR, vol. abs/1412.6980, 2014. [Online]. Available:http://arxiv.org/abs/1412.6980

[9] P. Dayan and L. F. Abbott, Theoretical Neuroscience: Computational andMathematical Modeling of Neural Systems. The MIT Press, 2005.