8/2/2019 Analog Neural Computer

1/13

University of Pennsylvania

ScholarlyCommons

Departmental Papers (ESE) Department of Electrical & Systems Engineering

1-1-1992

An Analog Neural Computer with ModularArchitecture for Real-Time Dynamic

ComputationsJan Van der SpiegelUniversity of Pennsylvania, jan@seas.upenn.edu

Paul MuellerUniversity of Pennsylvania

David BlackmanUniversity of Pennsylvania

Peter Chance

Corticon, Inc.

Christopher DonhamUniversity of Pennsylvania

See next page for additional authors

Copyright 1992 IEEE. Reprinted fromIEEE Journal of Solid State Circuits, Volume 27, Issue 1, January 1992, pages 82-92.

Publisher URL: http://dx.doi.org/10.1109/4.109559

This material is posted here with permission of the IEEE. Such permission of the IEEE does not in any way imply IEEE endorsement of any of the

University of Pennsylvania's products or services. Internal or personal use of this material is permitted. However, permission to reprint/republish this

material for advertising or promotional purposes or for creating new collective works for resale or redistribution must be obtained from the IEEE by

writing to pubs-permissions@ieee.org. By choosing to view this document, you agree to all provisions of the copyright laws protecting it.

This paper is posted at ScholarlyCommons.http://repository.upenn.edu/ese_papers/252

For more information, please contact repository@pobox.upenn.edu.

http://repository.upenn.edu/http://repository.upenn.edu/ese_papershttp://repository.upenn.edu/esehttp://repository.upenn.edu/ese_papers/252mailto:repository@pobox.upenn.edumailto:repository@pobox.upenn.eduhttp://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/esehttp://repository.upenn.edu/ese_papershttp://repository.upenn.edu/

8/2/2019 Analog Neural Computer

2/13

Author(s)

Jan Van der Spiegel, Paul Mueller, David Blackman, Peter Chance, Christopher Donham, Ralph Etienne-Cummings, and Peter Kinget

This journal article is available at ScholarlyCommons:http://repository.upenn.edu/ese_papers/252

http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252

8/2/2019 Analog Neural Computer

3/13

82 I E E E J O U R N A L OF SOLID-STATE CIRCUITS. VO L 27 . NO I . J A N U A R Y 1g92

An Analog Neural Computer with ModularArchitecture for Real-Time DynamiccomputationsJan Van der Spiegel , Sen ior Member, IEEE, Paul Muel ler , David Blackman, M em b er , IEEE, Pe te r Chance ,Chr i s tophe r Donham , M em b er , IE E E , Ralph Et ienne-Cummings , and Peter Kinget , Student Member, IEEE

Abstract-The paper describes a multichip analog parallelneural network whose architecture, neuron characteristics,synaptic connections, and time constants are modifiable. Thesystem has several important features, such as time constantsfor time-domain computations, interchangeable chips allowinga modifiable gross architecture, and expandability to any ar-bitrary size. Such an approach allows the exploration of dif-ferent network architectures for a wide range of applications,in particular dynamic real-world computations. Four differentmodules (neuron, synapse, time constant, and switch units)have been designed and fabricated in a 2-pm CMOS technol-ogy. About 100 of these modules have been assembled in a fullyfunctional prototype neural computer. An integrated softwarepackage for setting the network configuration and character-istics, and monitoring the neuron outputs has been developedas well. The performance of the individual modules as well asthe overall system response for several applications have beentested successfully. Results of a network for real-time decom-position of acoustical patterns will be discussed.

I. INTRODUCTIONR CENT years have seen a renewed interest in neuralnetworks for several reasons: a better understandingof the functions of the brain [ I ] , improved mathematicalmodels [2], development of new algorithms and net to-pologies, and the understanding of emergent collectiveproperties of neural networks [3]-[5]. While much of thework has been theoretical, technological advances are be-ginning to make hardware implementations of such sys-tems possible with the goal to build machines cap able ofsolving real-world problems. These tasks require com-putational power that is beyond the capabilities of currentvon Neumann-based digital computers. Biological sys-

Manuscript received May 27, 1991; revised August 27 , 199 1. This workwas supported by the Office of Naval Research under Grant N0014-89-J-12499, the National Science Foundation under Grant EE T 16685, and Cor-ticon, Inc. C. Donham and P. Kinget were supported by the National Sci-ence Foundation through Research Experience Undergraduates GrantsENG88-05282 and ENG89-00442.J. Van der Spiegel, P. Mueller, D. Blackman, C. Donham, and R.Etienne-Cummings are w ith the Departments of Electrical Engineering andBiophysics, University of Pennsylvania, Philadelphia, PA 19104-6390.P. Chance is with Corticon, Inc., Philadelphia, PA 19104.P. Kinget is with the Laboratory Electronika, Systemen, Automatisatie,Technologic (ESAT ), Departement Elektrotechniek, Katholieke Universi-teit Leuven, B-3001 Heverlee, Belgium.IEEE Log Number 9104598

tems solve these problems through parallel computationsin a highly interconnected structure consisting of manynonlinear processing elements.Several approaches have been taken to the implemen-tation of neural networks. Such efforts date back to theearly 1960's when electrical models of simple neurons andalso more com plex neural system s were first built [6], [7].Only recently have more powerful and compact systemsevolved. Tech nologies that are being used today includedigital systems [8]-[lo], either in serial or parallel form.analog electronic networks [ I l l - [2 11, and optical systems[22]-[23]. Digital approaches range anywhere from sim-ulation on von Neumann machines to special-purpose dig-ital neural networks. The advantages of digital ap-proaches are programming flexibility, computationalaccuracy, noise immunity, as well as the fact that digitaltechnology is well established. However, digital methodshave some distinct drawbacks which are inherent in theessentially sequential nature of the computations, andwhich remain even in a highly parallel architecture. Forexample, though high-speed parallel computers can cal-culate the state of each neuron in parallel, neurons stillevolve sequentially from state to state. The digital com-puter calculations must converge fo r each time slice. Sincethe neural network elements are nonlinear elements of in-put and time, convergence can be a problem for networksconfigured with feedback and lateral inhibition. Anotherserious bottleneck in the performance of digital computernetworks is the bandwidth between processors and mem-ory. In order to calculate the input for a given neuron, aprocessor must have access to all other neuron outputs.Hence, each processor must be able to access a largeshared memory where the previous neuron states arestored. An extremely large bandwidth is required so thatthe memory can communicate with the processors.In contrast, analog systems can sum and scale arbitrarynumbers of inputs truly simultaneously. The limited ac-curacy of analog components is not a serious problem be-cause neural networks are forgiving to component errorsfor the following reasons: 1) the output of the network isthe aggregate response of many neurons acting collec-tively, and thus the result is less dependent on minor com-ponent variations; and 2) learning can be used to tune the

001 8-9200192$03.00 1992 IEEE

http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252

8/2/2019 Analog Neural Computer

4/13

V A N DE R S P l E GE L ci u l . : ANAL OG NE UR AL C O M P U T E R W I T H M O D U L A R ARCHITECTURE 83

network response in spite of specific component error.Another advantage of analog systems is that time can beeasily represented as a continuous variable through a timeto potential or current transformation, thereby avoidingthe clocks and counters required in digital systems to keeptrack of time. As a result, analog neural networks can pro-vide not only a mechanism for summation and scaling ofincoming signa ls, but for temporal integration as well, ina fashion similar to biological networks. Biology has re-alized the advantages of analog methods and has achievedthe impressive computational rates that approach theequivalent of 1 0 ' ~ - 1 0 ~ ~loating-point operations persecond' in spite of its slow hardw are.This paper describes a large-scale neural network thatis loosely modeled after the biological system. It is fullyparallel and operates in analog mode. The architecture,the neuron characteristics, synapses, and synaptic timeconstants are modifiable. The system has several impor-tant attributes: 1) in addition to synaptic weights that areadjustable over a large dynamic range, the system con-tains modifiable synaptic time constants which are re-quired for time-domain com putations; a s a result the net-work does spurio-temporal processing, which is aconsiderably more complex task than parallel processingdone at discrete time steps; 2) the system is constructedfrom interchangeable nlodules with two-dimensionalsymmetrical pinout allowing a m odifiable gross architec-ture in which the numerical ratios and arrangement ofneurons, synapse, and connection modules can be easilyselected; 3) except for global control lines, the moduleshave only connections to their nearest neighbors, result-ing in a simplified packaging and board design; and 4) thenetwork is expandable to arbitrary size and can thereforebe adapted to the complexity of the task.The network can be used in both learning and compu-tation mode. However, learning algorithms are imple-mented through a host comp uter by sampling the networkstate (as described in Section IV) , and reprogramming thenetwork as required. Thus, the full potential of the ma-chine is obtained, after learning, in situations involvingneural computation of dynamic systems, such as acousticpattern recognition, sonar signal classification, motiondetection, etc. A prototype system consisting of 99 cus-tom chips has been designed, assem bled, and tested. Theoverall system is fully operational and has been success-fully used for several dynamic computa tions. In addition,supporting software for programming the network andmonitoring the state of the net has been developed. Re-

Il'licrc are about 10" neurons in the hurnan brain which are richly in-tcrconncctcd with fcedback and lateral inhibitions. Each neuron has on theaverage 1000 synapses resulting in a total of lo J 4 onnections. Each co n-ncction has a tinic constant of approximately I Ins which corresponds tothc summing of charge on the postsynaptic membrane capacitance. To sim-ul;~te his set of .simultaneous nonlinear differential equations describingsuch a system, a time step as small as 0.1 ms would have to be used. Thus,10IH ostsynaptic potentials would be updated per second. If each potentialupdate took 1 to 100 floating-point operations, the etfective processing ratew ou ld be I O ~ ~ - I O ~ "LOPS, assuming all neurons are active.

Fig. 1. Block diagram of the modular gross architecture of the neural net-work. The system consists of programmable neuron modules (N ), synapsemodules (S), ime constants (T) , and analog crosspoint arrays (blankboxes). The modules are directly interconnected through parallel analoglines. The state of each neuron is sampled and multiplexed on a single line(OM) that is connected to a digital host.

sults and performance of the individual modules and theoverall system are given below.

As shown in Fig. 1, the machine architecture is a mul-tichip system containing neurons, synapses, modifiabletime constants, and analog crosspoint switches. The ap-proach is to a certain extent inspired by biological sys-tems, in particular the cerebral cortex, in which there areseparate neurons, axons, and synapses with a limitednumber of inputs for each neuron. In contrast to the bio-logical system, the system described here is fully modi-fiable including the neuron, synapse, and synaptic timeconstants, as well as the network topology. Such a designwill allow the exploration of different network architec-tures for a wide range of applications, in particular dy-namic real-world problems.The modules of the neuron arrays are placed on a reg-ular grid and connected directly to neighboring synapticand analog switch arrays. During operation the analogsignal flow is as follow s: the outp ut signals exit from thenorth and south of the neuron chip; then the signals arerouted via the analog switches until they are steered northor south into synapse modules, through a time constantmodule if required; the synapses provide the input to theneighboring neurons along the east or west direction. Theswitch modules allow the neuron outputs to be routed toany other neuron. Hence, fully interconnected multilay-ered networks can be easily implemented. However, thenumber of neurons that can be fully interconnected be-tween layers is limited to the number of synaptic inputsper neuron. A lso, a wide range of sparsely interconnectednetworks can be mapped into the system. However, in

http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252

8/2/2019 Analog Neural Computer

5/13

84 I E E E J O U R N A L OF SOLID-STATE CIRCUITS, V O L . 27 . NO . I . J A N U A R Y 1992

general, the machine architecture presents more oppor-tunities for short-range connections. Thus, the architec-ture favors local computations and a hierarchical organi-zation of data flow, which also seems to be true for thebiological system.The ratio between the number of neurons, routing chan-nels, and synapses on each module is chosen to optimizethe usage of available components for typical applica-tions. Earlier experience with a discrete system based ona similar architecture for the neural computation ofacoustical patterns [24] has served as a guide in the choiceof these module sizes. However, the modules are de-signed so that they can be repeated w here required (e.g . ,placing two synapse modules on each side of the neuronsto double the fan-in, or using additional rows of switchmodules to increase the routing capabilities).The system operates fully analog and in parallel. How -ever, the switch positions, neuron characteristics, synap-tic weights, and time constants are set by a digital hostand are stored in a local memory on the individual mod-ules. In additio n, all or a selected number of neuron out-puts can be sampled continuously or during a specifiedtime segment and multiplexed on a single line (called theoutput monitor line or OM) which is connected to anAID converter and read into the digital host (Fig . 1 ). Thisoperation, which is independent of the analog neural com-putations, allows the host com puter to display the outputsas a function of time, or to use them for implementinglearning algorithms. The interaction between the neuralnetwork and the host compu ter is described in Section I V .111. DESCRIPTIONN D P E R F O R M A N C EF T H E M O D U L E S

Four different modules have been designed and fabri-cated in a 2-pm n-well CMOS technology through theMOSIS servicee2The function, major design issues, andresults of each chip will be described. These chips areused in a prototype system of 72 neurons whose overallperformance is discussed in Section IV. For this proto-type, synapse arrays of 16 x 8 synapses, switch arrays of16 x 16 switches, and neuron chips of eight neurons werefabricated. The number of synapses and switches is cho-sen so that the overall size of each die is not excessive,yet the chips are complex enough to allow evaluation ofthe performance of the whole system. The choice of eightneurons per module gives an optimum ratio between neu-rons and routing ch annel s, as mentioned in Section 11. Byusing larger dies and more advanced packaging methods,the number of components per modu le can be scaled up .

ClockResetOR1

In1In2In3In4

Fig. 2. Block diagram of a neuron module. Each unit consists of modifi-able neurons, an analog multiplexer, and control logic for addressing thechip .tem we choose eight neurons per module. Fig. 2 shows aschematic block diagram of the neuron chip.

The neuron is implemented as a piecewise-linear devicehaving a transfer function with a variable magnitude ofthe step at threshold, and variable threshold current. Theinput signal is a current and the output is a voltage. Thisrepresentation is chosen because currents can be easilysummed and scaled while voltage outputs have a largefan-out. Hence, the input op amp is used in a trans-impedance configuration, with a rectifier, and the outputop amp is a unity-gain inverting amplifier, as shown inFig. 3 . The transimpedance gain has to be matched to thesynapse characteristic that provides the input current. Thegain is 0.1 V /p A using nominal resistance values of 100kQ . The diodes are implemented as diode-connected MOStransistors. The comparator, which triggers the step whenthe input current exceeds the threshold value, is a PMOS-input differential pair. The step is provided by a currentmirror, which is biased off chip by the synapse module.The network is designed for real-world applicationswhere the input signals have frequency components lim-ited to tens of kilohertz. This is taken into account in thedesign of the neurons by intentionally slowing them down .This has the additional advantage that crosstalk will beminimized in neighboring lines of the network, particu-larly in the switch chip . Special attention is paid to main-taining the stability of the op amps under all conditions.The stability is influenced by the input and output capac-A. Neuron Module itances, which can be as large as 40 and 200 pF, respec-

This module consists of individual neurons, an analog tively, the output impedance of the synapses, which ismultiplexer, and digital control logic for addressing the highly dependent on the selected weight, and the diodechip and driving the multiplexer. For the prototype sys- resistance in the feedback loop, which is also current de-pendent. Furtherm ore, the neuron must be able to providean output voltage between 0 and 4 V . Miller op amps are2 ~ 0 ~ ~ ~ervice is a fast prototyping service offering fast turnaroundstandard cell and full-custom VLSI circuits at low cost. M OSISW s a reg- and ar e designed to have a minimum gain an distered trademark of the University of Southern California. gain bandwidth of 25K and 150 kHz, respectively. The

http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252http://repository.upenn.edu/ese_papers/252

8/2/2019 Analog Neural Computer

6/13

V A N DE R SPI E G E L el 0 1 . : A N A L O G N E U R A L C O MPU T E R WITH MO D U L A R A R C H I T E C T U R E 85

( p a r d )

Vout (1 Vldiv.)

0lin (12 fldiv.)

Fig . 3 . Schemat ic o f a neu ron employed in the ne twork . E ach neu ron has ( a )minimum output voltage at threshold, set by I,,,,. Vout (1VIdiv.)

comparator, on the other hand, requires a gain of 500 andunity-gain bandwidth of 750 kHz.The analog multiplexer is designed with support cir-cuitry to enable random sampling of a chip, and to gen-erate protocol signals to indicate the completion of a sam -pling sequence. The multiplexer consists of an 8-b shiftregister , which activates transmission gates and reads theneuron outputs. These outputs are then presented to abuffer, to be read off chip via the OM line. Th e shift reg-ister is externally reset before every sampling sequence.The OR1 line signals the start of a sam ple sequence whilethe O R 0 signals its completion. D uring standby, thebuffer is disconnected from the multiplexer by a read-en-able switch. The buffer is also designed as a Miller opamp, and is required to be stable for a large capacitiveload and to be fast. Howev er, a comprom ise was reachedbetween these two requirements, and the simulated bufferis found to be slew -rate limited, having a slew rate of 0.5V /p s. This is sufficiently fast for the applications of thisnetwork.The measured neuron characteristics are found to bewithin the specifications. Each neuron has an area of 228x 566 pm2 and consumes 12.2 mW when operat ing be-tween a -5- and +5 -V p ower supply. Th e value of theresistors is 30% larger than the nominal design value.However, the standard variation of the resistors over amodule and ove r different chips is 2% . The larger trans-impedance gain can be easily com pensated for by the syn-apses and does not pose a problem. The measured transfercharacteristics of the neurons is shown in Fi g. 4. Fig. 4(a)gives the output voltage versus input current from -60 tof 6 0 pA for different minimum output voltages at thresh-old, ranging from 0 to about 5 V. Fig. 4(b) gives the neu-ron outp uts for threshold currents (I,,,,) of -3 0 pA , 0 ,an d +30 pA a nd a minimum outpu t at threshold of about1 . 7 V .B. Synapse Module

The synapse module is made up of a variety of com-ponents (see Fig. 5): voltage-to-current converters, cur-rent splitters, current recombination units, shift registers,digital control logic, analog control logic, and offset con-trol. The voltage-to-current (V-to-I) converter receives an

0

0lin (12 fldiv.)

(b )Fig . 4. Measured neuron output voltage versus input current. (a ) T rans fercharacterist ics for different minimum outputs at threshold ranging from 0to 5 V . (b) Characterist ics for three different threshold currents (-30, 0 ,and +30 PA) wlth a 1 .7 -V s tep . Ver t ica l : 1 V/d i v . , ho r izon ta l : 12 pA/d iv .

input voltage from a neuron an d produces an output of 10pA/V. The current splitter multiplies the output of theV-to-I converter by several gain factors to produce therange of available output currents. The current recombi-nation unit adds together an arbitrary combination of thesplitter outputs, and produces the final output current thatgoes to the neuron. The shift register controls the currentsplitter based on a user programmed weight. The digitalcontrol logic is used in loading the shift register. Finally,analog control logic is needed to autobias both the V-to-Iconverter and the offset control.The num ber of units required in each synapse chip is alarge factor in determining how to implem ent the circuitsoutlined above. With neural networks expected to have32 inputs and 16 outputs per module, the circuit for eachsynapse has to be designed to be relatively small. Anotherimportant factor is the signal sw ing. T he neural networkoperates with signal voltages between 0 and 4 V, henceall transistors have to remain correctly biased for largeinput swings. Also, a large dynamic range of fo ur ordersof magnitude is required for the synaptic weights, whichare distr ibuted equally on a logarithmic scale. Lastly, asshown in Fig. 5 , the V-to-I converter and current splitterare common to all synapses connected to the same inputline. As a result, wherever possible, circuitry has beenmoved from the current recombination unit into the V-to-I converter and current splitter in order to minimize thearea.

8/2/2019 Analog Neural Computer

7/13

86 IEEE J O U R N A L OF SOLID-STATE CIRCUITS. VOL 27. NO I . J A N U A R Y 1992

VinlI

Voltage to

I 1 Current splitter 1lout1

'r 'i'Current Current -Recomb. - Recomb. -

I 1 I IShift Shift -Register Register

Voltage to

Recomb.

I u 'i' 'r IICurrent Current CurrentRecomb. Recomb. Recomb.

I I I I I IShift Shift Shift

Register Register Register

Autobiasm1 I 1 - Current Converter

Generator

T 'rCurrent OffsetRecomb. Control

1 I 1 IShift Shift-Register Register

Shift Shift

1 Current Splitter IVoltage to

Current Converter,ig. 5. Block diagram of the synapse module; the inputs arc common to all synapses in the same column and the output l ine5

sum the currents of all synapses in the same row.

The voltage-to-current converter is designed based ona six-transistor analog, four-quadrant multiplier devel-oped by Bult and Wallinga [25]. This circuit provides adifferential current that is dependent on the bias and theinput voltages. The original design has been augmentedto provide a single-ended output, to increase the outputimpedance and the current mirroring accuracy, resultingin a final design of 16 2 pm X 239 p m .Two major methods are considered for implementingthe current splitter and current recombination unit. In thefirst method, currents are divided into logarithmic units( i . e . , 10, 1, 0.1, 0.01), and are passed to the current re-combination circuit for further scaling (i e., 1, 1 2, 1 4)to create the final range of synapse outp uts. Such a designresults in a minimal number of analog bus lines commonto each synaptic input. The second method concentratesas much circuitry as possible in the current sp litter, in lieuof circuitry in the current recombination circuit. All in-dividual synaptic weights are generated in the currentsplitter (i.e., 10, 5,2. 5, 1, 1 2, 114, 1/ 10, 1 20, 1 40,1 loo, 1/200, an d 1 400). Twelve analog bus lines arerequired for each synapse input. The second method isfound to be significantly more compact for two reasons.First, as mentioned previously, since there are many morerecombination units than splitters, a reduction in the sizeof the recombination unit at the sacrifice of space in the

splitter is desirable. Second, the bus lines are routed overother circuitry in the synapse such that the additional buslines do not require additional space in the recombinationunit. The re is anoth er reason for using the second method:since scaling for a series of synapses is done in a singlecircuit, variation from synapse to synapse is significantlyreduced, transistors can be sized more appropriately forthe currents being scaled, and more accurate gain ratioscan be made.As shown in Fig. 6(a), the splitter is made of a seriesof ratioed current mirrors which results in effective cur-rent gains of 10, 5, 2.5 , l , l /2, l /4, l / 10, l /20, l /40,1/ 100, 1 /200, an d 11400. Each scaled current is con-nected to a diode-connected transistor. thus allowing avoltage to be passed to the recombination unit. Full-scalecurrents in these transistors range from 500 pA for a gainof 10 to 125 nA for a gain of 1/400. Size constraintslimited the dimensions of some of the diode-connectedtransistors. For the weights 10, 5, 2.5, 1, 1 /2. 114,1/ 10, an d 1 20, transistors with large widths and lengthscould be used to improve matching between the splitterand the current recombination unit. However, transistorsused for the smaller weights (1 140, 1/ 100, 11200, an d1/400) could not be sized appropriately, hence thesetransistors will operate in weak inversion fo r small currentflows. Therefore, the smaller weights are more suscepti-

8/2/2019 Analog Neural Computer

8/13

V A N DE R SPIEGEL er u l . : A N A L O G N E U R A L CO MP U TER WI T H M O D U L A R A R C H I T E C T U R E 87ybrbCurr ent Converter jQ2CvddT

1 I l l lout'

Shift Register (6 bits)(b)

Fig 6 S~mpl~fiedlr cu~ t chema t~c f the synapse (a) The top representsthe V-to-I converte r and the current sp l~t ter (b) The bottom part 1s therecomblnat~on unlt and sh ~f t egister Only the bottom part needs to berepeated for each synapse

ble to noise and exhibit larger gain errors. Th e final sizeof a single layout containing both a V-to-I and a currentsplitter is 332 pm x 4 0 8 p m .The current recombination unit is a relatively simplecircuit as can be seen in Fig. 6(b). The voltage producedin the splitter on each line of the analog bus is convertedback into a current via a PMOS transistor in saturation.NM OS transistors in linear mode are used to switch cur-rents on or off to select the required weight.A standard dynamic shift register is used to store thedigital data associated with each synapse. Each bit hasbeen modified slightly s o that if the on-chip clock is lockedin a certain phase, and an auxiliary control line is enabled,

the shift cells become static. The shift register is designedto operate at 2.5 MHz. A maximum of 1 ms is allowedbetween operation in dynamic mode and static mode.Each of the units described above has been fabricatedand tested individually. Outlined below are some of thetest results for the syn apse module. F ig . 7shows the out-put of a synapse with weights of 2.5, 1 , 0.5 , and 0 .25

Vl n (V )Fig. 7. Output current versus Input voltage of a synapse. measured overfour chips of the same run for weights of 2 .5 , I . 0.5. and 0 . 2 5 .

measured on four different chips . The V-to-I outp ut rangesfrom approximately 1 nA for 0-V input to 45 pA for 4.0-Vinput. The curved shape at the lower limit of the charac-teristic and slight bowing in the center are caused by theV-to-I converter, as was verified in a test synapse wherethe output of the V-to-I converter was measured directly.Currently, a slightly curved synaptic response is consid-ered beneficial for back-propagation learning algorithms,though future system testing will indicate whether thenonlinearity in the transfer characteristic will have to bemodified.In order to study the weight variations between syn-apses within a chip and between different chips, a total ofabout 1400 synapses were tested. A computer was usedto scan for all weights for each synapse. The synapticweights were determined by measuring the current outputfor two input voltages, and calculating the slope of thetransfer characteristic. The results of these measurementsfor 128 synapses on a single chip, as w ell as the resultsof 1400 synapses over 11 chips of the same run are tab-ulated in Table I. As expected the error in matching issmallest for the range between 0. 01 and 1 for which rel-atively large transistors could be used. The standard de-viation for the inhibitory weights are som ewhat large r thanfor the excitatory on es, which is du e to the sign inversionmirror used to create the inhibitory weights. By compar-ing the results within on e chip to those between chips onecan conclude that the chip-to-chip variations are consid-erably larger, as expected.The measured weights as a function of the synaptic codeare shown in Fig. 8 . These weights are normalized by themeasured w eight for a theoretical gain of 1. As can beseen, the normalized gains are very close to the theoreti-cal ones. As mentioned previou sly, the neural network issensitive to the ratio of the weights, not the absolute value.

8/2/2019 Analog Neural Computer

9/13

88 IEEE JOURNAL OF SOLID-STATE CIRCUITS, VOL. 27 , NO . 1, J A N U A R Y 1992

TABLE IM EANV A L U END S T A N D A R DEVIATIONF M E A S U R E DY N A P T I CE I G H TVALUESVER ONE CHIP 128 SYNAPSES)N D O V E R A G R O U PF 1 C HIP SB ELONGINGO T H E S A M E U NPer Chip Group of Chips

Nominal Weight Mean St. Dev. Mean St. Dev.

0 10 20 30Code

Fig. 8. Measured and theoretical weight factors of the synapse versus thedigital code programmed in the synaps e memory. The measured values arenormalized by the measured weight value for a gain of 1 .

Since the weight ratios are very close to the expectedvalue, the synaps e performance is more than ad equate forproper operation of a neural network.C. Time Constants

Modifiable time constants ranging between 5 and 1000ms are required for time-domain computations. Th ese time

constants are implemented fully monolithically on a rel-atively sm all area. The values are set by a 4-b word, storedin a local memory. The time-constant value varies loga-rithmically with the digital code.There are different ways to obtain these large time con-stants: a passive resistance and capacitor, a transconduc-tance amplifier used as an active resistance, and switched-capacitor circuits. The first method would result in exces-sive chip area. The potential problems with the othermethods are the large offsets and noise problems. The ap-proach used here follows the second method and is basedon a time-constant multiplication technique, by using aload-compensated operational transconductance amplifier(OTA) as the resistance. In the design of the circuit a verysmall current ratio between the input stage and outputbranch of the OTA is chosen [26]. This accomplishes twogoals: first, it provides a large transconductance reduc-tion; and seco nd, it limits the small current to the outputstage only and thus reduces the leakage current and theoffset voltage. In addition, the input stage is designed tohandle signals up to 4 V . Hence, the overall area of theamplifier and capacitance is minimized. The area of a1000-ms time co nstant, including switching for selectingdifferent time constants, memory for storing the digitalcode, the bias circuit, and an output buffer, is 1075 pmx 540 pm. Offset voltages in the range of 5 to 20 mVwere obtained for time-constant values up to 60 ms andincreased to about 200 mV for the larger time constants.Fig. 9 shows typical values of the measured time con-stants as a function of the digital code. The standard de-viation of the time constants m easured on 16 circuits overfour chips lies between 2 .5 and 7 .5 % depending on thetime-constant value .

D. Crosspoint Switch ChipThe function of the switch chip is to route the analogsignals between neuro n, syn apse, and time-constant mod-ules. Each sw itch is set by 1 b, stored locally. When aswitch is activated it connects a horizontal line to a ver-tical line. The interconnection architecture is thus set bythe switch positions. A module consists of an array ofsuch crosspoint switches. In addition, there are switchesat the end of the horizontal and vertical lines. Theseswitches interrupt the connection to the next switch, al-lowing the interconnection buses to be partitioned in sev-eral sections to optimize the available routing space.These switches can also ground unused lines in order toprevent floating lines. A 2-b memory is used to set eachof these last switches.The block diagram of a switch module consists of theswitch fabric and of digital logic circuitry used to controlloading of the memory. Each switch is realized as aCMOS transmission gate, clocked between f V . Inmaking the floor plan and corresponding layout, specialattention was paid to prevent latch-up in this module. De-signing the input pads properly is critical in reducing thechance for latch-up in the switch chip.

8/2/2019 Analog Neural Computer

10/13

V A N DE R SPIEGEL et at.: ANALOG NEURAL COMPUTER WITH MODULAR ARCHITECTURE 89

CodeFig. 9. Average time-constant value measured on 16 circuits. The stand-ard deviation lies between 2.5 an d 7.5%.

The switch modules have been found to be very reliableand no latch-up was found after taking the precaution,mentioned above. The on-resistance varies between 2 and3 kQ depending on the level of the inpu t sign al, which liesbetween 0 and 5 V. Off-resistance was in the teraohmsrange.

The neural network is programmed from one of twoperspectives: the physical view or the logical view (seeFig. 10). The physical editor displays a map of all thechips in the neural network. The user then selects a chipto edit, and makes the desired change to a specific syn-apse, switch, or time constant on the indicated chip. W henchanges are complete, the physical editor uses a devicedriver to send the chip settings to the network controller,which in turn drives the digital control wires ap propriatelyto program the network.The logical view displays an abstract representation ofa neural network in a graph form using symbols for theneurons and synapses. The user places neurons, and ar-bitrary weighted connections between the neurons, with-out particular regard for the organization of the networkhardware. A router is used to convert the logical view toa physical representation that can be examined with thephysical editor.The network comp iler converts the user-readable inputfile to an input file for the place-and-route program. Pro-gramming the network then proceeds in the same manneras with the logical editor.Finally, network operation can be observed through thenetwork state display. A 12-b AID converter in the net-work controller can sample the state of the neuronsthrough the OM line, as shown in Fig. 1. The samplingprocess is transparent to the operation of the neurons. Thenetwork state display supports two output modes . In mode

1 Network Compiler / I Logical Editor /Physical Editor

Device Driver

1 Neural Network I(analog)

Fig. 10. Overview of the supporting sof tware system , residing on a digitalhost, used to control the network configuration and to monitor the opera-tion. The neural network can be program med through a logical or physicaleditor or a network com piler. Learning algorithms are implemented on thehost computer as well.

I (dlgltal) T Host

1, a strip chart is shown for each neuron. The magnitudeof a neuron voltage at successive time slices is graphed ineach chart. In mode 2, each neuron is represented by pix-els whose intensity is proportional to the neuron outputvoltage.Currently, all of the above systems are functional, ex-cept for the network compiler and the place-and-routesystems, which are still in development. An 80386-basedpersonal computer (PC with ISA bus) is used as the hostcomputer. The network controller consists of a buffermemory and PAL-based finite state machine. Together,the network controller and 80386 PC can serially programthe network at 2. 5 Mb /s . Lastly, the network controllerhas random access read and write capability. H ence, anyindividual chip can be read or written without modifyingthe other chips in the system .

I I

V . PERFORMANCEND RESULTS F A PROTOTYPENETWORKA prototype analog neural system has been completelyassembled and tested based on the units described abov e.As discussed in Section IV , software has also been writtento control the configuration and monitor the network op-eration from the host com puter. Th e prototype consists of72 neurons, 2 466 s ynapses, 2 1 120 switches, and 36 timeconstants. Each neuron has 32 inputs. The system con-tains in total over 100 separate chips connected togetheras shown in Fig. 1. The total memory to digitally set thenetwork configuration is about 40 kb, which can be down-loaded in 1 6 ms. Th e system is realized on three processor

boards. The current system is a test version of a largerone that is currently being developed and that will haveover 1024 neurons with 64 inputs per neuron.As mentioned earlier, the components and the overallsystem have been designed for real-world applications, inparticular speech. As a result, the time response of theneurons has been optimized for such applications whereresponse times in the tens of microseconds up to ten of

8/2/2019 Analog Neural Computer

11/13

90 IEEE. J O U R N A L OF SOLID-STATE CIRCUITS, VOL. 27. NO . I , J A N U A R Y 1992milliseconds range are required in order to be able to han-dle and process the incoming signals in real time. Al-though the network could run at higher sp eeds, it was de-liberately slowed dow n. The overall computational powerof the network comes not so much from the individualcomponents but from the highly interconnected and par-allel architecture and the spatio-temporal processing of theincoming signals. For this reason, it is not so meaningfulto characterize the performance of the network by thespeed of the individual comp onents. It is more instructiveto look at the overall response of the network for partic-ular applications.The network has been configured for a number of ap-plications, including a "winner take all" net, an associ-ative network, a neural inte grator, and several circuits forthe computation of time-domain pattern primitives [27]-[29]. The latter ones have been realized by using feedbackand synaptic time constants, which generate dynamic ac-tivity patterns in the absence of external inputs. For the"winner take all" net, 1 6 neurons were fully intercon-nected with mutual inhibitory connections. Experimentsfor different inhibitory gains were performed. For inhib-itory connections of 0. 9 a clear ' 'winner" emerged , whilefor inhibitory connections of 0 .5 contrast enhancement ofthe input patterns was observed. The network settledwithin one time constant of the neurons. Also experimentsto verify the fan-in and fan-out capabilities were per-formed in which the outputs of 32 neurons were routedthrough synapses with gain of 0.03 and summed togetherinto one neuron, proving that noise was no problem evenat these small gains.Another application of the network is the real-timeanalysis of acoustical patterns. This network is pro-grammed for the primary decomposition of acoustical pat-terns into primitives which are functions of energy, space(frequency), an d time [30], a s shown in Fig . 1 1. The an-alog input signals to the network come from a bank ofhigh-Q bandpass filters (Q,,, = 300 dBloctave), withcharacteristics similar to the ones found in the coc hlea. Inthis example eight bandpass filters were used with the fol-lowing center frequencies: 400, 600, 750, 900, 1100,1700, 2500, and 4000 Hz. The primary neurons (layer 1)receive inputs from the bandpass filters. These neuronsare mutually inhibiting in a center-surround fashion withspatially decaying gains. Lateral inhibition is applied toenhanc e the frequency tuning of the primary neurons. Th enext stage involves the O N an d OFF neurons, which com-pute the temporal positive and negative derivative of theamplitudes. The ON units receive undelayed excitatory anddelayed inhibitory inputs from the neurons in layer 1,whereas the OFF units receive delayed excitatory and un-delayed inhibitory inputs. O N an d OFF neurons are mu-tually inhibiting. Th e units in layer 3 are the complementsof the ON an d OFF units. They are normally O N via a biasindicated by the arrow in the figure and are inhibited bythe activity of the neurons in layer 2. Th e last stage com-putes changes in formant frequency and their direction(local rise and fall of frequency, i.e., motion) through a

i Cochlea iLayer 1

Layer 2

Layer 3

Layer 4

OutputsFig. 11 . Logical view of part of the network used for the primary decom-position of acoustical patterns. The neurons receive input from eight (onlyfour are shown) bandpass filters.

combination of outputs of neurons of level 2 and 3, as isshown in Fig. 11.The resulting outputs of the 5 6 neurons used in this ex-periment are given in Fig. 12 for a time period of 1000ms. These waveforms are generated and recorded in realtime. Fig. 12(a) gives the primary decomposition of an"ah" sound and Fig. 12(b) of a "dah" sound. Noticethe ON an d OFF units and their complementary outputs.The "dah" sound shows a formant transition at the lowerfrequencies (output of neuron 63) which is absent in the"ah" of Fig. 12(a). The network easily keeps up with thespoken word and decom poses the sound into its primitivesin real time. Th ese outputs can be used as input to a pat-tern recognition circuit to recognize individual phonemes.How ever, this is beyond the scope of this paper. T he cur-rent network is too small to be useful for speech recog-nition. At least a 1000-neuron net will be required to per-form all of the needed functions for speed recognition.Even for this simple prototype network (72 neurons) thespeed advantage ove r digital simulations is impressive. ASun 411 10 workstation was used to simulate the netw ork,modeled as a set of differential equations, using a fourth-order Runge-Kutta algorithm [21]. We found that a dig-ital compu ter with a speed of at least 10" FLOP S is re-quired to match the real-time performance of this neuralnetwork.

The network described here is an attempt to constructa fully analog, parallel, and dynamic neural computer in-tended for real-world application s. Among its features areprogrammable synaptic time constants and weights overa large dynamic range. The performance of a prototypeneural network, consisting of over 100 programmablechips, has been evaluated . The network has been used forseveral applications including real-time acoustical patternanalysis. The individual modules consisting of neurons,synapses, time constants, and analog switches are foundto be functioning well within the overall network. The

8/2/2019 Analog Neural Computer

12/13

V A N DER SPIEGEL et a l . : A N A L O G N E U R A L CO MP U TER WI T H MO D U L A R A R C H I T E C T U R E

[I] T. Sejnowski , C. Koch, and P. S. Churchland, "Computational neu-roscience," Science, vol . 241, pp. 1299-1306, 1988.121 S. Grossberg, Studies ofMind and Brain, vol . 70, R. Cohen and M.Wartofskey, Eds. Boston, MA: D. Reidel, 1982.[3] J. Hopfield, "Neur ons with graded response have collective compu-tational properties like those of two-state neurons," Proc. Nut. Acad.S c i . , vol . 81, pp. 3088-3092, 1984.[4 ] D. E. Rumelhart and J . L . McClelland, Eds., Parallel DistributedProcessing: Explorations in the Microstructures of Cognition, vols.1, 2. Cambridge, MA: MIT Press, 1986.[5] T . Kohonen, "Correlat ion matrix memories," IEEE Trans. Comput. ,VOI.C-21, pp. 353-359, 1972.[6] P. Mueller, T. Mart in, and F. Putzrath, "General principles of op-erations in neuron nets with application to acoustical pattern recog-nition," Biological Prototypes and Synthetic Systems, vol . 1. NewYork: Plenum, 1962, pp. 192-212.[7 ] B. Widrow and M. E. Hoff, "Adaptive switching circuits," in 1960IRE WESCON, Conv. Rec. (New York), vol. IRE , 196 0, pp. 96- 104.[8] L. E. Atlas and Y. Suzuk i, "Digital system s for artificial neural net-works," IEEE Circuits Devices, pp. 20-24, 1989.[9] M. Ysunaga e t a l . , "Design, fabrication and evaluation of a 5-inchwafer scale neural network LSI composed of 576 digital neuron s," inProc. IJCNN Int. Joint Con$ Neural Netw orks, vol. 11, 1990, pp.527-535.[ l o ] R. Hecht-Nielsen, "Neurocomputing: Picking the human brain,"IEEE Spectrum , pp. 36-41, in 1988.[ I l l H . P . G r af a nd L . Jackel, "Analog electronic neural network cir-cui t ," IEEE Circuirs Devices Mag., pp. 44-49, July 1989.[12] H. Graf and D. Hend erson, "A reconfigurable CM OS neural net-work," in ISSCC Dig. Tech. Papers, 1990, pp. 144-145.1131 B. Boser and E. Sa ckinger , "An analog neural network processorwith programmable network topology," in ISSCC Dig. Tech. Papers ,1991, pp. 184-185.[14] S. Satyanarayana, Y. Tsividis , and H. P . Graf, "A reconfigurableanalog VLSI neural network chip," in Advances in Neural Informa-tion Processing Systems 2 , D . Touretzky, Ed. San Mateo, CA:Morgan Kaufmann, 1990, pp. 758-768.

[ 151 J . Raffel. J . Mann, R . Berger. A. Soares, and S. Gilbert , "Electronicimplementation of neuromorphic systems," in Proc. IEEE CustomInt. Circuirs Cunf., 1988, pp. 10.1.1-10.1.7.1161 J . Alspector and R. B. Allen, "A neuromorphic VLS I learning sys-tem," in Advanced Research VLSI, Proc. 1987 Stanford Conf.1171 1. White e t a l . , "Parallel architecture of 2D Gaussian convolution of>Fig. 12. Output voltages of 56 neurons during a 1000-ms time slice from images," Neural Networks. vol . 1 , p. 415. 1988.a network configured for pattern analysis: (a ) pattern for an 1181 M. H oller, S. Tam, H . Castro. and R. B enson, "An electrically t rain-sound, and (b) for a "dah" sound. Horiz ontal: 1000 ms; vertical: 5 V . able artificial neural network etann w~ t h10240 floating gate syn-apses," in IJCNN Int. Joint Conf. Neural Networks, vol . 2, 1989,

accuracy, noise levels, and stability proved to be entirelyadequate, illustrating that scaled-up networks are feasi-ble.The main advantage of analog hardware realizations,working in continuous time, is the computational speed.Simulations on a digital computer show that a machinewith a processing power of 10" FLOPS is required inorder to keep up with the neural computer. T he next ver-sion, which will have 1024 neurons and 64 synaptic in-puts per neuron, will have a performance in the order of1012- 1013quivalent FLO PS and is expected to be able toperform speech analysis in real time.ACKNOW L E DGM E NT

The authors would like to thank M. Massa and V .Agami for the support with the software developmen t, S .Fernando for help with testing of the switch modules, andProf. M . Steyaert for suggestions with the design of thetime-constant circuits.

pp. 191-196.[I91 A. Moopenn, T. Duong, and A. Thakoor, "Digital-analog hybridsynapse chips for electric neural network s," in Advances in NeuralInformation Processing Systems, 2 , D . Touretzky, Ed. San Mateo,CA: Morgan Kaufmann, 1990, pp. 769-776.1201 C. Mead, Analog VLSI and Neural System s. Reading, MA: Addi-son-Wesley, 1989.[21] P. Mueller e t a l . , "Design and fabrication of VLSI components fora general purpose analog neural compute r," in IEEE Workshop An-alog VLSI Implementation of Neural Systems, Circuits Syst. Con f .(Port land), C. Mead and M. Ismail . Eds. Boston: Kluwer, 1989.[22] N. Farhat, "Optoelectronic neural networks and learning mach ines,"IEEE Circuits Devices, pp. 32-41, 1989.(231 K. Kyum a, "Optical architectures for neuron circuits," in Proc. 1991IEEE Int. Symp. Circuirs Svst. (lSCAS91), vol. 3, pp. 1384-1387.1241 P. Mueller and J. Laz zaro, "A machine for neural computation ofacoustical patterns with applic at~o n o real speech recognition," inAIP Con5 Proc. , vol . 151, 1986, pp. 321-326.[25] K. Bult and H. Wallin ga, "A CM OS four-quadrant analog multi-pl ier, ' ' IEEE J. Solid-State Circuits , vol . SC-21, no. 3, pp. 430-435,June 1986.[26] M. Steyaert , P . Kinget , W. Sansen, and J . Van der Spiegel , "Fullintegration of extremely large time constants in CMOS," Electron.Let t . , vol . 27, no. 10, pp. 790-791, 1991.[27] P. Mueller, e t a l . , "Design and performance of a prototype generalpurpose analog neural com puter," in Proc. Int. Joint Conf NeuralNetworks ( IJCN N) , vol . I , 1991, pp. 463-468.[28] J. Van der Spiegel er al. , "A m ultichip analog neural netwo rks," in

8/2/2019 Analog Neural Computer

13/13

92 IEEE JO U RN A L O F S O LID -STA TE CIRCU ITS , V O L. 27 , NO . I . JANUARY 1992

Proc. Int. Syrnp. VLSI Tech. Syst. Applications, VLSITSA, 1991, pp. Peter Chance received the B.S. and M.S. degrees in computer science64-68. from the University of Pennsylvania, Philadelphia.[29] P. Mueller et a l . , "Design and performance of a prototype analog From 1974 to 1984 he worked in robotics and machine learning. Hisneural compu ter," in Proc. 2nd DOD Workshop Neural Networks current interests include speech recognition and robotic control using neural(Huntsville, AL), Sept. 1991. networks.[30] P. M ueller, "Computation of pattern primitives in a neural networkof acoustical pattern recognition," in Proc. Int. Joint ConJ NeuralNetworks ( IJCNN),vol. I, 1990 , pp. 149-151.

Jan Van der Spiegel (S'73-M'79-SM'90) wasborn in Aalst, Belgium. He received the engi-neering degree in electromechanical engineering,and the Ph.D. degree in electrical engineeringfrom the Katholieke University of Leuven, Bel-gium, in 1974 and 1979, respectively.From 1980 to 1981 he was a postdoctoral fellowat the University of Pennsylvania, Philadelphia,where he became an Assistant Professor of Elec-trical Engineering in 19 81. Currently, he is an As-sociate Professor of Electrical Engineering and ofMaterial Science and Engineering. He ho lds the Bicentennial Chair of theClass of 19 40 and is Director of the Cen ter for Sensor Technologies at theUniversity of Pennsylvania. His research interests are in artificial neuralnetworks, CCD sensors, microsensor technology, and low-noise, low-power, and high-speed analog integrated circuits.Dr. Van der Spiegel is the recipient of the Presidential Young Investi-gator Award, an IBM Award for Young Faculty D evelopment, the Reid S.Warren Award, and the C.M. Lindback Award for distinguished teaching.He is Editor for Sensors and Actuators (North and South America).

Paul Mueller received the M.D. degree fromBonn University, Germany.He has worked in molecular and systems neu-roscience since 195 3 and has been involved in the-oretical studies and hardware implementation ofneural networks since the early sixties.

David Blackman received the B.S. degree in computer science and engi-neering from the University of Pennsylvania, Philadelphia, in 1981, andthe M.S. degree in computer science from Rochester Institute of Technol-ogy in 1986.From 1981 to 1 986, he worked as an Engineer at Xerox Corporation inRochester, NY. He is currently a graduate student in the Department ofNeuroscience at the University of Pennsylvania where he is working onmodels of the visual cortex and VLSI implementation of neural networks.

Chrostopher Donham was born in Greenwich,CT, in 1967. He received the B.S.E. and M.S.E.degrees in electrical engineering from the Univer-sity of Pennsy lvania, Philadelph ia, in 1989 and1991, respectively. He is currently working to-wards the Ph.D. degree in electrical engineeringat the University of Pennsylvania where he is re-searching neural network based speech recogni-tion.Mr. Donham received the Outstanding StudentMember A ward for the Philadelphia branch of theIEEE in 1988.

Ralph Etienne-Cummings was born in Victoria.Seychelles, in 1967. In 1988 he graduated Vale-dictorian and summa cum laude from LincolnUniversity, PA , with the B.Sc. degree in physics.He is currently pursuing the Ph.D. degree in elec-trical engineering at the University of Pennsyl-vania, Philadelphia. His research interests includehardware implementation of neural networks andvisual tracking systems.Mr. Etienne-Cummings was a Fountain Fellowfrom 1988 to 1990 and is currently a Hams Fel-low.

Peter Kinget (S'88) was born in Buffalo, NY, onJune 24, 1967. He received the M.Sc. degree inelectrical and mechanical engineering in 1990from the Katholieke Universiteit of Leuven. Bel-gium. D uring the summ er of 1989 he participatedin the summer research program on sensor tech-nologies for undergraduate students at the Uni-versity of Pennsylvania. Philadelphia. Currently.he has a N.F.W.O. fellowship which allows himto work as a R esearch Assistant at the ESAT Lab-oratory, Katholieke Universiteit Leuven. He isworking towards the Ph.D. degree on the analog VLSI implementation ofneural networks. His research interests include neural networks and analogVLSI design.