Neural Networks and Their Applications

John Paxton

Montana State University

August 14, 2003

Yellowstone National Park

Problem Domains

• Storing and recalling patterns

• Classifying patterns

• Mapping inputs onto outputs

• Grouping similar patterns

• Finding solutions to constrained optimization problems

Human Brain

• 10 billion neurons

• 60 trillion connections (synapses)

soma

axon

dendrite

Human Brain

• Plastic

• Nonlinear

• Parallel

• Distributed Memory

• Distributed Processing

Artificial Neural Network

input layer middle layer output layer

Comparison

Biological Neural Net Artificial Neural Net

Soma Neuron

Dendrite Input

Axon Output

Synapse Weight

Neuron

x1

x2

x3

y

w1

w2

w3

Neuron Input

• -1 (absent)

• 0 (unknown)

• 1 (present)

Neuron Output

• X = ( ∑ xiwi )

• y = fn ( X )

• Common functions (fn)– sign function (-1 if X < 0, else 1)– sigmoid function (1 / (1 + e-x) )

Perceptron

• OR concept

• sign function

x0 (1)

x1

x2

-1

2

2

Perceptron

• Can automatically learn the weights in a provable fashion!

• But can only learn linearly separable concepts.

no

yes

yes

no

Multilayer Neural Network

• Can include zero or more hidden layers

• Has a learning algorithm (backpropagation) that works in practice!

XOR

Backpropagation

• Determine network topology

• Initialize weights

• Present a training example

• Apply inputs, calculate activations in middle layer, then calculate activations in output layer

• Calculate errors in output layer

• Calculate errors in hidden layer

Backpropagation

• Update weights

• Repeat process until some stopping condition is met

• Possible stopping conditions– largest error is below some threshold– total error is no longer decreasing– a time limit is exceeded

Clustering (Unsupervised)

• Traveling Salesperson Problem (6 cities)

• Kohonen Nets

Strengths

• Very versatile. They can predict, they can classify, they can cluster.

• They produce good results in complicated domains.

• Can handle categorical and continuous data.

• Available in many off-the-shelf products

Weaknesses

• All inputs must be massaged onto the range [-1 .. 1 ].

• Can not explain results.

• May converge on an inferior solution.

• Determining the topology is as much an art as it is a science.

• Might take a long time to converge.

Commercial Applications

• Neural networks were involved in more than 1 billion U.S. dollars in 1997!

Business

• Marketing– Microsoft. Direct mail marketing.– Albertsons. Determine the connection

between buying diapers and buying beer.– BehavHeuristics Inc. Forecasts demand of

airline flights.

• Real Estate– HNC. Automated Real Estate Appraisal.

Document and Form Processing

• Machine Printed Form Processing– Caere Corporation. Optical character

recognition (FoxMaster).– Synaptics. Check reader.

• Hand Written Character Recognition.– Eastman Kodak. Forms processing for UK

motor vehicle registration.– Fujitsu. Input to pen based computers.

Document and Form Processing

• Cursive Handwriting Character Recognition– Apple Newton 120. Input to PDA.

• Graphic Recognition– Fein-Marquet Associates, Inc. Converts a

hand drawn chemistry picture into a table.

Food Industry

• Odor/Aroma Analysis– Sharp. Cooking control via an electronic nose

in a microwave oven.

• Produce Development– M&M/Mars. Improved chemical formulations

of products.

Food Industry

• Quality Assurance– Anheuser-Busch. Beer testing.– Florida Department of Citrus. Pulp wash

detection.– Frito-Lay. Potato chip testing.

Financial Industry

• Market Trading– Gerber Baby Foods. Cattle futures trading.– John Deere. Pension management.– Walkrich Investment Advisors. Stock

valuation.

• Credit Rating– Chase Financial. Forecast credit worthiness.

Financial Industry

• Fraud Detection– Dunn and Bradstreet. Check approval.– HNC. Credit card fraud detection (Falcon).– Mastercard. Deviations in spending habits.

Energy

• Electrical Load Forecasting– Bayernwerk AG.

• Hydroelectric Dam Operation– Tauernkraftwerke. Dam displacement

prediction.

• Natural Gas– Northern Natural Gas. Predict gas index

prices.

Manufacturing

• Process Controllers– Nippon Steel. Continuous casting.– Siemans. Rolling mill.

• Quality Control Systems– Dunlop. Tires.– Intel. Computer chips.– Volvo. Diesel knock testing. Paint inspection.

Medical and Health Care

• Image Analysis– NeuroMedical Systems, Inc. Pap smears.

• Drug Development– Vysis Inc. Protein analysis

• Resource Allocation– Anderson Memorial Hospital. Predict use of

hospital resources.

Science and Engineering

• Chemical Engineering– StellarNet Inc. Spectroscopy.

• Electrical Engineering– NeuroCad Inc. Optimized circuit routing.

• Weather– National Weather Service.

Transportation and Communication

• Transportation– London Underground. Fault detection.– Rolls Royce. Fault detection.

• Communication– AT&T/Lucent. Echo cancellation systems

(more than 20 years).

Questions?