I

Intelligent Systemsfor

Data Analysis & Modeling

Washington State UniversitySummer, 1997

A Module for Secondary TeachersInvolving Neural Networks

Written byBeau Oswalt-Rumsey and Lyle Becker

This work was supported in part by the NSF Education Grant NSF-ESI-9254358 andwas performed at the Data Analysis and Modeling: Non-linear Laboratory, School ofElectrical Engineering and Computer Science at Washington State University underthe supervision of Professor Zoran Obradovic.

/t

Table of Contents

I. IntroductionA. Mathematics reform policy ........................................ -1B. Biological vs. artificial neural networks .................... 2C. Training a neural network ........................................... 3D. Applications of neural networks ................................ 4

II. BackgroundA. Neural network research ............................................ .4B. Feature selection process ............................................. 5C. Description of teaching module ................................. 6

III. ActivitiesA. Activity l(Number Comparison Tutorial) ........................... 7B. Activity 3 (The Poison hhshroom Problem) ..................... .13

IV. References ............................................................................ 17V. Appendices

A. Appendix A(Glossary of Terms) ..................................... 18B. Appendix B&st of Topics) ............................................ .20C. Appendix C(L esson Plans, Descriptions, and Worksheets) .21

1. Outline of Activities .................................................... .222. Activity 1 Lesson Plan ................................................ .243. Observation Form ...................................................... 25a4. Number Comparison.. .............................................. .25b5. s-ary ..................................................................... 25c6. mm. dat ....................................................................... 25d7. num.in .......................................................................... 25g8. Activity ~(How do NeuralNetworks Work?) ................... .269. Activity 3 Lesson Plan ................................................. 2910. Problem Trial Form .................................................... 30a

INTRODUCTION

In the late 198Os, the mathematics educational community felt a strongneed to reform mathematics education across the elementary and secondarycurriculum. As a result, the National Council of Teachers of Mathematicsdeveloped the “Standards” in an attempt to bmet mathematics teachers acrossthe United States to incorporate activities that focus on communication,reasonmg, problem solving, and connections to real-world applications intotheir classrooms. Washington State has developed its own educationalreform policy as well in the form of the “Essential Academic LearningRequirements for Mathematics.” Both reform policies are focused onapplying mathematics into real-life situations. Data collection, analysis, andinterpretation each play an integral role in an application-based approach tomathematics. Artificial neural networks allow students to experience anapplication-based approach to mathematics through statistics and predictions,as well as gain an understandin,D for how computer technology will be used tobenefit society in the future.

Humans have always searched for methods to make their lives easier.Just as the wheel allowed people to transport themselves in a more proficientmanner, machines have been developed to assist them in their physical andmental labors. The development of high speed personal computers andsoftware enable computers to exhibit many characteristics attributed tohuman intelligence. They can reason deductively, they perform complexmathematical calculations, they can learn new solutions to old problems, anddisplay good judgment and massive knowledge.’ People are able to look atproblems, quickly eliminate branches of possible outcomes, and then come upwith correct solutions. Can computers think like humans? Artificial neuralnetworks have been developed to simulate human thought patterns anddecision making abilities.

Artificial neural networks (ANNs) are a form of artificial, or computer,intelligence. ANNs sometimes have other names: artificial neural systems,connectionist systems, neurocomputers, adaptive systems, parallel distributedprocessors, collective decision circuits, and natural intelligence.’ Historically,the term artificial intelligence has been used to name the field of computerscience dedicated to producing programs that attempt to be as smart ashumans.3 AlNNs attempt to take the data processing architecture of biologicalsystems and implement it into an artificial system.4

ANNs get their name from the way that they resemble complicatedbiological neural networks found in the brain and nervous system. Thedifference between ANNs and biological systems lies in the data processing.For example, in a biological system, a neuron is composed of a number ofinput fibers (dendrites), a cell body or nucleus (soma), and an output fiber

1

(axon).” Suppose a person were to burn their finger on a hot stove. Impulsescoming from sensory transducers in the hand would be carried down thedendrites and into the soma through electrical impulses. The soma thenemits an output impulse that travels down the axiom fibers. The outputimpulse then becomes the input impulse for additional dendrites, and theoriginal impulse is carried throughout the nervous system to the brain. Thebrain then emits a signal back to our finger to jerk away from the source ofpain. In most instances, the person is then able to learn that a burner on astove could cause them pain, and they will not make the same mistake in thefuture. Learning takes place during the process.

An ANN is a computer program with an internal structure thatimitates the human brain’s interconnected system of neurons. A neuron inan ANN is called a processing element or node.6 Layers of nodes within theprogram resemble the human biological system:

Neurons are located in one of three types of places: the input layer, theoutput layer, or the hidden layers. The input neurons receive data fromthe outside world, such as a digitizing pad, a data file, or anotherprogram. The input layer neurons send information to the hidden layerneurons. The hidden neurons are all the neurons in between the input andoutput layers. We do not see their inputs and outputs because theyconnect only to other neurons. They are part of the large internal abstractpattern which represent the neural network’s solution to the problem.The output neurons send information out to us, or do something else, suchas a mechanical control. The output neurons provide us with thenetwork’s response to the input data.7

In other words, we can present information into the input layers and thenetwork gives you an answer in the output layer. Diagram 1’ illustrates thedesign of a human biological neuron whereas Diagram 2 illustrates an ANNin terms of its individual neurons and layers.

Diagram 1: Biological Neuron Diagram 2: A Simple ANN

Table 1 lists the common analogies between the components in a biologicalneural system and man-made artificial neural networks.

Biolog;ical Svstems Artificial Neural Networksneuron w processing element, nodesynapse w weighted connection

axon fiber w output channel, output portdendrite fiber ti input channel, input port

neural network w system of highly interconnected,nonlinear nodes

Table 1

ANNs function by means of inductive reasoning based on exposure totraining examples. One process used to train ANNs is called “BackPropagation.” For example, pairs of inputs and outputs are collected andpresented into the network. Each time an input is presented to the network,the network will guess as to what the correct output will be. If an incorrectguess is made, the network will compare its output to the training set andrearrange the pathways in the hidden layer so that the same mistake will notbe made again. This allows the network to have a better chance of guessingcorrectly on additional tests. Consider the following example. Suppose youare training an ANN to detect the classifications of flowers based on theirname. We could use a training set that tests whether a flower is a rose, adandelion, or a carnation. The inputs for this network may be stem type,petlzl type, and C&V. If you enter in thorny stem, soft petal, and red color andthe network is in a training session, it may guess dandelion in its initial run.The network would then compare the correct answer in the training set. Ifthe network needs to make changes, it will back propagate to change thepathways in the hidden layer so that the ANN would not make the sameincorrect guess in the future. Once the network is trained using pre-selectedinputs and outputs, we can run the ANN (without any supplied outputs) ona testing set to determine the accuracy of the network. After the testing sethas run, we can use our network to recognize, generalize, and predict thecorrect answer for us when given a new set of data.

A neural network is an excellent candidate for any applicationrequiring pattern recognition. A pattern may consist of visual, numeric, orsymbolic data. If your problem involves pattern recognition or classification,a neural network has the potential to do it faster, more consistently, and oftenbetter than a person.” There are many applications for the use of artificial

3

neural networks in the real-world. A very brief list of ANN abilities is givenin Table 2 along with an example of each.

Abilitv or Talentpattern recognition

generalizationtrend prediction

behavior predictionevaluation

good at optimizingiast & smooth operation

filtering

Example Apulicationidentify submarines from sonaraccess real estatedecide when to buy and sell stockpredict outcome of surgeryaccept/deny loan applicationsflight schedulingrobotic arm controlcleaning up video signals

Table 2

We have included Appendix B” to provide a list of the topics that havebeen explored with artificial neural networks. Even with the abundance ofuses of neural networks, the exciting thing is that, even with all of the theseachievements, the technology is still in its infancy.” ANNs have beenpredicted to be common household items by the year 2000.”

This teaching module is designed to give high school students a hands-on approach to learning artificial neural networks with the assistance of theBrainMaker Macintosh 1.0 software package. BrainMaker is a simulatedbiological intelligence, created by California Scientific Software, that enablesnovice users of AlNNs to create, train, and test data files to make predictionson real-world data. BrainMaker may be purchased by contacting:

California Scientific Software10024 Newtown Road, Nevada City, California 95959

Sales (800) 284-8112 l (916)478-9040 l Technical Support (916)478-9035Fax (916)478-9041

Total Price w/shipping: $203.00 (Julv 1997)

Background Information

Our 6 week program at Washington State University was spentworking with Dr. Zoran Obradavic and Pedro Romero, a doctoral candidate,in the School of Electrical Engineering and Computer Science. We werebombarded with foreign terms and concepts. What did neural networks,feature selection, disorder in proteins, amino acids, Calcineurin, Mahalanobisdistance, back propagation, probabilistic distance, ren,, stimpy, zmaj, UNIX,and tooldiag mean? As the first week progressed, we read two books:

4

“Introduction to Neural Networks: Design, Theory, and Applications,” byJeannette Lawrence, and “Building Neural Networks,” by David M. Skapura.These books, along with articles off the World Wide Web, helped us gain abetter understanding of ANNs. We were trained to know what we would besearching for, what a neural network was, and how it was used. In addition,we learned the investigative approaches that we would be taking during ourresearch.

Our research project involved setting up a neural network that wouldidentify proteins having amino acids with disorder and their respectivelocations. In our mentor’s previous research, they determined that neuralnetworks could recognize pattern similarities in disordered regions inproteins. Disordered regions are hard to determine by x-ray diffraction fromprotein crystals. Many proteins have both structured and disordered regions.The structured regions scatter x-rays coherently and are observed while thedisordered regions do not crystallize and scatter x-rays incoherently.Therefore, they are invisible in the resulting electron density maps.13Obradovic and Romero were able to identify proteins containing disorderedregions by using a neural network. The prediction accuracy ranged from 59%-74% based upon the length classes used. With this success in prediction,neural networks would be set up to do a more extensive and exhaustivesearch for disordered regions in other proteins.

Our first task became feature selection with Calcineurin (CaN). Themethod for the selection of the best 10 out of 24 features was very thorough.The first run of selections was based on a matrix of 24 features with 7different window sizes. The 24 features were the 20 different amino acidsfound in proteins plus flexibility, hydrophobic moment, alpha-mom andbeta-mom. The window sizes were lengths of strands of amino acids, i.e., astring of 7, 9, 11, 13, 15, 17, 19 and 21. Using feature selection software on theUNIX called tooldiag, our first search was a sequential forward search,probabilistic distance, Mahalanobis distance, selecting the 10 best features.This method picked the best feature, then the second best, etc., until the first10 had been selected. We then did a sequential backward search thateliminated the features until the best 10 were left. We tried some of thedifferent distance options, i.e., quadratic Guassian distance. The results werenot the same. We then tried selecting different numbers of features. This wasto determine if a different combination of features would alter which oneswere selected. The results were not conclusive as to which group was the best.

Dr. Obradovic discovered an article on feature selection written by AnilJain and Douglas Zongker, of Michigan State University. The article,“Feature Selection: Evaluation, Application, and Small SamplePerformances,” convinced us that a branch-and-bound search would be thebest. The branch-and-bound feature selection algorithm can be used to findthe optimal subset of features much more quickly than an exhaustive

5

1

search.14 We then did a series of feature selection using CaN with varyingwindow sizes and number of features. The selection process was withwindow sizes of 9, 15 and 21 with feature selection numbers of 2 to 24. Thisprocess was very time consuming requirin,0 that tooldiag be started and leftrunning for hours to complete each selection. The results were recorded andcompared. After graphing the results, it appeared that the larger window with10 features would bring the best results. Further testing would be conductedon CaN with higher window sizes to determine which will be the mostappropriate. A similar exhaustive approach will be conducted with the nextprotein Bcl to see if the feature selection for CaN can be used on the otherones. If the feature selection process is the same it will speed the results forthe other proteins.

In the neural network process, feature selection is the most crucial step.Good feature selection will allow a high prediction rate for our outcomes.Once the features are selected, the training of the neural network can begin.The training is composed of training data and testing data. These have knownoutcomes. The training data will run through epochs, or training runs, usingback propagation until the network gets the right results. The network willthen be tested with data with known outputs to determine accuracy of thenetwork. If there is a high percentage of accuracy, the network will be trainedand operational. Once trained, the network can be used to identify otherproteins that have disorder in their amino acids and their location.

Five weeks into the research, Romero discovered a software packagecalled Partek that involves a feature reduction process. Obradovic andRomero discovered that feature reduction, when paired with featureselection, will allow the prediction accuracy to increase from the 59%-74%range to approximately 92%. This discovery lead to changes in the directionof their research, and we were unable to assist in this portion of the projectdue to time constraints in the Summer Teachers Institute.

Description of Teaching: Module

The activities included in this module are set up for students to workin groups of 2 to 4 students (depending on the access of computers in yourschool that are set up to run BrainMaker). The goal of this teaching moduleis to allow students to apply statistics using real-world data to makepredictions on the outcomes of future events. Three neural networkactivities follow. You may find Lesson Plans, descriptions, and worksheets tothe activities by looking in Appendix C.

ACTIVITY 1:Number Comparison Tutorial.

This tutorial is designed as an introduction to the use of BrainMaker.BrainMaker allows students to create neural networks programs. The neuralnetwork for Activity 1 will determine if the first number in each row of the“Activity 1: Number Comparison” handout is less than, equal to, or greaterthan the second number found in the row. The tutorial will allow studentsto create input and output files, train a neural network, test their trainedneural network, and make predictions. Students should read all directionscarefully and proceed slowly through the tutorial. If students have anyquestions, feel free to ask the teacher for help.

PROCEDURE:

I. Complete the directions for First and Second on the Activitv 1:Number Comuarison handout. Be sure to fill in the handout.

II. Creating: a Running Fact File for Predictions.

1.3_.

3.4.5.6.

7.8.9.

Double click on the Activity 1 folder.Double click on the NetMaker alias icon, select File and drag down toOpen.Select Excel 3.x, then click OK.Double click on num.in.Select File, drag down to Create Running Fact File.In the dialog box on Column Type, mark the columns as follows:

First . . . . . . . . . . . . Input . . . . . . . . . OKSecond . . . . . . . Input . . . . . . . . . OK

Enter the filename num.trained.in, click on Save.Select File, drag down to Close.Select Cancel.

III. Creatine a Neural Network.

10. Select File, drag down to Open.11. Select Excel 3.x with your mouse, then click OK.12. Double click num.dat.13. Select File, drag down to Create BrainMaker Files.

7

14. In the dialog box on Column Type, mark the columns as follows:

First . . . . . . . . . . . . Input . . . . . . . . . OKSecond . . . . . . . Input . . . . . . . . . OKRating . . . . . . . Pattern . . . . . . . OK.

15. Accept the default filenames num.net, num.fct, and num.tst byclicking on Save.

16. Select File, drag down to Quit, and click on Cancel.

IV. Training Your Neural Network.

17.18.

19.

20.

21.

22.23.

Double click on the num.net icon.Hit OK to find neuron.tbl, hit OK, and click on Save to acceptneuron. tbl.Select File and drag down to Select Fact Files.

Find:Read Running Facts From: num.in.Hit Close first, then click on Select,Click on Open to choose num.trained.in.

Click on OK.Select Parameters, drag down to Learning Parameters.

l Set the Maximum Number of Training Runs: 100.Then click OK.Select Display and drag down to Edit Network Display.

l Input and Number circles must be highlighted together.l Output and Number circles must be highlighted together.l Pattern and Number circles must be highlighted together.

Then click on OK.Select Operate and drag down to Train Network.The network will take about 7 to 10 minutes to train. Take out theTraining Network Observation Form and follow directions #lthrough #3.

V. TestinP Your Trained Neural Network and Comparing Learning; Rates.

24. Select Operate and drag down to Test Network. Look at the Bad andGood Sections in the Statistics Display. Take the number found inGood and divide by the sum of the number found in (Good + Bad).Hint: 1 means 100% whereas .98 means 98%. This represents the“Test Data % Correct.”

25. Take out the Activity 1: Number Comparison (AlNC) handout.Enter the Test Data % Correct (found in step 24) into the row labeled

Test Data % Correct which is located at the bottom of AlNC, in thecolumn labeled Case 1: LR = 1.0.

2:Case Predicting number comparisons using a neural network with 10Neurons and a Learninp Rate = 1.0.

26.27.

25.

29.

1

Case 1: Predicting number comparisons using a neural network with 10Neurons and a Learning Rate = 0.5.

Select Operate, dra,0 down to Single Step Network.Select Operate, drag down to Run Train Network.@Hit Return once on your keyboard.

l Look at the Input and Output display boxes.l The Input box shows the first two numbers on the AlNC.l The Output display shows the prediction that the neural

network has made for the correct answer.Record the result onto the AlNC handout in the column labeledCase 1, LR = 1.0.

l Hint: Record 1 for a range of .5 to 10 for a range of -.499 to .499-1 for a range of -1 to -.5.

l Hit Return on your keyboard again.l Continue to record your results until all 25 number

comparisons have been recorded.

Compare your network predictions with the actual answers youplaced into the “First # c, =, >” column found on the handout.Determine the percentage correct by counting the number of incorrectresults. Subtract this number from 25 and then multiply the newanswer by 4. Enter this number into Input Data % Correct row foundat the bottom of the AlNC handout, in the column labeledCase 1: LR = 1.0.

30. Select Parameters, drag down to Learning Parameters.l Change the Learning Rate to 0.5l Change the Maximum Number of Training Runs to 200.

Hit OK.31. Select Operate, drag down to Single Step Network.32. Select Operate, drag down to Train Network.

9

.-

33.

34.

35.

36.37.

38.

39.

When the statistics display shows: Run: ZOO, turn to the AlNChandout.

l Enter the number Good into the row labeledNeural Network % Correct which is located on the AlNChandout in the column labeled Case 1: LR = 0.5.

Select Operate and drag down to Test Network. Look at the Bad andGood Sections in the Statistics Display. Take the number found inGood and divide by the sum of the number found in (Good + Bad).Hint: 1 means 100% whereas .98 means 98%. This represents the“Test Data % Correct.”Take out the AlNC handout.

l Enter the Test Data % Correct into its correct location onthe bottom of the AlNC handout in the column labeledCase 1: LR = 0.5.

Select Operate, dra g down to Single Step Network.Select Operate, dra,c down to Run Train Network.

l Hit Return once on your keyboard.Record the results onto the AlNC handout in the column labeledCase 1, LR = 0.5.

l Hint: Record 1 for a range of .5 to 10 for a range of -.499 to .499-1 for a range of -1 to -.5.

l Hit Return on your keyboard again.l Continue to record your results until all 25 number

comparisons have been recorded.

Compare your network predictions with the actual answers youplaced into the “First # c, =, >” column found on the handout.Determine the percentage correct by counting the number of incorrectresults. Subtract this number from 25 and then multiply the newanswer by 4. Enter this number into Input Data % Correct row foundat the bottom of the AlNC handout in the column labeledCase 1, LR = 0.5.

2 :Case Predicting number comparisons using a neural network with 5LVerrrons and a Learning Rate = 1.0

40. Select Parameters, drag down to Learning Parameters.l Change the Learning Rate to 1.0l Change the Maximum Number of Training Runs to 100.

Hit OK.41. Select Connections and drag down to Change Network Size. Change

the number of neurons in Layer 1 from 10 to 5. Hit OK.: IDLJ

42.43.44.45.

46.

47.

48.49.

50.

Select Operate and drag down to Single Step Network.Select Connections and drag to Randomize All Connections. Hit OK.Select Operate and drag down to Train Network.When the statistics display shows: Run: 100, turn to the AlNChandout.

l Enter the number Good into the row labeledNeural Network % Correct which is located on the AlNChandout in the column labeled Case 2: LR = 1.0.

Select Operate and drag down to Test Network. Look at the Bad andGood Sections in the Statistics Display. Take the number found inGood and divide by the sum of the number found in (Good + Bad).Hint: 1 means 100% whereas .98 means 98%. This represents the“Test Data % Correct.”Take out the AlNC handout.

l Enter the Test Data 96 Correct into its correct location onthe bottom of the AlNC handout in the column labeledCase 2: LR = 1.0.

Select Operate, drag down to Single Step Network.Select Operate, drag down to Run Train Network.

l Hit Return once on your keyboard.Record the results onto the AlNC handout in the column labeledCase 2, LR = 1.0.

l Hint: Record 1 for a range of .5 to 10 for a range of -.499 to .499-1 for a range of -1 to -.5.

l Hit Return on your keyboard again.l Continue to record your results until all 25 number

comparisons have been recorded.

51. Compare your network predictions with the actual answers youplaced into the “First # C, =, >" column found on the handout.Determine the percentage correct by counting the number of incorrectresults. Subtract this number from 25 and then multiply the newanswer by 4. Enter this number into Input Data % Correct row foundat the bottom of the AlNC handout in the column labeledCase 2: LR = 1.0.

11

2 :Case Predicting number comparisons using neural network with 5Neurons and a Learnina Rate = 0.5

52.

53.54.35.

36.

57.

58.59.

60.

61.

62.63.

Select Parameters and drag down to Learning Parameters.l Change the Learning Rate to 0.5l Change the Maximum Number of Training Runs to 200.

Hit OK.Select Operate and drag down to Single Step Network.Select Operate and dra,c down to Train Network.When the statistics display shows: Run: 100, turn to the AlNChandout.

l Enter the number Good into the row labeledNeural Network % Correct which is located on the AlNChandout in the column labeled Case 2: LR = 0.5.

Select Operate and dra,0 down to Test Network. Look at the Bad andGood Sections in the Statistics Display. Take the number found inGood and divide by the sum of the number found in (Good + Bad).Hint: 1 means 100% whereas .98 means 98%. This represents the“Test Data % Correct.”Take out the AlNC handout.

l Enter the Test Data % Correct into its correct location onthe bottom of AlNC in the column labeled Case 2: LR = 0.5.Select Operate, drag down to Single Step Network.Select Operate, dra,c down to Run Trained Network.

l Hit Return once on your keyboard.Record the results onto the AlNC handout in the column labeledCase 2: LR = 0.5.

l Hint: Record 1 for a range of .5 to 10 for a range of -.499 to .499-1 for a range of -1 to -.5.

l Hit Return on your keyboard again.l Continue to record your results until all 25 number

comparisons have been recorded.

Compare your network predictions with the actual answers youplaced into the “First # <, =, >” column found on the handout.Determine the percentage correct by counting the number of incorrectresults. Subtract this number from 25 and then multiply the newanswer by 4. Enter this number into Input Data % Correct row foundat the bottom of the AlNC handout in the column labeled Case 2,LR = 0.5.Select File and drag down to Quit. Hit Cancel.Complete the Activity 1 Summary.

12

3ActivityThe Poison Mushroom Problem

Mushrooms play a role in our diets. They are often used when makingsalads, hamburgers, and various types of casseroles. We often get ourmushrooms from stores, but what should we do if we were to come across anice, big mushroom patch while walking through the woods while campingwith family. Should you eat them based on their appearance? We willconsider this question with the assistance of a neural network.

This neural network will consider whether or not a mushroom isedible or poisonous based on twenty-three attributes. The attributes are listedbelow. Each attribute has various characteristics. The characteristics are basedon a description, a code letter, and a numeric representation of thecharacteristic. This information will help you when interpreting your dataset.

Attribute # 1 Attribute 1 Characteristics, Description, and Code#

(1)

(2)

(3)

(4)(5)

(6)

(7)(8)(9)

(10)

Cap Shape

Cap Surface

Cap Color

BruisesOdor

Gill Attachment

Gill SpacingGill Size

Gill Color

Stalk Shape

Bell=b=l, Conical=c=2, Convex=x=3,Flat=f=4, Knobbed=k=5, Sunken=s=6Fibrous=f=l, Grooves=g=2, Scaly=y=3Smooth=s=4Brown=n=l, Buff=b=2, Cinnamon=c=3Gray=g=4, Green=r=5, Pink=p=6,Purple=u=7, Red=e=8, White=w=9,Yellow=y=lOYes=t=l, No=f=2Almond=a=l, Anise=l=2, Creosote=c=3Fishy=y=4, Foul=f=5, Musty=m=6,None=n=7, Pungent=p=S, Spicy=s=9Attached=a=l, Descending=d=ZFree=f=3, Notched=n=4Close=c=l, Crowded=w=2, Distant=d=CBroad=b=l, Narrow=n=2Black=k=l, Brown=n=2, Buff=b=3,Chocolate=h=4, Gray=g=5, Green=r=6,Orange=o=7, Pink=p=8, Purple=u=9,Red=e=lO, White=w=ll, Yellow=y=12Enlarging=e=l, Tapering=t=2

13

(11)

(12)

(13)

(14)

05)

(16)(17)

(18)(19)

(20)

(21)

(22)

(23)

Stalk Root

Stalk SurfaceAbove RingStalk SurfaceBelow RingStalk Color

Above Ring

Stalk ColorBelow Ring

Veil TypeVeil Color

Ring NumberRing Type

Spore PrintColor

Population

Habitat

Edibility

Bulbous=b=l, Club=c=2, Cup=u=3,Equal=e=4, Rhizomorphs=z=5,Rooted=r=6, Missing=?=7Ibrous=f=l, Scaly=y=2, Silky=k=3,Smooth=s=4Ibrous=f=l, Scaly=y=2, Silky=k=3,Smooth=s=4Brown=n=l, Buff=b=2, Cinnamon=c=3,Gray=g=4,Orange=o=5, Pink=p=6,Red=e=7, White=w=8, Yellow=y=9Brown=n=l, Buff=b=IL, Cinnamon=c=3,Gray=g=4,Orange=o=5, Pink=p=6,Red=e=7, White=w=8, Yellow=y=9Partial=p=l, Universal=u=2Brown=n=l, Orange=o=2, White=w=3,Yellow=y=4None=n=l, One=o=2, Two=t=3Cobwebby=c=l, Evanescent=e=ZFlaring=f=3, Large=l=4, None=n=5,Pendant=p=6, Sheathing=s=7,Zone=z=SBlack=k=l, Brown=n=2, Buff=b=3,Chocolate=h=4, Green=r=5,Orange=O=6, Purple=u=7, White=w=8,Yellow=y=9Abundant=a=l, Clustered=c=2,Numerous=n=3, Scattered=s=4,Several=v=5, Solitary=y=6Grasses=g=l, Leaves=l=2,Meadows=m=3, Paths=p=4,Urban=u=5, Waste=w=6, Woods=d=7Edible=e=l, Poisonous=p= -1

It is very important that you and your partner share responsibilitieswhen taking part in this activity. Each group is required to do the followingthings.

1. Use the AlNC Tutorial to help you create a BrainMaker file for the datafile mushroom.tr. Use the attribute numbers as labels in you file.

l Attributes (1) through (22) are INPUT values whileAttribute (23) is a PATTERN.

14

2. Use NetMaker to create a neural network. It is essential that you helpeach other when creating your network. Use The Poison MushroomProblem Trial Form (PMPTF) as an aid to help you create differentsized neural networks.

l You are trying to make a network that will be the moreaccurate than those of other groups in the class.

l Creating different sized networks will allow you to seewhich network is the most accurate.

l Record all of you network sizes on the PMPTF.

3. Read through all of the Attribute Characteristics in the table above.

l Go to the school library to research the characteristics forAttributes (l), (2), (6) through (8), (10) through (13), (16j,and (19).

l Write a brief description for each characteristic, and thendraw pictures to represent the characteristics for Attributes(l), (2), (6), and (II).

4. Upon the completion of your neural network, bring the PMPTF toyour teacher to show them your results. Then complete the exercisebelow.

EXERCISE:

1. Use your PMPTF to help you answer the following question

l How did varying the size of you neural network increaseyour accuracy prediction for your training data? Providemathematical support from the PMPTF.

2. Below are four different types of mushrooms. Decode the mushroomsfrom their numeric representation to a verbal description usingattributes and characteristics. (Hint: The attributes are listed inascending order going from left to right.)

Mushroom A:6,4,10,2,1,2,2,2,12,1,2,4,4,9,9, 2,4,1, 5, 9,4,1

Mushroom B:6,4,3,2, I, 2,2,2, 7,2, 2,4,4,5,5, 2, 2,1, 5, 6, 2, 7

Mushroom C:1,3,2, 2, 7,3, 1,2,3, 2,2,2,4,2,2,2,3,2,4,3,4, I

Mushroom D:4,4,9, 2, 7, I, 1,2, 11,2,3,4,4,8,8,2,3,2,7,8,4,2

3. Using TeachText, create a white-spaced data file using the numberslisted above. Use the attribute numbers as labels in your file. Call thisTeachText file mushroom-in.

4. Use the AlNC Tutorial to help you create a BrainMaker file for the datafile mushroom.in. Use the attribute numbers as labels in you file.

l Attributes (1) through (22) are INPUT values.(You will predict if the mushrooms are poisonous oredible.)

5. Test you neural network using the file mushroom.in. Record yourresults and label Mushrooms A, B, C, and D as poisonous or edible.Then use the Internet to call up the location:

http://services.state.mo.us/conservation/nathil

Read through the different types of mushrooms on this web site, andtry to classify Mushrooms A, B, C and D by their names based on theattribute characteristics above, the predicted results discovered throughBrainMaker, and the information discovered on the web site.

16

References

1. J. Lawrence, Introduction to Neural Networks: Design, Theorv, andAuulications, (California Scientific Software Press, 1995), p. 4.

2.3.4.

Center,5.6.7.8.

J. Lawrence, p. 9.J. Lawrence, p. 4.Introduction to Neural Networks, (!vlolecular Science ResearchPacific Northwest Laboratory, 1994), p. 1.1.Introduction to Neural Networks, p. 1.3.J. Lawrence, p. 81.J. Lawrence, p. 82.http://wwwdse.doc.ic.ac.uk/-nd/surprise_96/journal/vo14/csl1/report.html#Human and Artificial Neurones -investigating the similaritiesJ. Lawrence, p. 22.9.

10. Introduction to Neural Networks, p. 1.2.11. J. Lawrence, p. 9.12. J. Lawrence, p. 3.13. I?. Romero, Z. Obradovich, C. Kissinger, J. E. Villafranca,

A. K. Dunker, Identifvine Disordered Reoions in Proteins from Amino AcidSequence, (Proceedin$, Volume 1 of 4, 1997 International Conference onNeural Networks, Houston, Texas, June 1997).

14. A. Jain, and D. Zongker, Feature Selection: Evaluation, _Apulication,and Small Sample Performance, (IEEE Transactions on Pattern Xnalysis andMachine intelligence, Vol. 19, No. 2, February 1997).

17

Appendix A

Glossary of Terms

Artificial Neural Network: A model made to simulate a biological neuralsystem. It may model brain processes or brain capabilities. They are a formcomputer intelligence and are referred to as artificial neural systems,connectionist systems, neurocomputers, adaptive systems, parallel distributedprocessors, collective decision circuits, and natural intelligence.

Artificial Intelligence: A branch of computer science in which computers areused to model or simulate some portion of human reasoning or brainactivity.

Axon: The cylindrical extension of the biological neuron which carries theoutput signal away to other neurons.

Back Propagation: A supervised learning method in which an error signal isfed back through the network altering weights as it goes, in order to preventthe same error from recurring.

Biological neural network: The neurons in an human or animal brain.

Connection: A unique line of communication between two neurons alongwhich impulse signals are sent.

Connectionists: People who accept the new electronic approach tocomputation using highly interconnected neuron-like components.

Delta Rule: The learning rule that states that, if there is a difference betweenthe actual output pattern and the desired output pattern during training, theweights in the hidden layer are changed to reduce the difference.

Dendrites: The input channels of a biological neuron; hair-like extensions ofthe soma that receive excitation or inhibition signals from other cells throughthe synapses.

Feature: A particular characteristic of an object, such as shape or color.

Hidden Layer: A layer of neurons in an artificial neural network which doesnot connect to the outside world but connects to other layers of neurons.

Impulse (nerve): the transmission of energy from one neuron to another; anoutput pulse of very short duration.

Input Layer: A layer of neurons in an artificial neural network that receivedata from the outside world, such as a digitizing pad, a data file, or anotherprogram. The input layer neurons send information to the hidden layerneurons.

Neuron: A nerve cell in a biological nervous system; a processing element inan artificial neural network. It has a number of inputs and a single output.

Nucleus: A collection of nerve cells in the gray matter of the brain wheremay synapses are present (nerve fibers interconnect).

Processing element: A neuron; it has a number of inputs and a signal output.

Synapse: the area of electrochemical contact between two neurons.

Training: A process which uses one of several learning methods to modifyweights, or connection strengths in an orderly fashion, so that the neuralnetwork will run correctly.

Training Set: A list of pairs of matched input and desired output patternsused in supervised learning of training an artificial neural network. Eachpair is called a fact. It is important that all the information the network needsto learn be in the training set and that the facts contain generally associatedinformation. The inputs can be symbols, pictures, or numbers.

: 9

Amendix B

List of topics that have been explored with artificial neural networksCredit ADD~OV~~

Stock Market PredictionEconomic ModelingPicking of Mutual FundsCommodity TradingCapital Market ForecastsCurrency MovementEconomic IndicatorsProperty Tax AnalysisInsuranceUnderwritingMortgage Backed SecuritiesEvaluation of Individual’s

Financial CredibilityCheck Processing

Business

Cash Flow ForecastingPrice Forecastscost AnalysisBusiness Strategy ModelsMarketing AnalysisSales Forecasts (Magazines,

Real Estate, etc.)Sales Prospect IdentificationInventory AnalysisPredicting EmployeeRetentionPurchase Order screeningWorkloadPredictionBankruptcy PredictionVehicle RoutingPredicting Consumer Expen-

ditureBehaviorService Quality EstimationClassification of Questionnaire

DataPredicting Shipping and

Delivering Schedules

Electric Circuit AnalysisElectric Load ForecastingElectric Circuit DesignPower System StabilityOptimization of Electrical

PowerComputer Aided Design

(CAD)Troubleshooting Scientific

instrumentsAdaptive Logic Elements

Adaptive Signal ProcessingEcho Cancellation Telephone

systemsError Connection in ModemsNoise FilteringData CompressionChannel Assignment of

Cellular Phone NetworkDemultiplexing in Fiber Optic

communicationOptimized Placement of

Broadcast SatellitesOptimized Network RoutingSpeech Analysis

Food and Aericulture

PoultryLnspectionMeat InspectionInspection and Quality

Control of ApplesExperiments in AgricultureFood and Beverage Odor

MonitoringFermentationProcessEstimator

ODtics

Adaptive FocusingAdaptive Optical TelescopesImage ClassificationSpectral Analysis

Medicine and Health

Early Diagnosis of Myocar-dial Infarction

Diagnosis of Coronary ArteryDisease

MRI Tissue ClassificationX-ray Spectral ReconstructionDetermination of Skeletal AgeDisease DiagnosisAnalysis of Medical TestsTumorDetectioninUltrasoundEKG DiagnosisDrug ScreeningAlDS and Cancer ResearchDiagnosis of Babies Vulner-

able to SIDSEpilepsy Classification

20

Education

Teaching Problem SolvingPredictjng Student P&or-

manCeSComputer Aided Instruction

IndustrialInspectionProcess ControlQuality ControlOptimizing Production

SchedulesDamaged Product Identifica-

tionPredicting LaborforProductionTroubleshooting Car Assem-

blyEquipment

Enrzineering

Vibration AnalysisTurbine Blade Fault DetectionArtificial Leg ControlVehicle Control Systems

National Defense

Target Recognition and TargetTracking

Pilot TrainingControl of Space VehiclesSubmarine Guidance

Criminal Iustice and Law

Predicting Parolee RecidivismFinger Print IdentificationFace Identification

Sports

Athletic TrainingPredicting Horse Race

winnersNFL Score Prediction

Other Auulications

Legal StrategiesGround Water ControlAnalysis of Music

Amendix C

Lesson Plans, Descriptions, and Worksheets

21

Outline of Activities for Teaching Module

A. Introduction to ANN1. What is a neural network?2. Where are they used?3. Introduce BrainMaker Software4. What is the process for developing a neural network?

B. Students form groups for ANN unit1. Expectations in a group2. Students given assessment tool

C. Activity 1 - Tutorial on BrainMaker Software

D. Activity 2 - How do Neural Networks Work?1. Student create four number data using spreadsheet

or word processer2. Student uses NetMaker to create BrainMaker files3. Student trains Net using output of 0 0 0 1 with

no hidden layers.4. Student trains Net using output of 0 1 0 1 with

no hidden layers5. Student adds hidden layer with 2 Neurons.6. Student trains Net using output of 0 1 0 1

adding a column Z with 1 0 0 0 to spread the data out into athree dimension space and using no hidden layers.

E. Activity 3 - The Poison Mushroom ProblemEach group is given the Activity #3 handout and data to train an ANN.1. Input data and train ANN2. Test ANN and determine % accuracy.3. Use ANN to predict outcomes for Exercise problems #2 and #3.4. Write up of results for Exercise problem #4 and #5.

22

Major Project1. Determine the area of study2. Determine data that is necessary to be collected.3. Collect data for training set and perform feature selection4. Input data and train ANN5. Test ANN and determine O/O accuracy.6. Collect data to do predictions.7. Use ANN to predict outcomes.8. Write up of results.

23

Activity 1 Lesson PlanNumber ComParison Tutorial

OBJECTIVE:Students will be introduced to neural networks and BrainMaker. Theywill create input and output files, train a neural network, test theirtrained neural network, and make predictions.

OUTLINE1. Human

;:C.

OF ACTIVITY #l:Brain vs. Computer vs. Neural Net(discussion - read background material)How do we learn?How do computers learn to do tasks?How do Neural Nets learn?1. Usage’s - Driving a car

“Data” on Star TrekRobots i.e. Short Circuitfootball, horse race, sports forecastingMedical diagnosis

2. Example problemTraining a neural net to discriminate between two numbers

a.

b.

3. Neural

Z:

::e.

f.

How do we compare numbers?small list of pairs of numbers on board:34 6712 324 4315 1529 945 90What are the symbols that we use to compare two numbers?Generalize the rule: Given any two Real numbers a and b, either

a < b, or a = b, or a > b. Rule of Trichotomy.

Network TrainingHandout with Activity 1: Number Comparison.Computer with folder containing NetMaker, BrainMaker, anddata file.Open NetMaker, load data file, create BrainMaker files.Open BrainMaker and load on files.Train Neural Network with a learning rate of 1.0, 10 neurons, and100 epochs(runs). Then test network and record percentage correct.Using a learning rate of .5, repeat for 100 more epochs, test and

24

record percentage correct.g. Repeat steps e and f using 5 neurons.h. Was there increased accuracy as the number of epochs increased?

What was the effects of changing the number of neurons?

4. Summarize experience with Neural Networks.

25

Name: Date: Class:_

ITRAINING NETWORK OBSERVATION FORM

Observe the Statistics Display while BrainMaker is training.

1. What is taking place in each of the following sections during thetraining session?

a. Fact Section:

b. Total Section:

c. Bad Section:

d. Last Section:

e. Good Section:

f. Last Section:

g. Run Section:

Look carefully at the Last Section (found between the Good and RunSections) of the Statistics Display. This represents the number of good,or correct, training results.

2. What happens in the Last Section as the number of runs increases?

3. After BrainMaker has displayed Run: 100, turn to the Activity 1:Number Comparison handout and enter the number Good into therow labeled Neural Network % Correct which is located at the bottomof the handout.

Return to the Number Comparison Tutorial and continue withProceedure V: Testing Your Trained Neural Network and Comparing

Learning Rates.

I

Name: Date: Class:

Directions:

Activity 1: Number Comparison(Al NC)

first-A

Second:

Use the table below to determine if the first number in each row is less than,equal to, or greater than the second number found in the row. Place the correct symbolin the “First #<,=,>‘I Column.For BrainMaker to read your data file, all outputs need to be in the form of a number.As a result, translate the symbol c to -1, = to 0, and > to 1.

Neural Network % correct:Test Data % correct:Input Data % correct:

Name: Date: Class:

Activity 1: Number Comparison Summary

Directions: You have just completed the Activity 1: Number Comparisontutorial. Review all information on the Activity 1: Number Comparisonhandout. Write a summary about what you learned during thisinvestigation. Compare each case on the handout, and explain how changingthe number of neurons and/or learning rates can change the effectiveness ofthe network. Also state which network provided the best results by providingmathematical support.

_ ^ . _ -

num.dat

first second rating112 135 -155 43 1

128 128 010 77 -124 55 -141 41 085 144 -1

113 53 1I 1421 1421 01

121 11 1104 104 0130 11 140 145 -1

9 9 0102 25 1137 137 075 46 159 76 -1

144 144 073 142 -163 145 -1

114 114 075 22 175 132 -13 3 0

83 137 -115 113 -162 4 1

131 131 0100 67 124 25 -1

Page1

num.dat

37 21 110 99 -1

129 129 0105 52 146 52 -178 110 -197 97 0

107 76 150 50 0

120 34 151 102 -117 17 061 53 145 111 -1

149 149 066 95 -166 26 1

117 117 0106 9 1 1

46 46 0143 37 1138 32 160 1 1

116 116 0

. _ L

441 1371 -11411 91

1011--81 131 -1

481 1491 -111401 1401 0169 71 -1

123 14 183 83 0 ,41 90 -15 39 -1

Page 2

num.dat

I 1411 161 146 30 161 61 081 66 18 126 -1

301 301 OJ1261

47 45 1121 121 053 41 1

I 431 731 -1170 70 0

139 138 1101 134 -1122 136 -183 141 -1

114 114 010 33 -149 0 18 107 -1

Page 3

I

num.in

38 3052 5354 100a9 5042 5223 2323 946 97

Page 1

Activity 2 Lesson PlanHow do Neural Networks Work?

OBJECTIVE:The student will work with a data set to demonstrate how inputs arelinked to the outputs, the need for hidden layers of neurons and howdata can be separated by adding noise.

A

xV

YWS) *(Lu Inputs

W1 W2Neural Net

t (no hidden units)P output

*(O/O) “(l/O)

x Y output

0 0 01 0 01 1 00 1 1

t - threshold is the tolerance foryes or no.

X, Y - inputsW - weight put on links.

How can you separate the data outputs for the above inputs in the graph?

The computer is doing the following:W,*X + W2*Y = t (equation of a line) if “yes” then 1 else 0. This means that

the line has separated the pattern correctly.

*If the prediction is OK then it proceeds to the next example and checks it out.

*If under prediction (prediction is 0, should be 1) so WlneW = W, + X andW,“ew = W2 + Y and then goes through the data again and checks it out.

*If over prediction (prediction is 1, should be 0) so WlneW = W, - X andW;eW = W_ - Y and then goes through the data again and checks it out.

This process will continue until it gets it right. The Neural Network Machineis now trained. This process is called the Perceptron Learning Algorithm.This simple algorithm only works when you have zero hidden units. The

26

machines that have zero hidden units cannot learn complex things. Themain breakthroughs in neural networks came when a new learningalgorithm called Back Propagation was developed. Back propagation is usedin the BrainMaker software program.

Now we will test out this pattern on BrainMaker and see if it can learn thepattern.

Using Simple Text, Teach Text or a Spread sheet program create thefollowing:

x Y Pattern0 0 01 0 01 1 00 1 1

Save the file as FourNum.datUsing NetMaker Open this file. Then go under File and down to Set test filepercentage to 0. Then b00 under File and Create BrainMaker Files create thefile labeling X as an Input, Y as an Input and P&tern as a Pattern. Then savethe files using the defaults that appear on the screen. Then double click onFourNum.Net. Under Connections go to Change Network Size and change itso that there are only 2 layers this means that there will be 0 hidden Neuronsin the Hidden layer. Now go under Operate and Train Neural Network. Willthis train? If yes, how many runs did it take to train it? If it trains, what hasthe Neural Net found? (The training tolerance can be increased to 0.400)

Now go to another file maker program and create FourNum2.dat this file willhave the following data set:

x Y Pattern0 0 01 0 11 1 00 1 1

Using NetMaker Open this file. Then go under File and down to Set test filepercentage to 0. Then go under File and Create BrainMaker Files create thefile labeling X as an Input, Y as an Input and P&tern as a Pattern. Then savethe files using the defaults that appear on the screen. Then double click onFourNum2.Net. Under Connections go to Change Network Size and changeit so that there are only 2 layers this means that there will be 0 hidden

27

Neurons in the Hidden layer. Now go under Operate and Train NeuralNetwork. Will this train? If yes, how many runs did it take to train it? If ittrains, what has the Neural Net found? (The training folerance can beincreased to 0.400)

This one will not train with no hidden layer. Now try training it with 3 layerswith 10 neurons. Remember to go under Connections and Randomize allConnections. Will it train? If yes, try reducing the number of neuron in thehidden layer until it will not train. Remember with each change in thenumber of neurons to Randomize all Connections before trying to train thenetwork. What was the least number of neurons that you were able to trainthis network. (The training tolermce can be increased to 0.400)

Now go to another file maker program and create FourNum3.dat this file willhave the following data set:

x Y Pattern Z0 0 0 11 0 1 11 1 0 10 1 1 1

Using NetMaker Open this file. Then go under File and down to Set test filepercentage to 0. Then go under File and Create BrainMaker Files create thefile labeling X as an Input, Y as an Input, Pattern as a Pattern and Z as anInput. Then save the files using the defaults that appear on the screen. Thendouble click on FourNum3.Net. Under Connections go to Change NetworkSize and change it so that there are only 2 layers this means that there will be0 hidden Neurons in the Hidden layer. Now go under Operate and TrainNeural Network. Will this train? If yes, how many runs did it take to train it?If it trains, what has the Neural Net found? (The training tolerance can beincreased to 0.400)

28

Activitv 3 Lesson PlanThe Poison Mushroom Problem

OBJECTIVE:To use a neural network to classify mushrooms as being poisonous oredible based on their attribute characteristics.

OUTLINE:

1. Start by distributing the handout for Activity 3: The PoisonMushroom Problem.

2. Provide students with a file folder containing the mushroom.tr data.Students will be creating their own BrainMaker file using the sameprocedures as the Activity 1 Tutorial.

3. Students will then create different sized neural networks for themushroom.tr data by using The Poison Mushroom Problem TrialForm to record their results.

4. After students are able to train a network so that it is close to 100%accuracy, students will complete the exercises found at the conclusionof the Activity 3 handout.

5. The information provided in Exercise problem 2 was taken from theweb site:

http://services.state.mo.us/conservation/nathi\

It was then transformed into code by using the Attribute Tableprovided on the handout.

6. Students will try to determine if Mushrooms A, B, C and D are edibleor poisonous. They will also use the web site above to try to classify themushrooms based on their actual names. Here is a list of the actualanswers if the neural network predicts the result correctly.

Mushroom A: (Edible) - - Chanterelles6,4,10,2,1,2,2,2,12,1,2,4,4,9,9,2,4,1,5,9,4,1

Mushroom B: (Poisonous) - - Jack-o-lanterns6,4,3,2,1,2,2,2,7,2,2,4,4,5,5,2,2,1,5,6,2,7

Mushroom C: (Poisonous) - - Green-spored Lepiota1,3,2,2,7,3,1,2,3,2,2,2,4,2,2,2,3,2,4,3,4,1

Mushroom D: (Deadly Poisonous) - - Amanitas4,4,9, 2, 7, 1, 1,2,11,2,3,4,4,8, 8,2,3,2,7,8,4,2

7. An extension exercise would be to have the teacher go to get different2 9

types of mushrooms and set up labs in the class. This way, studentscould classify the mushrooms by their different attributes andcharacteristics in a hands-on lab activity. Students could then create aTeachText for their real mushroom characteristics and use the neuralnetwork to see if the mushroom is edible or poisonous.

30

The Poison Mushroom Problem Trial FormActivitv #3

1st Trial 2nd Trial 3rd Trial

size: 3 layers size: 3 layers size: 3 layers

I H 0 I H 0 I H 022 11 I 22 II L. _ - -

LR = 1.0 SF = 0.9 L R = _ S F = _ L R = _ SF = _

Trn To1 =O.l Tst To1 = 0.4 Tm To1 =_ Tst To1 =_ Tm To1 =_ Tst To1 =_

%CorrecP=_ T i m e : _ %CorrecP=_ T i m e : _ %CorrecP=_ T i m e : _

#runs.*- %Cor Tst=_ # r u n s : %Cor Tst=_ #runs.. %Cor Tst=_

4th Trial 5th Trial 6th Trial

size: 4 layers size: _ layers size: _ layers

I H, H, 0 I H, H, 0 I I-I, H, 0_____-- _---

LR = 0.1 SF = 0.9 L R = _ SF = 0.9 L R = _ SF = 0.9

Trn To1 = 0.1 Tst To1 = 0.4 Tm To1 = _ Tst To1 = 0.4 Tm To1 = _ Tst To1 = 0.4

%CorrecP=_ T i m e : _ %CorrecP=_ T i m e : _ %Correct”“=_ T i m e : _

##runs.*- %Cor Tst=: #runs.. %Cor Tst=_ #runs*. %Cor Tst=_

7th Trial 8th Trial 9th Trial

size: _ layers’ size: _ layers size: _ layers

LR = 1.0 SF = 0.9 LR = 0.5 SF = 0.9 L R = _ SF = _

TmTol=_TstTol=_ TmTol=_TstTol=_ TmTol=_TstTol=_

%CorrecP=_ T i m e : _ %CorrecP=_ T i m e : _ %Correct”=_ T i m e : _

#runs.*- %Cor Tst=_ # r u n s : %Cor Tst=_ #runs.. %Cor Tst=_