K Nearest NeighborsSaed Sayad*www.ismartsoft.com

www.ismartsoft.com

KNN - DefinitionKNN is a simple algorithm that stores all available cases and classifies new cases based on a similarity measure*www.ismartsoft.com

www.ismartsoft.com

KNN different namesK-Nearest NeighborsMemory-Based ReasoningExample-Based ReasoningInstance-Based LearningCase-Based ReasoningLazy Learning*www.ismartsoft.com

www.ismartsoft.com

KNN Short HistoryNearest Neighbors have been used in statistical estimation and pattern recognition already in the beginning of 1970s (non-parametric techniques).Dynamic Memory: A theory of Reminding and Learning in Computer and People (Schank, 1982).People reason by remembering and learn by doing. Thinking is reminding, making analogies.Examples = Concepts???*www.ismartsoft.com

www.ismartsoft.com

KNN ClassificationAgeLoan$*www.ismartsoft.com

www.ismartsoft.com

KNN Classification Distance*www.ismartsoft.com

AgeLoanDefaultDistance25$40,000N10200035$60,000N8200045$80,000N6200020$20,000N12200035$120,000N2200052$18,000N12400023$95,000Y4700040$62,000Y8000060$100,000Y4200048$220,000Y7800033$150,000Y8000

48$142,000?

www.ismartsoft.com

KNN Classification Standardized Distance*www.ismartsoft.com

AgeLoanDefaultDistance0.1250.11N0.76520.3750.21N0.52000.6250.31N0.316000.01N0.92450.3750.50N0.34280.80.00N0.62200.0750.38Y0.66690.50.22Y0.443710.41Y0.36500.71.00Y0.38610.3250.65Y0.3771

0.70.61?

www.ismartsoft.com

KNN Regression - Distance*www.ismartsoft.com

AgeLoanHouse Price IndexDistance25$40,00013510200035$60,0002568200045$80,0002316200020$20,00026712200035$120,0001392200052$18,00015012400023$95,0001274700040$62,0002168000060$100,0001394200048$220,0002507800033$150,0002648000

48$142,000?

www.ismartsoft.com

KNN Regression Standardized Distance*www.ismartsoft.com

AgeLoanHouse Price IndexDistance0.1250.111350.76520.3750.212560.52000.6250.312310.316000.012670.92450.3750.501390.34280.80.001500.62200.0750.381270.66690.50.222160.443710.411390.36500.71.002500.38610.3250.652640.3771

0.70.61?

www.ismartsoft.com

KNN Number of NeighborsIf K=1, select the nearest neighborIf K>1,For classification select the most frequent neighbor.For regression calculate the average of K neighbors.*www.ismartsoft.com

www.ismartsoft.com

Distance Categorical Variables*www.ismartsoft.com

XYDistanceMaleMale0MaleFemale1

www.ismartsoft.com

Instance Based Reasoning IB1 is based on the standard KNN IB2 is incremental KNN learner that only incorporates misclassified instances into the classifier. IB3 discards instances that do not perform well by keeping success records.*www.ismartsoft.com

www.ismartsoft.com

Case Based Reasoning*www.ismartsoft.com

www.ismartsoft.com

New Case

Case Database(Prior Cases)

Similar Cases

Solved Case

Test/Repaired Case

Suggested Solution

Confirmed Solution

Retrieve

Retrieve

Reuse

Revise

Learned Case

Problem

Retain

Retain

KNN - ApplicationsClassification and Interpretationlegal, medical, news, banking

Problem-solvingplanning, pronunciation

Function learningdynamic control

Teaching and aidinghelp desk, user training

*www.ismartsoft.com

www.ismartsoft.com

SummaryKNN is conceptually simple, yet able to solve complex problems Can work with relatively little informationLearning is simple (no learning at all!)Memory and CPU costFeature selection problemSensitive to representation*www.ismartsoft.com

www.ismartsoft.com

*www.ismartsoft.comQuestions?

www.ismartsoft.com