ANFIS: Adaptive Neuro-Fuzzy Inference SystemANFIS: Adaptive Neuro-Fuzzy Inference System
Chapter 12: ANFIS
Chapter 12: ANFIS
2
OutlineOutlineSoft computingFuzzy logic and fuzzy inference systemsNeural networksNeuro-fuzzy integration: ANFIS
• ANFIS: Adaptive Neuro-Fuzzy Inference Systems• Learning methods for parameter ID
Input selection for ANFIS modeling• Heuristic and exhaustive searches• Performance index
Application examples• Hair dryer modeling• Box-Jenkins furnace data
Chapter 12: ANFIS
3
Neuro-Fuzzy and Soft ComputingNeuro-Fuzzy and Soft Computing
Neural networks
Fuzzy inf. systems
Model space
Adaptive networks
Derivative-free optim.
Derivative-based optim.
Approach space
SoftSoftComputingComputing
Chapter 12: ANFIS
4
Fuzzy SetsFuzzy Sets
Sets with fuzzy boundaries
A = Set of tall people
Heights(cm)
170
1.0
Crisp set A
Membershipfunction
Heights(cm)
170 180
.5
.9
Fuzzy set A1.0
Chapter 12: ANFIS
5
Membership Functions (MFs)Membership Functions (MFs)
About MFs• Subjective measures• Not probability functions
MFs
.5
.8
.1
“tall” in Taiwan
“tall” in the US
“tall” in NBA
180 Heights(cm)
Chapter 12: ANFIS
6
Fuzzy If-Then RulesFuzzy If-Then Rules
• Mamdani styleIf pressure is high then volume is small
high small
• Sugeno styleIf speed is medium then resistance = 5*speed
mediumresistance = 5*speed
Chapter 12: ANFIS
7
Fuzzy Inference System (FIS)Fuzzy Inference System (FIS)
If speed is low then resistance = 2If speed is medium then resistance = 4*speedIf speed is high then resistance = 8*speed
Rule 1: w1 = .3; r1 = 2Rule 2: w2 = .8; r2 = 4*2Rule 3: w3 = .1; r3 = 8*2
Speed2
.3
.8
.1
low medium high
Resistance = Σ(wi*ri) / Σwi= 7.12
MFs
Chapter 12: ANFIS
8
First-Order Sugeno FISFirst-Order Sugeno FIS
• Rule baseIf X is A1 and Y is B1 then Z = p1*x + q1*y + r1
If X is A2 and Y is B2 then Z = p2*x + q2*y + r2
• Fuzzy reasoning
A1 B1
A2 B2
x=3
X
X
Y
Yy=2
w1
w2
z1 =p1*x+q1*y+r1
z =
z2 =p2*x+q2*y+r2
w1+w2
w1*z1+w2*z2
Π
Chapter 12: ANFIS
9
Fuzzy Inference Systems (FIS)Fuzzy Inference Systems (FIS)
Also known as• Fuzzy models• Fuzzy associate memories (FAM)• Fuzzy controllers
Rule base(Fuzzy rules)
Data base(MFs)
Fuzzy reasoning
input output
Chapter 12: ANFIS
10
Neural NetworksNeural Networks
Supervised Learning• Multilayer perceptrons• Radial basis function networks• Modular neural networks• LVQ (learning vector quantization)
Unsupervised Learning• Competitive learning networks• Kohonen self-organizing networks• ART (adaptive resonant theory)
Others• Hopfield networks
Chapter 12: ANFIS
11
Single-Layer PerceptronsSingle-Layer Perceptrons
Network architecture
x1
x2
x3
w1
w2
w3
w0
y = signum(Σwi xi + w0)
∆wi = κ t xi
Learning rule
Chapter 12: ANFIS
12
Single-Layer PerceptronsSingle-Layer Perceptrons
Example: Gender classification
h
v
w1
w2
w0
Network Arch.
y = signum(hw1+vw2+w0)-1 if female1 if male=
y
Training data
h (hair length)
v (v
oice
freq
.)
Chapter 12: ANFIS
13
Multilayer Perceptrons (MLPs)Multilayer Perceptrons (MLPs)
Learning rule:• Steepest descent (Backprop)• Conjugate gradient method• All optim. methods using first derivative• Derivative-free optim.
Network architecture
x1
x2
y1
y2
hyperbolic tangentor logistic function
Chapter 12: ANFIS
14
Multilayer Perceptrons (MLPs)Multilayer Perceptrons (MLPs)
Example: XOR problem
Training data
x1
x2
y
Network Arch.
x1 x2 y0 0 00 1 11 0 11 1 0
x1
x2
x1
x2
y
Chapter 12: ANFIS
15
MLP Decision BoundariesMLP Decision Boundaries
A B
B A
AB
XOR Interwined General
1-layer: Half planes
A B
B A
AB
2-layer: Convex
A B
B A
AB
3-layer: Arbitrary
Chapter 12: ANFIS
16
Adaptive NetworksAdaptive Networks
Architecture:• Feedforward networks with diff. node functions• Squares: nodes with parameters• Circles: nodes without parameters
Goal:• To achieve an I/O mapping specified by training data
Basic training method:• Backpropagation or steepest descent
x
yz
Chapter 12: ANFIS
17
Derivative-Based OptimizationDerivative-Based Optimization
Based on first derivatives:• Steepest descent• Conjugate gradient method• Gauss-Newton method• Levenberg-Marquardt method• And many others
Based on second derivatives:• Newton method• And many others
Chapter 12: ANFIS
18
Fuzzy ModelingFuzzy Modeling
Unknown target systemUnknown target system
Fuzzy Inference systemFuzzy Inference system
y
y*
x1
xn
. . .
• Given desired i/o pairs (training data set) of the form(x1, ..., xn; y), construct a FIS to match the i/o pairs
• Two steps in fuzzy modelingstructure identification --- input selection, MF numbersparameter identification --- optimal parameters
Chapter 12: ANFIS
19
Neuro-Fuzzy ModelingNeuro-Fuzzy Modeling
Basic approach of ANFIS
Adaptive networks
Neural networks Fuzzy inferencesystems
Generalization Specialization
ANFIS
Chapter 12: ANFIS
20
ANFISANFIS
• Fuzzy reasoning
A1 B1
A2 B2
w1
w2
z1 =p1*x+q1*y+r1
z2 =p2*x+q2*y+r2
z = w1+w2
w1*z1+w2*z2
x y
• ANFIS (Adaptive Neuro-Fuzzy Inference System)A1
A2
B1
B2
Π
ΠΣ
Σ/
x
y
w1
w2
w1*z1
w2*z2
Σwi*zi
Σwi
z
Chapter 12: ANFIS
21
Four-Rule ANFISFour-Rule ANFIS
• ANFIS (Adaptive Neuro-Fuzzy Inference System)A1
A2
B1
B2
Σ
Σ
/
x
y
w1
w4
w1*z1
w4*z4
Σwi*zi
Σwi
z
Π
Π
Π
Π
• Input space partitioning
A1
B1
A2
B2
x
y x
y
A1 A2
B1
B2
Chapter 12: ANFIS
22
ANFIS: Parameter IDANFIS: Parameter ID
Hybrid training method
A1
A2
B1
B2
Σ
Σ
/
x
y
w1
w4
w1*z1
w4*z4
Σwi*zi
Σwi
z
Π
Π
Π
Π
nonlinearparameters
linearparameters
fixed
least-squares
steepest descent
fixed
forward pass backward passMF param.(nonlinear)
Coef. param.(linear)
Chapter 12: ANFIS
23
Parameter ID: Gauss-Newton MethodParameter ID: Gauss-Newton Method
Synonyms:• linearization method• extended Kalman filter method
Concept:general nonlinear model: y = f(x, θ)linearization at θ = θnow:
y = f(x, θnow)+a1(θ1 - θ1,now)+a2(θ2 - θ2,now) + ...LSE solution:
θnext = θnow + η(A A) A BT T-1
Chapter 12: ANFIS
24
Param. ID: Levenberg-MarquardtParam. ID: Levenberg-Marquardt
Formula:θnext = θnow + η(A A + λI) A B
Effects of λ:λ small Gauss-Newton methodλ big steepest descent
How to update λ:greedy policy make λ smallcautious policy make λ large
T T-1
Chapter 12: ANFIS
25
Param. ID: ComparisonsParam. ID: Comparisons
Steepest descent (SD)• treats all parameters as nonlinear
Hybrid learning (SD+LSE)• distinguishes between linear and nonlinear
Gauss-Newton (GN)• linearizes and treats all parameters as linear
Levenberg-Marquardt (LM)• switches smoothly between SD and GN
Chapter 12: ANFIS
26
FIS data structure in FLTFIS data structure in FLT
FIS file and FIS matrix
FIS file(on disk)
FIS Data Structure
MF 1Label: Small
Type: GaussianParams: [5 -10]
MF 2Label: Large
Type: TriangleParams: [-5 10 20]
[Input 1]Name: PositionRange: [-10 10]
MF #: 2
MF 1Label: SmallType: Gbell
Params: [5 2 -40]
MF 2Label: Large
Type: SParams: [-50 0 50]
[Input 2]Name: VelocityRange: [-50 50]
MF #: 2
MF 1Label: Neg. Big
Type: ZParams: [-10 -5 0]
MF 2Label: Neg. Small
Type: GbellParams: [5 2 -3]
MF 3Label: Pos, Small
Type: GbellParams: [5 2 3]
MF 4Label: Pos. Big
Type: SParams: [0 5 10]
[Output 1]Name: Force
Range: [-10 10]MF #: 4
[Rules]Rule list
Rule weightsRule types
[System]Name: mam21Type: mamdani
. . .
FIS matrix(in workspace)
readfiswritefis
Chapter 12: ANFIS
27
From Data Sets to FISFrom Data Sets to FIS
Flow chart: From data sets to FIS
FLTGUI tools
genfis1.m
genfis2.m
anfis.mTrainingdata
Initial FIS
Trainingdata
Checkingdata
Final FIS
Chapter 12: ANFIS
28
ANFIS: Structure IDANFIS: Structure ID
Input selection
Input space partitioning
Grid partitioning Tree partitioning Scatter partitioning
• CART method •C-means clustering•mountain method
To select relevant input for efficient modeling
Chapter 12: ANFIS
29
Input SelectionInput Selection
Optimal search: direct exhaustive search
x2x2x1x1 x3x3 x4x4 x5x51 input
x1, x2x1, x2 x1, x3x1, x3 x1, x4x1, x4 x1, x5x1, x5 x2, x3x2, x3 . . .2 inputs
x1, x2, x3x1, x2, x3 x1, x2, x4x1, x2, x4 x1, x2, x5x1, x2, x5
x1, x2, x3, x4x1, x2, x3, x4 x1, x2, x3, x5x1, x2, x3, x5
3 inputs
4 inputs
x1, x3, x4x1, x3, x4 . . .
x1, x2, x4, x5x1, x2, x4, x5 . . .
Chapter 12: ANFIS
30
Input SelectionInput Selection
Suboptimal search• One-pass ranking• Sequential forward selection• Generalized sequential forward selection• Sequential backward selection• Generalized sequential backward selection• ‘Add m, remove n’ selection• Generalized ‘add m, remove n’ selection
Chapter 12: ANFIS
31
Input SelectionInput Selection
Sequential forward selection
x2x2 x3x3 x4x4x1x1 x5x5
x2, x1x2, x1 x2, x3x2, x3 x2, x4x2, x4 x2, x5x2, x5
x2, x4, x1x2, x4, x1 x2, x4, x3x2, x4, x3 x2, x4, x5x2, x4, x5
x2, x4, x3, x1x2, x4, x3, x1 x2, x4, x3, x5x2, x4, x3, x5
1 input
2 inputs
3 inputs
4 inputs
Chapter 12: ANFIS
32
Performance IndexPerformance Index
How to effectively contruct a model and evaluate it properly?
• Model construction:ANFIS with one-epoch training(Use least-squares method only once)
• Model evaluation:- Training RMSE (root-mean-squared error)- Bipartite regularity criterion- Leave-one-out regularity criterion
Chapter 12: ANFIS
33
Performance IndexPerformance Index
Bipartite regularity criterion:
[ ] [ ]( ){ }( ){ }
RCA
t F xB
t F x
x t
x t
F
F
iA
B iA
iB
A iB
ii
iA
iA
iB
iB
A
B
= − + −⎧⎨⎩
⎫⎬⎭
•
•
∑∑1 12
2 2
| |( )
| |( )
;
;
( ):
( ):
r r
r
r
Data Set A =
Data Set B =
Model identified using data set A
Model identified using data set B
Chapter 12: ANFIS
34
Regularity CriterionRegularity Criterion
Bipartite RC (used in GMDH)
Data set A
Data set B
Model AModel A
constructionconstruction
MSEAevaluationevaluation
Model BModel B MSEB
constructionconstruction
evaluationevaluation
RC = (MSEA + MSEB)/2
Chapter 12: ANFIS
35
Regularity CriterionRegularity Criterion
Leave-one-out RC
( )rx ti i; Model kModel k
( )rx t1 1;
( )rx t2 2;
( )rx tn n;
...
...
...ni/o
pairs
constructionconstruction
MSEk( )rx ti i; evaluationevaluation
RC = (Σk MSEk)/n
Chapter 12: ANFIS
36
Regularity CriterionRegularity Criterion
Computational complexity of ANFIS:• Bipartite RC: TBRC = n*(tL+tE)• Leave-one-out RC: TLRC = 2n*tL + n*tE
(tL: time for one-entry sequential LSE update)(tE: time for one-entry model computation)
• If tL = 4tE, then TLRC/TBRC = 9/5 ~= 2.
Chapter 12: ANFIS
37
Input Selection Based on RCInput Selection Based on RC
Test example: y = (1 + x + x )Data source:
• Sugeno and Yasukawa, Tran. on Fuzzy SystemsData size:
• Total: 50, data set A: 25, data set B: 25Two dummy input variables: x3 and x4
-1.52
-21
2
Chapter 12: ANFIS
38
Input Selection Based on RCInput Selection Based on RC
x1x1(.57)(.57)
x2x2(.74)(.74)
x3x3(1.20)(1.20)
x4x4(1.47)(1.47)
x1, x2x1, x2(.33)(.33)
x1, x3x1, x3(1.18)(1.18)
x1, x4x1, x4(1.10)(1.10)
x1, x2, x3x1, x2, x3(21.49)(21.49)
x1, x2, x4x1, x2, x4(18.94)(18.94)
Level 1
Level 2
Level 3
Chapter 12: ANFIS
39
Gas Furnace ModelingGas Furnace Modeling
Data source: Box-Jenkins gas furnace data
Sampling time: 9 seconds
Gas furnaceGas furnace y(k)y(k)(CO(CO22 concentration)concentration)
u(k)u(k)(gas flow rate)(gas flow rate)
u(k)u(k)
y(k)y(k)
Chapter 12: ANFIS
40
Gas Furnace ModelingGas Furnace Modeling
10 potential inputs for ANFIS:• Group 1: y(k), y(k-1), y(k-2), y(k-3)• Group 2: u(k), u(k-1), u(k-2), u(k-3), u(k-4), u(k-5)
1 output: y(k+1)System model:
y(k+1) = F(y(k), ... y(k-3), u(k), ... u(k-5))Data size: 296
Chapter 12: ANFIS
41
Gas Furnace ModelingGas Furnace Modeling
Sequential forward selection with bipartite RC:• data set A: the first 148 entries• data set B: the second 148 entries
Level 1Level 1Level 2Level 2
Level 3Level 3
Selected inputs:Selected inputs:y(k) and u(ky(k) and u(k--2)2)
(RC = .186)(RC = .186)
(27.5 s)(27.5 s)
Chapter 12: ANFIS
42
Gas Furnace ModelingGas Furnace Modeling
Sequential forward selection with bipartite RC:• data set A: odd-indexed 148 entries• data set B: even-indexed 148 entries
Level 1Level 1Level 2Level 2
Level 3Level 3Level 4Level 4
Selected inputs:Selected inputs:y(k), y(ky(k), y(k--1), and u(k1), and u(k--3)3)
(RC = .107)(RC = .107)
(70.9 s)(70.9 s)
Chapter 12: ANFIS
43
Gas Furnace ModelingGas Furnace Modeling
Sequential forward selection with leave-one-out RC:
Selected inputs:Selected inputs:y(k), y(ky(k), y(k--1), and u(k1), and u(k--3)3)
(RC = .095)(RC = .095)
Level 1Level 1Level 2Level 2
Level 3Level 3Level 4Level 4
Chapter 12: ANFIS
44
Hair Dryer ModelingHair Dryer Modeling
Data source: Chap. 17 of System Identification., Ljung, 1984
u(k): binary random signal shifting between 3.41 and 6.41 Vsampling time: 0.08s
Hair dryerHair dryer y(k)y(k)(temperature)(temperature)
u(k)u(k)(voltage)(voltage)
y(k)
u(k)
Chapter 12: ANFIS
45
Hair Dryer ModelingHair Dryer Modeling
10 potential inputs for ANFIS:• Group 1: y(k), y(k-1), y(k-2), y(k-3)• Group 2: u(k), u(k-1), u(k-2), u(k-3), u(k-4), u(k-5)
1 output: y(k+1)System model:
y(k+1) = F(y(k), ... y(k-3), u(k), ... u(k-5))Data size: 600
• data set A: 300• data set B: 300
Chapter 12: ANFIS
46
Hair Dryer ModelingHair Dryer Modeling
Level 1Level 1Level 2Level 2
Level 3Level 3
Level 4Level 4
Sequential forward selection with bipartite RC:
Level 5Level 5
Selected inputs:Selected inputs:y(k), y(ky(k), y(k--1), u(k1), u(k--2) and u(k2) and u(k--3)3)
(RC = .002)(RC = .002)
(829 s)(829 s)
Chapter 12: ANFIS
47
Hair Dryer ModelingHair Dryer ModelingTo select 3 inputs out of ten:
Sequential forward selectionSequential forward selection(22.2 s)
Direct exhaustive searchDirect exhaustive search(54.7 s)
Chapter 12: ANFIS
48
Hair Dryer ModelingHair Dryer Modeling
ARX modeltraining RMSE = .114test RMSE = .072
ANFIS modeltraining RMSE = .038test RMSE = .044
Top Related