VI Всероссийская научно-техническая конференция...
200
Transcript of VI Всероссийская научно-техническая конференция...
-
,
- ( )
2004
2004
VI -
2
-
2004
-
004.032.26 (06) 32.818582
2004. VI -- 2004: . 2. .: , 2004. 200 .
, - , 2830 2004 VI - - 2004.
, - , - .
. . ,
ISBN 572620526X c - ( ), 2004
-
6
. . . 15 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 . . . 19 . . . . . . 26
. . . . . . . . . . . 26 . . 30 . . . . . . . . . . . . . . . . . . 34 . . . . . . . . . . . . . . . . . . 40
- . . . . . . . . . . . . . . . . 42 . . . . . . . . . . . . . . . . . . . . 43 . . . . . . . . . . . . . . 46 . . . . . . . . . . . 48 . . . . . . . . . . . . . . . 49 . . . . . . . . . . . . . . 49
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
. . . - 57 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 . . . . . . . . . . . 59
. . . . . . . . . . . . . . . . . . . . . . . . . . . . 60- . . . . . . 61
. . . . . . . . . . . 69 . . . . . . . . . . . . . . . . . . . . . . 70 . . . . . . . . . . . . . . . . . . . . . . . . . 72 . . . . . . . . . . . . . . . . . . . . . . . . 74
004.032.26 (06) 3
-
ISBN 572620526X
. . 77 . . . . . . . . . 88 -
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97 . . . . . . . . 97 . . . . . . . . . . 103
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
. . . - 111: - -
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112 . . . . . . . . 114 . . . . . . . . . . . . . . . . . 115
. . . . . . . . . . . . . . . . . . . . 115 -
. . . . . . . . . . . . . . . . . . . . . . . . 117- . . . . . . . . . . . . . . 120
. . . . . . . . . . . 122 . . . . . . . . . . . . . . . . . . . . . . . 125 . . . . . . . . . . . . . 127 . . . . . . . . . . . . . . . 128 . . . . . . 130 SARSA Q . . . . . . . . . . . . . . . . 133
. . . . . . . . . . . . . . . . . . 134 135 . 136
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
. . . - ? 140 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 . . . . . . . . . . . . . . . . . . . . . . 144
. . . . . . . . . . . . . . . . . . . . . . 146 . . . . . . . . . . . . . . . . . . . . . . 147 () . . . . . 149
4 004.032.26 (06)
-
ISBN 572620526X
- . . . . . . . . . . . . . . . . . . . . . . . . . . 151 . . . . . . . . . . . . . 153
: 158 : C
R2 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162 : . . . . 170
. . . . . . . . . . . . . . . . . . . . . 172 . . . . . . . . . . . . 175
: . . . . . . . . . 177 . . . . . . . . . 182
: . . . . . . . 187 H . . . . . . . . . . . . . . . . . . 191
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 194
004.032.26 (06) 5
-
1. ( ) , - -, 2830 2004 VI - - 2004.
- , - (. [15]). , - , -. -, , (. . , , , -), ( , - . .), , , , , , -.
, - , , , .
, - . , , , , , , - , .
2. - - 2003 - 1:
1. . . , . . . - .
1 1, 2 .
6 004.032.26 (06)
-
..
2. . . . - .
3. . . . : , -, .
4. . . . .5. . . . .6. . . . -
.7. . . . -
.8. . . . -
?
, 2 .
3. . . . . - - , - () 2002 [6]. - , . . , - - , . , , . . . . . - , -.
4. - . . ,
2 , , - - .
004.032.26 (06) 7
-
ISBN 572620526X
, 2001 - [7]. -, , - . , . . - , , -.
5. . . : ,, , - (), - . , , - . - , , - , - (). , - , . , - . - - 3. , - , .
3 (., , .. 2003 [8]), 2000 [8].
8 004.032.26 (06)
-
..
6. . . -, , - , , . (), -, . - , . . , , ( ). - , - ( , , -) , . . [10,11] - .
7. - (. . , . . . . ). - , , , - . , . . - , - (). , - , - () ( ) . , , - , , , ; , -, ; ;. - , - , .
8. , -, . . . ,
004.032.26 (06) 9
-
ISBN 572620526X
, - . . , - , , - . . - , - . , -. , . . . . [12], . . [13] 2001 .
9. - . . - - , , , , . . - 2002 [14] 2003 [15]. - . - , , - , - . , - -, - -. - - (reinforcementlearning), . - . - - -.
10. . . - ? - ,
10 004.032.26 (06)
-
..
2002 [16] 2003 [17]. - , - , . [17] , , . - : , - - , , . , - , , , , , , -, . , - .
, , ,
- , ResearchIndex ( - CiteSeer, . [19] -). , NEC , 2002 , - .
(, , , - . .) (PDF,PostScript .) , , (-, , / .), - -, - , . , , -
004.032.26 (06) 11
-
ISBN 572620526X
ResearchIndex - .
ResearchIndex, - [20], - [18].
, - - , , , , - , [15].
- , - , - , - , . , , ( ), (- .) . .
, , . . - [email protected] .
1. : - -
// III - -2001, 2326 2001 . / . .. . . .: - , 2001. 212 .
2. : - - // IV - -2002, 2325 2002 . / . .. . . 1. .: - , 2002. 164 .
3. : - - // IV - -2002, 2325 2002 . / . .. . . 2. .: - , 2002. 172 .
4. : - - // V -
12 004.032.26 (06)
-
..
-2003, 2931 2003 . / . .. . . 1. .: - , 2003. 188 .
5. : - - // V - -2003, 2931 2003 . / . .. . . 2. .: - , 2003. 180 .
6. (). , // .: . 1. .: -, 2002. . 66113.
7. . . -: // .: -. .: - , 2001. . 103141.
8. . . // .: . 2. .: - , 2003. . 2979.
9. : // 2- - - -2000, 1921 2000 .; III - - -2001, 2326 - 2001 . / . . . . . .: - , 2001. 104 .
10. . . - // .: . 1. .: - , 2002. . 114163.
11. . . // .: . 2. .: - , 2003. . 119179.
12. . ., . . - // .: -. .: - , 2001. . 1859.
13. . . // .: . .: - , 2001. . 60102.
14. . . - // .: -. 2. .: - , 2002. . 94120.
15. . . // .: -. 1. .: - , 2003. . 149187.
16. . . , , // .: . 2. .: - , 2002. . 121169.
004.032.26 (06) 13
-
ISBN 572620526X
17. . . // .: -. 1. .: - , 2003. . 86148.
18. (Matlab, Fortran, C++ . .)URL: http://www.mathtools.net/
19. NEC Research Institute CiteSeer (also known as ResearchIndex) ScienticLiterature Digital Library.URL: http://citeseer.nj.nec.com/
20. The Archive arXiv.org e-Print archive Physics, Mathematics, Nonlinear Sciences,Computer Science.URL: http://arxiv.org/
, . .
E-mail: [email protected]
-
. . ,
1
, (). : - , - , , . - , . - , , -, . . . -, , .
1 , 030100323
004.032.26 (06) 15
-
ISBN 572620526X
A. ZHDANOVInstitute for System Programming, Russian Academy of Sciences, Moscow
ABOUT A TECHNIQUE OF AUTONOMOUS ADAPTIVE CONTROL
AbstractA bionic-based approach to control system design is suggested named as au-tonomous adaptive control (AAC) technique. Some basic concepts of this ap-proach are considered. A principle of operation and structure of control systemare derived from general nervous system properties such as discrete structureand functioning, initial knowledge uncertainty as well as necessity to combinecontrol and learning processes within the same time period. Specic decisionsare proposed, which allow to make real-world control systems. These decisionsinclude techniques to build formal neurons, pattern generation and recognitionsubsystems, knowledge base, decision making and emotion subsystems. Someapplication problems are described solved by means of AAC technique. TheAAC approach is compared with alternative articial intelligence research ar-eas. It is demonstrated that the approach suggested could be regarded as somenew research area and could be named as autonomous articial intelligencesystems.
-. . , 1011 -, 1014 , 0.1 , , - - , , - . , , - . ,
16 004.032.26 (06)
-
..
, , , - , , . ., - . , -, , , , , - , , .
:1.
.2. , -
, , , , .
1 , : , 1011 -
, ;
- , , , ;
, , , ;
- , , - , , , - . , - - . - .
004.032.26 (06) 17
-
ISBN 572620526X
2 . - , - . - , , . , , . . , , . , , . -, , . , , . , . . - , . , - . , , - , , . ., , - .
, - . , -. - . . , , , , , , . .
, -, , . .
18 004.032.26 (06)
-
..
- - , .
. - - , . [1]. , - , . [2], . . [3],. . [4], . [5] , , - . -, , - .
, , , - , . . 1943 [6] . 1953 [7] ( - ()) , . , - , -, , . , , -, - , .
, - () [830].
004.032.26 (06) 19
-
ISBN 572620526X
() - , (), - , .
, - .
1. , -, , . . , , () ().
= , = .
2. , ( , , , ) ( , , ). , , .
3. - - , , , - , . . , , .
4. - , , -. , -.
, :
20 004.032.26 (06)
-
..
; .
, .
(.1), , , ( ). - , . , ( ) ( ).
. 1.
, h, -
h d i a h . . . ,
004.032.26 (06) 21
-
ISBN 572620526X
(), , . a , h - , d , i . . , -.
, - . . - v1, v2, . . . , vi, . . . , vn (, n ) . ,, - va(t1),vb(t2), . . . ,vc(tk), . . . ,vd(t), t - . , . . 1, - vc(tk) , i, b / p. i , - , b , , , p ( i b ). , i, b p, vc(tk).
, , - - - va(t1),vb(t2), . . . ,vc(tk), . . . ,vd(t). . 2.
, : ,
(, ) - ,
,
22 004.032.26 (06)
-
..
. 2. -
, - ,
. -
, . , - , , - , , . , (, . .) - , . , , , St. , St - . ( ) [21].
004.032.26 (06) 23
-
ISBN 572620526X
. St, - , . , St - ( - ).
, , Ok, - t , Otk = 1 ( - Ok t, Otk = 0). , -, - T , T tks, T < T . va(tks),vb(tks+1), . . . ,vc(tks+T ), . . . ,vd(tks+T ), , i (. . 1), , , , , St , -. -. , v(t), , b, . . , (, , - ). Yl. :
i) a,ii) l,iii) b.
, , - , -
.,
24 004.032.26 (06)
-
..
. 3. , -, ,
(. 3), ( a), - ( m) , , ( c).
, , , ,
, ,
St [10, 11].
, , , - , . . - , , [13].
, -, ,
004.032.26 (06) 25
-
ISBN 572620526X
, - , . . . , -. .
- .
, , - , .
, , .
. , , - , - .
, , . . .
, , .
() . - (, - . .). , - . - , . . . - . - .
26 004.032.26 (06)
-
..
. 4.
/ - , . , , - , - . , -, .
., - , , - , . . . - [12, 14, 15, 17, 18, 23, 25, 29]. (. 4) .
004.032.26 (06) 27
-
ISBN 572620526X
nw t
Xt = (x1, x2, . . . , xi, . . . , xm)
Stw. t + 1 Ot+1w St+1wk
Ot+1w = Stw &[(btw & l
tw) Otw
] St+1wk = b
tw & l
tw & g
tw.
btw b btw = 1, hw/m > p(N t) btw = 0 . hw xi Xt , 1 t;N t btw = 1 t = 0 t; p(N) . N = 0, 1, . . . p(0) = pmax, p 6100% p() = pmin, pmin < pmax. , p(M) = pM ,pmin < pM < pmax , M . pmin, pM , pmax M . ltw lw t 0 1 : ltw = 0, N t < M ltw = 1 .
T ,
(btw & ltw) = 1
(. c . 4) , Ot+1w 1, Stw = 1 , Ot+1w 0. gtw : gtw = 0, Zt < L gtw = 1 , Zt , c . L .
Xt, btw = 1, ( N -). M , N M , Xt -. , N M , ( ). N t = M nw, , nw Ow , t. N 0 M nw nw , ,
28 004.032.26 (06)
-
..
Ow. , ( ; - ). (Ot+1w = 1) (Ot+1w = 0). .
Ot+1w = 1 Stw = 1 , .
. 5. , Di , i
, -, Rw, lw gw, Tw. - . (. , . 6), - . , , p(N). ( p(N)) , ,
004.032.26 (06) 29
-
ISBN 572620526X
, .
. 5 , - i , i , , -, , , , Rw . , , - yi = 1 Ti, , i. - i i . , .
- , - [17, 25]. , .
, , ( - . 6), . - , . , - - .
- , ,
, ,
(. [9, 11, 12, 1417, 20]). - , . , p(N).
30 004.032.26 (06)
-
..
. 6. , , - -
-, . 4. , , p(N) - - . .
Ot1w &Yt1k & O
tv, (1)
Ot1w , t 1 ( 004.032.26 (06) 31
-
ISBN 572620526X
), Y t1k , t 1, Otv , t ( ). - . , , . , . 7( ).
O wt 1
Y kt 1
Y kt 1
n wY vt
. 7. , - Ot1w & Y t1k &Otv
nw m , . . (1), Ot1w ,Y t1k , Otv 1, . , (1), p(N). p(N) m 1, : Otv = 1 Ot1w = 1. :
(1) ;
, - (1).
() (. 8), Ot1w ,Y t1k , Otv . , . 7.
32 004.032.26 (06)
-
..
O tw1
O
n w
Y t 1
. 8. ()
, . t Ot1w , Y t1k , Otv Otw, Y tk ,Ot+1v , . t - Otw. Ot+1v . -, , Y tk . . Y tk . -, ., Ot+1v , - , . - , , , .
, , , . .
004.032.26 (06) 33
-
ISBN 572620526X
- , - . . - .
, , , ( ( III) [18, 25]). , .
- . , , , - . , . .
, - .
, - : - , , , , - [21]. -.
34 004.032.26 (06)
-
..
A. . - . - , , , . : , , - . , .
, , , , , . - - - . , - , S(t)., -
S ={s1, s2, . . . , sz, . . . , sZ
},
, , s1 = 5, s2 = 4, . . . , sZ = 5. S(t) = sz S pz,j - Oj . - p1,j = 5,pp2,j = 4, . . . , pZ,j = 5. , - , , 1, . . Oj - t, Oj(t) = 1, , Oj(t) = 0. pz,j - . , S(t), , , , - S. , .
- S(t). - , - . - ,
004.032.26 (06) 35
-
ISBN 572620526X
. - ( , , ), , . , , , -, , . , , - , , . , - , -, .
- . - , , , .
- . , , , , . . ,, - - , , - , - , .
36 004.032.26 (06)
-
..
B. - . Oj . - pz,j - S(t) , Oj(t) = 1, . S(t) , , ,, . , pz,j Oj S(t) , , . . Oj(t) = 1. , - , .
- S(t), . - , - , . . , . - - St , .
C. - . , -. , . - , , -, , . B(t).
B(t)) S(t), , , . . , ( ) -
004.032.26 (06) 37
-
ISBN 572620526X
. , :
B ={b1, b2, . . . , bh, . . . , bH
},
b1 = , b2= , b3 = , b4 = , b5 = , b6 = , b7 = , b8 = - . t B(t) = bh B.
S B, : S B , S(t) B(t). S , [Smin, Smax]. - Smin b1, Smax b2, B. . , - [Smin1, Smax1], [Smin2, Smax2], Smin2 < Smin1 Smax2 < Smax1. pz,j - , , b= , . Smin, - : , , , .
D. - . , - , [11]. . - - , . , , . . , , , - . - , - ,
38 004.032.26 (06)
-
..
, , - . , . . , - - , , . , , .
E. -. - . , . , , . k(t) , - . k(t) = f(S(t), dS/dt) - . Smin(t), (dS/dt)min, . ., , . - , , . f() , , -. , f() , , . , -, .
F. ., . -
004.032.26 (06) 39
-
ISBN 572620526X
, , . - , , , . - , , , . . , , - [31], - .
, , - (. 9). , . ,
, ,
, -, . . - . :
- - ;
; -
, .
40 004.032.26 (06)
-
..
. 9. , - ()
- . , .
1. 8, - [34, 37].
2. (AdCAS) [32].
3. ( ) [16, 19, 38].
4. ( -) [19].
004.032.26 (06) 41
-
ISBN 572620526X
- .
, . .
, , - , - , - . , , - - . .
, , . - , , . -, , , , . , , - , , . - , , , , , . - .
, , , -
42 004.032.26 (06)
-
..
. - , , . -, - , , - , .
, - , - - . - ().
- , . -. ( ) - . . , ,, , .
, , , - , . , , - (), . ( ) () . , , -. , :
( ) ;
004.032.26 (06) 43
-
ISBN 572620526X
, , ;
- .
, , - , . - , , . : () - , (, ) , , . - , , , - , , .
, , . . . , . , , .
- . -. . . - , - . - .
- , ,
44 004.032.26 (06)
-
..
, , . , .
, - [35], . - -, , - , . .
. , , , , , .
, :
; , , , -
, -, , . . , ;
; ; ; ,
-.
, . , .
004.032.26 (06) 45
-
ISBN 572620526X
, , , (). , , , - (. , - ), . . ,
.
(, - , ). - , , , . - () [36]. , , , .
, , , , , - , . - , .
, - , -, .
, , -, , , , -, - - -. , , -
46 004.032.26 (06)
-
..
, -, - .
, , , , - - , -. , - - , - , - . , , , , - , -, , , . , , , - .
, , , . , -, .
, - -, . - .
- , , , - .
:
- ;
; 004.032.26 (06) 47
-
ISBN 572620526X
, , , .
, .
(fuzzy logic), . . , - . (. . ) , , , , . - , , , . , : , , . - , , . , -, , , . .
- - , . . . - , [24].
, , ,, -, - .
, , . .
48 004.032.26 (06)
-
..
(reinforcement learning) - , , , , , -. , , , - , -, , . , .
, , - . , , -, , , . - .
, . - , (. . 9). - , , -, , , , - . , , - , . , - .
004.032.26 (06) 49
-
ISBN 572620526X
1. , , - , , . (. 9) . . , , . , .
2. - , , , , -. , - , . . , .
3. , , . . - , , . , , . , , -, - , .
4. - , , , , .
50 004.032.26 (06)
-
..
, . . - , , - , - (, , , - ), , - (, ). - , , - , , , - . , , - . . , , , . - , . , - . . - , . . - .
, - , , . - , . , , , - - , , , , -, , , , ,
004.032.26 (06) 51
-
ISBN 572620526X
, , , . . ( , , ). , - . -, - , , , 14 . -, . , - , , , -.
, - - . -, ? - - . - . - , - , , , ? - (, - ), . - .
, , - - (), , , , -. - () [13]. - , , , - .
52 004.032.26 (06)
-
..
, , , - , , .
- , - (). - , , ( 104 105) , - , , -. . , , , -. .
, - (), [13].
1. . : . . 2- . ., 1968.2. Cannon W. B. Bodily changes in pain, hunger, fear and rage. Appleon-Century,
1929.3. . . // . -
(, 2428. 1968 ). . - . .: , 1970.
4. . ., . . // .: . , 1971.
004.032.26 (06) 53
-
ISBN 572620526X
5. Mesarovic M. D. Conceptual basic for a mathematical theory of general systems// Kybernetes, 1, 1972.
6. McCulloch W. S., Pitts W. A. logical calculus of the ideas immanent in nervous activity// Bull. Math. Biophys. 1943, 5 [. .: ., . , // . . .: , 1956]
7. . . .: , 1965.8. . . . pp
H . ., 1991.9. Zhdanov A. A. Application of pattern recognition procedure to the acquisition and
use of data in control // Pattern Recognition and Image Analysis. v. 2, No. 2, 1992.10. Zhdanov A. A. A principle of pattern formation and recognition // Pattern Recognition
and Image Analysis. v. 2, No. 3, 1992.11. . .
// . . - . . 2. ., 1996.
12. . . - // . . . . 3. ., 1997.
13. . . // . .. . .: ,1997.
14. Zhdanov A. A. About an autonomous adaptive control methodology // ISIC/CIRA/(ISAS'98), NIST, Gaithersburg, Maryland. September 1417, 1998.
15. Zhdanov A. A. The mathematical models of neuron and neural network in autonomousadaptive control methodology // WCCI'98(IJCNN'98), IEEE World Congress onComputational Intelligence, Anchorage, Alaska, May 49, 1998.
16. . ., . ., . . // . . . . . 19. .: , 1998.
17. . ., . . - // .. . -. 99,. 1. .: , 1999.
18. . ., . . // . . . -. -99, . 1. .: , 1999.
54 004.032.26 (06)
-
..
19. . ., . . // . . . -. - 99, . 3. .: , 1999.
20. . ., . . - // . . V - (99). ., 1719 1999.
21. Zhdanov A. A., Vinokurov A. N. Emotions simulation in methodology of autonomousadaptive control // 14th IEEE International Symposium on Intelligent Control/Intelligent Systems and Semiotics ISIC/ISAS'99. Special session Emotions andIntelligent Systems. September 1517, 1999. Cambridge, Massachusetts, USA.Paper No. 99002I6.
22. . . // , 2. - . ., 1996. . 171206.
23. . . // - . . 1999, 5. . 127134.
24. . ., . . // - . . . . . .: - , 2002. . 119135.
25. Zhdanov A. A., Ryadovikov A. V. Neuron models in the autonomous adaptive controlmethod // Optical Memory and Neural Network. 2002. v. 9, No. 2. pp. 115-132.
26. . ., . . - // . - . 2. .: , 2001. . 141179.
27. Zhdanov A. A., Zemskikh L. V. The evolutionary growth of neural networks for theautonomous adaptive control system // The 5th World Multiconference on Systemics,Cybernetics and Informatics (SCI 2001) and the 7th International Conference onInformation Systems Analysis and Synthesis (ISAS 2001), Orlando, USA, July 2225, 2001. pp. 404409.
28. . ., . . // : , . 2, 2001.
29. . ., . . - // . . . . . 20. .: . 1999. . 111146.
004.032.26 (06) 55
-
ISBN 572620526X
30. . . // . 2000, 1, . 5864; 2, . 1416.
31. / . . . . - 2-. . .:, 2003. c. 267.
32. URL: http://www.ispras.ru/zhdanov/33. . . . .: , 1991.34. . ., . . . . -
// - (IMS'2002), . . 115118.
35. Hopeld J. J. Neural networks and physical systems with emergent collectivecomputational abilities // Proc. of the National Academy of Science. 1982. 79. pp. 25542558.
36. Bernold T. Expert systems and knowledge engineering. Elsevier, SciencePublishers, 1986.
37. . ., . ., . . - // . 2004. 1 ( ).
38. . ., . ., . . - // , 2004 ( -).
, -, ,. : , , . 60 .
-
. . -
E-mail: [email protected]
- ( , ), . , - . , - , , . .
LEV A. STANKEVICHSaint-Petersburg State Polytechnical University
E-mail: [email protected]
NEUROLOGICAL MEANS FOR CONTROL SYSTEMS OFINTELLECTUAL ROBOTS
AbstractUsage possibilities are discussed for neurological control means (basic modules,cognitive and actuator structures) as applied to control systems of intellectualrobots. These means are suggested to be able to learn and to form a rationalbehavior. It is shown control systems for the modern intellectual robots must besimilar functionally and structurally to the human nervous system. Such systemscould be developed using neurological means, sensor fusion and intelligent con-trol techniques as well as multiagent technology. An example of developmentand application of the neurological means with reference to control system forhumanoid robot is considered.
004.032.26 (06) 57
-
ISBN 572620526X
, - [1, 2]. . - , . - : -, - -. , , , , - .
, - , , . - , - , .
- , , [3]. - - , ( ) . -, , , .
-, . - - [4] [5, 6], , .
-. -
58 004.032.26 (06)
-
..
, [7], , [8]. - - .
- , - , . - . , , [9].
A 28- - [10]. .
- , -. , - .
. - , , . - , . , - , -, . . , , [11].
004.032.26 (06) 59
-
ISBN 572620526X
. , -
- , - . , [1, 2]:
, ;
, - ;
, - - .
, - , - .
. , - , - . - , .
. [12]. (GIG Generality InformationGranulation) . GIG , - (, , ) - .
60 004.032.26 (06)
-
..
, - , , -. Oc ,. .:
Oc = ins_g(G1, . . . , Gi, . . . , GN ),Gi = has_a(A1, . . . , Aj , . . . , AM ),Gi = has_a(A1, . . . , Aj , . . . , AM ),Aj = has_v(V1, . . . , Vq, . . . , VQ),
(1)
Aj j Gi; Vq q Aj ; ins_g ; has_a has_v . : - , , . ., , . . , , , . . , . . - ( , , ) (- , , .). , .
-
- (). , . (SAM) (MAM) .
AM, SAM AM :
AM = {X,Y, S, U, F},SAM = {Px, Py, AM},
MAM = {Px, Py, {AMt, t = 1, . . . , T}}.(2)
, , : X,Y (); S,U -
004.032.26 (06) 61
-
ISBN 572620526X
, ; F , ; Px, Py ; AMt SAM t ; T t, AMt, M .
- , , .
(C Cell) , - -. :
Mc = {X, Wc,Hx, Sc,Hy, BF, y}, (3) X = {x0, . . . , xn} ; Wc = {w0, . . . , wv} (m > n); Hx = {hx1, . . . , hxq} - , ; Hy = {hy1, . . . , hye} - , -; Sc = (sc1, . . . , scr) , ; BF = (bf1, . . . , bfk) ; y .
-- (.1):
C = ins_f(IG, A(a1, . . . , aN , Sc,Wc), DG, Tc), (4)
IG , Hx;A() , ai, i =1, . . . , N , - Hy Sc - Wc; DG , y; Tc , Sc Wc - X y, y = F (X);ins_f - .
(- ), n . -
62 004.032.26 (06)
-
..
X Hx Hy
IG
BF
A(a1,,aN;S,W) ID
Y
Yp
. 1.
, , - . - , . - X y :
X y : (Wx, X) (Be, Hx) (Wh, Sc, BFc) (Af, Hy) (wy, y),(5)
X - : (x1, . . . , xn) - ( ), x0 ; (Wx, X) - X Wx; (Be, Hx) (BeBefore), Hx ( : ; (Wh, Sc, BFc) , Wh Sc BFc; (Af, Hy) (Af After), - Hy ( ).(wy, y) .
, - , Wc = (Wx,Wh, wy).
004.032.26 (06) 63
-
ISBN 572620526X
BFc. , . , . BFc : ; - , : , - , - .
(N Nucleus) (.2). - (3), X = (X1 X2 XM ) , ; WN , Hx, Hy - , SN , , BF , - ; y Y .
, , , , . .
N = ins_f(IGN , AN , DGN , TN ). (6) -
, ( ). - - . , , , . .
Y = F (X) yj = fj(x1, . . . , xN ); j = 1, . . . , N. (7) ,
. - , . - , .
64 004.032.26 (06)
-
..
C1
Cm
y1
y
X Y
NN1
Nm
X1
Xk
LN
N1C1
NmCm
Y1
Ym
Px Py
N1
Nm
C
Px1
Pz1
N1
Nk
N
Pxq
Pzq
C
1 (LN)
q (LN)
Px1
Py
Ps
. 2.
004.032.26 (06) 65
-
ISBN 572620526X
(LN Local Network) . - (.2).
- :
M = {PX(X, CX),WLN , SLN , PY (Y, CY )}, (8) X CX PX , . . - , ( ) PX ; WLN , ; SLN , ; Y CY PY , . . - , (- ) PY .
:
LN = (C1, . . . , CQ; N1, . . . , NK , LLn), (9) Ci Nj i j , ;LLn (Learner), WLN .
( ) - . - . - .
(DN Distributed Network) -. (), () (.2). - , . , - .
: , ,
66 004.032.26 (06)
-
..
. :
PX/ti Fk1(PX , PZ1) . . . Fq(PX , PZq) PY/t1 (OY ) PS/t2 PX/t2,
(10)
t1 t2 , 1- 2- ; Fki i ; PS - ; PX , PY , PZ , . , wi,i+1. -, PZ .
- , . . , -. - . , , - , .
, . - - . - , ,, .
, , -. - . - - .
004.032.26 (06) 67
-
ISBN 572620526X
, . .
. - (Reinforcement learning RL) [13]. RL . , , . . , . : , RL . , - , RL. - RL . , c .
, - , . - .
, -, . - - . . - , . - ( ). - , .
- - , .
, , - () -
68 004.032.26 (06)
-
..
. , -
. .
, . - . , - - .
, - , - . , , , , .
, , -, - - . - - , - .
. , - ( - ). - .
, -, , -, - - , -
004.032.26 (06) 69
-
ISBN 572620526X
(.3).
X1
X2
Xn
Y1
Y2
Ym
1(X1)
2(X1)
1(Xn)
2(Xn)
k(Xn)
1(Y1)
2(Y1)
1(Ym)
2(Ym)
r(Ym)
. 3.
: , - .
(.4).
j(xi) j i , i(y) i -. , , ,
m(yr) =n
i=1
ki
j=1
j(xi)wl+m,p+j . (11)
: n , ; wij - ; ki i ; l , r ; p , i .
70 004.032.26 (06)
-
..
1(Y1)V V VV
ws11(Ym)ws2 wsp wsl wsh
w11 w12 w1p w1l w1hw1l+1
w21 w22 w2p w2l w2hw2l+12(Y1)
wt1 wt2 wtp wtl wthwtl+1t(Y1)
wk1r(Ym)wk2 wkp wkl wkh
ws+112(Ym)ws+12 ws+1p ws+1l ws+1h
wsl+1
ws+1l+1
wkl+1
1(X1)
2(X1)
p(X1)
1(Xn)
2(Xn)
p(Xn)
. 4.
004.032.26 (06) 71
-
ISBN 572620526X
. - - , .
: ; ; . , , .
, . , .
(.5).
ki(Xi) ki i , (y) ( ). , ,
(y) =m1...mn
k1...kn
(wk1...kn
n
i=1
ki(Xi)
). (12)
: n , ; wi,...j ; mi i.
- . , .
72 004.032.26 (06)
-
..
1(X1)
2(X1)
1(Xn)
2(Xn)
k(Xn)
p(X1)
w1...1
w2...1
wp...1
w1...2
w2...2
wp...2
w1...k
w2...k
wp...k
V
(Y)
. 5.
004.032.26 (06) 73
-
ISBN 572620526X
: ; ; -
.
, .
.6.
, - . , -, . , - ( -).
j(xi) j i . , ,
Y = sign (wk)m
i=1
win
j=1
i j(Xj). (13)
: n , ; m - ; wi i ; k .
- - .
74 004.032.26 (06)
-
..
X1
Xn
V Y
(X1, l11, c11, h11)
(Xn, l1n, c1n, h1n)
w1
(X1, lm1, cm1, hm1)
(Xn, lmn, cmn, hmn)
wm
. 6.
: ; -
; ; -
.
- -.
004.032.26 (06) 75
-
ISBN 572620526X
, - . - - - .
. 1 : ; ; .,
, - .
1. ,
-
2p n t1 + 3p n t2
p 2n n t1 +p 2n (n + 1) t2
e n t1+
e
n t2 -
p n
k=1
mk p n
k=1
mk 3 e n
-
- - -
- - -
: n ; mk k ; p ; e ; t1 ; t2 / .
PC Pentium 1 - : n: 412; mk:57; p: 812; e: 2002000; t1 = 10t2.
76 004.032.26 (06)
-
..
(cognition) - [14]. - . -, , . . - . - , - . - . , . . , . , , , .
, - , , . , -. , . - , . - , , . , - , . , - , - .
, - [1, 2]. -
004.032.26 (06) 77
-
ISBN 572620526X
-
-
-
-
( - )
. 7.
. - (.7). - () . , , . , . , - (), - . , ,
78 004.032.26 (06)
-
..
.
, - , : , ( - ) - (.7). - . , :
{E S CSP CSB B CSB CSC AS A R E},
E , S , CSP , CSB --, B , CSB --, CSC -,AS , A , R , E .
, -. , - ( ). - . - - . - , . .
- , - . , [15]. -, , - . ,
004.032.26 (06) 79
-
ISBN 572620526X
, - . ( ), -, . , , , . . - . - , - . . , - , . , . -, -, - . - , ( ) -. .
- [2]. , - , (.8).
(N Nucleuses), . . (LN Local Networks) (PN Projective Networks), - . - , - , - (ca cognitive agent), ,
80 004.032.26 (06)
-
..
ca1
cai
caM
CN
-
-
PN
LN
N
. 8.
004.032.26 (06) 81
-
ISBN 572620526X
, , - . - (CN Cross Networks). - (- , ).
-, [3]. , ATLANTIS[16], . - -. . - [17] SOAR [18]. . , . - , , .
, . - . , , -. , , , -, . - BDI (Belief-Desire-Intention) [19]. , , , BDI .
- . - . , - , .
82 004.032.26 (06)
-
..
: -, - (RoboCup), -, - [20]. - (, ). - : - , BDI - , BDI , .
- , , - , . -, , -, : , - , - .
(.9), , - (CF), - . - : , -.
, , - (G) , , . (WS) - (IS), (WM) (IM). -, , -. .
004.032.26 (06) 83
-
ISBN 572620526X
WM
IM
IP
WS, IS
WS
IS
CF
CF
CF
CF
CF
CF
CF
CF
CF
Pr
CM
-Rc
Rc
Pr
Dc
. 9.
() - , . , , , .
, , - , - . (Rc), - ( , ) (CM). -
84 004.032.26 (06)
-
..
, . , , , , , .
, , . (Pr), (IP). ( - ). - .
- -, , , - . - , . -, . ( ).
, - - (WS, IS) ( ), , , . , ( ).
, (IS) , , , - , . - ( ),
004.032.26 (06) 85
-
ISBN 572620526X
, .
, , - . - (.2), . - , - . .
-, - - (.10). - (InOut) - n X , m Y (Look). (Process Unit) - (fuz), , , - (Inference), (Memory), (dfuz). (LearningUnit) (fuz), (Current Weight), (wupdate), (Reg) (Pattern). :
fuz(xi, q(xi)) , - xi q ;
inference(ln, wi, qxi; i = 1, . . . , I; q = 1, . . . , K) -, yj q xi, wi ; I, K - ; ln , ;
dfuz(qyj , yj ; q = 1, . . . , K; j = 1, . . . ,M) , - yj , - ;
w-update(wp, y, y, p) ; wp - ; qy , y
86 004.032.26 (06)
-
..
y ( );p ; - p + 1 .
Out
Y
Look
w pi ( )1
In
X
Process Unit
hx1..
.
hxN
hy1.
.
hyMMemory W
(n) (m)
hx1
hxnhy1
w1 wL
Pattern
XpY*p
Learning Unit
h*y1.
h*yM
hy1.
..
.
hyM
wi(p) Current weight
(m)
hyi
wi(p+1)
p
(p+1)-1
wi
h*yi
p
(w-update)
RegReg wi(p+1)wi(p)
Y*p
Xp
Yp
(Inference)
dfuzfuz
fuz fuz
. 10.
fuz, inference, dfuz, w-update x1, x2 y. - . (q = 1, 2, 3
004.032.26 (06) 87
-
ISBN 572620526X
):
fuz : qx = ((x lq)c1ql lq < x < mq) ((rq x)c1qr mq < x < rq)(0 else); cql = mq lq; cqr = rq mq,
(14)
inference : qy = i=9i=1(wqi q,r=3q,r=1 (qx1, rx2)), (15)dfuz : yq = 0.5(yqn + y
qr) = 0.5(
qy/cqn +
qy/cqr);
y = Kq=1qyy
q/Kq=1qy,
(16)
w-update : wp+1 = wp(p/p + 1) + wp(y/y)/(p + 1);p = 1, . . . , np.
(17)
: lq , mq, rq - , - ; (min), q ; (max), 9 - .
- , . - - [5]. , .
- - , - . , - , .
88 004.032.26 (06)
-
..
, -, ( -, --, -). . - , - 19661972 [4]. , . , , -. . , -3, , - , [7]. , , - .
- - . -, ( , , .), - . , (, , , , .), - . (softrobots) , -, . -, , (RoboCup).
- (), -, - [6]. . , -, . , XXI , , -
004.032.26 (06) 89
-
ISBN 572620526X
, , . - [5, 6]. - , , , -.
- , , - ., , . - - . , .
- , , 5 : -, , , . - [7]. , . -. , - . . . -. . -
90 004.032.26 (06)
-
..
- .
, -, , - . , , , - , . , , - .
,
,
. 11.
, - , - (.11). , -
004.032.26 (06) 91
-
ISBN 572620526X
, , , ., - () () -. ( -). ( , -, .) . - ( , , , ) . - , , .
- -. - , . . - ( -). -, , . , .
- , , - . , - 1997 , 100 - . , -
92 004.032.26 (06)
-
..
.
- . , - - , , -, .
- . () :Honda P3, Asimo; Sony SDR-3 4; Fujtsu HOAP-1; JSC Laboratory H-7. - . , Honda , 1983 . 3 Asimo 3 . 10 , (P3 180 ., Asimo 120 .). Sony (SDR-4 53 .). (HRI) Waseda - . SDR-4, , - , - . , , .
: -, , , - . , - , , . . 1997 RoboCup, - , 2050 , - -.
004.032.26 (06) 93
-
ISBN 572620526X
. - - .
- ( 28) , - . . (.11), , , -, , ., ( ) . - - ( ) . - (, , ) (, ). ( , , , , ) - . ( ), - , . . , .
- . . -, . - (), [9].
, - . -
94 004.032.26 (06)
-
..
-
. 12.
, (.12). , . - - -. , . - , , . , . , . - , -. , , - - .
, - .
004.032.26 (06) 95
-
ISBN 572620526X
. - -, , ., , ,, - . -, , , , . , . - , . -. , .
, - . , , , , , , - , - . - , -, - , , - . , , , -, . - -, - , .
- - , -
96 004.032.26 (06)
-
..
, . - , - -. - , .
, - , . . . - ( ) ( ).
- , - , .
( ) -, . - 2001 .
2003 -02 28 , 123 53 . () - . ( ) , - . , - . , , . . -
004.032.26 (06) 97
-
ISBN 572620526X
. -, .
, , - , - - ( ,, .). , , -, (. 13).
. 13.
, . 13, -: , , , - , , , , - , , , -, .
98 004.032.26 (06)
-
..
, , , , - : , .
- . .
( ) - ( - - ) :
(, ,, );
;
( ); ; ( -
).
- ( , , ) :
; ; ; . -
- :
; -
(, , .);
004.032.26 (06) 99
-
ISBN 572620526X
;
( ); ( 10 ); ; -; -
; ; ( ,
. .).
- :
- ;
; ;
;
; ( 10 ); . -
: ; . -
: ; ; ; ;
100 004.032.26 (06)
-
..
(-, );
. -
- . , , - , . - , - . , - , , -, , . - , .
- - . .
- :
(); (); (). ,
, - :
; -
; ;
004.032.26 (06) 101
-
ISBN 572620526X
() - ;
;
; , -
; (, -
).
, , :
; ; ; ; ( -
RoboCup).
- (,, , ).
28 - :
- ( );
(-, , );
;
( - , );
102 004.032.26 (06)
-
..
( , );
- ( - );
.
. - ( ,, . .), . - , -. - (. 14). , - .
-, () , , .
, () :
-, ( );
--, - ( - );
004.032.26 (06) 103
-
ISBN 572620526X
-, ;
-, ;
--, -- ;
-, - ;
--, - .
M , . -, . .
, - , . .:
CA = {ca1, . . . , caM}; R = {r1, . . . , rM}, (ri 6= rj), (18)
cai i , ri i . , R, . .:
F = {R, (U1, . . . , UK)}; Ui R : U = {ri1, . . . , riK} (rq 6= rl), (19)
F , Ui i , .
CA R , - . , ( ).
: (CNP Contract Net Protocol), (NS Negotiation Strategy) . . - ,
104 004.032.26 (06)
-
..
(DSK Dynamical Social Knowledge), CNP NS [3]. : ContractFrame (CF ) DSKbase. CF :
CF = {goal(di), CA, alpha cut, deadline(dli),contractset(c1, . . . , cm), winners),
(20)
goal(di) ();CA , ; alphacut , CF ;deadline(dli) CF ; contractset(c1, . . . , cm) ; winners , CF .
, , ( ). - DSK. , - , (NS). - .
, - .
-. , , , . (, - .) .
. - , - , - , .
- .
004.032.26 (06) 105
-
ISBN 572620526X
(1) (7)
, ,
. 14.
, -. - . - - .
- . . - - . - ,
106 004.032.26 (06)
-
..
.
. , . . , , , . - . - . . , - : , - , , , .
- - -. - - , - .
COM Toolkit . COM (Component Object Model) ++. , -, - -. . - , - , -
004.032.26 (06) 107
-
ISBN 572620526X
. , .
- , , - . , , - . , - , , 22 . , , , - . - , , , .
- . , - , , , , (- , , .). - RoboCup Soccer, 1997 , - . - , RoboCup. , - .
- ( RoboCup Rescue), -
108 004.032.26 (06)
-
..
( - ).
1. .. // -
( XII , --, 1999), --, 1999.
2. .. - // IV ,2000 (, 1618 2000), ., 2000.
3. Weiss G. (Ed.). Multiagent Systems. A Modern Approach to Distributed ArticialIntelligence. The MIT Press, Cambridge, MA, London, UK, 1999.
4. . , // . .. . , , 1973.
5. URL: http://androidworld.com6. Tevatia G. and Schaal S. Inverse Kinematics for Humanoid Robots // In Proceedings
of IEEE Int. Conference on Robotics and Automation (ICRA 2000), 2000.7. .. . .: , 1985.8. .., ..
// 13- (1618 2002 .), - , , 2002.
9. .. // - (--, 2528 2002), --, 2002.
10. . , 21 2003, 40(515).11. Albus D. Brain Behavior and Robotics. BYTE Books, 1987.12. Zadeh L. Towards a Theory of Fuzzy Information Granulations its Centrally to
Human Reasoning and Fuzzy Logic // Fuzzy Set and Systems, vol. 90, 1997.13. Barto A. and Sutton R. Reinforcemenr Learning: An Introduction. The MIT Press,
1998.14. . . .: , 1967.15. . . 1. .: , 1987.16. Hayes-Roth B. An Architecture for Adaptive Intelligent Systems // Articial
Intelligence, vol.72, 1995, pp. 325365.
004.032.26 (06) 109
-
ISBN 572620526X
17. Anderson J. The Architecture of Cognition. Harvard University Press, Cambridge,MA, 1983.
18. Laird J. et al. Soar: An Architecturefor general intelligence // Articial Intelligence,33(1): 164, 1987.
19. Bratman M. Intention, Plans, and Practical Reasoning. Harvard University Press:Cambridge, MA, 1987.
20. . ., . . // , 3 (57), 2003.
, , - - , - - . , - , , -, , . 100 , 4 1 . - (RoboCup) . .
-
. . ALIFE,
, .E-mail: [email protected]
-
- . - - -, . - . - .
S. A. TEREKHOFFArticial Neural Network Laboratory (ALIFE),
Troitsk, the Moscow RegionE-mail: [email protected]
NEURO-DYNAMICAL PROGRAMMING FOR AUTONOMOUSAGENTS
AbstractThis lecture is devoted to multistage optimization methods in the control of com-plex dynamical systems. Statements of typical control tasks are discussed. Thekey technology neuro-dynamical programming method is based on the ap-proximation of cost-to-go function by articial neural networks. Examples showrelations between reinforced adaptation of autonomous agents and predictivecontrol problems.
004.032.26 (06) 111
-
ISBN 572620526X
: -
, -, - . - , , . - , . , -, . , , - , - . , . . .
1. . - . - real options.
2. (, ), , - . - . () - -.
3. . - - . -
112 004.032.26 (06)
-
..
(, - , ).
4. [1]. - . . - (. . ). , , .
5. . - . - . - .
6. . - . -, .
7. , , - , , ( , ), - .
- , . - : - , . -, , , , - , . -
004.032.26 (06) 113
-
ISBN 572620526X
, - . - - . [2] - - -, - , .
. - - - , . , . - - . - . - . - - (reinforcement learning) . .
() x = {s, a}, , - () . () F (x), - . - , . s(a), . , , - 1:
1 . . ., [35].
114 004.032.26 (06)
-
..
a?, ,
a? = arg maxaA
F[s(a), a
],
A , -
g[s(a), a
]= 0,
f[s(a), a
]6 0.
. , , , . . - - optimization-online.org. , . , - , . , .
, , - . -.
. , - , , - .
004.032.26 (06) 115
-
ISBN 572620526X
, , , - , . -, -. - , :
a?(t), , T
0
F[s?(t), a?(t)
]dt + Ft=T
[s?(T )
] max,
a? A ,
g[s(a(t)), a(t)
]= 0,
f[s(a(t)), a(t)
]6 0.
, - () . - [6] (, ).
, - . s(a(t)), , - . - , , .
- - (. . ). - .
, - .
116 004.032.26 (06)
-
..
- . , , - . 2.
- - ( -) . - , , [7].
- (MPC Model Predictive Control). MPC :
. -
, .
.
MPC - , . , , , , , - . , - , - , , -, , . , , , .
- - ,
2 - .
004.032.26 (06) 117
-
ISBN 572620526X
[8] 3.
t
. 1.
MPC : , -
. N (- ).
.
3 , , - .. : - // -. 1998. 1, 1. . 1124; - // . . 1998. 5 (432). . 69. . .
118 004.032.26 (06)
-
..
M - 4. - . - .
( N - ) . .
MPC :
, - . - .
, - , . , - ( , - , . .).
[9] MPC Pavilion Technologies (). - . 50 - .
MPC . , , -. , , , - .
. ,
4 M - N . .
004.032.26 (06) 119
-
ISBN 572620526X
.
-
:
- . , . , reinforcement . - (, - . .). - . .
- . . - . 5 [10].
, , . - - - [11]. - (., , [12]).
, MPC - ,
5, . - .
120 004.032.26 (06)
-
..
. . - , :
-;
, .
[10]. , - - , . .
F (x1, x2, . . . , xN ) =N
i=1
gi(xi) max,
xi > 0,N
i=1
xi = c.
g x. - F N ( ) c, fN (c). , f :
fN (c) = max06x6c
[gN (x) + fN1(c x)
],
, , fN1() - (N 1) , x (cx) .
f1(c) = g1(c).
, - , . - N .
004.032.26 (06) 121
-
ISBN 572620526X
- . , - , - . - , . - , , . , . - , .
x(t-1) x(t) x(t+1)
s(t-1) s(t) s(t+1)
a(t) r(t) y(t) a(t+1)
. 2.
, , , . .
122 004.032.26 (06)
-
..
, .
(, - ), - , ( ). . 2.
- 6. x(t), - . , - :
x(t) P [x(t)|x(t 1), a(t 1)]. - (, ).
y(t) :
y(t) P [y(t)|x(t), a(t)]. y(t) 7 (reward)
r(t) = E[R(t)|x(t), a(t)].
, , , . ,, .
. - s(t) :
s(t) = f[s(t 1), y(t), r(t), a(t)].
, - (s):
a(t) = [s(t)
].
6 , .
7 . . .
004.032.26 (06) 123
-
ISBN 572620526X
:
J(x, a) = E[
t
tr(t)] max .
-, , - . , , - , .
, - , . -, . , , - , .
( , ) , -, 8.
, , . - y(t) = x(t), , , ( - ). , + .
, -, - . , , ( -
8 , , .
124 004.032.26 (06)
-
..
), . - , ( -):
.
, , -.
, - , 9, - ( ). , :
P[s(t + 1) = s, r(t + 1) = r|s(t), a(t), s(t 1), a(t 1), . . . , s(0), a(0)] =
= P[s(t + 1) = s, r(t + 1) = r|s(t), a(t)]
. , - , .
, , (MDP Markov Decision Process). , - , .
.
T (s, a, s) = P[S (t + 1) = s|S (t) = s, A (t) = a].
9 , . . , t , - t, , t . , . . .
004.032.26 (06) 125
-
ISBN 572620526X
. 3. , .
s = f (s, a).
r (s, a, s) = E[R (t + 1) |S (t) = s,A (t) = a, S (t + 1) = s ]
10 . V (s) s . . -
, , , , s. , :
V (s) = E
[
k=0
kr (t + k + 1) |S (t) = s].
V (s) = max
V (s)
10 -, .
126 004.032.26 (06)
-
..
. - Q (s, a) , - s a,
Q (s, a) = E
[
k=0
kr (t + k + 1) |S(t) = s, A(t) = a].
, .
, :
Q (s, a) =
sT (s, a, s) [r (s, a, s) + V (s)].
:
V (s) = maxa
Q (s, a) = maxa
sT (s, a, s)
[r (s, a, s) + V (s)
].
:
Q (s, a) =
sT (s, a, s)
[r (s, a, s) + max
aQ (s, a)
].
- . - ( -). - , , , -.
, 11. - , , (. .
11, , - . ,
004.032.26 (06) 127
-
ISBN 572620526X
), , , , . .
. [13], - (-, , , . .). , . - . , , - , .
( ) (value iteration).
V j+1 (s) maxa
sT (s, a, s)
[r (s, a, s) + V j (s
)].
value iteration - , . , - (. . - ). - . , , -, , ?. , - . .
128 004.032.26 (06)
-
..
12. -
(policy evaluation). :
V j+1 (s)
sT (s, (s) , s)
[r (s, (s) , s) + V j (s
)].
s , s (s) . - , (policy iteration). , - , :
j+1 (s) arg maxa
sT (s, a, s)
[r (s, a, s) + V j (s
)].
. - - . - , . , , - 13
V = R + TV,
-
V 6 R + TV
.12 -
[14].13 -
, , . . .
004.032.26 (06) 129
-
ISBN 572620526X
- -. , , . .
, - . -, - , -, , -, . .
- . - , , . .
, .
(V (s) Q(s, a) - - , . , , , , - , - , .
, , , (). , - , - [15].
130 004.032.26 (06)
-
..
, . - value iteration V = g(V ) . - Vj+1 g(Vj) - .
. -, -
E(w) =12
[Vw(s) g(V )
]2,
w w + [g(V ) Vw(s)
]Vw(s)w
.
, , - [13]. -, . , - [16], valueiteration .
- , . , , (temporal difference methods, TD(). TD() [1].
(-) value iteration, , online- . - (). ,
004.032.26 (06) 131
-
ISBN 572620526X
, . .
-. , V , V , 2V . . - (eligibility trace).
1: TD() Require: V ; e 0
for all dos while do
s T [s, (s), s] r[s, (s)] + V (s) V (s)e(s) e(s) + 1V V + ee es s
end whileend for
(s) , . , - V , V . , , ( V ). .
132 004.032.26 (06)
-
..
- .
r[s, (s)] + Vw(s) Vw(s),
e e + Vw(s)w
,
w w + e., e -
w, ( - ). , TD() .
SARSA Q
, - V (s), - Q(s, a) . , Q , :
a = arg maxa
Q(s, a).
Q(s, a). - Q(s, a), . . ( on-policy).
Q[s(t), a(t)
] Q[s(t), a(t)]++
{r(t + 1) + Q
[s(t + 1), a(t + 1)
]Q[s(t), a(t)]}
.
SARSA ( s, a r sa). - , a(s),a(s) Q.
, Q (Qlearning) Q , . ,
004.032.26 (06) 133
-
ISBN 572620526X
, ( Q). - (off-policy). :
Q[s(t), a(t)
] Q[s(t), a(t)]++
{r(t + 1) + max
aQ
[s(t + 1), a
]Q[s(t), a(t)]}
.
:
SARSA Q , - -.
-
, - ( )
.
, (, , SARSA() Q()) - .
- - ( ), - , .
: - .
134 004.032.26 (06)
-
..
[17] - . - , 14 zt, Ft
{1, 0, 1}. - (long), , , - (short) . , . 15. , .
:
Ft = F[t;F1;
{zt, zt1, . . . ; yt, yt1, . . .
}].
, ytk - (, - , . .). , - T , :
PT =T
t=1
Rt = vT
t=1
{rft + Ft1
[rt rft
] Ft Ft1
}
.
v , , rt = zt zt1 -, f - (, ). , , , [18].
- . , ,
14 (, ) - , . () .
15 .
004.032.26 (06) 135
-
ISBN 572620526X
( ). - [17] .
S&P 500. 80 ( , ). - - 19501969 . 25- - 1970 1994 . 0.83 . 0.34 , (US Treasure Bills).
, , - , 1974 , 80- , 1984 1987.
- . .
. . , , (, ). , - , . , , , . , .
, - , -.
. -, ,
136 004.032.26 (06)
-
..
. [15].
, ( - ) (), ( -). , , (,, ).
, - . , - . -, , . - . .
, - - 16. , - , . , - . -, , -. , .
[15] - 50 70- , . TD( = 0) - , , .
16 !
004.032.26 (06) 137
-
ISBN 572620526X
. - 10 . , - . , , - .
- - , -, .
. . . . - .
1. Sutton R., Barto A. Reinforcement learning: An introduction. MIT Press, 1998.2. Wolpert D. H., Tumer K. An introduction to collective intelligence. NASA Ames
Research Center, 2000.URL: http://ic.arc.nasa.gov/ic/projects/coin_pubs.html
3. ., . - . .: , 1988. 440 .
4. ., ., . . .: , 1985.5. . . . : -
, 2001. 368 .6. . ., . ., . . -
. : - . . . , 2001.7. ., . . : -
, 2002.8. . .
// , 1998.
138 004.032.26 (06)
-
..
9. Piche S., Keeler J., Martin G., Boe G., Johnson D., Gerules M. Neural networkbased model predictive control // NIPS'1999. Vol. 12, Paper No. 1029.URL: http://nips.djvuzone.org/djvu/nips12/1029.djvu
10. . . .: , 1960.11. . . -
// - . 2, , , 2527 2002 . .:- , 2002.
12. . . // - - . 1, , , 2931 2003 . .: - , 2003.
13. Tsitsiklis J.N., Van Roy B. Feature-based methods for large scale dynamicprogramming // Machine Learning, 22:5994, 1996.
14. Sutton R. S. Open Theoretical Questions in Reinforcement Learning. AT&T.URL: www.cs.umass.edu/rich
15. Bertsekas D. P., Tsitsiklis J. N. Neuro-dynamic programming. Athena Scientic,Belmont, MA, 1996.
16. Kaelbling L. P., Littman M.L., Moore A. W. Reinforcement learning: A Survey //Journal of AI Research (JAIR), Vol. 4, 1996.URL: http://www.cs.brown.edu/people/lpk/rl-survey/rl-survey.html
17. Moody J., Saffell M. Reinforcement learning for trading // Advances in NeuralInformation Processing Systems, 11, M. S. Kearns, S. A. Solla and D. A. Cohn, Eds. MIT Press, Cambridge, MA, 1999.
18. . -. . , 2002. 567 .
C , - -, ALIFE(. , .). , , , , , . 1 50 .
-
. . , ,
E-mail: [email protected]
?
, , - , - . - , - .
N. MAKARENKOInstitute of Mathematics, Kazakhstan, AlmaAta
E-mail: [email protected]
HOW ARE TIME SERIES OBTAINED FROM GEOMETRY ANDTOPOLOGY OF SPACE PATTERNS?
Mathematical aspects of the analysis of extended dynamical systems
AbstractThe approaches applying computational, integral and stochastic geometry as wellas computational topology give possibility to transform a geometrical and topo-logical complexity of images into scalar time series appropriate for up-to-datenonlinear analysis. This lecture is a comprehensible introduction to contem-porary methods of analysis of spatiotemporal patterns, which are typical forextended dynamical systems.
140 004.032.26 (06)
-
. .
, , - . .
.
. , , - , - , , . . . , - , , () , . - : , . , . , , . n n - . , . , - . - . a la Takens. - , , , .
004.032.26 (06) 141
-
ISBN 572620526X
, . - , . , ( ) , snapshots, . - : , - -, , - . , , . - : - , , - -. . - - [1] :. . . , . , . .
. -, . , , ( -) . , , , - . , . , , - . , . - . , :
142 004.032.26 (06)
-
. .
. , , , . - . , ! , , - . , ;, . [2, 3]. ( -), , , [3], - , , ., , - , . . - , , - . , - , - .
, - . , , .
. . - .
004.032.26 (06) 143
-
ISBN 572620526X
, , . .
.
1 (1) . , . P (xp, yp) l : y = ax + b (. 1). ,
. 1.
l P .1 , ,
, (3), , - . , . . .
144 004.032.26 (06)
-
. .
, l , . . . - (xz, yz) , P , l; : yz = axp + b; xz = xp. , P l, yz > yp, , yz < yp. . : - q r. p, q, r ., S :
S =12[xq(yr yp) + xr(yp yq) + xp(yq yr)
].
, S = 0 P l. , q, r p , , . , P , - . , , . , -. , , .
. , - , , , - , .. 2. (2), , - . , -. - (), () - (). - (). .
2, Nomen est numen .
004.032.26 (06) 145
-
ISBN 572620526X
, - P R2 . , . , Vp - P , x VP p , q P .,
Vp ={x R2|q P {p}, x p x q},
{p} , p P . - 1908 (3), 3. - P , p q, : , - . , p q . () . , , ; , P . , - . , , ; . - . , , .
. . , , . , - P . . , , - ,
3 : , , -.
146 004.032.26 (06)
-
. .
, . ; - . - , - , , , . . . . 2 P , .
. 2.
, , , , Lp-, p 6=2. , L (4).
, , - . , 1915 (5) . - (6). (. 3). , 4 , - (). -
4 .
004.032.26 (06) 147
-
ISBN 572620526X
. 3.
- (7). , - : - . -, . 5 , . -! , , , , , - , .
., f(x, y) : f(xi, yi) = fi, i = 1, 2, . . . N . , , . , - f(x, y) -. , , - , (x, y)
Ni=1(x xi)(y yi),
5 .
148 004.032.26 (06)
-
. .
(x, y) . , , - . . - . - , . , , - . , , - . , . , , . f(x, y) , , - .
()
. - N . (8), - , . ,, , N. . 4. ,
. 4. 23
. N
004.032.26 (06) 149
-
ISBN 572620526X
, p, N {p} . p , . . , - . , . , . O(N2). , , .
. 5. Glass pattern
, - . - , , -, , -, : . . 5 6 , - , . ,
6 Glass pattern.
150 004.032.26 (06)
-
. .
. Glass pattern'
. - . . 6 , - (9) 1909 .
. 6.
- . , - , - . , , .
-
, - . , , . , -. , - . 3D .
004.032.26 (06) 151
-
ISBN 572620526X
, . , . 90 , , . - , , .
. , . . . , , . = 0, . - , , , = . - - (). 0 < < . . 7 -, , - . .
. 7. -
, ,
152 004.032.26 (06)
-
. .
, , . . - . - .
. 8. , 7
( ), - 7. , , , , -, . . . . 8 , .
, , (), - . - ; - [26, 27]. i, j = 1, . . . , N , - , Cij . Cij > 0, , i j. Cij yiyj , yi, yj
7exposed.
004.032.26 (06) 153
-
ISBN 572620526X
. v P (v), yi = R(v wi), R , w . :Cij yiyj , yiyj > ykyl Cij = 0 . - . [26, 27], , , {w}. , Cij [Cij(t )] = Aij , -, Aij {w} , Aij = 1, Vi
Vj 6= , Vi, Vj
wi, wj . , - . , . [28] , . . 9 , - .
. 9. ,
1. , , - . - , . , - 1978
154 004.032.26 (06)
-
. .
. , [5]. -, 1969 . . - [43]. - . , 4000 - . , . - (4)2 + (3)2 = (5)2 .
2. (17071778 .), , . , ,. . Systema naturae , . 1500 -, 24 . 6 : , , , , , -. , . , . [16] . , , -, ,, . . , - .
3. (18681908). , - (1907). 1889 , - . 1894 , 3 . 1897 - - , . .
004.032.26 (06) 155
-
ISBN 572620526X
4. - [8]. - , A, . . - , A . - >
-
. .
(b) P e2i , , e2i , P .
(c) P e1 e21 , P e21, e22 , e21
e22 = e
1.(d) P e21, -
e21, e22, . . . , e2n, P . - P .
8. , ; , , . - . , , , ; , - , .
9. (18701944), . - , 18 . 1893 - . , , -, 1870- . 1902 - . - . 1904 - . - . 1928 . .
. - [5]. - [6]; . [7]. [8]. - [9]. , , [10].
004.032.26 (06) 157
-
ISBN 572620526X
[1112] -. - [1314]. [15] (lamentary structure) . - . [16]. . [1721] [22]. 3D - [23], [24] [25] - . . [26, 27]. [28].
:
, - . ( , , - ), ( ). .
.
, (1) 1733 , - . , 2a. 2l < 2a 8 . , - ? -
8.. , .
158 004.032.26 (06)
-
. .
(. 10). - . P , . - 2x/2a. , [0, /2], ., x/a = (l/a) cos ,. . :
l/a
/2
/20
cos d = 2(2l)/2a.
. 10.
, , P , , :
P =2( )
a.
- (2) . :
004.032.26 (06) 159
-
ISBN 572620526X
, , , . , . - , (3), , .
, . - (. . 11).
. 11.
. , , - (. 11a), , . . , - . 1/3.
160 004.032.26 (06)
-
. .
. , -. - . , (. 11b), . .
3. -
, , , , , , , . , , . , 1/2.
. () -, , , . , , - (.11c). 1/2; , - , 1/4 . , , , 1/4.
, : 1/3, 1/2 1/4, . ., , , , . . .
1. , (17071788), -, , - (1753), . - -; - , [32]. ,, : -; , . , - , , .
004.032.26 (06) 161
-
ISBN 572620526X
1733 -, 1777 .
2. = 2l/P a, P = N1/N - P , N1 N -. [33]. , - , 355/113 3.1415929, N = 3408 , . . [29] -, , , - -, .
3. (18221900), , (1896) .
. - [29], - . [30] [31]. , , -[32].
: C R2
.
.
, , - . , , , -.
162 004.032.26 (06)
-
. .
m. , . . - . , , . A1, A2, . . . , An, . . . : , . . m(
Ai) =
m(Ai).
, . - , A1 A2, . , , - 9., , - . , ,
m(X) =
X
f(x, y)dxdy.
X . - , .
x, y P R2. G
x = a + x cos y sin,y = b + x sin + y cos.
}
f(x, y) , m(X) G. G (1), - , .
X
f(x, y)dxdy =
Xf(x, y)dxdy.
9 (. 11b) .
004.032.26 (06) 163
-
ISBN 572620526X
, ,
X
f(x, y)dxdy =
Xf(x, y)dxdy,
(x,y)(x,y) = 1. ,
Xf(x, y)dxdy =
Xf(x, y)dxdy.
X, -,
f(x, y) = f(x, y).
, -, f(x, y) , . .:
f(x, y) = const.
10, -:
G X P (x, y) X
m(X) =
X
dxdy
X
dP
, . (3) dP = dxdy .= dx dy -
11. (3).dx dy -
f(x)dx f(x(u))x(u)du. -
,
x = x(u, v), y = y(u, v),
10 .11 , ,
.
164 004.032.26 (06)
-
. .
dP = dxdy =
(x, y)(u, v)
dudv.
, dx =xudu + xvdv, dy = yudu + yvdv 12 (-) : dx dx = dy dy = 0; dx dy = dy dx, dxdy = (xuyvxvyu)dudv, .
- [34, 35]. -, . , , . .
G - G (0 6 6 2) p ( O (p > 0)). p, , - O . G (. 12)
x cos + y sin p = 0. (p, )
m(X) =
X
f(p, )dp d.
, dP , , :dG = dp d.
, -. D R2 F , G , D G
D . dp -
D, (0 6 6 ) :
GT
D
dG = F.
12 : xu = x/u.
004.032.26 (06) 165
-
ISBN 572620526X
. 12. G C
C - L,
x = x(s); y = y(s),
s . G , C - M(x, y). G C (.12). dG s, , - G. . 12
= + /2,
C x. M(x, y) G,
166 004.032.26 (06)
-
. .
p = x cos + y sin .
dp = cos dx + sin dy + (x sin + y cos )d.
dx = cos ds, dy = sin ds,
dp = cos( )ds + (x sin + y cos)d.
d dG = dp d =cos( )ds d.
d = d + ds, 13
dG = | sin |ds d.
, C.
L0
ds
0
| sin |d = 2L.
G n 14, - C. G, C, n = 0. C , ;
ndG = L.
. , C , , .
- (5). , - , .
, C , , - C, - .
13 | sin | . . .14, n !
004.032.26 (06) 167
-
ISBN 572620526X
, - . , C , C. , - , . , . [34, 35].
1. , , . , - , . , , -, , . . , G () -: , , .
2. - (u) =
ai(u)dxi. -
( 2 ) (u) =aij(u)dxi dxj ,
.
3. , R a, b, c, . . .. L n- R , , , . . .. p = 0, 1, 2 p pL - : 0L = R, 1L = L, 2L ,
ai(i i) ,
(a11 + a22) a1(1 ) a2(2 ) = 0, (b11 + b22) b1( 1) b2( 2) = 0,
= 0, + = 0.
168 004.032.26 (06)
-
. .
, -. ei L. =
iei, =
jej
=
i,j
aibj(ei ej) =
i
-
ISBN 572620526X
. - . [34, 35]. - , - , [36]. - . [31] ., - [37].
:
, , : , . . . .
.
{xi| i = 1, 2, . . . , N} Rd . , - , , . , , . xi Br(xi) = {x Rd| ||x xi|| 6 r} r. - (1): B(r) =
Ni=1 Br(xi).
. , , . , , . - .
K R3 c 15 - K C2. K K . ,
15. . .
170 004.032.26 (06)
-
. .
K , - :
K =
xKB(x).
. 13.
( F V) . , A a (. 13), :
F(A) = a2 + 4a + 4(/4)2.
, Q, , :
V(Q) = a3 + 6a2 + 12a(/4)2 + 8(4/24)3.
V d (2):
V(K) =d
i=0
(d
i
)Wi(K)i,
Wi(K) . d = 3 Wi [39]:W0(K) = V (K) K,3W1(K) = F (K) K,3W2(K) = H(K) K,
004.032.26 (06) 171
-
ISBN 572620526X
3W3(K) = G(K) = 4(K) K, K, .
d = 2 , - :
W0 = F , ,W1 = U/2 ,W2 = / . Wi :
: Wi(A
B) = Wi(A) + Wi(B)Wi(A
B).
. G gA(g G) A . : Wi(gA) =Wi(A).
. (3) An A, n : Wi(An) Wi(A).
. -. , . B Bi R2, Ai, . .
Bi B, Bi =N
i=1
Ai; B.
16 B, A, B B:
(A
B) = (A) + (B) (A
B)
A
(A) =
{1, A 6= ,0, A = .
16 .
172 004.032.26 (06)
-
. .
, , , -, . - ( ) . (3). , .
-: ( )( )+( )= 1, - . . - , [38], (. 14).
. 14.
, , , , .
, - . . , X R1 n . B, , = n. R2 (. 15).
(x) x. - X h(x).
004.032.26 (06) 173
-
ISBN 572620526X
(x), :
h(x) = lim0
{[X
(x)
] [X
(x + )]}
.
, h(x) 6= 0 xi, (x) X X . -
(X) =
i
h(xi).
h(x) X1, X2, X3 - . 15. , 1 - h(1) = 0. X1 : h(2) = 1 , ,(X1) = 1. X2 , h(2) = 1, h(3) = +1, h(4) = +1; (X2) = 1. , X3 h(2) = 1, h(3) = 0, h(4) = +1, (X3) = 0. , h(x) = +1, X (x) , . . , h(x) = 1, x. - ; , R3 , . X3 : (), .
. 15. R2
174 004.032.26 (06)
-
. .
. C, - s. . , (4). - : kg = d/ds.
2 =
C
kgds.
M () - M . - [42, 60] ,
n
i=1
( i) +
M
kgds +
M
KdA = 2(M)
1, 2, . . . , n M ; kg M ; K M ; ds dA . , () () ., , - [38, 41].
. . [43] , . . , , , . . (X) X , (1) (0) X . , (X) = 1, X (X) = 0 . k- , k , , - . , , X , p, q, r. - (p) = (q) = (r) = 1, X . , - , X .
004.032.26 (06) 175
-
ISBN 572620526X
, (X) , = 1. , , . . , X - , . . -, (X), [43], - . , X . , , , , . -, , , . . (X) = ( (X)) ( (X)) - .
1. (18641909) , . , , -. (Geometrie derZahlen; Diophantische Approzimationen .). - . . : Raumund Zeit (1909) .
2. (17961863) . - (). . . - (1834) , . , (1833), (1842).
3. - , . -
176 004.032.26 (06)
-
. .
[40]:
h(A, B) = inf{ : A B and B A
}.
, B, B - , A. 1. A 2. .
4. M0 s = M0M , M M0: k = limMM0 /s. k1 k2. () K = k1k2. , , k1 =k2 = 1/R, K = 1/R2
KdA =
1/R24R2 = 4.
. - [38], . [39, 40], -. [41]. - [10]; [42]. [43] - .
:
, .
,
- , , , - . ,
004.032.26 (06) 177
-
ISBN 572620526X
, -. , , - . , - . . 16 . -
. 16.
, . , , , - 17, . , , - . , , , . , (1).
, 18. ,
17 .18 . 1953 .
178 004.032.26 (06)
-
. .
. , - . , -. . , , N3. - , . - . , -. , , . - N2. . , . , . - , N1. , , . - . Ni, i = 1, 2, 3 , - . - . - [47, 49].
- , - X(t) Rn, t Rk. X(t) , , (n = 1) - (n = 3), (k = 2). - , sup X(t), , - u : Prob(supX(t) > u). , 19 , , . , . , , -
19 .
004.032.26 (06) 179
-
ISBN 572620526X
E() [47]:
Prob(sup X(t) > u) E() = 3
(2)2(u2 1) exp(u2/2)+
+(1/2)2
(2)3/2u exp(u2/2) + 2
2exp(u2/2)+
+
(2)1/2
0
exp(z2/2)dz.
R3 . - N2 . R1, . [45]. (t), t [a, b], - = h (. 17). - () h , (t) h, . . A(h) = {t| (t) > h}. A(h) .
t - : (t) < h; (t + dt) > h, dt . dt :
h (t)dt < (t) < h; (t) > 0, (t) .
Prob[h (t)dt < (t) < h] - (t) . 20, - , . . t, (t) h. . 1944 .
X(t) [0, T ], - h , X(0) 6= X(T ) 6= h. ,
20 , , .
180 004.032.26 (06)
-
. .
. 17.
X(t) h t0, > 0, X(t) 6 0 (t0 ), t0 X(t) > 0 t0, (t0 + ). t0 [0, T ] h Nh(X, T ). Nh(X, T ) Ah = Ah(X(t)) = {t [0, T ] : X(t) > h} . X(0) < h, Ah ; X(0) h,Ah = Nh + 1.
:
{t S : X(t) = h} = Ah(X, S), t RN , N > 1,
Ah Ah. R2, , Ah - . , . , , - . . [47, 48]. ,
004.032.26 (06) 181
-
ISBN 572620526X
. . - (), .
. 19491989 - 470 (90 -, 25 354 ). , , - . - : 14C ( 5730 ), 60Co (5.26 ),90Sr (28.5 ), 137Cs (30 ), 239Pu (24113 ), 241Am (14.5 ). , - , , . 21. , , - -. . . , 1 2, . . - . , : - 16000 . .
- - . - 1:1000000 - 10 . - -. , . . , -
21 1% .
182 004.032.26 (06)
-
. .
22 , - . , - , . - , . . - -. , : - k ? - . . 18 Cs . 20 50 . ; (2). K, Th, U .
0. 0 0. 1 0. 2 0. 3 0. 4 0. 5 0. 6 0. 7 0. 8 0. 9 1. 00. 0
0. 1
0. 2
0. 3
880
1080
1280
1480
. 18. Cs
-. . 19 - (3) IG - . , IG Cs
22 .
004.032.26 (06) 183
-
ISBN 572620526X
. - . [52, 53].
-6 -4 -2 0 2 4 6-50
-40
-30
-20
-10
0
10
20
30
40
50 Cs K Th U gauss
. 19.
. , -
