Lattice Coding and Decoding Achieve the Optimal Diversity ...
SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding...
Transcript of SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding...
![Page 1: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/1.jpg)
SC741– W6: An Introduction to Machine-
Learning and Modeling Techniques for Swarm-Intelligent, Embedded
Systems
![Page 2: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/2.jpg)
Outline• Machine-Learning Techniques
– Overview and definitions– Genetic Algorithms– In-line learning– From single-unit to multiple-unit systems
• Probabilistic Modeling Techniques– Overview and definitions– Microscopic modeling– Macroscopic modeling
• Combining Modeling and Learning Techniques
![Page 3: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/3.jpg)
An Introduction to Machine-Learning
Techniques
![Page 4: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/4.jpg)
Why Machine-Learning?
• Complementarity to a model-based/engineering approaches: when low-level details matter (optimization) and/or good solution do not exist (design)!
• When the design/optimization space is too big (infinite)/too computationally expensive to be systematically searched
• Automatic design and optimization techniques• Role of engineer reduced to specifying
performance requirements and problem encoding
![Page 5: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/5.jpg)
Why Machine-Learning?
• There are design and optimization techniques robust to noise, nonlinearities, discontinuities
• Individual real-time, adaptation to new environmental conditions; i.e. increased individual flexibility when environmental conditions are not known/cannot predicted a priori
• Search space: parameters and/or rules
![Page 6: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/6.jpg)
ML Techniques: Classification– Supervised techniques: “a trainer/teacher” is available.
• Ex: a set of in-out examples is provided to the system and the error at the output of the of the system vs. perfect answer of the teacher for the same input is calculated and fed to the system using a machine-learning algorithm so that error is reduced over trials
• The system generalization capabilities after training are tested on examples not presented to the system before (i.e. not in the training set)
– Unsupervised techniques: “trial-and-error”, “evaluative” techniques; no teacher available.
• The system judges its performance according to a given metric to be optimized
• The metrics does not refer to any specific input-to-output mapping• The system tries out possible design solutions, does mistakes, and
try to learn from its mistakes• Number of possible examples is very large, possibly infinite, and
not known a priori
![Page 7: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/7.jpg)
ML Techniques: Classification
– Off-line: in simulation, download of the learned/evolved solution onto real hardware when certain criteria are met
– Hybrid: most of the time in simulation (e.g. 90%), last period ( e.g. 10%) of the process on real hardware
– On-line: from the beginning on real hardware (no simulation). Depending on the algorithm more or less rapid
![Page 8: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/8.jpg)
ML Techniques: ClassificationML algorithms require sometimes fairly important computational resources (in particular for multi-agent search algorithms), therefoe a further classification is:
– On-board: machine-learning algorithm run on the system to be learned or evolved (no external unit)
– Off-board: the machine-learning algorithm runs off-board and the system to be learned or evolved just serve as phenotypical, embodied implementation of a candidate solution
![Page 9: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/9.jpg)
Selected Unsupervised ML Techniques Robust to Noisy
Fitness/Reinforcement Functions• Evolutionary computation
– Genetic Algorithms (GA) – Genetic Programming (GP)– Evolutionary Strategies (ES)– Particle Swarm Optimization (PSO)
• Learning – In-Line Adaptive Learning– Reinforcement Learning
Today
W9
W11Today
![Page 10: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/10.jpg)
GA: Coding & Decodingphenotype genotype phenotype
coding decoding(chromosome)
• phenotype: usually consists of real numbers • chromosome = string of genotypical segments, i.e. genes
G1 G2 GnG4G3 … Gi = gene = binary or real number
Coding: real to real or binary to real via Gray code (minimization of nonlinear jumping between phenotype and genotype)
Decoding: inverted operation
Rem:
• Artificial evolution: usually one-to-one mapping between phenotypic and genotypic space
• Natural evolution: 1 gene codes for several functions, 1 function coded by several genes.
![Page 11: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/11.jpg)
GA: Population & Fitness• Population: set of solutions tested in one generation; consists of m genetic
individuals (e.g. m = 100), each one provided with an own chromosome.
• Generation: new population after genetic operators have been applied (n = # generations e.g. 50, 100, 1000).
• Evaluation span: evaluation period of each genetic individual during one generation.
• Life span: number of generation an individual survive
• Fitness function: measurement of each genetic individual efficacy during life-span
• Population manager: applies genetic operators to generate the individuals of the new generation from the current one, at the end of the life span of the whole population.
![Page 12: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/12.jpg)
GA: Basic Operators• Selection: roulette wheel, ranked selection, elitist selection• Crossover: 1 point, 2 points (e.g. pcrossover = 0.2)
• Mutation (e.g. pmutation = 0.05)
Gk Gk’
Note: examples for fixed-length chromosomes!
![Page 13: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/13.jpg)
Generation Loop
Evaluation of Individual Candidate
Solutions
Population Replenishing
Selection
Crossover and Mutation
Decoding (genotype-> phenotype) Fitness Measurement
Population Manager
System
![Page 14: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/14.jpg)
Evolutionary Loop: Several Generations
Start
Initialize Population
Generation loop
End criterion met?N
Y
Ex. of end criteria:
• # of generations
• best solution performance
•… End
![Page 15: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/15.jpg)
Examples of GA Applications
• Mechanical design
• Single robot experiments
![Page 16: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/16.jpg)
An Example of Mechanical
Design using GA
+ +
Khepera external powering module• External power in each position and
orientation
• Power floor + redesigned basic module endowed with electrical contacts
![Page 17: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/17.jpg)
Powering Module: Problem Encoding
• Mixed approach: engineering intuition and GA
• Engineer: bands, minimal number of contacts (≥ 4) for limiting search space
• Genotype (9 reals): encoding of wcond, (r,ϕ) coordinates of each contact
• Fitness: fraction of position (translation and rotation) powered
• Simulated mechanical vibrations(gaussian noise) of contacts for evolving robust solutions
• m = 100, n = 50, pc = 0.2, pm = 0.01• Evolved solution symmetric, can be
backwards geometrically demonstrated that optimal solution
![Page 18: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/18.jpg)
Evolving a Neural Controller
1e12)( −
+= −xxf
S2
S3 S4S5
S6
M1 M2
∑=
+=m
jjiji IIwx
10
)( ii xfO =
inhibitory conn.excitatory conn.
Oi output
f(xi)
wijsynaptic weight
S1
neuron N with sigmoidtransfer function f(x)Ni
Ij S8 S7input
Note: In our case we evolve synaptic weigths but Hebbian rules for dynamic change of the weights, transfer function parameters, … can also be evolved (see Floreano course)
![Page 19: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/19.jpg)
Evolving Obstacle Avoidance(Floreano and Mondada 1996)
Defining performance (fitness function):
)1)(1( iVV −∆−=Φ
• V = mean speed of wheels, 0 ≤ V ≤ 1• ∆v = absolute algebraic difference
between wheel speeds, 0 ≤ ∆v ≤ 1• i = activation value of the sensor with the
highest activity, 0 ≤ i ≤ 1
Note: Fitness accumulated during evaluation span, normalized over number of control loops (actions).
![Page 20: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/20.jpg)
Evolving Robot Controllers
Note:Controller architecture can be of any type but worth using GA if the number of parameters to be tuned is important
![Page 21: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/21.jpg)
Evolving Obstacle Avoidance
Evolved path
Fitness evolution
![Page 22: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/22.jpg)
Evolved Obstacle Avoidance Behavior
• Generation 100• Direction of motion NOT encoded in the fitness function: GA
automatically discovers asymmetry in the sensory system configuration (6 proximity sensors in the front and 2 in the back)
![Page 23: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/23.jpg)
Evolving Homing Behavior(Floreano and Mondada 1996)
Robot’s sensorsSet-up
![Page 24: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/24.jpg)
• Fitness function:
• V = mean speed of wheels, 0 ≤ V ≤ 1• i = activation value of the sensor with the
highest activity, 0 ≤ i ≤ 1
)1( iV −=Φ
Evolving Homing BehaviorController
• Fitness accumulated during life span, normalized over maximal number (150) of control loops (actions).
• No explicit expression of battery level/duration in the fitness function (implicit).• Chromosome length: 102 parameters (real-to-real encoding).• Generations: 240, 10 days embedded evolution on Khepera.
![Page 25: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/25.jpg)
Evolving Homing BehaviorFitness evolution
Evolution of # control loops per evaluation span
Battery recharging vs. motion patterns
Battery energy
Left wheel activation
Right wheel activation
Reach the nest -> battery recharging -> turn on spot -> out of the nest
![Page 26: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/26.jpg)
Evolved Homing Behavior
![Page 27: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/27.jpg)
Moving BeyondController-Only Evolution
• Evidence: Nature evolve HW and SW at the same time …
• Faithful realistic simulators enable to explore design solution which encompasses off-line co-evolution (co-design) of control and morphological characteristics (body shape, number of sensors, placement of sensors, etc. )
• GA are powerful enough for this job and the methodology remain the same; only encoding changes
![Page 28: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/28.jpg)
Evolving Sensory Systems(Zhang, Antonsson, Martinoli 2002)
• Realistic simulator: Webots• Evolution of:
– number of sensors– cone of view– range– sensor placement
• Sensory system cost proportional to # of sensors, range, cone of view
• Fitness: maximal coverage at minimal cost ��������������
����� ������
����������������
�����
����
�������
������
�����
��
���
�
�����
��
������ ������
�����
������� ������!���
��
�
![Page 29: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/29.jpg)
Dealing with Variable-Length Chromosome
Zero line Zero line
Crossover line Crossover line
Zero line Zero line
Crossover line Crossover line
Parents Children
Crossover
• Evolution of number of sensors -> variable-length chromosome
• Additional operators: insertion, deletion
• Crossover: extremely disruptive operator
• Idea: take advantage of the problem genotype-to-phenotype mapping
• In our case: convex, closed structure on the vehicle -> fold chromosome in the same way -> mapping maintained!
![Page 30: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/30.jpg)
Evolving Control and Robot Morphology
(Lipson and Pollack, 2000)
http://www.mae.cornell.edu/ccsl/research/golem/index.html• Arbitrary recurrent ANN• Passive and active (linear
actuators) links• Fitness function: net distance
traveled by the centre of mass in a fixed duration
Example of evolutionary sequence:
![Page 31: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/31.jpg)
Examples of Evolved Machines
Problem: simulator not enough realistic (performance higher in simulation because of not good enough simulated friction; e.g., for the arrow configuration 59.6 cm vs. 22.5 cm)
![Page 32: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/32.jpg)
In-Line Adaptive Learning
![Page 33: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/33.jpg)
Not Always a big Artillery such a GA is the Most Appropriate Solution…
• Simple individual learning rules combined with collective flexibility can achieve extremely interesting results
• Simplicity and low computational cost means possible embedding on real simple individuals
• We will see similar learning rules in combination with threshold bases algorithms, foraging strategies, …
![Page 34: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/34.jpg)
In-Line Adaptive Learning(Li, Martinoli, Abu-Mostafa, 2001)
Key ideas:• Origin: adaptive bang-bang control• 2-steps algorithm• Learning step adaptive according to fixed rules• Randomness when stuck• Can be combined with a low-pass filter for noise reduction
GTP: parameter to be learned
∆d: learning stepd: direction
Performance
![Page 35: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/35.jpg)
In-Line Adaptive LearningDifferences with gradient descent methods:• Fixed rules for calculating step increase/decrease -> limited
descent speed -> no gradient computation -> more conservative but more stable
• Randomness for getting out from local minima (no momentum)
• Underlying low-pass filter is part of the algorithm
Differences with Reinforcement Learning:• No learning history considered (only previous step)
Differences with basic In-Line Learning:• Step adaptive -> faster and more stability at convergence
![Page 36: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/36.jpg)
Sample ResultsTest-bed: multi-agent system, 1 parameter (GTP)
Filter cut-off f high (short averaging window)
Filter cut-off f low (long averaging window)
0 100 200 300 400 500 6000
0.2
0.4
0.6
0.8
1
1.2
1.4
Initial gripping time parameter (sec)
Sti
ck−p
ulli
ng
rat
e (1
/min
)
2 robots
3 robots
4 robots
5 robots
6 robots
00
0.2
0.4
0.6
0.8
1
1.2
1.4
Sti
ck−p
ulli
ng
rat
e (1
/min
)Pe
rfor
man
ce
GTP
Perf
orm
ance
Learned (mean + std dev)Systematic (mean only)
100 200 300 400 500 600Initial gripping time parameter (sec)
2 robots
3 robots
4 robots
5 robots
6 robots
GTP
![Page 37: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/37.jpg)
Moving from a Single-Unit to Multiple-Unit
Systems
![Page 38: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/38.jpg)
Evolving Controllers
• Collective: fitness become noisy due to partial perception, independent parallel actions
![Page 39: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/39.jpg)
Credit Assignment ProblemWith limited communication, no communication at all, or partial perception:
![Page 40: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/40.jpg)
Example: Evolving the Sensory System of one Vehicle within a Traffic System
(Zhang, Antonsson, Martinoli 2002)
��������������
����� ������
����������������
�����
����
�������
������
�����
��
���
�
�����
��
������ ������
�����
������� ������!���
��
�
![Page 41: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/41.jpg)
Fitness Function
}0;)1max{(1
∑=
−=V
i
i
Vanf δ
a = cost factor for sensorsn = number of sensors usedδi = 0 if car not detected; 1 if car detectedV = actual number of vehicles in the detection region during
a whole evaluation span
![Page 42: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/42.jpg)
Ad Hoc Evolutionary Loop
• Goals:– Evaluating actual fitness– Eliminating as quickly as possible
lucky individuals
• Solution:– Dynamical population size– Individual life span = several
evaluation spans over generations
![Page 43: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/43.jpg)
Co-Learning and Co-Evolution
![Page 44: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/44.jpg)
Co-Learning or Co-Evolving of Competitive Behavior
No credit assignment problem: individual fitness!Example: co-evolution of prey’s and predator’s controllers
Φprey= T Φpredator= 1-T
T = normalized time of survival of the prey (before is caught)
![Page 45: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/45.jpg)
Prey-Predator Experiment(Nolfi and Floreano, 1998)
![Page 46: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/46.jpg)
Prey-Predator ExperimentFitness co-evolution in simulation
Fitness co-evolution in real robots
![Page 47: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/47.jpg)
Co-Learning or Co-Evolving of Collaborative Behavior
Key question: What metrics (group, individual, combination) should we measure for improving individual controllers?
![Page 48: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/48.jpg)
Ex.:Co-Evolving Collaborative BehaviorThree orthogonal axes to consider (extremities or balanced solutions are possible):
• Individual and group fitness• Private (non-sharing of parameters) and public (parameter sharing) policies• Homogeneous vs. heterogeneous systems
![Page 49: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/49.jpg)
An Introduction to Probabilistic Modeling
Techniques
![Page 50: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/50.jpg)
Why Modelling?
• Current target platform: self-organized robotic systems
• Individual robot control deterministic BUT mobility + robot-to-environment and robot-to-robot interactions -> overall individual behavior highly stochastic
• However, we do not care about microscopic interactions (part of the self-organization process) as long as the swarm behavior/performance is statistically predictable -> only way to get swarm-based (industrial) applications!
• This way of proceeding is revolutionary in engineering (while traditional in science – biology, physics, chemistry). E.g., classical roboticists tend to eliminate all forms of inaccuracies, limitations in their initial models and then relax the original assumption in order to come close to reality. In our models, we consider from the beginning noise and inaccuracies (even more important with miniaturization) very much as biologists, physicists, and chemists do.
![Page 51: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/51.jpg)
Why Modelling?• Understanding the role of system parameters in the team
performancecontrol, interaction geometry, body morphology, sensor, communication range, number of individuals
• Generalizingother hardware platforms, other experimental constraints, other class of experiments
• Prediction and fast evaluation -> design!before developing ad hoc embodied simulators or real embedded systems (104-105 faster than embodied simulations)
Probabilistic model
Embodied simulatorReal robots
![Page 52: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/52.jpg)
An Incremental Probabilistic Modeling Methodology for
Nonspatial Metrics
![Page 53: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/53.jpg)
Experimental LevelsAbstracted interactions, swarm behavior
• Simulation– Macroscopic models– Microscopic models– Non-embodied simulations– Embodied, sensor-based simulations
• Real robots– Actively controlled lab environments– Passively controlled lab environments– Real environments Detailed interactions,
individual behavior
![Page 54: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/54.jpg)
Methodology OverviewA priori known features of robots and environment;
desired swarm performance metric(s)
Characterization of microscopic interactions
(robot-to-robot, robot-to-environment)
Prediction of swarm performance/macroscopic system dynamics
![Page 55: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/55.jpg)
Modeling Assumptions• Nonspatial metrics for swarm performance
• Environment and controller can be described as ProbabilisticFSM; the state granularity of the description is arbitrarily established by the engineer as a function of the microscopic or macroscopic level of description.
• Both robots and environment are semi-Markovian systems: the system future state is a function of the current state and of how much time has been spent in this state.
• Mean spatial distribution of robots is homogeneous, as they were randomly hopping on the arena (trajectories not considered).
![Page 56: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/56.jpg)
Experimental Validation of Nonspatial Assumption – (1)
PositionNonembodied obstacles = detection surfaces
Shape
Numerical example (mean ± std dev, 3 locations, 100 h simulated time):
0.310.31 ± 0.030.32 ± 0.020.3 ± 0.030.31 ± 0.04robot
GeometryAll shapesRoundRect.SquareSize
![Page 57: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/57.jpg)
Experimental Validation of Nonspatial Assumptions (2)
DefaultSymmetryof Stick Distribution
# sticks
![Page 58: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/58.jpg)
Modeling Methodology
Microscopic characterization
Search Avoidance
Start
Obstacle?pa
τa
1-pa
Deterministic agent’s flowchart
Ss Sa
pa
τa
1
PFSM (Markov Chain)
Start
Deterministic robot’s flowchart
Search Avoidance
Obstacle?YN
![Page 59: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/59.jpg)
Microscopic Modeling Methodology
…
Caste 1
Ss Sa
Ss Sa
Ss Sa
R11
R12
R1l
…
Caste n
Se Sd
Si
Rn1
…Se Sd
Si
Rnm
…
O11 O1pOq1 Oqr
… …Sa Sb Sa Sb
Robotic System (N PFSM; N = total # agents)Coupling (e.g., manipulation, sensing)
Environment (Q PFSM; Q = total # objects)
![Page 60: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/60.jpg)
Macroscopic Modeling Methodology
Coupling
Type 1
Ss Sa
Se Sd
Si
Type q
Caste1
Caste n
Robotic (PFSM)
• average quantities• central tendency prediction (1 run)• continuous quantities: +1 DE per state
for all robotic castes and object types (metric/task dependent!)
• - 1 DE if substituted with conservation equations (e.g., total # of robots, total # of objects of type q, … )
Sa Sb
Environment (PFSM)
![Page 61: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/61.jpg)
Model Calibration• Goal: calibration method for achieving 0-
free parameter models!• Two types of parameters:
– Delays– Encountering probabilities
• Calibration procedure:– Systematic experiment with 1 real or realistic
simulated robot (robot-to-environment interaction)
– Systematic experiments with 2 real or realistic simulated robots (robot-to-robot interaction)
![Page 62: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/62.jpg)
Time Discretization: The Engineering Receipt
Time-discrete vs. time-continuous models:1. Assess what’s the time resolution needed for your swarm
performance metrics2. Consider unit-to-unit interaction essence: probabilistic/deterministic,
asynchronicity role, …3. Choose whenever possible the most computationally efficient model:
time-discrete less computationally expensive than emulation of continuity (e.g. Runge-Kutta, etc.) and maybe the prediction accuracy not statistically different from continuous models because of other (more important) sources of inaccuracies in the system
Often not even a tradeoff: just use the appropriate instrument for the appropriate problem!
![Page 63: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/63.jpg)
Delays
1. Measure all delays of interest in your system, i.e. those which might influence the swarm performance metrics. Note: often “delay states” can just summarize all what you need without getting into the details of what’s going on within the state
2. Consider only average values (we might consider also parameter distribution in the future, the modeling methodology does not prevent to do so)
3. For time-discrete systems: choose the time step T = MCF all the delays measured (e.g., 3 s obstacle avoidance, 4 s object manipulation, T = 1 s) -> no rounding error
![Page 64: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/64.jpg)
Encountering Probabilities• Idea: trajectories and object location do not count
(see assumption) -> 1D Montecarlo simulation
2D physical space 1D probability space
O1
O1O1
O2
O2O2
O2
Free space
![Page 65: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/65.jpg)
Encountering Probabilities
is
si g
AvWr =
As = detection area of the smallest object
v = mean robot speedWs = robot’s detection width for the
smallest object (center-to-center)
1. Measure geometric probabilities of encountering (detection areas)2. Calculate the encountering rate ri [s-1] for the object i from the
geometric probabilities gi :
3. For time-discrete models, calculate the encountering probabilities pi (per time step) from the encountering rates:
pi = riT
NOTE: not exactly the same of IJRR04 and AR04!
![Page 66: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/66.jpg)
Geometric Probabilities (Detection Areas)
ps, pw, …are function of sensor range, behavior, robot’s and object’s size, … : interaction characterization!
Example: stick
ps = As/Aarena
As
![Page 67: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/67.jpg)
Example 1: Linear Model –Wandering and
Obstacle Avoidance
![Page 68: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/68.jpg)
A Simple Linear ModelExample: search (moving forwards) and obstacle avoidance
© Gilles Caprari, 1999 © Nikolaus Correll, 2004
![Page 69: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/69.jpg)
A Simple Linear Model
Search Avoidance
1
+ paNs(k-Ta)- paNs(k)
pa
Linear DEs!
Ns(k)Ns(k+1) =
Na(k+1) = N0 – Ns(k+1)
Ta = obstacle avoidance durationpa = probability of encountering an obstacleNs = average # robots in searchNa= average # robots in obstacle avoidanceN0 = # robots used in the experimentk = 0,1, … (iteration index)
! Ns(k) = Na(k) = 0 for all k<0 !Ns(0) = N0 ; Na(0) = 0
![Page 70: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/70.jpg)
A Simple Linear Model
Group size
Micro to macro comparison(same robot density but wall surfacebecome smaller with bigger arenas)
Typical parameter analysis at steadystate
![Page 71: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/71.jpg)
Example 2:Nonlinear Model –
Collaborative Stick-Pulling
![Page 72: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/72.jpg)
Set-Up
Physical Set-Up Collaboration via indirect communication
Proximity sensors
Arm elevationsensor
• 2-6 robots• 4 sticks• 40 cm radius arena
IR reflectiveband
![Page 73: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/73.jpg)
Systematic Experiments
Real robots Embodied simulation
•[Martinoli and Mondada, ISER, 1995]•[Ijspeert et al., AR, 2001]
![Page 74: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/74.jpg)
Systematic Experiments’ ResultsNrobots > Nsticks
Nrobots ≤ Nsticks
• Real robots (3 runs) and embodied simulations (10 runs)• System bifurcation as a function of #robots/#sticks
![Page 75: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/75.jpg)
Modeling Methodology
Microscopic characterization
Probabilistic agent’sflowchart
Markov Chain (PFSM)Deterministic robot’s flowchart
![Page 76: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/76.jpg)
Geometric Probabilities
sgg
sg
aww
rR
arr
ass
pRp
ppAAp
NppAApAAp
=
==
−===
2
1
0
/)1(
//
Aa = surface of the whole arena
![Page 77: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/77.jpg)
Simplified Macroscopic Model
Τi,Τa,Τd,Τc << Τg →Τi=Τa=Τd=Τc=0
![Page 78: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/78.jpg)
Simplified Macroscopic ModelNonlinear DE coupling through unit-to-unit interaction (in this case through the stick)!
Search Grip
Ns(k+1) =
Ng(k+1) =
C(k) =
Ct(k) =
Ns(k) – pg1[M0 – Ng(k)]Ns(k)
successful
+ pg2Ng(k)Ns(k)
unsuccessful
+ pg1[M0 – Ng(k-Τg)]Γ(k;0)Ns(k-Tg)
N0 – Ns(k+1)
pg2Ns(k)Ng(k)
e
T
k
T
kCe
∑=0
)(
Ns = average # robots in searching modeNg= average # robots in gripping modeN0 = # robots used in the experimentM0 = # sticks used in the experimentΓ = fraction of robots that abandon pullingTe = maximal number of iterationsk = 0,1, …Te (iteration index)C = # of collaborations Ct = mean collaboration rate over Te
![Page 79: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/79.jpg)
Simplified Macroscopic Model
Fraction of robots which abandon pulling:
∏−
−−=
−=ΓSL
SLg
Tk
TTkjsgSL jNpTk )](1[);( 2
with
TSL = shift left duration [iterations] (=0 in this model)
Initial conditions and causality
Ns(0) = N0, Ng(0) = 0Ns(k) = Ng(k) = 0 for all k<0
![Page 80: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/80.jpg)
Results Simplified Model• Microscopic (100 runs) and macroscopic models overlapped• Only qualitatively agreement with embodied/real robots results• Discrepancies micro-macro: small numbers and combinatorial issues
• 16 robots, 16 sticks, Ra = 80 cm• 4 robots, 4 sticks, Ra = 40 cm
![Page 81: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/81.jpg)
Steady State Analysis• Steady-state analysis → can be computed analytically:opt
gT
gc
g
gg
optg R
forR
NRpT
+=≤
−
+−
−=
12
21
)1(2
1ln
)2
1ln(
10
1
βββ
β
with β = N0/M0 = ratio robots-to-sticks
• No for β>βc;• Depending on Rg, it’s possible:
i.e., optimal Tg also with more robots than sticks • [Lerman et al, Alife J., 2001], [Martinoli et al, IJRR, 2004]
optgT
cββ <<1
![Page 82: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/82.jpg)
Steady State Analysis
20 robots and 16 sticks (optimal Tg)
gg RR101~ =Example: (collaboration very difficult)
![Page 83: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/83.jpg)
Full ModelFor instance, for the average number of robots in searching:
)()()()()()()();()()(])()([)()1(
22
121
iasRaswcdascdagcascag
cgasacgagsRwggss
TkNpTkNpTkNTkTkNTkTkNTkTkkNppkkkNkN
−+−+−−∆+−−∆+−Γ−∆+++∆+∆−=+
with time-varying coefficients (nonlinear coupling):
∏−
−−=
−=Γ
=∆
−−=∆
SL
SLg
Tk
TTkjsgSL
ggg
dggg
jNpTk
kNpk
kNkNMpk
)](1[);(
)()(
)]()([)(
2
22
011
• 6 states: 5 DE + 1 cons. EQ• Ti,Ta,Td,Tc ≠ 0; Τxyz = Τx + Τy + Τz• TSL= Shift Left duration• [Martinoli et al., IJRR, 2004]
![Page 84: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/84.jpg)
Results (Standard Arena)
Webots (10 runs), microscopic (100 runs), macroscopic model (1 run)
![Page 85: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/85.jpg)
Results: 4 x #Sticks, #Robots and Arena Area
Webots (10 runs), microscopic (100 runs), macroscopic model (1 run)
![Page 86: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/86.jpg)
Example 3:Nonlinear Model – Collective
Aggregation
![Page 87: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/87.jpg)
To be Discussed in Week 10 Aggregation ProcessPhysical Set-Up
• 1-10 robots• 20-40 seeds• 80x80 cm and 114x114 cm arenas• With and without activity regulation
![Page 88: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/88.jpg)
Combining Modeling and Machine-learning
Techniques
![Page 89: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/89.jpg)
Key Idea• Microscopic modeling (or in general any form of abstraction) allow us
to consider certain parameters and leave others• Models as a “parameter” filter; allow for fast simulation and enhanced
understanding• Macroscopic models do not allow the study of specialization unless
precise castes can be identified a priori (caste = set of homogeneous individuals)
• In this case study we have identified a crucial parameter for our metrics and we wanted to understand what’s the best distribution of parameter values over the different individuals (learning as automatic way for discovering diversity/specialization)
![Page 90: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/90.jpg)
In-Line Adaptive Learning(Li, Martinoli, Abu-Mostafa, 2001)
• GTP: Gripping Time Parameter (parameter to be learned)• ∆d: learning step• d: direction• Performance/reinforcement signal: collaboration rate, i.e. #
sticks pulled/minute
![Page 91: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/91.jpg)
Co-Learning Collaborative BehaviorThree orthogonal axes to consider (extremities or balanced solutions are possible):
• Individual and group fitness• Private (non-sharing of parameters) and public (parameter sharing) policies• Homogeneous vs. heterogeneous systems
1
2
3
![Page 92: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/92.jpg)
Results #1 – Public & Group & Homogeneous Policy
Filter cut-off f high (short averaging window)
Filter cut-off f low (long averaging window)
0 100 200 300 400 500 6000
0.2
0.4
0.6
0.8
1
1.2
1.4
Initial gripping time parameter (sec)
Sti
ck−p
ulli
ng
rat
e (1
/min
)
2 robots
3 robots
4 robots
5 robots
6 robots
0 100 200 300 400 500 6000
0.2
0.4
0.6
0.8
1
1.2
1.4
Initial gripping time parameter (sec)
Sti
ck−p
ulli
ng
rat
e (1
/min
)2 robots
3 robots
4 robots
5 robots
6 robots
Note: 1 parameter for the whole swarm!
Systematic (mean only)Learned (mean + std dev)
![Page 93: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/93.jpg)
Diversity and SpecializationKey question: does team diversity enhance performance? I.e., canindividual become specialized?
Performance ratio between 2 caste and homogeneous system (microscopic model + Webots, systematic)
4 robots, one per color, microscopic model + learning
![Page 94: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/94.jpg)
Results #1,2,3 – All Three Learning Policies and Systematic
Systematic search
Learning
Public, group, homprivate, group, hetprivate, individual, het
Importance of specialization
• [Li, Martinoli, Abu-Mostafa, 2002] Note: high cut-off filter frequency!
![Page 95: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/95.jpg)
Impact of Diversity on Performance(Li, Martinoli, Abu-Mostafa, 2004)
2 3 4 5 6
1
1.05
1.1
1.15
1.2
1.25
1.3
Number of robots
Stic
k−pu
lling
rat
e ra
tio2−caste, GlobalHeterogeneous, GlobalHeterogeneous, Local
Performance ratio between heterogeneous (full and 2-castes) and homogeneous groups AFTER learning
![Page 96: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/96.jpg)
Measuring Diversity and Specialization(Li, Martinoli, Abu-Mostafa, 2004)
2 3 4 5 60
1
2
3
4
5
6
Number of robots
Div
ersi
ty
2−caste, GlobalHeterogeneous, GlobalHeterogeneous, Local
2 3 4 5 6
0
0.5
1
1.5
2
2.5
Number of robots
Sp
ecia
lizat
ion
2−caste, GlobalHeterogeneous, GlobalHeterogeneous, Local
Diversity (entropy measurement)
Specialization: diversity in order to fit
![Page 97: SC741– W6: An Introduction to Machine- Learning and Modeling … · 2008. 11. 21. · GA: Coding & Decoding phenotype genotype phenotype coding (chromosome) decoding • phenotype:](https://reader034.fdocuments.net/reader034/viewer/2022051912/6002bacef232d942e9544dba/html5/thumbnails/97.jpg)
Additional Literature – Week 6Books• Ross S. M., Introduction to Probability Models, Academic Press, San
Diego, CA, USA, 1997.
Papers• Ijspeert A. J., Martinoli A., Billard A., and Gambardella L.M.,
“Collaboration through the Exploitation of Local Interactions inAutonomous Collective Robotics: The Stick Pulling Experiment”. Autonomous Robots, 11(2):149–171, 2001.
• Lerman K., Martinoli A., and Galstyan A., “A Review of Probabilistic Macroscopic Models for Swarm Robotic Systems”. Proc. of the Workshop on Swarm Robotics, 8th Int. Conference on the Simulation of Adaptive Behavior, Los Angeles, July, 2004. Lecture Notes in Computer Science (2004), Sahin E. and Spears W., editors, to appear.
• Lerman, K. and Galstyan, A. Mathematical model of foraging in a group of robots: Effect of interference. Autonomous Robots, 13(2):127–141, 2002.