Axel Wolpert: Computergestütztes Lernen - was geht heute Ein Überblick
Wolpert Slides on Machine learning
description
Transcript of Wolpert Slides on Machine learning
![Page 1: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/1.jpg)
Probabilistic mechanisms in human sensorimotor controlDaniel Wolpert, University College London
• movement is the only way we have of– Interacting with the world – Communication: speech, gestures, writing
• sensory, memory and cognitive processes future motor outputs
Q. Why do we have a brain?
Sea Squirt
A. To produce adaptable and complex movements
![Page 2: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/2.jpg)
1000
1500
2000
2500
3000
3500
Why study computational sensorimotor control?
Year
Pag
es r2=0.96
1980 1990 2000 2010 2020 2030 2040 2050500
Experiments
Theory
Principles of Neural Science, Kandel et al.
![Page 3: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/3.jpg)
vs.
What to move where
vs.
Moving
The complexity of motor control
![Page 4: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/4.jpg)
Noise makes motor control hard
Noise = randomness
The motor system is Noisy
Perceptual noise– Limits resolution
Motor Noise– Limits control
NoisyPartial
Noisy
AmbiguousVariable
![Page 5: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/5.jpg)
David Marr’s levels of understanding (1982)
1) the level of computational theory of the system
2) the level of algorithm and representation, which are used make computations
3) the level of implementation: the underlying hardware or "machinery" on which the computations are carried out.
![Page 6: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/6.jpg)
Tutorial Outline– Sensorimotor integration
• Static multi-sensory integration• Bayesian integration• Dynamic sensor fusion & the Kalman filter
– Action evaluation• Intrinsic loss function• Extrinsic loss functions
– Prediction• Internal model and likelihood estimation • Sensory filtering
– Control• Optimal feed forward control• Optimal feedback control
– Motor learning of predictable and stochastic environments
Review papers on www.wolpertlab.com
![Page 7: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/7.jpg)
Multi-sensory integration
Multiple modalities can provide information about the same quantity• e.g. location of hand in space
– Vision– Proprioception
• Sensory input can be– Ambiguous– Noisy
• What are the computations used in integrating these sources?
![Page 8: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/8.jpg)
Ideal Observers
Maximum likelihood estimation (MLE)
i ix x ε= +
1 2( , ..., | )nP x x x x1
( | )n
ii
P x x=
=∏
2(0, )i iNε σ=
( )2
211
ˆ withn
ii i i n
i jj
x w x w σ
σ
−
−=
=
= =∑∑
( ) 12 2 2ˆ 1
nx i kj
kσ σ σ−
−=
= < ∀∑
2x
1x
x
Consider signals , {1 }in x i n= …
Two examples of multi-sensory integration
![Page 9: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/9.jpg)
Visual-haptic integration (Ernst & Banks 2002)
Two alternative force choice size judgment• Visual• Haptic• Visual-haptic (with discrepancy)
![Page 10: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/10.jpg)
Visual-haptic integration
Measure•Visual reliability •Haptic reliability•Predict
•Visual + Haptic noise•Weighting of
2Vσ2Hσ
2 Hσ
Prob
abili
ty
Size
Prob
abili
ty
Size difference
2 Hσ
Hσ Hσ
![Page 11: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/11.jpg)
Visual-haptic integration
( )2
21
2
2 2 1
ii n
jj
VH V H
V H
w
w w w
σ
σ
σσ σ
−
−=
=
= = −+
∑( )
2 212 2ˆ 2 21
n V Hx ij
V H
σ σσ σσ σ
−−
== =
+∑
Weights Standard deviation (~threshold)
Optimal integration of vision and haptic information in size judgement
![Page 12: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/12.jpg)
Visual-proprioceptive integrationClassical claim from prism adaptation
“vision dominates proprioception”
![Page 13: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/13.jpg)
Reliability of proprioception depends on location
(Van Beers, 1998)
Reliability of visual localization is anisotropic
![Page 14: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/14.jpg)
Integration models with discrepancyWinner takes all
Linear weighting of mean
Optimal integration
ˆ (1 )V Hw w= + −x x x
ˆ V HA B= +x x x
1 1 1
1 1
( )
( )PV P V
PV PV P P V Vµ µ µ
− − −
− −
Σ = Σ +Σ
= Σ Σ +Σ
![Page 15: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/15.jpg)
Prisms displace along the azimuth•Measure V and P•Apply visuomotor discrepancy during right hand reach•Measure change in V and P to get relative adaptation
Vision 0.33Prop 0.67
(Van Beers, Wolpert & Haggard, 2002)
![Page 16: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/16.jpg)
Visual-proprioceptive discrepancy in depth
AdaptationVision 0.72Prop 0.28
Visual adaptation in depth > visual adaptation in azimuth (p<0.01)> Proprioceptive adaptation in depth (p<0.05)
Proprioception dominates vision in depth
![Page 17: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/17.jpg)
Priors and Reverend Thomas Bayes
“I now send you an essay which I have found among the papers of our deceased friend Mr Bayes, and which, in my opinion, has great merit....”
Essay towards solving a problem in the doctrine of chances. Philosophical Transactions of the Royal Society of London, 1764.
1702-1761
![Page 18: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/18.jpg)
Bayes rule
A and B
( , )P A B = ( | )P A B ( | ) ( )P B A P A=
Belief instate sensory inpuAFTER t
Posterior
( )P B
P(state|sensory input)
Prior
BEFOREBelief in state sensory input
NeuroscienceA= State of the world B=Sensory Input
A = Disease B = Positive blood test
Likelihood
Evidence
P (sensoryinput|state) P(state)=P(sensory input)
Agiven B
![Page 19: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/19.jpg)
Bayesian Motor Learning
= Optimal estimate (Posterior)
Bayes rule
+ Task statistics (Prior)Not all locations are
equally likely
Sensory feedback (Evidence)Combine multiple cuesto reduce uncertainty
Estimate
Evidence
Prior
P (sensorP(state| y input|ssensory tate Pinput) (st )) ate∝
Real world tasks have variability, e.g. estimating ball’s bounce location
Does sensorimotor learning use Bayes rule?If so, is it implemented• Implicitly: mapping sensory inputs to motor outputs to minimize error?• Explicitly: using separate representations of the statistics of the prior and sensory noise?
![Page 20: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/20.jpg)
(Körding & Wolpert, Nature, 2004)
Prior
Lateral shift (cm)
Prob
abili
ty
0 1 2
Task in which we control 1) prior statistics of the task 2) sensory uncertainty
![Page 21: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/21.jpg)
(Körding & Wolpert, Nature, 2004)
Prior
Lateral shift (cm)
Prob
abili
ty
0 1 2
Task in which we control 1) prior statistics of the task 2) sensory uncertainty
![Page 22: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/22.jpg)
Sensory FeedbackLikelihood
Generalization
Learning
![Page 23: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/23.jpg)
After 1000 trials
2 cm shift
1cm
No visual feedback
Prob
abili
ty
0 1 2
Lateral shift (cm)
![Page 24: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/24.jpg)
0 1 2
Bayesian Compensation
Lateral shift (cm)
Models
Full Compensation
Lateral shift (cm)0 1 2
0 1 2Lateral Shift (cm)
Ave
rage
Err
or
Bia
s (c
m) 1
0
-1 0 1 2Lateral Shift (cm)
Ave
rage
Err
or
Bia
s (c
m) 1
0
-1
Mapping
0 1 2Lateral shift (cm)
0 1 2Lateral Shift (cm)
Ave
rage
Err
or
Bia
s (c
m) 1
0
-1
![Page 25: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/25.jpg)
Supports model 2: Bayesian
Results: single subject0
Full
Bayes
Map
0 1 2Lateral Shift (cm)
Ave
rage
Err
or
Bia
s (c
m)
![Page 26: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/26.jpg)
Supports model 2: Bayesian
Results: 10 subjects0
Full
Bayes
Map
0 1 2Lateral Shift (cm)
Ave
rage
Err
or
Bia
s (c
m)
-0.5 1 2.5
1
0
lateral shift [cm]Lateral shift (cm)
Infe
rred
Prio
r(n
orm
aliz
ed)
![Page 27: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/27.jpg)
Bayesian integrationSubjects can learn
• multimodal priors• priors over forces• different priors one after the other
(Körding& Wolpert NIPS 2004, Körding, Ku & Wolpert J. Neurophysiol. 2004)
![Page 28: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/28.jpg)
Statistics of the world shape our brain
Objects Configurations of our body
• Statistics of visual/auditory stimuli representation visual/auditory cortex• Statistics of early experience what can be perceived in later life
(e.g. statistics of spoken language)
![Page 29: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/29.jpg)
Statistics of action
• 4 x 6-DOF electromagnetic sensors• battery & notebook PC
With limited neural resources statistics of motor tasks motor performance
![Page 30: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/30.jpg)
Phase relationships and symmetry bias
![Page 31: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/31.jpg)
Multi-sensory integration
• CNS – In general the relative weightings of the senses is
sensitive to their direction dependent variability– Represents the distribution of tasks– Estimates its own sensory uncertainty– Combines these two sources in a Bayesian way
• Supports an optimal integration model
![Page 32: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/32.jpg)
Loss Functions in Sensorimotor system
What is the performance criteria (loss, cost, utility, reward)?
• Often assumed in statistics & machine learning – that we wish to minimize squared error for analytic or algorithmic tractability
• What measure of error does the brain care about?
Target Position
Post
erio
rPr
obab
ility
PriorLikePosterior lihood
P(sensory iP (state|sensory inpu nput|state) P(statet) )∝
[ ] ( , ) ( | _ ) _
ˆ ( ) arg min [ ]B actions
E Loss Loss state action P state sensory input dsensory input
Bayes estimator x s E Loss
=
=∫
![Page 33: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/33.jpg)
Loss function
312 2
Target
Scenario 1 Scenario 2
( )f error
2Loss error=
12Loss error=
Loss error=
Loss=4+4=8 Loss=1+9=10
Loss=2+2=4 Loss=1+3=4
Loss=1+1.7=2.7Loss=1.4+1.4=2.8
![Page 34: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/34.jpg)
Virtual pea shooter
MeanStarting location
Position (cm)Prob
abili
ty
-0.2 0 0.2
(Körding & Wolpert, PNAS, 2004)
![Page 35: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/35.jpg)
Probed distributions and optimal means
Possible Loss functions
2Loss ( )error=MEAN
-2 -1 0 1 2Error (cm)
ρ=0.2
ρ=0.3
ρ=0.5
ρ=0.8
Distributions
MODEMaximize Hits
-2 -1 0 1 2Error (cm)
MEDIANLoss error=
Robust estimator
![Page 36: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/36.jpg)
Shift of mean against asymmetry (n=8)
Mean squared error with robustness to outliers
![Page 37: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/37.jpg)
0.1 1.0 1.3 1.5 1.7 1.9 2.1 2.3 2.5 2.7 2.9 3.1 3.3 3.5 3.7 3.9α=
Loss ierrorα=∑
Personalised loss function
![Page 38: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/38.jpg)
Bayesian decision theory
Increasing probability of avoiding keeper
Increasing probability of being within the net
![Page 39: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/39.jpg)
Imposed loss function (Trommershäuser et al 2003)
0 -100 -500+100+100 +100
![Page 40: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/40.jpg)
Optimal performance with complex regions
![Page 41: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/41.jpg)
State estimation• State of the body/world
– Set of time-varying parameters which together with• Dynamic equations of motion• Fixed parameters of the system (e.g. mass)
– Allow prediction of the future behaviour
• Tennis ball– Position– Velocity– Spin
![Page 42: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/42.jpg)
State estimation
NOISE NOISE
Observer
![Page 43: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/43.jpg)
Kalman filter• Minimum variance estimator
– Estimate time-varying state– Can’t directly observe state but only measurement
1t t t tA B+ = + +x x u w
1t t tC+ = +y x v
1ˆ ˆ ˆ[ ]t t t t t tA B K C+ = + + −x x u y x
![Page 44: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/44.jpg)
State estimation
1
Forward DynamicModel
ˆ ˆ ˆ[ ]t t t t t tA B K C+ = + + −x x u y x
![Page 45: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/45.jpg)
Kalman Filter
• Optimal state estimation is a mixture– Predictive estimation (FF)– Sensory feedback (FB)
![Page 46: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/46.jpg)
Eye position
Location of object based on retinal location and gaze direction
PerceptActual
Motor command FM Eye
Position
![Page 47: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/47.jpg)
Sensory likelihood
(Wolpert & Kawato, Neural Networks 1998Haruno, Wolpert, Kawato, Neural Computation 2001)
P (sensorP(state| y input|ssensory tateinput) P(st )) ate∝
![Page 48: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/48.jpg)
Sensory predictionOur sensors report• Afferent information: changes in outside world• Re-afferent information: changes we cause
+ =
Internal source
Externalsource
![Page 49: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/49.jpg)
TicklingSelf-administered tactile stimuli rated as less ticklish than externally administered tactile stimuli. (Weiskrantz et al, 1971)
![Page 50: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/50.jpg)
Does prediction underlie tactile cancellation in tickle?
0
0.5
1
1.5
2
2.5
3
Self-producedtactile stimuli
Robot-producedtactile stimuli
Condition
Tic
kle
ratin
g ra
nk
Gain control or precise spatio-temporal prediction?
P<0.001
![Page 51: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/51.jpg)
Spatio-temporal prediction
0
0.5
1
1.5
2
2.5
3
3.5
4
Delay 0ms Delay100ms
Delay200ms
Delay300ms
Robot-produced
tactilestimuliCondition
Tic
kle
ratin
g ra
nk
P<0.001
P<0.001
0
0.5
1
1.5
2
2.5
3
3.5
4
0 degrees 30 degrees 60 degrees 90 degrees External
Condition
Tic
kle
ratin
g ra
nk
P<0.001P<0.001
(Blakemore, Frith & Wolpert. J. Cog. Neurosci. 1999)
![Page 52: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/52.jpg)
The escalation of force
![Page 53: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/53.jpg)
Tit-for-tat
Force escalates under rules designed to achieve parity: Increase by ~40% per turn
(Shergill, Bays, Frith & Wolpert, Science, 2003)
![Page 54: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/54.jpg)
Perception of force
70% overestimate in force
![Page 55: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/55.jpg)
Perception of force
![Page 56: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/56.jpg)
Labeling of movements
Large sensorydiscrepancy
![Page 57: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/57.jpg)
Defective prediction in patients with schizophrenic
• The CNS predicts sensory consequences
• Sensory cancellation in Force production
• Defects may be related to delusions of control
Patients Controls
![Page 58: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/58.jpg)
Motor LearningRequired if:
• organisms environment, body or task change• changes are unpredictable so cannot be pre-specified • want to master social convention skills e.g writing
Trade off between: – innate behaviour (evolution)
• hard wired• fast• resistant to change
– learning (intra-life)• adaptable• slow• Maleable
![Page 59: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/59.jpg)
Motor Learning
Actual behaviour
![Page 60: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/60.jpg)
Predicted outcome can be compared to actual outcome to generate an error
Supervised learning is good for forward models
![Page 61: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/61.jpg)
Weakly electric fish (Bell 2001)Produce electric pulses to • recognize objects in the dark or in murky habitats• for social communication.
The fish electric organ is composed of electrocytes, • modified muscle cells producing action potentials • EOD = electric organ discharges• Amplitude of the signal is between 30 mV and 7V • Driven by a pacemaker in medulla, which triggers each discharge
![Page 62: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/62.jpg)
Sensory filteringSkin receptors are derived from the lateral line system
Removal of expected or predicted sensory input is one of the very general functions of sensory processing. Predictive/associative mechanisms for changing environments
![Page 63: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/63.jpg)
Primary afferent terminate in cerebellar-like structures
Primary afferents terminate on principal cells either directly or via interneurons
![Page 64: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/64.jpg)
Block EOD discharge with curare
Specific for Timing (120ms), Polarity, Amplitude & Spatial distribution
![Page 65: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/65.jpg)
Proprioceptive Prediction
Tail bend affects feedbackPassive Bend phase locked to stimulus:
Bend
![Page 66: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/66.jpg)
Learning ruleChanges in synaptic strength requires principal cell spike discharge
Change depends on timing of EPSP to spike
Anti-Hebbian learning
T1
T2
T2-T1
• Forward Model can be learned through self-supervised learning• Anti-hebbian rule in Cerebellar like structure of he electric fish
![Page 67: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/67.jpg)
Motor planning (what is the goal of motor control)
Duration Hand Trajectory
Joint Muscles
• Tasks are usually specified at a symbolic level• Motor system works at a detailed level, specifying muscle activations• Gap between high and low-level specification• Any high level task can be achieved in infinitely many low-level ways
![Page 68: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/68.jpg)
Eye-saccades Arm- movements
Motor evolution/learning results in stereotypyStereotypy between repetitions and individuals
Time (ms)
• Main sequence• Donder’s law• Listings Law
• 2/3 power law• Fitts’ law
![Page 69: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/69.jpg)
Models
HOW models– Neurophysiological or black box models– Explain roles of brain areas/processing units in
generating behaviorWHY models
– Why did the How system get to be the way it is? – Unifying principles of movement production
• Evolutionary/Learning– Assume few neural constraints
![Page 70: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/70.jpg)
The Assumption of OptimalityMovements have evolved to maximize fitness
– improve through evolution/learning– every possible movement which can achieve a task has a cost– we select movement with the lowest cost
Overall cost = cost1 + cost2 + cost3 ….
![Page 71: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/71.jpg)
Optimality principles
• Parsimonious performance criteriaElaborate predictions
• Requires– Admissible control laws– Musculoskeletal & world model– Scalar quantitative definition of task
performance – usually time integral of f(state, action)
![Page 72: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/72.jpg)
Open-loop
• What is the cost– Occasionally task specifies cost
• Jump as high as possible• Exert maximal force
– Usually task does not specify the cost directly• Locomotion well modelled by energy minimization• Energy alone is not good for eyes or arms
![Page 73: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/73.jpg)
What is the cost?
Saccadic eye movements
• little vision over 4 deg/sec• frequent 2-3 /sec• deprives us of vision for 90
minutes/day
⇒Minimize time
![Page 74: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/74.jpg)
Arm movementsMovements are smooth – Minimum jerk (rate of change of acceleration) of the hand
(Flash & Hogan 1985)
2 2
04 5 3
0 04 5 3
0 0
( ) ( )
( ) ( )(15 6 10 )
( ) ( )(15 6 10 )/
T
T
T
Cost x t y t dt
x t x x x
y t y y yt T
τ τ τ
τ τ ττ
= +
= + − − −
= + − − −=
∫
![Page 75: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/75.jpg)
Smoothness• Minimum Torque change (Uno et al, 1989)
Shouldertorque
Elbowtorque2 2
0( ) ( )
T
s eCost t t dtτ τ= +∫
![Page 76: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/76.jpg)
The ideal cost for goal-directed movement
• Makes sense - some evolutionary/learning advantage• Simple for CNS to measure• Generalizes to different systems
– e.g. eye, head, arm
• Generalizes to different tasks – e.g. pointing, grasping, drawing
→ Reproduces & predicts behavior
![Page 77: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/77.jpg)
Motor command noise
MotorSystem
Noise
Position
Error minimized byrapidity
![Page 78: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/78.jpg)
Fundamental constraint=Signal-dependent noise
• Signal-dependent noise: – Constant coefficient of variation– SD (motor command) ~ Mean (motor command)
• Evidence from– Experiments: SD (Force) ~ Mean (Force)– Modelling
• Spikes drawn from a renewal process• Recruitment properties of motor units
(Jones, Hamilton & Wolpert , J. Neurophysiol., 2002)
![Page 79: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/79.jpg)
Task optimization in the presence of SDN
Given SDN, Task Given SDN, Task ≡≡ optimizing optimizing f(statisticsf(statistics))
An average motor command ⇒ probability distribution (statistics) of movement.
Controlling the statistics of actionControlling the statistics of action
![Page 80: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/80.jpg)
Finding optimal trajectories for linear systems
2 2 2( ) ( )u t k u tσ =
time
TM
x(t)0
[ ( )] ( ) ( )M
E x M u p M d Aτ τ τ= − =∫
2
0 0
2 2 2 2
0 0
[ ( )] [ ( ) ( )] [ ( )] ( )
( ) ( ) ( ) ( )
M M
M M
Var x T Var u p T d Var u p T d
k u p T d w u d
τ τ τ τ τ τ
τ τ τ τ τ τ
= − = −
= − =
∫ ∫∫ ∫
time
p(t)
Impulse responseSignal-dependent noise
( ) ( )
0[ ( )] ( ) ( ) 0
Mn nE x M u p M dτ τ τ= − =∫
Cost
Constraints
System
A
Linear constraints with quadratic cost:can use quadratic programming or isoperimetric optimization
![Page 81: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/81.jpg)
Saccade predictions
SDN
Motor command
Jerk
3rd order linear system
![Page 82: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/82.jpg)
0 100 2000
20
Time (ms)
Deg
rees
Prediction: very slow saccade
22 degree saccade in 270 ms (normally ~ 70 ms)
![Page 83: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/83.jpg)
Head free saccade
0 100 200 300 400 500 600 700 8000
20
40
60
80
100
120
Deg
rees
Time (ms)
Head
Gaze
Eye
Τ1=0.3 Τ2=0.3
Τ1=0.15 Τ2=0.08 Free parameter, eye:head noise
0 100 200 300 400 500 600 700 8000
20
40
60
80
100
120
Time (ms)
Deg
rees
Eye
Head
Gaze
(Tomlinson & Bahra, 1986)
![Page 84: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/84.jpg)
Coordination: Head and eyeFor a fixed duration (T), Var(A)=k A2
Var(A)=k A2 Var(A)= k (A/2)2 + k (A/2)2
= k A2 /2Var(A)=k A2
Eye only Head onlyEye &Head
![Page 85: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/85.jpg)
Movement extent vs. target eccentricity
Gaze amplitude
Ang
ular
dev
iatio
n at
ac
quis
ition
Eye
Head
Gaze amplitude
0 20 40 60 80 100 120 1400
10
20
30
40
50
60
70
80
90
100
0 100 200 300 400 500 600 700 8000
20
40
60
80
100
120
} }Head
Eye
![Page 86: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/86.jpg)
Arm movements
Drawing (⅔ power law) f=path errorObstacle avoidance f= limit probability of collision
Smoothness
Non smooth movement ⇒ requires abrupt change in velocity⇒ given low pass system⇒ large motor command⇒ increased noise
Smoothness ⇒ accuracy
Feedforward control•Ignores role of feedback•Generates desired movements•Cannot model trial-to-trial variability
![Page 87: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/87.jpg)
Optimal feedback control (Todorov 2004)
– Optimize performance over all possible feedback control laws– Treats feedback law as fully programmable
• command=f(state)• Models based on reinforcement learning optimal cost-to-go functions• Requires a Bayesian state estimator
![Page 88: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/88.jpg)
Minimal intervention principle
• Do not correct deviations from average behaviour unless they affect task performance– Acting is expensive
• energetically• noise
– Leads to • uncontrolled manifold• synergies
Uncontrolled manifold
![Page 89: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/89.jpg)
Optimal control with SDN• Biologically plausible theoretical underpinning
for both eye, head, arm movements
• No need to construct highly derived signals to estimate the cost of the movement
• Controlling statistics in the presence of noise
![Page 90: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/90.jpg)
What is being adapted?
• Possible to break down the control process:
• Visuomotor rearrangements• Dynamic perturbations• [timing, coordination , sequencing]
• Internal models captures the relationship between sensory and motor variables
![Page 91: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/91.jpg)
Altering dynamics
![Page 92: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/92.jpg)
Altering Kinematics
![Page 93: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/93.jpg)
Representation of transformations
Look-upTable
Non-physicalParametersθ=f(x,ω)
PhysicalParametersθ=acos(x/L)
High storageHigh flexibility
Low Generalization
Low storageLow flexibility
High Generalization
θ
x
L ÷asin
x
θ
L
x
θ
x θ1 103 35. .. .
![Page 94: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/94.jpg)
Generalization paradigm
• Baseline– Assess performance over
domain of interest – (e.g. workspace)
• Exposure– Perturbation: New task– Limitation: Limit the
exposure to a subdomain
• Test– Re-assess performance
over entire domain of interest
![Page 95: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/95.jpg)
Difficulty of learning
(Cunningham 1989, JEPP-HPP)
• Rotations of the visual field from 0—180 degrees
Difficulty• increases from 0 to 90• decreases from 120 to 180
• What is the natural parameterization
![Page 96: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/96.jpg)
Viscous curl field
(Shadmehr & Mussa-Ivaldi 1994, J. Neurosci.)
![Page 97: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/97.jpg)
Representation from generalization: Dynamic
1. Test: Movements over entire workspace
2. Learning– Right-hand workspace – Viscous field
3. Test: Movements over left workspace
Two possible interpretations force = f(hand velocity) or torque=f(joint velocity)
Joint-based learning of dynamics
(Shadmehr & Mussa-Ivaldi 1994, J. Neurosci.)
Left hand workspaceBefore After with Cartesian field
![Page 98: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/98.jpg)
Visuomotor coordinates
z
yx
(x,y,z)
Cartesian
z
y
x
Spherical Polar
θφ
r(r,φ,θ)
α1
α2
α3
(α1,α2,α3)
Joint angles
![Page 99: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/99.jpg)
Representation- Visuomotor1. Test: Pointing accuracy to a set of targets
2. Learning– visuomotor remapping– feedback only at one target
3. Test: Pointing accuracy to a set of targets
-50
-40
-30
y = 22.2/16.2 cm y = 29.2/26.2 cm y = 36.2 cm
-20 -10 0 10 20
20
30
40z = -43.6 cm z = -35.6 cm
-20 -10 0 10 20
z = -27.6 cm
-20 -10 0 10 20
x (cm)
Prediction of eye-centred spherical coordinates
(Vetter et al, J. Neurophys, 1999)
Predictions of eye-centred spherical coordinates
• Generalization paradigms can be used to assess– Extent of generalization– Coordinate system of transformations
![Page 100: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/100.jpg)
Altering dynamics: Viscous curl fieldBefore Early with force
Late with force Removal of force
![Page 101: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/101.jpg)
A muscle activation levels sets the spring constant k (or resting length) of the muscle
Stiffness control
Equilibrium point
![Page 102: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/102.jpg)
Equilibrium point control
• Set of muscle activations (k1,k2,k3…) defines a posture • CNS learns a spatial mapping
– e.g. hand positions muscle activations(x,y,z) (k1,k2,k3…)
![Page 103: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/103.jpg)
Equilibrium control
The hand stiffness can vary with muscle activation levels.
Low stiffness High stiffness
![Page 104: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/104.jpg)
Controlling stiffness
Burdet et al (Nature, 2002)
![Page 105: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/105.jpg)
Stiffness ellipses
• Internal models to learn stable tasks• Stiffness for unpredictable tasks
![Page 106: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/106.jpg)
Summary– Sensorimotor integration
• Static multi-sensory integration• Bayesian integration• Dynamic sensor fusion & the Kalman filter
– Action evaluation• Intrinsic loss function• Extrinsic loss functions
– Prediction• Internal model and likelihood estimation • Sensory filtering
– Control• Optimal feed forward control• Optimal feedback control
– Motor learning of predictable and stochastic environments
Wolpert-lab papers on www.wolpertlab.com
![Page 107: Wolpert Slides on Machine learning](https://reader034.fdocuments.net/reader034/viewer/2022051517/563dba4d550346aa9aa46ed9/html5/thumbnails/107.jpg)
References• Bell, c.(2001) Memory-based expectations in electrosensory systemsCurrent Opinion in
Neurobiology 2001, 11:481–487• Burdet, E., R. Osu, et al. (2001). "The central nervous system stabilizes unstable dynamics by
learning optimal impedance." Nature 414(6862): 446-9.• Cunningham, H. A. (1989). "Aiming error under transformed spatial maps suggest a structure for
visual-motor maps." J. Exp. Psychol. 15:3: 493-506.• Ernst, M. O. and M. S. Banks (2002). "Humans integrate visual and haptic information in a
statistically optimal fashion." Nature 415(6870): 429-33.• Flash, T. and N. Hogan (1985). "The co-ordination of arm movements: An experimentally
confirmed mathematical model " J. Neurosci. 5: 1688-1703.• Shadmehr, R. and F. Mussa-Ivaldi (1994 ). "Adaptive representation of dynamics during learning
of a motor task." J. Neurosci. 14:5: 3208-3224.• Todorov, E. (2004). "Optimality principles in sensorimotor control." Nat Neurosci 7(9): 907-15.• Trommershauser, J., L. T. Maloney, et al. (2003). "Statistical decision theory and the selection of
rapid, goal-directed movements." J Opt Soc Am A Opt Image Sci Vis 20(7): 1419-33.• Uno, Y., M. Kawato, et al. (1989). "Formation and control of optimal trajectories in human
multijoint arm movements: Minimum torque-change model " Biological Cybernetics 61: 89-101.• van Beers, R. J., A. C. Sittig, et al. (1998). "The precision of proprioceptive position sense." Exp
Brain Res 122(4): 367-77.• Weiskrantz, L., J. Elliott, et al. (1971). "Preliminary observations on tickling oneself." Nature
230(5296): 598-9.