Human Neural Machine
-
Upload
george-spithourakis -
Category
Education
-
view
327 -
download
0
Transcript of Human Neural Machine
Voice of the Machine-
Human Neural NetworkGeorgios SpithourakisPhD Candidate, UCL
Part 1Voice of the Machine
Humans Describe What They See
Encoding Decoding
???
Information Processing
Information Processing
Machines Describe What they See
Encoding Decoding
???
Encoding an Image• Convolutional Neural Networks (CNNs)
Encoding an Image• Convolutional Neural Networks (CNNs)
Layer 1 Layer 2
Encoding an Image• Convolutional Neural Networks (CNNs)
Layer 1 Layer 2
Generating Text (decoding) (1)
Nude Descending a Staircase,Duchamp, 1912
Generating Text (decoding) (1)Toe upon ___, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind.
We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __.
One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
Generating Text (decoding) (1)Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the ____ With nothing on. Nor on her mind.
We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __.
One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
Generating Text (decoding) (1)Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind.
We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging ___ That parts to let her parts go __.
One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
Generating Text (decoding) (1)Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind.
We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go __.
One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
Generating Text (decoding) (1)Toe upon toe, a snowing flesh, A gold of lemon, root and rind, She sifts in sunlight down the stairs With nothing on. Nor on her mind.
We spy beneath the banister A constant thresh of thigh on thigh-- Her lips imprint the swinging air That parts to let her parts go by.
One-woman waterfall, she wears Her slow descent like a long cape And pausing, on the final stair Collects her motions into shape.Nude Descending a Staircase,
Duchamp, 1912 X. J. Kennedy (1961)
• Recurrent Neural Networks (RNNs)
Input
Output
Generating Text (decoding) (2)
<START> Toe upon
Toe upon toe
• Recurrent Neural Networks (RNNs)
Input
Output
Generating Text (decoding) (2)
<START> Toe upon
Toe upon toe
• Recurrent Neural Networks (RNNs)
Input
Output
Generating Text (decoding) (2)
<START> Toe upon
Toe upon toe
Aardvark 0.1%
.
.
.
Toe 20%
.
.
.
Zebra 0.2%
Part 2Human Neural Network
Exercise Goals• Humans become a machine• Each group is a neural machine• Each individual is a neuron
• Adaptation• Communicate in words (not numbers)• More flexibility
Encoding an image• Describe what you see• Pieces of an image• Objects in a scene• A story for the scene
Encoding an Image – Pieces• For each piece, choose 3 words that describe its content
BlueUniformEmpty
SeaBlueCalm
SkyCloudSunny
CloudSkyStick
SkyCloudCotton
TightropeManBalancing
CloudsSkyLines
GreyCloudTripod
CityLandscapeRiver
SkyscrapersLandscapeHorizon
CornerBuildingsWindow
PuddleMirrorPavement
RiversideTownPark
BuildingsRoadsBelow
CityBrownishMetal
RiverBrownCity
Encoding an Image – Objects (individual)• Group together pieces to identify up to 5 objects• Describe each with 1 word• A story should start forming
BlueUniformEmpty
SeaBlueCalm
SkyCloudSunny
CloudSkyStick
SkyCloudCotton
TightropeManBalancing
CloudsSkyLines
GreyCloudTripod
CityLandscapeRiver
SkyscrapersLandscapeHorizon
CornerBuildingsWindow
PuddleMirrorPavement
RiversideTownPark
BuildingsRoadsBelow
CityBrownishMetal
RiverBrownCity
Sky
Man
Roof
City
Encoding an Image – Objects (group)• Reach an agreement as a group (up to 5 objects, 1 word each)
Sky
Man Tightrope
Roof
Town
Sky
Someone
Reflection
Town
Sky
Man Tightrope
Roof
City
Encoding an Image – Scene/Story• Decide on story of up to 5 words
Manwalkstightropeabovetown
Sky
Man Tightrope
Roof
Town
Man walks tightrope above town
BlueUniformEmpty
SeaBlueCalm
SkyCloudSunny
CloudSkyStick
SkyCloudCotton
TightropeManBalancing
CloudsSkyLines
GreyCloudTripod
CityLandscapeRiver
SkyscrapersLandscapeHorizon
CornerBuildingsWindow
PuddleMirrorPavement
RiversideTownPark
BuildingsRoadsBelow
CityBrownishMetal
RiverBrownCity
Sky
Man Tightrope
Roof
Town
The Encoded Image
Generating the Poem (Decoding)• React to what we saw• Write a poem word-by-word• Individually propose alternative continuations• Collaboratively select one
Decoding – Choose First Word
I
He
The
High
I 4
He 1
The 0
High 1
• Each person proposes 1 word
• Each person votes (up to 2 votes) for ‘best’ word
• Cannot vote yourself!
• Count votes
• Write highest scoring word to poem
• If tied, repeat voting only between tied words (or flip coin)
POEMI
Decoding – Choose Next Word
really
only
walk
stand
really 3
only 0
walk 0
stand 1
• Each person proposes 1 word
• Each person votes (up to 2 votes) for ‘best’ word
• Cannot vote yourself!
• Count votes
• Write highest scoring word to poem
• If tied, repeat voting only between tied words (or flip coin)
POEMI really
Decoding – Choose Next Word
knowing
walking
trotting
standing
knowing 0
walking 3
trotting 0
standing 1
• Each person proposes 1 word
• Each person votes (up to 2 votes) for ‘best’ word
• Cannot vote yourself!
• Count votes
• Write highest scoring word to poem
• If tied, repeat voting only between tied words (or flip coin)
POEMI really had an easy way of walking
Decoding – Speed it up• One word at a time is too slow for humans…
• Propose whole phrases (as many words as you like)
Decoding – Choose Next Phrase
However tall
I never thought
Looking at a small
Despite
However tall 0
I never thought 1
Looking at a small 3
Despite 1
• Each person proposes 1 phrase
• Each person votes (up to 2 votes) for ‘best’ phrase
• Cannot vote yourself!
• Count votes
• Write highest scoring phrase to poem
• If tied, repeat voting only between tied words (or flip coin)
POEMI really had an easy way of walkingLooking at a small
Part 3Conclusion
Image: Man leads caravan through desert
The camel holds the hand of the poor manHe’ll watch us tread, he’ll watch us fall
The long shadow of a donkey across the cracked sandShabbily clad but standing tall
We each of us must go, all
Surrounded by another expedition, Across an ocean on the sea of sand,
Mountains gaze upon a vast Egyptian, A field of sand beneath the silver strand.
Image: Man free falls to ground
We gotta hide behind the Beaver lake! I wanna know a better place or where,
Another day a little kiss and take, An angel on the other side of there.
The forest and cliffs standing against the skyA human being in free fall
Sailing through the air like a flyCome to me, the grass, a call.
Image: Crowd watches fishes at aquarium
In a box of Nothingfar from the deep
where waves are thrustingwhere light goes to sleep
Expecting something from an empty zoo, Surrounded by an ocean full of fish,
On the other side of me and you, Beneath the carpet like a jellyfish.
Image: Two men round a campfireA living fire becomes a doubles title.
To stay protected by the sons of men, We stuck together like a semi final,
The one and two and three or four of ten.
Marshmallows at dawnFreshly cut wood burns in the fire
Time slips out a wide yawnThey all sing out in choir
Acknowledgements• Zena Edwards, CV:iD• Daniela Paolucci, Apples and Snakes
• Sebastian Riedel, UCL• Piotr Mirowski, HumanMachine/Deepmind• Mandana Seyfeddinipur, SOAS
• Marjan Ghazvininejad, USC• Generating Topical Poetry, EMNLP 2016
Thank you!