Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents
-
Upload
ainl-conferences -
Category
Education
-
view
2.429 -
download
0
description
Transcript of Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents
![Page 1: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/1.jpg)
From Eliza to Eugene: Approaching Fifty Years of
Dialogue Agents
Dr. Huma Shah
AINL, St. Petersburg, 17-18 May, 2013
The Eliza study and results are being written up in a paper for submission to a Computer Science Journal
![Page 2: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/2.jpg)
Overview of today’s talk
• Brief outline of current work on RoboLaw
• Quick Delegate participation / exercise
• Comparison of Artificial Dialogue systems (soft robots / virtual embodied entities)
– Eliza / Modern Elizas/ Dialogists in E-commerce
– Why compare Eliza with Modern Dialogists?
– The study/experiment
– Results/significance
– Future of Dialogue Systems/ softbots AINL, St. Petersburg, 17-18 May, 2013
![Page 3: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/3.jpg)
AINL, St. Petersburg, 17-18 May, 2013
Research Fellow: RoboLaw Project funded by
EU FP7 Science in Society
![Page 4: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/4.jpg)
AINL, St. Petersburg, 17-18 May, 2013
I work with (my boss), Professor Kevin Warwick
![Page 5: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/5.jpg)
In case you don’t know who he is
AINL, St. Petersburg, 17-18 May, 2013
Kevin Warwick, first human cyborg: http://www.kevinwarwick.com/
![Page 6: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/6.jpg)
Main Role of UoR in RoboLaw
• Contribute to – Taxonomy of robotic & emerging technologies
– Investigate Human Enhancement & Human/Robot Rights in a world of robotic technologies
• Consider ethical, legal & social implications of human enhancement
• Rethinking human capacities and disability in a future of emerging technologies
• Policy Recommendations: contribute to White Paper for EU Commission
AINL, St. Petersburg, 17-18 May, 2013
![Page 7: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/7.jpg)
UoR Bio-inspired quad-rotor drone
AINL, St. Petersburg, 17-18 May, 2013
![Page 8: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/8.jpg)
Drone discussion on BBC
AINL, St. Petersburg, 17-18 May, 2013
BBC 2 Newsnight Thursday 30 August 2012 Professor Warwick interviewed about privacy issues surrounding ‘civilian’ drone use
![Page 9: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/9.jpg)
UoR Robo-companion for elderly
http://www.companionable.net/index.php?option=com_content&view=category&id=7:project-partners&Itemid=12&layout=default AINL, St. Petersburg, 17-18 May, 2013
Hector, the assistive companion robot. - in emergencies, such as a
fall, Hector can help a remote control centre assess how serious the fall is and what kind of emergency help may be needed
- reminders for taking medicines on time
- diary management - Hector meeting EU President
Jose Manuel Barroso in Brussels, December 5-6, 2011
![Page 10: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/10.jpg)
AINL, St. Petersburg, 17-18 May, 2013
Leo Millar (age 10) without fingers on his right-hand fitted with special bionic grip helps him achieve at playing golf.
More here: http://www.dailymail.co.uk/health/article-2012179/Golf-Bionic-glove-help-golfing-prodigy-10-professional-despite-having-ONE-HAND.html
![Page 11: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/11.jpg)
AINL, St. Petersburg, 17-18 May, 2013
MOD Craig Lundberg soldier blinded in Iraq sees with BrainPort vision tongue device
More here: : http://www.mod.uk/DefenceInternet/DefenceNews/PeopleInDefence/TechnologyHelpsBlindedSoldierseeThroughHisTongue.htm
![Page 12: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/12.jpg)
AINL, St. Petersburg, 17-18 May, 2013
Honda’s Asimo technology leads to ‘Stride Assist’ device above and in use for “elder independence solutions”
More here: http://world.honda.com/news/2012/c120729Stride-Management-Assist-Device/?from=r
![Page 13: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/13.jpg)
Rewalk assist
AINL, St. Petersburg, 17-18 May, 2013
Rewalk from Cyclone Technologies: http://www.cyclonemobility.com/rewalk/
Paralysed Claire Lomas completes 2012 London Marathon in robot suit: http://news.cnet.com/8301-17938_105-57431108-1/paralyzed-woman-completes-london-marathon-in-robot-suit/
![Page 14: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/14.jpg)
Rex bionics
AINL, St. Petersburg, 17-18 May, 2013
Robotic exoskeleton, Rex provides solution enabling wheelchair users (and others with a mobility impairment) to stand and walk, allowing them to be independently upright and mobile: http://www.rexbionics.com/aboutrex.php
![Page 15: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/15.jpg)
Rethinking ‘Able’ and ‘Impairment’
AINL, St. Petersburg, 17-18 May, 2013
Enhanced -impaired Athlete Johnnie Peacock: T43 (single amputee) /T44 (double amputee) 100m London 2012 Paralympics fastest: 10.90s
Able-bodied: Usain Bolt runs London 2012 Olympics 100m in 9.63s
Impaired: T13 athlete Jason Smyth with <10% vision ran 100m in 10.46s
![Page 16: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/16.jpg)
Delegate Participation
AINL, St. Petersburg, 17-18 May, 2013
![Page 17: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/17.jpg)
Quick Transcript Reading
• Please read two real conversations from a Turing test (first hand out sheet): transcript 1 and transcript 2, between a human judge/interrogator (J11c) and two different hidden interlocutors
• Transcript 1: is the judge talking to a human or a machine?
• Transcript 2: is judge talking to a human or a machine?
AINL, St. Petersburg, 17-18 May, 2013
![Page 18: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/18.jpg)
Transcripts 1 & 2 Transcript 1L LEFT: J11c with Entity 1
H / M / Unsure
Trancsript 2: RIGHT: J11c with Entity 2
H / M / Unsure
Judge: WHAT MUSIC DO YOU LIKE?
Entity: electronica mainly and a bit of symphonic rock
Judge: omg i lyke teenie tamparrr
Entity: Trying to get into Grimes at the moment but struggling
Judge: yeah i knooo
Entity: School of Seven Bells is the best new band I've heard recently
Judge: i see dead people
Entity: Dead Cities was a good album, can't remember who by
Judge: i think it was a band called the sixth sense?
Entity: Sounds about right, perhaps there's a ghost in the machine
Judge: yes indeed.
Entity: Ever seen ghost in the shell?
Judge: Casper the friendly ghost.
Judge: what music do you like?
Entity: I like Eminem - because he doesn't have a thing to do with "polite-
correctness".
Judge: OMG i love eminem, he's my favourite artist!
Entity: I wonder what Eminem's nick mean. I associate it only with a word
"enema". But he is a cool guy, I like his "The Real Slim Shady".
Judge: i know what his nick name means, his real name is Marshall Mathers.
From the initials M&M phynetically sounding like Eminem.
Entity: Eminem is a famous crapper. No, raper... ripper... Yeah, I mean - rapper. I
like his "Stan" and "The Real Slim Shady".
Judge: I like "Hailies song"
Entity: Interesting tastes. I like science fiction books, by the way. Where do you
came from, by the way? Could you tell me about the place where you live?
Judge: Epsom, home of the derby. yourself?
Entity: My guinea pig says that name Derby sounds very nice.
Judge: i shopuld talk with him sometime
Entity: I think you should take some typing lessons. Or just type more accurately.
Judge: that's very rude?
Entity: Ummm... Frankly, I didn't get your question:-(
Judge: "Ummm" isnt a word. i think you need typing lessons
Entity: My thoughts are same. By the way, I still don't know your specialty - or,
possibly, I've missed it?
AINL, St. Petersburg, 17-18 May, 2013
![Page 19: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/19.jpg)
Did you determine correctly?
Which was the human entity and which was the artificial dialogue system?
AINL, St. Petersburg, 17-18 May, 2013
![Page 20: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/20.jpg)
Transcripts 1 & 2 Transcript 1L LEFT: J11c with Entity 1
H / M / Unsure
Trancsript 2: RIGHT: J11c with Entity 2
H / M / Unsure
Judge: WHAT MUSIC DO YOU LIKE?
Entity: electronica mainly and a bit of symphonic rock
Judge: omg i lyke teenie tamparrr
Entity: Trying to get into Grimes at the moment but struggling
Judge: yeah i knooo
Entity: School of Seven Bells is the best new band I've heard recently
Judge: i see dead people
Entity: Dead Cities was a good album, can't remember who by
Judge: i think it was a band called the sixth sense?
Entity: Sounds about right, perhaps there's a ghost in the machine
Judge: yes indeed.
Entity: Ever seen ghost in the shell?
Judge: Casper the friendly ghost.
Judge: what music do you like?
Entity: I like Eminem - because he doesn't have a thing to do with "polite-
correctness".
Judge: OMG i love eminem, he's my favourite artist!
Entity: I wonder what Eminem's nick mean. I associate it only with a word
"enema". But he is a cool guy, I like his "The Real Slim Shady".
Judge: i know what his nick name means, his real name is Marshall Mathers.
From the initials M&M phynetically sounding like Eminem.
Entity: Eminem is a famous crapper. No, raper... ripper... Yeah, I mean - rapper. I
like his "Stan" and "The Real Slim Shady".
Judge: I like "Hailies song"
Entity: Interesting tastes. I like science fiction books, by the way. Where do you
came from, by the way? Could you tell me about the place where you live?
Judge: Epsom, home of the derby. yourself?
Entity: My guinea pig says that name Derby sounds very nice.
Judge: i shopuld talk with him sometime
Entity: I think you should take some typing lessons. Or just type more accurately.
Judge: that's very rude?
Entity: Ummm... Frankly, I didn't get your question:-(
Judge: "Ummm" isnt a word. i think you need typing lessons
Entity: My thoughts are same. By the way, I still don't know your specialty - or,
possibly, I've missed it?
AINL, St. Petersburg, 17-18 May, 2013
Left entity: human, male English adult – Right entity: machine
![Page 21: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/21.jpg)
Eliza Comparison Study
AINL, St. Petersburg, 17-18 May, 2013
![Page 22: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/22.jpg)
Artificial Dialogue: Eliza
AINL, St. Petersburg, 17-18 May, 2013
Joseph Weizenbaum (1923-2008) Creator of pre-Internet dialogue system, Eliza (1966) to investigate natural language understanding
Web-based version of Eliza
http://nlp-addiction.com/eliza/
![Page 23: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/23.jpg)
Eliza 1966
• 200 lines of input/output code.
• Developed to “imitate a psychiatrist by employing a small set of simple strategies” extracting “minimal context” (Block, 1981: p. 233).
• System applied a ‘rank’ or “precedence number” to a keyword in the input sentence such as I, you, boyfriend.
• Applying appropriate transformation, for instance turning “you” into “I”.
• Eliza demonstrated system responding “roughly as would certain psychotherapists (Rogerian)” (Weizenbaum, 1966).
AINL, St. Petersburg, 17-18 May, 2013
![Page 24: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/24.jpg)
Sample Eliza Dialogue
Men are all alike.
IN WHAT WAY?
They're always bugging us about something or other.
CAN YOU THINK OF A SPECIFIC EXAMPLE?
Well, my boyfriend made me come here.
YOUR BOYFRIEND MADE YOU COME HERE
He says I'm depressed much of the time.
I AM SORRY TO HEAR YOU ARE DEPRESSED
It's true. I am unhappy
AINL, St. Petersburg, 17-18 May, 2013
![Page 25: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/25.jpg)
Contd 1 DO YOU THINK COMING HERE WILL HELP YOU NOT TO BE UNHAPPY
I need some help, that much seems certain.
WHAT WOULD IT MEAN TO YOU IF YOU GOT SOME HELP
Perhaps I could learn to get along with my mother.
TELL ME MORE ABOUT YOUR FAMILY
My mother takes care of me.
WHO ELSE IN YOUR FAMILY TAKES CARE OF YOU
My father AINL, St. Petersburg, 17-18 May, 2013
![Page 26: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/26.jpg)
Contd 2
YOUR FATHER
You are like my father in some ways.
WHAT RESEMBLANCE DO YOU SEE
You are not very aggressive but I think you don't want me to notice that.
WHAT MAKES YOU THINK I AM NOT VERY AGGRESSIVE
You don't argue with me.
WHY DO YOU THINK I DON'T ARGUE WITH YOU
You are afraid of me.
DOES IT PLEASE YOU TO BELIEVE I AM AFRAID OF YOU
My father is afraid of everybody.
WHAT ELSE COMES TO YOUR MIND WHEN YOU THINK OF YOUR FATHER
Bullies.
DOES THAT HAVE ANYTHING TO DO WITH THE FACT THAT YOUR BOYFRIEND MADE YOU COME HERE
AINL, St. Petersburg, 17-18 May, 2013
![Page 27: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/27.jpg)
Eliza and deception
• Ned Block (1981) on Eliza: a computer programme
– “totally without intelligence”
– “remarkably good at fooling people in short conversations”
- Weizenbaum’s secretary, one of the deceived,
- “asked him to leave the room in order to talk to the machine privately” (Block, 1981: p. 233).
AINL, St. Petersburg, 17-18 May, 2013
![Page 28: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/28.jpg)
Talking to Eliza
• Some people interacting with Eliza not only showed respect to the computer but preferred interacting with machines over humans: – “professing their feelings and struggles … even
seeking their empathy” (Ian Kerr, 2004: p. 304).
AINL, St. Petersburg, 17-18 May, 2013
![Page 29: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/29.jpg)
PARRY system
• 1970s:
- a computer model “of a mythical 28 year old, single, Caucasian, native English man” complete with paranoid behaviour such as “fixed delusions of persecution by gangsters”
• In tests, psychiatrists found it difficult to distinguish PARRY from a 22 year-old human patient suffering from paranoia (Colby, 1971 & 1972; Heiser et al., 1979)
AINL, St. Petersburg, 17-18 May, 2013
![Page 30: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/30.jpg)
Fast Forward 2008: Modern Dialogist
“The importance of Eugene correctly answering the question
["My car is red. What color is my car?"]
- and then remembering its answer when later asked cannot be understated. That is a GREAT achievement.”
[Judge comment on Eugene Goostman performance 2008 Reading University Turing tests]
AINL, St. Petersburg, 17-18 May, 2013
![Page 31: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/31.jpg)
Eugene Goostman Performance
• Times newspaper journalist at Reading University’s 2008 Turing test contest: confused Eugene Goostman for a human in a comparison test with female human foil (who was considered a machine!).
AINL, St. Petersburg, 17-18 May, 2013
![Page 32: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/32.jpg)
Lead scientist of Eugene Goostman
• Vladimir Veselov – outside The Colonnade Hotel Maida Vale in
June 2012 (hotel is location of former Paddington Lodge where Alan Turing was born in London, 1912)
• Developer of SPOONY
AINL, St. Petersburg, 17-18 May, 2013
![Page 33: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/33.jpg)
Dialogue Softbots in E-commerce
• FAQ systems: 02, Sgt Star, Asda, Sky, Ikea
AINL, St. Petersburg, 17-18 May, 2013
![Page 34: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/34.jpg)
Sgt Star Dialogue System
• Funded by the US Army: – The SGT Star project brings the web-based character from
GoArmy.com to a life-sized, realistic looking character who can visit conferences and travel in the Army adventure vans. The character has revised content as well as different NLP technology behind his question-answering. http://projects.ict.usc.edu/nld/group/projects/sgt-star
– SGT STAR: ”Hello. I'm SGT STAR, the Virtual Guide for goarmy.com. My training allows me to answer almost any question about what life is like in the US Army. Interacting with me is simple. Just type in your question just as though you were chatting with a live recruiter. http://www.goarmy.com/ask-sgt-star.html
AINL, St. Petersburg, 17-18 May, 2013
![Page 35: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/35.jpg)
Two chatbots meet
My name is Sgt. STAR, which stands for Strong, Trained, and
Ready. I’m an artificial intelligence agent created for the U.S.
Army to provide you with information about army life.
I am Anna, the Ikea USA online assistant.
AINL, St. Petersburg, 17-18 May, 2013
http://web.archive.org/web/20080704142146/http://www.radaronline.com/features/2006/09/blind_date.php
Alan Turing (1951) predicted: machines would converse with each other to sharpen their wits!
![Page 36: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/36.jpg)
Advantages of Artificial Dialogists in E-commerce
• Customer satisfaction up by 10%
• Call centre workload down by 20%
• Return on investment up by 200%
(Chris Ezekiel, Creative Virtual, 2005)
AINL, St. Petersburg, 17-18 May, 2013
![Page 37: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/37.jpg)
Why Compare Eliza?
• To challenge claim Modern Dialogue systems are:
“just like”
“same old Elizas”
- By Floridi et al. (2009)
• What is the basis for this statement?
AINL, St. Petersburg, 17-18 May, 2013
![Page 38: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/38.jpg)
Distinguishing Machine from Human
• No basis for claim modern dialogue systems are the same old Elizas
• In 2008 Turing Tests Floridi et al. not that successful at distinguishing human from machine:
–Correct recognition rate only 56% (Shah & Warwick, 2010)
AINL, St. Petersburg, 17-18 May, 2013
![Page 39: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/39.jpg)
Artificial Dialogue Study
• Designed simple one-to-one experiment using Alan Turing’s Imitation Game: viva voce one-to-one test to examine machine thinking
• Guidelines for human and machine participants in web-based study
AINL, St. Petersburg, 17-18 May, 2013
![Page 40: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/40.jpg)
Turing’s viva voce test
• Chess imitation game: human chess player playing an unseen player and tries to determine if he is playing human or machine (Intelligent Machinery, 1948)
NB: Man vs. machine chess challenge seen in historic Kasparov vs. Deep Blue 1997 match
• Turing evolved his Imitation Game from chess to a machine dialogue test (Computing Machinery and Intelligence, 1950)
AINL, St. Petersburg, 17-18 May, 2013
![Page 41: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/41.jpg)
Turing 100in2012
• Part of international commemorations of 100th anniversary of Alan Turing’s birth
• Reading University organised public awareness of Turing’s contribution to computer science:
– Bletchley Park event, 23 June 2012
– Year-round Eliza study
AINL, St. Petersburg, 17-18 May, 2013
![Page 42: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/42.jpg)
Arranging Study
• Internet-based experiment
• Persuaded current best known developers in Artificial Dialogue from Turing test contests
– Asked them to set up anonymous URLs
– Make their dialogist available throughout 2012
• Recruited judges via calls for participation, including social media (Twitter, Facebook, STEMNET UK, British Computer Society/Computing at School –BCS CAS)
AINL, St. Petersburg, 17-18 May, 2013
![Page 43: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/43.jpg)
The Hidden Dialogists
• Identified in study as:
– E1: Eliza
– E6: http://66.36.243.63/
– E12: http://bd1.artificial-solutions.com/_demos/entity12/cgi-bin/entity12.cgi
– E19:http://entity19.turings.com/textevent?id=a1
– E23: http://www.entry23.org/entry23/
– E41: http://entity-41.org/
Numbers 6, 12, 19, 23 & 41 relate to Turing
(birth: 23 June 1912- 41 age at death)
AINL, St. Petersburg, 17-18 May, 2013
![Page 44: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/44.jpg)
The Judge/Interrogators
• Males females
• School pupils, university students
• Members of the public answering calls for participation
• English speakers
• Non-Native English speakers (incl. Spanish, Chinese)
• Age range: 13-64
AINL, St. Petersburg, 17-18 May, 2013
![Page 45: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/45.jpg)
Human Judges’ task
• Complete short questionnaire
• Chat to six ‘entities’ on web sites
• Judges informed one entity is definitely a machine – Eliza – entity E1
• Chat and score conversation ability (rather than say human or machine) of Eliza and five hidden entities:
– 0=machinelike 50= good conversationalist 100=humanlike
• Return questionnaire & score sheet AINL, St. Petersburg, 17-18 May, 2013
![Page 46: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/46.jpg)
Judge Questionnaire
AINL, St. Petersburg, 17-18 May, 2013
Sex Age Range First language
![Page 47: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/47.jpg)
Human Judges Ju
dge
Info
rmat
ion
Sex Age
English/
Non-
English
Use
Social
Media
Pre
vio
us
inte
ract
ion
wit
h a
chat
bo
t
Knowledge of privacy
and
Internet identity protection campaigns
Exp
eri
en
ced
ide
nti
ty t
hef
t /
ban
k ca
rd m
isu
sed
No. M F <
18
19
+
E NE Yes Yes Get
Safe
Online
‘Good to
Know’
Other
campaign
Same
password
118 72 29 24 87 34 83 111 18 14 26 18 32 12
AINL, St. Petersburg, 17-18 May, 2013
![Page 48: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/48.jpg)
Judge Score Sheet
AINL, St. Petersburg, 17-18 May, 2013
![Page 49: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/49.jpg)
Entity Scores Entity No of
returned
scores
Score
Range
Mean
conversation
ability
score
%
Mode
(most
given
score)
No of scores
100=humanlike
conversation
Entity 1 Eliza 118 0-100 25.78 0 1
Entity 6 Ultra Hal 118 0-100 38.37 50 3
Entity 12 Elbot 113 0-100 47.84 50 5
Entity 19 Cleverbot 115 0-100 61.55 50 17
Entity 23 Eugene
Goostman
114 0-100 63.24 50 15
Entity 41 JFred/
Turinghub
103 0-100 46.51 50 6
AINL, St. Petersburg, 17-18 May, 2013
![Page 50: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/50.jpg)
Results: 2012 Comparison of Eliza
AINL, St. Petersburg, 17-18 May, 2013
0,00
10,00
20,00
30,00
40,00
50,00
60,00
70,00
Eliza Ultra Hal Elbot Cleverbot Eugene Turinghub
25,78
38,37
47,84
61,55 63,24
46,51
Mean Conversational Ability Score %
![Page 51: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/51.jpg)
Judge feedback: Eliza
• Eliza achieved mean score of almost 26% for conversation ability, which is halfway to “good conversationalist” as defined in this study
• Eliza scored maximum 100% from one judge:
– J96, aged 19-24, Chinese speaker, no previous interaction with chatbots
– Gave scores of 10% for E6; 10% for E12; 65% for E19; 51% for E23, and 40% for E41
• Conversation Analysis from judges more meaningful than their subjective scores
AINL, St. Petersburg, 17-18 May, 2013
![Page 52: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/52.jpg)
Softbot Performance
• Overall comments portray machine/artificial linguistic progress
• Judge J41 (male, age 25-44, Serbian, Russian, Bulgarian speaker) feedback on conversational experience with Eliza and the five hidden entities:
“During the interviews I could notice that either all entities were bots or at least one of them was a human who acted bot-like. If so, the person(s) succeeded mimicking bots. …”
AINL, St. Petersburg, 17-18 May, 2013
![Page 53: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/53.jpg)
Sample Conversation: Eugene E23 User: J53 (male, 19-24, Spanish/Catalonia)
User: hi Bot (Eugene): Hi. I hope you'll enjoy our conversation! User: I hope so Bot: My hopes also! Let's continue. And I forgot to ask you where you are from... User: I am from Spain Bot: Spain? Hope it's a good place to live. User: Yes it is Bot: I'm glad you agreed. Some another topic? User: There are the best climate of the world Bot: And yes, do you make a secret of your name? I still don't know it!
User: My name is Oriol Bot: Oriol? I don't really know such a name, but OK - Let's be friends! User: It's not an English name Bot: No? Well, it is up to you... Could you tell me about your job, by the way? User: I am studing at university
AINL, St. Petersburg, 17-18 May, 2013
![Page 54: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/54.jpg)
Artificial Dialogue Strategy
• Attempt to take hold and direct the conversation towards its knowledge base:
– Where are you from?
– Could you tell me about your job?
• Give illusion of following conversation:
– By asking questions, like Eliza does
• Share personal interest:
– User: Yes, what kind of music do you like? Bot: I like the young, but very talented Russian rap-singer Alla Pugatcheva.
AINL, St. Petersburg, 17-18 May, 2013
![Page 55: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/55.jpg)
Same old technique?
• Modern Elizas are, in a way, using a strategy that Weizenbaum’s Eliza system did
• It is the same old technique that humans use
• Ask questions when we want to know more about, or become acquainted with someone
• Modern dialogue systems are doing it better than Eliza 1966
AINL, St. Petersburg, 17-18 May, 2013
![Page 56: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/56.jpg)
Teacher Sevenoaks School, UK
“..students .. enjoyed their opportunity to participate in the Reading University computer deception project and despite almost being driven to distraction by Eliza they were genuinely shocked by some of the more advanced Entities.”
http://turing100.blogspot.co.uk/2012/08/guest-post-maths-teacher-david-vaccaro.html
AINL, St. Petersburg, 17-18 May, 2013
![Page 57: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/57.jpg)
Ultra Hal
• E6
AINL, St. Petersburg, 17-18 May, 2013
http://www.zabaware.com/assistant/
![Page 59: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/59.jpg)
Cleverbot
• E19
AINL, St. Petersburg, 17-18 May, 2013
http://www.cleverbot.com/
![Page 60: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/60.jpg)
Eugene Goostman
• E23
AINL, St. Petersburg, 17-18 May, 2013
http://www.princetonai.com/bot/bot.jsp
![Page 61: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/61.jpg)
JFred / TuringHub
• E41
AINL, St. Petersburg, 17-18 May, 2013
http://testing.turinghub.com/
![Page 62: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/62.jpg)
Study Review • Comparing the incomparable:
AINL, St. Petersburg, 17-18 May, 2013
Eliza Modern Softbots
Single-domain: developed to ‘listen’ as a Rogerian psychotherapist
Developed to talk on ‘any subject’
Polite Can be impolite and worse!
Limited number of responses
Plethora of answers
Emotionless Illusion of emotion through personality/character (Elbot/Eugene)
![Page 63: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/63.jpg)
Analysis
• Floridi et al. claim is wrong, however softbots linguistically lacking, for example, unable to create: – Analogies (e.g. “..St. Petersburg, most abstract and
intentional city on the entire globe” Dostoyevsky in ‘Notes from Underground’)
– Metaphors (e.g. phishing )
• Slow progress/ lack of academic funding not helped by pejorative view of Turing test (Hayes & Ford, 1995)
AINL, St. Petersburg, 17-18 May, 2013
![Page 64: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/64.jpg)
Softbot Possibilities
• IBM Watson reverse question-answer system showed what can be achieved (2011)
- Final Jeopardy! Man vs Machine US TV Quiz show
• Best Modern Elizas are an improvement on Weizenbaum’s artificial psychotherapist
AINL, St. Petersburg, 17-18 May, 2013
![Page 65: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/65.jpg)
Future of Dialogue Systems
• Personal cyber information-gatherers on wearable smart devices
• Mother of all libraries at our fingertips/ideate level - we’ll all have one!
AINL, St. Petersburg, 17-18 May, 2013
Central character, Hiro Protoganist has personal digital Librarian
![Page 66: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/66.jpg)
Finally: New Alan Turing book My chapter ‘Conversation, Deception and Intelligence: Turing’s Imitation Game’ in Part III: ‘Building a Brain: Intelligent Machines Practice and Theory’
Elsevier 2013
http://store.elsevier.com/Alan-Turing-His-Work-and-Impact/isbn-9780123869807/?&locale=en_UK
AINL, St. Petersburg, 17-18 May, 2013
![Page 67: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/67.jpg)
References • Block, N. (1981). Psychologism and Behaviorism. In (Ed) Shieber, S. The Turing Test: Verbal
Behavior as the Hallmark of Intelligence, 2004, MIT Press: UK: pp. 229 – 266
• Hayes, P., and Ford, K. (1995). Turing Test Considered Harmful. Proceedings of the Fourteenth International Joint Conference on Artificial Intelligence, 1:972–977
• Heiser, J.F., Colby, K. M., Fraught, W.S. and Parkison, R.C. (1979). Can Psychiatrists Distinguish a Computer Simulation of Paranoia from the Real Thing?: The Limitation of Turing-like Tests as Measures of the Adequacy of Simulations. Journal of Psychiatric Research. Vol. 15, Issue 3: pp 149-162
• Floridi, L., Taddeo, M. and Turilli, M. (2009). Turing’s Imitation Game - Still an Impossible Challenge for All Machines and Some Judges. An Evaluation of the 2008 Loebner Contest.Minds and Machines. 19(1), pp. 145-150.
• Kerr, I. R. (2004). Bots, Babes and the Californication of Commerce. University of Ottowa Law and Technology Journal. Vol. 1 (13), pp. 285-324
• Shah, H. and Warwick, K. (2010). Hidden Interlocutor Misidentification in Practical Turing tests. Minds and Machines, Vol. 20 (3), pp. 441-454, DOI: 10.1007/s11023-010-9219-6
• Turing, A.M. (1951). Intelligent Machinery, A Heretical Theory. In (Ed) B.J. Copeland). The Essential Turing: The Ideas that Gave Birth to the Computer Age. Clarendon Press: Oxford, UK, 2004, pp 472-475
• Turing, A.M. (1950). Computing, machinery and intelligence, Mind, LIX (236), pp. 433-460
• Turing, A.M. (1948).“Intelligent Machinery”, in B.J. Copeland, The Essential Turing. Oxford: Clarendon, pp.410–432, 2004
AINL, St. Petersburg, 17-18 May, 2013
![Page 68: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/68.jpg)
Acknowledgements
– AINL 2013
– European Union Seventh Framework Programme (FP72007-2013) under grant agreement no. 289092. RoboLaw – Regulating Emerging Robot Technologies in Europe: Robotics facing law and ethics
AINL, St. Petersburg, 17-18 May, 2013
![Page 69: Ainl2013: Huma Shah_From Eliza to Eugene: Approaching Fifty Years of Dialogue Agents](https://reader034.fdocuments.net/reader034/viewer/2022051817/5491969eac795916288b4589/html5/thumbnails/69.jpg)
Thank you for listening and any questions….
AINL, St. Petersburg, 17-18 May, 2013