Using cognitive tools in robots dealing with people with dementia

Using cognitive tools for designing robots dealing with

people with dementia

Valentina Presutti

Semantic Technology LaboratoryInstitute of Cognitive Sciences and Technologies

Italian National Research Council

3rd Data Analytics Symposium @ Philips

Friday, November 18th 2016, Eindhoven (NL)

STLab team

Valentina Presutti Aldo Gangemi

Andrea NuzzoleseDiego Reforgiato

Martina Sangiovanni Mario CarusoGiorgia Lodi

Alessandro RussoLuigi Asprino

Piero Conca

This presentation reports the result of a team work. This is a recent picture of the STLab team although one of us is missing (Prof. Diego Reforgiato).

http://www.mario-project.eu

frames

ontologies

affordance

The main problem and domain addressed by this research effort is dementia. Mario is a EU H2020 project that contributes to support people with dementia by building a companion and service robot. CNR (and in particular STLab) is one of the main research and development partners.

Life expectancy is growing

Life expectancy is growing, but this positive phenomenon comes along with others that make our society face important challenges.

World population aged 60-79 and aged 80 or older

Popu

latio

n (m

illio

ns)

0

750

1500

2250

3000

year2000 2015 2030 2050

aged 60 years or olderaged 60-79 years aged 80 years or older

71607

125901

2021400

4342100>3*

>2*

Data Source: United Nations (2015). World Population Prospects: The 2015 Revision.

Older people are a rapidly growing proportion of the world’s population” (US NIA Director Dr. Richard J. Hodes)

One of them concerns the world population’s age, a significant part of which is growing olderIn 2000: 607 million people aged 60 years or over - in 2015: 901 million, an increase of 48% - By 2030 is projected to grow by 56% per cent, to 1.4 billion by 2050 is projected to more than double its size in 2015, reaching nearly 2.1 billion.The number of people aged 80 years or over, is growing even faster than the number of older persons overall. 2000: 71 million people aged 80 or over. 2015: grew by 77% to 125 million - 2030: it is projected to increase by 61% reaching nearly 202 million. 2050 projection to 434 million globally, having more than tripled in number since 2015.

Ten leading causes of disabilities among persons aged 60 years or over, by sex, 2012Data source: WHO (2014). Global Hearth Estimates.

*YLDs = Years of life Loss due to Disability

Society must prepare to address health concerns of older persons

Research must provide solutions to this huge societal challenge

This fact has an impact on the society, which has to ensure that these people can live dignifiedly during their elderliness.Hence, it is important to understand what are the main needs of elderly people, and be prepared to support them.The top ten causes of disabilities among elderly people include dementia and depression (which usually accompanies the former). Dementia is a very close problem to each of us. Every 60 second, 20 people in the world are diagnosed with dementia, meaning that by the time that we will be 85, 1/3 of us will suffer from dementia. And the older you are, the higher is the risk.

We can slow down the decline due to dementia

To slow down the decline due to dementia we have to keep being socially, mentally and physically active.Although this looks feasible to us today, this is not straightforward when you suffer from dementia.Suffering from dementia means to struggle with the simplest things such as using a mobile phone, remembering his friends’ or relatives’ names, details of a day spent together. And he finds this embarrassing to the point that he did not say he has dementia to anybody, because he does not want to be seen differently. This often leads to isolation and loneliness.

If you’re lonely you are not going to be socially, mentally and physically active

Lonely people with dementia need social support and companionship.

Robots,elderlyandcognitiveimpairment

It is here that social and companion robots come into the picture. They are a key technology to address this problem, hence it is desirable to concentrate research effort on their development


Mario is one relevant effort in this direction. An ongoing project, at half of his path. Most of the prototypes are already in place, and now it is in the phase of field trials which will drive further incremental prototype development

Taking to locations

Finding objects

Reminding

Helping with reminiscence

Reminding names, guessing words

Playing games

Connecting to family and carers

Performing Comprehensive Geriatric Assessment

Smart entertainment

playlist with familiar music

Memory loss

Loneliness

Depression

Monitoring


MARIO is able to show a number of behaviors for making people with dementia feel more autonomous and help the medical staff to assist them at best. MARIO can replace medical or nursing staff for conducting specific tests that are used to assess and monitor the cognitive abilities of a patient. The Comprehensive Geriatric Assessment is a test that MARIO can perform, allowing the medical staff to save significant time and perform it more frequently, hence collecting more data than it’s normally possible, to better and more frequently analyse the evolution of the disease. As for the patients, MARIO for example can remind them to take medicines, to find objects, or to reach a place at home or in the hospital. MARIO can help fighting loneliness by connecting the patient with family and carers, as well as by entertaining the patient by reading news or a favorite novel, listening to music, playing games, and revisiting reminiscence. Playing games and revisiting reminiscence impacts also on keeping the patient mentally active. Playing music, video and watching picture are developed as “smart entertainment” which can have a strong rehabilitation impact.

“Personalmemoriesareembeddedinthingslikemusic,especiallyinsongswhichonekneworsongswhichonelearntorsongsonesang.Sothepastwhichisnotrecoverableinanyotherwayis

embeddedinmusicandpeoplecanregainasenseofidentity,atleastforawhile…”

“Theeffectlastslongerthanthedurationofthesongs”

OliverSacks

https://www.youtube.com/watch?v=MdYplKQ4JBchttps://www.youtube.com/watch?v=NKDXuCE7LeQ

Playsmartmusic

Smart entertainment can have a huge impact on supporting people with dementia, both at early and severe stage.

https://www.youtube.com/watch?v=MdYplKQ4JBc

https://www.youtube.com/watch?v=MdYplKQ4JBc

https://www.youtube.com/watch?v=NKDXuCE7LeQ

https://www.youtube.com/watch?v=NKDXuCE7LeQ

Keylessonslearnt…sofarArchitecture

open and extensible

Arbitration behavior-based, cognitive theories

Interactionas natural as possible

Observationfield trials are essential

Lessons that I learnt so far as a computer scientist working in a robotic project in a very sensitive domain such as helping people with dementia.


open and extensible

Arbitration , cognitive theories



As a computer scientist with background in software engineering I am interested in the architectural design choices for the system. In particular for the software part.


affordance

frame

openness

This is MARIO’s architecture, which shows a pretty standard structure as for a robotic system.One of its characteristic features is its openness. Openness addresses two requirements: (i) it must be easily extensible, (ii) it must benefit from the large background knowledge available on the web. MARIO’s architecture allows any developer using any programming language to plug new abilities to MARIO via REST services. Furthermore, it reuses data available on the semantic web, as background knowledge. Another characterizing feature of MARIO’s architecture is that its internal knowledge representation and processes are designed according to cognitive theories. More specifically, MARIO relies on the concepts of affordance (from Gibson) and frame (Minsky and Fillmore).

Ontology,Knowledgebaseandreasoning

https://github.com/anuzzolese/lizard

http://www.ontologydesignpatterns.org/ont/mario/

Let’s zoom on the knowledge component of MARIO’s architecture. We use semantic web technologies for representing data and their related ontologies. To ensure MARIO’s extensibility this component include a middle layer, called Lizard, which generates REST services (and Java API) to ease the access (retrieval and storage) to the ontologies and its associated knowledge bases. In other words, even developers who are not familiar with semantic web technologies can easily use the shared MARIO knowledge base and ontology network to retrieve and store data, produced or used by their client application.

https://github.com/anuzzolese/lizard


MARIOOntologyNetwork

49 networked ontologies


In order to support MARIO’s behavior and store data about the patient, its interaction with MARIO and the observed events we have developed a network of ~50 ontologies that cover a number of topics. These are used for organising the stored data and support internal processes.


Competency Questions Which is the level of capability of daily living activities of a patient (e.g. bathing, dressing, toileting, transferring, controlling urination and bowel movement, feeding)?

CGA application

Reasoner

KB System

Use case

CGA ontology

For example, there is an ontology dedicated to represent the concepts used for performing the CGA test. This ontology stores questionnaires, answers and test evaluation. An ability dedicated to the CGA uses this ontology to retrieve and store data in the shared knowledge base.

Competency Questions What tags are associated with a certain photo or events?Do the tags identify relatives or friends?

Reminiscence application

Reasoner

KB System

Use case

Tagging ontology

Another example of ontology is one that represents the concepts around tagging a multimedia object. This is very important for smart entertainment such as the reminiscence app and the play music app, which use information about events, people, places, mood, and others associated with a multimedia object to generate music playlists, or show related pictures, and so on.


open and extensible




13 Another key point is the capability of a robot to arbitrate behavior.

The robot operates autonomously in a dynamic environment and interacts with users to perform complex tasks

Behaviours may be incompatible with each other or compete for accessing the agent's actuators and devices (motors, screen, etc.)

Affordance: the possibility of an action on an object or environment (Gibson, 1977)

Situation types: abstract possible states of the world, according to the agent's internal model

Approach the user

Propose entertainment Recharge

Low battery

User far

User inactive

Recharge

Approach user

Proposeentertainment

Affordance-based Behaviour Arbitration

It is very common in robotic system to address this issue by defining a complex model of the world, then proceed with a sense-plan-action process. However, we have to do with a dynamic environment and the robot has to be able to adapt to this environment and its changing nature. For example, mapping the environment compute a trajectory and a plan to reach e.g. the user, may fail as the user may move, the patient won’t wait for the robot. Also, we want the robot to consider what is the best thing to do given certain observed conditions, but this can change over time based on what has happened before or according to a user feedback. Here we borrow from cognitive theories and in particular the notion of affordance from Gibson. An affordance is the possibility of an action on an object or an environment. In our system the object/environment are abstract possible states of the world, simple models that represent situation types that afford to some extent the possible robot’s behavior (its abilities). Affordance can be positive or negative and have different weights.

Use simple models to obtain complex behavior, inspired by cognitive theories instead of complex modelling (e.g. planning)

Situation occurrences: actual state of the world resulting from:internal or external events, stimuli and perceptions (short term knowledge)user requests/commands (e.g. expressed in natural language)long term & background knowledge

Affordance-based Behaviour Arbitration

low battery

recharge

user inactive

propose entertainment

user wants music

play music

I’d like to listen to some jazz music!

MARIOMary

The robot continuously or frequently monitors the actual state of the world which is instantiated as situation occurrences. Such occurrences may result from internal or external events (from sensors) or from user’s requests (expressed in NL) or from long term or background knowledge (e.g. based on history of behavior). The computed resulting affordance allows the robot to decide what is the most appropriate action at the moment, and will affect the affordances of the internal model for future behavior. In a sense, it reacts intuitively to what is the best thing to do, which allows the robot to handle a dynamic environment. This approach also helps us in dealing with adaptation and with adjusting the robot behavior based on observation of the user’s feedback.


open and extensible




20 The third key lesson is about interaction.

Interactionmustbeasnaturalaspossible

WecannotaskPwDtolearnacontrolledlanguageWecannotexpectthemtoadapttothemachine

Ithastobetheotherwayaround

We just started our field trials and one thing that immediately emerged is that patients who have interacted with the robot have forgot to have done so the day after. This means that it’s always as it is the first time.

Minsky[1]

“Whenoneencountersanewsituation[…]oneselectsfrommemoryastructurecalledaFrame.Thisisarememberedframeworktobeadaptedtofitrealitybychangingdetailsasnecessary.”

“Aframeisadata-structureforrepresentingastereotypedsituation,likebeinginacertainkindoflivingroom,orgoingtoachild'sbirthdayparty”

“Wecanthinkofaframeasanetworkofnodesandrelations.”

“Collectionsofrelatedframesarelinkedtogetherintoframe-systems”

Fillmore[2]“[…]incharacterisingalanguagesystemwemustaddtothedescriptionofgrammarandlexiconadescriptionofthecognitiveandinteractional“frames”[…]”

“Theevolutiontowardlanguagemusthaveconsistedinpartinthegradualacquisitionofareportoryofframesandofmentalprocessesforoperatingwiththem,andeventuallythecapacitytocreatenewframesandtotransmitthem.”

“[…]inordertoperceivesomethingortoattainaconcept,whatis[…]necessaryistohaveinmemoryarepertoireofprototypes.Theactofperceptionorconceptionbeingthatofrecognizinginwhatwaysanobjectcanbeseensasaninstanceofoneoranotheroftheseprototypes.”

Frames

Frames for the Semantic Web [3]

Our approach here again borrows its principle from cognitive theories and in particular the notion of frames. Frames are data structures representing stereotyped situations according to Minsky. Everything in MARIO is a frame. The situation types and the situation occurrences I mentioned before are frames. MARIO abilities are represented in terms of frames. Situation occurrences can be sensed from a user’s request in NL, hence we look for frame occurrences in the sentences, by following Fillmore notion of frames.

CureHealer

Medicationhttp://framenet.icsi.berkeley.edu/

Patient

This is an example of a linguistic frame as it is represented in a repository called framenet. The frame defines a context, a Cure in this case, based on which we can interpret the terms that refer to things that participate in it such as a patient, a medication, and a healer. For examle the sentence “Doctors alleviated his suffering” is an occurrence of this frame.

http://framenet.icsi.berkeley.edu/

http://framenet.icsi.berkeley.edu/

Frames naturally support knowledge reconciliation, regardless the logical, conceptual or syntactic

representation of knowledge sources

Frames can be found in any form of expression or representation of knowledge, hence they emerge as acommon knowledge interpretation key. This is the way we want to use them. Frames naturally support knowledge reconciliation, regardless the logical, conceptual or syntactic representation of knowledge sources. They are the right abstraction level and can be used as units of meaning in knowledge representation and extractionIn natural language (text): they occur In ontologies (intensional): entities evoke or represent framesIn data: Individuals (extensional) are disambiguated in terms of the frames they participate toVisually you recognise them as scenes

N-ary relation f(e, e1,…en)

f is a first order logic relation

e is a variable for any event or situation described by f

ei is a variable for any of the entity arguments of f

A semantic web (OWL) n-ary relation pattern

the n-ary relation is the reification of f, i.e. e

the n objects represent the arguments of f

the n argument relations are binary projections of f including e

co-participation relations are binary projections of f not including e

Representing frames

“Hagrid rolled up a note for Harry in Hogwarts”

A general definition for frame is that of a n-ary relation where its argument include the event or situation that the frame describes and its participants.

Abstract, formalised frame model

generalised model of rolesRepresents all resources’ entities in terms of its frame semantics

Links linguistic data with ontologies and facts (~43M triples)

Outperforms Semafor and FrameBase on frame detection

Framester [4,5]

As part of MARIO background knowledge we created a resource that provides a formal model for the concept of frame that encompasses all possible types of frame manifestations, in language, from sensor, in data, etc. and can be used as common interpretation key for the observed, sensed situation occurrences.This resource includes linguistic frames and links together linguistic and world factual knowledge, that is available on the semantic web. All these entities are represented in terms of its frame semantics and link to each other. Hence it provides us with a way to navigate from situation occurrences, to MARIO knowledge base, which includes abilities representation.

Abstract, formalised frame model

generalised model of rolesRepresents all resources’ entities in terms of its frame semantics

Links linguistic data with ontologies and facts (~43M triples)

Outperforms Semafor and FrameBase on frame detection

Framester [4,5]

http://lipn.univ-paris13.fr/framester/en/wfd/

As part of MARIO background knowledge we created a resource that provides a formal model for the concept of frame that encompasses all possible types of frame manifestations, in language, from sensor, in data, etc. and can be used as common interpretation key for the observed, sensed situation occurrences.This resource includes linguistic frames and links together linguistic and world factual knowledge, that is available on the semantic web. All these entities are represented in terms of its frame semantics and link to each other. Hence it provides us with a way to navigate from situation occurrences, to MARIO knowledge base, which includes abilities representation.



Blah blah blah blahBlah blah blah blahBlah blah blah blahBlah blah blah blahBlah blah blah blahBlah blah blah blahBlahblahblahblahBlahblahblahblahBlahblahblahblah\

User-Robot KB

As we said situation occurrence may be sensed from users’ requests in natural language. Hence user speech has to be translated in a frame-based representation in order to be treated with this approach.

Frame-driven knowledge extractionhttp://wit.istc.cnr.it/stlab-tools/fred/ [6]

“FrancescoteachesvisualartsattheUniversityofBologna.HeisinfluencedbyDavidBowieandhedreamstoworkwithMatteoGarrone.”

We developed a method and a tool able to perform this transformation. Let us consider this example sentence: “” which can be a user’s sentence about his/her son, for example. We can see how the text is represented in terms of frame occurrences.

http://wit.istc.cnr.it/stlab-tools/fred/

FRED http://wit.istc.cnr.it/stlab-tools/fred/ [6]

“The Black Hand might not have decided to barbarously assassinate Franz Ferdinand after he arrived in Sarajevo on June 28th, 1914”

This method outputs together the result of many different natural language processing tasks, by integrating them in a whole semantic graph linked to MARIO background knowledge, based on frames.




Evaluation tasks [7]

33

Tool/Task Topics NER NE-RS

TE TE-RS Senses Taxo Rel Roles Events Frames+SRL

AIDA – + + – – + – – – – –Alchemy + + – + – + – + – – –

Apache Stanbol – + + – – + – – – – –CiceroLite – + + + + + – + + + +

DB Spotlight – + + – – + – – – – –FOX + + + + + + – – – – –FRED – + + + + + + + + + +NERD – + + – – + – – – – –Ollie – – – – – – – + – – –

Open Calais + + – – – + – – – + –PoolParty KD + – – – – – – – – – –

ReVerb – – – – – – – + – – –Semiosearch – – + – + – – – – – –

Tagme – + + + + – – – – – –Wikimeta – + – + + + – – – – –Zemanta – + – – – + – – – – –

Topic detection and Opinion holder detection [8]

Sentiment propagation through frames and roles [9]

Sentiment analysis

“People hope that the President will be condemned by the judges”

A nice thing about frame-based knowledge extractionis that you can define domain-specific frames,and re-design or annotate the resulting graph according to themThis for example worked very well with sentiment analysis

50 sentences from MPQA opinion corpus1 and Europarl corpus2

100 Sentence sentiment polarity of open rated hotel reviews (positive and negative)

Evaluation

Task Measure Value

Holder detection F1 0.95Topic detection F1 0.68

Sub-topic detection

F1 0.77

Review sentiment vs. user scores

Avg. correlation 0.81

2 http://www.statmt.org/europarl/ 1 http://mpqa.cs.pitt.edu/corpora/mpqacorpus/

3 http://www.stlab.istc.cnr.it/documents/sentilo/reviewsposneg.zip

http://www.statmt.org/europarl/

http://mpqa.cs.pitt.edu/corpora/mpqacorpus/



http://www.stlab.istc.cnr.it/documents/sentilo/reviewsposneg.zip



Key lessons learnt…so farArchitecture

open and extensible

Arbitration cognitive theories



29min Observe: do not make assumptions unless you’re one of themField trials are essential

One year effort started on 1st November 2016

Robocup@Home challenge

Three pilot sites: Stockport (UK), Galway (IE), and San Giovanni Rotondo (IT)

Private homes, Nursing homes, and Hospital

We can meet next year, I’ll bring MARIO with me!

Evaluation

MARIO pilot experiments started on November 2016The technologies being developed will be evaluated both by comparing performances with existing benchmarkAnd by testing with real users, i.e. people with dementia and medical staff

References

[1] Marvin Minsky: A Framework for Representing Knowledge. MIT-AI Laboratory Memo 306, June, 1974.

[2] Charles J Fillmore. Frame Semantics and the Nature of Language. Annals of the New York Academy of Sciences, 280(1):20-32, 1976.

[3] Aldo Gangemi, Valentina Presutti: Towards a pattern science for the Semantic Web. Semantic Web 1(1-2): 61-68 (2010)

[4] Aldo Gangemi, Mehwish Alam, Valentina Presutti, Luigi Asprino and Diego Reforgiato Recupero: Framester: A Wide Coverage Linguistic Linked Data Hub. In Proceedings of EKAW 2016

[5] Aldo Gangemi, Mehwish Alam, Valentina Presutti: Word Frame Disambiguation: Evaluating Linguistic Linked Data on Frame Detection. LD4IE@ISWC 2016: 23-31

http://dblp.uni-trier.de/pers/hd/g/Gangemi:Aldo

http://dblp.uni-trier.de/db/journals/semweb/semweb1.html#GangemiP10

http://dblp.uni-trier.de/pers/hd/a/Alam:Mehwish

http://dblp.uni-trier.de/pers/hd/p/Presutti:Valentina

http://dblp.uni-trier.de/db/conf/semweb/ld4ie2016.html#GangemiAP16

[6] Aldo Gangemi, Valentina Presutti, Diego Reforgiato Recupero, Andrea Giovanni Nuzzolese, Francesco Draicchio, Misael Mongiovì: Semantic Web Machine Reading with FRED. Semantic Web (To appear)

[7] Aldo Gangemi: A Comparison of Knowledge Extraction Tools for the Semantic Web. ESWC 2013: 351-366

[8] Aldo Gangemi, Valentina Presutti, Diego Reforgiato Recupero: Frame-Based Detection of Opinion Holders and Topics: A Model and a Tool. IEEE Comp. Int. Mag. 9(1): 20-30 (2014)

[9] Diego Reforgiato Recupero, Valentina Presutti, Sergio Consoli, Aldo Gangemi, Andrea Giovanni Nuzzolese: Sentilo: Frame-Based Sentiment Analysis. Cognitive Computation 7(2): 211-225 (2015)

References cont.

http://www.semantic-web-journal.net/content/semantic-web-machine-reading-fred-1

http://dblp.uni-trier.de/db/conf/esws/eswc2013.html#Gangemi13


http://dblp.uni-trier.de/pers/hd/r/Recupero:Diego_Reforgiato

http://dblp.uni-trier.de/db/journals/cim/cim9.html#GangemiPR14

http://dblp.uni-trier.de/pers/hd/r/Recupero:Diego_Reforgiato

http://dblp.uni-trier.de/pers/hd/c/Consoli:Sergio


http://dblp.uni-trier.de/pers/hd/n/Nuzzolese:Andrea_Giovanni

http://dblp.uni-trier.de/db/journals/cogcom/cogcom7.html#RecuperoPCGN15

Using cognitive tools in robots dealing with people with dementia

Presentations & Public Speaking

Transcript of Using cognitive tools in robots dealing with people with dementia