Deliverable D5.2: Report on use cases, performance …FP7 - MOBOT – 600796 1 EU FP7-ICT-2011.2.1...
Transcript of Deliverable D5.2: Report on use cases, performance …FP7 - MOBOT – 600796 1 EU FP7-ICT-2011.2.1...
-
FP7 - MOBOT – 600796 1
EU FP7-ICT-2011.2.1
ICT for Cognitive Systems and Robotics - 600796
Work Package 5: Specification of Use Cases, Performance Metrics, and User Evaluation
Studies
Deliverable D5.2: Report on use cases, performance metrics and user
study preparations
Release date: 20-02-2014
Status: public
Document name: MOBOT_WP5_D5.2
-
FP7 - MOBOT – 600796 2
Executive Summary
This report includes an update of the first report (D5.1) as submitted before. In D5.1 a
comprehensive documentation of background, definitions and assessment strategies has been
given. This update (D5.2) is restricted to amendments and additional information on future
validation of MOBOT prototypes to be developed.
It includes some minor amendments with respect to user groups and use cases as presented in
D5.1, which were necessary for the specification of the technical development. The target
group of MOBOT have been defined by impairment - rather than disease-oriented definitions.
The selected motor and cognitive criteria in addition to the clinical setting (geriatric rehab)
was not modified for this update and seem to perfectly fit for the MOBOT context.
A detailed description of validation studies on existing devices by a systematic review has
already been given in D5.1 and has been updated for this report (D5.2). Results of the
systematic review will be submitted for publication timely and will allow an evidence-based
development of a validation strategy for MOBOT prototypes.
In addition, this report includes performance metrics for evaluation studies, depending on the
assessment strategy to be selected. As some features and functionalities of the MOBOT
device remain to be specified by technical partners, assessments and performance metrics may
be modified or added for the first evaluation study. The report completes with the description
of present preparations of the upcoming user evaluation studies including tasks of clinical
partners such as ethics votes and subject recruitment. An update of performance metrics and
current status of user evaluation studies will be given in D5.3.
-
FP7 - MOBOT – 600796 3
Deliverable Identification Sheet
IST Project No. FP7 – ICT for Cognitive Systems and Robotics - 600796
Acronym MOBOT
Full title Intelligent Active MObility Assistance RoBOT integrating
Multimodal Sensory Processing, Proactive Autonomy and Adaptive
Interaction
Project URL http://www.mobot-project.eu
EU Project
Officer
Michel Brochard
Deliverable D5.2 Report on use cases, performance metrics and user study
preparations
Work package WP5 Specification of Use Cases, Performance Metrics, and User
Evaluation Studies
Date of delivery Contractual M 12 Actual 20-02-2014
Status Version 1.5 Final
Nature Report
Dissemination
Level
Public
Authors
(Partner)
Klaus Hauer (BETHANIEN), Phoebe Köpp (BETHANIEN), Angelika
Peer (TUM), Yiannis Koumpouros (DIAPLASIS), Pavlos Alevras,
Panagiotis Siavelis, Despoina Alexopoulou, Foteini Koureta
(DIAPLASIS)
Responsible
Author
Klaus Hauer Email [email protected]
Partner BETHANIEN Phone 0049-6221-3191532
Keywords Use Cases, Performance Metrics, User Evaluation Studies
Version Log
Issue Date Rev
No.
Author Change
12-11-2013 1.0 Phoebe Köpp First draft
06-12-2013 1.1 Klaus Hauer,
Phoebe Köpp
Input of revised chapter 2 and 5
06-12-2013 1.2 Yiannis
Koumpouros
Input of chapter 3 + contribution to chapter 4
06-12-2013 1.3 Phoebe Köpp Formal integration of chapter 3
http://www.mobot-project.eu/
-
FP7 - MOBOT – 600796 4
19-12-2013 1.4 Phoebe Köpp Input of revised chapter 4
27-01-2014 1.5 Klaus Hauer,
Phoebe Köpp
Integration of corrections after internal review
-
FP7 - MOBOT – 600796 5
TABLE OF CONTENTS
Executive Summary .......................................................................................................... 2
1. Introduction ................................................................................................................... 8
2. Update on definition of user groups and use cases (BETHANIEN) ............................. 9
2.1. Definition of user groups ........................................................................................ 9
2.1.1. Previous definition of user groups .................................................................. 9
2.1.2. Addenda for user groups ................................................................................. 9
2.1.3. Final inclusion criteria and adjusted assessment ............................................. 9
2.2. Definition of use cases ......................................................................................... 12
2.2.1. Previous definition of use cases .................................................................... 12
2.2.2. Addenda for use cases ................................................................................... 13
2.2.3. Final Use cases .............................................................................................. 13
3. Subjective and objective performance metrics for mobility device assessment
(BETHANIEN) ............................................................................................................... 17
3.1. Technical evaluation ............................................................................................ 18
3.2. Clinical Evaluation ............................................................................................... 18
3.2.1. Generic (established) clinical assessment, performance-based measures ..... 18
3.2.2. Subjective measures for Human-Robot Interaction (HRI) (DIAPLASIS) .... 19
3.2.3. Specifically tailored assessment tests focusing on specific functionalities to
document the added value of the device (BETHANIEN) ....................................... 24
4. User study preparations (BETHANIEN) .................................................................... 26
4.1. Results of a review on clinical validation studies of robotic devices ................... 26
4.2 Abbreviated presentation of results of the systematic review ............................... 31
4.3 Objectives to be achieved for MOBOT- validation studies .................................. 31
4.4 Ethical approval ..................................................................................................... 34
4.5 Time table and development of validation strategy .............................................. 34
4.6 Subject recruitment ............................................................................................... 35
5. Summary of report ...................................................................................................... 36
6. References ................................................................................................................... 37
Appendix 1 ...................................................................................................................... 43
Appendix 2 ...................................................................................................................... 75
-
FP7 - MOBOT – 600796 6
LIST OF TABLES
Table 2-1 Inclusion criteria for gait ................................................................................. 10
Table 2-2 Inclusion criteria for transfer .......................................................................... 11
Table 2-3 Inclusion criteria for cognition ....................................................................... 11
Table 2-4 Summary of categories for user groups according to motor and cognitive
impairment ...................................................................................................................... 12
Table 2-5 Anthropometric characteristics ....................................................................... 12
Table 2-6 List of use cases .............................................................................................. 13
Table 3-1 Different types, purposes and responsibilities of evaluations ......................... 17
Table 4-1 Evaluation studies of a mobility assistance robot ........................................... 27
Table 4-2 Objectives and solutions for evaluation .......................................................... 31
Table 4-3 Specific functions of the device to be evaluated in first user evaluation study
as described in the study proposal ................................................................................... 32
-
FP7 - MOBOT – 600796 7
LIST OF ABBREVIATIONS
Abbreviation Description
PR Public Report
WP Work Package
Partner Abb. Description
TUM TECHNISCHE UNIVERSITAET MUENCHEN
ICCS INSTITUTE OF COMMUNICATION AND COMPUTER
SYSTEMS
INRIA INSTITUT NATIONAL DE RECHERCHE EN
INFORMATIQUE ET EN AUTOMATIQUE
ECP ECOLE CENTRALE DES ARTS ET MANUFACTURES
UHEI RUPRECHT-KARLS-UNIVERSITAET HEIDELBERG
ATHENA RC ATHENA RESEARCH AND INNOVATION CENTER IN
INFORMATION COMMUNICATION & KNOWLEDGE
TECHNOLOGIES
ACCREA ACCREA BARTLOMIEJ MARCIN STANCZYK
BETHANIEN BETHANIEN KRANKENHAUS - GERIATRISCHES
ZENTRUM - GEMEINNUTZIGE GMBH
DIAPLASIS DIAPLASIS REHABILITATION CENTER SA
-
FP7 - MOBOT – 600796 8
D5.2: Report on use cases, performance metrics and user study preparations
1. INTRODUCTION
Changes due to the aging process result in manifold limitations crucial for quality of
life. An increasing number of frail older persons are not able to perform basic activities
of daily living such as walking, transfer, personal hygiene or shopping independently.
Acute diseases or hospitalization are associated with worsening of function and
abilities. The loss or decrease of motor and cognitive functions represents the main risk
factors for loss of autonomy, the most feared condition/event in older people`s life.
The use of rollators is established to support mobility, autonomy and independent living
of older frail persons. A lot of studies prove the benefit of rollator use such as support
by weight unloading and postural control [Bateni and Maki, 2005], reduction of risk for
falls [Graafmans et al., 2003] and support of mobility, activity and participation
[Salminen et al., 2009]. But there are also studies that indicate a higher risk for falls
while using a walker [Stevens et al., 2009]. So far, rollator-like devices have not, or
only in a limited way, targeted cognitive support.
The aim of the MOBOT project is to develop an intelligent active mobility assistance
robot for indoor environments (geriatric rehab) that provides user-centred, context-
adaptive and natural support. Such a robotic rollator can support older persons in a
number of every-day challenges. Features like automatic collision avoidance can
prevent falls, enhanced physical support can prevent exhaustion, navigation support can
prevent disorientation in unaccustomed settings such as geriatric ward-based
rehabilitation.
As stated in the DOW, technical features such as application of computer vision
techniques with modalities such as range sensor images, haptic information, as well as
command-level speech and gesture recognition and other functionalities will allow such
support.
The aim of WP 5 (clinical partners) is amongst other tasks (analysis of user needs,
definition of use cases, review of existing devices) the preparation and execution of
formal user evaluation studies and the definition of objective and subjective
performance metrics and adequate assessment strategies. With the user evaluation trials,
the specific improvements of the newly developed devices will be pointed out clearly
and the additional value for the user will be documented. The trials will be conducted
with potential users of the target sample of patients in geriatric rehab, who interact with
prototypes of the developed devices. The selection of adequate clinical assessment
strategies and associated metrics depend on the technical development of specific
functionalities of the MOBOT device. Based on the information as given by technical
partners, MOBOT working groups will assess technical functions (responsibility of
technical partners), clinical performance-based functions (responsibility of clinical
partner Bethanien), user specific perception (responsibility of clinical partner Diaplasis)
and mixed clinical–technical functionalities as developed for the MOBOT device
(responsibility of technical and clinical partners).
-
FP7 - MOBOT – 600796 9
2. UPDATE ON DEFINITION OF USER GROUPS AND USE CASES (BETHANIEN)
In this chapter the final definition of user groups and use cases is provided. In the
following we relate to the previous report D5.1 (Preliminary report on use cases and
user needs).
2.1. Definition of user groups
2.1.1. Previous definition of user groups
The MOBOT project consortium decided to focus on moderate to mild impaired
persons in institutionalized settings such as geriatric rehabilitation and seniors‟ homes.
This target user group is defined as frail, multi-morbid persons, partly with acute
impairments as in geriatric rehabilitation, with decreased motor status based on multiple
impairments and advanced age/frailty, and high incidence of cognitive impairment. This
highly impaired group ascertains that the devices to be developed and validated by
MOBOT will meet with a high demand for comprehensive support as induced by
multiple impairments.
With cognitive and motor status we chose two main criteria to define the sample as both
criteria will dominate the options and limitations in using the projected devices to be
developed by the MOBOT project consortium. Cognitive as well as motor impairments
leading to falls represent main risk factors for loss of autonomy. As established in
rehabilitation research we focus on impairments rather than single diseases as devices
will not be specified for disease symptoms but specific impairments.
The definition of user groups is required in this context to further describe the target
population for use of the projected device and as inclusion criteria for validation studies
as projected in MOBOT.
2.1.2. Addenda for user groups
In addition to the first version of the user definition as documented in our previous
report (cognitive and motor status), we decided to add anthropometric data (body
weight and body height) as safety, usability and functionality of the new device are
strongly associated with individualized customization.
2.1.3. Final inclusion criteria and adjusted assessment
We decided finally to focus on 3 criteria for the use of the device including motor and
cognitive impairment and specific anthropometric characteristics.
Motor status (See D5.1, page 110-115)
The development of MOBOT devices will focus on two motor key features: walking
ability and transfer ability in sit-to-stand situations. We therefore decided to use motor
assessment criteria which distinguish between gait assistance (rollator) and transfer
assistance (e.g. specific rollator type for sit-to-stand transfer) to achieve a specific motor
definition of the user group. We suggest combined assessment criteria which are based
on standardised clinical observation (rollator use, sit-to-stand transfer/ seat elevation) or
-
FP7 - MOBOT – 600796 10
are based on internationally established tests such as Gait tests or the Short Physical
Performance test (SPPB).
To ascertain that the case definition addresses the target population of persons using a
gait assistive device we will use a clinical marker/observation (current use of rollator:
yes vs. no) as a dichotomous clinical criteria.
We further suggest an established performance-based measure which is specific for gait
impairments (gait speed >0.6 m/sec without walking device) [Cesari, 2011; Fritz and
Lusardi, 2009; Studenski et al., 2003; Studenski, 2009]. Using established cut-offs as in
MOBOT, a classification of patients is feasible.
This selected assessment-based cut off represents a clinical relevant parameter for
indoor performance representing the main activity (indoor walking) in the target sample
of rehab patients. A number of other references use higher max speed especially for
outdoor performance/ or higher functioning persons, or lower values for higher
impairment levels. Gait speed as a continuous variable is highly associated with other
clinical outcomes [Guralnik et al., 1995] and allows evaluation of functional status in a
key motor performance for autonomy without ceiling or floor effects.
In table 2-1 the gait-related classification for in/exclusion of MOBOT user groups are
given. Patients without clinically relevant impairments are excluded (no demand for
rollator use), as are patients with severe impairment (no longer able to use a rollator
because of high risk exposure)
Table 2-1 Inclusion criteria for gait
Category Impairment level
Category
1
Persons with no or minor motor
impairment (no use of rollator, max
gait speed unassisted >0.6 m/sec)
excluded
Category
2
Persons with moderate motor
impairment (habitual use of rollator,
gait speed
-
FP7 - MOBOT – 600796 11
Table 2-2 Inclusion criteria for transfer
Categories Impairment level
Category 1 Patients with no or minor impairment
(Person is able to stand up and sit
down unassisted on a normal chair
without problem, 5-chair stand 16,7 sec or
able to stand up from a chair from
elevated chair (120% lower leg length)
included
Category 3 Patients with very severe transfer
impairment (unable to use transfer aid
without supervision, unable to stand up
from an elevated position (120% lower
leg length)
excluded
The majority of patients admitted to geriatric rehabilitation will meet with our inclusion
criteria. However, we expect a noticeable number of patients rated in category 3 to fail
in sit-to-stand tasks (floor effects for STS testing).
Cognitive status (See D5.1, page 110-115)
We will determine the cognitive status with the internationally well established
screening test for cognitive impairment (MMSE, Mini Mental State Examination). This
allows comparability to other studies also including cognitive criteria. The test has
proven high validity, reliability, feasibility to be used as a screening instrument
[Folstein et al., 1975; Tombaugh and McIntyre, 1992] and allows a rather
comprehensive documentation of cognitive sub-performances. We used established cut-
offs for the definition of the user groups with 27-30 scores to define persons without
cognitive impairment and scores ranging from 17-26 to define persons with mild to
moderate impairment. We used the cut-off value of 26 as a more sensitive (but less
specific) cut-off compared to another established value in MMSE assessment (n=24) to
include less impaired persons.
As we start with the development of a new clinical device we suggest not to include
persons with more severe impairment levels (MMSE
-
FP7 - MOBOT – 600796 12
Category 2 Mild to moderate impairment (MMSE
17-25)
included
Category 3 Advanced to severe impairment
(MMSE
-
FP7 - MOBOT – 600796 13
The use cases as listed in the table below identify tasks or situations which differ with
respect to duration, frequency of activity and clinical relevance. Some use cases will
only take seconds while other may take minutes, some situations will occur frequently
during a day while others will only occur once a year. Some of the situations occur
rarely but are crucial for the task of the device (to support motor stability) to the user (to
prevent injurious falls).
The suggested use cases are not strictly specified for the MOBOT device. Some of them
as described below will be useful for validation of functionalities of the device while
others may be less specific and more standardised use cases will have to be established.
As at the current state of the project not all projected functionalities have been fully
developed, a number of additional or modified use cases may be included depending on
availability of functionalities and the validation strategies chosen.
2.2.2. Addenda for use cases
We decided to delete some of the less specific use cases of our previous report and add
some others.
2.2.3. Final Use cases
The following table presents a final list of use cases we will focus on.
Table 2-6 List of use cases
No. Use case Description/Background
1 The user moves to
get up from a lying
position
The user is moving from a lying position to a seated
position, indicating an intention to get up
2 The user moves to
stand up from a
seated position
The user is initiating movements from a seated position
indicating an intention to stand up
3 The user calls for
the device
gesturally and/or
vocally from a
seated position
The user is sitting and is in need of the device, which is
out of reach; the user performs specific calling (come
here type) gestures and/or calls for the device issuing
specific vocal commands
4 The user calls for
the device
gesturally and/or
vocally from a
standing position
The user is in a standing position and is in need of the
device, which is out of reach (in a near stand-by or
parked mode); the user performs specific calling (come
here / come closer) gestures and/or calls for the device
issuing specific vocal commands
5 Mobility assistant
approaches user
while being in a
seated position
Mobility assistant approaches user after being called,
user is in a seated position
6 Mobility assistant
approaches user
while being in a
standing position
Mobility assistant approaches user after being called,
user is in a standing position
-
FP7 - MOBOT – 600796 14
7 The user starts
accelerating the
mobility assistant
Smooth acceleration is required from 0 to an adequate
walking velocity
8 The user walks
straight with the
device in obstacle-
free area
Stability and safety while walking are very important as
those older frail persons are vulnerable and without
adequate postural stability, especially at the beginning of
walking users may feel dizzy (common on older persons
due to balance problems like orthostatic problems). The
device has to observe the walking pattern and stabilize
the gait.
9 User changes the
velocity (e.g. in
dual-task situation)
or changes
orientation
Changing the velocity is a function that has to be
provided as user could be tired or in a dual task situation
(like talking, making a decision, avoiding obstacles while
walking). Changes of orientation or direction could occur
while avoiding obstacles or walls or if the user changes
his mind about the goal
10 User does not
recognize static or
dynamic obstacles
Avoiding obstacles if user does not recognize obstacles
as mentioned before is part of walking. Especially older
persons with sensory impairment (visual impairment)
detect these situations later and more inexact. Compared
to objects, other persons move, possibly very fast, and
change their position.
11 User walks on
slopes
Walking on slopes supposed to be a rare event especially
in indoor situation, but is a very difficult situation in
normal rollators or devices. To support this situation may
be useful for the frail user as it demands for braking
while walking and controlling if going downwards,
increasing pushing force to move and control the rollator
if going upwards and tracking path and controlling the
rollator on a sideway slope.
12 User walks/has to
manoeuvre in
narrow area
User needs support in collision avoidance, obstacle
avoidance and steering as narrow spaces are very
difficult to pass especially if there is visual or balance
impairment
13 User has to walk
through narrow
passages like doors
User needs support in collision avoidance, obstacle
avoidance and steering as narrow spaces are very
difficult to pass especially if there is visual or balance
impairment
14 User stands up or
sits down
Support in sit-to-stand or stand to sit transfer
15 User has to
open/close doors
Opening and closing doors is a typical everyday task
16 User takes
something and tries
to walk using one
Stability is required while holding the device with one
hand (partial pressure), no goal-directed movement with
one hand should occur
-
FP7 - MOBOT – 600796 15
hand only
17 User wants to lean
or sit on device
Device is blocked and provides leaning/sitting support if
needed
18 User wants to walk
with indirect
support by the
device
A following mode to support user if direct contact is not
needed or impossible. The device only approaches the
user in case of a potential fall or insecurity and otherwise
follows him/her at a safe distance.
19 User wants to stop
or brake or has to
stop or brake
accidently
Braking and stopping the rollator has to be smooth and
safe to reduce the risk for the user.
20 User does not want
to use or need the
device
Mobility assistant is in stand-by mode and waits for user
to call it in a safe parking position.
21 User is not oriented Orientation support in unfamiliar environments to enable
user to participate and to acclimatize and orient in the
new environment. Support of orientation is also
important to persons with cognitive impairment as
orientation is particularly difficult for example if a
demented patient is hospitalized.
22 User does not know
the way, and needs
to be guided to
desired location
User is guided to desired location after having
communicated it to mobility assistant
23 User acts hasty User needs to calm down and has a special need for
stability
24 User needs advice/
wants to advise
something
Communication advises if reorientation is required or if
user wants to command the device
25 User is insecure If the user is insecure or does not accept the suggested
movement by the device comment or explanation could
be convenient.
26 User does not
notice instructions
due to his/her
impairment or
distraction
The user is cognitively impaired or does not notice
instructions due to distraction. Repetitions are needed.
27 User does not hear
the instruction
The user has possibly hearing impairment or in case of
loud environment the user cannot hear instructions by the
device.
28 User falls down User falls down and is not able to stand up alone
29 Misuse of the
device
Protection of impairment of the persons or damage of
device
-
FP7 - MOBOT – 600796 16
30 External problems,
like uneven or slick
floor
Monitoring of environment
31 User or caregiver
wants to change
base setting
Operator mode to modify base setting or support specific
modes. The mode is important for basic instrument
adjustment and machine care.
32 User tests all
support modes of
the device in a
demo mode
User is guided through the different modes of the device
and instructed on the usage of it
33 User has to
recharge the device
Charging mode and simplicity of handling and operating
of charging. The mode is important to guarantee optimal
support while using the device.
*the use cases marked with light colour may not be evaluated during validation studies,
as they are limited in feasibility or would impair ethical requests.
*the use cases crossed have been deleted out of the original set because these represent
very special cases that require very tailored and individualized solutions.
-
FP7 - MOBOT – 600796 17
3. SUBJECTIVE AND OBJECTIVE PERFORMANCE METRICS FOR MOBILITY DEVICE ASSESSMENT (BETHANIEN)
For technical devices as projected by MOBOT to support mobility and independence in
a high risk population, a clinical evaluation is mandatory to ensure the safety, usability,
efficacy, and acceptance. Identified user needs should guide the development and
evaluation process to provide structured data of realistic experience with the device
(Guidelines on medical devices, 2009). The development of robotic mobility devices
enfolds different perspectives and has to bring together the technical developments as
well as human requirements. Some of the requirements may be mandatory for a
mobility device such as safety aspects, while others may represent service
functionalities. Hierarchical aspects of the development derive from the hierarchy of
user needs. For instance, mobility robots are developed mainly for older, disabled or
impaired persons with crucial mobility impairments. For these persons safety is
fundamentally important due to their vulnerability and has to be provided in every
possible situation (Feil-Seifer et al., 2007).
To ensure usability of a supportive device not only technical but also the subjective user
perceptions and perspective may be relevant including handling and usability of the
device, the satisfaction and acceptance, or the potential impact on activities of daily
living. In technically driven developments such user-specific demands have frequently
not or insufficiently been considered.
In the following we focus on technical and performance-based clinical evaluation. As a
prerequisite for a clinical evaluation, adequate assessment strategies will have to be
identified including associated metrics related to such assessments. For a clinical
validation the user perspective will be most relevant and will guide the selection of
measurements and associated metrics. Different aspects influence the use and
acceptance of a robotic device.
Beside the technical functionality and (task) performance in motor key performances as
supported by the MOBOT device (walking and sit-to-stand transfer), subjectively
perceived usefulness, user experience and usability are important. We therefore propose
qualitative and quantitative metrics that measure outcomes of the human-MOBOT
interaction. The outcomes will be consistent with the predetermined objectives of the
MOBOT project. Particularly those functions that are specific for the MOBOT device
may have to be targeted to document the added value. To address different technical and
clinical perspectives, the evaluation of the MOBOT device will be separated into
different types of evaluation. It will be further specified how clinical evaluation will be
attributed to clinical partners.
Table 3-1 Different types, purposes and responsibilities of evaluations
Type of
evaluation
Purpose/matter of evaluation Responsibility
Technical Test of technical systems/subsystems
and technical functionalities
Technical partners
Clinical Generic (established) clinical
assessment, performance-based
measures
Bethanien (clinical partner)
Subjective perception, feasibility Diaplasis (clinical partner)
-
FP7 - MOBOT – 600796 18
Specifically tailored assessment tests
focusing on specific functionalities
to document the added value of the
device
Validation I/rollator:
clinical partners/ technical
partners
Validation II/ rollator:
clinical partners/ technical
partners
Validation 3: nurse-type:
clinical/ technical partners
3.1. Technical evaluation
The specific functions of the MOBOT device are documented in the DoW. Technical
functionalities will have to be assessed with respect to accuracy, validity and reliability
of technical function. Both the single technical functionality as well as the function
within the device has to be ascertained. Responsibility for such assessment lies with the
technical partners responsible for the specific functionality. Specific technical
assessment strategies may translate into the specific clinical validation to document the
specific added value of the functionality. Optional integration of technical assessment
strategies into the validation studies will have to be discussed between clinical and
technical partners when functionalities have been fully developed. The technical
evaluation represents a first step to ensure the suitability for the intended use and will be
a prerequisite for clinical validation of a mobility aid classified as a medical product.
3.2. Clinical Evaluation
Clinical evaluation targets at the interaction between a human and the device. For
development of a comprehensive validation strategy we propose 3 different approaches.
3.2.1. Generic (established) clinical assessment, performance-based measures
As already detailed in Deliverable 5.1., we suggest established clinical measures such as
the Timed-up-and-go test (TUG) and the Short Physical performance battery (SPPB) to
document performance in motor key functions as supported by the MOBOT
development. Such motor key features include walking performance, static and dynamic
postural control, and sit-to-stand performance. Those clinical measures include detailed
metrics based on a validated and standardised assessment (see previous report D5.1.).
When the first validation study will take place at Bethanien hospital we will be able to
support those clinical measures with additional technical assessments (electronic
gaitway “Gait Rite”) and accelerometer-based assessment of transfer by “Dynaport”
measures) which give a large number of detailed temporo-spatial metrics for evaluation.
Clinical as well as additional technical measures hold a variety of metrics specific for
the assessment (for detailed information see also D5.1.) which we will not summarize in
this context.
As one of the main objectives of the MOBOT device is support in walking and a
number of use cases developed in the MOBOT project (see also D5.1 and chapter 2 of
this report) include walking, different walking tasks may be used as test scenarios in the
validation. In the following we give a list of scenarios which are typically required in
daily routine: (see also list of use cases as reported in D5.1)
-
FP7 - MOBOT – 600796 19
- Starting the rollator: Acceleration from 0 to adequate/individual velocity
- Changing the velocity/adjust motor driven speed of rollator to individual speed
of user
- Changing orientation or direction while walking
- Walking up/down slopes
- Steering the device/ avoiding obstacles
- Decelerating/ stopping the rollator
Those specific tasks are not part of the standardised clinical assessment as reported
above (TUG/SPPB), but may be included in the design to evaluate specific
functionalities of the MOBOT development (see chapter below).
3.2.2. Subjective measures for Human-Robot Interaction (HRI) (DIAPLASIS)
3.2.2.1. Human-Robot interaction metrics
In the early years of many technical fields, the research community often utilizes a wide
range of metrics that are not comparable due to a bias towards application specific
measures. The primary difficulty in defining common metrics is the incredibly diverse
range of human-robot applications. Thus, although metrics from other fields (HCI,
human factors, etc.) can be applied to satisfy specific needs, identifying metrics that can
accommodate the entire application space may not be feasible [Steinfeld et al., 2006].
Attempts to categorize both objective and subjective metrics have been made.
According to the USUS Evaluation Framework for Human-Robot Interaction [Weiss et
al., 2009] the factors usability, social acceptance, user experience, and societal impact
are considered the main categories of evaluation factors. Each category is divided to
specific metrics, either objectively or subjectively measured:
Usability: Effectiveness, Efficiency, Learnability, Flexibility, Robustness, Utility.
Social acceptance: Performance Expectancy, Effort Expectancy, Attitude toward Using
Technology, Self Efficacy, Forms of Grouping, Attachment, Reciprocity.
User experience: Embodiment, Emotion, Human-Oriented Perception, Feeling of
Security, Co-Experience with Robots.
Societal impact: Societal impact describes all effects the introduction of robotic agents
consequences for the social life of a specific community (taking into account cultural
differences) in terms of quality of life, working conditions and employment, and
education.
Among these, the authors propose that the following could be tested using end-user
questionnaires:
Utility (refers to how an interface can be used to reach a certain goal or to perform a certain task. The more tasks the interface is designed to perform, the
more utility it has)
Performance Expectancy (the degree to which an individual believes that using the system will help him or her to attain gains in performance)
-
FP7 - MOBOT – 600796 20
Effort Expectancy (indicates to which extent the user perceives a system will be easy to use)
Attitude toward Using Technology (the sum of all positive or negative feelings and attitudes about solving working tasks supported by a humanoid robot)
Self Efficacy (relates to a person‟s perception of their ability to reach a goal)
Attachment (an affection-tie that one person forms between him/ herself and another person or object - a tie that binds them together in space and endures
over time)
Reciprocity (the positive or negative response of individuals towards the actions of others)
Embodiment (describes the relationship between a system and its environment and can be measured by investigating the different perturbatory channels like
morphology, which has impact on social expectations)
Emotion (As emotion is an essential part in social interaction it has to be incorporated in the assessment and design of robots)
Feeling of Security (it is important to investigate how to design human-robot interaction in a way that humans experience them as safe)
Co-Experience (Co-experience describes experiences with objects regarding how individuals develop their personal experience based on social interaction
with others)
Societal Impact (Societal impact describes all effects the introduction of robotic agents consequences for the social life of a specific community -taking into
account cultural differences- in terms of quality of life, working conditions and
employment, and education)
The above categorization is the most full and detailed, including aspects that are rarely
taken into account when it comes to evaluating a robotic assistant. Most researchers,
however, when evaluating an assistive device/technology tend to use questionnaires that
give information on the aforementioned fields. However, as Bartneck et al. mention, due
to their naivety and the amount of work necessary to create a validated questionnaire,
developers of robots have a tendency to quickly cook up their own questionnaires. This
conduct results in two main problems. Firstly, the validity and reliability of these
questionnaires has often not been evaluated. Secondly, the absence of standard
questionnaires makes it difficult to compare the results from different researchers
[Bartneck et al., 2009].
For example, on a feasibility study of a robotic medication assistant for the elderly,
Tiwari et al. created a ten-itemed questionnaire (items answered on a 5-point Likert
scale, 1 indicating strong agreement and 5 indicating strong disagreement). Some of the
questions were: “I felt very confident using the system”, “It would be easy to practically
use this system in our living quarters” or “I would imagine that most people would learn
to use it very quickly” [Tiwari et al., 2011].
In the same way, Heerink et al. examined the acceptance of a robotic agent by elderly
users. The subjects were asked questions on perceived social abilities and technology
acceptance. At this case questions were only partly chosen from an existing model
(UTUAT) and partly created from the researchers. Again, they were formed in a 5-point
Likert scale [Heerink et al., 2000].
-
FP7 - MOBOT – 600796 21
This type of choosing tailored questionnaires is the rule in robotics assessment.
However, few valuated tools from the assessment of assistive technology exist and are
sometimes used for assessing robotic assistants.
3.2.2.2. Existing questionnaires for subjective HRI measures
The existing variety of questionnaires that could be useful for the evaluation procedure
of the MOBOT prototypes is narrow, for the reasons described above. The first one that
we should mention is the “Quebec User Evaluation of Satisfaction with assistive
Technology” (QUEST2.0) [Demers et al., 2006]. According to Holz et al. the QUEST
2.0 is the only standardized satisfaction assessment tool designed for assistive
technologies [Holz et al., 2013]. Moreover, it is considered as the most relevant tool in
functional assessment for assistive Technology [Jardón et al., 2011]. It is the most
commonly used assessment tool for HRI. An assistive technology evaluation should, for
the above reasons, include the QUEST2.0 instrument.
A strategy that would target maximum coverage of the subjective measures spectrum
would require the combined use of two (or more) questionnaires, since the QUEST2.0
only covers some subjective aspects (mainly: feeling of security, perceived
effectiveness, ease of use). The only two validated and relevantly common used
questionnaires we found in the bibliography were:
- The Assistive Technology Device Predisposition Assessment-Device Form (ATDPA)
[Scherer et al., 2005].
- The Psychosocial Impact of Assistive Devices Scale (PIADS) [Palmer et al., 2005].
The ATDPA-Device Form was more relevant in context than the PIADS, targeting on
evaluating overall user experience with assistive technology, while PIADS only
emphasizes to the psychosocial impact of assistive devices, without targeting on
evaluating the actual experience of interacting with a robot, but rather on the impact that
this interaction has in quality of life (QoL).
Also, other questionnaires such as the USE-IT [Michel-Verkerke and Hoogeboom,
2013] questionnaire were ruled out from the very beginning, since they were not well-
valuated or not widely used from researchers in the bibliography.
So, for the first evaluation measurements, we propose the combined use of the
QUEST2.0 questionnaire and the Assistive Technology Device Predisposition
Assessment-Device Form (ATDPA), as the combination that covers most of the
desirable user-experience aspects, with assured validity and reliability.
3.2.2.2.1. The “Quebec User Evaluation of Satisfaction with Assistive
Technology” (QUEST2.0) questionnaire
The Quebec User Evaluation of Satisfaction with assistive Technology (QUEST) is an
instrument specifically designed to measure satisfaction with a broad range of assistive
technology devices in a structured and standardized way. Although its experimental
version consisted of 24 items, an item analysis subsequently resulted in a reduced 12-
item scale: the QUEST 2.0. These 12 items relate to device characteristics (n=8) and
-
FP7 - MOBOT – 600796 22
assistive technology services (n=4) [Demers et al., 2002]. The 8 device characteristics
items assess the user‟s degree of satisfaction with device properties and the remaining 4
items are related to assistive technology services. Items are rated on a 5-point Likert
scale indicating level of satisfaction with the aspect of the device or service. The user is
asked to identify the satisfaction variables most important to them. The QUEST was
developed for use with a wide range of assistive technology devices and not all items
are relevant to every device (appendix 2).The device characteristics items are the
following:
How satisfied are you with:
1. the dimensions (size, height, length, width) of your assistive device? 2. the weight of your assistive device? 3. the ease in adjusting (fixing, fastening) the parts of your assistive device? 4. how safe and secure your assistive device is? 5. the durability (endurance, resistance to wear) of your assistive device? 6. how easy it is to use your assistive device? 7. how comfortable your assistive device is? 8. how effective your assistive device is (the degree to which your device meets
your needs)?
The measurement properties of the QUEST 2.0 have been investigated with respect to
reliability, test-retest stability, alternate form reliability, construct validity and
applicability [Demers et al., 2002] by its developers. Also, the QUEST 2.0 has been
used at many studies in order to evaluate perceived satisfaction among users of assistive
devices.
Zickler et al. used the QUEST2.0 in order to measure user satisfaction with the revised
prototype of the brain–computer interface (BCI). Brain Painting application was
evaluated in its target function – free painting – and compared to the P300 spelling
application by four end users with severe disabilities [Zickler et al., 2013]. The
Assistive Technology Device Predisposition Assessment (ATD PA) Device Form was
also used in combination, in order to complement the subjective measurements. Four
items of the QUEST2.0 (the assistive technology services category: durability, service
delivery, repairs/servicing, follow-up services) were not adequate for the evaluation of a
BCI prototype and were, thus, removed from the questionnaire. To render the
QUEST2.0 more suitable for evaluation of BCI-controlled AT the four items reliability,
speed, learnability, and esthetic design were added. This BCI adapted QUEST 2.0 was
referred to as “extended QUEST2.0.” [Scherer and Cushman, 2002].
In the same field, Holz et al. (2013) used the QUEST2.0 in order to measure user
satisfaction with “Connect-Four”, a new sensorimotor rhythm (SMR) based brain–
computer interface (BCI) gaming application. The interface was evaluated by four
severely motor restricted end-users. The authors used the same extended form of the
QUEST2.0 as Zickler et al., and also complemented the QUEST2.0 with The Assistive
Technology Device Predisposition Assessment (ATD PA) Device Form.
The QUEST2.0 has already been used for assessing assistive robots. Jardón et al. (2011)
used the QUEST2.0 on a population of six spinal cord injury patients, in order to assess
the usability of a sophisticated technical aid, the “ASIBOT”, a personal assistance robot
totally developed by RoboticsLab at the University Carlos III of Madrid. Their
-
FP7 - MOBOT – 600796 23
methodology was based on the QUEST test with some specific and local changes to
adapt the questionnaire to their population and product. The designed questionnaire was
similar to QUEST regarding device aspects such as usefulness, training, robustness,
safety, dimensions, simplicity of use, appearance, and effort of installation. However,
they –as most prototype assessors- chose not to use the service components for the
“ASIBOT” prototype.
Concerning mobility assistive devices, QUEST2.0 has been used in several studies, for
assessing user acceptance of mobility aids such as rollators or electric and manual
wheelchairs. Kirby et al. used the QUEST2.0 combined with a device-specific
questionnaire (the Wheelchair Skills Test -WST, version 3.2.) in order to test the
hypothesis that, in comparison with a commercially available tilt-in-space wheelchair, a
light-weight manual wheelchair equipped with a new, rear anti-tip device (Arc-RAD)
provides caregivers with improved wheelchair-handling performance, less exertion, and
greater satisfaction. Again, the authors used only the 8 components for assistive devices
[Kirby et al., 2008].
Hill et al. used the QUEST2.0 to assess satisfaction with the use of a typical rollator among persons with Chronic Obstructive Pulmonary Disease. Supplementary, a
structured questionnaire was used to obtain information regarding daily utility of the
device and barriers to its use [Hill et al., 2008].
Samuelsson and Wressle, in order to follow-up user satisfaction with and the use and
usefulness of rollators and manual wheelchairs, interviewed a sample of 262 users, 175
rollator users and 87 wheelchair users. In this study, also, the QUEST 2.0 was not used
alone: an additional ten-itemed questionnaire was used for collection of subjective data
[Samuelsson and Wressle, 2008].
Finally, Laffont et al., used the QUEST2.0 in combination with some basic objective
metrics (trial duration, number of failures, etc) in order to evaluate a stair-climbing
power wheelchair (“TopChair”), designed for indoor and outdoor use, including stair-
climbing. The sample was consisted of 25 people with tetraplegia [Laffont et al., 2008].
3.2.2.2.2. The “Assistive Technology Device Predisposition Assessment-
Device Form” (ATDPA)
The ATD PA-Device Form is a 12-item questionnaire that examines consumer‟s
subjective satisfaction with achievements in a variety of functional areas, when using
assistive technology (AT). It is the last part of the 66-itemed “ATD PA”, a
questionnaire based at the Matching Person and Technology (MPT) Model [Fuhrer,
2001]. The MPT process is validated for use by persons with disabilities (ages 15 and
up) and is applicable across a variety of users and settings. The measures have been
determined to have good reliability and validity and they have been used in research
studies within the US, Canada, and Europe [Scherer et al., 2005]. A complete list of
validation studies of the “ATD PA” instrument and for the MPT model can be found at
the Institute of Matching Person and Technology
[http://www.matchingpersonandtechnology.com/validation.html].
The questionnaire rates the AT-person match and the expected support in using the
device, in other words the expected technology benefit. The 12 items (such as “will help
http://www.matchingpersonandtechnology.com/validation.html
-
FP7 - MOBOT – 600796 24
me to achieve my goals”, “will fit in accustomed routine”), are rated on a 5-point
Likert-scale from 1 (not at all/0% of the time) to 5 (all the time/100% of the time).
Users have the option to indicate a “0” if the item is not applicable. By averaging the
total of the items a mean score can be calculated. The highest possible score would then
be 5.0. The items are:
1. This device will help me to achieve my goals. 2. This device will benefit me and improve my quality of life 3. I am confident I know how to use this device and its various features 4. I will feel more secure (safe, sure of myself) when using this device 5. This device will fit well with my accustomed routine 6. I have the capabilities and stamina to use this device without discomfort,
stress and fatigue
7. The supports, assistance and accommodations exist for successful use of this device
8. This device will physically fit in all desired environments (car, living room, etc.)
9. I will feel comfortable using this device around family 10. I will feel comfortable using this device around friends 11. I will feel comfortable using this device at work 12. I will feel comfortable using this device around the community.
(Possible answers: 5 = all the time/100% of the time, 4 = often/around 75% of
the time, 3 = half the time, neutral/about 50% of the time, 2 = sometimes/around
25% of the time, 1 = not at all/0% of the time, 0 = not applicable)
According to the MPT Institute [http://www.matchingpersonandtechnology.com/], the
ATD PA Device Form is compatible with the World Health Organizations‟ ICF and,
thus, may be considered measures relevant for use in assessing ICF domains as
impacted by technology use.
It has already been mentioned that the ATD PA-Device Form has been used in the
evaluation of very recently developed brain–computer interfaces (BCI) [Zickler et al.,
2013; Holz et al. 2013]. However, although only in some more researches the ATD PA
has been used, the ATD PA-Device Form, as a reliable and valid instrument that covers
several aspects of subjective human-robot interaction measurements, could be used
supplementary to the use of the QUEST2.0, offering items that give information on
learnability, performance expectancy, effort expectancy, attitude toward using
technology, perceived effectiveness and self-efficacy [Scherer and Fischer, 1998].
Concluding, we propose, for the evaluation of the first stage of MOBOT, the use of
QUEST 2.0 combined with the ATD PA-Device Form. Thus, we could acquire the most
significant subjective measures utilizing already valid and evaluated questionnaires.
3.2.3. Specifically tailored assessment tests focusing on specific functionalities to
document the added value of the device (BETHANIEN)
The Rollator-type device as projected by MOBOT will include specific functionalities
which allow a number of additional support aspects compared to low-tech established
rollators. (For a comprehensive list of functionalities and state of development see study
protocol).
http://www.matchingpersonandtechnology.com/
-
FP7 - MOBOT – 600796 25
Functionalities to be developed will be finalised at different points in time during the
lifecycle of the project depending on the complexity of the task, the current
development and preliminary work. For the first validation studies we may include
those functionalities which will achieve stable functions at that time (original projected
start of validation I: Month 16, due to delay in development of device postponed to
M18). Responsibility for reliable functionalities lies with the technical partners and
represents a precondition for participation in the clinical validation.
Within the MOBOT project organisation ICCS for technical partners and Bethanien for
clinical partners have been appointed at the last project meeting (Munich 11.-12.12;
2013) to organise and develop the validation strategy of such functionalities. The task
force will address other technical and clinical partners. However, technical partners will
hold full responsibility for technical reliability and functionality and tasks associated to
the validation process such as transport and maintenance of the technical devices for
validation purpose. The task force will start the development of specific assessment
strategies associated to functionalities in January 2014 to allow a timely finalisation for
the first round of validation studies.
Specific roles of clinical partners will have to be specified for the upcoming validation
studies. As functionalities represent innovative approaches, a tailored validation strategy
may have to be developed. For such a validation, scenarios as already listed in D5.1 will
be used and extended by additional scenarios specific for the functionality. A number of
established clinical tests may be modified to be used for the tailored validation. Results
of the systematic review (see chapter 4 of this report, validation) as performed by
Bethanien will be used to implement validated, standardised test procedures as used in
comparable, previous validation studies.
In case specific technical assessment strategies for technical validation can be used for
clinical validation as well, their usability for validation purpose will be checked.
Technical partners will have to participate and support the decision to use and
implement their functionalities in the validation process. A report on a detailed
assessment strategy and results of the first validation will be given in Deliverable 5.3.
-
FP7 - MOBOT – 600796 26
4. USER STUDY PREPARATIONS (BETHANIEN)
The technical devices as developed by MOBOT partners will be evaluated in a clinical as well
as technical validation process. Depending on the technical solutions and their feasibility to be
included in the prototypes, associated functionalities may also be part of the clinical
validation. The primary focus of the clinical validation will be the subjective perception and
usability of devices and performance-based measures as established in clinical settings and
validations. In our previous report we described different assessment strategies (e.g. screening
vs. performance-based measures, clinical vs. technical measures) with focus on the key motor
features as supported by the MOBOT devices (transfer support, support in walking). We
provided information and requirements on natural human movement patterns, support and
compensation and established assessment strategies in frail older persons (see D5.1, pages 49-
82).
4.1. Results of a review on clinical validation studies of robotic devices
As a first step in developing an evidence-based validation design we performed a systematic
review on previous clinical validation studies in comparable mobility support devices. Results
of this review are currently summarized in a manuscript in detail and will be submitted for
publication timely. Results of the review have already been documented in the previous report
(D5.1, pages 136-149) and have been updated for the present report.
-
FP7 - MOBOT – 600796 27
Table 4-1 Evaluation studies of a mobility assistance robot
Name of device:
author
Tec
hn
ical
ev
alu
atio
n/
Use
r-in
tera
ctio
n t
est*
Type of assessment (Standardized/Non-
Standardized; Validated/Non-Validated;
Technical Evaluation/Performance test), description
of assessments
Number of
participants,
description of
sample
Po
ten
tial
u
sers
/ O
ther
un
con
cern
ed p
erso
ns
*² Results of tests (Technical evaluation/ User-interaction test: objective
quantification or subjective comment or rating;
comment on statistical analysis ( no stats: no statistical analysis
mentioned, with stats: with statistical analysis mentioned)
1. CAIROW: Mou
et al., 2012
U S/NS; NV:
P: 80 meters walking path → standard deviation of
velocity and step length;
Subjective user opinion
n=6 (persons with
Parkinson Disease)
P U: Presentation of performance results without detailed subsumption;
Subj (user): positive comments (stimulating force by device);
no stats
NS; NV:
P: walking with usual device vs. Cairow → abnormal
gait patterns and gait velocity
Subjective quantification of gait by experts
n=7 (persons with
Parkinson disease,
ø 86 years)
U: Obj: smaller deviation of velocity (more stable);
Subj (experts): less episodes of abnormal steps while walking with
Cairow;
no stats
2. Care-o-bot II:
Graf, 2009
TU S, NV:
T: collision avoidance in natural environment
P: Robotic walker vs. conventional walker, 2
different walking courses/conditions → duration,
distance and collisions
Subjective user opinion (questionnaire)
n=6 (older persons
with need for
mobility aid)
P T: collision avoidance increased (no collision vs. 3);
U: Obj: duration with robot longer and force to control robot high;
Subj (experts): difficulties to control conventional walker in slopes;
Subj (user): 80% felt safe and in control of Care-o-bot;
no stats
3. CMU Robotic,
XR4000
platform: Morris
et al., 2003
TU NS; NV:
T: Usability and capacity of navigation system with
different control modes
Subjective user opinion
n=4 (older persons,
not specified; target
group: persons with
cognitive
impairment)
P? T: demonstration of control concept and technical feasibility;
U: Subj (user): acceptance and interest in robotic walker high,
difficulties mentioned manipulating the haptic interface;
no stats
4. Cool Aide:
Alwan et al., 2005
T
S; NV:
T: Walking straight and with turns with vs. without
controller; human intent vs. controller conflict
situation
n=22 (15 older + 7
younger; all
persons without
motor impairment)
O T: assisted steering enhanced stability of user when intents of user and
robot aligned, possibly endangering problems if this is not the case;
no stats
5. Electric Motor
Based Gait
Rehabilitation
System: Lee et al.,
2004
U S; NV;
P: body weight support of 0%, 10%, 20%, 30% and
40% in own comfortable walking speed: double
limb support time and
single limb support time
n=10 (ø 37 years old) O U: double limb support time decreased and single limb support time
increased with increased body weight support; heart rate: tendency of decrease with increasing body weight support;
no stats
-
FP7 - MOBOT – 600796 28
Impact on user: heart rate measurement
6. GRSR: Jang et
al., 2008
TU NS; NV:
T: Test of Walking guide system
P: Impact on user: muscle activities tested while
walking with body weight support mode
n=2 (young, healthy
persons)
O T: small path tracking error due to friction and slip of wheels;
U: Impact on user: EMG activity decreased in the quadriceps muscles
while walking without full body weight (-40%);
no stats
7. Guido:
Rentschler et al.,
2008
U S; NV:
P: Guido vs. low-tech mobility aid on 36.6 m
obstacle course → travel time, obstacle/ wall
contacts and reorientations
Subjective user opinion (questionnaire)
n=45 (persons with
visual impairment
and limited
mobility)
P U: no significant performance differences between the devices;
Subj (user): more positive after experience of Guido;
descriptive stats given, no statistical analysis
8. Hitachi: Tamura
et al., 2001
U S; NV:
P: Usual walker vs. caster walker vs. power-assisted
walker, 2x6-meter straight path → walking speed,
body acceleration
Impact on user: gastrocnemius electromyogram
measured
n=6 (older persons, ø
82 years) state of
frailty not clear
P? U: Best performance results with conventional walker;
Impact on user: Results of electromyogram show gradual decreases as
the speed decreases;
no stats
9. i-walker: Annicciarico,
2012
U S; V:
Impact on user: 4-weeks training with i-Walker vs.
traditional ambulatory treatment
n=20 (stroke patients) P U: Impact on user: significant improvements in i-walker training
regarding functional outcome (Tinetti, 10MWT, Barthel);
with stats
10. i-Walker:
Kikuchi et al.,
2010
TU S;NV:
P: walking course → with/without controller
N=7 (frail older
persons)
P U: improvement of trajectory, collision avoidance with controller;
deficit: not helpful for a blind person;
no stats
11. iWalker:
Kulyukin et al.,
2007
TU S; NV:
T: navigation mistakes
P: different walking routes → time of trials
Subjective user opinion
n=4 (frail older
adults; clients of
senior services
agency)
P T: system is able to provide way finding task with location
information;
U: Subjective user comments and feedback regarding the interface;
no stats
12. MOBIL
Walker: Bühler et
al., 2001
TU NS; NV:
T: feasibility
P: walking course and practical trials
Subjective user opinion (questionnaire)
n=?, number of
subjects not clear
(persons in foster
home)
P T: Robot is feasible;
U: Stability and speed satisfying, comments regarding the braking
system, 2/3 of users found different control modes easy to handle;
no stats
13. Pam-AID:
Lacey and
MacNamara, 2000
TU S; NV:
P with active demonstrator: Course with turns and
obstacles → two control modes
Subjective user opinion (questionnaire, comments
and rating on usability and support)
n=5 (persons with
visual and partially
mobility
impairment, ø 82
years)
P U: Subj (user): adaptive better than fixed control mode; safety,
usability and utility of device quite to very useful depending on
impairment level, with stats: mean rating and standard deviation;
no stats
-
FP7 - MOBOT – 600796 29
NS; NV:
Subjective user opinion: passive demonstrator 5 days
test (no detailed information)
n=19+? (older
persons, part of
patients not
defined)
U: Subj (user): little heavy and difficult to push, sonar sensor errors
occurred, device was in great demand;
no stats
14. PAMM Smart
walker: Yu et al.,
2003
TU S; NV:
T: wall distance, path deviation
P: walking path of 35 meters → 3 control modes
Subjective user opinion
n=6 (older persons in
assisted living
facility)
P T: adaptive shared control mode is “effective”;
U: improves user performance by shared control mode and not noticed
by user,
Subj (user): averseness of full computer control mode;
no stats
15. Robotically
augmented
walker: Glover et
al., 2003
TU NS; NV:
T: navigation mode
P test
Subjective user opinion
n=6 (older persons,
no potential user
group)
O T: Navigation “successful”;
U: Subj (user): redesign of interface due to difficulties, self-parking
and retrieval mode useful;
no stats
16. Robuwalker: Rumeau et al.,
2012
U S; V and NV:
P: Usual walking vs. regular walking frame vs.
Robuwalker → 4 meter walking test and modified
Timed get up and go test
n=9 (older persons
with motor and
cognitive
impairment)
P U: tested walker is less efficient than a regular walking frame; results
presented;
no stats
17. RoTa: Mori et
al., 2002
U NS; NV:
P: small test course
Subjective user opinion
n=60 (older persons
with visual
impairment)
P U: Subj (user): helpful for the blind due to protection against obstacles,
deficit: cannot go up and down stairs;
no stats
18. RT Walker:
Hirata et al., 2007
T NS; NV:
T: Obstacle, step or wall detection/ path follow
mode/environment with stairs, obstacles and down
slope
n=4/5/1 (young
persons,
blindfolded)
O T: Walker provides obstacle/step avoidance, gravity compensation,
variable motion characteristics, fall-prevention; no detailed results,
annotation of authors: further validation needed;
no stats
19. Simbiosis:
Frizera-Neto et al.,
2011
U S; NV:
P: Walking path (2x10meter + turn)
Subjective user opinion (manoeuvrability, security,
posture/comfort)
n=8 (persons with
spinal cord injury)
P U: subj (user): results given, trend: positive rating without
subsumption;
no stats
20. Monimad:
Saint-Bauzel et al.,
2009
TU NS; NV:
T: biomechanical analysis of temporo-spatial forces
P: Sit-to-stand movement test
n=10 (Multiple
sclerosis patients)
P T: natural style of interaction: temporo-spatial course of forces (shape
of forces similar to human-human sit-to-stand experiments);
U: quick learning rate of adequate use of the device;
no stats
21. Robotic walker:
Chugo et al., 2009
U S; NV
P: Standing up motion with/without body stability
control scheme
Subjective user opinion (effort and fear of falling)
N=7 (older persons,
67 years+ with
need of care)
P? U: subj (user): easy standing up movement, no fear of falling
-
FP7 - MOBOT – 600796 30
*Aim of test: T = Technical evaluation; U = User interaction, handling or satisfaction evaluation; Type of assessment: S = Standardized; N = Non-Standardized; V = Validated;
NV = Non-Validated; T = Technical evaluation; P = Performance test; *²User group: P = Potential-user of the developed device; O = Other unconcerned persons: Subjects
employed for the evaluation study; Results of tests T = Technical evaluation results; U = User-interaction test: objective quantification or subjective comment or rating
comment on statistical analysis: no stats = no statistical analysis mentioned; with stats = with statistical analysis mentioned
-
FP7 - MOBOT – 600796 31
4.2 Abbreviated presentation of results of the systematic review
We included 21 studies based on predefined inclusion criteria focusing on evaluation of
a robotic walking transfer device. Three different types of validation approaches were
identified: 1. technical evaluations (n=2); 2. user-oriented validation (n=9) and mixed
strategies (n=10).
Assessment methods: Assessment methods varied substantially according to aim of
specific testing and quality of assessment methods. Most of assessment tools seemed
not to be validated in general or seem not to be validated for the specific target group
(19 out of 21), one study used validated and non-validated measures [Rumeau et al.,
2012] and one study used validated assessment methods [Annicchiarico, 2012]. A
number of assessments was not clearly described or were based on subjective ratings
with uncertain or insufficient test quality. We could not identify studies focusing on
long-term effects of devices such as the impact on habitual physical activity or quality
of life. None of the studies showed an overall convincing and comprehensive
assessment strategy and design. No standardised assessment protocol specifying
primary outcome variables seems to be established for such clinical validation studies,
severely limiting comparability of results. The added values by new functionalities of
devices were only partly or not addressed at all in assessment strategies.
Target samples: In most of the studies it remains unclear whether tests and the devices
itself were targeted for a specific user group. A clear use case definition was missing.
Statistical analysis: With the exception of 1 study, results were all presented without
statistical analysis thereby substantially limiting the value of validation results by mere
descriptive case presentation without statistical back up. No sample size calculation was
mentioned or presented.
4.3 Objectives to be achieved for MOBOT- validation studies
The table presents objectives, solutions and the actual state of preparation.
Table 4-2 Objectives and solutions for evaluation
Objectives Solutions for MOBOT State of
preparation
Clear definition of target sample
including sub-patient categories
according to motor and cognitive
status
See chapter 2.1 √
Clear definition of goals for the
development of the device and use-
case definition
See chapter 4 (status of
development for evaluation
studies)
√-(√)
Clear definition of use cases See chapter 2.2 √
Selection of standardized, validated (at
best for the specific sample e.g.
cognitive impairment) assessment
tools
Concerning inclusion criteria
to reach target group: see
chapter 2.1.3
Concerning assessment tests
√-(√)
-
FP7 - MOBOT – 600796 32
for new device functionalities:
to be determined
Definition of procedures and metrics
for user evaluation studies
See chapter 4 √-(√)
Adjusted assessment strategy to cover
specific targets of development. The
assessment should clearly separate
between technical assessment of
technical features, requesting mainly
engineering standards and assessment
of user oriented perspective
See chapter 4 √-(√)
√ - fulfilled
(√) – partially fulfilled, to be finalised until validation
According to the study protocol a number of functionalities have been described for the
technical development of the MOBOT device. Depending on the progress of the
construction of the device and the development of additional functionalities, appropriate
targets for validation of the prototypes have to be identified in tight cooperation with
technical partners. Based on study goals as detailed in the study proposal we list the
following objectives as potential targets for the first evaluation study. This list may have
to be modified by technical partners.
Table 4-3 Specific functions of the device to be evaluated in first user evaluation
study as described in the study proposal
Objective,
functiona-
lities
Background Proposed assessment
User
localisation
and activity
detection
3D coordinates of
user
Evaluation will be performed using the
intersection-over-union (IOU) performance metric
based on the November data set, not the data
recorded during evaluation: saying that we provide
a 3D cube A that presumably contains the person,
B (understood as a volume), the IOU metric= |A
and B|/|A or B| measures the extent to which
volumes A and B overlap. If IOU > .7, A and B
overlap substantially, if IOU >.5, they somehow
overlap, but not too much. Having IOU around .8
or .9 is understood as perfect localization.
Basic pose
estimation (state in
which the user is
with respect to the
rollator: distant,
close, in contact)
Detection accuracy, quantified in terms of
precision-recall curves will indicate the trade off
between false alarms and recall; the goal is in
general to have the minimum number of false
alarms at a given level of recall. A scalar metric
that can summarize the performance of a detector
is the Average Precision (AP) which averages the
detector's precision as recall varies.
Recognition of user
gestures and
Recognition performance (success rate, false
-
FP7 - MOBOT – 600796 33
recognizing and
interpreting user
voice commands
detection) will be evaluated off-line.
Detection
of environ-
ment
Obstacle detection
possible if obstacle
is known/static
(static map)
Evaluation metrics include: (a) measurement of a-
posteriori distance from detected obstacles, and (b)
contour matching against known obstacles.
SLAM: off-line evaluation of the similarity of the
created map against the known map.
Localizatio
n of
rollator
itself
Basic localisation
on a static known
map
Evaluation can be performed using known
markers in the map (e.g. a door, a corridor,
known markers on the floor etc.), and measuring
localisation precision in space, especially in loop
closure situations.
Approach
the user
from a
distance
Autonomous
navigation will be
prototype. User
position will be
known. Navigation
will provide
planning, tracking
and approach using
static map.
Approach metrics will consist of evaluating the
difference between the final robot pose with
respect to the user position, as well as the velocity
profile during the near approach. A third metric
can be the time-to-docking, i.e. from first call to
actual user docking.
Assistance
to the user
Transfer/Sit-to-
stand: brakes to
block rollator, fixed
trajectory
Evaluation should assess whether this approach
really improves the functionality (assist sit to
stand) compared to no-actuated rollator handles
and which of the realized versions is best. Precise
estimations of joint torques will not be possible for
evaluation unless we decide to use the Qualisys
tracking system again, so that we can compute
inverse dynamics. Thus, other measures for
evaluation would need to be considered like
subjective measures derived from questionnaires
or objective ones like for example heart rate.
Following the
user`s demand:
Support is passive
but facilitative
Manoeuvrability measured based on interaction
forces, jerk and smoothness of trajectories
Avoiding obstacles Number of collisions with positive obstacles,
distance to positive/negative obstacles,
manoeuvrability assessed by measure based on
jerk
Assist the user
localization
“Cognitive localization” is trivially performed
using a point-in-polygon test. User feedback
should be used to investigate the performance of
this feature.
Guide/navigation Number of arrivals at desired location, time to
-
FP7 - MOBOT – 600796 34
desired location compared to no assisted version,
interaction forces as measure of agreement of user
input and autonomous guidance
4.4 Ethical approval
For series of experimental testings which successfully took place in November 2013 at
Bethanien, a positive ethical vote was given by the ethical board of the University of
Heidelberg (Ethikkommission I der medizinischen Fakultät der Universität Heidelberg).
For the validation studies we plan to submit an amendment which may cover the
projected future validation studies. Based on the comprehensive discussion with the
ethical board, an amendment may not suffice, as the validation of the MOBOT devices
may be qualified as MPG-Studies (medical product studies), which request a much
stricter formal proceeding, comparable to pharmacological studies. The application for
the ethical vote is planned to be submitted in January 2014 to allow a timely start of
validation and will cover the projected 2 series of validation studies (see comment
below). So far no ethical approval for the Greek clinical partners (Diaplasis) is
available.
4.5 Time table and development of validation strategy
According to the milestones as documented in the project‟s DOW, 3 separate validation
studies (validation I/IIrollator and III/nurse-type) are projected during the MOBOT
project. Validation I was supposed to start at month 16 (May 2014). In cooperation with
technical partners (lead ICCS) clinical partners (lead Bethanien) will develop an
assessment strategy for this first validation. Responsibilities were decided at the Munich
project meeting (11.-12.12.2013)
Since the state of construction of the first version of the device and associated
functionalities is slightly delayed, the Consortium decided to postpone the start to
month 18 (July 2014). Depending on the vote of the ethical board, and the development
of the study design and technical functionalities, the start of validation may lead to
further delay. The projected duration of validation also depends on the study design and
the projected sample calculation with consequences on the test burden for participants,
the number of test sessions and total number of participants to be recruited.
A decision by technical partners at the last project meeting (Munich 11.-12th of
December 2013) to include only the 20-80 percentile (based on anthropometric
parameters) of the potential target group of rehab patients substantially reduces the
recruitment base of the validation studies, which were already limited by predefined
inclusion criteria (see case definition) and medical reasons in this multi-morbid, frail,
acutely impaired sample. As the number of patients to be included in the validation
study represents the bottle neck of the duration of validation studies, the decision will
have substantial consequences for the time table of validation. Based on the long-term
experience of the clinical WP-leader (Prof. Hauer) the projected duration as documented
in the DOW of 2 months may not suffice to recruit an adequate number of study
participants in the target population of the MOBOT project (Geriatric Rehabilitation).
Within upcoming project meetings a solution of such problems will be found according
to previous agreements of technical and clinical partners.
-
FP7 - MOBOT – 600796 35
The definition of prototypes to be validated still has to be finalized. According to the
current state of information, the first validation will cover a first version of the MOBOT
device with focus on support of walking performances and sit-to-stand transfer support.
At a later time within the project further validations will cover an advanced prototype,
which may also include additional technical functionalities as developed during 2014
and 2015 by technical partners. A positive ethical vote for the Greek clinical setting will
be mandatory for future testing at their premises.
4.6 Subject recruitment
Clinical centres, at which the validation will take place, will be responsible for
recruitment. The recruitment will include geriatric patients during rehabilitation as the
target sample for the development according to predefined inclusion criteria (see
comments above for general classification of study participants/user definition),
additional ethical (written informed consent) and medical inclusion criteria (e.g. severe
medical condition excluding participation). Recruitment of subjects will start when
rollator prototypes and study protocols for validation will be finalised and positive
ethical votes will be given.
-
FP7 - MOBOT – 600796 36
5. SUMMARY OF REPORT
This deliverable D5.2 presents an update of the comprehensive D5.1 report and gives an
additional outlook on preparation of user evaluation studies as described in the DoW for
WP5.
We present an update of user groups with only minor amendments for the final
inclusion criteria in chapter 2 including motor, cognitive and anthropometric
characteristics and the final use cases as basis for the development of specifically
adjusted performance measurements. Assessment strategies to determine user needs and
limitations of existing mobility devices are given in chapter 3. Chapter 4 covers a
preliminary conception of performance metrics and types of evaluation for the
upcoming validation of the first MOBOT prototype. The specific functionalities of this
prototype will determine a specifically tailored evaluation strategy apart from a
standardised clinical evaluation. The results of a systematic review on previous clinical
validation studies or comparable devices as presented