FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION
Transcript of FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION
![Page 1: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/1.jpg)
FRONTIERS OF AI-ENHANCED
HUMAN -MACHINE INTERACTION
European Union’s Horizon 2020 research and innovation programme,
Grant Agreement N. 723386
Aleksandra MankiewiczDariusz Cieslar
Leo HoogendoornGianluca Di Flumeri
LiorLimonad
AptivAptivTMSiBrainSignsIBM
![Page 2: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/2.jpg)
ALEKSANDRA MANKIEWICZ,
M.SC. Advanced Engineering, Perception and Features
Test and Verification Engineer
![Page 3: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/3.jpg)
AGENDA
SIMUSAFE DRIVER SENSING AND AUTOMOTIVE SAFETY
ACQUISITION OF BIOSIGNALS
APTIV
01
APTIV
02
TMSi
03
BrainSigns
FROM BIOSIGNALS TO NEUROMETRICS
04
IBM
IOT INCREASING CONNECTIVITY
05
![Page 4: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/4.jpg)
SIMUSAFESimulation of Behavioural Aspects for Safer Transport
http://simusafe.eu/
![Page 5: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/5.jpg)
SIMUSAFE OBJECTIVES
2
Influence of road situations and infrastructure
types on human interaction
1
4Multi-driver
and multi model simulationwith
variousenvironmental
features
3
Risk metrics based on vehicle dynamicsand psychophysiological
state indexes
5
Realisticartifficial trafficagent models
for trafficsimulators
Behaviouralmodels, driving
and walking patterns for traffic
paticipants
![Page 6: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/6.jpg)
• Trackreplicatedin the simulation• Vehicleand pedestriandynamics
modelledin the simulation• Drivingand walkingcockpits• Registrationof correspondingdataset
• Full instrumentationof vehicles• Real test track• Carefully designed test cases• Registrationof gazecoordinates• Biometricdata and face recording
REAL ENVIRONMENT SIMULATED ENVIRONMENT
![Page 7: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/7.jpg)
DRIVER SENSING AND AUTOMOTIVE SAFETY
01
Driver Sensing Camera
![Page 8: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/8.jpg)
DARIUSZ CIESLAR, PHD
Advanced Engineering, Perception and Features
Engineering Group Manager
![Page 9: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/9.jpg)
Interior Sensing | EU Regulations and Certification
2019 2020 2021 2022 2023 2024 2025 2026 2027
Expected Regulation Release
Distraction Recognition (All Vehicles)System capable of recognition of the level visual attention of the driver to the traffic situation and warning the driver if needed
Driver Readiness Monitoring (L2 – L3)Assess if the driver is in a position to take over the driving function from an automated vehicle in appropriate situations
Driver Monitoring (All Vehicles)Systems to mitigate driver distraction & impairment through alcohol, fatigue, etc.
2020
Protocol Release
Implementation
EU GENERAL SAFETY
REGULATIONS
Application to New Vehicle TypesApplication to all new Vehicles
Child Presence Detection (All vehicles)Technological solutions that can monitor a child’s presence in the vehicle and alert the car owner or emergency services should the situation become dangerous
![Page 10: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/10.jpg)
SAFE GREEN CONNECTED
CENTRAL COMPUTE
PLATFORMS
SENSING AND PERCEPTION
SYSTEMS
SENSING AND PERCEPTION
SYSTEMS
HIGH-SPEED, HIGH-RELIABILITY
POWER AND DATA
DISTRIBUTION
![Page 11: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/11.jpg)
Interior Sensing | Features Overview
AVAILABILITY ACTIVITY / READINESS
DRIVER CABIN
• Gesture Recognition• Pointing direction
• Presence• Gaze Direction
• Identification• Mood & Emotion
• Distraction level• Fatigue level• Drowsiness
• Child presence detection• Health Parameters• Occupant position / state
Comfort/ Convenience
In-CabinSafety
• Mood & Emotion• People presece• Identification
• Video Call• Object
detection/Anomaly
• Body position• Occupant
classification
• Hands on wheel• Activity: eating,
reading, using phone
![Page 12: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/12.jpg)
Smarter Safety | Interior and Exterior Sensor Fusion
Combines interior with exterior sensing to unlock safety features
Presented by Aptivat CES 2020
https://www.aptiv.com/newsroom/article/smarter-safety-through-interior-and-exterior-sensor-fusion
![Page 13: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/13.jpg)
Multi-agent VRsimulator as a tool for the development of safety features
Risk perception and evaluation
SELECTED OBJECTIVES OF EXPLORATORY RESEARCH:
Investigation of altered driving conditions (ADC)
SIMUSAFE | Improving Safety with AI
![Page 14: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/14.jpg)
NEW S
Multi-agent VRsimulator as a tool for the development of safety features
Risk perception and evaluation
SELECTED OBJECTIVES OF EXPLORATORY RESEARCH:
Investigation of altered driving conditions (ADC)
SIMUSAFE | Improving Safety with AI
![Page 15: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/15.jpg)
Naturalistic behavior in a
non-controlledenvironment
Behavior captured with enhanced
instrumentation in controlled
environment
Behavior under altered driving
conditions*
Current Status
OTA recording of everyday driving
Driving in a virtual city
Real test track
Virtual test track
Driving in a virtual city
Real experiments:
Sim experiments: * Stress, alcohol, THC, diabetes, etc.
SIMUSAFE | Structured Research Process
![Page 16: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/16.jpg)
SIMUSAFE | Electrophysiological Measurements
• EEG• ECG• GSR
MEASURE
• Vigilance• Stress• Workload• Fatigue
ESTIMATE
![Page 17: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/17.jpg)
Definition of test procedures and individual analysis follows a SORC model
• Stimuli• Organism• Reaction• Consequences
SIMUSAFE | Traffic Psychology
![Page 18: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/18.jpg)
SIMUSAFE | Test Scenario and Area Design
Laps designed to stimulate:• Habituation• Interactions with
other users • Interactions with
infrastructure • Stress
![Page 19: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/19.jpg)
SIMUSAFE | Experiments
![Page 20: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/20.jpg)
ACQUISITION OF BIOSIGNALS
02
![Page 21: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/21.jpg)
LEO HOOGENDOORN,
M.SC.Amplifierand CommunicationTechnologyfor
ElectrophysiologicalApplications
Senior Consultant
![Page 22: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/22.jpg)
Highlights of TMSi (Tw ente Med ica l Systems Interna t ional B.V.)
• Unique amplifier technology for measuring a large variety of electrophysiological signa• Signals are actively shielded to eliminate electrical interference and cable movement
artefact• Therefore very suited for measuring in ‘’hostile’’ environments and moving subjects
IDEAL IN SIMUSAFE!
![Page 23: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/23.jpg)
And further:
• All data collected without analog filters and very high resolution• Open interfaces to Windows, Linux and Matlab• Systems certified for medical use
![Page 24: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/24.jpg)
Neurometric tests performed w ithin Simusafe:
• EEG (16 or 24 channel)• ECG single channel to derive Heart Rate
(HR) and Heart Rate Variation (HRV)• Galvanic Skin Response (GSR)
![Page 25: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/25.jpg)
Devices used :
In initial recordings Porti and Mobi were used on simulators and cars
All data was synchronized with parameters from non-biometric sensors and from the vehicle
During the project, the Saga was developed and fully certified, to allow for further ease of use, on body mounting in all test modalities and a large number of neurometric parameters (up to 64 channels of EEG, 4 channels of ECG, EOG or EMG, GSR, Respiration, 3d accelerometers etc.)
![Page 26: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/26.jpg)
Conclusions from Simusafe p roject for TMSi
• With the Saga, we have developed the ideal system for use in road-traffic research, both in simulators as well as in real life.
• At the same time, it is clear that biometric measurements of this nature, where sensors are mounted on subjects, will not be suitable in daily life.
• Depending on the outcome of the in-depth analysis of the neurometric data by Brainsigns, TMSi may therefore seek to develop or validate technology that allows measurement in daily life, in order to enhance road traffic while at the same time not hampering the active participants in any way.
![Page 27: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/27.jpg)
FROM BIOSIGNALS TO NEUROMETRICS
03
![Page 28: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/28.jpg)
GIANLUCA DI FLUMERI, PHD
NeuroscientistR&D Project Manager
Cognitive States in Operational Environment
![Page 29: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/29.jpg)
Biometrics is the technical term for body measurementsand calculations. Biometricidentifiers are the distinctive, measurablecharacteristicsused to label and describeindividuals. Examplesinclude, but are not limited to body measurements,fingerprint,facerecognition, DNA,palmprint andhandgeometry, iris-recognition,etc.
PRELIMINARY CONCEPTS
![Page 30: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/30.jpg)
Neurometrics is a technical term developed in the last decade within the field ofneuroscienceand standsfor measuresof human mental states (thus the prefix neuro-becauseof the relation with human neurophysiologicalactivities), such as the level ofattention, workloador stresswhile performinga task.
PRELIMINARY CONCEPTS
Credit to:STRESS project, H2020 SJU GA n. 699381, http://www.stressproject.eu/
![Page 31: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/31.jpg)
Preliminary concepts | Approach
Cognitive Neuroscienceapplied to operationalenvironments
Neurometrics of specificmental states
![Page 32: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/32.jpg)
Preliminary concepts | Approach
Cognitive Neuroscienceapplied to operationalenvironments
Neurometrics of specificmental states
In SIMUSAFEthe followingneurometrics will beestimated:• WORKLOAD• VIGILANCE• STRESS• FATIGUE
![Page 33: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/33.jpg)
Why neurometrics?
Lackof objectiveinformation about the user’spsychophysiologicalstatuswhiledealingwith operativeactivities.
! Self-assessed measures are subjective and cannot be collected while operating. Also, theuser could be not aware of an incoming psychophysiological impairment.
! Supervisor assessment could have a certain subjective bias. Also, sometimes mental statedegradation could be covert (i.e. not perceivable from his behaviour).
! System data often highlight risky behaviours “after the fact”.
![Page 34: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/34.jpg)
Why neurometrics?Neurophysiologicalmeasurescould provide objectiveinformation about humanmental states. Theassessmentof the differentmentalstateswould allow to solvealso the Human Factor issue related to the Human Performance Envelopecharacterization. (Parasuraman et al., 2008; Borghini et al., 201
Humanperformancedegradationresultsfrom the interactionofmultipleHFsandthisinteractionisstill mostlyunderexplored.
Theconceptof HumanPerformanceEnvelope(HPE), a functiondefinedby relevantHFsand associatedscales,aimsto predictoperator’s performancedefininga region where performancewill be tolerable, and where it starts to become hazardous(H2020FutureSkySafetyprogram).
![Page 35: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/35.jpg)
Why neurometrics?Neurophysiologicalmeasurescould provide objectiveinformation about humanmental states. Theassessmentof the differentmentalstateswould allow to solvealso the Human Factor issue related to the Human Performance Envelopecharacterization. (Parasuraman et al., 2008; Borghini et al., 201
Neurometrics are able to provide objective measures, even online, of the user’spsychophysicalstatus.
With respect to behavioural data (e.g. vehicular data), through neurometrics it is possibleto immediatelydetect potentially riskyconditions (e.g. your driving behaviour is going tobecome dangerous once you are already fatigued and prone to drowsiness, neurometricsare able to anticipate fatigue detection).
With respect to self-assessed measures, neurometrics are able to point out unconsciousreasonsof riskybehaviours(e.g. sometimes you cannot be aware of your stress).
![Page 36: FRONTIERS OF AI-ENHANCED HUMAN-MACHINE INTERACTION](https://reader031.fdocuments.net/reader031/viewer/2022012005/61d9287e1fd94c23d12730d1/html5/thumbnails/36.jpg)
CASEHISTORIES