ICT4Life objective on information fusion and algorithm training

Workshop on Open Platforms in the Field of Independent Living and Active Ageing | www.alfred.eu

“Information fusion and algorithm training framework”

1st International Workshop on Open Platforms in the

Field of Independent Living and Active Ageing

Alejandro Sánchez-Rico de las Heras@alejandrosr


Information is everywhere

Ambient Sensors

Wearabledevices

Cameras

3D

Zenith

Normal

Behaviourdata

Data Analysis(Algorithms)

Clinical data

ProfileDisease informationFamily historyMedicationLaboratory analysisMedical imaging…

Analyzeddata


Multimodal Fusion

Behaviourdata

Clinicaldata

ProfileDisease informationFamily historyMedication…

Multimodal Fusion

Algorithm sensor

analysis

Health Monitoring Data

FeedbackProfessional feedbackPatient evolution


ICT4Life Integrated Care Project

This project has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement Nº 690090.

@ICT4LifeEU

http://www.ict4life.eu/


ICT4Life Overall Structure


@ICT4LifeEU



ICT4Life Setup


@ICT4LifeEU

Kinect

Binarysensor



ICT4Life Framework


ICT4Life Open API on Multimodal Fusion• Open framework for platform flexibility to accommodate various sensors and information channels.• Ready-to-use algorithms for re-training in new environments.• Flexibility in amount and type of modalities.

@ICT4LifeEU

Abnormal BehaviourDetection of abnormal human activities using extensive training through:• Video analysis algorithms.• Ambient assisting living.• Skeleton track data

Multimodal Fusion• Detected abnormal behaviours.• Clinical Data (EHR)• Interactions with social networks and

gamified interfaces.

Decision Support module• Personalization of interfaces• Support caregivers and

professionals.



Abnormal BehaviourDetection of abnormal activities• Set of activities to detect: Apathy, Freezing, …• Set of data for algorithm training.

– Sensor data capture (Normal and abnormal conditions)– Professional data feedback.

Skeleton information3D Cameras

Spatial informationZenith Cameras

Bracelet data

Binary sensors data

Training

Algorithm ExecutionFeedback

Sensor data

ClassificationLearn from errors

Multiple sensor informationSensor Calibration

Common Axis and Time system.

Initial training data

Output

Normal data (x2)Abnormal data

Activities


Abnormal BehaviourOpen API Framework strategy• Objective:

– Provide tools for training on new activities– Be able to adapt platform to new disease scenarios.

• Needs flexibility on:– Set of monitored activities.– Algorithm training data for new activities.– Output description and representation. (E.g. type of output: count, time, …)– Final user feedback for algorithm improvement in time.– Sensor data acquisition.

• Functionality:– Activity register (Basic data, output representation)– Training data load (Normal and abnormal data)

• E.g. Image captures, 3D skeleton data, …


Abnormal BehaviourActivity Register Training data load

CRUD OF NEW ACTIVITIESCREATION, READ, UPDATE, DELETE

Activity structure{ code: num, name: Activity name, Type: count / time / ... Scenarios: Scenario Ids, where analysis apply Output: { title: Output title of activity, description: Output description of activity }}

Activity identification (code).

Activity data:• Image captures.• 3D camera data.• Other sensor data.• Calibration data for multisensor.

User assessment. • Situation identification.

Process of user validation & feedback.

Validation


Multimodal FusionOpen API Framework strategy• Objective:

– Increase data structure flexibility and scalability considered on fusion analysis.– Be able to adapt platform to new disease scenarios.

• Needs flexibility on:– Set of data structures and associated patient data.– Multimodal fusion rules.

• Adapt current rules to consider new data.• Register new rules.

– Output description and representation.– Final user feedback and patient evolution data for fusion improvement.

• Functionality:– Adherence-Impact on existing standards (LOINC, SNOMED CT)– Patient data for analysis.– Rules update and creation.– Efficient and easy integration of experts feedback on patient analysis and progress.


Multimodal Fusion

Electronic Health Record(Input and Output)

Fusion outputDecision support tools or

other modules

Health Monitoring(sensors data)

Data Pre-processing

Fusion techniques

Feature Extraction

New Patient Data

Abnormal Behaviour Information

Previous algorithm results

EHR added sets

Adapted fusion rules

FeedbackProfessional user input

Patient evolution

New Data structures


Multimodal Fusion using late fusion

Probabilistic framework, where each low-level sub-subsystem outputs a probability measure for each type of results.

+ + wn * P N()Patient unique identifier

weights associated with each modality, based on knowledge or learnt during training.

i

wa , wh we, wn

*

Rule Creation and Adaptation• Rule adaptation

• Weights matrix for data modalities.• Probabilistic results on new data structures.

• Listener probabilistic functions.• Rule creation / New infers results

• Add probabilistic functions for each data modality needed. Fusion of each module output (classification results)


Multimodal Fusion and standardsData Structure Integration strategy• Common terminology understanding.• Integration with current data structures.• Facilitate interoperability.

Data structure definition• Data structure tree.• Internal code.• LOINC code match. (if possible)• SNOMED on observation values. (if possible)

• E.g. HL7 ORU messages.

Data Observation Functionality• Patient identifier.• Data associated to previous registered structure.

• LOINC codes / Internal codes• SNOMED CT Values / Internal values

• Time association if needed.• To support fusion of coherent data.

Standards on health data terminology

Clinical terminologyDiseases.Clinical findings.Procedures.

Universal codes to identifyTestsObservations


Development Community

ICT4Life targets to create a community for interested developers and researchers in ICT and Health.

– Platform integration with other monitoring needs.– Collaboration with other initiatives.– Share integration ideas.– ICT4Life in the crossroads of ICT, Medicine, Psychology.


@ICT4LifeEU

Stay [email protected]@ict4life.euVisit our website, join our newsletter and RRSS.



Thank you!Alejandro Sánchez-Rico de las Heras@alejandrosr

ICT4Life objective on information fusion and algorithm training

Technology

Transcript of ICT4Life objective on information fusion and algorithm training