Field Informatics Human Sensing ACCMS, Kyoto University Yuichi Nakamura 1/46 Copyright (C) 2010...

Field InformaticsHuman Sensing

ACCMS, Kyoto University

Yuichi Nakamura

Introduction to Field Informatics Chapter4

Human Sensing:Measuring Human Activities and Social Actions

• Human Activities– simple activities

• walking, eating, house keeping, etc.• simple tasks in daily life

– social actions• conversation, meeting, lectures, etc.• tasks with other people

– others• peoples in a panic

• Measuring what and how?• How store and retrieve the data?

Overview: Purpose of Human Sensing

(a) External information– who did what, how,....

(b) Internal information– thought, intention, feeling,...– physiological conditions

(c) Communication– communicated information– communication intention

Overview: Information on Humans

(a) External information– bodily movements, behaviors, ...– bodily characteristics

(appearance ， sweat ， smell ， etc.)(b) Internal information

– psychological conditions (tension ， fear ， emotion ， comfort/discomfort ， etc.)

– physiological conditions (table 4-3)(c) Communication information

– verbal/non-verbal communication– interpersonal contact, interpersonal

distance, mutual interaction with group

Overview: Tools for Human Sensing

(a) External information– physical sensors– esp. non-invasive, non-intrusive

sensors

(b) Internal information– physiological sensors– brain measurements– introspection, reflection

(c) Communication information– physical sensors– ethnography, ethnomethodology

Information on Humans (examples)

(a) External conditions– bodily movements, behaviors, ...– bodily characteristics

(appearance ， sweat ， smell ， etc.)

(b) Internal conditions– psychological conditions (strain ， fear ，

emotion ， pleasant ， etc.)– physiological conditions

(c) Communication conditions– verbal communication– non-verbal communication

Human positions and movements

• image sensors

• magnetic sensors• ultrasonic wave sensors• RFID

Human Position

• Image sensors– real-time tracking– frequently used for

security and surveillance purpose

• Fish-eye lens and omni-directional cameras– omni-directional– low spatial resolution

Face Detection

• Image sensors with image recognition software.– face detection– face identification

• Many embedded system, e.g., digital camera.

3D Measurements

• Stereo Vision

Multiple Stereo Video CameraStereo Still Camera

Real-time Stereo Machine

• 1995 Carnegie Mellon University

Magnetic Sensors (Motion Capture)

• Comparing with image sensors– better accuracy– no occlusion effect

• Characteristics– ▲ cost, size– × non-intrusive– ◎ accuracy– ◎ occlusion– ◎ lighting– ×other constrains (affected by metal)

Eye Tracking, Gaze Tracking

• Head mount type– measuring eye ball

direction by projecting infrared light

• Table mount type– measuring the pupil

position by video camera(s)

Eye/Gaze Tracking

• Gazing properties– a sequence of fixations– order, duration– movements, saccades

• Internal conditions– intention, attention,

interest

• Object’s characterisitics– features– characteristics

(appearance ， sweat ， smell ， etc.)(b) Internal conditions

– physiological conditions (table 4-3)(c) Communication conditions

Physiological Conditions (examples)

• electrocardiogram (ECG), heart rate, blood pressure, pulse pressure, O2/CO2 concentration in the blood

• breathing rate ， O2/CO2 concentration in breath

• electrooculogram (EOG) ， blink ， pupil size ， focus

• electromyography (EMG) ， evoked electromyography

• skin potential activity, flicker value, body temperature, facial skin temperature, perspiration ， etc.

• electroencephalogram (EEG), magnetoencephalograpy (MEG), functional mgnetic resonance imaging (fMRI) ，near infrared spectroscoping topography (NIRS)

Measuring Brain Activity

• electric activities of neurons

• magnetic field caused by electric activities

• blood flow and brain metabolism

Electroencephalogram

• electrical alterations in accordance with neural activity

• small potential changes on the scalp

－５０

Ｉ１

－５０

－１００

Ｉ１

－５０

Measuring Method

• 10-20 system ： distances between adjacent electrodes are either 10% or 20% of the total front-back or right-left distance of the skull. 鼻根部

Nasion

後頭結節Inion

左耳右耳

Brain Waves• delta wave δ ： 1 ～ 3Hz • theta wave θ ： 4 ～ 7Hz • alpha wave α ： 8 ～ 13Hz • beta wave β ： 14 ～ 30Hz • gamma wave γ ： 30 ～ 64Hz 　　　　• omega wave ω ： 64 ～ 128Hz • rho wave ρ ： 128-512Hz • sigma wave σ ： 512-1024Hz

relaxed

active

exited

Near Infra-Red Spectroscopic Topography (NIRS)

• Near infrared light (around 800nm) is projected and traverses the scalp and skull

• Reflectance from the brain are measured on the scalp

• Brain metabolism can be measured by the ratio of oxidized hemoglobin and deoxidized hemoglobin

Near Infrared Light

MagnetoEncepharoGraph (MEG)

• Magnetic field arising from neural electrical activity

• Large-scale system with high-performance probes

• high temporal resolution

• noise elimination is a serious problem

functional Magnetic Resonance Imaging ( ｆ MRI)

• Brain activity (blood flow, metabolism)

• Magnetic resonance difference between oxidized hemoglobin and deoxidized hemoglobin

ElectroMyoGraphy (EMG)

• Membrane potential changes in muscle contraction

muscle fiber

nerve muscle connection

muscle

motor nerve

surface EMG

AmplifierElectrodes

• Electric potential changes ( 10mV) ≦ on skin surface

• Electrodes and relatively simple electronic circuits

• problem: noise elimination, MU estimation

０．５ sec

1 ｍ v

Multi Channel Measurement

Example of EMG signal

０．５ sec

1 ｍ v

Condition 1

Condition 2

similar motions with different conditions

Subjective or introspective analysis for psychological conditions

• Conversation Analysis

• Protocol Analysis

• Narrative Analysis

Protocol Analysis

Ask a subject to tell anything which comes up to the subject’s mind, and analyze the internal process of the subject’s.

1: Which one?2: I got it!3: Difficult to find, ...4: Hmm, push it, ... really?5: I’m afraid all are gone...

Protocol Analysis

• think aloud method: speak synchronously what the subject thinks during actions

• retrospective report method: explain actions after it is finished

• In both methods, actions are takes in a video or some recording devices, and those data are minutely analyzed afterwards.

ref. Narrative Analysis

• The subject reconstructs real experiences as a personal story

• Originated from narrative therapy• Linear causality is a dominant feature of

narrative structure • A subject is prompted story telling by a

question addressing what to tell.– Type1: as less interruption as possible– Type2: guided by appropriate questions

(appearance ， sweat ， smell ， etc.)(b) Internal conditions

– physiological conditions (table 4-3)(c) Communication conditions

Sensing of Communication Conditions

• Nonverbal information– 70 ～ 80 ％ of information is carried

through nonverbal behaviors

• Various kinds of nonverbal information– attitude, behaviors– body characteristics– perspiration ， smell– clothes, accessories– others

ref. distance communication

• Asynchronous communication– e-mail– Web

• Realtime communication– chat– video conference– distance lecture

Key Points on Human Sensing

• Objective measurements– physical sensors as much

as possible– non-invasive, non-intrusive

sensors• Multiple sensors

– synchronization– large amount of data

• Data handling– indexing– browsing– retrieval

Some examples

• Smart meeting recording

• Lifelog

• Data Browsing

Smart Meeting Recorder• Non-intrusive sensing and recording

– tracking each person from entering room to sitting down

– tracking each person’s face while talking

• Video capturing with typical picture compositions

カメラ制御コンポーネント

観測カメラ：人物の位置検出

制御指令・映像選択コンポーネント

撮影カメラ：首振りカメラによって追跡撮影映像切替器

MPEG エンコード， HDD に録画

pan/tilt 制御

映像を提示• Two types of cameras

– sensing camera– contents capturing

camera• Contents capturing

cameras are guided by the sensing camera.

Sensing camera:detecting participants positions

control, selecting views

capturing camera: tracking and capturing

video switching

MPEG encoding

pan/tilt control with tracking a face

Automatic Editing Examples

Smart Meeting Browser• Smart meeting browser with realtime meeting

capture• Toward realtime meeting support

Lifelog (personal experience log)

video,sound,location,temperature,time,etc.

experiences

large amount of logs

automatic indexingstructure analysisefficient retrieval

browsing

memory aidseducation supportdisability supporthuman factor analysis

• Browsing– skimming

– gathering related actions

used it here

took it herecame into a room

went out a room

Lifelog (Personal View Records)

Browsing

Browsing for indoor activities browsing by related events

Data Indexing

• Meta-data– author, title, date, keywords, etc.– index, tag, ...

• Automated indexing by video, audio, and text processing.

• Examples– XML– MPEG7– ANVIL

Example of Index<?xml version="1.0" encoding="ISO-8859-1"?><annotation> <head> <specification src="C:\Documents and Settings\nakamine\kijou_sagyou.spec" /> <video src="C:\Documents and Settings\nakamine\fish_nagai.mov" /> <info key="coder" type="String"> Our server </info> <bookmark name="scene01" time="73.43333" /> <bookmark name="scene02" time="234.89999" /> (...) <bookmark name="scene07" time="852.73334" /> <bookmark name="scene08" time="900.59998" /> </head> <body> <track name="situation" type="primary"> <el index="0" start="71.13333" end="203.5"> <attribute name="token">scene01</attribute> </el> <el index="1" start="203.5" end="234.33333"> <attribute name="token">show a sample</attribute> </el> <el index="2" start="234.89999" end="294"> <attribute name="token">scene02</attribute> </el>( 以下省略 )

Field Informatics Human Sensing ACCMS, Kyoto University Yuichi Nakamura 1/46 Copyright (C) 2010...

Documents

Transcript of Field Informatics Human Sensing ACCMS, Kyoto University Yuichi Nakamura 1/46 Copyright (C) 2010...

Ubiquitous and Wearable Vision Systems...Ubiquitous and Wearable Vision Systems Takashi Matsuyama Graduate School of Informatics Kyoto University Sakyo, Kyoto, 606-8501, Japan Capturing

ACTIVITY KYOTO ACTIVITY KYOTO 1 Connection ......ACTIVITY KYOTO ACTIVITY KYOTO 1 Connection ACTIVITY KYOTO 'vVish ACTIVITY O,'OTO Amulet ACTIVITY KYOTO 4 Night View ãLL15R (*hutàD

Graduate School of Informatics Kyoto University€¦ · 4. Recommendation Each applicant must be officially recommended to the president of Kyoto University by the president of the

Intercultural Collaboration Experiment ICE 2002 Department of Social Informatics, Kyoto University Japan Science and Technology Corporation NTT Communication.

1 UPKI-Federation based on Shibboleth National Institute of Informatics Motonori Nakamura Toshiyuki Kataoka, Kyoto University Yasuo Okabe.

Graduate School of Informatics Kyoto University · 2016-12-08 · Graduate School of Informatics Kyoto University ... format allowing examinees to select questions from various fields

Daytime Kobe・Osaka・Kyoto Nagoya 1 bus · Universal Studios Japan − − − − − ‖ − − − − ‖ ... Departure place Kyoto Osaka Kyoto Kobe Kyoto Kyoto Osaka Kyoto

Kourosh MESHGI Shin-ichi MAEDA Shigeyuki OBA Shin ISHII 18 MAR 2014 Integrated System Biology Lab (Ishii Lab) Graduate School of Informatics Kyoto University.

Department of Social Informatics Graduate School of Informatics Kyoto University, Japan July 8, 2004 The Social Informatics of Healthcare Infrastructure.

Constructing immersive virtual space for HAI with photos Shingo Mori Yoshimasa Ohmoto Toyoaki Nishida Graduate School of Informatics Kyoto University GrC2011.

Statistical Modeling of Creole Genesis - ACL Member … Modeling of Creole Genesis Yugo Murawaki Graduate School of Informatics, Kyoto University Yoshida-honmachi, Sakyo-ku, Kyoto,

Exploitation of 3D Video Technologies...Exploitation of 3D Video Technologies Takashi Matsuyama Graduate School of Informatics Kyoto University Sakyo, Kyoto 606-8501, Japan tm@i.kyoto-u.ac.jp

KYOTO-OSAKA KYOTO KYOTO-OSAKA SIGHTSEEING PASS … · KYOTO-OSAKA SIGHTSEEING PASS < 1day > KYOTO-OSAKA SIGHTSEEING PASS [for Hirakata Park] KYOTO SIGHTSEEING PASS KYOTO-OSAKA

Graduate School of Informatics Kyoto University

Structural Phrase Alignment Based on Consistency Criteria Toshiaki Nakazawa, Kun Yu, Sadao Kurohashi (Graduate School of Informatics, Kyoto University)

Arif Bramantoro and Toru Ishida Department of Social Informatics Kyoto University Japan

Statistical Modeling of Creole Genesis · 2016. 6. 28. · Statistical Modeling of Creole Genesis Yugo Murawaki Graduate School of Informatics, Kyoto University Yoshida-honmachi,

1/33 Nobuaki ARAI, Junichi OKUYAMA (Graduate school of Informatics, Kyoto Univ.) Copyright (C) 2010 Field Informatics Research Group. Kyoto University.

Keiji Maekawa Graduate School of Informatics, Kyoto University Yasuo Okabe Academic Center for Computing and Media Studies, Kyoto University.

Graduate School of Informatics Kyoto University · October 2016 Admission to the Master’s Program Admissions Policy of the Graduate School of Informatics The field of study referred