Digitising Human Emotions for Virtual Reality Arvind … · Digitising Human Emotions for Virtual...
Transcript of Digitising Human Emotions for Virtual Reality Arvind … · Digitising Human Emotions for Virtual...
Digitising Human Emotions for Virtual Reality
Arvind Sanjeev
Introduction
The article (http://diyhacking.com/digitisinghumanemotionsvirtualreality) is based
on the learnings I gained from a short study I did on existing research to understand the
possibility of digitising our emotions for remote transfer between individuals and to
incorporate it as feedback to virtual reality platforms. My concept was to introduce a
wearable emotion interface that can measure our emotions through facial EMG
(electromyography) and consequently digitise and transfer them to another remote
individual wearing the same wearable headset and invoke the similar feelings. Invoking
an emotion is a concept that could be achieved through facial EMS (Electrical Muscle
Stimulation). Moreover, this system can be used with Virtual Reality to use the
emotions of a person as feedback to the platform and even invoke genuine emotions
through it. Thereby improving the immersive experience for Virtual Reality platforms
using psychophysiology.
For example: the emotional episode of a person sitting in California could be remotely
experienced (feel similar emotions) by a person based in India through an advanced
wearable headset.
All the research work that has been referred can be found here and the pdf version of
this article can be viewed here.
This is the structure of the rest of the article:
Contents
1. Problem statement
2. Corroborating the concepts
3. Dimensions of emotion
4. Conceptual prototyping: Facial EMG and EMS
1. Sensing of emotion through facial EMG 2. a. Physiological effect of emotion on facial muscles 3. b. Facial EMG measurement 4. Electrical stimulation of muscles to invoke emotions
5. Similar research
6. Interesting research: Measure emotion using Galvanic Skin Response
7. Further research
8. Applications of the proposed wearable emotion interface
9. Conclusion
1. Problem statement
Today in the 21st century, people are capable of talking with each other from different
parts of the globe. This is done mainly through instant messaging systems, emails or
other text based platforms. And they rely only on emoji’s :) ;) :( for conveying emotions.
Today with the advancement of futuristic technologies, it is due time we created a
platform that can understand and evoke emotions within people. A wearable
interface that can allow remote transfer of emotions between people and which can
help in making the communication more organic is thus required.
Check this small snippet from the movie "Martian" that clearly explains the problem
statement:
https://youtu.be/CpqAx304Q7c
2. Corroborating the concepts
The research work done till date will be used to corroborate the feasibility of the two
part system:
1. Sensing of emotion through facial EMG.
2. Electrical stimulation of muscles to invoke emotions.
1. Sensing of emotion through facial EMG
Dr.Paul Ekman, a noted American Psychologist has researched for more than 40 years
on the emotions and facial expressions of people. He found out that all the emotions we
experience are associated and mapped to a specific facial expression or group of facial
muscles: Ekman, P., Friesen, W. V., & Ancoli, S. (1980). Facial Signs of Emotional
Experience. Journal of Personality and Social Psychology, 39(6), 11251134.
As part of his research, he had studied and worked with the native tribes of New Guinea,
who haven’t had any interaction with the outside world before him. Through the
interaction with them, he learned that the expressions made by them can be easily
predicted just like any other person and that every human is programmed to have the
same expressions. Thus it has been proved that the emotions made by any human is
universal and can be easily understood by any other human by reading their face. His
work is a modern day extension of the research done by Charles Darwin and Tomkins on
breaking down human facial expressions.
This opens up a huge avenue for emotion based communication interfaces and a
veritable platform capable of understanding the emotions of any individual from the
world irrespective of their origin or the language they speak.
To sense a person’s emotion, there are two ways: self report and concurrent expression.
Self report is what we use normally while IM’ing, we use emojis to report our feelings,
for doing this we have to spare time to stop what we are doing and report it. However,
the system I propose uses concurrent expression, where the user’s emotion is recorded
unconsciously and the genuine feeling is recorded and transmitted. (Picard, R.W.:
Toward computers that recognize and respond to user emotion. IBM Systems Journal
39 (2000) 705–719 )
Courtesy: indiana.edu
The sensing of our facial expressions and thus our emotions can be done by measuring
the muscle activity in our face. Facial Electromyography will be used to detect muscle
activation and corresponding emotions that is invoked in an individual. Research done
by U Dimberg has corroborated that Facial EMG is an accurate technique for sensing a
person’s emotions: Dimberg, U. (1990), For Distinguished Early Career Contribution
to Psychophysiology: Award Address, 1988. Psychophysiology, 27: 481–494. doi:
10.1111/j.14698986.1990.tb01962.x.
1. Electrical stimulation of muscles to invoke emotions.
Along with this concept, I am also researching on whether we can also evoke genuine
emotions through a wearable interface. For corroborating this concept, I rely on Dr.Paul
Ekman’s work, wherein he found that voluntary activation of the facial muscles
concerned with any particular emotion can in turn evoke that emotion.
The concept which triggered the second part of my idea is based on his experiment
where a person can trigger a genuine emotion by voluntarily recreating the facial
expression associated with it, as proved in his work: Voluntary smiling changes
regional brain activity Psychological Science, Vol. 4, No. 5. (1993), pp. 342345 by Paul
Ekman, Richard J. Davidson.
The conceptual wearable emotion interface will be capable of activating the muscles
involved in natural smiling as per the Duchenne marker (Duchenne de Boulogne GB. In:
The Mechanism of Human Facial Expression. Cuthbertson RA, translator and editor.
New York: Cambridge University Press; 1990), which states that the facial muscles:
Zygomatic Major and Orbicularis Oculi are involved in our natural smile. You can
read about them in detail in section 4, part 1.
On stimulating these facial muscles through electric currents, corresponding emotions
have the potential to be recreated in the individual. Prior research had been done to
figure out if facial muscle stimulation can trigger emotions through this paper: Zariffa,
J., Hitzig, S. L. and Popovic, M. R. (2014), Neuromodulation of Emotion Using
Functional Electrical Stimulation Applied to Facial Muscles. Neuromodulation:
Technology at the Neural Interface, 17: 85–92. doi: 10.1111/ner.12056. According to
which, researchers found out that emotions such as: daring, concentration and
determination were brought out in individuals.
3. Dimensions of emotion
Here, emotion can be described as a point in two dimensional space of affective valence
and arousal. Valence is used to describe our positive (happiness based) and negative
(sadness based) emotions and arousal is used to express our intensity levels from calm
to excited, even fear or anger can be included in this. Neutral condition of emotion can
also be incorporated.
There is also a concept called mixed emotions where a person experiences simultaneous
emotions. For example; a person feels happy when he sees his old grandmother after a
long time but at the same time he is sad because of her deteriorating health. Four
emotion categories were used for testing in the research stated below: neutral, mixed,
positive and negative (valence only).
Moreover, it was found that the EMG of frontalis muscle is insignificant with emotions.
Zygomaticus major muscle is the most discriminatory signal and its mean, absolute
deviation, standard deviation and variance was calculated. Reference: Computing
Emotion Awareness Through Facial Electromyography, Broek, Egon L, Schut,
Marleen H, Westerink, Joyce H. D. M, Herk, Jan, Tuinenbreijer, Kees, Huang, Thomas
S, Computer Vision in HumanComputer Interaction: ECCV 2006 Workshop on HCI,
Graz, Austria, May 13, 2006. Proceedings
4. Conceptual prototyping: Facial EMG and EMS
1. Sensing of emotion through facial EMG
Here, an EMG sensor: link could be used along with a platform like Arduino to sense
facial muscle activation. Using this setup, different muscles on the face will be tested
which corresponds to different emotions. Paul Ekman’s work: Unmasking the Face
could be used as a guide to understanding the various emotions and the underlying
muscles involved for displaying them on the face. Following muscles are associated with
the given emotions:
Happiness: Zygomatic major and orbicularis oculi
Sadness/anger: Corrugator muscle
They can be visualised as shown: (reference video)
Interesting Fact: All the muscles in the face are controlled by the VII cranial nerve
(CN VII) i.e., the facial nerve. And research could be done to find whether this nerve can
be tapped to decode our facial expressions from a single point.
The research paper: H. C. Ning, C. C. Han, C. H. Yuan, “The Review of Applications and
Measurements in Facial Electromyography,” Journal of Medical and Biological
Engineering, vol. 25, pp 1520, 2004 also helped in understanding the process involved
in measuring facial EMG. This paper talks about the measurement problems seen in
recording facial EMG. It has divided the process of recording into three stages:
1. Electrode selection and placement
2. Facial EMG recording
3. Signal conditioning
Surface EMG being noninvasive is very safe and easy to use compared to invasive
methods even though it experiences crosstalk signal originating from surrounding
muscles and signal to noise ratio is less. However, safety is more important than
accuracy, so most people adopt noninvasive surface electrodes (Fridlund AJ and
Cacioppo JT, "Guidelines for human electromyographic research," Psychophysiology,
23(5): 567589, 1986) ( Cole KJ , Ko, packi RA, Ab, and JH, "A miniature electrode for
surface electromyography during speech," J Acoust Soc Am, 74: 13621366, 1983)
a. Physiological effect of emotion on facial muscles
The paper: Larsen JT, Norris CJ, Cacioppo JT (September 2003). "Effects of positive
and negative affect on electromyographic activity over zygomaticus major and
corrugator supercilii". Psychophysiology 40 (5): 776–85.
doi:10.1111/14698986.00078.PMID 14696731 has helped me in understanding in detail
the effects of emotion on our facial muscles, mainly zygomaticus major and corrugator
supercilli. The following learning is based on the above mentioned research.
Courtesy: indiandentalacademy.com
Pleasant stimuli exhibits greater EMG activity over the Zygomaticus Major than the
Corrugator Supercilii than do unpleasant stimulii.
Corrugator supercilii is sparsely represented in the motor cortex and is therefore likely
to be involved in fine voluntary motor behaviours such as articulation and nuanced
display designed to mask affective reactions. Zygomaticus major and other neighbouring
parts are well represented in the motor cortex, affording it greater involvement in
display rules and other fine voluntary motor behaviours. Moreover, like the muscles of
the back, the corrugator tends to be bilaterally innervated, another characteristic that
impedes fine voluntary motor control. In contrast, the zygomaticus like the muscles of
our fingers offers greater contralateral innervation.
The zygomaticus offers a quadratic effect of valence rather than the linear effect. Activity
over zygomaticus major appears to be characterized by a threshold effect and few
stimuli in the current study exceeded that threshold.
When people were exposed to different pictures to record their emotions, very negative
and mildly negative stimuli elicited greater activity over corrugator supercilii muscle
than did neutral pictures. And very positive pictures elicited less activity than neutral
pictures.
Very positive pictures potentiated activity over zygomaticus major and neutral pictures
inhibited activity. Due to this inhibition, mildly positive and very negative pictures
elicited greater activity over zygomaticus than neutral pictures. Increase in self reported
negative affect potentiated activity over corrugator muscle whereas increase in self
reported positive affect inhibited activity.
Increase in positive affect potentiated activity in zygomaticus but increase in negative
effect had little effect. Showing that it is not a reciprocal relationship. Thus with respect
to affective variables (positive, negative), activity over zygomaticus major may give
greater specificity than activity over corrugator.
Their findings that positive affect decreases and negative affect increases activity over
corrugator supercilii suggests that an ambivalent stimulus may have antagonistic effects
on activity over corrugator supercilii, resulting in little change in activity. In contrast,
findings that positive affect increased and negative affect had no effect on activity over
zygomaticus major suggests that ambivalent stimuli may increase activity over
zygomaticus major.
b. Facial EMG measurement
Electrode specification for facial EMG
In EMG measurements, all recording electrodes should be made of the same material to
minimize halfcell potential differences. Circular electrode made with Ag/AgCl is most
often adopted (Hermie J Hermens, Bart Freriks, Catherine DisselhorstKlug, and
Gunter Rau, "Development of recommendations for SEMG sensors and sensor
placement procedures," Journal of Electromyography and Kinesiology, 10: 361374,
2000.)
Best site for electrode placement
The best site for electrode placement is the midline of the muscle belly (C.J.De Luca,
"The use of surface electromyography in biomechanics," Journal of Applied
Biomechanics, 1997). Lapati et al suggested that we could find the detail muscle position
by touching the contracting muscles (Lapatki, B. G., Stegeman, D. F., and Jonas, I. E.,
"A surface EMG electrode for the simultaneous observation of multiple facial muscles,"
Journal of Neuroscience Methods, 123(2): 117128, 2003)
Reducing measurement errors
Movement/deformation of skin under the electrode can result in measurement error by
changing the skinelectrode impedance. To reduce this, the most convenient method is
to use separating conductive gel. Any mechanical disturbances caused between the
electrode and skin can be damped by this intermediate gel.
To reduce power line interference and clear power noise in measurement, shielding the
electrode lead wires and measurement devices is done.
Likelihood of crosstalk from other muscles can be eliminated by attaching the electrode
in the midline of the muscle belly. The double differential technique of using three
electrodes can eliminate crosstalk.
A block diagram of the signal acquisition process (from H. C. Ning, C. C. Han, C. H.
Yuan, “The Review of Applications and Measurements in Facial Electromyography,”
Journal of Medical and Biological Engineering, vol. 25, pp 1520, 2004):
Interesting fact: Facial EMG was recorded for different taste effects. Subjects
described the hedonic sense and facial EMG activities in the levator labii
superioris/alaeque nasi muscle regions were recorded. And thus experiments showed
that facial EMG could be used as a palatability factor (Hu, S., Player, K. A., Mcchesney,
K. A., Dalistan, M. D., Tyner, C. A., and Scozzafava, J. E., "Facial EMG as an indicator
of palatability in humans," Physiology & Behavior, 68(1): 3135, 1999).
2. Electrical stimulation of muscles to invoke emotions
On the other hand to output emotions by stimulating the facial muscles (zygomatic
major and orbicularis oculi) we can use electric stimulation through the existing
methods of EMS (electronic muscle stimulation) or TENS (Transcutaneous Electric
Nerve Stimulation).
EMS is mainly used for treating muscle fatigue, by helping the muscles contract and
relax. Whereas TENS is mainly used to suppress pain by interfering with the nerves and
their signals. Experimentation has to be done to find out which of the two methods are
effective in recreating emotions. A proposed EMS machine: link. Voltage in the range of
DC 12V and current in the range of 0100mA is usually applied in pulses through these
systems.
5. Similar research
1. Facial Performance Sensing HeadMounted Display
Hao Li, Laura Trutoiu, Kyle Olszewski, Lingyu Wei, Tristan Trutna, PeiLun
Hsieh, Aaron Nicholls, Chongyang Ma ACM Transactions on Graphics,
Proceedings of the 42nd ACM SIGGRAPH Conference and Exhibition 2015,
08/2015 – SIGGRAPH 2015
Here, as part of a research done by Oculus and Facebook, researchers have
figured out a way to allow face to face communication in virtual worlds. Since
head mounted displays (HMD) occlude our face, cameras cannot be used to
record the facial expressions in the upper part of our face. Hence, they have
created a system where the inner foam lining of the HMD is embedded with
strain gauges that record facial muscle activity. A camera is also used to record
the facial expressions near the area of our mouth. Check out their demo video:
https://youtu.be/rgKkEnaaSDc
2. Wearable Device for reading facial expressions
Gruebler, A., Suzuki, K., “Design of a Wearable Device for Reading Positive
Expressions from Facial EMG Signals“, IEEE Transactions on Affective
Computing, (accepted), 2014.
This wearable device measures distal EMG signals (signals from a particular
distance away from the facial muscles) and estimates emotions using that data.
Since the data was not taken directly from the affected muscles, techniques like
independent component analysis and artificial neural networks have been used to
achieve a high emotion recognition rate.
3. Expression Glasses from MIT Media Lab
Scheirer, J., Fernandez, R., & Picard, R. W. (1999). Expression glasses a
wearable device for facial expression recognition. Computers in Human
Interaction (pp. 262264). Pittsburgh, USA.
Another interesting research to sense a person’s emotion was: “Expression
Glasses”. This wired system uses piezoelectric sensors on the forehead to detect
either confusion or interest is evoked in a person and the system is modelled in
the form factor of spectacles. This is mainly focussed on obtaining feedback for
different processes. The wired system, not truly a wearable relied on a windows
95 based desktop having labview to process the information.
6. Interesting research: Measure emotion using Galvanic Skin
Response
Based on: The Galvactivator: A Glove that Senses and Communicates Skin
Conductivity In Proceedings of the 9th International Conference on HCI (2001), pp.
15381542 by Picard, J. Scheirer
The Galvactivator is a device which measures electrodermal response or skin
conductivity, this response is directly associated with our emotional arousal and can be
a good predictor of emotions like: attention & excitement. The conductivity of our skin
effectively changes based on the intensity of our emotional state and thus the
researchers have compressed the form factor of the system to a wearable and enabled
the measure of responses for a person engaged in everyday activities and made it free
from the tethered laboratory setup.
However, electrodermal response makes it difficult to predict what made you excited
unless several potentially confounding factors are controlled. It requires a person to
remain seated in a place where temperature and humidity are fairly constant and there
is minimal physical activity and the wearable shouldn’t be tampered with. This further
bolsters the need for a more robust and accurate wearable that can effectively track our
emotions and which is why the current research is pursued.
7. Further research
In order to successfully create a wearable headset capable of tracking emotions, further
research has to be done on the following areas:
1. To create a minimalistic design for a headset/individual micro sticker modules
that can be placed in contact with the facial muscles for sensing. Thus eliminating
the labyrinth of wires usually associated with the system.
2. Research can also be done to find out if there is a central point on our face or
elsewhere, through which all the facial muscle’s electrical activity can be sensed.
This idea is akin to the myo band, where a single area of the arm is used for
detecting a wide range of hand gestures (tapping the facial nerve or CN7 for data
is a possibility).
3. Find correlation between body’s autonomic activity to certain emotions. Research
should also be done to incorporate sensors that can measure galvanic skin
response, heartrate, sweat gland, respiration, blood pressure, body
temperature on to the wearable headset.
8. Applications of the proposed wearable emotion interface
VR In Virtual Reality, more and more technologies are being developed to make the
experience more real and organic, starting from omnidirectional treadmills to body
movement trackers. However, there still hasn’t been any system that has been developed
that can convey the emotion of the person in real time to the system. An emotion based
feedback to the system can allow real time adaptation of the virtual environment taking
into effect the person’s current emotional state and hence allow the system to evolve the
story/scenario around that input.
Media feedback and output Here, realtime feedback can be obtained from a
person while he/she is listening to music or any other media. The emotional feedback
can allow the intelligent playback system to identify the kind of content and media that
the person likes and hence create a curated list of similar songs/media.
It also opens up avenues in the form of big data for corporations which will allow them
to find which content/media that the people are more attracted to.
Moreover while using any media, for example music, the wearable interface can also
amplify the overall experience of the user by stimulating or evoking emotions pertaining
to the song via the interface and augment the pleasure.
New emotion based organic communication platforms The wearable interface
tries to make long distance communication more organic by helping convey real time
emotions of people during text based/other kinds of conversation. The interface can
allow sensing and transfer of emotions from one end and also mimic the same emotion
on the other end using electrical stimulation of the corresponding facial muscles.
Training autistic individuals to understand emotions (reference) Most autistic
individuals are unable to understand or recognise emotions due to the improper
development of the amygdala and temporoparietal junction in the brain. Using this
wearable interface, the autistic person can understand the emotions of another person
who is wearing the interface by the stimulation of his/her own facial muscles through
the second interface they itself are wearing and thus mimic and evoke the same emotion
in the individual. This system can hence be used as a training platform to help autistic
individuals understand facial expressions and thus support them in becoming better
social creatures.
Gaming Here, the emotion of an individual can be fed to the system to allow
remodelling of the storyline of the game to cater to the feelings of the user.
Lie detection This system proposes the creation of a new lie detection platform
which can sense all the micro expressions the face makes while communicating verbally.
Paul Ekman has proved in his work that these micro expressions, which are made
unconsciously by the brain are reliable for detecting whether a person is honest about
something or not. Thus the wearable emotion interface can allow the development of a
new breed of lie detectors. The TV series: Lie to Me is based completely on this concept.
Facial muscle based control for paraplegics This interface can allow paraplegics
to use their facial muscles to control digital systems or even move a wheelchair in the
desired direction.
9. Conclusion
The concept presented here also raises the fundamental question of whether emotions
should be manipulated (recognised and induced)?
For answering this, I rely on the expertise of R.W.Picard (Founder, Affective Computing
Research Group, MIT Media Lab) who states that there is nothing wrong with either
sensing emotions nor invoking them; the worst that could happen is emotion
manipulation for malicious purposes.This sort of manipulation is already happening
(cinema, music, marketing, politics, etc) why not use computers to better understand it
(R.W.Picard. Affective Computing, M.I.T Press, Cambridge, MA)
The need for a wearable emotion interface is expressed clearly in H. C. Ning, C. C. Han,
C. H. Yuan, “The Review of Applications and Measurements in Facial
Electromyography,” Journal of Medical and Biological Engineering, vol. 25, pp 1520,
2004. According to their research, while recording the EMG, they hope to decrease the
number of electrodes on the face so that the design of the system is easier and reduces
development cost. Furthermore, the existence of electrodes will affect the movement of
facial muscles and decrease validity of measurement. The facial electrodes will impair
the appearance of subjects and reduce acceptance for facial EMG products. For design of
facial sensor, it is a critical task to find the position of interesting facial muscles and fix
the electrodes on the face without disturbing facial movement. As a result, more
research would have to be done as explained in section 7 to create systems that can
understand human emotions and cater more relevant content to us.
By Arvind Sanjeev Founder, DIY Hacking