Smart Data Webinar: See Me, Feel Me, Touch Me, Heal Me - The Rise of the Cognitive Interface

See Me, Feel Me, Touch Me, Heal Me The Rise of the Cognitive Interface

Adrian Bowles, PhDFounder, STORM Insights, Inc.

Lead Analyst, AI, Aragon Research

[email protected]

NOVEMBER 9, 2017

mailto:[email protected]

Context: Advanced Human Computer Interfaces Perception & Perceptive Input Management Emotive Output When to Use Advanced IO

Copyright (c) 2017 by STORM Insights Inc. All Rights Reserved.

AGENDA


NATURAL HCI HAS BEEN AN AI GOAL SINCE THE BEGINNING

PERCEPTIONUNDERSTANDING

LEARNING

PROBLEM-SOLVING

NATURAL LANGUAGE

PROCESSING

Big Data

Classic AI

Deep Learning

Context


HOW DO WE COMMUNICATE?


NLP

Natural Language Understanding NLU

Natural Language Generation NLG

?

COMMUNICATIONS GOES BEYOND NATURAL LANGUAGE PROCESSING (NLP)


People are good - but not perfect - when communicating in natural languages. We…

understand in context (environment & our own frame of reference) attempt to resolve ambiguity have to deal with competing signals, noise fill in words and meaning and may not hear/understand use our knowledge about the speaker and the situation to filter the input use visual and auditory cues to distinguish between what was said vs what was meant…

X

X’

Y

HOW DO PEOPLE COPE WITH IMPRECISION?


WHAT DO WE COMMUNICATE?

Facts

I send you a text at 2AM something bad happened

Concepts

I am angry

I am inconsiderate

I exaggerate


“You’re Not Just Responsible For What You Say, You Are Responsible For What People Hear”

August 9, 2016

General Michael Hayden (Retired), former Director, Central Intelligence Agency and National Security Agency

on advice to his senior staff.

NATURAL LANGUAGE PROCESSING IS HARD - FOR PEOPLE

Watch out for slang, context…


PICTURES VS WORDS


GESTURES ADD RICHNESS TO LANGUAGE


“THAT’S A BIG HOUSE” - MEANING MAY BE DIFFERENT FOR DIFFERENT SPEAKERS

Bob Mary

Al (Capone)

Wikipedia contributors. "Alcatraz Federal Penitentiary." Wikipedia, The Free Encyclopedia. Wikipedia, The Free Encyclopedia, 30 Oct. 2017. Web. 8 Nov. 2017.


DRAW A QUARTER, TO SCALE - RESULTS DIFFER ACCORDING TO HIDDEN CONTEXT


Hearing (audioception) ~12,000 outer hair cells/ear

~3,500 inner hair cells

Vision (ophthalmoception) Photoreceptors - Per Eye~120,000,000 rod cells

(triggered by single photon)~6,000,000 cone cells

(require more photons to trigger)~ 60,000 photosensitive ganglion cells

Touch (tactioception) Thermoreceptors, mechanoreceptors, chemoreceptors and nociceptors for touch, pressure, pain, temperature, vibration

Smell (olfacoception) Chemoreception

Taste (gustaoception) Chemoreception

Human Cognition~100,000,000,000 (100B) Neurons

~100-500,000,000,000,000 (100-500T) Synapses

THE HUMAN ADVANTAGE FOR PERCEPTION & COGNITION

LearnReason

Understand


Listen Look

Touch

Smell Taste

Your System

LearnModel

Reason

Understand

PERCEPTIVE INPUT: IT’S A NEW SENSATION Perception


HOW DO WE COMMUNICATE?

A Mathematical Theory of Communication, C. E. Shannon,

The Bell System Technical Journal, Vol. 27, pp. 379–423, 623–656, July, October, 1948.

Systems

Controls

LearnReason

Understand

Model

Data Mgmt

Human

Machine

Input Output

Gestures

Emotions

Language

Narrative Generation

Visualization

Reports

Haptics

Sensors (IOT)

SystemsControls


COGNITIVE SYSTEMS: COMMUNICATIONS & CONTROL

Perc

eptio

n

Human Input

Human-Oriented Output

GesturesEmotions

Language

Narrative GenerationVisualization

Reports

Haptics


COGNITIVE SYSTEMS: COMMUNICATIONS & CONTROL

Perc

eptio

n

MeaningConcepts

Intent

Context

Animation

LearnReason

Understand

Model

Data Mgmt

Human Input

Gestures

Language


PERCEPTIVE INPUT CLASSIFICATION

Context

LearnReason

Understand

Model

Data Mgmt

Detected by Human Senses

Derived

ImagesSee

Hear

Touch

SmellTaste

Sounds

ObjectsEmotions

Meaning

Concepts

Intent

MULTI-DIMENSIONAL INPUT ANALYSIS

Source: IBM Bluemix


TONE ANALYSIS OF TEXT: IBM WATSON

Emotion tones: Anger, Disgust, Fear, Joy, and Sadness.

Writing tones: Analytical, Confident, and Tentative.

Social tones: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism.

Source: IBM


UNDERLYING ALGORITHMS ARE COMPLEX

Source: Affectiva

Depend on detection and classification of word use for text and physiological state changes for audio and video.


DEEP LEARNING FOR FEATURE, ENTITY, CONCEPT EXTRACTION/RESOLUTION (SIMPLIFIED!)

Visible Layer

Hidden Layers

Output Layer

Input: Observable Variables

H

IGH

AB

STR

ACTI

ON

LO

W

Output

Pixels

Depthof the Model

Edges

Object

Shapes/Parts

Object Class

Brightness/Contrast

GeometryRules

Featuresto

Extract

Methods

1. Feed Forward

2. Back-Propagate(adjust weights)

Model


CONCEPTS IN SPACE: IMPORTANT ASSUMPTIONS

Principles that control the development and representation of natural intelligence in the neocortex provide a guide to the implementation of machine intelligence.(Numenta Hierarchical Temporal Memory) A function applied to a string representing data or a concept results in a value or vector meaningful for comparison.

e.g. Using Kolmogorov complexity to measure the strength of relationships in Memory-Based Reasoning.

DistanceA metric for the similarity of two items based on their relative locations in n-space according to an algorithm.


Proximity Mapping

BoyBay

Map

Mop

Man

Nay May

MopeBuy

Hop

Mapped with vectors, proximity algorithm based on purpose.

Mapping for autocorrect/complete vs Mapping for meaning

BoyBay

Map

Mop

Man

Nay

May

Mope

BuyHop

Hope

HopeSimilar structure -> similar meaning in vision, not always in language.

CONCEPTS IN SPACE

Model


COGNITIVE COMPUTING FUNDAMENTALS: SAMPLE ASSUMPTIONS

ModelThe Corpus, Assumptions, Algorithms

Used to Generate & Score Hypotheses

or Calculate The Strength of a Relationship

Relationships can be discovered without understanding the meaning of any individual string, fact, or (NL-free concept modeling)


RECOGNIZING CONCEPTS VS UNDERSTANDING

Courtesy of LoopAI Labs.


EMOTION DETECTION IN VIDEO

7 EmotionsAnger Sadness Disgust Joy Surprise Fear Contempt

Source: Affectiva


KEY TREND: EMOTION DETECTION AS A FEATURE OR SERVICE

Source: Affectiva


HCI ADVANCES AUGMENT UNDERSTANDINGG

estu

re R

ecog

nitio

n/An

alyt

ics


NEXT TREND? USING SMARTPHONE SENSORS FOR RICHER CONTEXT

Type Typical Current Use

Accelerometer/motion Rotate screen, Switch screen to landscape/portrait

Ambient Light Adjust screen brightness

Barometer Measure altitude

Geo-Location (wifi/cellular) Location/Alerts

3-Axis Gyroscope Rotation rate for games, VR…

Proximity Turn off screen when phone is by your head

Touch ID fingerprint Security


EXPRESSIVE OUTPUT: EMOTIONS, URGENCY

Model

Emotions Meaning

Concepts IntentContext

Narrative GenerationImages

HapticsAnimation

Using Language, Avatars, Emotive Text to Speech

Emotive Output


EMOTIVE TEXT TO SPEECH IS NOT NEW


DATA Model/Article

Structure Template

Generated Narrative

NLG TECHNOLOGY CAN USE EMOTIONS IF THE MODEL SUPPORTS IT

Needs Context-Relevant Tone Assignment Data

Model


Copyright (c) Narrative Science.

COMMERCIAL NLG TECHNOLOGY WITH TONE ATTRIBUTES


NEUROBEHAVIORAL ANIMATION


Sentiment/Emotion/Theme/Concept Analysis

Don’t let the search for perfection interfere with the path to progress.

WHEN SHOULD YOU USE PERCEPTIVE INPUT/EMOTIVE OUTPUT? Getting Started


WHEN SHOULD YOU USE PERCEPTIVE INPUT/EMOTIVE OUTPUT?

When It Improves the User Experience

The System Tone Should Complement,Not Mirror, the User Tone

When You Can Select or Generate Appropriate Responses/Actions

[email protected]

Twitter @ajbowles Skype ajbowles

If you would like to connect on LinkedIn, please let me know that you that you registered for the Smart Data webinar series.

KEEP IN TOUCH

Upcoming SmartData Webinar Dates & Topics

Dec. 14 The Road to Autonomous ApplicationsJan. 11 AI At The Edge: Pushing Intelligence to Fog Computing NodesFeb. 8 A Pragmatic AI Maturity Model: Choosing the Right AI Technologies Based on Application Requirements and User Attributes

mailto:[email protected]?subject=

Smart Data Webinar: See Me, Feel Me, Touch Me, Heal Me - The Rise of the Cognitive Interface

Technology

Transcript of Smart Data Webinar: See Me, Feel Me, Touch Me, Heal Me - The Rise of the Cognitive Interface