Lecture 10

Smart u-Things and Ubiquitous Intelligence

What are smart u-things?

Classifications of smart u-things

Features of smart u-things & ubiquitous intelligence

Challenges of smart u-things & ubiquitous intelligence

e-Thing & e-Activity in Cyber World

e-Things: digital things in Cyber World on cyber space

Computers/Networks/Internet

Cyber Space

Web, WbS, SmW, Grid, P2P, EaaS, Cloud

e-Thing

e-Activity

Cyber World Physical World

Physical Thing

Usual Activity

u-Things Physical Things with AEB Two Fundamental Technology Trends

Continuing miniaturization of devices (Moore’s law, new material, nanotech., …)

Available interconnections by ubiquitous/pervasive wired and wireless networks

u-Things: Physical things with some kind of Attachment, Embedment, Blending

(AEB) of computers, sensors, tags, networks, and/or other devices

0.15x0.15 mm, Hitachi/06

u-Things in Physical/Real World

Physical World

u-Thing

Usual Activity

Sensor/M/NEMS, Comps & Per. Nets

Physical Space

UC, ID, Context, Emb. Sys., etc.

u-Things: on the physical space in physical-digital form

e-Thing

e-Activity

Cyber World

Computers & Networks/Internet

Cyber Space

WbS, SmW, Grid, P2P, EaaS, Cloud

Characteristics in 3 Computing Waves/Eras

m-to-1MC

1-to-1PC

1-to-mUC

Main Role/Obj

Proc. Content

Calculations Information Services/Intelligence (?)

Data Media

u-Things + Cloud Mainframes Personal ComputersComp Element

Context

Size/Form/User

Weiser’s3 Relations3 Waves

Basic Behavior Passive ActiveInteractive

Large/Stationary/Few Small/Portable/Many Invisible/Everywhere/All

Theory/Tech Comp Sci/Eng Info Sci/Eng (??? Science)Smartness/Intelligence

Central Goal Fast/Precise/Reliable Rich/Convenient/Secure Aware/Comfort/Safe

Intelligent Computing Perspectives

Social-driven + Web-oriented- Autonomous software- Multi/Massive agents- Agent language and semantics- Agent negotiation & cooperation- Personal/social behavior- Web Intelligence (WI)

Physical-Cyber Aware- Physical/everyday things’ intelligence- Atop of the above intelligent comp- Life-like, diverse things and all users- Uncertain, unpredictable, unsafe,

incomplete, controllable, manageable, diverse, exception, emotion, ethic, …

- Others-aware & self-aware mind/spirit?

Human-like- Machine learning- NLP & comp-vision- Robot & game theory- Expert system- Knowledge/reasoning- DAI (Distributed AI)

Nature-inspired- Probabilistic computing- Fuzzy logic- Neural network- GA/Evolutionary computing- Chaotic & Swarm computing- Biologic computing- Deep Learning

Smart u-things Ubiquitous Intelligence (UI)

Essential Questions

Can machines think ? AI (Artificial Intelligence)

- By A. Turing in “Computing Machinery and Intelligence” in 1950

Can things think ? Intelligent real/virtual entity

- Consortium of Things That Think (TTT) in MIT Media Lab since 1995

- “When things start to think” by Neil Gershenfeld of MIT in 1999

Can smart u-things be everywhere ? UI (u-Intelligence)

- Ubiquitous Intelligence (UI) or Pervasive Intelligence ( ) from 2004, by Ma

- Residing in everyday objects, environments, systems, ourselves, plant, animal, …

- Real Things u-Things Smart Things Smart World/Ubiquitous Intelligence

p

IBM’s Smart Planet, New Intelligence 2008

Hyperworld and Smart World Hyperworld (1995)

- “Hyperworld Modeling”, Keynote in VIS, Melbourne, Feb. 1996 (Kunii, Ma, Huang)- “Its basic characteristic is direct mapping between virtual and real worlds via

active devices including sensors, actuators, micro-machines, robots, etc.” - 1996- “A Study on a Hyperworld System of One-to-Many Interaction”, ICAI’97 (Weiser’s 1-to-m relationship)

- “Towards Direct Mapping between Information Worlds and Real Worlds”, LNCS1306, 1997

Smart World (2003)- Based on ubiquitous computers, sensors, actuators, networks, etc.- Pervaded with smart, life-like ubiquitous things/u-things- Characterized by u-services & u-intelligence in the real world

Smart e-Thing

e-Service

Cyber World Smart World

Smart u-Thing

u-Service

Hyperworld

Cyber-Physical-Social Spaces

Web-Intelligence

WIUbi-Intelligence

UI

Ubiquitous IntelligencePervasive smart real things able to sense, think, act, …

Information Explosion Intelligence Explosion!

- The Intelligence Revolution, Interview by Wise Media, ID People Magazine, Apr./05- Ubiquitous Intelligence Summit 2005, A Think Tank, Oulu, Finland, June 15-17, 2005- Journal of Ubiquitous Computing and Intelligence (Yang & Ma), American Sci. Pub./05- IEEE Task Force on Intelligent Ubiquitous Computing (Duman, Yang, Ma), since 2006 - Journal of Autonomic and Trusted Computing (Yang and Ma), ASP, USA, ASP, 2006

Smart Object (Smart u-Object, Smartifact)- A physical object with AEB devices and some smartness/intelligence- Handheld, card, label, sensor, artifact, appliance, goods, furniture, textile, robot, ……

Smart Environment (Smart u-Environment)- A physical spatial environment/site integrating smart u-objects &/ devices- Smart u-Services via these objects/devices and their commun./cooperation

Smart System (Smart u-System)- May be a physical system/network for management, monitoring, emergence, …- May be a platform or middleware for smart u-objects/environments/services

Ubicomp/Percomp/AmI/CPS/IoT Computing of Smart u-Thing

Smart u-Things

Plant talks, IEEE Spectrum, May/09

Smart Grid, IEEE Spectrum, July/09

Basic features of smart u-things- Sense & effect for perception and interaction- Networking & naming for interconnection and communication- Computation & storage for information processing and memory

Smart u-Things – MIT Lover’s Cups

Explore the idea of sharing feelings of drinking between two people in different places

Two cups are connected with each other, and glow when your lover is drinking.

Smart u-Things – Smart Slippers & Nike

Smart u-Things – GrowCaps, etc.

Smart u-Things – Smart Home

Videos About Smart Object and System

Smart Trashbox – 勝手に入るゴミ箱

Smart Systems – Toyota’s Smart Mobility Society

Sony SmartWatch2 , Samsung Galaxy Gear ,

Galaxy Gear Concept 2013

Smart Shoe – Nike smart trainers

Smart Pill Bottle , Smart Pill Box , Smart Pill

Smart Toy – Smartphone + Toy

Smart Space/Environment

Smart Space/Environment Computerized active one

- Physical environments but digitally enhanced/integrated

- Awareness of contexts: users, ambience, resources, etc. - Context-Aware responses or services with certain intelligence- Where are users? Supervisors & computing elements!!

Sensors

Actuators

Space

Environment

Object

Artifact

Plant

Body

(User)

Ubi/Pervasive

Information

- Acquisition

- Archive

- Analysis

- Awareness

- Application

- …

MM Information

Comfort Service

Context-Aware

Context

Loop

Smart Space/Environment

Status/ attend

Status/ absent

RFID

Reader

Room １ Room ２

Room ３

Motion‘Organ’

Memory‘Organ’

Percep.‘Organ’

Brain

BathToilet

StudySleep

Entrance Kichen

Living

Dinning

Action‘Organ’

Think‘Organ’

Control‘Organ’

Smart Physical Objects Management

Things Mapping - toy- game- key- …

Thing Reminder- umbrella- book- pencil- …

Users’ IndoorLocation Sharing

ユーザの屋内位置情報の共有

ROOM

サーバー　　　

ROOM

アクセスポイント１

データベース

位置推定行動推定

　アクセスポイント２ログデータ

ログデータ

電波強度加速度データ

EV２

EV１

User Location-Aware Music Player

Music data sent to speaker(s) near a user’s location

Sound-Aware Smart Space

Status/ attend

Status/ absent

RFID

Reader

Room １ Room ２

Room ３

Sound-aware Reminder- Ambient sound capture- Ambient analysis- Advice judgment- Advice presentation- …

Sound Disturbance

- home

- classroom- theater- …

Robots forMobile Sensing & Object Finding

R-Reader

R-Controller (PDA)

R-Machine

Ultrasonic Sensor

Motor

Camera

R-Reader

R-Controller (PDA)

R-Machine

Ultrasonic Sensor Motor

Lifelog – Personal Experience/DB

LabLog – Info Sensing & Management

Sun SPOTS

Wireless networks

& Sensors

Java program

SSG

SSGOn Robot

(Time)

24℃

(Raw) SSG (Door)

(Time)

(Raw)

25℃

SSG (Robot)

HomeLog – Kid’s Activity Summarization

ID Tag and Sensor for User2007/11/10

12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11

1 Sensor A(study)

Sensed Activity Period

4

3 Sensor F(study)Sensor G(study)

2 Sensor E(study)

5 Sensor H(study)

Sensed Activity Change Time

Sensor for User12 1 2 3 4 5 6 7 8 9 10 11 12 1 432 5 6 7 8 9 10 11

2007/12/11

部屋入口で認識した子どものRFIDタグ

机の椅子のタッチセンサー

教科書のRFIDタグ

机付近で認識した子どもののRFIDタグ

漫画本のRFIDタグ

ID Tag and Sensor for User2007/11/10

12 1 2 3 4 5 6 7 8 9 10 11 12 1 2 3 4 5 6 7 8 9 10 11

1 Hosei CIS

Kid’s Name12 1 2 3 4 5 6 7 8 9 10 11 12 1 432 5 6 7 8 9 10 11

2007/12/11

Sleep Game Study Sleep

Videos About Smart Things and Services

Green Wheel - Smart Mobility & Ubiquitous Computing- 10 scenarios on Smart Ubiquitous Services

Display Glass - Smart Ubiquitous Displays- Displays in wall, surface, mirror, veneer, etc.

Smart Home - Home Energy Management System (HEMS)(community) - Smart energy supply, use, control, etc.

IEEE CIS Smart World Technical Committee

Smart & Smartness: Terms and Characters

Smart related computing terms- Smart Intelligent Life-like- Sentient, Aware, Context-aware, Active, Reactive, Proactive, Assistive, Adaptive, Automated,

Autonomic, Amorphous, Organic, Spray, Perceptual, Cognitive, Thinking, Self-*, …

Smartness features in general- Some kind/level of intelligence, but softer, wider and flexible than “intelligence”- From simple reactive functions to complex intelligent or life-like behaviors- Anything from real to e/virtual ones, but with emphases of real/physical ones- Felt relatively, not absolutely two co-exist facets: smart and stupid!- Complicated or abstruse philosophical, culture social, ethical and other implications

Ultimate goal of UI and SW- To make u-Things calm, trustworthy, safe with context-/self-awareness- To move u-Services from ANY place/time/means to RIGHT place/time/means

Smart u-Things challenges Technology Complexity/Challenges: discussed widely and realized relatively well Real World Complexity/Challenges: addressed rarely and realized not very well

Smart u-Things – Technology Challenges

Three basic types of functions in smart u-things- Computation & storage for information processing and memory- Networking & naming for interconnection and communication- Sense & effect for perception and interaction

Technology challenges (not exhausted) Both in hardware and software devices miniaturization, power management, sensor nets, universal ID, ubiquitous/pervasive networks, ad hoc mobility, open service architecture, sensed information overload & database, context semantics & management, autonomic system administration, user interface, operating system, language, middleware, integration, cooperation, scalability, heterogeneity, dependability, availability, security, privacy, test, evaluation, standards, etc.

Ubiquitous Computing Grand Challenge: Manifesto, UK, 2006, EPSRC, UK-UbiNet

Beyond-The-Horizon Workshop on Pervasive Computing and Communications (PCC-05), Vienna

Smart u-Things – Real World Challenges

Smart u-Things are emerging- Many scenarios: Weiser’s Sal, AmI’s Maria/Dimitrios/Carmen, Aura’s Jane/Fred, …- Lots of research and many various prototypes Research Booming comes- But rare practical ones Application Booming is to come real world complexity

Ideal Smart u-Things expected and related challenges Able to act adaptively and automatically according to

1. Surrounding Situations Challenge 1: Situation Approximation

2. Users’ Needs Challenge 2: Knowing Users’ Needs

3. Things’ Relations Challenge 3: Complex Things’ Relations

4. Common Knowledge Challenge 4: Knowledge Management/Growth

5. Own Goal, Role, etc. Challenge 5: Self Awareness

6. Error and Exception Challenge 6: Looped Self Adjustment

7. Safety & Satisfaction Challenge 7: UbiSafe

Above challenges from real world intrinsic characteristics- RW == physical + social + natural + … uncertain, unpredictable, changing, …- RW computing: complicated/abstruse phil., social, ethical & other implications- Understanding real world (RW) diversity and complexity Extremely Hard !!- Novel cyber dimensions newly added in physical-digital combined u-things

Challenge 1 – Situation Approximation

Context- Information characterizing the situation of an entity (Dey) or 5Ws (Abowd)

- Whole contexts are a collection of various values from sensors or other sources

C(t) = {Ci(Bi, L, th), i∈I(t), th∈[t-h, t]}

Situation (Activity)- Relatively compact, more semantic, directly used for judgment/decision

S(t) =φ[C(t)] = {Sj(Wj, t), j∈J(t)}Designing a Context-aware System to Detect Dangerous Situations in School Routes for Kids Outdoor Safety Care, LNCS, Proc. of EUC'05.

C(t), S(t) approximations to a real environment- Are the contexts C(t) sufficient and precise enough to characterize a real environment?

- How correctly be S(t) determined using available but incomplete & uncertain contexts?

- What are consequences of situation judgment errors to context-aware/situated u-things?

Nyquist Sampling Theorem & Shannon Entropy/R(D) Theory Theory for context/situation?

Keep incompletion, uncertainty and misjudgment as basic design assumptions !

Challenge 2 – Knowing Users’ Needs

Interactive Mechanism- A request-response dialog process betw. users (activator) & computer (passive)- 3 user limits: too small/many/complex computers to be visible/interact-able/manageable

Proactive and Autonomic Mechanisms- Take actions proactively by anticipating users’ needs (Tennenhouse, Intel, 2000)

- Manage themselves under human supervisions/needs (Paul Horn, IBM, 2001)"Proactive Control of Group Revision Assistance Management Using P2P Technology" (Takata & Ma, ISCIT04)

Journal of Autonomic and Trusted Computing (JoATC, EiC, Laurence T. Yang and Jianhua Ma), ASP, USA

1st Int’l Wksp on Trusted and Autonomic Ubiquitous & Embedded Systems (TAUES-05, Japan, Dec. 2005)

2nd International Workshop on Trusted and Autonomic Computing Systems (TACS-06, Austria, Apr. 2006)

3rd Int’l Conf. on Autonomic & Trusted Computing (ATC06, China) ATC07 (HK) ATC08 (NW) ATC09 (Australia)

Users’ Needs- Multi aspects and subtle affection by many factors

N(t) = {Nu,v[Pu,v(t), S(t), S(t-Δ)], u∈U(t), v∈V(t)}

Hardness of knowing users’ true needs- How much can be known correctly and promptly for users’ true needs in changing situations?

- “Know you and know your face, but don’t know your mind” - Chinese saw

Challenge 3 – Complex Things’ Relations

Complex Relations among u-Things- “Everything will be connected to everything else” - by R. Lucky, 1999

- “what are the consequences of any given action (including simply walking into a room)”

- by M. Weiser, 1999

Cyber dimensions, except spatial/temporal/other conventional dimensions Complex dynamic relations among users, u-things, and other things

R(t) = {Rq(Tq, Uq, t), Uq∈U(t), q∈Q(t)}

- How to define, find, describe and use complex relations necessary for u-things’ systems?Smart Hyperspace: a set of interconnected smart spaces with situational-spatial-temporal relations

(“Modeling Interface with a Multimedia Hyperworld”, HIS’96, “Smart Hyperspaces and Project Ubikids”, USW05)

users

non u-thingsu-things

Challenge 4 – Knowledge Mgmt/Growth

Why common knowledge needed?- Knowledge is the base of analysis, reasoning, anticipation and judgment (Ref AI)

- Smart u-things are in the physical environments and serve people’s daily life and work- Smart u-things need some common knowledge about physical world, human society, etc.

K(t) = {Km(Fm, th), m∈M(t), th∈[t0, t]}

Issues are- What knowledge is necessary for smart u-things? - What knowledge should be initially set? - What knowledge can be added later on?- What knowledge may be self-learned during uses?- How knowledge is used for rich and varied real situations?

Challenge - how to abstract, learn, use complex knowledge about human & world

How about DAI, swarm, softcomp, agent, semantics, etc.?

Challenge 5 – Self-Aware u-Things

Why smart u-things should be self-aware- Physical, not virtual, things with attached/embedded/blended (AEB) computers/etc.- AEB devices are function parts or components of the real physical things- AEB is for enhancing original functions of real things following common rules- Smart u-things should be aware their roles and function without against the rules

A(t) = {Az(Gz, t), z∈Z(t)}

Context-awareness vs self-awareness- Context-awareness: knowing others- Self-awareness: knowing oneself

“Knowing others is wisdom, knowing yourself is enlightenment” - Tao Tzu

- What this exactly means? How to equip self-awareness to u-things?

Extreme Challenging in this research“we seem so ill equipped to understand ourselves”, The Society of Mind, Minsky, 1986

Workshop on Self-Aware Computer Systems - Chaudhri & McCarthy, SRI/DARPA, 2004

Challenge 6 – Looped Self-Adjustment

Decisions cannot be correct always in 100% Imprecise decision & exception are common

Loop mechanism is indispensable !

Challenge - adaptively correct errors & make adjustments How is this related to automatic control theory?

D(t) =ψ[S(t), N(t), R(t), K(t), A(t), D(t-Δ), E(t)]

ψS(t)

N(t)

R(t)

D(t)

K(t) A(t)

actions

D(t-Δ)

E(t)χ

Error(t)

Exception(t)

Challenge 7 – Ubisafe (Ubiquitous Safety)

Unobtrusive AEB devices in real objects and spaces

Physical unsafe factors

Diverse Users with different ages/backgrounds/demands

Human unsafe factors

Desired States of ubicomp environment

Calm Technology by Weiser and Brown

UbiSafe Ubiquitous Safety & Satisfaction

Basic Question

How to build a computing environment in which all people can fully enjoy the u-services under complex situations without worrying or thinking the safety problem any more?

Ubiquitous Comp Ubiquitous Services Ubiquitous Dangers UbiSafe(Zixue Cheng)

Japan Smart City Pilot Projects

http://jscp.nepc.or.jp/

Japan Smart Community Alliance

Established on April 6, 2010 to promote Smart Community through government-private sector cooperation

As of Nov. 2012, 397companies and organizations (including universities)

Cooperation across companies and organizations

https://www.smart-japan.org/default.aspx

Smart City Project

On Sep. 17th, 2009, Smart City Planning Inc. was established as Joint Venture Company of leading companies for the management of Smart City Project.

As of Sep. 2012, 22leading companies joined the project.

http://www.smartcity-planning.co.jp/

Nikkei Smart City Consortium

http://expo.nikkeibp.co.jp/scw/2011/exhibitors/english/http://expo.nikkeibp.co.jp/scw/2012/exhibitors/english/

http://bizgate.nikkei.co.jp/smartcity/

Numbers of Smart City Projects

http://itpro.nikkeibp.co.jp/article/Keyword/20120906/421044/ As of September 12, 2012

China

JapanIndia

Africa

Far-East

NorthAmerica

SouthAmerica

PacificOcean

Europe

Toyota Smart Center in 20xx

APEC Smart City Forum - Cooperation Road

Distinguished Panelists:

-- Dr. Nattapon Chayopitak，National Institute of Sci. & Tech. (Thailand)

-- Prof. Ching-Tarng Hsieh，National Jiaotong U (Chinese Taipei）

-- Dr. Steshin Alexander Andreevich，Russian Information Belau (Russia)

-- Prof. Qing Li，Hong Kong City U (Hong Kong)

-- Prof. Qun Jin，Waseda U (Japan)

Page 42

To share experiences of smart city R&D&A in different countries/regions

To explore cooperative mechanisms, approaches, programs, etc.

Smarter Cities, Smarter Worlds – Chaired by Jianhua Ma

Read the papers below and access the related websites to learn more about smart u-things include smart objects, smart environment/space, smart systems, as well as ubiquitous intelligence, ambient intelligence, etc.

“Smart u-Things: Challenging Real World Complexity” by J. Ma

“Ubiquitous Intelligence – The Intelligence Revolution” by J. Ma

Smart Environment – Wikipedia

Home Automation – Wikipedia, Introducing the Smart Home

Smarter Planet – Wikipedia, IBM Smarter Planet Home Page

Ambient Intelligence – Wikipedia

IEEE CIS Smart World Technical Committee

Others you like Important to get materials from Web!!

Homework