Pretty Pictures Zen, Data Visualization and the Art of Real-Time Decision-Making

Brandon Satrom @BrandonSatrom

SO. MUCH. DATA.

18% 25%

81%

!

"

#

1

2

4

3

Social Apps

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

App Analytics

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

DevOps

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

WHAT ARE WE DOING WITH ALL THIS DATA? $

-

15

30

45

60

What effect, good or bad, did the last release of your web or mobile

app have on customer conversion?

CONVERSION

15

30

45

60

Did that recent system outage negatively impact website

engagement? How do you plan to address it?

ENGAGEMENT

13

25

38

50

Why have only 40% of Android users installed the latest version of

your app?

ADOPTION

18

35

53

70

What’s causing all of those app crashes anyway?

BLOCKERS

QUESTIONS. QUESTIONS. QUESTIONS.

18

35

53

70

Did that new pair of shoes affect your speed or running style?

PERFORMANCE

15

30

45

60

How can your checkin history tell help you choose a restaurant or a beer or wine from this 3 page list?

PREFERENCES

13

25

38

50

What effect does meeting your step count goal for the day have on

your energy, diet or overall well-being?

GOALS

18

35

53

70

What impact does mood tracking or journaling have on your career

choices?

ASPIRATIONS

QUESTIONS. QUESTIONS. QUESTIONS.

Building better experiences for our customers?

DOES ANY OF THAT DATA MATTER IF WE’RE NOT…

1

2

3

Using insights to change and improve our behavior?

Improving the apps and systems we use to run our businesses?

HOW ARE WE MANAGING THE NOISE?

-

!

., /

0 1

2

3

4

5

6

7

89

:

;

?

=

>

@

#

A

B

D

E

F

G

H

300 billion

3 billion

1 billion

14 million

“BIGGER THAN THE INTERNET”

Internet Growth from 1993 - 2015

The Number of “Connected Things” by 2020

(Projected)

HOW DO WE SEPARATE THE SIGNAL FROM THE NOISE?

Tools for Data Ingestion Tools for Data Visualization Tools for Recommendation

THE RIGHT TOOLS + THE RIGHT APPS = TRUE INSIGHT

Apps that can be human-directed Apps that can learn Apps that take action

THE CONTINUUM OF DATA INSIGHT

INSIGHT

DIRECTIONHuman defines rules or conditions in the system in advance; System takes action when conditions are

met.

RECOMMENDATION

System makes suggestions based

on data & a human takes

action

VISUALIZATION

System provides visuals to help you

reason about data.

COLLECTION

System gathers data and stores it in some location

LEARNINGSystems that take action based on past behavior,

public information or other factors

Real-Time

SPEED OF DECISION-MAKING

SPEED OF DATA INGESTION

Data Collection

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

DATA COLLECTION/INGESTION SYSTEMS

1. Data is created (and stored locally)

2.Data is sent to another location for storage

3.The rest is up to you…

@

#

B

EH

J

KL

M

EXAMPLES OF DATA COLLECTION/INGESTION SYSTEMS

Microsoft Azure

AT&T M2X

Wolfram Data Drop

Telerik Backend Services

Parse

Building Collection Systems

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

• Data storage is foundational, so your options are endless…

• HOWEVER, if you have a choice, optimize for speed (of entry and retrieval)

• Real-time apps begin with real-time transport & storage

• Socket.io • Meteor • Firebase • MongoDB

CONSIDERATIONS FOR DATA COLLECTION SYSTEMS

CONSIDERATIONS FOR DATA COLLECTION SYSTEMS

• Even better, consider backends with built-in analytics capabilities

• InfluxDB + Grafana

• Wolfram Alpha

Data Visualization

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

DATA VISUALIZATION SYSTEMS

1. System aggregates and presents data in a consumable way.

2. Action taken in response, if any, is manual

3. Many Experiences insert “gamification” here to trigger action or improve engagement

,

N

|

• Favor tools that provide automated visualizations of your data…

•OR tools that make it easy to configure and analyze data.

• Automated Data Visualization Tools • Jupyter.Org • Grafana + InfluxDB •Wolfram Alpha •AT&T M2X

CONSIDERATIONS FOR DATA VISUALIZATION SYSTEMS

Data Recommendations

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

DATA RECOMMENDATION SYSTEMS

1. System processes data and suggests one or more actions

2. Human intervenes and takes action or redirects the decision

3. First popular in E-Commerce and Marketing Systems, but applicable elsewhere

18%

Likelihood to purchase a related product based on past history

25%

Category

81%

TYPES OF “PRODUCT RECOMMENDATION” SYSTEMS

MANUAL CROSS-SELL

CROWD-SOURCED

PERSONA- BASED

RULE- & ALGORITHM-

BASED

RECOMMENDATIONS - NOT JUST FOR E-COMMERCE

• Recommendations are applicable to nearly any problem we’re solving with software

• Many examples of recommender tools are popping up on Mobile

• Uses

• Prioritizing bugs based on crash report frequency/geo/other factors

• Suggesting web-pages for optimization based on automated funnel analysis

• Suggesting modification to watering frequency based on soil-moisture readings

78%Enterprise

Software Developers

19%Solo

Developers & Entrepreneurs

{{

• Note: not many tools exist outside of e-commerce/marketing for general use

• HOWEVER, a few open-source options and example applications do exist

• Consider looking at:

• LensKit • PredictionIO • Wikipedia SuggestBot • Cyclopath/Cycloplan

CONSIDERATIONS FOR RECOMMENDER SYSTEMS

Human-Directed Systems

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

HUMAN-DIRECTED SYSTEMS

1. System provides capability for defining rules or conditions in the system

2. Includes

• Rules & Workflow

• Triggers

• Data Tagging/Categorization

PQ

R

S T&

|

EXAMPLES OF HUMAN-DIRECTED SYSTEMS

Tagging (RunScribe)

Triggers (AT&T M2X)

Rules & Workflow (NodeRed)

Learning Systems

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

LEARNING SYSTEMS

1. Can apply insights based on publicly-available information or history

2. Takes action without intervention

3. Informs a human after the fact, who can refine and adjust the parameters for the next decision

)

UV

'

!

$X

*

♥

Z

[

=

• This is the wild-west of insight

• There are few tools available today to support this, out of the box

• Start with your recommendation and visualization tools and build from there

• Learning Systems Require:

• Enough Data Volume to be meaningful

• A Facility for automated decision-making and refinement (incl. manual)

•Rules++

CONSIDERATIONS FOR LEARNING SYSTEMS

1) \ 2 3] ^

Building Decision-Making Systems

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

$ % & ' ( $ ) * + , !

% & ' ) $ % ! ' ( ' (

“REAL-TIME”

• Real-time means…

• Speed of entry

• Speed of insight

• Speed of action@

#

B

EH

J

KL

M

BUILDING “REAL-TIME” APPS - TIPS

1. Use tools and transports that make real-time simple

• Meteor, Modulus.io & MongoDB

2. Push insights outward

• Push Notifications

• Triggers to other systems

3. Build with rules and workflow in mind

• Rules defined in advance

• Workflow between systems

4. Use public data to make decisions simple (or automatic)

@

#

B

EH

J

KL

M

Create an Arduino-based weather station that:

• Monitors environmental data from on-board sensors • Temp, pressure, wind speed & direction, rainfall, soil moisture, etc.

• Powered by a small solar panel with integrated battery monitoring • Posts all data to a cloud-hosted source • Can respond to environmental conditions and external inputs

SCENARIO. GARDEN WEATHER STATION

_

`V

|

a

Ingestion

• Store environmental data as a ContentType in Telerik Backend Services

• Store Battery Data (Charge, Voltage) as a separate ContentType

Visualization

• Create a monitoring dashboard that shows summary views for environmental and battery data

• Built with • Node.js, Express and Modulus (hosting) • Meteor for real-time communication • Kendo UI for web dashboard widgets/UI • NativeScript and UI for NativeScript for

Mobile

GARDEN WEATHER STATION - INGESTION AND VISUALIZATION

_

`V

|

a

∠

∠

Recommendations & Triggers

• Send a notification when: • Soil moisture falls below a certain level • Rainfall is greater than a certain amount • The temperature is nearing freezing • Device battery drops below a threshold or isn’t charging

enough over time • Recommendations can be combined with triggers to suggest

action

Human-Directed (Rules)

• Instruct the device to enter “low-power” mode when battery level is low

• Set thresholds for soil moisture, rainfall and wind speed • Trigger sprinkler system when moisture-level is low • Delay sprinkler system when rainfall surpasses a threshold

GARDEN WEATHER STATION - RECOMMENDATIONS & RULES

_

`V

|

a

∠

∠

GARDEN WEATHER STATION - LEARNING SYSTEMS

_

`V

|

a

Learning Systems

• Set a watering schedule for the week/day based on public forecast data

• Combine history and rules to adjust moisture and rain delay thresholds

• Re-position the solar panel (w/ servo) to obtain optimal charge • Self-monitor sensors and hardware and send notifications

when future malfunctions are likely

∠

THE CONTINUUM OF DATA INSIGHT

INSIGHT

DIRECTIONHuman defines rules or conditions in the system in advance; System takes action when conditions are

met.

RECOMMENDATION

System makes suggestions based

on data & a human takes

action

VISUALIZATION

System provides visuals to help you

reason about data.

COLLECTION

System gathers data and stores it in some location

LEARNINGSystems that take action based on past behavior,

public information or other factors

Real-Time

SPEED OF DECISION-MAKING

SPEED OF DATA INGESTION

THANK YOU!

@BrandonSatrom

? Brandon.Satrom@Telerik.com

=

c

dbsatrom

e