PLOTCON NYC: Building a Flexible Analytics Stack

Benn Stancil November 16, 2016

1

Building a flexible analytics stack

2

Benn Stancil, Analyst

5

Drive decisions

10

It needs a backstory

13

Editors in an Organizations Each Month

Sales

month/month

Lost accountsYear over year

125 8 $20,187,921.56

Gross revenueNew accountsmonth/month

1.75% 0.5% 0.5%

Accounts by State

Search…Sales

Editors in an Organizations Each Month

Sales

Personal

SPACES

General

Home

Financial Reports

Sept. Board Meeting

Marketing

Retention Reports

Churn analysis

14

Analytics applications

15

Business application

Website

Central warehouse

Third party applications


Event logging

Transformation and aggregation

E(T?)L

Communication and information platforms


E(T?)L

16

How this system gets built (and what we learned building it)

17

How this system gets built (and what we learned building it)

18


19


• Web application • Mobile app • Inventory management system • Point of sale system • Sales records in Excel

20



• Plotly • Mode • (Others)

21

1. You can start small

THINGS WE LEARNED

22

Still a cowboy

23



24


Website


25


Website



• Google Analytics • Mixpanel • Chartbeat • Kissmetrics • Amplitude

29


Website



30


Website

Central warehouse


Event logging


31

Website

Central warehouse

Event logging


• PostgreSQL • MySQL • Oracle • MS SQL Server • Redshift • BigQuery • Hive • Spark

32

Website

Central warehouse

Event logging


• PostgreSQL • MySQL • Oracle • MS SQL Server • Redshift • BigQuery • Hive • Spark

33

Website

Central warehouse

Event logging


• Google Analytics • Amplitude • Heap • Segment • Snowplow • Custom built

34


Website

Central warehouse


Event logging


35


Website

Central warehouse


Event logging

E(T?)L


36


Central warehouse

E(T?)L• Fivetran • Segment • ETLeap • Stitch Data • Data Virtuality • Xplenty • Alooma • Blendo • AWS Data Pipeline • Custom built

37

2. One step at a time

THINGS WE LEARNED

38


Website

Central warehouse


Event logging

E(T?)L


39


Website

Central warehouse


Event logging

E(T?)L


• Salesforce • Zendesk • Stripe • Mailchimp • Twilio • JIRA • Hubspot

• Marketo • NetSuite • Square • Github • Adwords • FB Ads . . . .

40


Website

Central warehouse



Event logging

E(T?)L


E(T?)L

41



E(T?)L

E(T?)L• Fivetran • Segment • ETLeap • Stitch Data • Data Virtuality • Xplenty • Alooma • Blendo • AWS Data Pipeline • Custom built

42

3. Evaluate data productsby their data

THINGS WE LEARNED

43


Website

Central warehouse



Event logging

E(T?)L


E(T?)L

44


Website

Central warehouse



Event logging


E(T?)L


E(T?)L

45


• Informatica • Pentaho • Alooma • Keboola • Airflow • Custom built • Analytics applications

46


Website

Central warehouse



Event logging


E(T?)L


E(T?)L

50


Website

Central warehouse



Event logging


E(T?)L


E(T?)L

51


• Salesforce • Zendesk • Mailchimp • JIRA • Hubspot • Marketo • NetSuite • Adwords • FB Ads . . . .

• Email • Slack • Internal wikis • Customer sites

52


• Salesforce • Zendesk • Mailchimp • JIRA • Hubspot • Marketo • NetSuite • Adwords • FB Ads . . . .

• Email • Slack • Internal wikis • Customer sites

• Analytics applications • Custom built • Processes

53

4. Be unavoidable

THINGS WE LEARNED

54


Website

Central warehouse



Event logging


E(T?)L



E(T?)L

55


WebsiteThird party applications



56

4. Start by asking questions

THINGS WE LEARNED

57


Website

Central warehouse



Event logging


E(T?)L



E(T?)L

58

Website

Website



ETL

Event logging






ETL

60

Mature data infrastructure

New data infrastructure

Architect Optimize

61

Smaller team

Larger team

componentsWork on

the systemWork on

62

componentsArchitect

componentsOptimize

the systemOptimize

the systemArchitect

Mature data infrastructure

New data infrastructure

Smaller team

Larger team

63

The best plots don’t have holes

PLOTCON NYC: Building a Flexible Analytics Stack

Data & Analytics

Transcript of PLOTCON NYC: Building a Flexible Analytics Stack