From Business Intelligence to Big Data:The Evolution of Business Analytics

@hackreduce – Dec. 3, 2014

Adam Ferrari@AJFerrari

(All opinions expressed are my own / I’m not here representing my employers)

Adams-MacBook-Pro:~ ferrari$ whoami

ferrari

2012- 2014-

BA ‘91 MS ‘94 PhD CS ’98

2000-2012(CTO 2005-2012)

Endeca did this…

Which led us into this…

This talk

What did I learn as CTO of a BI product company as we jumped into the BI market mid-stream, and then later as we were acquired by one of the biggest “traditional BI” vendors?

Most Importantly:

Stay focused on real business value, not technology.

Note: My context is very “product provider” oriented, but I believe the lessons are equally interesting to “product consumers” – after all, we’re all interested in where the toolset is going and why

A note about scope

Analytics is a highly overloaded term

The vast majority of my experience, and the focus of this talk, is around “BI-style” analytics, i.e.,

Delivering historical and aggregate views of data (e.g., charts, reports, dashboards, etc.) to business decision makers

There are many other important forms of “analytics”

E.g., Data mining, statistics, data science, etc.

These are very important and complementary,but not in my scope here

Part 1 (of 3)Some Ancient History

(or, a bunch of important stuff that happened before my time)

In the beginning…

…there was the cube

(well, there was a bunch of stuff before that – Hans Peter Luhn coins the term Business Intelligence in 1958, Edgar Codd invents the relational data model in 1970, etc…but we’ll start with the beginning of modern Business Intelligence, which is OLAP)

Image source: oracle.com

Research sponsored by Arbor Software in 1993,defined the “12 Rules for OLAP Products”Rule #1 – “Multidimensional Conceptual View”

OLAP = Multidimensional Analysis

Notable “traditional” OLAP Products• Express

(IRI - Oracle)

• Essbase(Arbor - Hyperian - Oracle)

• Microsoft Analysis Services (Panorama - MS)

Image source: microsoft.com

[1995]

Notable “traditional”ROLAP Products• Microstrategy

• Business Objects(Business Objects - SAP)

• Cognos(Cognis - IBM)

• OBIEE(nQuire - Siebel - Oracle)

• Actuate, Birst, Pentahoetc…

Image source: microstrategy.com

ROLAP Modeling

• Manage mapping between physical data stores, “logical view” (core dimensional model), and “business view”

• Definition of metrics, dimensions

• Management of pre-computed aggregates

Image source: rittmanmead.com

Data Warehousing: go big or go home

HW

• Teradata

• Netezza (IBM)

• Oracle Exadata

SW – Traditional DBMS

• Oracle

• MS SQL Server

• IBM DB2

SW – Analytical DBMS

• Vertica (HP)

• ParAccell (/ RedShift)

• SAP HANA

Image source: teradata.com

ETL- Extract/Transform/Load

Image source: informatica.com

Notable ETL Products• Informatica Power Center• Ascential DataStage (IBM)• Ab Initio• … numerous others

• Capture History

• Manage dimensions– E.g., what happens if a

customer moves?“slow changing dimensions”

• Pre-compute aggregates

• Serve as the versionablemanaged record of how the dimensional model of the warehouse is derived from the raw data

Best Practices

[First Edition, 1992]

Image source: wiley.com

[Founded, 1995]

[First Edition, 1996]

“Business Analytics” 1.0 Architecture

Image source: ibm.com

Business Analytics 1.0 - Pros & Cons

• Governance, re-use, and quality– “One Version of the Truth” – correct, agreed upon, reusable definitions of core

business metrics and dimensions

But…

• Poor Agility – development process requires:– Creating or modifying a dimensional model

– Creating ETL to populate the new model

– Creating report or dashboard content on top of the model

– Iterating to make the model perform

• Lack of self-service for end users

• Historically, poor user experience for end consumers

• Cost and Complexity – large, complex stack of components, code, and configuration to manage, scale, troubleshoot, etc.

Part 2 (of 3)Some Recent History

(or, where I joined the story already in progress)

Data Discovery & VisualizationKey Features

• Visual data presentation

• Interactive data exploration –“facets,” “lassos,” etc.

• Simplified stack – DBMS and Server optional

• Self-service: data loading & content creation,no dimensional modeling

Notable products:

• QlikView (Qlik Tech)

• Tableau

• Spotfire (TIBCO)

• Endeca Latitude(now Oracle Information Discovery)

• EdgeSpring (now Salesforce.com Wave)

• Business Objects Explorer

Image source: tibco.com

Image source: sap.com

Source: http://www.tcsnycmarathon.org/analytics

Image source: community.qlik.com

QlikView configuration example…

Source: http://www.tableausoftware.com/learn/gallery

Image source: vizwiz.blogspot.com

Tableau configuration example…

Data Discovery Lessons• Improved User Experience, Self-serviceBut…• BI is still really hard

– Reading from raw, real-world operational schemas is messy and complicated

– And the requisite history may not even be available

• The usability benefits of discovery tools come with significant scalability limitations

• Additional data types – so called “unstructured” data (logs, text, etc.) is even harder, as discovery tools (generally) target structured, tabular data (didn’t address “Big Data”)

And…• Traditional BI tools are rapidly adding better UX, Visualization,

and Self-service

Part 3 (of 3) (woohoo!)

Future History(or, stuff that’s still anyone’s guess)

Our analytics ambitions have only grown!We want BIG, EASY, DEEP analytics

• [BIG] the headline grabber:More data from more sources, aka: Big Data

• [EASY] the real issue (IMHO):Faster time to value, at lower cost of ownership

• [DEEP] increasingly important:Deeper intelligence from data…not just data, but actions, predictions, etc…

… Can we solve these problems without creating an ever larger mess of technology and products?

[BIG]: the Hadoop Solution Posits that what we need is a better, more flexible and scalable foundation for the Data Warehouse – more like a “data operating system” than a DBMS

Image source: cloudera.com

[BIG] and [EASY] “On-Hadoop” Solutions

Image source: gigaom.com

Platfora Architecture

Posit that although Hadoop is indeed a powerful platform, it’s complexity needs to be wrapped in a BI / analytics application

Notable Products

• Platfora

• Datameer

• Oracle Big Data Discovery(based on Endeca)

[BIG+]: The Logical Data Warehouse*Posits that what is needed is a variety of data stores to constitute the “Data Warehouse,” along with integration to allow data to be stored and processed where most appropriate with little or no additional development effort or operational management overhead

Image source: teradata.com

* From Understanding the Logical Data Warehouse: The Emerging Practice, 21 June 2012, Mark A. Beyer and Roxane Edjlali

[EASY] The Cloud Solution

• Agility via all of the traditional cloud benefits –reduced setup, less customization, reduced ongoing management, etc…

• SaaS-based BI tools, such as– GoodData

– Domo

• SaaS-based BI applications, such as– Numerify (IT analytics on ServiceNow, etc.)

– InsightSquared (Sales analytics on Salesforce)

Other notable examples

• [DEEP] and [EASY]: BeyondCore – data discovery with automatic/algorithmic analysis of attribute relationships

• [DEEP]: Ayasdi – deeper insight into data based on novel topological data visualization

• [DEEP] Alteryx – democratizing more complex analytical workflows

• [EASY 2.0]: Looker – lightweight BI without sacrificing modeling, yet avoiding the need for a warehouse

• [BIG] and [EASY]: Tamr, Trifacta - curating and wrangling data into usable forms

My guesses about the future?

• I voted with my feet. My beliefs:– Fast time to real value is of paramount importance

• Zero-friction SaaS applications targeted to specific business problems are an essential enabler – essential to amortizing the cost of developing meaningful analytics and quickly disseminating best practice updates – DIY just doesn’t cut it any more in many cases.

– Our ability to do basic BI (dashboards, data discovery, etc.) is mature, and the real action is in deeper analysis of data• Yet highly custom data science efforts are at odds with

fast time to value, and hard to advance in many cases

Crisply – quantified work for CRMmodel & activity

activity

quantifiedwork

• Algorithmic quantification of the human effort behind each customer, opportunity, support case, etc.

• Determine the true cost to acquire a specific customer or type of customer, and understand the true profitability of that customer or segment over time

Thanks!

And stay focused onthe value that analytics creates

(the technology with follow from that)