Adding Big Data to your Organization to create true 360-Degree Customer Insight

T : +44 (0) 1273 911 268 (UK) or (888) 631-1410 (USA) or +61 3 9596 7186 (Australia & New Zealand) or +91 997 256 7970 (India)

E : [email protected] W : www.rittmanmead.com

Adding Big Data to your Organization to create true 360-Degree Customer InsightMark Rittman, CTO, Rittman Mead November 2015





About the Speaker

•Mark Rittman, Co-Founder of Rittman Mead •Oracle ACE Director, specialising in Oracle BI&DW •14 Years Experience with Oracle Technology •Regular columnist for Oracle Magazine •Author of two Oracle Press Oracle BI books •Oracle Business Intelligence Developers Guide •Oracle Exalytics Revealed •Writer for Rittman Mead Blog :http://www.rittmanmead.com/blog

•Email : [email protected] •Twitter : @markrittman

http://www.rittmanmead.com/blog

mailto:[email protected]





About Rittman Mead

•Oracle BI and DW Gold partner •Winner of five UKOUG Partner of the Year awards in 2013 - including BI •World leading specialist partner for technical excellence, solutions delivery and innovation in Oracle BI

•Approximately 80 consultants worldwide •All expert in Oracle BI and DW •Offices in US (Atlanta), Europe, Australia and India •Skills in broad range of supporting Oracle tools: ‣OBIEE, OBIA, ODIEE ‣Big Data, Hadoop, NoSQL & Big Data Discovery ‣Essbase, Oracle OLAP ‣GoldenGate ‣Endeca



Everyone’s … Was Talking About Big Data

•Explosion in volume and variety of data that’s now available •New, cheap and open-source technology makes it economic to store + process it •New businesses are being built, andexisting ones disrupted, by rise of big data

• IT departments are using Hadoop + othertechnologies to complement, and replace,proprietary databases, storage etc

•But we’ve now moved beyond just talkingabout “build it and they will come” projects…



The Most Popular Big Data Applications Being…MoreCompleteDataAbout

YourCustomers

Advancedanalyticsandmachinelearning

MoreAttributesandActivitiesStoredatScale

True360°CustomerProfile

Connectdisparatedata

Targeted,personalizedcustomertreatment

Customer 360-Degree Analysis



MoreCompleteDataAboutYourCustomers

Advancedanalyticsandmachinelearning

MoreAttributesandActivitiesStoredatScale

True360°CustomerProfile

Connectdisparatedata

Targeted,personalizedcustomertreatment

… but what does it ACTUALLY mean?



Consider How Consumers Used to Be Marketed To…

• Marketing used to be generic, one-way “broadcasting” to public

• Then Web 2.0 gave customersa voice, they could talk back…

• But they expected an immediate answer • More work, but more intimate relationship

• Big data, smart technology + complex algorithmsmakes a “360-degree view of customers now possible

• Customers volunteer much data themselves • But equilibrium of relationship now moved

irrevocably to the customer



Consumers Now Drive Their Own Purchase Decisions



Customers Share Data with You .. With Expectations

•Customers now share huge amounts of data willingly, and perhaps unknowingly ‣Through your channels and applications - with potential privacy issue ‣Through tweeting, posting on Facebook and other social networks ‣But they also want to be in control

-Ability to delete their data -Understand what data you hold -For what purposes -And how it was collected



Datasets for Marketing Need to Reflect Today’s Consumer



Customer Touch-points Now Cover Many More Channels

•The days of a single, high-street retail channel are long gone •Prospects often now find you via web searches, social media connections •Shopping and browsing “on the go” using mobile devices, wearables

•Web increasingly the main sales channel •“Order and go” collection at stores •Call centre helplines, •Customer service desks •Forums, blogs, product reviewsand other user-generated content

1980s 1990s 2000s 2010s

EmpoweredEmployees

Digital isHumanized

KnowledgeEverywhere

InternetofThings

MobileasPrimaryChannel

Cross-ChannelService

WHAT’SNEXT



What is Customer 360-Degree Analysis?

•Gather together all meaningful information about the customer (“360-degree view”) •Organizing, matching, profiling & storing every interaction in real time

•Matched and combined; factual, interpreted, learned ‣Across all channels, and on public forums and social media

•Captures interactions across all-touch points and all channels ‣Including activity on social networks, forums, blogs

•Typically stored and processed in a Hadoop “data reservoir” •Dynamic customer profiles with segmentation, behavioural analysis “at scale”

•Downstream feeds into DW, CRM and other systems



But … Isn’t This Just CRM?

•Typically built for call centres, sales automation •Core data is customer service activity •Supplemented by purchase history •CRM system typically system of record forservice activity, with links to transactions ‣LoB application focused on particular tasks



•Detailed events and transactions now combined with granular behavioural & attitudinal data

Adding “Who” and “Why” to Customer & Transaction Data

SingleCustomerViewEnriched

CustomerProfile

Correlating

Modeling

Machine Learning

Scoring

“How” InteractionData

Voice+ChatTranscripts In-person

dialogs

Webserverlogs

Blogs

Surveys

SocialMedia

“Why”AttitudinalData

“What” BehaviouralData

Transaction History

Retail Activity

PaymentHistory

BasketAnalysis

Attributes

Segments

Relationships

“Who”DescriptiveData

Demographics



Requires Partnership Between IT and Business Functions

•Many Big Data projects start as IT initiatives or business “skunk works” that then need scale •Customer 360 projects require co-operation across the organization •Marketing, Customer Service & Sales provide

•Business need to create a single customer view ‣Need to reflect multi-channel nature of interactions ‣Increasing need for analysis + modeling at scale

• IT provides the scaleable long-term platform ‣Provisioning of platform & analytical tools ‣Governance, security, ‣Data integration and/or virtualization ‣Adding robust real-time data + sensor ingestion ‣Integration with rest of enterprise architecture



•Customer 360-Degree view typically used as central data store for digital marketing •Provides key data for real-time decision engines, next-best offer, personalisation

Customer 360-Degree View Powering Marketing + Offers

DataTransfer DataAccess

Real-TimeContext Environmental

UserJourney

OfferFeedback


CustomerProfile

Correlating

Modeling

Machine Learning

Scoring

RealTime

Offers&Suggestions

Up-Sell/Cross-Sell

DecisioningService

Self-LearningPredictiveModels




Customer 360-Degree View as Driver of Digital Marketing

?

?

?



Typically Stored on Flexible, Scalable Hadoop + NoSQL

Voice+ChatTranscripts

CallCenterLogsChatLogs iBeaconLogs WebsiteLogsCRMData Transactions SocialFeeds Demographics

Real-timeFeeds,batchandAPI

$50k

HadoopNode

$50k

HadoopNode

$50k

HadoopNodeHadoopNodeHadoopNode

$50k$50k

HadoopNode HadoopNode

$50k

EnrichedCustomerProfile

Modeling

Scoring HadoopDataReservoirRawcustomerdatastoredatdetail Enrichedandprocessedforinsights

$50k



But Wait … Isn’t This Just Data Warehousing & Data Mining?

•Data warehouses were conceived as a single source of reporting truth •Formally accept, model and integrate data to provide analytical reporting platform •Well-established design patterns for long-term data storage •Stored in structured, indexed, optimised “schema on write” storage •Data moved through layers via formal ETL •Extreme Performance, Highly Secure •Analytic SQL, In-Database Analytics ‣So why not use for this Customer 360 data?



DW Databases Can Hold Customer 360 Data - But Should They?

$1m

AnalyticDBMSNodeDBInstance

Compute

ETL

DataModel

ETL Developer

DataModeller

$1m

AnalyticDBMSNode

Compute

$1m

AnalyticDBMSNode

Compute

$1m

AnalyticDBMSNode

SingleDBInstance

Compute

DatawarehousedatabasescanscaleviaclusteringButlimitsonhowfarthiscango

Maximumsizeofclusteraround5-10nodesAndcost-eachnodetypicallycosts$1m

ButETLandloadingsimpleassingleDBinstance



Hadoop & Model-on-Write : Cheap, Flexible Storage

•Hadoop & NoSQL better suited to exploratory analysis of newly-arrived data ‣Flexible schema - applied by user rather than ETL ‣Cheap expandable storage for detail-level data ‣Better native support for machine-learning anddata discovery tools and processes

‣Potentially a great fit for our new and emergingcustomer 360 datasets, and great platform for analysis



Architected using “Data Reservoir” Design Pattern

•Data for customer 360 system typically landed into a Hadoop & NoSQL-based •Applies aggregation, joining and machine-learning processes to extract insights


DataFactory DataReservoir

BusinessIntelligenceTools

HadoopPlatform

FileBasedIntegration

StreamBased

Integration

Datastreams

Discovery&DevelopmentLabsSafe&secureDiscoveryandDevelopment

environment

Datasetsandsamples

Models andprograms

Marketing/SalesApplications

Models

MachineLearning

Segments

OperationalData

Transactions

CustomerMasterata

UnstructuredData


ETLBasedIntegration

RawCustomerData

Datastoredintheoriginal

format(usuallyfiles)suchasSS7,ASN.1,JSONetc.

MappedCustomerData

Datasetsproducedbymappingandtransformingrawdata



Interfaces with CRM Tools, and Accessible

‣CRM can be a consumer of data from the Hadoop-based Customer 360 ‣And provide key customer attributes and sales events from CRM activity ‣Allows CRM tools to focus on their core strengths ‣With ability to interface with the Customer 360 system as appropriate

DataReservoir

BusinessIntelligenceTools

CRMSystem

Models

Machine Learning

Segments

Raw CustomerData



MappedCustomerData





•Oracle Engineered system for big data processing and analysis •Start with Oracle Big Data Appliance Starter Rack - expand up to 18 nodes per rack •Cluster racks together for horizontal scale-out using enterprise-quality infrastructure

OracleBigDataApplianceStarterRack+Expansion

• ClouderaCDH+Oraclesoftware• 18High-specHadoopNodeswith

InfiniBandswitchesforinternalHadooptraffic,optimisedfornetworkthroughput

• 1CiscoManagementSwitch• SingleplaceforsupportforH/W+S/W

Deployed on Oracle Big Data Appliance Engineered System

OracleBigDataApplianceStarterRack+Expansion

• ClouderaCDH+Oraclesoftware• 18High-specHadoopNodeswith

InfiniBandswitchesforinternalHadooptraffic,optimisedfornetworkthroughput

• 1CiscoManagementSwitch• SingleplaceforsupportforH/W+S/W

EnrichedCustomerProfile

Modeling

Scoring

Infiniband



Oracle Big Data Discovery for Cataloging Customer 360 Data

•Access a rich, interactive catalog of all data in Customer 360 data reservoir

•Familiar search and guided navigation for ease of use

•See data set summaries, user annotation and recommendations

•Add personal and enterprise data to Customer 360 datasets via self-service ‣Make sense of the wider customerdata now loaded into data reservoir



The Data Integration Challenge

•Data from all the sources will need to be integrated to create the single customer view ‣Hadoop technologies (Flume, Kafka, Storm) can be used to ingest events, log data ‣Files can be loaded “as is” into the HDFS filesystem ‣Oracle/DB data can be bulk-loaded using Sqoop ‣GoldenGate for trickle-feeding transactional data

•But nature of new data sources brings challenges •May be semi-structured or unknown schema ‣Joining schema-free datasets

•Need to consider quality and resolve incorrect, incomplete, and inconsistent customer data


BatchLoadfromfiles,DB:

Easy

StreamfromAPIs,HTTP:Moderate

Loadrawtextfromfiles:

Easy

DataReservoir

Raw CustomerData



MappedCustomerData


Cleanse, enrichand

obfuscaterawfiles:

Lotsofwork..!

Joinstructured+semi/

unstructured:How..?



•Landing raw data is easy; then the real work needs to be done - can be > 90% of project •Four main tasks to take raw data and apply schema and combine together

1. Apply Schema to Raw and Semi-Structured Data 2. Remove Sensitive Data from Any Input Files 3. Identify joins, further enrichments and transforms 4. Store as “mapped” data in data reservoir

Ingesting Raw Customer Data : Two Key Challenges



•Data enrichment tool aimed at domain experts, not programmers •Uses machine-learning to automate data classification + profiling steps

•Automatically highlight sensitive data,and offer to redact or obfuscate

•Dramatically reduce the time requiredto onboard new data sources

•Hosted in Oracle Cloud for zero-install ‣File upload and download from browser ‣Automate for production data loads

RawData

Datastoredintheoriginalformat(usuallyfiles)suchasSS7,ASN.1,JSONetc.

MappedData

Datasetsproducedbymappingandtransforming

rawdata


Oracle Big Data Preparation Cloud Service



Step 1: Apply Schema to Raw and Semi-Structured Data

NLPEmbeddedInformationin

unstructuredtext Entities

EmbeddedInformationNoreliablepatterns

InvalidandmissingdataSensitivedata

Invalidemails

StreamfromAPIs,HTTP:Moderate

BatchLoadfromfiles,DB:

Easy

Loadrawtextfromblogentries,reviews



Step 2: Remove Sensitive Data from Any Input Files

•Automatically profile and analyse datasets •Use Machine Learning to spot and obfuscate sensitive data automatically



Step 3 : Identify Common Keys and Joins using BDD

•Data ingest process automatically applies some enrichments - geocoding etc •Can apply others from Transformation page - simple transformations & Groovy expressions



Step 4 : Process Joined and Enriched Data Back to Hadoop

•Data joined and transformed within Big Data Discovery can be saved back to Hadoop •Export to HDFS, register with Hive (optional) •Supports creation of

DataReservoir

Raw CustomerData

Datastoredintheoriginalformat(usuallyfiles)suchasSS7,ASN.

1,JSONetc.

MappedCustomerData

Datasetsproducedbymappingand

transformingrawdata

Cleanse, enrichand

obfuscaterawfiles:Lotsofwork..!

Joinstructured+semi/unstructured:

How..?



Solution Component Logical View

•Data Factory for file, event and DB transaction batch and streaming ingestion •Big Data Management Platform for combined Hadoop + RDBMS data storage •Discovery Labs for innovation and sandboxing




Customer 360-Degree View as Driver of Digital Marketing



UserJourney

OfferFeedback


CustomerProfile

Correlating

Modeling

Machine Learning

Scoring

RealTime

Offers&Suggestions

Up-Sell/Cross-Sell

DecisioningService




•Customer 360 systems can use machine learning across all data to build predictive models •Decision engines (for example, Oracle RTD) can use its data as source

•Wider range of attributes and faster velocity •Aim to process data and not just to store it

- Identify customers likely to defect, work proactively to retain

- foster enhanced engagement • increase revenue •pass events and contextual data to real-time decisioning engines

Use Machine Learning and Real-Time Decisions to add Value

RealTime

Offers&SuggestionsUp-Sell / Cross-Sell

Decisioning Service



UserJourney

OfferFeedback


CustomerProfile

Correlating

Modeling

Machine Learning

Scoring

OperationalData

Transactions

Customer MasterData

UnstructuredData


TouchPointsStore

Web

Service



•Real-time decisioning engine from Oracle, part of Oracle BI product family •Predictive and Personalised Real-Time Recommendations ‣Behaviour-based models that take intoaccount activity across all channels ‣At very low levels of granularity ‣Micro-segmentation to individual customer ‣Multi-contextual recommendations based on predicted customer needs

•Real-time offers, delivered to any channel •Feedback loop to improve recommendation

Oracle Real-Time Decisions



Event-Driven Personalised Marketing & Offers

•Vastly increased size, time-relevance and scope of customer data into decision models

TouchPointsStore

Web

Service

OperationalData

Events&Workflow

Transactions

Customer MasterData

Profile-Based(Demographic)

Prioritization

Offers&Suggestions

GuidedSearch GuidedSearch

Pricing Marketing

Engagement

RealTime

Offers&SuggestionsUp-Sell / Cross-Sell

Decisioning Service



CustomerProfile

Correlating

Modeling

Machine Learning

Scoring


UserJourney

OfferFeedback

Relevant PersonalizedExperiences

•Then close the loop with real-time context back into decision engine



Connect the Silos, Understand Customers, Drive Decisions

execute smarterlisten better

consumption logs, clickstream & devices

demographic, user and credit data

customer contacts and service cases

transactions and subscriptions

content metadata, ratings, comments

marketing campaign response

social mediaactivity

programmatic advertising

audience acquisition, retention

multi-channelmarketing

targeted promotions

next bestoffer

personalized content

product & service strategy

content acquisition


CustomerProfile

Correlating

Modeling

Machine Learning

Scoring



•Start with pilot for area of the business that needs a single view of customers •Then, over time, iterate and build out the Customer 360-degree view

Delivering a Successful Customer 360-Degree View

Startwithabusinessareathat

needsasinglecustomerview

Obtainclearunderstandingof

customeronline&offlinebehaviour

Buildout PredictiveModels

andDecisionEnginestodelivervaluenow

BuildoutHadoopDataReservoir,Feeds

andlinktoDW+CRM

IterateandBuild-out,addnewintegrations,incrementallybuilding

capability

DevelopandImplementStrategy,DeliverBusinessValue

BuildDevOpsCapability

Pilot&QuickWin

CreateFullProductionInfrastructurePilot(Virtualised/Commodity)HadoopInfrastructure



Packaged Customer 360 Applications

•Packaged Customer 360 applications seem a good way to start…? •Risks around degree of fit and integration limits, but can be useful for mature projects

Seeeverythingtogether–comparisonswithaSetdefinedbyyou,andevolvingtrendscoresforeachcustomer

FromDatatoDNA– 1000sofmetricsdetermineindividualDNA–common,industryandcustomermetrics



Rittman Mead Customer 360 + Real-Time Decisions Projects

Startwithabusinessareathat

needsasinglecustomerview

Obtainclearunderstandingofcustomeronline&offlinebehaviour

Buildout PredictiveModels

andDecisionEnginestodelivervaluenow

BuildoutHadoopDataReservoir,Feeds

andlinktoDW+CRM

IterateandBuild-out,addnewintegrations,incrementallybuilding

capability





Thank You for Attending!

•Thank you for attending this presentation, and more information can be found at http://www.rittmanmead.com

•Contact us at [email protected] or [email protected] •Look out for our book, “Oracle Business Intelligence Developers Guide” out now! •Follow-us on Twitter (@rittmanmead) or Facebook (facebook.com/rittmanmead)

http://www.rittmanmead.com





Adding Big Data to your Organization to create true 360-Degree Customer InsightMark Rittman, CTO, Rittman Mead November 2015

Adding Big Data to your Organization to create true 360-Degree Customer Insight

Data & Analytics

Transcript of Adding Big Data to your Organization to create true 360-Degree Customer Insight