Post on 08-May-2015
description
Why Big Data is Really
About Small Data: The Big Data Paradox
Judith Hurwitz
President & CEO, Hurwitz & Associates
Agenda
§ What is so big about Big Data? § What is a data scientist § Data at rest, data in motion § Is Big Analytics more important? § Rethinking data modeling in a big data world § A couple of examples § What you should think about § Questions?
Meet the Speaker
§ Judith Hurwitz § President and CEO of Hurwitz & Associates, Inc., a strategy consulting and research firm
focused on distributed computing technologies. A pioneer in anticipating technology innovation and adoption, Judith advocates for a pragmatic adoption of an architectural and business approach to the emerging market for cloud computing, service orientation, and service management. She has served as a trusted advisor to many industry leaders over the years. Judith has helped these companies make the transition to a new business model focused on the business value of emerging platforms. Judith is an accomplished author and most recently co-author of Big Data for Dummies.
4
Our Team’s Latest Book
What is so big about big data?
§ Definition of Big Data § Volume – How much data § Variety – Various types of data (structured, unstructured) § Velocity – Speed that data moves from one location to another § Veracity – Accuracy (Do the results of a big data analysis make
sense?)
§ Big Data is not new § So, why now?
§ Impacting the way you collect, store, manage, analyze, and visualize data
What is the Purpose of Big Data?
§ Gather, store, manage, and manipulate vast amounts of data at the right speed, at the right time to get the right results
§ Gather enough data so that you can find patterns
§ Put those patterns to work to gain insights in context
6
Examples of Big Data
§ Analyze multiple data sources to detect and protect against insider trading, money laundering, credit card theft
§ Monitoring market feeds § Managing risk models § Log files § Spatial data from sensors § Medical device data – data from sensors connected to
medical equipment § GPS data § Unstructured data in emails, text messages, call center
notes 7
Why do we need to think about Big Data?
§ What big data means to business
§ More data for better decision making
§ Integration of data across business units and silos
§ Detecting risks in real time
§ Focus on putting information in context with supporting business decisions
§ Improving the customer experience by leveraging customer feedback from many different sources
8
From Big to Small
• Big data is only the first step in the journey
• Big data requires that you reduce the amount of data to a subset so that your organization can take a deeper look
• Once this subset of data is cleansed and verified, it can help analyze, predict, and prepare to address the future
9
The Role of a Data Scientist?
§ Combining computing science, math, statistics, and business (domain) knowledge
§ Looking for answers when you don’t know the question you want to ask
§ Asking new types of questions: finding nuggets of actionable information in huge volumes of data
§ Making analytics consumable: real-time analysis to help the business take the right action at the right time
§ Predictive analytics: What is the next best action?
10
Representation Technology Stack
11
Big$Data$Tech$Stack$
Redundant$Physical$Infrastructure$Security$Infrastructure$
“Organizing”$Databases$and$Tools$
Analy@cal$Data$Warehouses$and$Data$Marts$
Interfaces$and
$feed
s$from/to$the$Internet$
Interfaces$and$feeds$from/to$internal$applica@ons$
Big$Data$Applica@ons$
Repor@ng$&$Visualiza@on$Analy@cs$(Tradi@onal$and$Advanced)$
Opera@onal$Databases$(Structured,$Unstructured,$SemiMstructured)$
12
Where Most of This Began
TransactionalSystem
(Production Data)
Data WarehouseDataMart
Then It Got “Better”
13
TransactionalSystem
(Production Data)
Data WarehouseDataMart
TransactionalSystem
(Production Data)
Data WarehouseDataMart
Then It Got “More Better”
14
Operational System
Operational System
TransactionalSystem(s)
Data Warehouse
LOBDataMart
LOBDataMart
LOBDataMart
And Better Still
15
Operational System
StagingArea
Operational System
TransactionalSystem(s)
Data Warehouse
LOBDataMart
LOBDataMart
LOBDataMart
Oops. Data at rest vs. data in motion
16
Operational System
StagingArea
Operational System
TransactionalSystem(s)
????
Data At Rest, Data In Motion
§ Data in motion is no longer a bad thing § Trend is combining “traditional” with
streaming § Instant analysis isn’t fast enough
§ It’s all about real-time
§ What data to keep?
Is Big Analytics More Important?
§ In a word YES § We are looking for answers to questions we haven’t
asked yet § Patterns, patterns, patterns § But…
§ Current generation analytics engines can be overwhelmed § Results may be too difficult to understand even with visualization § You may be looking in the wrong place or at the wrong things
Is Hadoop the New EDW?
§ No one type of Big Data platform is optimal for all requirements
§ Hadoop is changing the economics of storing and analyzing large volumes and variety of data
§ Results of Hadoop analytics needs to be understood in context
§ Increasing importance of hybrid big data architectures – combine Hadoop with your systems of record
§ Hadoop for specific roles § Exploratory data-science sandboxes § Staging platform for unstructured data
19
Rethinking Data Modeling
§ Traditional data models assume: § Relational data § Clean data § A few clearly identifiable data sources
§ Next generation data model – the rules have changed § Some relational data, some NoSQL § Some of the data is dirty § Lots of data sources coming from many different places § Some of the data you will keep and some you will not
§ Design your data model to account for new world of large and varied data sources
20
Big Data Use Cases
§ “Voice of the Customer”, 360-degree view of customer § Strengthen brand and increase customer loyalty § Improve operational analytics § Target and reduce fraud and improve security § Use sensors to provide real-time information about rivers
and oceans to predict impact of environmental changes
21
Correlating Varied Data Sources in Finance
§ Financial services is highly competitive and highly regulated. Financial services needs to create innovative customer experience while protecting IP. Companies need to anticipate the next best action.
§ What type of data is needed? § Transaction data § Threat data § Log data § Customer survey data § Customer support data § Customer social media data § Partner data § News and event data, ……
§ Need to be able to correlate all types of structured and unstructured data to predict the future and provide opportunities for growth and expansion
22
Advanced Security Analytics to Predict and Protect
§ Government agency needed more visibility into all system traffic
§ Concern about the unknown – needed to look for and protect from malicious activity
§ Used advanced security analytics to correlate data across seemingly unrelated events
§ Real-time § Analyze variety data sources- emails, documents, social
media data, business process data, DNS transactions § Analyze massive amounts structured and unstructured
data
23
Matching Capabilities to Business Problems
§ Text Analytics § Next Best Action § Data in Motion § Adding business
process and rules § Anamoly Detection § Data Visualization
§ Correlation between customer service, comments in the market, customer management
§ Putting a lot of data types together to determine best actions
§ Detecting Fraud 24
How Do You Manage Big Data?
§ Big data is not clean – it is massive and much is unstructured
§ Resulting patterns from big data analytics needs to be culled, cleaned and matched to enterprise data
§ Culled data now must be analyzed in context with your systems of record
§ Apply data visualization and best practices to determine how to apply data to actions
25
You need to think about the following:
§ Where are the sources of the data that could be important?
§ How often do you need access to particular types of data?
§ How long and how much data do you need to keep?
§ Can you trust the data and its sources?
§ Use Big Data analytics to overcome conventional wisdom and conventional thinking.
§ If you already know the questions to ask you aren’t moving forward.
26
Q & A
§ Thank you!
§ Contact info: § Judith Hurwitz: judith.hurwitz@hurwitz.com
27