Intro to Data Science Big Data

55
Introduction To BIG DATA & DATA SCIENCE

Transcript of Intro to Data Science Big Data

Page 1: Intro to Data Science Big Data

Introduction To BIG DATA &

DATA SCIENCE

Page 2: Intro to Data Science Big Data
Page 3: Intro to Data Science Big Data
Page 4: Intro to Data Science Big Data

See It & Tell Me What You Have Seen....

Page 5: Intro to Data Science Big Data

What’s Big Data?

No single definition; here is from Wikipedia:

• Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on-hand database management tools or traditional data processing applications.

• The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization.

• The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.”

5

Page 6: Intro to Data Science Big Data

Big Data: 3V’s

6

Page 7: Intro to Data Science Big Data

Volume (Scale)

• Data Volume– 44x increase from 2009 2020– From 0.8 zettabytes to 35zb

• Data volume is increasing exponentially

7

Exponential increase in collected/generated data

Page 8: Intro to Data Science Big Data

12+ TBs of tweet data

every day

25+ TBs oflog data

every day

? TB

s of

data

eve

ry d

ay

2+ billion

people on the Web

by end 2011

30 billion RFID tags today

(1.3B in 2005)

4.6 billion camera phones

world wide

100s of millions

of GPS enabled

devices sold annually

76 million smart meters in 2009…

200M by 2014

Page 9: Intro to Data Science Big Data

Data Analytics Challenge

Page 10: Intro to Data Science Big Data

Variety (Complexity)

• Relational Data (Tables/Transaction/Legacy Data)• Text Data (Web)• Semi-structured Data (XML) • Graph Data

– Social Network, Semantic Web (RDF), …

• Streaming Data – You can only scan the data once

• A single application can be generating/collecting many types of data

• Big Public Data (online, weather, finance, etc)

10

To extract knowledge all these types of data need to linked together

Page 11: Intro to Data Science Big Data

A Single View to the Customer

Customer

Social Media

Gaming

Entertain

BankingFinance

OurKnownHistory

Purchase

Page 12: Intro to Data Science Big Data

Velocity (Speed)

• Data is begin generated fast and need to be processed fast

• Online Data Analytics• Late decisions missing opportunities• Examples

– E-Promotions: Based on your current location, your purchase history, what you like send promotions right now for store next to you

– Healthcare monitoring: sensors monitoring your activities and body any abnormal measurements require immediate reaction

12

Page 13: Intro to Data Science Big Data

Real-time/Fast Data

Social media and networks(all of us are generating data)

Scientific instruments(collecting all sorts of data)

Mobile devices (tracking all objects all the time)

Sensor technology and networks(measuring all kinds of data)

• The progress and innovation is no longer hindered by the ability to collect data• But, by the ability to manage, analyze, summarize, visualize, and discover

knowledge from the collected data in a timely manner and in a scalable fashion

13

Page 14: Intro to Data Science Big Data

Real-Time Analytics/Decision Requirement

Customer

InfluenceBehavior

Product Recommendations that are Relevant

& Compelling

Friend Invitations to join a

Game or Activitythat expands

business

Preventing Fraud as it is Occurring

& preventing moreproactively

Learning why Customers Switch to competitors

and their offers; in time to Counter

Improving theMarketing

Effectiveness of a Promotion while it

is still in Play

Page 15: Intro to Data Science Big Data

Some Make it 4V’s

15

Page 16: Intro to Data Science Big Data

Harnessing Big Data

• OLTP: Online Transaction Processing (DBMSs)• OLAP: Online Analytical Processing (Data Warehousing)• RTAP: Real-Time Analytics Processing (Big Data Architecture & technology)

16

Page 17: Intro to Data Science Big Data

The Model Has Changed…

• The Model of Generating/Consuming Data has Changed

Old Model: Few companies are generating data, all others are consuming data

New Model: all of us are generating data, and all of us are consuming data

17

Page 18: Intro to Data Science Big Data

What’s driving Big Data

- Ad-hoc querying and reporting- Data mining techniques- Structured data, typical sources- Small to mid-size datasets

- Optimizations and predictive analytics- Complex statistical analysis- All types of data, and many sources- Very large datasets- More of a real-time

18

Page 19: Intro to Data Science Big Data

Big Data: Batch Processing &

Distributed Data StoreHadoop/Spark; HBase/Cassandra

BI ReportingOLAP &

Dataware house

Business Objects, SAS, Informatica, Cognos other SQL

Reporting Tools

Interactive Business Intelligence &

In-memory RDBMS

QliqView, Tableau, HANA

Big Data:Real Time &Single View

Graph Databases

THE EVOLUTION OF BUSINESS INTELLIGENCE

1990’s 2000’s 2010’s

Speed

Scale

Scale

Speed

Page 20: Intro to Data Science Big Data

Big Data Analytics

• Big data is more real-time in nature than traditional DW applications

• Traditional DW architectures (e.g. Exadata, Teradata) are not well-suited for big data apps

• Shared nothing, massively parallel processing, scale out architectures are well-suited for big data apps

20

Page 21: Intro to Data Science Big Data
Page 22: Intro to Data Science Big Data

Big Data Technology

22

Page 23: Intro to Data Science Big Data
Page 24: Intro to Data Science Big Data
Page 25: Intro to Data Science Big Data
Page 26: Intro to Data Science Big Data
Page 27: Intro to Data Science Big Data

The 5 Questions DATA SCIENCE Can Answer....

• 1. Is This A or B?• 2. Is This Weird?• 3. How Much - OR - How Many?• 4. How is this Organized?• 5. What should I do next?

• That's It - there are only 5!

Page 28: Intro to Data Science Big Data

How Does DATA SCIENCE Works?

• Algorithm = Recipe• Your Data = Ingredients• Computer = Blender• Your Answer = Strawberry Smoothie

Page 29: Intro to Data Science Big Data

1. Is This A or B ? - Classification Algorithms2. Is This Weird ? - Anomaly Detection

Algorithms3. How Much? or How Many? - Regression

Algorithms4. How is this Organized? - Clustering

Algorithms5. What Should I do Now? - Reinforcement

Learning Algorithms

Page 30: Intro to Data Science Big Data

What is Datascience?

Data Science Is Nothing but Extracting “Meaningful” and “Actionable” Knowledge

From DATA• “A Science or practice of collecting and analyzing

numerical data in large quantities especially for the purpose of inferring proportions in a whole from those in a representative sample”

• In simple terms using DATA to solve problems or answer questions.

• Difference between Datascience, BI and Reporting

Page 31: Intro to Data Science Big Data

Why Datascience?

• Data collection is increasing– Explosion of data from many sources

• Online: Social media, web ready devices, digital content• Enterprise: Transactional data, operational data, R & D data

• By end of 2020 , 1.7 MB of new information will be created for each and every human being on earth

• Data storage is increasing– Costs of storage devices going down– More machine learning algorithms to work on the data– Technology is growing

• Many kinds of data: Structured, unstructured etc• BIGDATA• Data is used not just for Descriptive but for Predictive and Prescriptive

Analytics

Page 32: Intro to Data Science Big Data

Data Science Competence

Building Data Science Profession

Data Science Competence includes 5 areas/groups• Data Analytics• Data Science Engineering• Domain Expertise• Data Management• Scientific Methods (or Business Process

Management)

Scientific Methods• Design Experiment• Collect Data• Analyse Data• Identify Patterns• Hypothesise Explanation• Test Hypothesis

Business Operations• Operations Strategy• Plan• Design & Deploy• Monitor & Control• Improve & Re-design

Page 33: Intro to Data Science Big Data

Data Science Competence includes 5 areas/groups• Data Analytics• Data Science Engineering• Domain Expertise• Data Management• Scientific Methods (or Business Process

Management)

Scientific Methods• Design Experiment• Collect Data• Analyse Data• Identify Patterns• Hypothesise Explanation• Test Hypothesis

Business Process Operations/Stages• Design• Model/Plan• Deploy & Execute• Monitor & Control• Optimise & Re-design

Data Science Competences Groups – Business

Design

Modelling

ExecutionMonitoring

Optimisation RESEARCHDATA

ANALYTICS

ALGORITHMSANALYTICSYSTEMS

ENGINEERINGCOMPETENCES

DOMAINEXPERTISE DATA

SCIENCE

Data Management

Scientific Methods

Business ProcessManagement

Building Data Science Profession

Page 34: Intro to Data Science Big Data

Identified Data Science Skills/Experience Groups

• Group 1: Skills/experience related to competences– Data Analytics and Machine Learning– Data Management/Curation (including both general data management and scientific data management) – Data Science Engineering (hardware and software) skills – Scientific/Research Methods– Application/subject domain related (research or business)– Mathematics and Statistics

• Group 2: Big Data (Data Science) tools and platforms– Big Data Analytics platforms – Math & Stats apps & tools – Databases (SQL and NoSQL)– Data Management and Curation platform– Data and applications visualisation– Cloud based platforms and tools

• Group 3: Programming and programming languages and IDE– General and specialized development platforms for data analysis and statistics

• Group 4: Soft skills or Social Intelligence– Personal, inter-personal communication, team work (also called social intelligence or soft skills)

Big Data Tools and Programming Languages• Big Data Analytics platforms• Math& Stats tools • Databases• Data/applications visualization• Data Management and Curation

Building Data Science Profession

Page 35: Intro to Data Science Big Data

Identified Data Science Skills/Experience Groups

• Group 1: Skills/experience related to competences– Data Analytics and Machine Learning– Data Management/Curation (including both general data management and scientific data management) – Data Science Engineering (hardware and software) skills – Scientific/Research Methods– Application/subject domain related (research or business)– Mathematics and Statistics

• Group 2: Big Data (Data Science) tools and platforms– Big Data Analytics platforms – Math & Stats apps & tools – Databases (SQL and NoSQL)– Data Management and Curation platform– Data and applications visualisation– Cloud based platforms and tools

• Group 3: Programming and programming languages and IDE– General and specialized development platforms for data analysis and statistics

• Group 4: Soft skills or Social Intelligence– Personal, inter-personal communication, team work (also called social intelligence or soft skills)

Big Data Tools and Programming Languages• Big Data Analytics platforms• Math& Stats tools • Databases• Data/applications visualization• Data Management and Curation

Building Data Science Profession

Page 36: Intro to Data Science Big Data
Page 37: Intro to Data Science Big Data
Page 38: Intro to Data Science Big Data
Page 39: Intro to Data Science Big Data
Page 40: Intro to Data Science Big Data
Page 41: Intro to Data Science Big Data
Page 42: Intro to Data Science Big Data
Page 43: Intro to Data Science Big Data
Page 44: Intro to Data Science Big Data

Cloud Computing

• IT resources provided as a service– Compute, storage, databases, queues

• Clouds leverage economies of scale of commodity hardware– Cheap storage, high bandwidth networks &

multicore processors – Geographically distributed data centers

• Offerings from Microsoft, Amazon, Google, …

Page 45: Intro to Data Science Big Data

wikipedia:Cloud Computing

Page 46: Intro to Data Science Big Data

Benefits

• Cost & management– Economies of scale, “out-sourced” resource management

• Reduced Time to deployment– Ease of assembly, works “out of the box”

• Scaling– On demand provisioning, co-locate data and compute

• Reliability– Massive, redundant, shared resources

• Sustainability– Hardware not owned

Page 47: Intro to Data Science Big Data

Types of Cloud Computing• Public Cloud: Computing infrastructure is hosted at the vendor’s

premises. • Private Cloud: Computing architecture is dedicated to the

customer and is not shared with other organisations. • Hybrid Cloud: Organisations host some critical, secure

applications in private clouds. The not so critical applications are hosted in the public cloud– Cloud bursting: the organisation uses its own infrastructure for normal

usage, but cloud is used for peak loads.• Community Cloud

Page 48: Intro to Data Science Big Data

Classification of Cloud Computing based on Service Provided

• Infrastructure as a service (IaaS) – Offering hardware related services using the principles of cloud computing. These

could include storage services (database or disk storage) or virtual servers. – Amazon EC2, Amazon S3, Rackspace Cloud Servers and Flexiscale.

• Platform as a Service (PaaS) – Offering a development platform on the cloud. – Google’s Application Engine, Microsofts Azure, Salesforce.com’s force.com .

• Software as a service (SaaS) – Including a complete software offering on the cloud. Users can access a

software application hosted by the cloud vendor on pay-per-use basis. This is a well-established sector.

– Salesforce.coms’ offering in the online Customer Relationship Management (CRM) space, Googles gmail and Microsofts hotmail, Google docs.

Page 49: Intro to Data Science Big Data

Infrastructure as a Service (IaaS)

Page 50: Intro to Data Science Big Data

More Refined Categorization• Storage-as-a-service• Database-as-a-service• Information-as-a-service• Process-as-a-service• Application-as-a-service• Platform-as-a-service• Integration-as-a-service• Security-as-a-service• Management/ Governance-as-a-service• Testing-as-a-service• Infrastructure-as-a-service

InfoWorld Cloud Computing Deep Dive

Page 51: Intro to Data Science Big Data
Page 52: Intro to Data Science Big Data
Page 53: Intro to Data Science Big Data
Page 54: Intro to Data Science Big Data

Thank You ! & Have a nice Day Ahead….

Page 55: Intro to Data Science Big Data

Thank You !

We hope you enjoyed this Demo / Beginner’s Session (and learned a little something along the way)

Please Contact Sales Team…..

For Further Details.

Have a Nice Day Ahead!....