Trivadis Azure Data Lake
-
Upload
trivadis -
Category
Technology
-
view
63 -
download
0
Transcript of Trivadis Azure Data Lake
The world is changing
Agenda
- New challenges and ways
- On-premises or cloud (or both?)
- The union of all
- Making sense of it
- How to get there...
Today, 80% of
organizations
adopt c loud-fi rst
strategies
AI investment
increased by
300% in 2017
Data wi l l grow to
44 ZB in 2020
Today, 80% of
organizations
adopt c loud-fi rst
strategies
AI investment
increased by
300% in 2017
Data wi l l grow to
44 ZB in 2020 C LO U D A IDATA
C LO U D
DATA A I
Organizations that harness data, cloud, and AI outperform
Rely on a modern data estate
Patrik Borosch TSP / DP
• 02.07.1971/married/Daughter(19)
• Music(Squared Circle/The Midcrise Liars)/climbing/skiing/cycling/Tango
• EDV-Kaufmann (german IHK 1995) = Data Processing with Cobol and Databases = the early BI guys ;)
• Reporting and data processing in controlling departments
• BI Consultant/Senior Consultant: ASTECH Solutions/T-Systems/Trivadis/Avanade
• Discipline Manager Microsoft BI and Power Pivot Trainer @ Trivadis
• Head of BI: Allianz Global Assistance
• TSP DP: Microsoft
• SQL Server/SSIS/SSAS/SSRS/MDM
• PowerBI/PowerPivot/PowerQuery
• Azure SQL DB/DW/Data Lake/Azure Stream Analytics/Data Factory/AAS
• SQL/DAX/MDX/(PowerShell)/(C#)
• Informatica/Microstrategy/Essbase/Enterprise Architect/UML/Perl/Unix/Linux had that... been there...
• First «Big Data»-Project in 2006: Teradata/Informatica/Microstrategy, 1.4TB = eight weeks for init load = lots of fun :D
8
9
Dr. John Snow (1854)
One of the first visual investigations of collected data helped to solve a cholera epidemic in Soho…
- You need the facts- BUT you also need to
make sense of it...
Therefore you need to have the right tools and methods...
The many sources and rapid growth of data requires a new approach
• Sentiment Analysis
• Social Media / Sales Connection
• Customer Segmentation
Data lake
From Wikipedia, the free encyclopedia
A data lake is a method of storing data within a system or repository, in its natural format,[1] that
facilitates the collocation of data in various schemata and structural forms, usually object blobs or files.
The idea of data lake is to have a single store of all data in the enterprise ranging from raw data (which
implies exact copy of source system data) to transformed data which is used for various tasks including
reporting, visualization, analytics and machine learning. The data lake includes structured data from
relational databases (rows and columns), semi-structured data (CSV, logs, XML, JSON), unstructured
data (emails, documents, PDFs) and even binary data (images, audio, video) thus creating a centralized
data store accommodating all forms of data.
James Dixon / Pentaho (2010)
Data Lakes
BUSINESS APPS
CUSTOM APPS
ANALYTICAL DASHBOARDSAZURE SQL DATA WAREHOUSE
AZURE CLI
AZURE DATA FACTORY
BCP COMMAND LINE UTILITY
SQL SERVER INTEGRATION SERVICES
AZURE ANALYSIS SERVICES
BUSINESS APPS
CUSTOM APPS
ANALYTICAL DASHBOARDS
DATA FACTORY
ANALYTICAL DASHBOARDS
Polybase
AZURE SQL DATA WAREHOUSE
DATA FACTORY
AZURE ANALYSIS SERVICES
AZURE MACHINE LEARNING
& MACHINE LEARNING SERVER
AZURE COSMOS DB
AZURE STORAGE
BUSINESS APPS
CUSTOM APPS
ANALYTICAL DASHBOARDS
DATA FACTORY
ANALYTICAL DASHBOARDS
Polybase
AZURE SQL DATA WAREHOUSE
DATA FACTORY
AZURE ANALYSIS SERVICES
AZURE MACHINE LEARNING
& MACHINE LEARNING SERVER
AZURE COSMOS DB
AZURE HDINSIGHT
(Hadoop)AZURE STORAGE
BUSINESS APPS
CUSTOM APPS
ANALYTICAL DASHBOARDS
DATA FACTORY
ANALYTICAL DASHBOARDS
Polybase
AZURE SQL DATA WAREHOUSE
DATA FACTORY
AZURE ANALYSIS SERVICES
AZURE MACHINE LEARNING
& MACHINE LEARNING SERVER
AZURE COSMOS DB
AZURE STORAGE AZURE DATABRICKS
(SPARK)
BUSINESS APPS
CUSTOM APPS
ANALYTICAL DASHBOARDS
DATA FACTORY
AZURE DATA LAKE STORE AZURE DATA LAKE ANALYTICS
ANALYTICAL DASHBOARDS
Polybase
AZURE SQL DATA WAREHOUSE
DATA FACTORY
AZURE ANALYSIS SERVICES
AZURE MACHINE LEARNING
& MACHINE LEARNING SERVER
AZURE COSMOS DB
CONTROL EASE OF USE
Azure Data Lake
Analytics
Azure Data Lake Store
Azure Storage
Any Hadoop technology,
any distribution
Workload optimized,
managed clusters
Data Engineering in a
Job-as-a-service model
Azure MarketplaceHDP | CDH | MapR
Azure Data Lake
Analytics
IaaS Clusters Managed Clusters Big Data as-a-service
Azure HDInsight
Frictionless & Optimized
Spark clusters
Azure Databricks
BIG
DA
TA
S
TO
RA
GE
BIG
DA
TA
A
NA
LYT
ICS
Red
uced
Ad
min
istr
ati
on
K N O W I N G T H E V A R I O U S B I G D A T A S O L U T I O N S
Drag & Drop
Azure ML
Big Data is driving transformative changes
Cost
Culture
Data
Characteristics
Traditional Big Data
Relational(with highly modeled schema)
All Data(with schema agility)
Expensive(storage and compute capacity)
Cloud(storage and compute capacity)
Rear-view reporting(using relational algebra)
Intelligent action(using relational algebra AND ML, graph, streaming, image processing)
Cognitive Services
• Faces, images, emotion recognition and video intelligence
• Spoken language processing, speaker recognition, custom speech recognition
• Natural language processing, sentiment and topics analysis, spelling errors
• Complex tasks processing, knowledge exploration, intelligent recommendations
• Bing engine capabilities for Web, Autosuggest, Image, Video and News
Intelligence
Cortana
Bot
Framework
Cognitive
Services
Microsoft BI, the agile way…
Azure
Analysis Services
Data Sources Ingest Prepare Analyze Publish Consume
Sensors and devices
StreamAnalytics
DiagnosticStreaming
Power BI
Sources- Oralce HFS- SAP BW- …
Azure Data Lake Store
Data Factory: Move data, orchestrate, schedule and monitor
Azure Data LakeIoT Hubs
MachineLearning
HDInsight
Data ScienceWorkbench
StreamAnalytics
Power BI Report Server
Architecture Blueprint
SSIS
SQL Server 2017: Security, Performance, Polybase, ML Services, Analytics
SQL Server2017
SSAS
BI Bot
Apps
Lab- and other Apps
Azu
re D
ata
Pla
tfo
rmSQ
L se
rver
20
17
AI built-in | Most secure | Lowest TCO
M I C R O S O F T F O R Y O U R M O D E R N D A T A E S T A T E
Data warehouses
Data lakes
Operational databases
Data warehouses
Data lakes
Operational databases
SQL Server Azure Data Services
Industry leader 4 years in a row
#1 TPC-H performance
T-SQL query over any data
70% faster
2x the global reach
99.9% SLA
HYBRID
Easiest lift and shift
with no code changes
SocialLOB Graph IoTImageCRM
Security and
performance
Flexibility
of choice
Reason over
any data, anywhere
Tools for your migration journey
SQL Server Migration Assistant (SSMA)
Automates database migration to SQL Server from
Microsoft Access, DB2, MySQL, Oracle, and SAP ASE.
Data Migration Assistant (DMA)
Enables upgrade to SQL Server and Azure SQL
Database.
Database Experimentation Assistant (DEA)
Assists in evaluating a targeted version of SQL for
a given workload.
Azure Hybrid Benefit for SQL Server
Maximizes current on-premises license investments
to facilitate migration.
Azure SQL Database Managed Instance
Facilitates lift and shift migration from
on-premises SQL Server to PaaS.
Azure Database Migration Service (Azure DMS)
SQL Server Migration Assistant (SSMA)
Data Migration Assistant (DMA)
Database Experimentation Assistant (DEA)
SQL Database
Managed Instance
Azure Hybrid
Benefit for
SQL Server
Empower today’s innovators to unleash the power of data
and reimagine possibilities that will improve our world