1

In-Memory Technologie

Adastra

20.4.2016

Josef Pinkr, Consultant

Josef.Pinkr@adastragrp.com

cz.linkedin.com/in/josefpinkr

In-memory technologies

3

Excel + Power BI add-insQuery, Pivot, View, Map

SharePointPower Pivot Gallery, Power View

ExcelData Mining

Power BI Desktop Power BI Portal

Azure ML

Power BI Mobile App

Analytics Platform System(APS)

Faster transactions, Faster Queries, Faster Insights

In-Memory Technologies

4

Up to 30x faster transactions

In-memory objects

Row-store data format

Non/Durable data

Max. table size 2 TB

Many other improvements

Up to 10x query performance

Up to 10x data compression

Column-based data storage

• xVelocity (analytics engine)

Nonclustered columnstore index

• Updatable

• Filtered

Many other improvements

Billions of rows per second

with PowerPivot in Excel

Greater performance with

In-Memory Analysis Services

F A S T E R

Business scenarios

5

• ODS / OLTP systems

• No reporting / No analysis

• Operational reporting

• OLTP reporting

• Static reporting

• Management reporting

• Data analysis

• BI / Ad-hoc reporting

6

In-Memory OLTP

In-Memory OLTP

7

• Up to 2 TB (from 256 GB)

• Support for LOB types

• Fully transactional

• ALTER TABLE

• Non/Durable

• Accessed Using T-SQL

• 1 – 8 indexes

• Columnstore index

• Foreign Key

• Check/Unique Constraints

• All collations

• Operate against memory-optimized tables

• Is compiled when it is created

• Support for LOB types

• OUTPUT clauses

• UNION / UNION ALL

• SELECT DISTINCT

• OUTER JOIN

• Subqueries in SELECT(EXISTS, IN)

• Must be schema bound

• Natively compiled storedprocedures

• Natively compiled scalaruser-defined functions

• Traditional T-SQL modules

Tables Stored procedures User-defined functions Triggers

• AFTER trigger

• On memory-optimized tables

Performance Improvement of In-memory OLTP

Application Layer

SQL Server

OLTP system

Presentation Layer

INSERT SELECT

Application Layer

SQL Server

OLTP system

Presentation Layer

INSERT SELECT

Application Layer

SQL Server

OLTP system

Presentation Layer

INSERT SELECT

Memory-optimized tableMemory-optimized table

Natively compiled SP

2x 11x

Test: 4 Workloads \ 10 Threads \ 1000 Iterations

10

Columnstore

Columnstore

11

• Technology for storing, retrieving and managing data by using a columnar data format

• Better performance for new BATCH execution mode

• Preferred for data warehousing and analytics workloads

• Not the only index

• Physical storage for the entire table

• Updatable

• Filtered

• Compressed (from 2014)

• Secondary index on row-store table

• Contains a copy of part/all of the rows and columns

Nonclustered Columnstore Index (NCI)

12

Key Points

• Create an updateable non-clustered columnstore index (NCCI) for analytics queries

• Drop all other indexes that were created for analytics.

• No OLTP Application changes.

• ColumnStore index automatically keeps up with DML operations

• Query Optimizer will choose columnstore index where needed

Relational Table (disk-based)(Clustered Index/Heap)

Btree IndexD

ele

te b

itm

ap

Dele

te B

uff

er

Delta rowgroups

Minimizing Columnstore Index overhead – Filtered index

13

Key Points

• Create Columnstore only on cold data – using filtered predicate to minimize maintenance

• Analytics query accesses both columnstore and ‘hot’ data transparently

• Example:

– Order Management Application: create nonclustered columnstore index … where order_status = ‘SHIPPED’

Relational Table (disk-based)(Clustered Index/Heap)

Btree IndexD

ele

te b

itm

ap

Dele

te B

uff

er

HOT

Delta rowgroups

Minimizing Columnstore Index overhead – Compression delay

14

Syntax – delay in minutes:

Create nonclustered columnstore index <name>

on <table> (<columns>)

with (compression_delay = 30)

Relational Table (disk-based)(Clustered Index/Heap)

Btree Index

HOT

Delta rowgroups

Dele

te b

itm

ap

Dele

te B

uff

er

Key Points

• Delta RG is only compressed after ‘Compression_Delay’ duration

• Minimizes/Eliminates index fragmentation

16

In-Memory Analytics

Operational analytics

17

Benefits

• No Data Latency

• No ETL

• No Separate DW

Challenges

• Minimizing Impact on OLTP

workload

• Delivering Performant Analytics

It is NOT for

• Extreme analytics – needs star-

schema, pre-aggregated cubes

Presentation Layer

Application Layer

SQL Server

OLTP system

Add columnstore index

(Create in-memory table)

Insert into <transactions>

values (‘<upc-code>, ‘flowers’, $20.00)

Select ProduceName, ExpiryDate, Quantity, sum (sales)

From <transactions>

Where ProduceType =‘perishables’

Group By ProduceName, ExpiryDate

Reporting and analytics

Operational analytics

ETL

SQL Server Analysis Services

SSAS Tabular

Data Mart

SSAS

Data refresh

Reporting and analytics

SQL Server

Scenario 2

Data Stage

Analytical query

Data upload (L1)

In-MemoryAnalytics in Data Warehouse

18

Tabular Mode

Reporting and analytics

DWH / SQL Server

Scenario 1

Disk based tables

Data Stage

Data upload (L1)

Analytical query

Data

Late

ncy

Response T

ime

Data

Late

ncy

Response T

ime

Columnstore indexes

(non/clustered)

Reporting and analytics

SQL Server

Scenario 3

Data Stage

Analytical query

Data upload (L1)

Columnstore indexes

(non/clustered)

25x

+ 3%

10M rows \ Aggregations: Sum, Max, DistinctCount

40x

+ 50%

ADASTRA CZECH REPUBLICAdastra, s.r.o.

Karolinská 654/2, 186 00 Praha 8

Tel.: +420 271 733 303

infocz@adastragrp.com

www.adastra.cz

ADASTRA GROUP North America8500 Leslie St.

Markham, Ontario, L3T 7M8

Tel: +1 905 881 7946

info@adastragrp.com

Restrictions for public release and use:This document can comprise confidential information. As such it may not, without Adastra’s prior consent, be copied or transferred.

Important:All brands and names of products given in this documentation are or can be registered trademarks of their owners.© 2016 Adastra, all rights reserved.

19

Thank you!