Loading Data Using Sq l Bulk Copy

8/11/2019 Loading Data Using Sq l Bulk Copy

1/21

SQL Azure and SqlBulkCopyPresented by Herve Roggero

Managing Partner, Blue Syntax Consulting

SQL Azure MVP

Email: [email protected]

Twitter: @hroggero

LinkedIn: SQL Azure and SQL Server Security

LinkedIn: Azure Florida Association

www.bluesyntax.net


2/21

AgendaAvailable tools to load data

INSERT commands

SqlDataAdapterAnalyzing BCP

Analyzing SqlBulkCopy

Performance test


3/21

Available tools/techniques INSERT

INSERT (2)

INSERT BULK SqlDataAdapter

Migration Wizard

SSIS

BCP / INSERT BULK

SqlBulkCopy


4/21

What makes up load time? Connection Time

Send Query

Compile Query Execute Query

Send Data


5/21

INSERT Classic way to insert records

Logged operation

Usually single record at a timeAll packets are encrypted => too verbose

VERY SLOW...

INSERT INTO dbo.MyProducts (Name, ListPrice) VALUES ('Helmet', 25.50)

INSERT INTO dbo.MyProducts (Name, ListPrice) VALUES ('Wheel', 30.00);


6/21

INSERT (2) New INSERT command in SQL Server 2008

Table Value Constructor

Not as verbose Operation is still logged

Faster than regular INSERT, but still SLOW

INSERT INTO dbo.MyProducts (Name, ListPrice) VALUES

('Helmet', 25.50),

('Wheel', 30.00);


7/21

INSERT (summary)

INSERT INTO

INSERT INTO

INSERT INTO SQL Azure

SQL Azure

SQL Azure


8/21

BULK INSERT NOT SUPPORTED in SQL AZURE

INSERT BULK INTO

SQL Server

Raw packets

Raw packets

Raw packetsTERMINATION

File


9/21

INSERT BULK Supported but requires feeding raw TDS packets

=> Must be used by a Tool

Very efficient and lean Sends 1 INSERT statement

Everything that follows are raw data packets

NO FURTHER COMPILATION/QUERY EXECUTION

SQL Azure waits until Session sends TERMINATION Operation not logged

FAST


10/21

INSERT BULK (cont.)

INSERT BULK INTO

SQL Azure

Raw packets

Raw packets

Raw packets

TERMINATION


11/21

BCP Uses INSERT BULK

Performance of BCP depends on INSERT BULK

However BCP can also extract data

FAST


12/21

Tools Tools that use the BCP program:

SSIS: Bulk Insert Task

SQL Azure Migration Wizard http://sqlazuremw.codeplex.com/

Tools that use the SQL Bulk Copy API

Enzo Data Copy


13/21

SqlDataAdapter Based on XML Datagrams (differences)

Very memory intensive

Can be very slow depending on approach Provides a way to work in Batches

Sends INSERT operations

SLOW


14/21

SqlBulkCopy Uses INSERT BULK operation

Programmatic access, options similar to BCP

Can send data in Batch FAST


15/21

Techniques SummaryTool/Technique Operation Logged SpeedINSERT INSERT Yes SLOW

BULK INSERT BULK INSERT No n/a

INSERT BULK INSERT BULK No FASTBCP INSERT BULK No FAST

SSIS/ MigrationWizard

INSERT BULK No FAST

Data Adapter INSERT Yes SLOW

SqlBulkCopy INSERT BULK No FAST

Enzo Data Copy INSERT BULK No FAST


16/21

About the performance test Load 40,000 records (on-premise -> SQL Azure)

Uses multithreading (TPL) 8 threads

Comparing Data Adapter

Stored Procedure

SqlBulkCopy API

INSERTs


17/21

Performance ChartX: Batch Size; Y: Load Time {40K records; DOP = 8}

Batch Size

LoadTime(sec)

2000 3000 40001000500100

10

20

30

Data Adapter

Stored Procedure

SqlBulkCopy

INSERT 2 [limited to batch of 1000]


18/21

Comparing Tools

40,000 Records (insecs)

2,000,000 Records(in mins)

SSIS 4.2 2.17

SQL Bulk Copy 2.18 4.3

Enzo Data Copy 25 1.41

0

5

10

15

20

25

30

SSIS

SQL Bulk Copy

Enzo Data Copy


19/21

Demos Sample Code

Uses parallel processing on all methods

Compares: Data Adapter

Inline INSERTs

Stored Procedure INSERTs

SqlBulkCopy

SSIS

Enzo Data Copy (www.bluesyntax.net)


20/21

Considerations Resilience to Throttling

Adapting to throttling conditions

Changing workload depending on errors Connection Retries

Retrying for connection loss

Causes include throttling, load balancing, network

http://social.technet.microsoft.com/wiki/contents/articles/sql-azure-connection-management.aspx


21/21

Thank youPresented by Herve Roggero

Managing Partner, Blue Syntax Consulting

SQL Azure MVP

Email: [email protected]

Twitter: @hroggero

LinkedIn: SQL Azure and SQL Server Security

LinkedIn: Azure Florida Association

Blog: www.geekswithblogs.net/hroggerowww.bluesyntax.net

Loading Data Using Sq l Bulk Copy

Documents

Transcript of Loading Data Using Sq l Bulk Copy