《 中国科学指标数据库 》 （ CSI ） China Science Indicators Database 功能及应用
SQL Server White Paper Templatedownload.microsoft.com/download/D/4/9/D490CB39-5… · Web viewSQL...
41
SSIS 操操操操操操操 SQL Server 技技技技 操 ：Alexei Khalyako、Carla Sabotta、Silvano Coriani、Sreedhar Pelluru、Steve Howard 操 ：Cindy Gross、David Pless、Mark Simms、Daniel Sol 操 ：2012 技 12 技 操 ：SQL Server 2012；Windows Azure SQL Database 操 ：SQL Server Integration Services (SSIS) 技 技技技 一，、 (ETL) 技 技技技 一一， Windows Azure (WA) SQL Database 技技技技技技 。 SSIS 技技技技技技 技技技 ，。 SSIS 技技技技 技技技技 ， SSIS 技技技技技技 技技技技技技技技技技技技 （），一。
Transcript of SQL Server White Paper Templatedownload.microsoft.com/download/D/4/9/D490CB39-5… · Web viewSQL...
SQL Server White Paper TemplateSQL Server
Alexei KhalyakoCarla SabottaSilvano CorianiSreedhar PelluruSteve Howard
Cindy GrossDavid PlessMark SimmsDaniel Sol
2012 12
SQL Server 2012Windows Azure SQL Database
SQL Server Integration Services (SSIS) (ETL) Windows Azure (WA) SQL Database SSIS SSIS SSIS
“” URL Internet
© 2011 Microsoft
5 5 5 Azure 6 7 7 8 9 9 SQL Server Integration Services (SSIS) 9 ADO.NET SqlBulkCopy 10 (BCP.exe) 11 Azure Blob 11 12 12 13 13 14 14 Azure SQL Database 14 15 16 16 16 22 23 25 25 SSIS 30 31 31 31 SSIS 32 BLOB 34 SSIS 2012 36 36 37 42 47
1
Azure
(http://social.technet.microsoft.com/wiki/contents/ articles/1926.how-to-shard-with-windows-azure-sql-database.aspx) 20 GB/ Azure SQL Database WA (VM) SQL Server
SSIS Azure Blob SSIS
Azure SQL Database Windows Azure (http://msdn.microsoft.com/zh-cn/library/windowsazure/ jj156154.aspx)
Azure SQL Database WA VM SQL Server WA Azure
SSIS WA VM SSIS DBA/ETL SSIS ETL
WA VM SQL Server Azure SQL Database
Windows Azure SQL Database SSIS
SSIS
SSIS
ADO.NET SqlBulkCopy
SqlBulkCopy ReliableSqlConnection (http://msdn.microsoft.com/zh-cn/library/microsoft.practices.enterpriselibrary. windowsazure.transientfaulthandling.sqlazure.reliablesqlconnection(v=pandp.50).aspx) (http://msdn.microsoft.com/zh-cn/library/hh680934(v=PandP.50).aspx) SqlConnection (http://msdn.microsoft.com/zh-cn/library/system.data.sqlclient. sqlconnection.aspx)
10 1000 SqlBulkCopy SqlBulkCopy (TVP)“INSERT INTO Destination SELECT * FROM @TVP”
API SqlBulkCopy (http://msdn.microsoft.com/zh-cn/library/system.data.sqlclient.sqlbulkcopy.aspx)
(BCP.exe)
BCP.EXE BCP.EXE SSIS BCP.EXE SSIS SqlBulkCopy
Azure Blob
Azure Blob Azure .NET Azure Blob REST Azure Blob (http://www.windowsazure.com/ en-us/develop/net/how-to-guides/queue-service/ ) .NET Windows Azure Blob (http://www.windowsazure.com/en-us/develop/net/how-to-guides/blob-storage/)
Azure SQL Database
Azure SQL Database SQL Server WA VM Azure Blob “”
4
Azure VM
Web Azure WA VM
Windows Server System Center
SSIS Azure SDK Azure VM (.msi)
SSIS
WA VM Windows Server Azure Web Windows .NET
/Windows Azure Blob SSIS (DtExec.exe) VM
Azure SQL Database
SQL Server Windows Azure SQL Database Windows Azure SQL Database SQL Server
Windows Azure SQL Database SQL Server Windows Azure SQL Database (HA) Windows Azure SQL Database SQL Server (Windows Azure SQL Database) (http://msdn.microsoft.com/zh-cn/ library/windowsazure/ff394115.aspx)
Windows Azure SQL Database SQL Server Windows Azure SQL Database Microsoft Windows Azure SQL Database /
WA Windows Azure SQL Database SQL Server Windows Azure -
SSIS
“ Azure SQL Server 2012 SSIS” WA MSDN SQL Server 2012 Microsoft
ETL ETL ETL
CREATE TABLE [Production].[TransactionHistory]
[ProductID] INT NOT NULL,
[ReferenceOrderID] INT NOT NULL,
[ReferenceOrderLineID] INT NOT NULL,
[TransactionDate] INT NOT NULL,
[TransactionType] NCHAR(1) NOT NULL,
[Quantity] INT NOT NULL,
[ActualCost] MONEY NOT NULL,
[ModifiedDate] DATETIME NOT NULL,
TransactionID
SSIS
3. ORDER BY TransactionID TransactionID
SSIS SQL
SELECT MAX(TransactionID) FROM Production.TransactionHistory
TransactionID Null SQL TransactionID TransactionID TransactionID ORDER BY TransactionID
ISNULL WHERE CASE
SQL Server ORDER BY SQL Server SORT SSIS
4. SSIS minTransactionIDAtSource SQL Int32 TransactionID
5. “ TransactionID”
a. “”
b. SQLStatement SQLSourceType “” SQLStatement
SELECT ISNULL(MIN(TransactionID), 0) FROM Production.TransactionHistory
SQL SSIS SSIS
5 SQL transactionID
c. “”“ SQL ”
d. ResultSet “”
e. SQL “”
f. “”
g. “” 0 User::minTransactionIDAtSource 4 “” SQL
h. “ SQL ”
6. SQL “ TransactionID”“ TransactionID”
7. maxTransactionIDAtDestination Int32 TransactionID
8. “ SQL ”
a. ResultSet “”
b.
?
f. “”
iii. LONG 32
iv. “” 0 0
g. “”“”
i. “” 0
ii. “” 7 User::maxTransactionIDAtDestination
OLE DB SQL SSIS ADO.NET OLE DB
9. “”“ TransactionID”
SQL
10. sourceQuery
a. “”
b. “” 7 User::MaxTransactionIDAtDestination Int32 String DT_WSTR “” (DT_WSTR, <<length>>)
c. “”“”
"SELECT TransactionID, ProductID, ReferenceOrderID, ReferenceOrderLineID, TransactionDate, TransactionType, Quantity, ActualCost, ModifiedDate FROM Production.TransactionHistory WHERE TransactionID > " + (DT_WSTR, 12 ) @[User::maxTransactionIDAtDestination] + " ORDER BY TransactionID"
12
12 SQL INT BIGINT 22
“” SSIS maxTransactionIDAtDestination
SQL ORDER BY ORDER BY
d. “” SQL SourceQuery
11. “”
“ Production.TransactionHistory ” SSIS
13. “ TransactionHistory ”“OLE DB ”
a. “OLE DB ”“”
b. “”“ SQL ”
c. “” 10 User::sourceQuery
d. “”
f. “”“OLE DB ”
14. OLE DB “TransactionHistory ”“ TransactionHistory ”
a. “OLE DB ”“”
b. “”“ - ”
c. “” Production.TransactionHistory
d. TransactionHistory “” AdventureWorks “”
e. “”
(http://msdn.microsoft.com/zh-cn/library/ ms140083.aspx)
f. “”“OLE DB ”
15.
“ TransactionID”“ TransactionID”
·
· Shard00 key value 1000
· Shard01 key value 1100
· Shard02 key value 1050
ShardID == 0 && TransactionID > @[User::maxTransactionIDAtShard00]
ShardID == 1 && TransactionID > @[User::maxTransactionIDAtShard01]
ShardID == 2 && TransactionID > @[User::maxTransactionIDAtShard02]
2 3 maxTransactionIDAtShardXX
· SQL Server SSIS SQL SQL MaxTransactionIDAtDestination MinTransactionIDAtSource “ TransactionID” SQL SQL_VARIANT
· WHERE
1.
3.
[TransactionID] <= @[User::maxTransactionIDAtDestination]
·
SELECT MAX(EmployeeID) AS EmployeeID,
(SELECT MAX(EffectiveSequence)
FROM HumanResources.EmployeeHistory h
EffectiveDateEffectiveSequence EmployeeID
2. SSIS attemptNumber
3. FOR
4. FOR
6. SQL FOR FOR
7. SQL FOR
8. SQL
10. 10 false
11. FOR FailPackageOnFailure false FOR
12. SQL “”
SSIS
· –
· –
9000 (NIC) MaxJumboBuffers 8192
SQL Server SSIS 32676 SQL Server SSIS “”
“SSIS ”“”“”“”
SSIS
SSIS SSIS SSIS
SSIS
[] = [] x [DefaultBufferMaxRows]
DefaultBufferSize
DefaultBufferSize DefaultBufferMaxRows SSIS SSIS Buffers spooled
SSIS (http://msdn.microsoft.com/zh-cn/library/ms137622.aspx)
EngineThreads 10 10 1
(http://msdn.microsoft.com/zh-cn/library/ ms141031.aspx)
BLOB
SSIS BLOB XML BLOB SSIS BLOB BLOB BLOB SSIS
1. BLOBTempStoragePath BufferTempStoragePath TEMP TMP SSIS BLOBTempStoragePath SSIS
2. DefaultBufferSize DefaultBufferMaxRows BLOB BLOB DefaultBufferSize DefaultBufferMaxRows
a. MaxBufferSize BLOB 100 MB 104857600
b. 2 52428800 BLOB
c. BLOB + blob 2 blob 98% SSIS
· SQL Server
d. b c DefaultBufferMaxRows
DefaultBufferMaxRows MaxBufferSize blob
1. DefaultMaxRowsInBuffer Int32 MaxBufferSize
2. SQL DefaultBufferMaxRows 1 DefaultMaxRowsInBuffer
SQL SSIS SQL (http://technet.microsoft.com/zh-cn/library/cc280492.aspx)
3. DefaultBufferMaxRows “”“”
4. “”“” DefaultBufferMaxRows
5. 1 “”“”“”“”
6. “”“” BLOB
SSIS 2012
SQL Server 2012 Integration Services (SSIS) SSIS
3
(http://msdn.microsoft.com/zh-cn/library/ms345174.aspx)
“”
2. “”
3. “”
“”
1. “SQL Server ”“”“”
--
2. “”“”“”
3. “”“SQL Server Integration Services ”
4. “”“”“SSIS ”“”
5. “”“”“”
6.
Declare @execution_id bigint
“Integration Services ”“”“”
“Integration Services ” 24
“Integration Services ”
“”“”“”
“” 10 “”“”“”“”
“”“”
“” SQL Server “Integration Services ”“”
“”
“” SQL Server
SQL Server Management Studio SSISDB
SSISDB
catalog.execution_component_phases
“”“”
ID 33 sum DATEDIFF
Declare @execution_id bigint
Set @execution_id = 33
sum (DATEDIFF(ms, start_time, end_time)) as active_time,
DATEDIFF(ms,min(start_time), max(end_time)) as total_time
order by active_time desc
dm_execution_performance_counters
select * from [catalog].[dm_execution_performance_counters] (34)
EXEC [SSISDB].[catalog].[start_execution] @execid
create_execution Integration Services SQL Server Management Studio create_execution SSIS SSIS Visual Studio SSIS “” SSIS
SQL
2. “”
3. add_data_tap start_execution
add_data_tap task_package_path Visual Studio PackagePath Visual Studio “”“”“” PackagePath add_data_tap task_package_path
add_data_tap dataflow_path_id_string IdentificationString dataflow_path_id_string“” IdentificationString
<Program Files>\Microsoft SQL Server\110\DTS\DataDumps output[1].txt
catalog.add_data_tap_by_guid ( http://msdn.microsoft.com/zh-cn/library/hh230991 ) add_data_tap ID task_package_path Visual Studio ID
DECLARE @tap_id bigint
EXEC [SSISDB].[catalog].remove_data_tap @tap_id
select * from [SSISDB].[catalog].execution_data_taps where execution_id=@execid
execution_id create_execution catalog.executions ID
use SSISDB
http://www.microsoft.com/sqlserver/ SQL Server
http://technet.microsoft.com/zh-cn/sqlserver/ SQL Server
http://msdn.microsoft.com/zh-cn/sqlserver/ SQL Server
1 5
·
Alexei KhalyakoCarla SabottaSilvano CorianiSreedhar PelluruSteve Howard
Cindy GrossDavid PlessMark SimmsDaniel Sol
2012 12
SQL Server 2012Windows Azure SQL Database
SQL Server Integration Services (SSIS) (ETL) Windows Azure (WA) SQL Database SSIS SSIS SSIS
“” URL Internet
© 2011 Microsoft
5 5 5 Azure 6 7 7 8 9 9 SQL Server Integration Services (SSIS) 9 ADO.NET SqlBulkCopy 10 (BCP.exe) 11 Azure Blob 11 12 12 13 13 14 14 Azure SQL Database 14 15 16 16 16 22 23 25 25 SSIS 30 31 31 31 SSIS 32 BLOB 34 SSIS 2012 36 36 37 42 47
1
Azure
(http://social.technet.microsoft.com/wiki/contents/ articles/1926.how-to-shard-with-windows-azure-sql-database.aspx) 20 GB/ Azure SQL Database WA (VM) SQL Server
SSIS Azure Blob SSIS
Azure SQL Database Windows Azure (http://msdn.microsoft.com/zh-cn/library/windowsazure/ jj156154.aspx)
Azure SQL Database WA VM SQL Server WA Azure
SSIS WA VM SSIS DBA/ETL SSIS ETL
WA VM SQL Server Azure SQL Database
Windows Azure SQL Database SSIS
SSIS
SSIS
ADO.NET SqlBulkCopy
SqlBulkCopy ReliableSqlConnection (http://msdn.microsoft.com/zh-cn/library/microsoft.practices.enterpriselibrary. windowsazure.transientfaulthandling.sqlazure.reliablesqlconnection(v=pandp.50).aspx) (http://msdn.microsoft.com/zh-cn/library/hh680934(v=PandP.50).aspx) SqlConnection (http://msdn.microsoft.com/zh-cn/library/system.data.sqlclient. sqlconnection.aspx)
10 1000 SqlBulkCopy SqlBulkCopy (TVP)“INSERT INTO Destination SELECT * FROM @TVP”
API SqlBulkCopy (http://msdn.microsoft.com/zh-cn/library/system.data.sqlclient.sqlbulkcopy.aspx)
(BCP.exe)
BCP.EXE BCP.EXE SSIS BCP.EXE SSIS SqlBulkCopy
Azure Blob
Azure Blob Azure .NET Azure Blob REST Azure Blob (http://www.windowsazure.com/ en-us/develop/net/how-to-guides/queue-service/ ) .NET Windows Azure Blob (http://www.windowsazure.com/en-us/develop/net/how-to-guides/blob-storage/)
Azure SQL Database
Azure SQL Database SQL Server WA VM Azure Blob “”
4
Azure VM
Web Azure WA VM
Windows Server System Center
SSIS Azure SDK Azure VM (.msi)
SSIS
WA VM Windows Server Azure Web Windows .NET
/Windows Azure Blob SSIS (DtExec.exe) VM
Azure SQL Database
SQL Server Windows Azure SQL Database Windows Azure SQL Database SQL Server
Windows Azure SQL Database SQL Server Windows Azure SQL Database (HA) Windows Azure SQL Database SQL Server (Windows Azure SQL Database) (http://msdn.microsoft.com/zh-cn/ library/windowsazure/ff394115.aspx)
Windows Azure SQL Database SQL Server Windows Azure SQL Database Microsoft Windows Azure SQL Database /
WA Windows Azure SQL Database SQL Server Windows Azure -
SSIS
“ Azure SQL Server 2012 SSIS” WA MSDN SQL Server 2012 Microsoft
ETL ETL ETL
CREATE TABLE [Production].[TransactionHistory]
[ProductID] INT NOT NULL,
[ReferenceOrderID] INT NOT NULL,
[ReferenceOrderLineID] INT NOT NULL,
[TransactionDate] INT NOT NULL,
[TransactionType] NCHAR(1) NOT NULL,
[Quantity] INT NOT NULL,
[ActualCost] MONEY NOT NULL,
[ModifiedDate] DATETIME NOT NULL,
TransactionID
SSIS
3. ORDER BY TransactionID TransactionID
SSIS SQL
SELECT MAX(TransactionID) FROM Production.TransactionHistory
TransactionID Null SQL TransactionID TransactionID TransactionID ORDER BY TransactionID
ISNULL WHERE CASE
SQL Server ORDER BY SQL Server SORT SSIS
4. SSIS minTransactionIDAtSource SQL Int32 TransactionID
5. “ TransactionID”
a. “”
b. SQLStatement SQLSourceType “” SQLStatement
SELECT ISNULL(MIN(TransactionID), 0) FROM Production.TransactionHistory
SQL SSIS SSIS
5 SQL transactionID
c. “”“ SQL ”
d. ResultSet “”
e. SQL “”
f. “”
g. “” 0 User::minTransactionIDAtSource 4 “” SQL
h. “ SQL ”
6. SQL “ TransactionID”“ TransactionID”
7. maxTransactionIDAtDestination Int32 TransactionID
8. “ SQL ”
a. ResultSet “”
b.
?
f. “”
iii. LONG 32
iv. “” 0 0
g. “”“”
i. “” 0
ii. “” 7 User::maxTransactionIDAtDestination
OLE DB SQL SSIS ADO.NET OLE DB
9. “”“ TransactionID”
SQL
10. sourceQuery
a. “”
b. “” 7 User::MaxTransactionIDAtDestination Int32 String DT_WSTR “” (DT_WSTR, <<length>>)
c. “”“”
"SELECT TransactionID, ProductID, ReferenceOrderID, ReferenceOrderLineID, TransactionDate, TransactionType, Quantity, ActualCost, ModifiedDate FROM Production.TransactionHistory WHERE TransactionID > " + (DT_WSTR, 12 ) @[User::maxTransactionIDAtDestination] + " ORDER BY TransactionID"
12
12 SQL INT BIGINT 22
“” SSIS maxTransactionIDAtDestination
SQL ORDER BY ORDER BY
d. “” SQL SourceQuery
11. “”
“ Production.TransactionHistory ” SSIS
13. “ TransactionHistory ”“OLE DB ”
a. “OLE DB ”“”
b. “”“ SQL ”
c. “” 10 User::sourceQuery
d. “”
f. “”“OLE DB ”
14. OLE DB “TransactionHistory ”“ TransactionHistory ”
a. “OLE DB ”“”
b. “”“ - ”
c. “” Production.TransactionHistory
d. TransactionHistory “” AdventureWorks “”
e. “”
(http://msdn.microsoft.com/zh-cn/library/ ms140083.aspx)
f. “”“OLE DB ”
15.
“ TransactionID”“ TransactionID”
·
· Shard00 key value 1000
· Shard01 key value 1100
· Shard02 key value 1050
ShardID == 0 && TransactionID > @[User::maxTransactionIDAtShard00]
ShardID == 1 && TransactionID > @[User::maxTransactionIDAtShard01]
ShardID == 2 && TransactionID > @[User::maxTransactionIDAtShard02]
2 3 maxTransactionIDAtShardXX
· SQL Server SSIS SQL SQL MaxTransactionIDAtDestination MinTransactionIDAtSource “ TransactionID” SQL SQL_VARIANT
· WHERE
1.
3.
[TransactionID] <= @[User::maxTransactionIDAtDestination]
·
SELECT MAX(EmployeeID) AS EmployeeID,
(SELECT MAX(EffectiveSequence)
FROM HumanResources.EmployeeHistory h
EffectiveDateEffectiveSequence EmployeeID
2. SSIS attemptNumber
3. FOR
4. FOR
6. SQL FOR FOR
7. SQL FOR
8. SQL
10. 10 false
11. FOR FailPackageOnFailure false FOR
12. SQL “”
SSIS
· –
· –
9000 (NIC) MaxJumboBuffers 8192
SQL Server SSIS 32676 SQL Server SSIS “”
“SSIS ”“”“”“”
SSIS
SSIS SSIS SSIS
SSIS
[] = [] x [DefaultBufferMaxRows]
DefaultBufferSize
DefaultBufferSize DefaultBufferMaxRows SSIS SSIS Buffers spooled
SSIS (http://msdn.microsoft.com/zh-cn/library/ms137622.aspx)
EngineThreads 10 10 1
(http://msdn.microsoft.com/zh-cn/library/ ms141031.aspx)
BLOB
SSIS BLOB XML BLOB SSIS BLOB BLOB BLOB SSIS
1. BLOBTempStoragePath BufferTempStoragePath TEMP TMP SSIS BLOBTempStoragePath SSIS
2. DefaultBufferSize DefaultBufferMaxRows BLOB BLOB DefaultBufferSize DefaultBufferMaxRows
a. MaxBufferSize BLOB 100 MB 104857600
b. 2 52428800 BLOB
c. BLOB + blob 2 blob 98% SSIS
· SQL Server
d. b c DefaultBufferMaxRows
DefaultBufferMaxRows MaxBufferSize blob
1. DefaultMaxRowsInBuffer Int32 MaxBufferSize
2. SQL DefaultBufferMaxRows 1 DefaultMaxRowsInBuffer
SQL SSIS SQL (http://technet.microsoft.com/zh-cn/library/cc280492.aspx)
3. DefaultBufferMaxRows “”“”
4. “”“” DefaultBufferMaxRows
5. 1 “”“”“”“”
6. “”“” BLOB
SSIS 2012
SQL Server 2012 Integration Services (SSIS) SSIS
3
(http://msdn.microsoft.com/zh-cn/library/ms345174.aspx)
“”
2. “”
3. “”
“”
1. “SQL Server ”“”“”
--
2. “”“”“”
3. “”“SQL Server Integration Services ”
4. “”“”“SSIS ”“”
5. “”“”“”
6.
Declare @execution_id bigint
“Integration Services ”“”“”
“Integration Services ” 24
“Integration Services ”
“”“”“”
“” 10 “”“”“”“”
“”“”
“” SQL Server “Integration Services ”“”
“”
“” SQL Server
SQL Server Management Studio SSISDB
SSISDB
catalog.execution_component_phases
“”“”
ID 33 sum DATEDIFF
Declare @execution_id bigint
Set @execution_id = 33
sum (DATEDIFF(ms, start_time, end_time)) as active_time,
DATEDIFF(ms,min(start_time), max(end_time)) as total_time
order by active_time desc
dm_execution_performance_counters
select * from [catalog].[dm_execution_performance_counters] (34)
EXEC [SSISDB].[catalog].[start_execution] @execid
create_execution Integration Services SQL Server Management Studio create_execution SSIS SSIS Visual Studio SSIS “” SSIS
SQL
2. “”
3. add_data_tap start_execution
add_data_tap task_package_path Visual Studio PackagePath Visual Studio “”“”“” PackagePath add_data_tap task_package_path
add_data_tap dataflow_path_id_string IdentificationString dataflow_path_id_string“” IdentificationString
<Program Files>\Microsoft SQL Server\110\DTS\DataDumps output[1].txt
catalog.add_data_tap_by_guid ( http://msdn.microsoft.com/zh-cn/library/hh230991 ) add_data_tap ID task_package_path Visual Studio ID
DECLARE @tap_id bigint
EXEC [SSISDB].[catalog].remove_data_tap @tap_id
select * from [SSISDB].[catalog].execution_data_taps where execution_id=@execid
execution_id create_execution catalog.executions ID
use SSISDB
http://www.microsoft.com/sqlserver/ SQL Server
http://technet.microsoft.com/zh-cn/sqlserver/ SQL Server
http://msdn.microsoft.com/zh-cn/sqlserver/ SQL Server
1 5
·
