Tips for Improving INSERT Performance in DB2 Universal Database

8/13/2019 Tips for Improving INSERT Performance in DB2 Universal Database

1/28

Sign in (or register)EnglishIBM

Technical topicsTechnical topics Evaluation softwareEvaluation software CommunityCommunity EventsEvents Search developerWorks

Tips for improving INSERT performance in DB2 Universal Database

Bill Wilkins([email protected]), DB2 Vendor Enablement, IBM

Summary: Improving the performance of database inserts is very important to managing a database, especially in an OLTP environment. This articleexplains exactly what happens when an insert occurs, looks at alternatives, and examines issues that affect insert performance, such as locking, indexmaintenance, and constraint management. And, to back up his advice, the author includes performance measurement results.

Date: 11 Mar 2004Level: Intermediate

Activity: 74917 viewsComments: 0 (View| Add comment- Sign in)

Average rating (71 votes)Rate this article

Introduction

IBM DB2 e-kit for Database Professionals

Learn how easy it is to get trained and certified for DB2 for Linux, UNIX, and Windows with the IBM DB2 e-kit for Database Professionals. Register now,and expand your skills portfolio, or extend your DBMS vendor support to include DB2.

The insertion of rows is one of the most common and important tasks you will perform when using a DB2 Universal Database (UDB). This article is acompilation of techniques for optimizing the performance of inserts, particularly high volume inserts. As in most any performance discussion, there aretradeoffs. We'll discuss the tradeoffs that optimizing inserts can introduce. For example, a technique you use for inserts may require additional processingfollowing the inserts, or may impact query performance. I will provide some performance measurement results to give you an idea of the significance of many

of the optimization techniques. Appendix Acontains a numbered summary of the results., and throughout the article I'll refer to the results by their testnumbers. In the Conclusionsection below is a summary of the most beneficial techniques, and Appendix Blists all of the suggestions made in the paper.

Tips for improving INSERT performance in DB2 Universal Database http://www.ibm.com/developerworks/data/library/tips/dm-0403wilkins/

1 of 28 12/6/2013 4:17 PM


2/28


3/28


4/28


5/28


6/28


7/28


8/28


9/28


10/28


11/28


12/28


13/28


14/28



15/28

Total elapsed asynchronous write time = 25000

LSN Gap cleaner triggers = 21

Dirty page steal cleaner triggers = 0

Dirty page threshold cleaner triggers = 0

Update/Insert/Delete statements executed = 100000

Rows inserted = 100000

Take a look at "Rows inserted" in consecutive snapshots to see if the rate of inserts varies during the course of a bulk insert. Most of the other values reflectthe amount of I/O and the effectiveness of page cleaning. Regarding the latter, ideally you'll see that all of the data writes are asynchronous, and all of thebuffer pool write time is asynchronous (as is the case in the above output); if not, try lowering CHNGPGS_THRESH and/or increasing NUM_IOCLEANERS.

Additional information on the insert can usually be found in the dynamic SQL snapshot. It can be very useful to look at the total elapsed time and compare

that with the user and system CPU times. Most of the difference between the elapsed and CPU times should taken up by I/O, so it may be clear where thebulk of the time is being spent and where to put your tuning effort.

Here's a subset of a dynamic SQL snapshot entry for a CLI array insert of 100,000 rows. Note that the "Number of executions" is one per row, even thoughthe application sent only 1/10 of that number of arrays.

Number of executions = 100000

Number of compilations = 1

Rows written = 100000

Buffer pool data logical reads = 102120

Total execution time (sec.ms) = 13.830543

Total user cpu time (sec.ms) = 10.290000

Total system cpu time (sec.ms) = 0.130000

Statement text = INSERT into test1 values(?, ?, ?, ?, ?)

Some additional information to look at in snapshot output:

"Lock waits" and "Time database waited on locks" -- Use these to see if the locks on inserted rows are causing concurrency issues for otherapplications.Table Snapshot -- "Rows Written" will reflect the number of rows inserted (or updated).

Event monitoring

DB2 event monitor lets you obtain performance information on events as they occur on the server, such as statement or transaction completion. For thepurpose of insert performance analysis, you may want to create an event monitor for statements and activate it during a period of insert executions. Although

15 of 28 12/6/2013 4:17 PM



16/28

it may be overkill, the event monitor output will show the elapsed time of each insert statement. For OLTP-like applications, the overhead of running eventmonitoring for statements is quite high, and the output voluminous, so be careful not to have the monitor running for too long; even a few seconds canproduce megabytes of output. You could write the monitor information to a table to facilitate analysis of the results, such as the performance trend over time.

The following is a condensed sample statement event from a series of 10,000 CLI array inserts of 10 rows each. There is one event per array, in this case per

10 rows.

17) Statement Event ...

Appl Handle: 9

Appl Id: *LOCAL.wilkins.0953D9033443

-------------------------------------------

Type : Dynamic

Operation: Execute

Section : 4 Creator : NULLID

Package : SYSSH200

Text : INSERT into test1 values(?, ?)

-------------------------------------------

Start Time: 01-28-2004 22:34:43.918444

Stop Time: 01-28-2004 22:34:43.919763

Exec Time: 0.001319 seconds

Number of Agents created: 1

User CPU: 0.000000 seconds

System CPU: 0.000000 seconds

Fetch Count: 0

Rows read: 0

Rows written: 10

Internal rows deleted: 0

Internal rows updated: 0

Internal rows inserted: 0

Bufferpool data logical reads: 10

SQLCA:

sqlcode: 0 sqlstate: 00000

Explain

If the performance of your inserts is not as good as expected, it may be because there is "hidden" processing taking place. As discussed earlier, this processingcould take such forms as index maintenance, constraint verification, or trigger execution. Running a form of Explain (such as Visual Explain, or the Explainstatement plus db2exfmt) against your insert will reveal most of the extra processing (but not index maintenance). You may be able to eliminate the extraprocessing if it seems to be the cause of the performance problem.

16 of 28 12/6/2013 4:17 PM



17/28

As a simple example, the following graph (produced by db2exfmt) shows the access plan for an "Insert into Test1 values (?, ?, ...)". Can you guess what theextra processing is here? The answer is below.

Rows

RETURN

( 1)

Cost

I/O

|

0.333333

TBSCAN

( 2)

28.2956

1

/----+---\ 0.04 1

FILTER TABFNC: SYSIBM

( 3) GENROW

28.2266

1

|

1

NLJOIN

( 4)

28.1268

1

/---------+--------\

1 1

INSERT IXSCAN

( 5) ( 7)

25.5248 2.60196

1 0

/---+--\ |

1 116 120 TBSCAN TABLE: WILKINS INDEX: SYSIBM

( 6) TEST1 SQL0401290925513

0.0048

0

|

1

TABFNC: SYSIBM

GENROW

17 of 28 12/6/2013 4:17 PM


18/28


19/28



20/28

1 CLI -- SQLExecDirect for each row 10,000 180.632 CLI -- Prepared statement but 1 row/insert 10,000 92.053 CLI -- Prepared statement and array insert (10 rows/insert) 10,000 12.85

Tests 1-3 had autocommit on and thus had a Commit for each insert. That is, there was a Commit for each row in tests 1 and 2, and for every 10 rows in test

3. For the performance impact of Commit, see tests 61-78.

Table 2. Populating one table from another one

Test # Insert method / Notes # Rows Elapsed secs.6 INSERT ... SELECT with logging 100,000 15.667 INSERT ... SELECT with NOT LOGGED INITIALLY 100,000 16.438 Same as test 7, but with CHNGPGS_THRESH = 5 100,000 11.809 Load from a cursor (same Select as tests 6-8) 100,000 12.95

Tests 6-9 timed the population of one table from another, including the time to Commit. Test 7 show that using NOT LOGGED INITIALLY (NLI) actuallycaused a slowdown in performance, because of the new pages needing to be written to disk at Commit time: the Commit took over half of the time. However,with the more aggressive page cleaning in test 8, performance was improved substantially, mostly through the Commit time being reduced by over fiveseconds. Test 9 showed a nice 17% improvement from using Load instead of insert ... Select, without the risk of NLI as in test 6.

Table 3. Effects of check constraints, foreign keys and triggersTest # Insert method / Notes # Rows Elapsed secs.11 CLI -- Same as test 3, but 100K rows, commit 1000 100,000 31.5112 CLI -- same as test 11, but 1 check constraint 100,000 33.6513 CLI -- same as test 11, but 2 check constraints 100,000 36.6314 CLI -- same as test 11, but 1 foreign key (FK) 100,000 55.3715 CLI -- same as test 11, but 2 foreign keys (FK's) 100,000 72.71

16 CLI -- same as test 14, but FK added after Inserts 100,000 32.8417 CLI -- same as test 15, but FK's added after Inserts 100,000 38.8918 CLI -- same as test 11, but 1 trigger doing Insert 100,000 67.5719 CLI -- same as test 11, but 2 triggers doing Insert 100,000 175.9520 CLI -- same as test 11, but 1 trigger doing Update 100,000 54.7121 CLI -- same as test 11, but 2 triggers doing Update 100,000 150.1822 CLI -- same as test 11, but with 1M rows 100,000 282.0223 CLI -- same as test 22, but with APPEND ON 100,000 281.64

Clearly the use of check constraints has a minor effect on performance, but foreign keys and triggers can be extremely significant. In tests 18 and 19, each

20 of 28 12/6/2013 4:17 PM


21/28



22/28

Table 6. Tests showing effect of large arrays and large number of inserts per COMMIT

Test # Insert method / Notes # Rows Elapsed secs.1 CLI -- SQLExecDirect for each row (repeated from above) 10,000 183.5561 CLI -- same as test 1, but with a Commit every 5 rows 10,000 118.41

62 CLI -- same as test 1, but with a Commit every 10 rows 10,000 114.2363 CLI -- same as test 1, but with a Commit every 100 rows 10,000 103.4364 CLI -- same as test 1, but with a Commit every 1000 rows 10,000 102.862 CLI -- Prepared statement but 1 row/insert (repeated from above) 10,000 92.0565 CLI -- same as test 2, but with a Commit every 5 rows 10,000 22.3466 CLI -- same as test 2, but with a Commit every 10 rows 10,000 20.7867 CLI -- same as test 2, but with a Commit every 100 rows 10,000 10.11

68 CLI -- same as test 2, but with a Commit every 1000 rows 10,000 7.5869 CLI -- as test 3 (10x rows); array size 10, Commit per 10 rows 10,000 118.1870 CLI -- same as test 69, but array size 10, Commit per 100 rows 10,000 56.7171 CLI -- same as test 69, but array size 10, Commit per 1000 rows 10,000 30.0972 CLI -- same as test 69, but array size 100, Commit 100 rows 10,000 50.6573 CLI -- same as test 69, but array size 100, Commit 1000 rows 10,000 24.2774 CLI -- same as test 69, but array size 1000, Commit 1000 rows 10,000 23.4375 CLI -- same as test 69, but array size 2, Commit 2 rows 10,000 471.82

76 CLI -- same as test 69, but array size 2, Commit 10 rows 10,000 155.2177 CLI -- same as test 69, but array size 2, Commit 100 rows 10,000 74.8278 CLI -- same as test 69, but array size 2, Commit 1000 rows 10,000 48.5179 CLI -- Using Load via SQLSetStmtAttr (array size 10) 10,000 13.68

The above tests show the great benefit of using large arrays and committing a large number of rows at a time, with the latter being more important.

Table 7. Using SMS or DMS

Test # Insert method / Notes # Rows Elapsed secs.11 CLI -- SMS with db2empfa run (repeated from above) 100,000 31.5181 Same as 11 but with DMS file tablespace 100,000 25.5282 Same as 11 but db2empfa NOT run (SMS) 100,000 62.8783 Same as 11 but extentsize = 4 (and prefetchsize = 4) 100,000 38.3783 Same as 11 but extentsize = 4 (and prefetchsize = 4) 100,000 38.37

84 Same as 11 but extentsize = 8 (and prefetchsize = 8) 100,000 34.6185 Same as 11 but extentsize = 16 (and prefetchsize = 16) 100,000 31.75

22 of 28 12/6/2013 4:17 PM



23/28

The above tests show that for SMS, running db2empfa is critical to good insert performance, and that using a smaller extentsize than 32 pages is detrimental.DMS is somewhat preferable to SMS.

Table 8. Impact of a CLOB column

Test # Insert method / Notes # Rows Elapsed secs.11 CLI -- (repeated from above) 100,000 31.5191 Same as 11 but with CHAR(10) column now CLOB(10) 100,000 286.49

The above comparison shows the extreme impact of having a CLOB column, even a very short one. Variations of test 91 were tried, but the results were verysimilar, regardless of the choice of logged or not logged, or compact or not compact, for the CLOB column.

Table 9. Using table locks instead of row locks

Test # Insert method / Notes # Rows Elapsed secs.11 CLI -- (repeated from above) 100,000 31.51101 Same as 11 but with LOCKSIZE TABLE in use 100,000 30.58

The above comparison shows that causing table locks to be used instead of row locks saved about 3% in elapsed time.

Table 10. Varying LOGBUFSZ

Test # Insert method / Notes # Rows Elapsed secs.104 Same as test 71, but Commit 10K rows (LOGBUFSZ=8) 100,000 28.53105 Same as 104 but with LOGBUFSZ = 256 100,000 24.91

Test 105 showed about a 13% improvement from raising LOGBUFSZ high enough such that there were no log writes forced by having the log buffer fill upduring the Inserts.

Table 11. Using multiple optimizations

Test # Insert method / Notes # Rows Elapsed secs.22 CLI -- (repeated from above) 1,000,000 282.02111 Same as 22 but with all optimizations (see below) 1,000,000 160.451(*) CLI -- (as test 1, but extrapolated to 1,000,000 rows) 1,000,000 ~18000.00

Test 111 used all of the optimizations tried above as alternatives to the baseline: DMS instead of SMS, arrays of 1000 rows (vs. 10), committing every 10000rows (vs. 1000), LOCKSIZE TABLE, APPEND ON, and LOGBUFSZ 256. Note that test 22 was already somewhat optimized as compared to the originalbaseline (test 1). The time for test 1 has been extrapolated here to 1,000,000 rows to show the vast difference in performance that is possible with only a few

23 of 28 12/6/2013 4:17 PM


24/28



25/28

Use Explain to discover "hidden" processing.26.And, it goes without saying, use the fastest possible CPU's and disks.27.

Resources

Learn

"DB2 for OS/390 and z/OS V7 Load Resume and SQL INSERT Performance": (developerWorks, Mar 2003): Read the results of different ways ofadding data to a DB2 table in DB2 for z/OS: SQL INSERT, online load resume, and offline load resume.

"Allocating for High-Velocity Inserts on DB2 UDB for iSeries" (developerWorks, Mar, 203):Improve the performance of those applications in

concurrent environments by preallocating storage.

developerWorks Information Management zone: Learn more about DB2. Find technical documentation, how-to articles, education, downloads, productinformation, and more.

Stay current with developerWorks technical events and webcasts.

Get products and technologies

Build your next development project with IBM trial software, available for download directly from developerWorks.

Discuss

Participate in developerWorks blogsand get involved in the developerWorks community.

About the author

Bill Wilkinsis a senior technical specialist in the IBM Information Management Business Partner Enablement group in Toronto. Prior to this he worked inDB2 performance analysis in the IBM Toronto development laboratory for six years. He is an IBM Certified Advanced Technical Expert in DB2 UDB, hascontributed to two books on DB2, and has over twenty-six years of experience in the IT industry.

Close [x]

25 of 28 12/6/2013 4:17 PM



26/28

developerWorks: Sign in

IBM ID:Need an IBM ID?

Forgot your IBM ID?

Password:Forgot your password?Change your password

Keep me signed in.

By clicking Submit, you agree to the developerWorks terms of use.

The first time you sign into developerWorks, a profile is created for you. Select information in your profile (name, country/region, and company) isdisplayed to the public and will accompany any content you post. You may update your IBM account at any time.

All information submitted is secure.

Close [x]

Choose your display name

The first time you sign in to developerWorks, a profile is created for you, so you need to choose a display name. Your display name accompanies the contentyou post on developerWorks.

Please choose a display name between 3-31 characters. Your display name must be unique in the developerWorks community and should not be your emailaddress for privacy reasons.

Display name: (Must be between 3 31 characters.)

By clicking Submit, you agree to the developerWorks terms of use.

26 of 28 12/6/2013 4:17 PM



27/28

Print this page Share this page Follow developerWorks

All information submitted is secure.

Average rating (71 votes)

1 star 1 star

2 stars 2 stars

3 stars 3 stars

4 stars 4 stars

5 stars 5 stars

Add comment:

Sign inor registerto leave a comment.

Note: HTML elements are not supported within comments.

Notify me when a comment is added1000 characters left

Be the first to add a comment

27 of 28 12/6/2013 4:17 PM



28/28

About

Help

Contact us

Submit content

Feeds Report abuse

Terms of use

Third party notice

IBM privacy

IBM accessibility

Faculty

Students

Business Partners

28 of 28 12/6/2013 4:17 PM

Tips for Improving INSERT Performance in DB2 Universal Database

Documents

Transcript of Tips for Improving INSERT Performance in DB2 Universal Database