Sybase ASE 15.7- Two Case Studies of Successful Migration

(c) 2015 Independent SAP Technical User GroupAnnual Conference, 2015

- ISUG TECH 2015- ISUG TECH 2015ConferenceConference

ASE 15.7: 2 case studies of successful migration ASE 15.7: 2 case studies of successful migration

(shhhhh… preps floods & sc)(shhhhh… preps floods & sc)

, Andrew Melkonyan Senior DB Architect , Andrew Melkonyan Senior DB Architect


AgendaAgenda

Welcome Speaker Introduction ( Session Title add presentation

)title &Q A

2 [30]


WelcomeWelcome

, , , , , - ISUG ASUG UKSUG SAUG TechEd D CODE& .c

• I ndependent S ( ) AP Technical U ser G ( . . )…roup www isug com• A ’ mericas S AP U ’ sers G ( . . )…roup www asug com• UK ( ) and Europe S ( ) AP Database and Technology products U serG ( . . )…roup www uksug com

• SA P Australian U ser G ( . . . )…roup www saug com au• & - ( , , – SAP TechED d code technologists engineers developers

. .www sapteched co )…m

, …Sybase is dead long live SAP Sybase• …Most user groups are heavily SAP oriented• / - - …I UK SUG leads the way with ex Sybase technologies

3 [30]


: Ecce homo : Ecce homo . Mr Andrew. Mr AndrewMelkonyanMelkonyan

Over 15 years working with Sybase( / )ASE RS

… … … … Developer DBA Lead DBA Team Leader…Consultant

- Ex Ness Employee - … Official Sybase Products Re seller in Israel Responsible

( - )…for local case management via case express

Passionate for Sybase :// . . …http andrewmeph wordpress com :// . . / / . . . .http scn sap com people andrew melkonyan

4 [30]


( . . ) . ( . & Safety Zone 12 5 x vs Futuristic Dreams 15 7)…beyond

Migration to 15.7 – Motivation

15.7

T5-4

PRDR

IMDRRDR15.7 T5-4

3000+ TPS

PROD

REPIMDB

DRP

64 bit

MS2

M’

’RM R

ETL 32 bit

12.5.4

M5

M51212

12.5.4 M5

1000 TPS

PROD

DRP

PRDR

32 bitMS2

REPDRREP

MS1

5 [30]


Migration to 15.7 – 2 case studies

In the past 2 years I witnessed two reasonably large production ASE servers failing to migrate from ASE 12.5.x to the pre ASE 15.7 releases of ASE 15:

One upgrade (Customer#1) was performed on an HP Integrity BL890c i2 host with 8 Intel(R) Itanium(R) Processor 9340s (1.6 GHz), 32 logical processors (4 per socket), 256 GB RAM – 24 Engine ASE, HPUX.The other upgrade (Customer#2) was performed on an M5000 server with 8 SPARC VII Processors (2.4 GHz), 32 logical processors (4 per socket), 128 GB RAM – 16 Engine ASE, Solaris.

Disproportionally high degree of engine utilization coupled with significant drop in throughput turned each migration attempt into a failure.

6 [30]


Migration to 15.5 – Cust#1 Profile

ASE 12.5.3: Runs about 300 transactions per second with an average of 25% engine utilization. The system runs about 600 procedure requests and 1700 statements per second (Total Rows Affected: ~6.5K, Total Index Scans ~72K, Total Lock Requests ~300K, 1.4 M bytes received/sent per second).

Customer #1: an OLTP server (12.5.3) accessed by a mixture of clients – mostly Power Builder CTLIB software.

7 [30]


Migration to 15.5 – Cust#1

ASE 15.5: Run about 100 transactions per second with an average of 30% Engine utilization. The system run ~600 procedure and ~500 statement requests per second. No apparent problems with ASE configuration. No apparent reasons for slowing down. TF753 did not help. Migration was aborted.

Early during migration ASE started to exhibit higher than expected engine utilization while the throughput sunk.

8 [30]


Migration to 15.5 – Cust#1, TSESybase TS initial direction: procedure cache configured too small (12 GB for 24 Engine ASE) & statement cache configured too large (2 GB). The two together were thought to result in high spinlock contention on ASE resources (SSQLCACHE and RPROCMGR spinlocks).

9 [30]


Migration to 15.5 – Cust#2 Profile # : ( . . ) – , Customer 2 an OLTP server 12 5 4 accessed by a mixture of clients JDBC BDE and native

.CTLIB software

ASE 12.5.4: Runs about 900 transactions per second with an average of 40% engine utilization. The system runs approximately 1400 procedure and 800 statement requests per second (Total Rows Affected: ~15K, Total Index Scans ~65K, Total Lock Requests ~250K, 1 M bytes received/sent per second).

10 [30]


Migration to 15.5 – Cust#2Early during migration ASE started to exhibit extremely high engine utilization while the throughput sunk.

ASE 15.[0|5]: Run about 200 transactions per second with an average of 95% Engine utilization. The system run ~650 procedure requests per second. Again, no apparent problems with ASE configuration. No apparent reasons to slow down. TF753 did not help. Migration aborted – twice (massive code review in-between).

11 [30]


Migration to 15.5 – Cust#2,TSESybase TS initial direction: procedure cache fragmentation (4 GB for 16 engine ASE). In the aftermath of work done for Customer #1 it has become clear that here too the problem is around RPROCMGR (visible neither in regular sp_sysmon invocation nor through MDAs – at that time).

12 [30]


Migration to 15.x – AftermathFollowing the migration, new simulators were written by customers teams in order to reproduce failed migration better. None of the customers succeeded to reproduce comparable throughput drop.Only under extreme stress the same degree of spinlock contention or drop in throughput started to surface.Narrowing down the checks, it was found that what causes a high degree of contention is running a high volume of prepared statements simultaneously.

13 [30]


Migration to 15.x – AftermathIn order to deconstruct migration failures we had to:

1.Learn the difference in impact of prepared statement on ASE 12.5.x versus ASE 15.x correspondingly.

2.See what is so peculiar about these customers ASE environment from the prepared statement perspective.

3.Learn the right way to handle a high influx of prepared statement calls with the tools available at the DBMS side.

4.

14 [30]


To Prep or not to Prep, is it a question?When application code uses prepared statement API both client host and DBMS server host prepare internal memory structures designed for subsequent reuse. For ASE’s this structure is a lightweight procedure – or LWP.

Depending on client connection settings ASE may receive and handle:

A.Fully prepared statements (sent via TDS_DYNP).B.Partially prepared statements (sent via TDS_LANG).

The footprint of each on ASE is different. When these structures are created without the purpose of being reused and sent en-masse to ASE they may cause substantial damage.

The distribution of prepared statement types may be inspected either with dbcc cis trace flags or in monSysSQLText (DYNPs are identified as “DYNAMIC_SQL... CREATE PROC…”).

15 [30]


DYNP impact on ASE 15.xConsider the table below:

We use three JDBC clients running (fully) prepared statements in a loop. Instead of reusing them in client code, we just create them and drop right after being used once. All the 15.x versions prior to 15.7 produce significant spinlock contention and the throughput drops. Only the 15.7 with the “streamline dynamic SQL” option turned on fixes the issue(the last column: note the change in procedure removal and statement reuse).

For ASE 12.5.x this was not an issue at all.

16 [30]


DYNPs are better StreamlinedGiven the impact of fully prepared statements on ASE 15.x, the following recommendations apply for application hitting ASE at high rate with fully prepared statements:

If the application layer mustuse prepared statement semantics and there is a high volume of fully prepared statements hitting ASE (generated and dropped at once rather than reused), run ASE 15.7 ESD#1 or later and turn the “streamlined dynamic SQL” option on.

Earlier versions of ASE 15.x cannot handle high rate of fully prepared statements well. For earlier ASE 15.x versions dynamic prepared option must be turned off on driver/connection level (e.g. DYNAMIC_PREPARE = false for JDBC/ODBC client).

17 [30]


Non-DYNP impact on ASE 15.xWith the statement cache sized properly ASE 15.x handles load generated by partially prepared statements much better than ASE 12.5.x.

The problem arises only when client code generates high volume of unique SQL statements forced on the application layer into prepared statement API – again generated and dropped rather than being reused. In this case, statement cache becomes inefficient. Statements come in and out of statement cache at high rate causing high statement cache turnover. Throughput goes down and engine utilization goes up.

In fact, in this case fully and partially prepared statements as well as regular callable statements have the same negative impact.

18 [30]


Non-DYNP impact on ASE 15.xConsider the table below:

We use three JDBC clients running (partially) prepared (or callable) statements in a loop – again with no reuse: create->execute once->free. We generate unique SQL statements. Statement cache turnover go up. ASE 15.5 may handle this situation only if the “bad” unique code is executed with statement cache turned off. ASE 15.7 handles the situation gracefully.

For ASE 12.5.x this was not an issue at all.

19 [30]


Statement Cache WondersStatement cache management in 15.7 has been improved to handle reuse of statements much more effectively. Still, flooding ASE with unique statements is risky:

If the application layer must use prepared statement logic or there is a high volume of unique statements streamed into ASE (either due to bad coding or third party application components – e.g. data-window), ASE 15.7 ESD#1 or later will in most cases deal with it gracefully. However: if the statement cache will get over-flooded by statements contention will arise to the degree of making ASE inoperable. In this case it is better to cut them off statement cache altogether.

Earlier ASE 15.x versions cannot handle situation unless this stream of unique code hitting the statement cache is isolated on the connection level and the statement cache is turned off (e.g., set statement_cache off in login trigger).

20 [30]


Cust#1: Migration Fiasco RevisitedLet’s inspect prepared statement situation in 12.5.3 for Customer#1.

We may see here that there is a medium rate of procedure requests. However, there is almost no procedure removals. Client application here has little or no fully prepared statement calls (confirmed through monSysSQLText inspection).

In addition, we see ~2000 statement requests per second.

21 [30]


We have experimented with statement cache size during simulation sessions and found that the greater the cache the lower is engine utilization – all the way up to 2GB statement cache. So 2GB statement cache by itself did not constitute a problem here (compare throughput below vs. migration data).

Cust#1: Simulation & SC size

22 [30]


Cust#1: Potential IssueThe simulation gave us a chance to inspect statement cache distribution. Cache buckets were found to contain 1 to ~2000 hashed statements in very uneven distribution.

Below is a sample statement from a sc bucket:

What we see here is that this is the same SQL which is slightly modified by the PB client data-window component.

23 [30]


Cust#1: Migration to 15.7Customer#1 used a lot of unique statements. Versions of ASE prior to 15.7 could not handle this influx with SC enabled.

For 15.7 Migration to succeed, we had to monitor SC influx rate closely. High rate of statement influx is more detrimental to ASE 15.x than it has been back in the days of 12.5.4.

During the final migration tests it has been discovered that not only PB data-window component generated a huge number of large unique statements that land in statement cache, but the same code is run by different logins. Statement residing in SC may be reused ONLY if it is invoked by the same login – unless a specific TF is applied to ASE.

The combination of sizing SC properly + controlling SC influx through forcing ASE to reuse statements across different logins made the migration possible.

24 [30]


Cust#2: Migration Fiasco RevisitedLet’s inspect prepared statement situation in 12.5.4 for the Customer#2.

In contrast with Cust#1, Cust#2 has uses fully prepared statement API extensively. It has also a large number of other statements hitting SC.

25 [30]


Cust#2: Migration to 15.715.7 provided solution for the Customer#2 as well. Streamlined dynamic SQL option allowed fully prepared statements to be reused rather than discarded.

Here too, we had to monitor SC influx rate closely since high rate of SC turnover is detrimental to ASE 15.x.

During the migration it has been discovered that there are certain applications that wake up periodically and flood the SC, bringing about high spinlock contention on SC.

The combination of “streamlined” option and controlling SC influx tightly through turning off statement cache access for particular logins made the migration possible.

26 [30]


Migration to 15.7 – A Happy TwinBoth customers have been successfully migrated to ASE 15.7

Customer#2*:

moved from 12.5.4 to 15.7 in July 2013 with a performance boost of over 100% in TPS. There were initial issues around ASE Procedure Cache which were eventually solved using a combination of ASE settings, different access patterns to ASE Statement Cache for different logins and custom PC monitoring scripts.

Customer#1**:

moved from 12.5.3 to 15.7 in March 2014. In addition to the pressure on Statement Cache generated by the high volume of large statements, it was discovered that the situation was aggravated by the fact that ASE generates separate entries in SC for the same statement run by different logins. Customer has over 1000 logins running the same code. Luckily there is a TF to loosen SSQL-suid link.

* Migration orchestrated and performed under my lead.

** Migration has been performed with my personal involvement on-site.

27 [30]


ASE 15.7 Migration – an AsideIn order to prepare/analyze/monitor migration I’ve had to write quite a number of custom monitoring applications. You may find some of the tools @ https://andrewmeph.wordpress.com

28 [30]

Annual Conference, 2015 (c) 2015 Independent SAP Technical User Group

Questions and Answers Questions and Answers

Annual Conference, 2015 (c) 2015 Independent SAP Technical User Group

Thank You for Attending Thank You for Attending

Please complete your Please complete your session feedback form session feedback form

Sybase ASE 15.7- Two Case Studies of Successful Migration

Technology

Transcript of Sybase ASE 15.7- Two Case Studies of Successful Migration