The SAP Transaction Model: Know Your Applications...Operational: Run on multiple DBMS products...
Transcript of The SAP Transaction Model: Know Your Applications...Operational: Run on multiple DBMS products...
SYSTEMATIC THOUGHT LEADERSHIP FOR INNOVATIVE BUSINESS
The SAPTransaction Model:
Know YourApplications
Shel Finkelstein, Rainer Brendle, Dean Jacobs,Manfred Hirsch and Ulrich Marquard
SAP Labs{firstname.lastname}@sap.com
© SAP ACM SIGMOD Conference, June, 2008 2
Outline
SAP Applications and DataExamples, Requirements, Properties
SAP Data Management ArchitectureApplication ArchitectureDatabase Execution Model
SAP Scalability and PerformanceApplication ScalabilityBenchmarks
© SAP ACM SIGMOD Conference, June, 2008 3
SAP Applications and Data
© SAP ACM SIGMOD Conference, June, 2008 4
SAP Customer Application Requirements
SAP ERPOn a laptop
SAP for ConsumerProducts5000+ concurrentlyactive users
SAP SCM4.5 million characteristiccombinations & 256 GBmemory in live cache
SAP for Retail1.3+ million salesorder items per daySAP
NetWeaver BICurrently productive:10 TB database(planned 60 TB)
SAP ERP - HCMPayroll calculation for 500,000employees and retirees in 4 hours
SAP for Utilities25 million business partners85 million service and salesorders per year
SAP NetWeaverPortal100,000 concurrent users
© SAP ACM SIGMOD Conference, June, 2008 5
Solution Areas and Data Access
Order ProcessingEnter order with line items; schedule delivery; handle changesRead mostly; most updates are inserts; most updates don’t conflict
Supply Chain Management (SCM)Planning for procurement, sales and demand, production, distribution; controland managementRead mostly, most updates are inserts; most updates don’t conflictBusiness processes handle impacts of updates
Integrated Product and Process Engineering (IPPE)Master data that provide high integration of engineering processes andmanufacturing processesRead mostly; most updates don’t conflictCollaboration, but changes usually involve independent subtrees
© SAP ACM SIGMOD Conference, June, 2008 6
Data Access Patterns for Applications
Applications may be conversational, with multiple conversational userinteraction steps in a transactionMost business data is read-onlyUpdate operations are insert-mostlyMost database transactions are shortFor all updates, conflicts are rare in practicePotential hotspots (e.g., inventory stock; sequence numbers that must beconsecutive for legal reasons) must be addressed
© SAP ACM SIGMOD Conference, June, 2008 7
Data Management Principles
Technical: Maintain integrity of Business Data across appsMultiple solution areas use the same Business Objects
Legal: Handle most updates as inserts of new dataSupport for Governance, Risk and Compliance (GRC) and auditing
Operational: Run on multiple DBMS productsDon’t depend on special properties of any DBMSDB must not be a bottleneck
Programming Model: Message-based consistencyLike normalization and serializability, transactions should only be used whereappropriateUse transactions within a deployment unit; avoid distributed commit acrossdeployment units
© SAP ACM SIGMOD Conference, June, 2008 8
SAP Data Management Architecture
© SAP ACM SIGMOD Conference, June, 2008 9
App
licat
ion
Serv
er DispatcherGateway
SharedMemory
andBuffers
Basic Application Server Architecture
DialogHandler
User User User User
DialogHandler
DialogHandler
DialogHandler
UserUser
Database
© SAP ACM SIGMOD Conference, June, 2008 10
Screen
COMMIT WORK
Screen Screen
Two-Stage Execution Model
Read from databaseAcquire locksQueue descriptors of update operationsNo writesRollback possible
Perform update operations in deadlock-free orderSingle message database transactionNo rollbacksRelease locks upon completion
Dialog Stage Update Stage
Rediscovered in H-Store, where it is used to eliminate the undo log
© SAP ACM SIGMOD Conference, June, 2008 11
Lock Manager
The ProblemMany applications have multiple screens with long user think timesMaintaining open database connections would hurt performanceImplicit Pessimistic and Optimistic Concurrency Control have disadvantages
E.g., scalability; rollback after user completes work
The SolutionA business-level logical lock manager (Enqueue Manager) outside the database
Business-level locking is very flexible approachNo danger of lock escalation or index locking in the database
Locks are explicitly requested; a “contract” between applications, using frameworkLock types: shared, intention, exclusiveRather than getting blocked, applications get errors that they can handleCan request that individual locks be held after commit
© SAP ACM SIGMOD Conference, June, 2008 12
Database
App
licat
ion
Serv
er 2
App
licat
ion
Serv
er 1
DialogHandler
DialogHandler
DialogHandler
Dispatcher Dispatcher
LockManager
User User User
MessageServer
LockTable
SharedMemory
1
2
34
56
Steps in Two-Stage Execution
Gateway
© SAP ACM SIGMOD Conference, June, 2008 13
Asynchronous Updates
The packaging of rollback-free queued updates into a bundle allows them to beexecuted after the transaction completes (write behind)Doing so allows the user to continue on to other work soonerThe locks are released after the updates have been performed so subsequent writeoperations never get stale dataAppropriate when subsequent operations go on to other work, so they never readstale dataThe updates are performed by an Update Handler, which reduces the number ofhandlers that do writesThe updates from several transactions may be delayed indefinitely until acoordinator picks them up and performs them all together
Example: A background process that updates statistical data
© SAP ACM SIGMOD Conference, June, 2008 14
Screen
SAVE COMMIT WORK
Screen Screen Screen
Three-Stage Execution Model
Dialog Stage Update Stage Write StageDB COMMIT
Read from databaseAcquire logical locksNo writesRollback possible
Read update operations fromdatabase and perform them indeadlock-free orderSingle message databasetransactionNo rollbacksRelease locks uponcompletion
Write descriptors ofupdate operations tothe databaseNo rollbacks
© SAP ACM SIGMOD Conference, June, 2008 15
Database Manager
Database
App
licat
ion
Serv
er 2
App
licat
ion
Serv
er 1
DialogHandler
DialogHandler
UpdateHandler
UpdateHandler
Dispatcher Dispatcher
LockManager
Application data Data to be updated
MessageServer
LockTable
1
2 3
4
56
Steps in Three-Stage Execution
7
Gateway
User User User
© SAP ACM SIGMOD Conference, June, 2008 16
Comparison of the Models
Two-StageUpdates always visible immediatelyafter transaction completesLower database load for singletransactionSynchronous execution of updateshelps to build process chains
Three-StageUser can continue sooner, since returnto user as soon as update descriptor isin DBLower database load when multipletransactions are boxcarred togetherShorter locking times and/or log forsequence number assignment andother hotspot operations
© SAP ACM SIGMOD Conference, June, 2008 17
What Does SAP Use Database For?
Store/Retrieve dataTransaction commit
Internal locks are held very brieflyCanonical ordering of locked resources avoids deadlocksRollbacks for internal DB reasons only (can retry)
Operational utilities, including disaster recoveryIndexing
Internal indexesBut external indexes are maintained in main memory, kept consistent by post-commit processing
© SAP ACM SIGMOD Conference, June, 2008 18
Hot Spots: Sequence Numbers
Many ERP and Financial applications require that sequence numbers beassigned to transactions or documentsNumber allocation requirements depend on legal requirements
Weaker requirements permit implementations that scale betterStrictest guarantee: chronological assignment with no gaps
Number assignment must occur inside the transaction and be rolled back incase of abortForces serialized access to the next available number, which reducesconcurrencyEssential to hold the (database) lock for as short a time as possibleTwo-stage model: lock held during updates and writesThree-stage model: lock held during updates but not writes
Weaker guarantee: Gaps must be logged for auditingThree-stage model: pending updates provide a log
© SAP ACM SIGMOD Conference, June, 2008 19
Hot Spots: Aggregates
Applications often must update shared aggregate data,as well as exclusive data
Order Processing focuses on a specific OrderProcessing an Order also affects aggregate data
Total expense for Contract associated with that OrderStock remaining for Products sold in that Order
Transaction execution strategy is designed for scalabilityAcquire exclusive logical lock on specific OrderAfter Commit, perform updates on OrderAfter Commit, perform (commutative) delta update on aggregatedataSAP also can perform merge update, e.g., for collaborativeapplications such as Integrated Product and Process Engineering
© SAP ACM SIGMOD Conference, June, 2008 20
Reviewing SAP Transaction Execution Stages
During transactionLogically lock data that needs to be changedRead Business Data into an application contextBefore commit, perform updates only on the application context
After commitStage 1: Application does logical consistency checks, then queuesupdate descriptor
Due to logical locks, there can be no conflicts with concurrent transactionsStage 2: Transaction’s queued update descriptor is written to DB insingle message transaction (with boxcarring)
Return to user after its single message transaction completesStage 3: Updates to Business Data in DB are performed
Updates are executed in canonical order to avoid deadlocksDelta updates to aggregated executed last, with coarsest aggregationsexecuted last of all
© SAP ACM SIGMOD Conference, June, 2008 21
Scalability and Performance
© SAP ACM SIGMOD Conference, June, 2008 22
Performance typically defined by two metricsResponse time - speed of task completion (latency)
System throughput - amount of work done in a givenamount of time
Good performance if metrics match customerneeds or Service Level Agreements
Most important Key Performance Indicators (KPIs)CPU time consumedPeak memory usedDisk spaceNetwork load
Proven and predictable scalabilityPrecondition for translating business requirements intohardware configurations (sizing) with linear growthMinimize load on central resources (e.g., DB)No system wait times, no bottlenecks
What Exactly Is Good Performance?
Application ServerApplication Server(s)
Database
Web Server(s)
ResponseTime = t[i]
ResponseTime
t[1]
t[2]t[3]
t[4]
t[5]
t[6]t[i]
© SAP ACM SIGMOD Conference, June, 2008 23
Vertical & Horizontal Scalability(Cross-Tier Scaling and ScaleOut)
From one tier (laptop demo) tomulti-tier systems
PresentationMore than 150K very activeusers (fat transactions)
InternetMore than tens of thousands ofhits /sec10 servers at one of our largestcustomers
Application and IntegrationOver 150 app serversconnected to DB
DatabaseMore than 100 CPUs; morethan 10 TB DB sizeScalability through SMPdatabase serverScalability through parallelpartitioned databases
©SAP 24
Architecture of SAP Solutions
CentralServicesMessage Services
Application Server
ABAP
UserUser
HTTP Dispatcher HTTP Dispatcher
Workprocess
SAP Web Dispatcher
DBMS
I/O sub system
HTTP, SOAPHTTP
RFC
DB WPDB WP
RFC
Application Server
ABAPABAP
Dispatcher
Workprocess
Enqueue Services (Logical Locks)
ABAP Dispatcher
Sharedmemory &buffers
Sharedmemory &buffers
© SAP ACM SIGMOD Conference, June, 2008 25
Simulation of User Interaction Load
The Sales and Distribution(SD)benchmark is the mostcommon SAP benchmark
Core business process (selling, shipping, delivery)High scalability demands from customersFocuses on CPU utilizationEasy to installEasy to use"Well understood and accepted"Highly credible and influential high-end OLTPbusiness application benchmarkCertified benchmarks run since 1995
© SAP ACM SIGMOD Conference, June, 2008 26
0. Logon 10. Choose customer order1. Main screen 11. Change delivery info for order
2. Create customer order 12. Posts goods issue (schedule delivery)3. Enter order info 13. List orders4. Enter details for 5 items 14. Choose set of orders
5. Save 15. Create invoice6. Create a delivery 16. Save
7. Enter delivery info 17. End8. Save 18. Confirm logoff
9. Display customer orderSteps 2 to 16 are repeated n times; each 15 step run takes at least 150 seconds due to 10 secondthink time between steps.
Business aspect:One run (steps 2 to 16) corresponds to the selling of 5 line items.
User Interaction Steps –SAP SD Standard Application Benchmark
© SAP ACM SIGMOD Conference, June, 2008 27
2005
Proven Scalability Using SD Benchmarks
SD Benchmark, Two-Tier Internet Architecture: Scale with Hardware1996 1997 1998 1999 2000 2001 2002 2003 2004
http://www.sap.com/benchmark
21,000
6001,410 1,708
3,0004,100
7,8008,000
13,000
20,000
2006
0
5,000
10,000
15,000
20,000
25,000
30,000
35,000
23,456
30,000
35,4002007 2008
Number of SDBenchmark users
© SAP ACM SIGMOD Conference, June, 2008 28
Key Figures for the 168,300 UserThree-Tier SD Benchmark (2005)
8,392 DB transactions per second (commits)
323 MB written to disk per second on average
611 GB data written in one hour (~ 170 MB per second on average)
1,157,000 network packets per second on average, with packet size of 512 bytes
469,091 SQL statements per second
2,240 GB database disk space
5,632 SAP (as opposed to DB) transactions / second
14,081 Screen changes / second
Database used a 32-processors SMP server
R/3 was running on 712 processors
16,896,670 fully business processed order line items per hour ( 4,694 per second )
© SAP ACM SIGMOD Conference, June, 2008 29
Conclusion
Data and transaction management should be based on applications andtheir data utilization
Traditional DBMS are not ideally suited for thisSAP has had its own external data and transaction layer since 1985– This works; DB hasn’t been a bottleneck for us
Get the programming model right for your appsExplicit locking for applications, based on frameworkUse distributed commit only within a deployment unit– Across deployment units, use message/process-based (loose) consistency
Know your applicationsOne size hasn’t fit all, for SAP, for other companies, and for our customers
© SAP ACM SIGMOD Conference, June, 2008 30
Further Information
Public ReferencesSAP Standard Application BenchmarksSAP Standard Benchmark Certification 2005021Rudiger Buck-Emden, “The SAP R/3 System: A
Client/Server Technology”, Addison-Wesley, 1996Horst Keller, “The Official ABAP Reference”, SAP Press,
2005Gerhard Knolmayer, “Supply Chain Management Based on
SAP Systems: Processes and Architecture”, Springer, 2002Burkhard Neidecker-Lutz, Frying your infrastructure: Are
transactions really useful in a distributed realtimeenterprise system?, HPTS 2005[H-Store] Mike Stonebraker, The End of an Architectural
Era (It's Time for a Complete Rewrite), VLDB 2007
© SAP ACM SIGMOD Conference, June, 2008 31
Further Information
SAP Service Marketplace (SAP customers)http://service.sap.com/benchmarkhttp://service.sap.com/performancehttp://service.sap.com/sizinghttp://service.sap.com/quicksizing
© SAP ACM SIGMOD Conference, June, 2008 32
Copyright 2008 SAP AGAll rights reservedNo part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changedwithout prior notice.
Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors.
SAP, R/3, xApps, xApp, SAP NetWeaver, Duet, SAP Business ByDesign, ByDesign, PartnerEdge and other SAP products and services mentioned herein as well as their respective logos aretrademarks or registered trademarks of SAP AG in Germany and in several other countries all over the world. All other product and service names mentioned and associated logos displayedare the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary.
The information in this document is proprietary to SAP. This document is a preliminary version and not subject to your license agreement or any other agreement with SAP. This documentcontains only intended strategies, developments, and functionalities of the SAP® product and is not intended to be binding upon SAP to any particular course of business, product strategy,and/or development. SAP assumes no responsibility for errors or omissions in this document. SAP does not warrant the accuracy or completeness of the information, text, graphics, links, orother items contained within this material. This document is provided without a warranty of any kind, either express or implied, including but not limited to the implied warranties ofmerchantability, fitness for a particular purpose, or non-infringement.
SAP shall have no liability for damages of any kind including without limitation direct, special, indirect, or consequential damages that may result from the use of these materials. This limitationshall not apply in cases of intent or gross negligence.
The statutory liability for personal injury and defective products is not affected. SAP has no control over the information that you may access through the use of hot links contained in thesematerials and does not endorse your use of third-party Web pages nor provide any warranty whatsoever relating to third-party Web pages
Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durchSAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden.
Einige von der SAP AG und deren Vertriebspartnern vertriebene Softwareprodukte können Softwarekomponenten umfassen, die Eigentum anderer Softwarehersteller sind.
SAP, R/3, xApps, xApp, SAP NetWeaver, Duet, SAP Business ByDesign, ByDesign, PartnerEdge und andere in diesem Dokument erwähnte SAP-Produkte und Services sowie diedazugehörigen Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und in mehreren anderen Ländern weltweit. Alle anderen in diesem Dokument erwähnten Namenvon Produkten und Services sowie die damit verbundenen Firmenlogos sind Marken der jeweiligen Unternehmen. Die Angaben im Text sind unverbindlich und dienen lediglich zuInformationszwecken. Produkte können länderspezifische Unterschiede aufweisen.
Die in diesem Dokument enthaltenen Informationen sind Eigentum von SAP. Dieses Dokument ist eine Vorabversion und unterliegt nicht Ihrer Lizenzvereinbarung oder einer anderenVereinbarung mit SAP. Dieses Dokument enthält nur vorgesehene Strategien, Entwicklungen und Funktionen des SAP®-Produkts und ist für SAP nicht bindend, einen bestimmtenGeschäftsweg, eine Produktstrategie bzw. -entwicklung einzuschlagen. SAP übernimmt keine Verantwortung für Fehler oder Auslassungen in diesen Materialien. SAP garantiert nicht dieRichtigkeit oder Vollständigkeit der Informationen, Texte, Grafiken, Links oder anderer in diesen Materialien enthaltenen Elemente. Diese Publikation wird ohne jegliche Gewähr, wederausdrücklich noch stillschweigend, bereitgestellt. Dies gilt u. a., aber nicht ausschließlich, hinsichtlich der Gewährleistung der Marktgängigkeit und der Eignung für einen bestimmten Zwecksowie für die Gewährleistung der Nichtverletzung geltenden Rechts.
SAP übernimmt keine Haftung für Schäden jeglicher Art, einschließlich und ohne Einschränkung für direkte, spezielle, indirekte oder Folgeschäden im Zusammenhang mit der Verwendungdieser Unterlagen. Diese Einschränkung gilt nicht bei Vorsatz oder grober Fahrlässigkeit.
Die gesetzliche Haftung bei Personenschäden oder die Produkthaftung bleibt unberührt. Die Informationen, auf die Sie möglicherweise über die in diesem Material enthaltenen Hotlinkszugreifen, unterliegen nicht dem Einfluss von SAP, und SAP unterstützt nicht die Nutzung von Internetseiten Dritter durch Sie und gibt keinerlei Gewährleistungen oder Zusagen überInternetseiten Dritter ab.
Alle Rechte vorbehalten.
© SAP ACM SIGMOD Conference, June, 2008 33
The SAP Transaction Model:Know Your Applications
Shel Finkelstein, Rainer Brendle, Dean Jacobs,Manfred Hirsch and Ulrich Marquard
SAP Labs{firstname.lastname}@sap.com
© SAP ACM SIGMOD Conference, June, 2008 34