Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through...
Transcript of Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through...
![Page 1: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/1.jpg)
Take Your Oracle WebLogic Applications to The
Next Level with Oracle Enterprise Manager 12c
Mojahedul Hoque Abul Hasanat CTO, Therap Services
Neelima Bawa Consulting Tech. Lead, SCP, EM, Oracle
![Page 2: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/2.jpg)
Agenda
• Background of Therap Services
• The problem
• Application Performance Management
• Quick description of Oracle’s APM offering
• How OEM and RUEI helped us
• Actual scenarios
• The future
• Tips
• Q&A
Therap Services - OOW 2013 2
![Page 3: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/3.jpg)
Therap Services, LLC
• Documentation and Communication Software for MR/DD
• EHR for the DD industry is the closest for describing us
• Niche segment in the health sector
• Improve quality of life for people with DD by improving efficiency of delivery through communication
• SaaS business model
• 150K+ active users
• 1000+ providers in 48 states
• State customers
• Extensive usage for DD in DHS ND and DHHS NE
• 150+ employees
• Based in CT, dev center in Bangladesh
• http://www.therapservices.net
Therap Services - OOW 2013 3
![Page 4: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/4.jpg)
The Application
• The application is our business
• 1M+ lines of code
• 60+ modules
• 1M+ sustained HTTP requests/hour
• 30K+ peak requests/minute
• 6000+ concurrent users
• Based on JEE and the Spring Framework
• Hibernate
• Seam
• GRAILS
Therap Services - OOW 2013 4
![Page 5: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/5.jpg)
Delivery Platform
• 2 identical sites in two states
• Primary hosts (per site):
• 4 WebLogic application servers in cluster
• 1 Memory based data server (in-house, java)
• 1 Oracle database server
• 1 NetApp storage (SAN)
• 1 F5 Load balancer
• Supporting hosts
• Use Dyn for site high availability
• Data replication with Oracle Golden Gate
Therap Services - OOW 2013 5
![Page 6: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/6.jpg)
What Matters
• Availability
• Application is used 24x7
• Application use is critical to the business of our customers
• Performance
• A user needs to spend as little time as possible in our application
• Most users use it daily, multiple times
• Data integrity
• Fast development turnaround
Therap Services - OOW 2013 6
![Page 7: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/7.jpg)
Evolution of Therap
• Improved testing
• Formal code review
• Improved processes
• Removed repeating problems
• Now all problems we face are new
• Acquired large customers
• Availability and performance have become critical factors
Therap Services - OOW 2013 7
![Page 8: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/8.jpg)
Before OEM & RUEI
• Heavy use of logging
• Nagios
• Cacti
• kill -3
Therap Services - OOW 2013 8
![Page 9: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/9.jpg)
The Problem
• Diagnosis of performance issues
• Has become much harder with the growing system
• Application availability
• With a larger customer base, uptime has become a major factor
• Complexity of the system increases difficulty
• Limits of logging
• Works for known unknowns
• Need infrastructure to visualize and store historic data
• Limits of OS based monitoring
• Limited metrics
• Limits of simple JMX monitoring
Therap Services - OOW 2013 9
![Page 10: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/10.jpg)
Application Performance Management
• Deep insight into running application
• Profiling at runtime
• Some bottlenecks are only visible at runtime
• Historic data
• Invaluable for preventing performance regression
Therap Services - OOW 2013 10
![Page 11: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/11.jpg)
Which Vendor?
• We were moving from JBoss to WebLogic
• JVM Diagnostics
• Extensive WebLogic metrics
• Probably the best database diagnostics
• Our team was already familiar with OEM
• Deep integration with the database
• Integration of JVMD and app server metrics
• Expect better support from Oracle
Therap Services - OOW 2013 11
![Page 12: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/12.jpg)
Oracle’s APM
• Oracle Enterprise Manager 12c
• WebLogic Metrics
• Middleware Diagnostics Advisor
• JVM Diagnostics
• Configuration Management
• Incident Management
• Lifecycle Management
• Oracle Real User Experience Insight
• Oracle Business Transaction Management
Therap Services - OOW 2013 12
![Page 13: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/13.jpg)
WebLogic Metrics
• Pro-active monitoring
• Helps us in avoiding downtime
• Correlation between various metrics
• Middleware Diagnostics Advisor
Therap Services - OOW 2013 13
![Page 14: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/14.jpg)
JVM Diagnostics
• Deep insight into the JVM
• Invaluable for understanding application performance issues
• Helped us in identifying log4j bottleneck
• Early identification of performance problems
Therap Services - OOW 2013 14
![Page 15: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/15.jpg)
Oracle Real User Experience Insight
• Measure performance seen from the customer end
• Detect performance regression
• Enables shorter release cycles
• Quick and real feedback for performance tuning
operations
Therap Services - OOW 2013 15
![Page 16: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/16.jpg)
Our Journey
Therap Services - OOW 2013 16
![Page 17: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/17.jpg)
Timeline
• Evaluation of various vendors – July 2012
• Purchase of WebLogic, OEM, RUEI – Nov 2012
• Start JBoss to WebLogic migration
• Start building expertise on OEM
• Start using OEM in test environment
• Fix problems found through OEM, JVMD
• Production deployment – Mar 2013
Therap Services - OOW 2013 17
![Page 18: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/18.jpg)
How OEM and RUEI helped us
Therap Services - OOW 2013 18
![Page 19: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/19.jpg)
The log4j bottleneck
• During load testing, we could not increase load beyond a
certain point
• CPU load was low
• JVMD showed us something that we could hardly believe
• Many threads were contending for lock for writing to the
log file
• The contention only shows up at high loads
• Used JVMD heavily to find the best logging backend and
the best configuration
Therap Services - OOW 2013 19
![Page 20: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/20.jpg)
log4j…
Therap Services - OOW 2013 20
![Page 21: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/21.jpg)
log4j…
Therap Services - OOW 2013 21
![Page 22: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/22.jpg)
log4j...
Therap Services - OOW 2013 22
![Page 23: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/23.jpg)
EJB Transaction Optimization
• Noticed abnormally high number of bean transaction
commits
• We had forgotten to optimize some frequently used EJB
• Read-only methods do not need to be transactional
Therap Services - OOW 2013 23
![Page 24: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/24.jpg)
EJB Transaction Optimization…
Therap Services - OOW 2013 24
![Page 25: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/25.jpg)
Unexpected Top Method
• Noticed a JMS listener in the top method list
• In production!
• Did not show up during synthetic load testing
• We forgot to add a “message selector” on the listener
Therap Services - OOW 2013 25
![Page 26: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/26.jpg)
Top Method…
Therap Services - OOW 2013 26
![Page 27: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/27.jpg)
The MDA Catch
• MDA reported an unexpected “The EJB is taking too long
to execute”
• Related method was showing in the top methods list
• There were extraneous calls to the EJB
• The method did not need to be in an EJB
Therap Services - OOW 2013 27
![Page 28: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/28.jpg)
The MDA Catch…
Therap Services - OOW 2013 28
![Page 29: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/29.jpg)
The MDA Catch…
Therap Services - OOW 2013 29
![Page 30: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/30.jpg)
The Slow Library
• A library call for producing JSON showed on the top
method list
• JSON is needed for AJAX
• It was totally unexpected
• The library was old and inefficient
• Replaced it with a newer and more efficient library
Therap Services - OOW 2013 30
![Page 31: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/31.jpg)
The Slow Library…
Therap Services - OOW 2013 31
![Page 32: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/32.jpg)
The Slow Library…
Therap Services - OOW 2013 32
![Page 33: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/33.jpg)
The Slow Library…
Therap Services - OOW 2013 33
![Page 34: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/34.jpg)
In-efficient Network Write
• Initially discovered in production through JVMD
• There were instances of high network waits
• Methods a certain module in the application showed up in
the top list during the high network wait periods
• Discovered a 3 level loop that writes data
• Further inspection through JProfiler confirmed it
Therap Services - OOW 2013 34
![Page 35: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/35.jpg)
In-efficient Network Write…
Therap Services - OOW 2013 35
![Page 36: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/36.jpg)
In-efficient Network Write…
Therap Services - OOW 2013 36
![Page 37: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/37.jpg)
Automatic Thread Snapshots
• Previously, relied on kill -3
• Manual, missed dumps at crucial moments
• Now, JVMD takes thread snapshots when an abnormal
thread state is reached on any WebLogic server
• Combined with auto-restart from WebLogic, eliminated
unplanned downtime
Therap Services - OOW 2013 37
![Page 38: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/38.jpg)
Compliance
• Helps us identify patches needed for:
• WebLogic
• Oracle Database
• OEM
• Downtime log
• Useful for tracking operations improvement
Therap Services - OOW 2013 38
![Page 39: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/39.jpg)
The JDK Upgrade
• Upgraded JDK from 1.6.0_29 to 1.6.0_45
• At that time, we have not brought RUEI into our regular
operations process
• Started getting slowness complaints after a few days
• A look into RUEI instantly revealed performance
regression
• Downgrading the JDK fixed the regression completely
Therap Services - OOW 2013 39
![Page 40: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/40.jpg)
The JDK Upgrade…
Therap Services - OOW 2013 40
![Page 41: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/41.jpg)
The JDK Upgrade…
• We could not reproduce the performance regression in lab
environment
• We now have a new procedure to upgrade JDK
• Do normal tests and load tests as before
• In production, upgrade the JDK of one server only
• Wait a few days
• Compare performance
• Decide whether to upgrade or rollback
Therap Services - OOW 2013 41
![Page 42: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/42.jpg)
The Results
• 1 unplanned downtime in the last 4 months!
• Improving DevOps culture
• Foster collaboration between dev, ops and DBA
• No more fighting between dev and DBA
• With RUEI, even the business team joins the fun
Therap Services - OOW 2013 42
![Page 43: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/43.jpg)
Most
Importantly
I can do this!
Therap Services - OOW 2013 43
![Page 44: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/44.jpg)
Future with OEM and RUEI
Therap Services - OOW 2013 44
![Page 45: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/45.jpg)
Future
• Involve more people
• Use KPI in RUEI
• Integrate KPI with OEM by defining a Business
Application
• Leverage alerting and incident management in OEM
Therap Services - OOW 2013 45
![Page 46: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/46.jpg)
Challenges
• Steep learning curve
• Took us time to understand the role of BTM and JRF
• Better documentation available now
• Lack of full WebLogic 12c support when we started
• Should be solved in latest release of OEM
• Support has been fanatical!
• Thank you Oracle!
Therap Services - OOW 2013 46
![Page 47: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/47.jpg)
Tips
• Understand what you need
• Any web app with performance requirements needs RUEI
• If you face performance issues, you will need JVMD
• Engage as many people in your team as possible
• Give wide access to many
• The learning curve
• The problems these tools help you with are also complex
Therap Services - OOW 2013 47
![Page 48: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/48.jpg)
Q&A
Therap Services - OOW 2013 48
![Page 49: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/49.jpg)
Other Sessions
• JVM Diagnostics: Java Profiling in Production
Environments [CON9571]
• Thursday 2:00pm – 3:00pm @ Moscone North - 130
Therap Services - OOW 2013 49
![Page 50: Take Your Oracle WebLogic Applications to The Next Level with … · •Further inspection through JProfiler confirmed it Therap Services - OOW 2013 34 In-efficient Network Write](https://reader036.fdocuments.net/reader036/viewer/2022090602/605529cada9549091546490d/html5/thumbnails/50.jpg)
Therap Services - OOW 2013 50
Contact
http://neelimabawa.blogspot.com
@Masum6