Top .NET, Java & Web Performance Mistakes - Meetup Jan 2015
-
Upload
andreas-grabner -
Category
Software
-
view
419 -
download
0
Transcript of Top .NET, Java & Web Performance Mistakes - Meetup Jan 2015
1 @Dynatrace
- More on http://blog.dynatrace.com
- Dynatrace Free Trial: http://bit.ly/dttrial
Andreas Grabner - @grabnerandi
Biggest Performance Mistakes (.NET, Java, Web …)
2 @Dynatrace
That’s why I ended up talking about performance
3 @Dynatrace
Nobody wants this …
4 @Dynatrace
Unless you work for Google or Microsoft
5 @Dynatrace
Nor this …
6 @Dynatrace
7 @Dynatrace
As it leads to this …
8 @Dynatrace
The “War Room”
Facebook – December 2012
9 @Dynatrace
And potentially to this …
10 @Dynatrace
11 @Dynatrace
And this …
12 @Dynatrace
13 @Dynatrace
And that’s why Business doesn’t like it either …
14 @Dynatrace
~80% of problems
caused by ~20% patterns
YES we know this
80% Dev Time in Bug Fixing
$60B Defect Costs
BUT
15 @Dynatrace
6 Situations on
WHY this happened,
HOW to avoid it,
METRICS to look at
16 @Dynatrace
17 @Dynatrace
Incorrect Sizing of Pools and
Queues
18 @Dynatrace
Online Banking: Slow Balance Check
1.69s (=101s!) To Check Balance!
87% spent in IIS 600! SQL Executions
19 @Dynatrace
#1 Time really spent in IIS?
Tip: Elapsed Time tells us WHEN a Method was executed!
Finding: Thread 32 in IIS waited 87s to pass control to Thread 30 in ASP.NET
Tip: Thread# gives us insight on Thread Queues / Switches
20 @Dynatrace
#2 What about these SQL Executions?
Finding: EVERY SQL statement is executed on
ITS OWN Connection!
Tip: Look at “GetConnection”
21 @Dynatrace
#2 SQL Executions! continued …
#1: Same SQL is executed 67! times
#2: NO PREPARATIONbecause everything
executed on new Connection
22 @Dynatrace
Lessons Learned!
ASP.NET Worker Thread Pool Sizing!
DB Connection PoolsMore Efficient SQL
23 @Dynatrace
Idle vs. Busy Threads
# SQLs / Request
# GetConnection
24 @Dynatrace
25 @Dynatrace
“Blindly” (Re)use Existing
Components
26 @Dynatrace
Requirement: We need a report
27 @Dynatrace
Using Hibernate results in 4k+ SQL Statements to display 3 items!
Hibernate Executes 4k+ Statements
Individual Execution VERY FAST
But Total SUM takes 6s
28 @Dynatrace
• Proper Sizing Specifications
• Understand the Frameworks you are using
Lessons Learned
29 @Dynatrace
# SQLs / Request
# Same SQL / Request
30 @Dynatrace
31 @Dynatrace
Wrong Tools for the Job
32 @Dynatrace
#1 Banking: Transaction History CSV Download!
Remember: High GC doesn’t necessarily mean that this request has a memory problem – BUT - …
Root Cause: Heavy use of String.Concat to build the CSV output in Memory
Recommendations:#1: Use System.Text.StringBuilder instead of String.Concat#2: Write directly back to OutputStream
Problem: Takes 207s! To download. 87% of Time spent in Garbage
Collection
33 @Dynatrace
#2 Store: Rendering Search ResultProblem: 4.4s to render
result page
Root Cause: Custom RegExLibrary with performance
issues on large strings
34 @Dynatrace
#3 Store: Occasional Performance Spikes
Problem: Occasional Spikes!Never seen in Development
35 @Dynatrace
First Analysis: His own code
36 @Dynatrace
DNS Lookup Issue during Login
37 @Dynatrace
Reverse Lookup by IP fails on these IPs
Root Cause: Issues from Data Center Hoster
38 @Dynatrace
• CSV Export Example
• Don’t build large Strings in Memory with String.Concat
• Write to output stream and use StringBuilder
• Search Result Page
• Don’t implement custom RegEx Library – existing Tools are better
• Test with large results and not just small test data
• Geo Lookup
• Don’t trust your Hosting Provider
• E.g: Use GeoIP Databases for accurate reverse lookups
Lessons Learned – Use Better Tools / Libraries
39 @Dynatrace
Memory Usage
Time Spent in APIs
40 @Dynatrace
41 @Dynatrace
Not every Architect makes good decisions
42 @Dynatrace
• Symptoms
• HTML takes between 60 and 120s to render
• High GC Time
• Assumptions
• Bad GC Tuning
• Probably bad Database Performance as rendering was simple
Project: Online Room Reservation System
43 @Dynatrace
Developers built own monitoring
void roomreservationReport(int roomid)
{
long startTime = System.currentTimeMillis();
Object data = loadDataForRoom(roomid);
long dataLoadTime = System.currentTimeMillis() - startTime;
generateReport(data, roomid);
}
Result:
Avg. Data Load Time: 45s!
DB Tool says:
Avg. SQL Query: <1ms!
44 @Dynatrace
#1: Loading too much data24889! Calls to the
Database API!
High CPU and High Memory Usage to keep all
data in Memory
45 @Dynatrace
#2: On individual connections 12444!individual
connections
Classical N+1 Query Problem
Individual SQL really <1ms
46 @Dynatrace
#3: Putting all data in temp Hashtable
Lots of time spent in
Hashtable.get
Called from their Entity Objects
47 @Dynatrace
• Custom Measuring
• Was impacted by Garbage Collection
• Just measured overall time but not # SQL Executions
• Learn SQL and don’t use Hashtables as Workaround
Lesson Learned
void roomreservationReport(int roomid)
{
long startTime = System.currentTimeMillis();
Object data = loadDataForRoom(roomid);
long dataLoadTime = System.currentTimeMillis() - startTime;
generateReport(data, roomid);
}
48 @Dynatrace
# SQL Executions
# of SAME SQLs
Conn. Acquisition Time
49 @Dynatrace
50 @Dynatrace
#Push without a Plan
51 @Dynatrace
Mobile Landing Page of Super Bowl Ad
434 Resources in total on that page:230 JPEGs, 75 PNGs, 50 GIFs, …
Total size of ~ 20MB
52 @Dynatrace
Fifa.com during Worldcup
http://apmblog.compuware.com/2014/05/21/is-the-fifa-world-cup-website-ready-for-the-tournament/
53 @Dynatrace
# Images
# Redirects
Size of Resources
54 @Dynatrace
55 @Dynatrace
#No “Agile” Deployment
56 @Dynatrace
Load Spike resulted in UnavailabilityAd
on
air
57 @Dynatrace
Alternative: “GoDaddy goes DevOps”
1h before SuperBowl KickOff
1h after Game ended
58 @Dynatrace
# of Domains
Total Size
59 @Dynatrace
Waiting for Bug Reports
60 @Dynatrace
Look behind the scenes
61 @Dynatrace
•# Images
•# Redirects
•Size of Resources
•# SQL Executions
•# of SAME SQLs
•# Items per Page
•# AJAX per Page
Look at these Metrics Before Checking In Code
•Time Spent in API
•# Calls into API
•# Functional Errors
•3rd Party calls
•# of Domains
•Total Size
•…
62 @Dynatrace
Commit Stage• Compile• Execute Unit Test• Code Analysis• Build installers
Automated Acceptance
Testing
Automated Capacity Testing
Manual testing• Key showcases• Exploratory testing Release
Unit & Integration Tests
Functional Tests
Performance TestsProductionMonitoring
Functional Tests
(R)Evolutionize Performance Optimization
63 @Dynatrace
64 @Dynatrace
Example from Web Diagnostics 282! Objects
on that page9.68MB Page Size
8.8s Page Load Time
Most objects are images delivered from your main
domain
Very long Connect time (1.8s) to your CDN
65 @Dynatrace
Example from Server-Side Diagnostics526s to render that
report
1 SQL running
210s!
Lots of time spent in logging to Log4J
Lots of time spent in rendering
66 @Dynatrace
Online Performance Clinics
Every Other week @
bit.ly/onlineperfclinic
67 @Dynatrace
Your Benefits
• Free Performance Review
• Extended Dynatrace License
“Share Your PurePath”
bit.ly/sharepurepathMy Benefits
• More blog material for next year
• Gratification that I could help you
68 @Dynatrace
Questions and/or Demo
Slides: slideshare.net/grabnerandi
Get Tools: bit.ly/dttrial
YouTube Tutorials: bit.ly/dttutorials
Contact Me: [email protected]
Follow Me: @grabnerandi
Read More: blog.dynatrace.com
69 @Dynatrace
Andreas GrabnerDynatrace Developer Advocate
@grabnerandi
http://blog.dynatrace.com