Haproxy web performance

Powering Your Uptime

HAProxy Technologies

HAProxy and web performance

EMEA Headquarters3, rue du petit RobinsonZAC des Metz78350 Jouy en JosasFrancehttp://www.haproxy.com

Powering Your UptimeHAProxy Technologies - HAProxy and web performance Private document

http://www.haproxy.com

HAProxy and web performance

Agenda

• Disclaimer

• About this presentation

• How HAProxy works

• Watching live traffic

• Configuring the stats sockets

• Demo 1: using the stats sockets

• Logging

• Information logged

• Demo 2: exploiting HAProxy logs

• Queueing in HAProxy demystified

• What about the performance metric

• Demo 3: measuring performance

• Demo 4: Boosting performance with HAProxy

• Conclusion


ContactFor Any questions related to the slides presented here, feel free to contact the author: Baptiste Assmann<bassmann (at) haproxy (dot) com>

Keep updated with HAProxy Technologies:

• http://www.haproxy.com/

• http://blog.haproxy.com/

• http://twitter.com/haproxy_tech/

• http://github.com/haproxytech/

• http://www.slideshare.net/haproxytech/

For any question related to our products and services, send a mail to <info (at) haproxy (dot) com>


http://www.haproxy.com/

http://blog.haproxy.com/

http://twitter.com/haproxy_tech/

http://github.com/haproxytech/

http://www.slideshare.net/haproxytech/

DisclaimerDon't apply blindly all the tips presented here

• Each application is different and deserves a deep understanding before applying / changing aconfiguration

• Each workload, network stack may interfere

• Don't hesitate to ask the experts on HAProxy:

• Community user: [email protected] (no registration required)

• HAProxy Technologies customer: [email protected]

• Wear 3D glasses to enjoy the pictures of this presentation

• This is a non-exhaustive presentation

• 19 slides and Pi+1 (4.14) live demos!


mailto:[email protected]

mailto:[email protected]

DisclaimerOne of those is a liar, guess who:



About this presentationLabs in this presentation are hosted by docker 1.8.1 running in my laptop:

+------------------------------------laptop-------------------------------------+| || +-----------------------docker-----------------------+ 172.16.0.0/16 || | | docker0: 172.16.0.1 || | +---------+ +---------+ +----------+ | || | | HAProxy | | appsrv1 | | rsyslogd | | || | +---------+ +---------+ +----------+ | || | | || +----------------------------------------------------+ || |+-------------------------------------------------------------------------------+

NOTE about docker: https://groups.google.com/forum/#!topic/docker-dev/KzCF7ouVhe8


https://groups.google.com/forum/#!topic/docker-dev/KzCF7ouVhe8

Watching live trafficHAProxy collects many statistics and information about traffic passing through.Here is a "high" level view:

• internal states:

• object administrative and operational status, check result

• number of failed check, time since the last transition

• server total downtime

• connection layer:

• number of failed connection attempts

• number of retries and redispatch

• session layer:

• number of current, max, limitation or total sessions

• number of bytes in and out

• number of denied request or responses

• number of request or response in error

• rate information

• time since last session


Watching live traffic• internal processing:

• queue information

• load-balancing statistics

• HTTP related:

• number of response per status code family

• HTTP request rate and max reached

• HTTP compression information

• termination states:

• data transfer aborted by either the client or the server

• These statistics can be read through a couple of ways:

• telnet-style, over UNIX, TCP or TLS socket: CSV data

• web interface: HTML or CSV data

• SNMP, for HAPEE or ALOHA users


Configuring the stats socket• Unix socket:

global ... stats socket /var/run/haproxy/socket stats timeout 10m

• TCP socket or TLS socket:

global ... stats socket 0.0.0.0:82 stats socket 0.0.0.0:83 ssl crt server.pem stats timeout 10m

Note: we can force a client certificate for this socket

• HTTP page:

frontend f_stats bind :88 stats enable stats uri / #stats scope <frontend or backend name>

Note: Of course, can be over HTTPs and require a client certificate


Demo #1: using the stats sockets1. run the lab: make run LAB=demo1 EX=1

2. check stats works as expected:

• open HTTP stats page: HTML content or CSV content

• open TCP and TLS stats socket

3. inject some traffic and check collected statistics

4. httperf some traffic and observe the result:

• look for the errors in HTML and CSV pages

• look for the errors using stats socket

•

Conclusion: in the HTML page, devil stands in the underlined information:


http://haproxy.local:88/

http://haproxy.local:88/;csv

Demo #1 output• inject output summary, no errors:

Reponse : 220 hits/sErreurs : 0

• httperf output, errors occur when many requests are processed in parallel:

Total: connections 500 requests 500 replies 500 test-duration 1.623 sReply status: 1xx=0 2xx=232 3xx=0 4xx=0 5xx=268

• socat output after httperf command has run:

# pxname svname hrsp_5xxf_myapp FRONTEND 268f_myapp sock-1b_myapp appsrv1 268b_myapp BACKEND 268f_stats FRONTEND 0f_stats sock-1


Logging• HAProxy logs are very verbose

• When the traffic workload allows it, they should be enabled all the time! Worst case, we must have theability to enable them on demand

• HAProxy can be configured to selectively log part of the traffic

• the log line can be customized to your needs (beware to not break the compatibility with halog)

• Logging can be enabled either in the global or in the defaults/frontend section

• Log format is configured per frontend, but log level must be reported in the backend too

• To setup your own log format, use the .... log-format directive


Logging• Example in global section

global log 172.16.0.1:8514 local0

defaults log global # 'pointer' to the global section option httplog

• Example in frontend/backend section

frontend fe log 172.16.0.1:8514 local0 option httplog default_backend be

backend be option httplog

• Split traffic and events logs:

global log 172.16.0.1:8514 local0 # traffic logs log 172.16.0.1:8514 local1 notice # event logs


Logging• Log only errors:

defaults option dontlog-normal

• Don't log empty connections or browser's pre-connect "feature"

defaults option dontlognull option http-ignore-probes

• Log only dynamic traffic:

frontend fe http-request set-log-level silent unless { path_end .php }


Information logged• path followed in HAProxy: frontend, backend, server

• client information: IP and port

• timers: client request, server connection, server response, queue, total time

• bytes read and written

• termination state

• number of connections on the process, frontend, backend, server

• number of connections in the server's or backend queue

• HTTP information:

• HTTP status code

• captured request and response information: cookie and header fields

• URL

• not only: Thanks to log-format directive, we can also log any information available through afetch: IE session rate on the frontend, any TLS information (cipher, SNI, CN, etc...), and muchmore


Demo #2: exploiting HAProxy logs1. read logs from rsyslogd container: docker attach rsyslogd

2. run the lab: make run LAB=demo2 EX=1

3. perform an HTTP request: GET http://haproxy.local/

4. inject some traffic

5. run httperf

6. Check what happened on the HTTP stats page: HTML content

7. read the logs from rsyslogd container

8. analyze logs with halog

9. it's a capacity problem, increase capacity!!!!



12. run httperf

13. Wait, I've heard about this magic maxconn parameter

14. Check the stats page and the logs


• Conclusion: queueing at HAProxy layer is magic !!!



Demo #2 output• halog output for URL response time, okavg column report the average response time per URL whenthe response isn't an error:

docker exec -it rsyslogd sh -c 'halog -uao </var/log/haproxy/traffic | column -t'4438 lines in, 42 lines out, 0 parsing errors#req err ttot tavg oktot okavg bavg btot src500 487 61759 123 1267 97 203 101869 /test.php88 76 1046 11 953 79 86 7605 /orders/o/tracking.php89 41 3422 38 3395 70 141 12601 /catalog/catalog/promotions.php89 20 4341 48 4335 62 218 19482 /catalog/catalog/search.php88 60 1808 20 1733 61 153 13471 /inventory/allocation.php88 71 1141 12 1027 60 100 8806 /orders/o/validation.php88 72 931 10 836 52 120 10614 /payment/process.php...90 71 146 1 97 5 103 9313 /s/img23.png89 61 160 1 124 4 107 9580 /s/img34.png89 64 165 1 108 4 100 8947 /s/img32.png90 59 184 2 139 4 137 12407 /s/img25.png

• halog output for server response time, pct_ok report the percentage of responses which aren't errors,avg_ct is the average connection time (TCP) and avg_rt is the average application response time:

docker exec -it rsyslogd sh -c 'halog -srv </var/log/haproxy/traffic | column -t'4438 lines in, 2 lines out, 0 parsing errors#srv_name 1xx 2xx 3xx 4xx 5xx other tot_req req_ok pct_ok avg_ct avg_rtb_myapp/appsrv1 0 1489 0 0 1554 1392 4435 1834 41.4 5 27f_stats/<STATS> 0 3 0 0 0 0 3 3 100.0 0 0


Demo #2 output• httperf output with 4 servers, adding server did not improve anything:


• Same httperf after setting up maxconn, errors have disappeared:



Queueing in HAProxy demistified• HAProxy can queue requests to protect weak applications and servers

• HAProxy prevents the application server to run too many requests in parallel (remember slide 4 andHAProxy's frontend/backend architecture)

• Queueing allows the following benefits:

• the server never crashes

• the application remains fast

• response time remains good

• no (or almost none) errors are reported to the clients

• saving resources

• Simply setup maxconn parameters on the server line statement in HAProxy's backend

• there are no magic values. Benchmarking the application is the only way. Benchmarking with highconcurency is the key

• From our experience, a good maxconn value is from 50 to 300 (as seen in the previous example...)

• coupled with HAProxy's TCP/HTTP buffering, queueing hides the client side latency and allows quicklyrecycling connection slots for an other client while carry on delivering response to a first one


What about the performance metricsWe saw the log line contains the URL and the server response time amongst many other information.So I want to:

• sort URLs by response time

• sort app server by response time

• much more

• this is when halog enters in the party and demo #3 as well!!!


Demo #3: measuring performance1. run the lab: make run LAB=demo3 EX=1

2. perform an HTTP request: GET http://haproxy.local/


4. Check status on the HTTP stats page: HTML content


• docker exec -it rsyslogd sh -c 'halog -uao </var/log/haproxy/traffic | column -t'

• docker exec -it rsyslogd sh -c 'halog -srv </var/log/haproxy/traffic | column -t'

• docker exec -it rsyslogd sh -c 'fgrep appsrv1 /var/log/haproxy/traffic | halog -uao | column -t'

• docker exec -it rsyslogd sh -c 'fgrep /catalog/ /var/log/haproxy/traffic | halog -srv | column -t'

6. split static/dynamic content, lab: make run LAB=demo3 EX=2


• Conclusion: we can clearly highlight where the latency occurs in the application



Demo #3 output• halog server response time output for EX=1, avg_rt is the average application response time:

docker exec -it rsyslogd sh -c 'halog -srv </var/log/haproxy/traffic | column -t'1009 lines in, 4 lines out, 0 parsing errors#srv_name 1xx 2xx 3xx 4xx 5xx other tot_req req_ok pct_ok avg_ct avg_rtb_myapp/appsrv1 0 256 0 0 0 0 256 256 100.0 0 117b_myapp/appsrv2 0 258 0 0 0 0 258 258 100.0 0 116b_myapp/appsrv3 0 273 0 0 0 0 273 273 100.0 0 106b_myapp/appsrv9 0 222 0 0 0 0 222 222 100.0 0 147

• halog server response time output for EX=2, avg_rt is the average application response time:

docker exec -it rsyslogd sh -c 'halog -srv </var/log/haproxy/traffic | column -t'695 lines in, 8 lines out, 0 parsing errors#srv_name 1xx 2xx 3xx 4xx 5xx other tot_req req_ok pct_ok avg_ct avg_rtb_myapp/appsrv1 0 96 0 0 0 0 96 96 100.0 0 228b_myapp/appsrv2 0 99 0 0 0 0 99 99 100.0 0 223b_myapp/appsrv3 0 98 0 0 0 0 98 98 100.0 0 216b_myapp/appsrv9 0 74 0 0 0 0 74 74 100.0 0 315b_static/appsrv1 0 79 0 0 0 0 79 79 100.0 0 4b_static/appsrv2 0 86 0 0 0 0 86 86 100.0 0 3b_static/appsrv3 0 69 0 0 0 0 69 69 100.0 0 6b_static/appsrv9 0 94 0 0 0 0 94 94 100.0 0 3

We can obviously see that:

• the application is very slow compared to delivering static content

• appsrv9 is 50% slower than the other servers


Demo #4: boosting performance with HAProxy1. run the lab: make run LAB=demo4 EX=1

2. Check status on the HTTP stats page: HTML content

3. run httperf against the server, observe the result

4. run httperf against the haproxy, observe the result

• conclusion

• HAProxy protects the application

• HAProxy improves response time

• YES, there is some magic in queueing



Demo #4 output• httperf output when hammering the server itself:

httperf --uri=/test.php --server=server1 --port 80 --num-conn 1000 --rate 1000000 | egrep "^(Reply status|Total)"Total: connections 1000 requests 1000 replies 1000 test-duration 7.085 sReply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0

• httperf output with HAProxy in front of the server to protect it with maxconn:

httperf --uri=/test.php --server=haproxy.local --port 80 --num-conn 1000 --rate 1000000 | egrep "^(Reply status|Total)"Total: connections 1000 requests 1000 replies 1000 test-duration 1.619 sReply status: 1xx=0 2xx=1000 3xx=0 4xx=0 5xx=0

For the same set of requests, the server responded much faster when it runs behind HAProxy!


Conclusion• HAProxy protects weak application and servers

• HAProxy reduce time when troubleshooting issues

•

All the tips introduced in those slides are available in both HAProxy Community and Enterpriseversion, use them, abuse them, call us if you need expertise!

• HAProxy doesn't lie

• I'm not a laundry detergent seller!

•

During this presentation, only HAProxy did not lie!


• NON EXHAUSTIVE presentation !!!! HAProxy can do much more for your safety


Haproxy web performance

Engineering

Transcript of Haproxy web performance