Drupal PerformancePhilip Norton

Blog at #! code (www.hashbangcode.com)

@philipnorton42 on Twitter

Creator of Vlad

Technical Lead at

Philip Norton

Not a sysadmin!

Improving

Performance

Improving

Performance

Performance

Testing

“Performance enhancements without

benchmarks is little more than guesswork.”

Probably heard it at PHPNW...

Setting a benchmark.“How does the site perform?”

Identifying bottlenecks.“Where is the site slow?”

Comparing improvements with the benchmark.“Has that change improved things?”

Gauging the limits of resources.“If the current traffic doubles will the site fall over?”

1

2

3

4

Performance Testing Goals

Google Chrome

More than a web browser.

Lots of profiling tools available.

Measure all kinds of things like raw network speed, page rendering, and JavaScript performance.

YSlow / ShowSlow

YSlow

YSlow is a Firebug plugin developed by Yahoo! It measures factors that govern page speed and gives an overall score.

ShowSlow

ShowSlow can be used to store YSlow results and show difference of scores over time.

Drupal Devel Module

Devel module can be used to render statistics on the page load.

Shows all SQL queries run and how long they took to execute.

Queries highlighted in red are slow and should be addressed.

1

2

3

MySQL Process List

Provides a window on what the MySQL server is doing, and can pinpoint problems.

> SHOW PROCESSLIST;

Keep an eye out for sleeping or lengthy processes or processes that transmit lots of data.

Only provides a snapshot of processes.

MySQL Slow Query Log

Often overlooked but provides a valuable information on queries that are slowing things down.

Set a minimum execution time for queries to be entered into log.

Use EXPLAIN to figure out why queries are slow.

MySQL Slow Query Log

Add the following to my.cnf:

slow-query-log = 1slow-query-log-file = /var/log/mysql/mysql-slow.loglong_query_time = 2log-queries-not-using-indexes

Use mysqldumpslow to analyse the logs.

Xdebug Profiler

Part of the Xdebug tool.

Shows everything that happened during a page request.

Use tools like KCachegrind and Webgrind to view the information and identify bottlenecks.

Very resource heavy so run when needed (never in production!).

1

2

3

4

XHProf

Generates profiling data for PHP.

Performant enough to run on production environment.

Can profile 1 request in 100 or 1000 to get a window on performance.

Use XH Gui to view XHProf results and compare results over time.

1

2

3

4

Siege / JMeter / Loader.io

Benchmark a site by throwing traffic at it.

Find out how much traffic you can send before you start getting errors.

Siege and JMeter are a local tools so all traffic comes from one place.

Loader.io is a distributed service.

1

2

3

4

New Relic

Fully featured performance statistics package for front and back end monitoring.

Has knowledge of Drupal and can identify slow modules/views/hooks.

Lots of information available and shows changes over time.

Paid service and can be a little expensive.

1

2

3

4

Improving Performance

There is no magic bullet!Speed improvements are made through multiple

improvements and usually have diminishing returns.

Drupal Caching

Too obvious? Cache all the things!

Turn on anonymous page caching.

Use Block Cache Alter module to enable/force block level caches.

https://www.drupal.org/project/blockcache_alter

1

2

3

Drupal Caching

Turn on Views query and render caching.

Use Entity Cache module to cache entities.

https://www.drupal.org/project/entitycache

1

2

Boost Module

Creates a static cache of page content.

Will also gzip content, including HTML, CSS, JavaScript, XML.

https://www.drupal.org/project/boost

1

2

CSS / JavaScript Aggregation

Combines CSS and JavaScript files.

Less files being downloaded means:- Less strain on the server.- Quicker rendering times for users.

Drupal also removes comments from files which makes them smaller.

1

2

3

Advanced CSS / JS Aggregation

Easy to install module.

Further reduces JS/CSS footprint by minifying and compressing the files.

Not a substitute for slow JavaScript code.

https://www.drupal.org/project/advagg

1

2

3

Turn off database Logging

Drupal tends to log a lot of user activity.

PHP warnings and notices can bring a site down if this module is active.

(You should really be removing these errors where possible.)

Use syslog to log any errors to file based logs.

1

2

3

StatisticsWrites to the database on every page load.Use an analytics package instead.

PHP Input FilterBypasses most caches. No. Just no.

1

2

Other Bad Modules

Custom Code

Where problems mostly stem from?

Cache where possible.

Try to avoid potentially heavy Drupal hooks like hook_init() and hook_boot().

Profile and experiment, especially with lots of data. e.g. Test with 5,000 or 50,000 nodes instead of 5.

1

2

3

Queues

Doing more than one thing in a page request?

Stick that in a queue and process it later!

Built into Drupal 7, easily extendable.

Should form a key part of any batch process or APIservice you implement.

1

2

3

PHP Micro Optimisations

Single quotes '' vs double quotes "".

Printing stuff using echo vs print.

Pre increment (++$i) vs post increment ($i++).

Creating arrays from strings using explode() vs preg_split().

Micro optimizations are basically pointless*

*unless you are Facebook.

PHP Micro Optimisations

Performance analysis and benchmarking should feed into any code modifications.

Don’t optimise code this way unless you have evidence that you need to.

Don’t sacrifice code maintainability to increase site speed by 0.00001%.

1

2

3

Drupal Fields

Don’t overuse fields.

node_load() and user_load() will pull in all field data (plus revisions) into memory.

50 fields on a node means 50 delete and 50 insert statements on every update.

Try to use shared fields where possible.

1

2

3

4

Cron

Don’t get your users to run cron. Always do it via a crontab.

Use Elysia cron module to split the cron up.

Set cache clearing cron runs to hourly.

https://www.drupal.org/project/elysia_cron

1

2

3

PHP 5.5+

New versions of PHP are faster than ever.

Especially with objects.

You should be upgrading to take advantage of this.

Use PHP-FPM for high performance PHP.

1

2

3

4

PHP Opcode Cache

e.g. APC or Zend Cache.

Caches PHP scripts so they don’t have to be recompiled when run again.

Easy to install and gives a speed increase of up to 30%.

Make sure your cache size is big enough to accommodate the number of files!

1

2

3

4

Add indexes to tables. Slow query log will tell you where indexes are needed.

Add enough memory for MySQL to use (you don’t want lots of read/writes to disk).

Use a master/slave setup - Drupal will use slave server for expensive read only queries (eg. search).

MySQL Performance Tweaks

1

2

3

Apache Performance Tweaks

Turn off KeepAlive.

Set your max clients to a sensible value.- Too high and server crashes when busy.- Too low and you can’t serve traffic correctly.

Remove any modules that you don’t need.

Turn off AllowOverride and include .htaccess files in server configuration.

1

2

3

4

Solr Search

Moves search out of your database server.

Much more efficient and has a bunch of nice features (e.g. facets).

Hard to set up, but services exist to help you.

1

2

3

Memcache / Redis

Store your Drupal cache outside of a database.

Much faster than traditional database servers (but only for this kind of data).

Can be scaled easily.

1

2

3

Split Site Up

Complex sites can be slow. Breaking them into sub-sites allows for better experience.

Most users don’t care about the URL, as long as they get a coherent experience.

Other sites can be hosted elsewhere.

Bakery module provides single sign on.

1

2

3

4

Server Side Caching

i.e. Varnish.

Provides a reverse proxy that caches all page content.

Repeat requests never hit Drupal, Apache, or MySQL and are instead served by Varnish.

Huge speed increase for little effort.

1

2

3

4

Load Balancers

Balances the load across multiple web servers.

More complex to setup, but some providers have good solutions (Rackspace).

Stops Apache being the bottleneck.

Can have some negative effects on Drupal (e.g. file uploads available on all servers).

1

2

3

4

Content Delivery Network (CDN)

Move the files or even the entire site out of your hosting environment.

Pull vs push CDN.

CDN content nodes closer to users mean less distance to travel for request/response.

Make sure you allow authentication traffic through.

1

2

3

4

Throw money at it!

Throw money at it!

Buying monster servers does not necessarily mean performance improvements.

Can sometimes solve the wrong problem.

If you plaster over cracks, the cracks will still be there.

“We’ve spent all this money on servers and the site is still slow.” - An annoyed client

1

2

3

4

To Recap...

Server

PHP 5.5+ with APC

Optimised MySQL

Optimised Apache

Memcache/Redis

Solr

1

2

3

4

5

Drupal

Cache all the things

JS/CSS aggregation

Split complex sites up

Disable expensive modules

1

2

3

4

External

Varnish

Load balancers

CDN

1

2

3

Essentially...

“Making a Drupal site fast is mostly about removing

Drupal from the page request.”

Philip Norton

Failing GracefullyIf at first you don’t succeed, fail!

Drupal Log Levels

Set a level that won’t print error logs to screen.

Maintenance theme

Config item in Drupal’s settings.php file.

Create a nice looking error page that fits in with site theme.

Link out to social media channels.

Prevent Drupal from printing SQL error messages.

1

2

3

4

Varnish error page

If your web server doesn’t respond then Varnish can be made to issue an error page.

Or even redirect traffic to another (static) site.

1

2

Static pages

Don’t be afraid to present users with a static page.

“We’re busy, please come back later” is better than an error page.

With load balanced solutions you can give a percentage of your users a static page and serve the rest correctly.

1

2

3

Case StudyA quick case study about dealing with

peak traffic on awards nights

BAFTA: Peak traffic

High levels of traffic on major awards nights.

1,400% increase from normal traffic levels.

Needed a scalable hosting solution that could be ramped up on selected evenings.

Rackspace cloud chosen as the hosting platform.

1

2

3

4

BAFTA: Peak traffic

Film and TV awards

Games Awards

Nominees Announced

Children’s Awards

1

2

4

BAFTA: Peak traffic

Sites are hosted on a number of web nodes.

Rackspace Load Balancer provides unified front end.

Each web node consists of a Varnish/Apache setup.

1

2

3

BAFTA: Peak traffic

In the run-up to awards night new nodes are provisioned and added to the load balancer.

Caches on each web node are warmed and site is load tested using loader.io service.

After awards night the nodes are scaled down to normal service.

1

2

3

BAFTA: Peak traffic

Rackspace Load Balancer

Web Nodes

SolrNFSMySQL Master/Slave

BAFTA: Peak traffic

Varnish

Apache

Load BalancerEach web node setup as a Varnish/Apache pair.

Varnish configured to produce nice error page if Apache not available.

1

2

BAFTA: Peak traffic

Contingency plans put in place just in case things went wrong.

Error page templates put in place for:- Drupal- Varnish- Rackspace load balancer

Secure login module ready to disable member access to reduce load on database server.

1

2

3

BAFTA: Peak traffic

Team Access on call during the weekend in case of issues.

Rackspace also had a dedicated technician on hand.

Monitoring of servers during awards ceremony in order to adapt to any problems.

1

2

3

BAFTA: Peak traffic

Cloudflare chosen as a CDN.

Caches content externally for anonymous traffic.

Cloudflare module used to clear external caches from within Drupal.

>90% of all traffic served through Cloudflare.

1

2

3

4

BAFTA: Peak traffic

BAFTA: Results

No downtime.

Cloudflare served most of the traffic.

Site stayed responsive throughout the peak traffic levels.

Client happy.

1

2

3

4

BAFTA: Peak traffic

Fin.