PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story
-
Upload
vanphp -
Category
Technology
-
view
3.453 -
download
1
Transcript of PHP At 5000 Requests Per Second: Hootsuite’s Scaling Story
PHP at 5000 Requests / SecHootsuite’s Scaling Story
Lead Technical Engineer - Platform@bmonkman
Bill Monkman
Overview - Selected Current Architecture
Users lb1 lb2 lb3 ... Nginx Load balancers
web1 web2 web3 ... Nginx web serversPHP-FPM PHP-FPM PHP-FPM PHP-FPM
mem1 ...
Memcached cluster
master slave
Mysql cluster
master slave
MongoDB cluster
slavemaster
shard1
shard2
geard1 geard2
Gearman cluster
worker1 ... ...
Services
...
• Apache
• PHP
• MySQL
Technologies - at first
Then...
It’s hard to scale MySQL horizontally
Problem
Memcached.
● Distributed cache, cluster of boxes with lots of RAM, trivial to scale
● Cache as much as possible, invalidate only when necessary
● Use cache instead of DB
● No joins - decouple entities (collection caching)
● Twemproxy!
Solution - Caching
• Phil Karlton
“There are only two hard things in Computer Science: cache invalidation and naming things.”
Solution - Caching
SocialNetwork
MvcModelMysql
MvcModelBase
MvcModelBaseCaching
SELECT * FROM member WHERE org_id=888
set individual cache records
set collection cache
Automatic invalidation of collection cache
Solution - Caching
member_1 {data}
member_5 {data}
member_9 {data}
member_org_888 [1,5,9]
It’s hard to scale MySQL horizontally
Now:
● No need to scale MySQL
● Able to serve the whole site on 1 MySQL server
● 500 MySQL SELECTs per second. 50,000 Memcached GETs.
● 99+% hit rate
Solution - Caching
Then...
Need a way to perform asynchronous, distributed tasks using a single-threaded language.
Problem
Gearman.
● Distribute work to other servers to handle (workers also using
PHP, same codebase)
● Precursor to SOA where everything is truly distributed
● Many other solutions, queueing systems.
Solution - Gearman
Solution - Gearman
geard1 geard2
gearworker1 gearworker2 gearworker6
Need a way to perform asynchronous, distributed tasks using a single-threaded language.
Now:● Moved key tasks to Gearman
● Another cluster, scalable separately from web
● Discrete tasks, callable sync or async
Solution - Gearman
Then...
Need to store data with the potential to grow too big to handle effectively with MySQL.
Problem
MongoDB.
● Certain data did not need to be highly relational
● NoSQL DB, many other solutions these days
● Mongo can be a pain, lots of moving parts
● Had to make our own sequencer where auto-incremented ids were
necessary
Solution - MongoDB
Need to store data with the potential to grow too big to handle effectively with MySQL.
Now:● Multiple clusters containing amounts of data that likely would
have crushed MySQL● Billions of rows per collection, many TB of data on disk
Solution - MongoDB
• Apache
• PHP
• MySQL
• Memcached
• Gearman
• MongoDB
Technologies
Then...
With a codebase and an engineering team increasing in size, how do we keep up the pace of development and maintain control of the system?(SVN, big branches, merge hell)
Problem
Dark Launching.
● Wrap code in block with a specific name
● That name will appear in a management page
● Can control whether or not that block is executed by modifying it’s value
● Boolean , random percentage, session-based, member list, organization
list, etc.
Solution - Dark Launching
if (In_Feature::isEnabled(‘TWITTER_ADS’)) {
// execute new code
} else {
// execute old code
}
Solution - Dark Launching
• Control your code
• Limit risk -> raise confidence -> speed up pace of releases
• “Branching in Production”
• Learning happens in Production
Dark Launching - Reasons
With a codebase and an engineering team increasing in size, how do we keep up the pace of development and maintain control of the system?
Now:● Work fast with more confidence
● Huge amount of control over production systems
● Typically 10+ code releases to production per day
● Push-based distribution with Consul
Solution - Dark Launching
Then...
With a rapidly increasing codebase and amount of users / traffic how do we keep visibility into the performance of the code?
Problem
Statsd / Graphite.Logstash / Elasticsearch / Kibana. Sensu
● Statsd for metrics
● Logstash for log events
● Sensu for monitoring / alerting
Solution - Monitoring
Statsd::timing('apiCall.facebookGraph', microtime(true) - $startTime);
Solution - Monitoring
Logger::event('user liked from in-stream', In_Log::CATEGORY_UX, $logData);
Solution - Monitoring
• Visibility into the performance and behaviour of your application
• Iterate upon your code, measure results
• Pairs well with dark launching
• Also systems like New Relic
Solution - Monitoring
With a rapidly increasing codebase and amount of users / traffic how do we keep visibility into the performance of the code?
Now:● Able to watch performance / behaviour in real time.● Able to view important events both in the aggregate or very
granular● Able to control the system and watch the effect of changes
Solution - Monitoring
Optimizations
• Things expand beyond their initial scope
• Case in point: Translations
Optimizations
• Within reason, push work up to users
• Make your users into a distributed processing grid
• e.g. Stream rendering
Optimizations - Push work to users
• Performance is more important than clean code, business reqts (in the instances where they may be mutually exclusive)
• Fine line between future proofing and premature optimization
• Don’t add burdensome processes, but make it easy for your team to do things the right way
• Know your weak spots, protect against abuse
Optimizations - Performance / Risks
Technologies
LinuxNginx
MySQL
PHP-FPM
Jenkins
Scala
MongoDB
GearmanRedis
Akka
Python
Memcached
HAProxy
jQuery
ZeroMQ
RabbitMQBackbone
EC2Zend
Docker
Cloudfront CDN
VarnishElasticSearch
Logstash
Zookeeper
Kibana
Statsd/Graphite
Packer
Vagrant
Nagios
VirtualBox
Sensu
SymfonyRiak
Websockets
Comet
Hadoop
Spark/SharkAnsible
Consul
Composer
Git
RedshiftWebpack
With a huge and growing monolithic codebase and over 80 engineers, how to keep scaling in a manageable way?
Problem
SOA.
● Split up the system into independent services which communicate only via APIs
● Teams can work on their own services with encapsulated business logic and have their own
deployment schedules.
● We chose to use Scala/Akka for services, communicating via ZeroMQ
● SOA transition made easier by the “no joins” philosophy
● Tons of work
Solution - SOA
SOM.
● “Service Oriented Monolith”
● When splitting up a monolithic codebase, dependencies are what kill you
● Fulfill dependencies by writing interim services using existing PHP code
● Maintain the contract and future scala services will be drop-in replacements
Solution - SOA
With a huge and growing monolithic codebase and over 130 engineers, how to keep scaling in a manageable way?
Today:● Transitioning to Scala SOA● PHP will still be used as the Façade, a thin layer built on top of
the business logic of the services it interacts with.
Solution - SOA
Conclusion
Thank You!Bill Monkman@bmonkman
More Info:code.hootsuite.com