Post on 18-Jul-2015
How we built Caibangzi.com
Robin Lu (caibangzi.com)
IntroductionPersonal Investment Management. Tool + Community
Founders: We are engineers!
Technology:
Ruby On Rails
MySQL,Linux,Lighttpd ...
TopicsProblems we are facing
Architecture evolution of caibangzi
Maintenance
Q&A
Peaks!
High Peak
Heavy Computation
Visiting Peak, Computation Peak and Cache expiration Peak are in the same time. :...(
Evolution - StartLaunched at March, 2007
Hardware: 1 x 1U server
1xAMD Opteron 250, 2G Memory, 250G SATA, RAID 1
Bandwidth
Shared
less than 3Mbps
Evolution - StartApache 2 + mod_proxy_balancer
mongrel x 8
caibangzi
rails 1.1.6
MySQL 5
memcached
All in one box
First ChallengeMay, 2007
Extremely time consuming actions blocked the site.
Performance tuning
Optimize Rails cache
Second Challenge
August, 2007
We were running out of bandwidth
Solution:
Money
upgrade to 5Mbps
Server upgrading
High growth rate after August, 2007
Hardware upgrading in Sept. 2007
still 1 x 1U
2 x AMD 2214 2.2GHz
4G RAM
Until Sept. 2007Apache 2 + mod_proxy_balancer
mongrel x 20
caibangzi
rails 1.1.6
MySQL 5
memcached
Still in one box
Something happened in Oct. 2007
Stock Index reached 6000
Daily pv reached 1 million
Bigger challenge came
Several Changes We Made
Upgrade bandwidth to 10Mbps
Separate APP server and DB server
Load Balancer
Distributed Computation
2 BoxesWeb Server
event mongrel x 30
caibangzi
rails
MySQL 5
memcachedAPP
Tianyuanjia
Load balancer
Run with 2 mongrels
behindbalancer
Lighttpd 1.4
Load balancerSlow actions block the others with balancer:
by request
by traffic
round-robin
Apache
HAproxy
Nginx
Note: Nginx fixed it with ‘fair’ balancer later
Caibangzi Choice of Balancer
Lighttpd 1.5 beta
proxy-core.balancer = "sqf"
http://redmine.lighttpd.net/issues/show/1362
For Rails application, choice of balancer is sometimes more important than choice of web server
More on Web Server
disable keep-alive
serve static files with web server instead of mongrel
tune configurations according to your web server documentations
Distributed solution
BackgrounDRB
based on Distributed Ruby (DRB)
use ruby thread
high overhead during context switching
very slow when concurrent connections higher than 10
Distributed solution
FastBackground
our own distributed computation solution
based on Eventmachine
simple, but fast and works for us
Rails upgrading
Early of 2008, we upgraded to Rails 2.0
expired several plugins
refactory to be more RESTful
cookie session
multi view => wap.caibangzi.com
Open API
API Based on OAuth
integrate caibangzi into your own sites
Other APPs support several F8 clones
xiaonei.com, hainei.com, UCHome
Maintenance
Munin
easy to use and extend
Monit
Crontab
With all that....we handled:
pv/day 1Mrails req/day 1.8M
peak rails req/sec 160total req/day 5Mpeak req/sec 430
It’s the fact. Not the capacity.Our CPU usage never goes higher
than 30%
Q&A