Inside Sqale's Backend at YAPC::Asia Tokyo 2012
-
Upload
gosuke-miyashita -
Category
Technology
-
view
4.422 -
download
3
Transcript of Inside Sqale's Backend at YAPC::Asia Tokyo 2012
![Page 1: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/1.jpg)
Inside Sqale’s BackendYAPC::Asia Tokyo 2012
Gosuke Miyashitapaperboy&co., Inc.
![Page 2: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/2.jpg)
Technical Managerat
paperboy&co.
![Page 3: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/3.jpg)
cpan:mizzygithub.com/mizzy
mizzy.org@gosukenator
![Page 4: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/4.jpg)
Inside Sqale’s Backend
![Page 5: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/5.jpg)
http://www.facebook.com/sqalejp
![Page 6: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/6.jpg)
WARNINGThere are no topics
about Perl in this talk
![Page 7: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/7.jpg)
What is Sqale?
![Page 8: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/8.jpg)
![Page 9: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/9.jpg)
Cloud Application Platform like Heroku
![Page 10: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/10.jpg)
![Page 11: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/11.jpg)
Architecture Overview
![Page 12: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/12.jpg)
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
![Page 13: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/13.jpg)
Containers
![Page 14: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/14.jpg)
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
![Page 15: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/15.jpg)
Virtual Environments Assigned To Users
![Page 16: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/16.jpg)
Similar to Dynos of Heroku
![Page 17: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/17.jpg)
Containers made by LXC (Linux Containers)
![Page 18: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/18.jpg)
EC2 Instance (1 Virtual Machine)
Container for
user B
Container for
user A
Container for
user A
Container for
user B
Container for
user B
Container for
user D
Container for
user D
Container for
user C
Container for
user E
Container for
user E
Container for
user F
Container for
user F
Container for
user E
Container for
user F
Container for
user F
![Page 19: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/19.jpg)
NginxUnicorn
sshdsupervisrod
on each container
![Page 20: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/20.jpg)
Amazon Linux+
Patched kernel(3.2.16)
![Page 21: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/21.jpg)
grsecurity kernel patchfor various restrictions
![Page 22: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/22.jpg)
original kernel patchesto restrict tcp port
bind and fork bomb
![Page 23: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/23.jpg)
![Page 24: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/24.jpg)
![Page 25: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/25.jpg)
Anti fork bomb patch makes some changes to cgroup and fork process
![Page 26: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/26.jpg)
Seepaperboy-sqale/sqale-patches
on GitHub
![Page 27: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/27.jpg)
Web Proxy
![Page 28: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/28.jpg)
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
![Page 29: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/29.jpg)
nginx
Container for
user A
Container for
user B
Container for
user B
Container for
user C
Container for
user C
Container for
user C
ELB
nginx
HTTP/HTTPS
![Page 30: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/30.jpg)
nginxlua-nginx-module
redis2-nginx-module
![Page 31: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/31.jpg)
Container for
lokka-mizzy
Container for
lokka-mizzy
Container for
i4pc-mizzy
Container for
i4pc-mizzy
nginx
http://www.i4pc.jp/
Redis
nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084
Which containers?
host001:8083, host001:8084
host001
or
![Page 32: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/32.jpg)
location / { set $container ""; set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
nginx.conf (excerpt)
![Page 33: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/33.jpg)
local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)end
local containers, type = parser.parse_reply(reply.body)
dynamic-proxy.lua (excerpt)
![Page 34: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/34.jpg)
while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
dynamic-proxy.lua (excerpt)
![Page 35: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/35.jpg)
ngx.var.container = containerngx.var.next_containers = luabins.save(containers)
dynamic-proxy.lua (excerpt)
![Page 36: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/36.jpg)
location / { set $container ""; set $next_containers "";
error_page 502 = @failover;
rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}
nginx.conf (again)
![Page 37: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/37.jpg)
location @failover { error_page 502 = @failover;
rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
nginx.conf (excerpt)
![Page 38: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/38.jpg)
failover.lua (excerpt)local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end
![Page 39: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/39.jpg)
failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend
![Page 40: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/40.jpg)
if not container then ngx.exit(503)end
ngx.var.container = containerngx.var.next_containers = luabins.save(containers)
failover.lua (excerpt)
![Page 41: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/41.jpg)
location @failover { error_page 502 = @failover;
rewrite_by_lua_file failover.lua; proxy_pass http://$container;}
nginx.conf (agin)
![Page 42: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/42.jpg)
Seehttp://bit.ly/UHbHIb
by @hiboma
![Page 43: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/43.jpg)
SSH Router
![Page 44: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/44.jpg)
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
![Page 45: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/45.jpg)
SSH Router
File Repositories(Git Server)
Git SSH Login
File Repositories(File Server)
Containers
SFTP
![Page 46: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/46.jpg)
How implement this routing?
![Page 47: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/47.jpg)
OpenSSH with script authentication patch
![Page 48: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/48.jpg)
![Page 49: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/49.jpg)
Seemizzy/openssh-script-auth
on GitHub
![Page 50: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/50.jpg)
Change routes by SSH_ORIGNAL_COMMAND
![Page 51: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/51.jpg)
![Page 52: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/52.jpg)
In case of SSH_ORIGINAL_COMMAND
is “git-*”
![Page 53: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/53.jpg)
SSH Router
File Repository(Git Server)
git push(ssh [email protected] git-recieve-pack ‘/mizzy/lokka.git’)
MySQL
Run AuthorizedKeys Script
Verify the public keyand get the user’s git server command=“ssh [email protected] git-recieve-pack ‘/var/repos/mizzy/lokka.git’”
![Page 54: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/54.jpg)
In case of SSH_ORIGINAL_COMMAND
is “sftp-server”
![Page 55: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/55.jpg)
SSH Router
File Repository
(File Server)
sftp [email protected](ssh [email protected] sftp-server)
MySQL
File Repository(Git Server)
git push
Run AuthorizedKeys Script
Verify the public keyand get the user’s file server command=“ssh [email protected] sftp-server”
![Page 56: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/56.jpg)
In case of SSH_ORIGINAL_COMMAND
is empty
![Page 57: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/57.jpg)
SSH Router
Container
MySQL
Run AuthorizedKeys Script
Verify the public keyand get the user’s cotainers list
command=“ssh sqale@ users001.sqale.lan -p 8081”
Display the user’s containers list and wait the user’s selection
![Page 58: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/58.jpg)
![Page 59: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/59.jpg)
![Page 60: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/60.jpg)
Deploy Servers
![Page 61: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/61.jpg)
AWS
SSH Router
Containers
Web Proxyto Containers
Deploy Servers
File Repositories
SFTPGit over SSHSSH
HTTP/HTTPS
![Page 62: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/62.jpg)
Please ask to@kyanny
![Page 63: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/63.jpg)
Other
![Page 64: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/64.jpg)
About Sqale’s Server Build Automation
http://bit.ly/NBbj9Fby @lamanotrama
![Page 65: Inside Sqale's Backend at YAPC::Asia Tokyo 2012](https://reader031.fdocuments.net/reader031/viewer/2022030316/58776f341a28ab5b568b5671/html5/thumbnails/65.jpg)
Thanks