Inside Sqale’s BackendYAPC::Asia Tokyo 2012

Gosuke Miyashitapaperboy&co., Inc.

Technical Managerat

paperboy&co.

cpan:mizzygithub.com/mizzy

mizzy.org@gosukenator

Inside Sqale’s Backend

http://www.facebook.com/sqalejp

WARNINGThere are no topics

about Perl in this talk

What is Sqale?

Cloud Application Platform like Heroku

Architecture Overview

AWS

SSH Router

Containers

Web Proxyto Containers

Deploy Servers

File Repositories

SFTPGit over SSHSSH

HTTP/HTTPS

Containers

AWS

SSH Router

Containers

Web Proxyto Containers

Deploy Servers

File Repositories

SFTPGit over SSHSSH

HTTP/HTTPS

Virtual Environments Assigned To Users

Similar to Dynos of Heroku

Containers made by LXC (Linux Containers)

EC2 Instance (1 Virtual Machine)

Container for

user B

Container for

user A

Container for

user A

Container for

user B

Container for

user B

Container for

user D

Container for

user D

Container for

user C

Container for

user E

Container for

user E

Container for

user F

Container for

user F

Container for

user E

Container for

user F

Container for

user F

NginxUnicorn

sshdsupervisrod

on each container

Amazon Linux+

Patched kernel(3.2.16)

grsecurity kernel patchfor various restrictions

original kernel patchesto restrict tcp port

bind and fork bomb

Anti fork bomb patch makes some changes to cgroup and fork process

Seepaperboy-sqale/sqale-patches

on GitHub

Web Proxy

AWS

SSH Router

Containers

Web Proxyto Containers

Deploy Servers

File Repositories

SFTPGit over SSHSSH

HTTP/HTTPS

nginx

Container for

user A

Container for

user B

Container for

user B

Container for

user C

Container for

user C

Container for

user C

ELB

nginx

HTTP/HTTPS

nginxlua-nginx-module

redis2-nginx-module

Container for

lokka-mizzy

Container for

lokka-mizzy

Container for

i4pc-mizzy

Container for

i4pc-mizzy

nginx

http://www.i4pc.jp/

Redis

nginx port 8081 nginx port 8082 nginx port 8083 nginx port 8084

Which containers?

host001:8083, host001:8084

host001

or

location / { set $container "";    set $next_containers "";

error_page 502 = @failover;

    rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}

nginx.conf (excerpt)

local reply = ngx.location.capture("/redis")if reply.status ~= ngx.HTTP_OK then ngx.exit(503)end

local containers, type = parser.parse_reply(reply.body)

dynamic-proxy.lua (excerpt)

while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend

dynamic-proxy.lua (excerpt)

ngx.var.container = containerngx.var.next_containers = luabins.save(containers)

dynamic-proxy.lua (excerpt)

location / { set $container "";    set $next_containers "";

error_page 502 = @failover;

    rewrite_by_lua_file dynamic-proxy.lua; proxy_pass http://$container;}

nginx.conf (again)

location @failover { error_page 502 = @failover;

    rewrite_by_lua_file failover.lua; proxy_pass http://$container;}

nginx.conf (excerpt)

failover.lua (excerpt)local downed_container = ngx.var.containerif downed_container then ngx.shared.downed_containers:set( downed_container, 1, sqale.NEGATIVE_CACHE_SECONDS )end

failover.lua (excerpt)while #containers > 0 do tmp = table.remove( containers, math.random(#containers)) if ngx.shared.downed_containers:get(tmp) then ngx.log(ngx.DEBUG, tmp .. " is down") else container = tmp break endend

if not container then ngx.exit(503)end

ngx.var.container = containerngx.var.next_containers = luabins.save(containers)

failover.lua (excerpt)

location @failover { error_page 502 = @failover;

    rewrite_by_lua_file failover.lua; proxy_pass http://$container;}

nginx.conf (agin)

Seehttp://bit.ly/UHbHIb

by @hiboma

SSH Router

AWS

SSH Router

Containers

Web Proxyto Containers

Deploy Servers

File Repositories

SFTPGit over SSHSSH

HTTP/HTTPS

SSH Router

File Repositories(Git Server)

Git SSH Login

File Repositories(File Server)

Containers

SFTP

How implement this routing?

OpenSSH with script authentication patch

Seemizzy/openssh-script-auth

on GitHub

Change routes by SSH_ORIGNAL_COMMAND

In case of SSH_ORIGINAL_COMMAND

is “git-*”

SSH Router

File Repository(Git Server)

git push(ssh sqale@gateway.sqale.jp git-recieve-pack ‘/mizzy/lokka.git’)

MySQL

Run AuthorizedKeys Script

Verify the public keyand get the user’s git server command=“ssh sqale@git001.sqale.lan git-recieve-pack ‘/var/repos/mizzy/lokka.git’”

In case of SSH_ORIGINAL_COMMAND

is “sftp-server”

SSH Router

File Repository

(File Server)

sftp sqale@gateway.sqale.jp(ssh sqale@gateway.sqale.jp sftp-server)

MySQL

File Repository(Git Server)

git push

Run AuthorizedKeys Script

Verify the public keyand get the user’s file server command=“ssh sqale@file001.sqale.lan sftp-server”

In case of SSH_ORIGINAL_COMMAND

is empty

SSH Router

Container

ssh sqale@gateway.sqale.jp

MySQL

Run AuthorizedKeys Script

Verify the public keyand get the user’s cotainers list

command=“ssh sqale@ users001.sqale.lan -p 8081”

Display the user’s containers list and wait the user’s selection

Deploy Servers

AWS

SSH Router

Containers

Web Proxyto Containers

Deploy Servers

File Repositories

SFTPGit over SSHSSH

HTTP/HTTPS

Please ask to@kyanny

Other

About Sqale’s Server Build Automation

http://bit.ly/NBbj9Fby @lamanotrama

Thanks