Advanced nginx in mercari - How to handle over 1,200,000 HTTPS Reqs/Min
-
Upload
masahiro-nagano -
Category
Technology
-
view
17.737 -
download
1
Transcript of Advanced nginx in mercari - How to handle over 1,200,000 HTTPS Reqs/Min
Advanced nginx in Mercari
「nginx実践入門」出版記念!執筆者らが語る nginx Tech Talks
Masahiro Nagano
How to handle over 1,200,000 HTTPS Reqs/Min
Me
• Masahiro Nagano @kazeburo
• Mercari, inc.
• Principal Engineer, Site Reliability Engineering Team
• BASE, inc. Technical Advisor
MercariYour Friendly Mobile MarketPlaceJP US
Mercari KPI
DOWNLOAD
GMV
ITEMS
3100万DL(JP+US)
月間数十億円
1日数十万品以上
API Traffic
1,200,000 reqs/min (HTTPS only)
nginx in mercari
©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
User Users Client Multimedia Corporate data center
Traditional server
Mobile Client
Internet AWS Management Console
IAM Add-on Example:IAM Add-on
Amazon Mechanical Turk
On-Demand Workforce
Human Intelligence Tasks (HIT)
Assignment/Task
RequesterWorkersAmazon Mechanical Turk
Non-Service Specific
©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
User
Users
Client
Multimedia
Corporate data centerTraditional server
Mobile Client
Internet
AWS Management Console
IAM Add-on
Example:IAM Add-on
Amazon Mechanical Turk
On-Demand Workforce
Human Intelligence Tasks (HIT)
Assignment/Task
Requester
Workers
Amazon Mechanical Turk
Non-Service Specific
©2011 Am
azon W
eb Servi
ces L
LC or its
affiliate
s. All r
ights re
serve
d.
User
Users
Client
Multimedia
Corpora
te
data ce
nter
Traditio
nal
serve
r
Mobile C
lient
Inte
rnet
AWS M
anagement
Console
IAM A
dd-on
Exam
ple:
IAM A
dd-on
Amazo
n Mechanical T
urk
On-Dem
and Work
forc
e
Human In
tellig
ence
Tasks (
HIT)
Assignm
ent/
Task
Requester
Work
ers
Amazo
n
Mechanica
l Turk
Non-Serv
ice Specifi
c L7 load balancerReverse Proxy TLS Terminator
Contents Distribution©2011 Amazon W
eb Services LLC or its affiliates. All rights reserved.
User
Users
Client
Multim
edia
Corporate
data center
Traditional
server
Mobile Client
Internet
AWS M
anagement
Console
IAM Add-on
Example:
IAM Add-on
Amazon M
echanical Turk
On-Demand W
orkforce
Human Intelligence
Tasks (HIT)
Assignment/
Task
Requester
Workers
Amazon
Mechanical TurkNon-Service Specific
nginx Extension in mercari
• ngx_dynamic_upstream
• Operate upsteam servers Up/Down dynamically
• OpenResty (ngx_lua)
• More powerful L7 load balancer
• Application log collector
Agenda
• Continuous updating nginx
• TLS Optimization
• Automated TLS session ticket key updating
• SPDY to HTTP/2
Continuous updating nginx
Keep nginx the latest• Use nginx mainline version
• stableは安定しているという意味ではなく新機能が追加されず、バグ修正も緊急な修正を除いて行われないということ。すべての新機能やバグ修正はmainlineにて行われるので結果的にmainlineが安定しているhttps://www.nginx.com/blog/nginx-1-6-1-7-released/
• nginxは無停止でのアップデートが可能
• Periodic OpenSSL vulnerability report 😓
nginx-build
• nginx builder command written in Gohttps://github.com/cubicdaiya/nginx-build
• Automate download, configure and make
• Download OpenSSL, zlib and PCRE. build them statically ( independent from OS’s library version)
• Support Tengine and OpenResty
nginx-build$ EXPORT $VERSION=1.9.10$ nginx-build \ -clear \ -d work \ -v $VERSION \ -c nginx-build-conf/configure \ -m nginx-build-conf/modules3rd.ini \ -zlib \ -zlibversion=1.2.8 \ -verbose \ -pcre \ -pcreversion=8.38 \ -openssl \ -opensslversion=1.0.2f \$ cd work/$VERSION/nginx-$VERSION$ sudo make install
TLS Optimization
TLS Optimization
• PFS - Perfect Forward Secrecy
• TTFB - Time To First Byte
Perfect Forward Secrecy
• Mozilla Wikihttps://wiki.mozilla.org/Security/Server_Side_TLS
• Mozilla SSL Configuration Generatorhttps://mozilla.github.io/server-side-tls/ssl-config-generator/
• *最新の情報を参照し、プロダクション環境に投入する前にテストを行ってください
TTFB
• TLS Session Cache
• TLS Session Tickets
• OCSP Stapling
• SSL buffer size
TLS Session Cache
• Server side session cache
• Client software does not matter
• nginx could’t share session cache between servers.
ssl_session_cache shared:SSL:120m;ssl_session_timeout 5m;
CONS
PROS
TLS Session Cache
• OpenResty will support it?https://twitter.com/agentzh/status/686655229828403201
TLS Session Tickets
• Client Side Session Cache(like a CookieStore)
• No need to share cache between servers
• It’s required share ssl_session_ticket_key file
• Client support required
ssl_session_tickets on;ssl_session_ticket_key /path/to/ssl_session_ticket;
PROS
CONS
TLS Session TicketsUser Agent Session Tickets
Android 2.3.7 NoAndroid 4.0.4 YES
Android 5 YESChrome 47 YESFirefox 42 YESIE11/Win7 No
IE11/Win10 YESEdge/Win10 YESSafari 9/iOS9 No
Safari 9/MacOSX10.11 No!!!!!!!!
https://www.ssllabs.com/ssltest/clients.html
OCSP Stapling
https://wiki.mozilla.org/Security/Server_Side_TLS
Client required to checking the revocation status. If OCSP Responder is down, page load is failed.
Server retrieves OCSP record, caches it and servers it to client directly. Fast!
OSCP Stapling
• All Android version don’t support yet
• iOS < 9 doesn’t support it
CONS
CONS
ssl_stapling on;ssl_stapling_verify on;ssl_trusted_certificate /path/to/server.crt;resolver 8.8.8.8 valid=30s;resolver_timeout 5s;
TLS buffer size
• TLS record size. Default 16KB, it’s too large. That’s effect to latencyhttps://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/
ssl_buffer_size 1k;
Automated TLS session tickets key
updating
TLS Session Tickets
• Should share ssl_session_ticket_key file between servers
• Mozilla recommends replace it every day for forward secrecy
• Previously we changed the key file almost every month in manual operation😓
Update Tickets with Consul
• Inspired by Stretcher (fujiwara👍)
• Key_file is very small (48byte), It can store into a payload of consul event
• We’re updating key_file twice a day by using Consul
Update Tickets with Consul
JOB
Consul Servers
nginx
Consul Agent
$ consul event \ -name=“ssl-session-ticket-refresh" \ $(openssl rand 48 | base64)
nginx
Consul Agent
nginx
Consul Agent
cat | jq -r '.Payload' | \ base64 -d | base64 -d \ > /path/to/ticket.keyservice nginx reload
❶
❷ ❷ ❷
❸
SPDY to HTTP/2
SPDY is alive
Request to a webview728
1,009,268
270,527
HTTP/1.1 SPDY HTTP2
SPDY to HTTP/2
• In Mobile App, transition to HTTP/2 from SPDY is not so quickly
• Android 4.x, iOS 8...
• nginx dropped SPDY in 1.9.4, but we need to transition gracefully in a few and more years
• We need a server that speak both HTTP/2 and SPDY
Tengine
Now we’re trying Tengine (at a previous slide)
cloudflare patch?
https://blog.cloudflare.com/introducing-http2/
End