Fastly - Inaugural NYC Varnish Meetup
-
Upload
fastly -
Category
Technology
-
view
335 -
download
5
description
Transcript of Fastly - Inaugural NYC Varnish Meetup
Varnish NYC Meetup | 23 July 2014
Varnish | NYC Meetup23 July 2014
Varnish NYC Meetup | 23 July 2014
• First major user of Varnish
• Varnish contributor
• Fastly Founder and CEO
• @crucially
• Please ask questions
Artur Bergman
Varnish NYC Meetup | 23 July 2014
• Large distributed Varnish
• Realtime
• Purging
• Logfiles
• Config
• Stats
•Since July 2011
Fastly
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• Reverse proxy
• Sits between you and origin/backends
• Caches content
• Load balancer
• Advanced configuration language
Varnish
Varnish NYC Meetup | 23 July 2014
FASTLY
VCL
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• Simple scripting language
• Compiles into c code
• Shared library dynamically loaded into app
VCL
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
sub vcl_recv { if (req.http.Accept-Encoding) { if (req.http.User-Agent ~ "MSIE 6") { unset req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } }
VCL
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• “The Vary field value indicates the set of request-header fields that fully determines, while the response is fresh, whether a cache is permitted to use the response to reply to a subsequent request without revalidation. For uncacheable or stale responses, the Vary field value advises the user agent about the criteria that were used to select the representation"
• Second level cache key
• You need to look up the object, get the vary signature, then compute the vary and match it
Vary
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• varnishlog -i RxHeader | grep Accept-Encoding: | head -1000 | cut -c 22- | sort | uniq -c | sort -rn
• 399 Accept-Encoding: gzip, deflate• 285 Accept-Encoding: gzip• 262 Accept-Encoding: gzip,deflate,sdch• 42 Accept-Encoding: gzip,deflate• 3 Accept-Encoding: identity• 3 Accept-Encoding: deflate, gzip• 1 Accept-Encoding: identity,gzip• 1 Accept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3• 1 Accept-Encoding: gzip,deflate,lzma,sdch• 1 Accept-Encoding: gzip, deflate, identity• 1 Accept-Encoding: gzip, deflate, compress• 1 Accept-Encoding:gzip
Vary - Accept Encoding
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
Varnishlogvarnishlog -i RxHeader
| grep Accept-Encoding:
| head -1000 | cut -c 22- | sort | uniq -c | sort -rn
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• So should be 1-4 different versions
• Vary is literal
• 10 different version!• Massive DOS vector
• Linear scan across all Varys
Vary - Accept Encoding
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
sub vcl_recv { if (req.http.Accept-Encoding) { if (req.http.User-Agent ~ "MSIE 6") { unset req.http.Accept-Encoding; } elsif (req.http.Accept-Encoding ~ "gzip") { set req.http.Accept-Encoding = "gzip"; } elsif (req.http.Accept-Encoding ~ "deflate") { set req.http.Accept-Encoding = "deflate"; } else { unset req.http.Accept-Encoding; } }
VCL
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• Bounded number of possible combinations
• Vary: User-Agent, Referer, Accept-Encoding, Cookie • Seen in the wild!
• Be careful with what you let into you vary
Vary - Accept Encoding
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
sub vcl_recv { set req.http.Vary-Cookie = regsub(req.http.Cookie, “.*(userid=[^;]+).*”, “\1”);}
sub vcl_fetch {set beresp.http.Vary = “Vary-Cookie”;
}
VCL
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• Ignore cookies from• ad networks• google analytics• random 3rd party tracking cookies
• Cache-Control: private for cookie content • Now you have hitpass objects for logged in users
Vary - Cookie
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• return(pass) in vcl_recv• bypass all caching• bypass request collapsing• bypass hash lookup
• return(pass) in vcl_fetch• Cache that the object is uncacheable• On next request will closely follow vcl_recv return(pass)• Except it does go into vcl_hash
• Useful to avoid bad request collapsing
Hitpass
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
Inline CC{asm ("\t.LD2496:\n""\t.string \”xxxx.services.fastly.net\"\n""\t.LD2497:\n""\t.string \”yyy.services.fastly.net\"\n""\t.LD2498:\n""\t.string \”zzzz.services.fastly.net\"\n""\t.LD2499:\n"\tcmpb $103, %r14b # g\n""\tleaq .LS5230(%rip), %r8 # xxxxxx -> 5705\n""\tleaq .LD23896(%rip), %r9\n""\tje compare # foo\n""\tcmpb $114, %r14b # r\n""\tje L177\n""\tcmpb $48, %r14b # 0\n""\tleaq .LS1407(%rip), %r8 # zzzzzzzzzz -> 2264\n""\tleaq .LD9381(%rip), %r9\n""\tje compare # foo\n"
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
Inline C
Varnish NYC Meetup | 23 July 2014
FASTLY
Purging
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• Cache forever• Purge on changes• Hook into CMS/Wordpress/Whatever• Fastly supports it :)
Cache invalidation
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• Tag responses• Surrogate-Key: objectid userid templateid
• Purge by tag
• ban obj.http.surrogate-key ~ “objectid”
Surrogate key purging
Varnish NYC Meetup | 23 July 2014
FASTLY
Running Varnish
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• -p thread_pool_stack=524288• Save ram
• -p syslog_cli_traffic=off• Don’t block if syslog hangs
• -p thread_pool_max=32000• Depends on your count of CPU
• -p thread_pools=8• -p thread_pool_min=1000• -p thread_pool_add_delay=0
• Don’t spend time waiting for threads when you need them
Start parameters
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• -s file,<dir_or_file>,<size>
• Has concurrency problems• Holds lock while paging (reading) from disk!
• Solution, use many per disk (or rewrite it)• (And obviously use a SSD)
Disk storage
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• vm.dirty_background_bytes = 531838208• vm.dirty_expire_centisecs = 3000• vm.dirty_ratio = 50• vm.dirty_writeback_centisecs = 500• vm.lowmem_reserve_ratio = 1 1 32• vm.max_map_count = 524288• vm.min_free_kbytes = 15000000• vm.mmap_min_addr = 65536• vm.swappiness = 0• vm.zone_reclaim_mode = 0
sysctl
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• vm.dirty_background_bytes = 531838208• vm.dirty_expire_centisecs = 3000• vm.dirty_ratio = 50• vm.dirty_writeback_centisecs = 500• vm.lowmem_reserve_ratio = 1 1 32• vm.max_map_count = 524288• vm.min_free_kbytes = 15000000• vm.mmap_min_addr = 65536• vm.swappiness = 0• vm.zone_reclaim_mode = 0
Control writeback
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• vm.dirty_background_bytes = 531838208• vm.dirty_expire_centisecs = 3000• vm.dirty_ratio = 50• vm.dirty_writeback_centisecs = 500• vm.lowmem_reserve_ratio = 1 1 32• vm.max_map_count = 524288• vm.min_free_kbytes = 15000000 <<< WARNING• vm.mmap_min_addr = 65536• vm.swappiness = 0• vm.zone_reclaim_mode = 0
keep memory free
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• vm.dirty_background_bytes = 531838208• vm.dirty_expire_centisecs = 3000• vm.dirty_ratio = 50• vm.dirty_writeback_centisecs = 500• vm.lowmem_reserve_ratio = 1 1 32• vm.max_map_count = 524288• vm.min_free_kbytes = 15000000• vm.mmap_min_addr = 65536• vm.swappiness = 0• vm.zone_reclaim_mode = 0
Don’t fucking swap
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• 2 Intel 2690 v2 (Sandy Bridge)• 10 Cores @ 3 Ghz
• 768 GB of Ram• 4x10Gb Ethernet EB82599• 24*500GB SSD
• Intel 3500• Samsung 840 Pro
Hardware
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
• 16 Servers
• 12 TB Ram• 192 TB of SSD• 640 Gbit/sec
Rack
Varnish NYC Meetup | 23 July 2014Varnish NYC Meetup | 23 July 2014
Varnish NYC Meetup | 23 July 2014
FASTLY
Questions?
Varnish NYC Meetup | 23 July 2014
FASTLY
Bonus!
Varnish NYC Meetup | 23 July 2014
FASTLY
Thank you.