System design and maintenance Real Time Bidding · System design and maintenance rg-dev#3 -...
Transcript of System design and maintenance Real Time Bidding · System design and maintenance rg-dev#3 -...
~ # whoamiKonrad Kaplita
CTO at AppliscaleSoftware Engineer (Erlang)Operations Engineer (AWS)
https://twitter.com/konradkaplitahttps://pl.linkedin.com/in/konradkaplitahttps://github.com/konradkaplita
Hire us! www.appliscale.io/contactJoin us! www.appliscale.io/career
RTBReal Time Bidding
Every AD placeholder can be a fierce battlefield where different advertisers try
to outbid each other
source: http://www.iab.net/media/file/OpenRTB-API-Specification-Version-2-3.pdf
HTTP layer● Thousands concurrent HTTP requests per host● More than 50 pools of connections to DSP partners● Up to 50 connections per single HTTP pool
Issues with network infrastructure
● Undersea fiber maintenance● Rogue fiber shooters● Classical fiber cuts● ISP datacenter floods● BGP issues - “scenic packet routing” included free of
charge
Issues with external partners
● Datacenter outages● AWS misconfigurations● HTTP protocol abuse
○ 204 No Content with content○ 204 Pas de Contenu instead of No Content
● Death by Time-Wait
Challenges● Soft real time - users hate waiting for ads to display
(and eventually install adblock)● Concurrency - sending bid requests to multiple partners at the same time● Fault tolerance - we cannot crash the entire service because of a simple
bug in business logic● Fast feature delivery - simple, frequent and safe deployments● System awareness - with such big and complex system it is really hard to
know what is the status of all parts● Failure detection - when something breaks identify what was that
● Functional language● Supports actor model concurrency● Developed in 1986, open source since 1998● Soft real time● Distribution● Hot code reload● Garbage collection● Pattern matching● Runs on Erlang VM (BEAM)
Datadog
● Simple client-server architecture● Very stable agent● Lightweight on resources● Tons of out-of-the-box integrations (MySQL, SNMP)● If you have big infrastructure it will cost you● You have to decide if it’s worth it
https://www.datadoghq.com
Jenkins job-dsl
● Keep your jenkins jobs defined as groovy scripts● Configuration as code● All job changes are tracked and versioned● PR review process for job changes (important on production!)● Simplify jobs by extracting common parts● Bootstrapping of Jenkins infrastructure is a breeze
https://github.com/jenkinsci/job-dsl-plugin
Ansible
● One of many automation frameworks out there● Simple, declarative syntax (YAML format)● No need for agent modules● Only dependency is SSH and Python● Can run deployments simultaneously to hundreds of machines
https://www.ansible.com/