Copyright © 2015 Criteo

Extreme Client Side Tweaking on Couchbase

Sébastien Foutrier

Lead Software Developer

June 3rd 2015

Copyright © 2015 Criteo 2

Criteo

• French start up since 2005• Keeping the spirit

• Machine Learning and Big Data oriented

• Algorithms as root

• Ad Tech business

• 230 in R&D 1500 worldwide• R&D teams in Paris

Copyright © 2015 Criteo 3

Criteo – Personalized performance advertising at scale

• Personalized• Per user per view • Real time product recommendation

• Performance• Cost of sales model• Cross device

• Scale• 994 millions unique users reach globally• 741 billions ads in 2014

Copyright © 2015 Criteo 4

Site Reliability Engineering at Criteo

50 DevOps engineers and growing with the company

Software infrastructure

Systems (CentOS, Windows, monitoring)

Automation (Chef)

Big Data infrastructure (Hadoop, Storm, Kafka, HBase)

NoSQL infrastructure (Memcached, Couchbase, Graphite,ElasticSearch, MongoDB, Cassandra, Redis)

Escalation

Predictive monitoring

Build & continuous integration

Copyright © 2015 Criteo 5

Hardware Infrastructure

• 6 Datacenters

• 10k+ server (50/50 windows/linux)

• Private worldwide dedicated network up to 10Gb/s

• One of the biggest Hadoop clusters in Europe(1k+ servers, 37 PB)

Copyright © 2015 Criteo 6

From SQL to some NoSQL

• Criteo started with• User data in client side cookies• Client and campaign configurations in MS SQL server

• First issues• RAM cache in IIS servers• Server to server calls

• Added some Memcached …. big instances

Copyright © 2015 Criteo 7

Next steps

• Real Time Bidding • 100 ms to respond

• Server to servers calls => bye bye client side cookies

• Additional challenge : multi datacenter => sync !

• Lots of instances

• Need for durability• Couchbase comes in

Copyright © 2015 Criteo 8

Grown big !

• 550 Couchbase servers (from 1.8 to 3)

• 24 clusters

• 107 TB Total RAM and SSD space

• 10M hits/sec worldwide

• 2M hits/sec peak single cluster

Copyright © 2015 Criteo 9

Things get serious, issues also

• 3.5K+ Clients

• Slow cross datacenter links (70ms)

• Blocking call

• 9K TCP sockets reached !• Client buffering blows memory/threads

Copyright © 2015 Criteo 10

Solutions ?

• Moxi• Unstable• Heavy CPU load• Bandwidth x3

• Prototype !• https://code.google.com/p/memcached/wiki/BinaryProtocolRevamped

Couchbase

Moxi Memcached

Couchbase

Moxi Memcached

Copyright © 2015 Criteo 11

Raw socket, from scratch

• Send(Byte[])• Synchronous call• request buffering• sending thread(s)

• BeginSend(…) / EndSend(…)• Overhead

• SendAsync(SocketAsyncEventArgs e)• Efficient when reusing ‘e’

Copyright © 2015 Criteo 12

The asynchronous API

if (!client.Get("Hello", (Status s, byte[] v) =>

{

if (s == Status.NoError && v != null)

Console.WriteLine("Hello, " + Encoding.UTF8.GetString(v));

else

Console.WriteLine("Get failed with status " + s);

}))

Console.WriteLine("Get failed ");

Copyright © 2015 Criteo 13

Asynchronous gain

• Less context switches• Less threads• Easy to pipeline !

Copyright © 2015 Criteo 14

TCP socket

Sequential model

Web Server Couchbase

Copyright © 2015 Criteo 15

TCP socket

Pipelining model

Web Server Couchbase

Copyright © 2015 Criteo 16

TCP socket

Socket pooling

TCP socket

Web Server Couchbase

Copyright © 2015 Criteo 17

TCP socket

TCP Windows

Web Server Couchbase

Copyright © 2015 Criteo 18

Couchbase

Redundancy (Memcached bucket)

Web Server

Couchbase

Couchbase

Couchbase

OKDrop

Z z zz

Copyright © 2015 Criteo 19

Couchbase

GetWithReplica (Couchbase bucket)

Web Server

Couchbase

Couchbase

Couchbase

OKOK

Z z zz

Copyright © 2015 Criteo 20

Benchmark

• Couchbase 1.Xusing Enyim.Caching

• Ping 277 ms(Paris-Tokyo)

• 20 nodes• 20 socket per node• 40 client threads

Copyright © 2015 Criteo 21

Benchmark

• Criteo.Memcache• Ping 277 ms

(Paris-Tokyo)• 20 nodes• 1 socket per node• 4 client threads

Copyright © 2015 Criteo 22

What is supported?

• Get/Set/Update/Add• GetAndTouch• Increment/Decrement• Ketama locator• Couchbase locator• GetWithReplica• Stats

• The features you’ll add!

Copyright © 2015 Criteo 23

Conclusion

• Open source (feel free to contribute):• https://github.com/criteo/memcache-driver

• Specific use cases• Async in Couchbase SDK 2.1

Copyright © 2015 Criteo 24

Conclusion (bis)

CRITEO IS HIRING

Copyright © 2015 Criteo

Thanks !Questions ?