Protecting MySQL network traffic

Daniël van Eeden | 25 April 2017

Booking.com at a glance● Started in 1996; still based in Amsterdam

● Member of the Priceline Group since 2005 (stock: PCLN)● Amazing growth; continuous scaling challenges

● Online Hotel/Accommodation/Travel Agent (OTA):● Over 1.2 million active properties in 227 countries● Over 1.2 million room nights reserved daily● 40+ languages (website and customer service)● Over 13,000 people working in 187 offices in 70 countries

● We use a lot of MySQL and MariaDB:● Thousands (1000s) of servers, ~90% replicating● >150 masters: ~30 >50 slaves & ~10 >100 slaves

2

Why protect MySQL network traffic?

● Protect leaking of authentication data (passwords, etc)● Protect leaking of sensitive data (PII, credit card numbers,

medical records)● Ensure data is not tampered with.● Because of regulations● Because why not? Are you still using telnet to manage

servers?

How?● Use SSL!● Done!

SSL Support in MySQL● MySQL doesn't have SSL support● MySQL never had any SSL support● MySQL has TLS support.. this is what is called SSL but isn't● Supported since 4.0.0 (~2003)● For now just assume SSL and TLS are the same

What is NOT protected by TLS● Data-at-rest

○ InnoDB and MyISAM data files○ Binlogs, redo logs, slow query logs○ Backups

● Does not protect against a DoS○ e.g. corrupting traffic

● Might not protect the query text○ performance_schema etc.

● Does not hide the traffic pattern

First steps with TLS1. Get a certificate2. Restart MySQL3. Enable TLS on the client4. Check if the connection actually uses TLS

Generating the certificate● With 5.7 and up: Might already be done by your installation● If not use mysql_ssl_rsa_setup● For older versions: https://github.com/dveeden/mysslgen● Or use the openssl commandline utilities as described in the reference

manual on https://dev.mysql.com/doc/mysql/en/creating-ssl-files-using-openssl.html

● Did you know MySQL Workbench has a SSL Wizard?

Configuration● On 5.7+: Place the ca.pem , server-cert.pem and

server-key.pem in your datadir. (already the case if you use mysql_ssl_rsa_setup)

● Or set ssl-ca , ssl-key , ssl-cert in your my.cnf● Restart MySQL● Enable SSL in your application. You probably want to copy

your ca.pem file to your client

Checking your connection

● 'status' or \s● Look for 'Cipher in use'● Or check the 'Ssl_cipher' session

status.

What if it doesn't work?● Check your mysqld.log● Check the permissions on the pem files

○ Should be readable for the mysql user● Try to connect with --ssl-mode=REQUIRED● Use the OpenSSL commandline tools to see what's in the certificate.● Use Wireshark.

ERROR 2026 (HY000): SSL connection error: error:00000001:lib(0):func(0):reason(1)ERROR 2026 (HY000): SSL connection error: unknown error number ERROR 2026 (HY000): SSL connection error: SSL certificate validation failureERROR 2026 (HY000): SSL connection error: SSL_CTX_set_default_verify_paths failedERROR 2026 (HY000): SSL connection error: protocol version mismatchERROR 2026 (HY000): --ssl-mode=REQUIRED option forbids non SSL connectionsERROR 2026 (HY000): SSL connection error: Failed to set ciphers to useERROR 2026 (HY000): SSL connection error: Unable to get certificateERROR 2026 (HY000): SSL connection error: Unable to get private key

Now let's make it more secure● Require the use of TLS on the server● Require the use of TLS on the client● Enable more security checks● Security updates

Make TLS a requirement● On a per user basis:

○ ALTER USER foo REQUIRE SSL○ Undo: ALTER USER foo REQUIRE NONE

● But what happens if you accidentally create a user?○ e.g. GRANT on a nonexistent user?○ Set: sql_mode=NO_AUTO_CREATE_USER,…

● On a server level:○ SET GLOBAL require_secure_transport=ON○ This still allows UNIX socket connections w/o TLS

Issues with full-on TLS● Is your monitoring capable of using TLS connections?● What about load balancer health checks?

On the client● Use --ssl-mode=REQUIRED

○ The default in 5.7 is PREFERRED○ Older releases default to DISABLED

● This only makes a TLS connection a requirement○ Does not check if issued by a trusted CA○ Does not check if the hostname matches the cert○ To do this use VERIFY_CA or VERIFY_IDENTITY

● On older versions:○ Use --ssl-ca to allow TLS and enable CA checks○ Use --ssl-verify-server-cert to do hostname checks.○ Often not possible to force the use of TLS: this is the

BACKRONYM vulnerability● Use --ssl-ca=/path/to/ca.pem to specify which CA(s) are

trusted.

Client checks● The client could do these checks:

○ Is the certificate signed by a trusted CA?○ Does the CommonName (CN) in the certificate match

the hostname we are connecting to?○ Is the certificate expired?

Certificate Authority validation● Validates that the server certificate is signed by one of the CA's

present in the specified CA file.● Note that a CA file can have multiple CA's● There is also a CA path option.● The auto generated certificates from mysql_ssl_rsa_setup all

have their own CA.

Hostname validation● mysql_ssl_rsa_setup generates certificates with

○ CN=MySQL_Server_5.7.18_Auto_Generated_Server_Certificate● So generate the certificates manually if want this to match your

hostname● A certificate can have a list of hostnames in SubjectAltName

○ MySQL doesn't check those... Bug #68052● So if you use a virtual-IP, cname, etc. it might be difficult to match

this.● What if your clients connect on a CNAME and your replicas connect

on the hostname? You can't have both!

Security updates● I reported a few issues to Oracle.● CVE-2017-3590 for Connector/Python● CVE-2017-3469 for MySQL Workbench● CVE-2017-3467 for libmysqlclient● Those are fixed. See the Oracle Critical Patch Update for

details.● But if you care about security you should follow the release

notes and Critical Patch Update anyways...

What library does MySQL use?● Community Edition: YaSSL

○ Because GPL and the OpenSSL license are not really compatible

○ This library is maintained by WolfSSL○ This not CyaSSL/WolfSSL○ WolfSSL made a patch to include WolfSSL in MySQL

5.6.30 (https://github.com/wolfSSL/mysql-patch)● Enterprise Edition: OpenSSL● If you build MySQL yourself: you can compile against either

of them.

Why not TLS?● Because it is SLOW!● Because we trust our network!● Because we encrypt with:

○ The application (store encrypted data)○ SSH (Also works great with Workbench)○ VPN

● Because we want to inspect our network traffic!○ Wireshark can decrypt it if you hand over your private

key. Some ciphers require you to somehow extract session keys.

How slow is slow?● Overhead in milliseconds for

setting up a TLS connection on localhost with TCP.

● Client: go● 5.7 is faster than 5.6● OpenSSL is faster than YaSSL● Using TLS tickets (OpenSSL only)

helps● Best case: 0.99ms (5.7 OpenSSL w/

tickets) vs. 0.60ms (no TLS)● TLS does need more roundtrips,

but this will change with TLS 1.3● OpenSSL performs better because

it uses AVX2 and AES-NI

Bulk transfer performance● Easy to test: mysqldump with and without TLS● Different ciphers do make a difference.

mysqldump performance with MySQL 5.6.35 (YaSSL)

No TLS 4.5s

TLS Default 10.4s

RC4-MD5 7.1s

DES-CBC3-SHA 23.2s

Monitoring● Monitor the Expiry of certificates● Not just the certificate on disk, also the one in memory.● Use TLS for your monitoring on 5.6 and earlier, otherwise

you might not see the status vars● Performance schema can show you the ciphers and TLS

versions in use by all connections● Using SYS is even easier:

○ SELECT * FROM session_ssl_status

Client certificates● This allows mutual authentication● Often used together with a password● You might want to use REQUIRE SUBJECT or REQUIRE

ISSUER on accounts.● At least use REQUIRE X509 instead of REQUIRE SSL

Replication● Use CHANGE MASTER TO MASTER_SSL=1, etc● Think about what happens if your certificate(s) expire● Does the hostname match the certificate?

Changing certificates● Needs restart● Moving slaves around might not work until you restart..● Same for a switchover.

CRL and OCSP● Only possible with OpenSSL● Does not auto download the CRL from the distribution

point● Does not use OCSP● Basically restart MySQL every time your CRL changes.. which

is not practical

Where to get your certificate?● Official CA?● Internal CA?● Self signed?

TLS handshake with MySQL● server helo with ssl flag set● 'empty' login packet with ssl flag set● Start SSL handshake● Basically STARTTLS-ish

○ SSL and non-SSL on the same port

Protection of authentication data● native password with nonce● sha256 password with RSA keys or TLS● cleartext plugin

TLS ciphers● Possible to set restriction on Server and Client● How are you going manage and maintain that?● 'REQUIRE cipher' also requires client certificates● One practical use case would be to use a faster cipher for

mysqldump● Might help with compliance● 5.7.10 already places more strict requirements on the list of

ciphers

TLS versions● Can be limited on the server and client● Note that YaSSL only has TLS 1.0 and TLS 1.1 support● Minimum is TLS 1.0

What about MariaDB?● Doesn't use --ssl-mode ● Does have good TLS support● MariaDB Connector/C has support for

○ fingerprint verification○ password protected private keys

● 19 Open MDEV's tagged with SSL

Connector support● Works for C, C++, Python (multiple), Perl, Java, ODBC, Go, etc● The Go MySQL driver lets you specify a TLS Config, which is

really flexible.● Do update your Connector.. Many connectors did have

security updates related to TLS.

Don't forget these● MySQL Cluster (NDB) communication within the cluster● Galera communication● Sending backups to a central location (xbstream etc)● Network traffic for iSCSI, FCP, NFS

Future● TLSv1.3 with 0-RTT● WolfSSL?

Oh, and Booking.com is hiring!● Almost any role:

● MySQL Engineer / DBA● System Administrator● System Engineer● Site Reliability Engineer● Developer● Designer● Technical Team Lead● Product Owner● Data Scientist● And many more…

● https://workingatbooking.com/39

Thank you!

All references to “Booking.com", including any mention of “us”, “we” and “our” refer to Booking.com BV, the company behind Booking.com™

Title● Item● Item