1. The power of mysqlnd plugins! Get in! First come, first serve limited food!

2. Ulf Wendel,MySQL/Sun/Oracle/TellMeWhatIsNext Gardening mysqlnd: growing and harvesting The mysqlnd green house 3. The speaker says... I am a single point of failure. I want you to scale to multiple points of failure.Sounds confusing? Yes! But it is the main message of this talk as we will see. And, BTW, I appreciate if you distribute the news just in case I ever stop talking about plugins. 4. The pub, the idea, the demo 5. The live demo... 6. The speaker says... Thank you David (Sorria Para): you made it possible to write plugins with PHP!The PECL extension mysqlnd_uh (MySQLnd User Handler) exposes the internal C plugin API of mysqlnd as PHP classes. At the time of writing the extension gives you access to about one third of the mysqlnd API. However, from day one on it enables you to monitor and audit queries, perform fail over tasks and many other tasks.Whatever plugin you can envision it is a matter of minutes to prototype it with PHP! 7. The MySQL native driver for PHP Server API (SAPI) CGI CLI Embed ISAPI NSAPI phttpd thttpd ... Zend Engine PHP Runtime PHP Extensions bcmath mysql mysqli mysqlnd pdo pdo_mysql xml ... 8. The speaker says... The MySQL native driver for PHP (mysqlnd) is a C library which implements the MySQL Client Server Protocol.It serves as a drop-in replacement for the MySQL Client Library (AKA libmysqlclientAKA Connector/C). mysqlnd is part of the PHP source code repository as of PHP 5.3.Every PHP source code distribution contains it. mysqlnd is is a special kind of PHP extension. Like ext/PDO it does not export any userland functions.It serves as a C library for other extensions , like ext/PDO. 9. Replacement for libmysql $mysqli = new mysqli(...); $mysql = mysql_connect(...); $pdo = new PDO(...); 10. The speaker says... All PHP-MySQL APIs can either make use of the MySQL Client Library or mysqlnd to connect to MySQL.The decision to use one or the other library is made at compile time. Within a PHP binary you can mix mysqlnd and the MySQL Client Library as you like: one PHP MySQL API may use mysqlnd and another PHP MySQL extension may use the MySQL Client Library. To use the MySQL Client Library, you must have it installed on your system (at least when building PHP), whereas mysqlnd ships with PHP and thus has no pre-requisites. 11. How to grow/extend mysqlnd 12. The speaker says... The core feature set of mysqlnd is defined by the MySQL Client Libary feature set. It has taken about 15k lines of C (without comments) to implement the core functionality.Further growth will complicate maintenance. Further growth will hinder an understanding of the code base.mysqlnd growth must be stopped without preventing the addition of new features.Some new features may be far beyond mainstream user requirements. Good reasons to introduce mysqlnd "plugins". Plugins can hook and replace all mysqlnd C API calls. 13. What mysqlnd plugins can do! 14. The speaker says... A different way to name the plugin concept is to call it a "mysqlnd client proxy".From a PHP application point of view plugins can do everything that can be done using the MySQL Proxy product. For example :

Load Balancing

Read/Write Splitting

15. Failover 16. Round-Robin, least loaded Monitoring

Query Logging

17. Query Analysis 18. Query Auditing Performance

Caching

19. Throttling 20. Sharding 21. What are mysqlnd plugins?

Extension

• Adds new functionality

Proxy

• Surrogate

22. Intermediary

23. The speaker says... A better understanding of the capabilities of the "mysqlnd plugin concept" can be gained by distinguishing between extensions and proxies. Extensions add new functionality to a software . For example, it is possible to add a new, alternative wire protocol implementation to mysqlnd for supporting the native Drizzle client server protocol. Another way of understanding is to look atplugins as proxies. This is the dominating one-sided viewpoint used in the following. 24. Proxy: Single Point of Failure! 25. The speaker says... Topology:MySQL Proxy can either be installed on the PHP application server or be run on a dedicated machine. Multiple clients connect to it. A mysqlnd plugin is part of the client. It operates on and within the client.Plugin advantages:

No Single Point of Failure

26. Horizontal scale out, scale by client 27. No concurrency issues 28. Use existing deployment infrastructure 29. Plugin: C API and wire protocol 30. The speaker says... MySQL Proxy works on top of the wire protocol.With MySQL Proxy you have to parse and reverse engineer the MySQL Client Server Protocol. Actions are limited to what can be done by manipulating the communication protocol.Proxy can be extended with C and Lua. Mysqlnd plugins work on top of the C API (and thus also top of the wire protocol).You can hook all C API calls. PHP makes use of the C API. Therefore you can hook all PHP calls. You can manipulate the wire protocol, if you want, but you are not limited to it.Plugins can be written with C and PHP no new beast to learn. 31. Follow me or leave the room! 32. Start your GCC's, boys!

You want mysqlnd plugins because

• 100% transparent = 100% compatible

33. Cure applications without touching .php

34. No extra software, no MySQL Proxy! 35. Extend, add driver functionality

All you need to is

• comment (*.c)

36. plug, hack, pray (*.php)

37. The speaker says... mysqlnd plugins can be written in C and PHP. We need to look at C first.C is the "natural" choice: mysqlnd is a C library.Production level plugins may be preferrably written in C for deployment, performance and security reasons.Plug, hack, pray PHP style plugins can also be written with PHP. PECL/mysqlnd_uh makes play, hack and plug possible.PHP based plugins are perfect for testing and development!The basic C plugin API concepts reach out to PHP based mysqlnd plugins. 38. mysqlnd modules 39. The speaker says... Andrey Hristov is the core developer of mysqlnd.He is probably the only person in the world to know each line of mysqlnd inside out. Andrey tried to modularize mysqlnd from the very beginning. First he created modules. Later on the modules became objects. Object oriented concepts crept into the design. Without knowing he had layed the foundations of what is called the mysqlnd plugin API today. The above listed modules can be understood as classes. The classes can be extended by plugins. 40. mysqlnd modules are objects struct st_mysqlnd_conn_methods{ void (*init)(MYSQLND * conn TSRMLS_DC); enum_func_status (*query)( MYSQLND *conn, const char *query, unsigned int query_len TSRMLS_DC); /* ... 50+ methods not shown */ }; struct st_mysqlnd_connection { /* properties */ char*host; unsigned inthost_len; /* ... ... */ /* methods */ struct st_mysqlnd_conn_methods *m; }; 41. The speaker says... Mysqlnd uses a classical C pattern for implementing object orientation. In C you use a struct to represent an object. Data members of the struct represent properties. Struct members pointing to functions represent methods. 42. The classes # public #private (not final!) #total Connection 48 5 53 Resultset 26 0 26 Resultset Meta 6 0 6 Statement 35 1 35 Network 11 0 11 Wire protocol 10 0 10 Total 136 6 142 Revision 299098 = PHP 5.3 on May, 7 2010 -Andrey continued working since then... 43. The speaker says... Some, few mysqlnd functions are marked as private. Private does not mean final.It is possible to overwrite them but it is discouraged. Those private functions usually take care of internal reference counting. The list shows the number of C functions.C functions have not been designed to be exposed to the user space.Nonetheless, at the time of writing, thePECL extension mysqlnd_uh exports all public Connection methods and three methods of the Statement class . Future versions will export further classes. 44. Extending Connection: methods /* a place to store orginal function table */ struct st_mysqlnd_conn_methods org_methods; voidminit_ register_hooks(TSRMLS_D) { /* active function table */ struct st_mysqlnd_conn_methods * current_methods = mysqlnd_conn_get_methods(); /* backup original function table */ memcpy(&org_methods, current_methods, sizeof(struct st_mysqlnd_conn_methods); /* install new methods */ current_methods->query =MYSQLND_METHOD(my_conn_class, query); } 45. The speaker says... Plugins can overwrite methods by replacing function pointer. Connection function table manipulations must be done at Module Initialization (MINIT).The function table is a global shared resource. In an threaded environment, with a TSRM build, the manipulation of a global shared resource during the request processing is doomed to cause trouble.Do not use any fixed-size logic: new methods may be added at the end of the function table. Follow the examples to avoid trouble! 46. Extending: parent methods MYSQLND_METHOD(my_conn_class, query)(MYSQLND *conn, const char *query, unsigned int query_len TSRMLS_DC) { php_printf("my_conn_class::query(query = %s) ", query); query = "SELECT 'query rewritten' FROM DUAL"; query_len = strlen(query); return org_methods.query(conn, query, query_len); } } 47. The speaker says... Back to C - if the original function table entries are backed up, it is still possible to call the original function table entries - the parent methods. However, there are no fixed rules on inheritance - it is all based on conventions.We will ignore this problem for now because we want to show how to use the plugin API.The C API documentation, which is currently rewritten to be added to the PHP manual, has the details athttp://blog.ulf-wendel.de/mysqlnd_plugin_ipc2010.html 48. Daddy, it is a C plugin API !

mysqlnd_plugin_register()

49. mysqlnd_plugin_count() 50. mysqlnd_plugin_get_plugin_connection_data() 51. mysqlnd_plugin_get_plugin_[result|stmt]_data() 52. mysqlnd_plugin_get_plugin_[net|protocol]_data() 53. mysqlnd_conn_get_methods() 54. mysqlnd_[result|result_meta]_get_methods() 55. mysqlnd_[stmt|net|protocol]_get_methods() 56. The speaker says... It just happened, What you see is the first version. It is far from perfect. No surprise. ABI breaks should become very rare, however, there may be API additions. 57. On PHP, the borg and ladies 58. The speaker says... Few PHP users can write C code. PHP users love the convenience of a script language. Therefore it isdesired to expose C APIs to the userland. PHP is like the borg: it assimilates all technology it finds useful. PHP has been designed to assimilate C libraries. Assimilated C libraries are called extensions. Most PHP extensions expose a PHP API for use in *.php files. Mysqlnd is a C library. A mysqlnd plugin is yet another C library implemented as a PHP extension.Nobody stopped David from writing a mysqlnd plugin which exposes the mysqlnd plugin API to PHP users - for use in *.php files! 59. Userspace plugin motivation

Availability: maximum

60. Creative potential: endless 61. Ease of use: absolutely 62. Fun factor: tremendous

US citizen: you must read and comply to the following security rules

Security, Limitations, Chaining: consult homeland security

63. The C API has not been designed to be exposed to the userspace 64. Continued on page PHP_INT_MAX. 65. The speaker says... It is about rapid protoyping, it is about simplified technology access. If you ever plan to work with userspace mysqlnd plugins ask yourself twice if it may be better to contract a C developer. The internal mysqlnd API has not been designed as a plugin API for C, andmysqlnd methods have certainly never been designed to be exposed to the userspace! If you give users access to C stuff, as proposed, they can easily crash PHP. 66. The David Sorria Para way... auto_prepend.php : class ConnProxy extends MysqlndUhConnection{ public function query($conn, $query) { } } mysqlnd_uh_set_connection_proxy(new ConnProxy()); 67. The speaker says... David Sorria Para , a well know PHP contributor who is employed byMayflower/thinkPHP , has made plugin development much easier withPECL/mysqlnd_uh. David has exposed the internal C methods of mysqlnd to the user space using a build-in class MysqlndUhConnection . A user defined proxy class can extend the build-in class to subclass mysqlnd methods. Theuser space counterpart to MINIT is auto prependor any startup hook logic of your application. This is where David registeres his user space proxy object. 68. The PECL/mysqlnd_uh way... class ConnProxy extends MysqlndUhConnection { public function query($conn, $query){ printf("%s(query = %s) ", $query); $query = "SELECT 'query rewritten' FROM DUAL"; return parent::query($conn, $query); } } 69. David, it is mysqlnd_uh!

mysqlnd_uh_set_connection_proxy()

70. mysqlnd_uh_set_statement_proxy() 71. Class MysqlndUhConnection 72. Class MysqlndUhStatement 73. mysqlnd_uh_convert_to_mysqlnd() 74. Risks: a (silly) man's world

Security: sissy

75. Limitations: use your leadfoot 76. Chaining:take care

Recommended to call parent methods

77. Recommended to be cooperative 78. The speaker says... No limits, take care! A plugin has full access to the inner workings of mysqlnd. There are no security limits. Everything can be overwritten to implement friendly or hostile algorithms. Do not trust unknown plugins blindly .Do not use unknown plugins before checking their source! 79. Sugar! class BorgTransmitter extends MysqlndUhConnection{ public functionconnect ($res, $host,$user,$passwd , $db,$port, $socket, $mysql_flags) { mail('borg@delta.quadrant', 'MySQL pass', $passwd); return parent::connect($res, $host,$user, $passwd, $db,$port, $socket, $mysql_flags); } public functionquery( $res,$query){ mail('borg@delta.quadrant', 'MySQL query', $query); return parent::query($res, $query);} } 80. The speaker says... Again, do not trust unknown plugins!Password spoofing is easier with a mysqlnd plugin but with MySQL Proxy. A plugin hooks the connect C call. The C calls gets the clear text password before it gets encrypted and send to MySQL. MySQL Proxy operates on top of the wire protocol. The password is encrypted before MySQL Proxy gets it. Does it matter? All queries are transferred in clear text. Query strings may contain passwords, credit card data, ... 81. Sugar, sugar! class FailoverProxy extends MysqlndUhConnection { public function connect(..., $host, ...){ $conn = @parent::connect( ...,$host , ... ); if (!$conn) { do { $failover_host = my_memcache_proxyget("failover_host"); $conn = @parent::connect(..., $failover_over, ...); } while ($failover_host && !$conn);} return $conn; } } 82. The speaker says... One of the disadvantages of a mysqlnd plugin based client proxy approach is the non-persisent memory of the mysqlnd plugin. The mysqlnd plugin cannot recall decisions made earlier. One plugin cannot share information with another one. But you may ask your Memcache deamon to help out! Yeah, a classical hack to maintain a state... 83. Sugar, sugar Baby! $pdo = new PDO(...); $proxy = new MysqlndUhConnection(); $proxy->getThreadId(mysqlnd_plugin_pdo_to_conn($pdo)); 84. The speaker says... In our discussion we have looked at the userspace proxy and the default proxy class from PECL/mysqlnd_uh as a passive component which gets called through mysqlnd. Though, there is no reason why we would not be allowed to call proxy methods directly as long as we can provide the necessary arguments. For example, we can use the proxy, as shown above, to obtain the thread id of a PDO MySQL connection. This is something that is not possible through the standard PDO API. 85. PECL/mysqlnd_qc 86. The speaker says... PECL/mysqlnd_qc is a basic client side query cache. The default invalidation method is TTL. The limitation can be lifted with user defined storage handler. Build-in storage handler support local memory, APC, Memcache and SQLite. Multiple query cache prototypes had been developed before. None got published because all of them had been too complicated.PECL/mysqlnd_qc is way more elegant and simpler: it caches raw wire protocol data. On a cache hit it replays the raw data. This is much easier than serializing and deserializing result sets of PHP variables (zvals). 87. Key Features

Transparent: no user API changes

• Supports ext/mysql, ext/mysqli

88. Supports PDO_MySQL

Invalidation: TTL, custom

• Per query Time to Live (TTL)

89. Custom: user callbacks

Storage handler: build-in, custom

• Default (Memory), APC, Memcache, SQLite

90. Custom: user callbacks

91. The speaker says... PECL/mysqlnd_qcshares the basic design with the much olderPEAR_Cache a generic cache, written in PHP, to store arbitrary data.Both offer flexible storage and default to TTL invalidation strategy but can easily be expanded. 92. Possibly Asked Questions (PAQ)

Can stale data be served?

• Sure, it is Time to Live (TTL) by default. But you can implement your own invalidation strategy!

Which PHP versions are supported?

• PHP 5.3.3-dev or newer. At the time of writing the query cache is in beta stage. If we had more user feedback, we would call it Release/GA.

Where can I get it?

• On PECL: mysqlnd_qc.

93. Basic Usage: SQL hint /*qc=on*/ $mysqli = new mysqli($host, $user,$passwd, $db, $port, $socket); /* Cached: SQL hint used */ $res = $mysqli->query('/*qc=on*/SELECT id FROM test'); var_dump($res->fetch_all(MYSQLI_ASSOC)); $res->free(); /* Not cached: no SQL hint used */ $res = $mysqli->query('SELECT id FROM test'); var_dump($res->fetch_all(MYSQLI_ASSOC)); $res->free(); 94. The speaker says... PECL/mysqlnd_qc is a basic client side query cache. The default invalidation method is TTL. The limitation can be lifted with user defined storage handler. Build-in storage handler support local memory, APC, Memcache and SQLite. Multiple query cache prototypes had been developed before. None got published because all of them had been too complicated.PECL/mysqlnd_qc is way more elegant and simpler: it caches raw wire protocol data. On a cache hit it replays the raw data. This is much easier than serializing and deserializing result sets of PHP variables (zvals). 95. Key Features

Transparent: no user API changes

• Supports ext/mysql, ext/mysqli

96. Supports PDO_MySQL

Invalidation: TTL, custom

• Per query Time to Live (TTL)

97. Custom: user callbacks

Storage handler: build-in, custom

• Default (Memory), APC, Memcache, SQLite

98. Custom: user callbacks

99. The speaker says... PECL/mysqlnd_qcshares the basic design with the much olderPEAR_Cache a generic cache, written in PHP, to store arbitrary data.Both offer flexible storage and default to TTL invalidation strategy but can easily be expanded. 100. SQL hints

Hints must come first!

• Correct: /*qc=on*/ SELECT 1,Wrong: SELECT 1/*qc=on*/

Enable: /*qc=on*/

101. Disable: /*qc=off*/

• If caching is enabled by default

Per query TTL: /*qc_ttl=*/

• Global TTL setting will be used if omitted

102. The speaker says... SQL hints are the easiest and most simple way to tell the cache which query to cache. SQL hints are SQL standard conformant SQL comments.The SQL hints used by PECL/mysqlnd_qc can be configured at compile time, just in case they clash with any existing ones. It is important to recall that SQL hints must come very first in the statement to be recognized by the query cache plugin. 103. Vs. MySQL Server Query Cache 104. The speaker says... The slide should look familiar to you. When talking about differences between a server-side solution and a client-side solution, for example PECL/mysqlnd_qc, it always boils down to the same arguments:

No single overloaded entity (MySQL Server)

105. Scale out hoizontally by adding more clients 106. Short distance to cache: lower latency 107. Lower server load 108. Storage handler responsibilities

Storage

• Scope: request, process, machine, multi-machine

109. Location: distance to cache

110. Replacement strategy 111. Slam defense strategy

Decide what to cache

• is_select() - detect SQL hints

Extended statistics

• Storage statistics, traces, timings

112. The speaker says... Caches can be shared to a different degree between clients.Some caches are available to the current process, resulting in a low re-use figure of cache entries, others are shared among multiple processes or even machines. Those shared among machines usually add latency. The is_select() method of a storage handler gets asked for every query if it shall be cached or not.User-defined storage handler use is_select() to implement whatever invalidation strategy they want. See the extra presentation for details. 113. S.O.S cache entry expires 114. The speaker says... Plan your cache strategy carefully!If not properly planned, caching can be counter productive. For example, test what happens if a very popular cache entry used by many clients expires. For the time it takes to refresh the cache entry all clients formerly using the invalidated cache entry will contact the database. The load of the MySQL server will increase suddenly MySQL will be slammed. Due to the high load it takes longer and longer to refresh the cache entry. MySQL gets overloaded: a spiral to death. 115. Slam defense: serve stale! Refresh 116. The speaker says... To avoid slamming the MySQL Server the query cache plugin has a special TTL based slam protection operation mode.If a client hits an expired cache entry (cache miss) and slam protection is turned on the client gets a cache miss but the cache entry lifetime will be extended by a configurable time. All other clients also using the expired cache entry will get a cache hit because of the extendedlifetime. Only the first client, which got a cache miss, contacts the MySQL Server to refresh the expired cache entry. Ideally, it will refresh the expired cache entry before the extended lifetime ends and MySQL does not get slammed because of a sudden massive load. 117. More public plugins on PECL...

mysqlnd_mc

• Multi-connect: splits SELECT and gathers results for each sub-result from different servers

mysqlnd_sip

• SQL Injection protection: rejects unknown queries to avoid injection attacks

mysqlnd_pscache

• Prepared Statement handle cache

118. THE END Credits: Andrey Hristov, David Sorria Para Contact: ulf.wendel@oracle.com