Lessons from 4 years of driver develoment

4 years of Node.js and Mongodb

Christian Amor KvalheimNow at 10gen

Friday, August 16, 13

How did I get into Node.js


Thu Jan 7 13:53:14 2010

First Commit


Growth


IT was crap

• It had a “Hash” and “Ordered Hash” class

• Complete mess with horrible structure

• Used Hacked JSpec for testing at the start


Node was liquid

• Constant changes in Node.js

• Constant changes in driver to work with Node.js

• OH yeah no NPM yet


Challenges


Types Oh Types

• No int64 support in Javascript

• Oh yeah Numbers is a Double (53 bit int max)

• Decided to use Google Long closure library class


Features

• Started adding features like GridFS in late jan 2010

• Replicaset and Mongos support in 2010/2011

• Stream support for Cursor and GridFS

• High Availability


C++

• Profiled and found BSON parser to be main bottleneck

• Wrote horrible C++ parser

• Later cleaned up by a real C++ developer from Lucasarts (RIP)


C++

• Initial version had own implementation of all BSON types

• C++ Long, Binary etc.

• Hard as hell to maintain

• Bought me nothing in the end as V8 changed


Making JS BSON fasterBuffers

• Another major refactoring to support properly

• Calculate size of Document

• Single allocation instead of multiple small ones

• Using Buffers (optimized by v8)

• Indexed entry a[0] direct memory access


Pool handling

• No traditional Connection Pool

• No checking OUT and IN Connections

• Connection pinning

• The “Spice” must flow


Write concerns

• Mongodb wire protocol

• Insert/Update/Remove + GetLastError

• Operations must be pipelined

• Write the op + GetLastError in one operation


Reconnecting

• No Blocking == difficult reconnect

• Buffer operations until reconnect

• Still a possible problem waiting for a solution


Custom Serialization off Object to BSON

• Added toBSON support

• Specify on your objects and return the true representation of the object

• Modeled after toJSON


Replicaset

• Problem

• Servers come and leave the cluster

• Need to update status of servers

• Need to pick the right server for the readPreference


Replicaset

• Single thread SUCKS

• Have to use setTimeout/setInterval

• Has tons of subtle state changes

• Never going to get Webworkers in Node.js

• Processes are to granular


Multi Platform


Windows

• Hello Windows Users

• Node.js goes Windows

• What to do About C++ ?

• Build DLL’s for win32 and win64 in Windows 7 and package with driver


Windows Build Hell

• Oh yeah Node extensions only build with Visual Studio 2010 + Windows 7 64 bit SDK

• Maybe better now but I doubt it

• Node-gyp will fail to build extensions on Win causes confusion


Windows Build Hell

• Confusion about build process on Windows

• More “Cannot” install module questions on Win than you can swing a cat at

• Mostly PBKC

• Who the Hell still uses XP (SERIOUSLY)


ARM

• Build C++ extension on ARM

• Learned about memory alignment difference between ARM and X86

• Runs on Raspberry PI (HUZZA)


Regret


Regrets

var A = function() { this.something = "a"}

A.prototype.some = function() { return this.something}


Should have

var B = function() { var something = "a"

this.some = function() { return something }}


Lesson

• If it’s possible to access internal state they will access internal state

• Only expose state and functions you wish end user should be able to use

• Examples of My BAD

• Collection class

• Cursor class


Testing It


Testing it ALL

• In the beginning

• Simple integration testing with JSpec

• Later moved to JUnit


3D Reality hitsMongo 2.0 Mongo 2.2 Mongo 2.4

Node 0.8

Node 0.10

Node 0.X

with authreplicasetsharding

+2.2 features

+2.4 features

etc etc etc

etc etc etc


A New Beginning

• New test framework

• Introduce configurations (replset/etc)

• Introduce runners (in auth mode, etc)

• Run tests against configurations

• Test does not manage configurations

• Still not there

• Need to work on the concept more


Documenting It


Documentation

• Had to provide tons of usage examples

• Realized I had them in code already

• Built some tools to help me generate most of the docs

• Spaghetti JS + Python Spinx


Documentation/** * Example of inserting a document containing functions * * @_class collection * @_function insert * @ignore */exports.test = function(configuration, test) { var db = configuration.db(); // DOC_LINE var db = new Db('test', new Server('locahost', 27017)); // DOC_START // Fetch a collection to insert document into var collection = db.collection("simple_document_insert_with_function_safe"); // Insert a single document collection.insert({hello:'world' , func:function() {}}, {w:1, serializeFunctions:true}, function(err, result) { test.equal(null, err);

// Fetch the document collection.findOne({hello:'world'}, function(err, item) { test.equal(null, err); test.ok("function() {}", item.code); test.done(); }) }); // DOC_END}


Result


Open Source


On Backwards Compatibility

• Drivers should be boring

• Change is bad

• Yeah some of the API sucks but to many depends on it

• This week CoffeeScript next week LiveScript (REALLY?)


On Managing The Project

• Challenge people to help

• Be specific about what you need

• Give credit for pull requests

• Ignore flames or criticism

• Just ignore bad faith issues

• Your code is always going to suck, roll with it


Wrapping Up


Take aways

• No Environment is Static

• Testing Drivers is Very Hard

• Don’t take criticism personally

• Challenge people to help you

• Most people are nice and will help

• Credit people who help you


In Conclusion

• It’s been A crazy 3 1/2 years of serving the Node community

• I’ve made tons of mistakes a long the way

• I’m sure to make a ton of mistakes in the future

• I would do it all again in a second


Feel free to Ask Questions


Lessons from 4 years of driver develoment

Technology

Transcript of Lessons from 4 years of driver develoment