Versant Innovation Leveraging your Knowledge of ORM Towards Performance-based NoSQL Technology

Versant Corporation U.S. Headquarters 255 Shoreline Dr. Suite 450, Redwood City, CA 94065 www.versant.com | 650-232-2400

NoSQL at it’s Core

Pole-Position – Overview

About the Code

Circuits Description

RDB JPA Code

MongoDB Code

Versant JPA Code

Results – Winners of the Race

Developer Challenge

Overview

NoSQL at its Core

A Shift In Application Architecture

• Google – Soft-Schema • IBM – Schema-Less

Inefficient CPU destroying

Mapping

Excessive Repetitive data

movement and JOIN calculation

Why the Shift is Needed

• Think about it – How Often do Relations Change? – Blog : BlogEntry , Order : OrderItem , You : Friend

Stop Banging Your Head on the Relational Wall Relations Rarely Change, Stop Recalculating Them You don’t need ALL your Data, you can distribute

Measured Value

How NoSQL performance stacks up to Relational

PolePosition Performance Benchmark

• Established in 2003 – NoSQL -vs- ORM

• Code Complexity Circuits – Flat Objects – Graphs – Inherited Objects – Collections

• Data Operations – CRUD – Concurrency

• Scalability

Contenders | Methodology

• Contenders – ORM

• Hibernate – MySQL / Postgres • OpenJPA – MySQL / Postgres (other RDB, cannot publish – same

basic results) – NoSQL

• MongoDB • Versant Database Engine

• Methodology – External RDB Experts, Internal NoSQL Experts – Open Source Benchmark Code for all contenders

About the Code

• ORM - Common Code Base – Hibernate – MySQL / Postgres – OpenJPA – MySQL / Postgres – MongoDB – Versant

Circuit Structure • Simulate CRUD on embedded graph of different Classes. • Class model - Complex Circuit: class Holder0 { String _name; List <Holder0> _children; Holder0[] _array; } class Holder1 extends Holder0 { int _i1; } class Holder2 extends Holder1 { int _i2 <indexed>; } ...class HolderN extends…..

Hibernate JPA Code

• Write – Generates Holder’s to user spec depth ComplexHolder cp = new ComplexHolder( ); em.makePersistent(cp);

• Read – Access root of graph and traverse cp = em.find( ComplexHolder.class, id ); cp.getChildren().touch();

• Query String query = "from org.polepos.teams.hibernate.data.Holder2 where

i2=" + currentInt; Iterator it = em.iterate(query);

• Delete – Deletes Graph - cascading operation

ORM JPA Mapping

• - <hibernate-mapping package="org.polepos.teams.hibernate.data" default-cascade="none" default-access="property" default-lazy="true" auto-import="true"> • - <class name="ComplexHolder0" table="tComplexHolderNew0" polymorphism="implicit" mutable="true" dynamic-update="false" dynamic-insert="false" select-before-update="false"

optimistic-lock="version"> • <id name="id" column="fid" type="long" /> • - <discriminator type="string" not-null="true" force="false" insert="true"> • <column name="DISCRIMINATOR" /> • </discriminator> • <property name="name" column="fname" type="string" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • - <list name="children" access="field" cascade="all" inverse="false" mutable="true" optimistic-lock="true" embed-xml="true"> • <key column="parentId" on-delete="noaction" /> • <index column="elementIndex" /> • <many-to-many class="ComplexHolder0" embed-xml="true" not-found="exception" unique="false" /> • </list> • - <array name="array" access="field" cascade="all" inverse="false" mutable="true" optimistic-lock="true" embed-xml="true"> • <key column="parentId" on-delete="noaction" /> • <index column="elementIndex" /> • <many-to-many class="ComplexHolder0" embed-xml="true" not-found="exception" unique="false" /> • </array> • - <subclass name="ComplexHolder1" discriminator-value="D" dynamic-update="false" dynamic-insert="false" select-before-update="false"> • <property name="i1" column="i1" type="int" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • - <subclass name="ComplexHolder2" discriminator-value="E" dynamic-update="false" dynamic-insert="false" select-before-update="false"> • <property name="i2" column="i2" type="int" index="i2_idx" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • - <subclass name="ComplexHolder3" discriminator-value="F" dynamic-update="false" dynamic-insert="false" select-before-update="false"> • <property name="i3" column="i3" type="int" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • - <subclass name="ComplexHolder4" discriminator-value="G" dynamic-update="false" dynamic-insert="false" select-before-update="false"> • <property name="i4" column="i4" type="int" unique="false" optimistic-lock="true" lazy="false" generated="never" /> • </subclass> • </subclass> • </subclass> • </subclass> • </class> • </hibernate-mapping>

XML Mapping File for each Persistent Class ( 11ea Files )

MongoDB Code private void storeData(Mongo mongo) { collection(mongo).insert(new BasicDBObject("test", "object")); } public ComplexHolder0 convertFromDocument(DBObject data, DeserializationOptions deserializationOption) { ComplexHolder0 instance = createInstance(getAsString(data, TYPE_ATTRIBUTE)); instance.setName(getAsString(data, NAME_ATTRIBUTE)); if (null != data.get(CHILDREN)) { instance.setChildren(fromMongoObjectToList(getAsList(data, CHILDREN), deserializationOption)); } if (null != data.get(ARRAY)) { final List<ComplexHolder0> arrayContent = fromMongoObjectToList(getAsList(data, ARRAY),

deserializationOption); instance.setArray(arrayContent.toArray(new ComplexHolder0[arrayContent.size()])); } readAttributes(data, instance); return instance; }

MongoDB Mapping private static void writeSubTypeAttributes(ComplexHolder0 holder, BasicDBObject

dataStorage) { if (holder instanceof ComplexHolder1) { dataStorage.put(FIELD_I1, ((ComplexHolder1) holder)._i1); } if (holder instanceof ComplexHolder2) { dataStorage.put(FIELD_I2, ((ComplexHolder2) holder)._i2); } private static void readAttributes(DBObject dataStorage, ComplexHolder0 holder) { if (holder instanceof ComplexHolder1) { ((ComplexHolder1) holder)._i1 = (Integer) dataStorage.get(FIELD_I1); } if (holder instanceof ComplexHolder2) { ((ComplexHolder2) holder)._i2 = (Integer) dataStorage.get(FIELD_I2); }

MongoDB Mapping class Serialisation { static final String TYPE_ATTRIBUTE = "_t"; private static final String PACKAGE_NAME = ComplexHolder0.class.getPackage().getName(); static final String NAME_ATTRIBUTE = "name"; static final String CHILDREN = "children"; static final String ARRAY = "array"; static final String REFERENCE_TO_ORIGINAL_DOCUMENT = "_refToOriginal"; static final String FIELD_I1 = "_i1"; static final String FIELD_I2 = "_i2"; static final String FIELD_I3 = "_i3"; static final String FIELD_I4 = "_i4"; private final OneArgFunction<BasicDBObject, DBRef> refCreator; Serialisation(OneArgFunction<BasicDBObject, DBRef> refCreator) { this.refCreator = refCreator; } public static Serialisation create(OneArgFunction<BasicDBObject, DBRef> refCreator) { if (null == refCreator) { throw new ArgumentNullException("requires a reference creator"); } return new Serialisation(refCreator); } public static OneArgFunction<BasicDBObject, DBRef> createReferenceCreator(final DBCollection collection) { if (null == collection) { throw new ArgumentNullException("requires a collection"); } return new OneArgFunction<BasicDBObject, DBRef>() { @Override public DBRef invoke(BasicDBObject basicDBObject) { final DB db = collection.getDB(); final Object id = basicDBObject.get("_id"); if (null == id) { throw new IllegalStateException("Expected an '_id' on the object"); } return new DBRef(db, collection.getName(), id); } }; }

public SerialisationResult convertToDocument(ComplexHolder0 holder) { List<BasicDBObject> holder2Objects = new ArrayList<BasicDBObject>(); BasicDBObject document = convertToDocument(holder, holder2Objects); return new SerialisationResult(document, holder2Objects); } public ComplexHolder0 convertFromDocument(DBObject data) { return convertFromDocument(data, DeserializationOptions.FULL_DESERIALISATION); } public ComplexHolder0 convertFromDocument(DBObject data, DeserializationOptions deserializationOption) { ComplexHolder0 instance = createInstance(getAsString(data, TYPE_ATTRIBUTE)); instance.setName(getAsString(data, NAME_ATTRIBUTE)); if (null != data.get(CHILDREN)) { instance.setChildren(fromMongoObjectToList(getAsList(data, CHILDREN), deserializationOption)); } if (null != data.get(ARRAY)) { final List<ComplexHolder0> arrayContent = fromMongoObjectToList(getAsList(data, ARRAY), deserializationOption); instance.setArray(arrayContent.toArray(new ComplexHolder0[arrayContent.size()])); } readAttributes(data, instance); return instance; } private BasicDBObject convertToDocument(ComplexHolder0 holder, List<BasicDBObject> referencedDocumentsCollector) { BasicDBObject dbObject = createDocumentWithAttributesOnly(holder); dbObject.put(CHILDREN, toMongoObject(holder.getChildren(), referencedDocumentsCollector)); if (null != holder.getArray()) { dbObject.put(ARRAY, toMongoObject(asList(holder.getArray()), referencedDocumentsCollector)); } return dbObject; } private BasicDBObject createDocumentWithAttributesOnly(ComplexHolder0 holder) { BasicDBObject dbObject = new BasicDBObject("_id", new ObjectId()); dbObject.put(NAME_ATTRIBUTE, holder.getName()); dbObject.put(TYPE_ATTRIBUTE, holder.getClass().getSimpleName()); writeSubTypeAttributes(holder, dbObject); return dbObject; } public List<ComplexHolder0> fromMongoObjectToList(List data, DeserializationOptions deserializationOption) {

List<ComplexHolder0> objects = new ArrayList<ComplexHolder0>(data.size()); for (Object o : data) { objects.add(restoreDocumentOrReference(o, deserializationOption)); } return objects; } private ComplexHolder0 restoreDocumentOrReference(Object o, DeserializationOptions deserializationOption) { DBObject dbObj = (DBObject) o; if (null != dbObj.get(REFERENCE_TO_ORIGINAL_DOCUMENT)) { return deserializationOption.deserialize(this, dbObj); } else { return convertFromDocument(dbObj, deserializationOption); } } private static List getAsList(DBObject data, String attributeName) { return (List) data.get(attributeName); } private static String getAsString(DBObject data, String attribute) { return (String) data.get(attribute); } private static ComplexHolder0 createInstance(String className) { try { return (ComplexHolder0) Thread.currentThread().getContextClassLoader() .loadClass(PACKAGE_NAME + "." + className).newInstance(); } catch (Exception e) { throw rethrow(e); } } private static void writeSubTypeAttributes(ComplexHolder0 holder, BasicDBObject dataStorage) { if (holder instanceof ComplexHolder1) { dataStorage.put(FIELD_I1, ((ComplexHolder1) holder)._i1); } if (holder instanceof ComplexHolder2) { dataStorage.put(FIELD_I2, ((ComplexHolder2) holder)._i2); } if (holder instanceof ComplexHolder3) { dataStorage.put(FIELD_I3, ((ComplexHolder3) holder)._i3); } if (holder instanceof ComplexHolder4) {

} if (holder instanceof ComplexHolder4) { dataStorage.put(FIELD_I4, ((ComplexHolder4) holder)._i4); } } private static void readAttributes(DBObject dataStorage, if (holder } if (holder instanceof ComplexHolder2) { ((ComplexHolder2) holder)._i2 = (Integer) dataStorage.get(FIELD_I2); } if (holder instanceof ComplexHolder3) { ((ComplexHolder3) holder)._i3 = (Integer) dataStorage.get(FIELD_I3); } if (holder instanceof ComplexHolder4) { ((ComplexHolder4) holder)._i4 = (Integer) dataStorage.get(FIELD_I4); } } private List<Object> toMongoObject(List<ComplexHolder0> children,) { List<Object> objects = new ArrayList<Object>(children.size()); for (ComplexHolder0 child : children) { final BasicDBObject document = convertToDocument(child, if (isDocumentOnlyReferenced(child)) { referencedDocumentsCollector.add(document); DBObject copy = createDocumentWithAttributesOnly(child); copy.put(REFERENCE_TO_ORIGINAL_DOCUMENT, objects.add(copy); } else { objects.add(document); } } return objects; } private static boolean isDocumentOnlyReferenced(ComplexHolder0 child) { return child.getClass().equals(ComplexHolder2.class); }

Versant JPA Code

• Write – Generates Holder’s to user spec depth ComplexHolder cp = new ComplexHolder( ); em.makePersistent(cp);

• Read – Access root of graph and traverse cp = em.find( ComplexHolder.class, id ); cp.getChildren().touch();

• Query String query = "from org.polepos.teams.versant.data.Holder2 where i2="

+ currentInt; Iterator it = session.iterate(query);

• Delete – Deletes Graph – Cascading Operation

Versant JPA Mapping

<class name="ComplexHolder0"> <field name="name" /> <field name="array" /> <field name="children"> <collection element-type="ComplexHolder0"/> </field> </class>

single XML File with primitive declaration entries

Sh@t’s and Grins • JDBC Code JDBC StringBuilder sb = new StringBuilder(); sb.append("select " +

HOLDER_TABLE0 + ".id from " + HOLDER_TABLE0); sb.append(" INNER JOIN " + HOLDER_TABLES[0]); sb.append(" on " + HOLDER_TABLE0 + ".id = " + HOLDER_TABLES[0] + ".id "); sb.append(" INNER JOIN " + HOLDER_TABLES[1]); sb.append(" on " + HOLDER_TABLE0 + ".id = " + HOLDER_TABLES[1] + ".id "); sb.append(" LEFT OUTER JOIN " + HOLDER_TABLES[2]); sb.append(" on " + HOLDER_TABLE0 + ".id = " + HOLDER_TABLES[2] + ".id "); sb.append(" LEFT OUTER JOIN " + HOLDER_TABLES[3]); sb.append(" on " + HOLDER_TABLE0 + ".id = " + HOLDER_TABLES[3] + ".id "); sb.append(" where " + HOLDER_TABLES[1] + ".i2 = ?"); PreparedStatement stat = prepareStatement(sb.toString()); ResultSet resultSet stat.executeQuery();

PolePosition BenchMark Summary Results

FlatObject - Single Thread

Results – ComplexConcurrency @mongo - no safe mode

Results – QueryConcurrency @mongo – no safe mode

Results – InsertConcurrent @mongo – no safe mode

Results – ComplexConcurrency @mongo – safe mode

Results – QueryConcurrency @mongo – safe mode

Results – InsertConcurrent @mongo – safe mode

Conclusions

• Wake-up, smell the coffee – JOINs are not needed - unless adhoc analytics

• Take care of business, dump a load of code – Serialization is bad, data useful once structured – Mapping is Mapping, even when it’s not an ORM

• Get on your Bad Motor Scooter and Ride – NoSQL without Mapping rules the road

Get the Code: http://www.polepos.org

Contact

Robert Greene Vice President, Technology

Versant Corporation

rgreene@versant.com 650-232-2431