Dexjava Technical Seminar Dec 2011

Dex G

rap

h D

ata

base

http://www.sparsity-technologies.com

Technical Seminar

Sergio Gómez

December, 2011

Dex G

rap

h D

ata

base


Introduction

Basic Concepts

Database construction

Query database

Loaders

Script loaders

Tips & tricks

Index

Dex G

rap

h D

ata

base


Index

Introduction

Basic Concepts


Query database

Loaders

Script loaders

Tips & tricks

Dex G

rap

h D

ata

base


Introduction

Graph database

Graph databases focus on the structure of the model. Nodes and edges instead of tables. Relationships are first-class citizens.

Explicit in the model.

DEX is a programming librarywhich allows to manage agraph database.

Very large datasets. High performance

query processing.

Dex G

rap

h D

ata

base


IntroductionDex Definition

Persistent and temporary graph management programming library.

Data model: Typed and attributed directed multigraph.

Typed: Node and edge instances belong to a type (label). Attributed: Node and edge instances may have attribute

values. Directed: Edge can be directed or undirected. Multigraph: Multiple edges between two nodes.

Dex G

rap

h D

ata

base


IntroductionGraph Model

Dex G

rap

h D

ata

base


Introduction

Basic Concepts


Query database

Loaders

Script loaders

Tips & tricks

Index

Dex G

rap

h D

ata

base


Basic Concepts

Java library public API Private native

dynamic library Automatically

loaded

System requirements: Java Runtime Environment, v1.5 or higher. Operative system:

Windows, MacOSX, Linux 32 and 64 bits

Dex G

rap

h D

ata

base


Basic Concepts

Dex Session1

N

Graph

1

1

Persistent DB

Objects

1

N

Database

1

N

Set of OIDs

Dexjava Class Diagram

Dex G

rap

h D

ata

base


Basic ConceptsMain methods

Dex

open(filename) Databasecreate(filename) Databaseclose()

DatabasenewSession() Session

Session

getGraph() Graphclose()

Graph

newNodeType(name) intnewEdgeType(name) intnewNode(type) longnewEdge(type) longnewAttribute(type, name) intsetAttribute(oid, attr, value)getAttribute(oid, attr) value

select(type) Objectsselect(attr, op, value) Objectsexplode(oid, type) Objectsneigbors(oid, type) Objects

ObjectsIterator

hasNext() booleannext() long

Objects

add(long)exists(long)

copy(objs)union(objs)intersection(objs)difference(objs)

Dex G

rap

h D

ata

base


Index

Introduction

Basic Concepts


Query database

Loaders

Script loaders

Tips & tricks

Dex G

rap

h D

ata

base


Database construction Graph

Dex: Loads library and manages graph db instances. Database: Manages a graph db instance. Session: Manages queries and temporary data.

Nodes & Edges Type:

Dex identifier (integer) Public identifier (string)

Instance: DEX identifier (long) – OID belongs to a type

Attributes Attribute:

DEX identifier (int) public identifier (string) Scope: type or global Temporary (per Session) or persistent

Dex G

rap

h D

ata

base



Database Dex#create(String path, String alias)Creates a new graph database instance.Returns the Database instance to manage a new persistent graph.

Database Dex#open(String path, bool read)Opens an existing graph database instance.Read-only mode.Returns the Database instance to manage the persistent graph.

Session Database#newSession()Initiates a new user Session.

Graph Session#getGraph()Gets the Graph instance which represents the graph data.

Create a graph database

Dex G

rap

h D

ata

base



import com.sparsity.dex.gdb.*;…Dex dex = new Dex(new DexConfig());Database db = dex.create(“C:/image.dex”, “graphdb”);Session s = db.newSession();……s.close();db.close();dex.close();

Create a graph database example

Dex G

rap

h D

ata

base



int Graph#newNodeType(String name)Creates a new node type with the given unique name.Returns the Dex node type identifier.

long Graph#newNode(int nodeType)Creates a new node belonging to the given node type. Returns the Dex object identifier.

Add nodes

Dex G

rap

h D

ata

base



int Graph#newEdgeType(String name, bool directed,bool neighbors)

Creates a new edge type with the given unique name. Directed or undirected edge type.

Create neighbor-index or not. Returns the Dex edge type identifier.

int Graph#newRestrictedEdgeType(String name, int srcNodeType, int dstNodeType, bool neighbors)

Creates a new directed edge type with the given unique name.(Integrity restriction) Source and destination of the edge

instances are restricted to the given node types.Create neighbor-index or not.Returns the Dex edge type identifier.

Add edges

Dex G

rap

h D

ata

base



int Graph#newEdge(int edgeType, long tail, long head)Creates a new edge belonging to the given edge type. Tail is the source and head is the target.Returns the Dex edge identifier.

Add edges

Dex G

rap

h D

ata

base


…Graph g = s.getGraph();

int person = g.newNodeType(“PERSON”);long p1 = g.newNode(person);long p2 = g.newNode(person);long p3 = g.newNode(person);

int friend = g.newEdgeType(“FRIEND”, false, false);

long e1 = g.newEdge(friend, p1, p2);long e2 = g.newEdge(friend, p2, p3);

int loves = g.newEdgeType(“LOVES”, true, false);

long e3 = g.newEdge(loves, p1, p3);…

Add nodes and edges example


p1

p2

p3

p1

p2

p3

Dex G

rap

h D

ata

base



class ValueEncapsulates a value and its domain (data type).Use them to set and get attribute values for the objects.

int Graph#newAttribute(int type, String name, DataType dt, AttributeKind kind)Creates a new attribute with the given unique name for the given node or edge type. Returns the Dex attribute identifier.

“dt” can be: Boolean, Integer, Long, Double, String, Text, Timestamp, OID.

“kind” can be:Basic: Just set and get operations are allowed.Indexed: Select operations are allowed as well as set and get operations.Unique: As indexed. Unique integrity restriction: no two objects with the same value for the attribute but NULL.

Manage attributes

Dex G

rap

h D

ata

base



Graph#setAttribute(long oid, int attr, Value v)Sets the given Value for the given attribute to the given object identifier. Given attribute must be defined for the object’s type. Value ‘s data type must match attribute’s data type or NULL.

Graph#getAttribute(long oid, int attr, Value v)Gets the Value for the given attribute and for the given object identifier. Given attribute identifier must be defined for the object’s type.

Manage attributes

Dex G

rap

h D

ata

base



…int name = g.newAttribute(person, “NAME”,

String, Unique);int age = g.newAttribute(person, “AGE”,

Integer, Indexed);Value v = new Value();

g.setAttribute(p1, name, v.setString(“JOHN”));g.setAttribute(p1, age, v.setInteger(18));g.setAttribute(p2, name, v.setString(“KELLY"));g.setAttribute(p3, name, v.setString(“MARY"));…

int since = g.newAttribute(friend, “SINCE”, Integer, Indexed);

g.setAttribute(e1, since, v.setInt(2000));g.setAttribute(e2, since, v.setInt(1995));…

JOHN18

KELLY

MARY

1995

2000JOHN18

KELLY

MARY

Manage attributes example

Dex G

rap

h D

ata

base



…int phones = g.newEdgeType("phones“, true, true);int when = g.newAttribute(phones, "when",

String, Indexed);

long e4 = g.newEdge(phones, p1, p3);g.setAttribute(e4, when, v.setString("4pm")));

long e5 = g.newEdge(phones, p1, p3);g.setAttribute(e5, when, v.setString("5pm"));

long e6 = g.newEdge(phones, p3, p2);g.setAttribute(e6, when, v.setString("6pm"));…

g.getAttribute(p1, name, v);System.out.println(v);g.getAttribute(e5, when, v);System.out.println(v);g.getAttribute(e4, when, v);System.out.println(v);

1995

2000JOHN18

KELLY

MARY

4pm5pm

6pm

Manage attributes example

Dex G

rap

h D

ata

base


Index

Introduction

Basic Concepts


Query database

Loaders

Script loaders

Tips & tricks

Dex G

rap

h D

ata

base


Query database

Manage node and edge types

int Graph#findType(String name)Returns the Dex type identifier for the given type name.

Type Graph#getType(int type)Returns the metadata for the given Dex type identifier.

TypeList Graph#findTypes()Returns the list of all existing Dex type identifiers.

Dex G

rap

h D

ata

base


Query database

Manage attributes

int Graph#findAttribute(int type, String name)Returns the Dex attribute identifier for the given Dex type identifier and attribute name.

Attribute Graph#getAttribute(int attr)Returns the metadata for the given Dex attribute identifier.

AttributeList Graph#findAttributes(int type)Returns the list of all existing Dex attribute identifiers for the given Dex type identifier.

Dex G

rap

h D

ata

base


Query database

Objects

class ObjectsUnordered set of OIDs for large collections.Implements Set<Long>, Iterable<Long>.

boolean Objects#add(long oid)Adds the given OID to the collection.Returns true if added, false if the OID was already into the collection.

boolean Objects#exists(long oid)Returns true if the given OID exists into the collection, false otherwise.

boolean Objects#remove(long oid)Removes the given OID from the collection.Returns true if removed or false if the OID was not into the collection.

Dex G

rap

h D

ata

base


Query database

Objects

long Objects#union(Objects objs)this = this UNION objsReturns the new size of the collection.

long Objects#intersection(Objects objs)this = this INTERSECTION objsReturns the new size of the collection.

long Objects#difference(Objects objs)this = this DIFFERENCE objsReturns the new size of the collection.

Dex G

rap

h D

ata

base


Query database

Objects Graph#select(int t)Retrieves object identifiers belonging to the given node or edge type.

Objects Graph#select(int attr, Condition c, Value v)

Retrieves object identifiers which satisfy the condition for the given Value.“c” can be:

Equal, NotEqual, GreaterEqual, GreaterThan, LessEqual,LessThan, Between.Also, for String attributes: Like, LikeNoCase, RegExp.

long Graph#findObject(int attr, Value v)Randomly retrieves an object identifier which has the given value for the given attribute (or Objects.InvalidOID if not found).Useful for unique attributes.

Retrieve data

Dex G

rap

h D

ata

base


Query database

Objects Graph#explode(long oid, int edgeType, EdgesDirection dir)

Retrieves out-going or in-going edges (or both) from or to the given node identifier and for the given edge type.“dir” can be:

Ingoing, Outgoing, Any.

Objects Graph#neighbors(long oid, int edgeType, EdgesDirection dir)

Retrieves neighbor nodes to the given node identifier which can be reached through the given edge type and direction.“dir” can be:

Ingoing, Outgoing, Any.

Navigation

Dex G

rap

h D

ata

base


Query database

…Graph g = s.getGraph();Objects persons = g.select(person);ObjectsIterator it = persons.iterator();while (it.hasNext()) {

long p = it.next();dbg.getAttribute(p, name, v);String name = v.toString();

}it.close();persons.close();…

JOHN18

KELLY

MARY

JOHN18

KELLY

MARY

1995

4pm

5pm

6pm

2000

Retrieve data example

Dex G

rap

h D

ata

base


Query database

…Objects objs1 = g.select(when, GreaterThan, “5pm”);// objs1 = { e5, e6 }Objects objs2 = g.explode(p1, phones, Outgoing);// objs2 = { e4, e5 }objs1.intersection(objs2);// objs1 = { e5, e6 } ∩ { e4, e5 } = { e5 }…objs1.close();objs2.close();…

JOHN18

KELLY

MARY

JOHN18

KELLY

MARY

1995

4pm

5pm

6pm

2000

Navigation & Objects operations example

Dex G

rap

h D

ata

base


Index

Introduction

Basic Concepts


Query database

Loaders

Script loaders

Tips & tricks

Dex G

rap

h D

ata

base


Loaders Package com.sparsity.dex.io

NodeTypeLoader Requires a RowReader

CSVReader … or write your own implementation.

Creates a node instance for each row and sets its attributes with the values within each column.

EdgeTypeLoader Requires a RowReader, too. Creates an edge instance for each row and sets its

attributes with the values within each column. Two special columns to identify source and target

nodes for the edge.

Dex G

rap

h D

ata

base


Index

Introduction

Basic Concepts


Query database

Loaders

Script loaders

Tips & tricks

Dex G

rap

h D

ata

base


Script loaders

LOAD NODES ‘file_name’ [LOCALE loc]COLUMNS attribute_name [alias_name], …INTO node_type_name[IGNORE (attribute_name|alias_name), …][FIELDS

[TERMINATED char][ENCLOSED char][ALLOW_MULTILINE [max]]]

[FROM num][MAX num][MODE (ROWS|COLUMNS [SPLIT [PARTITIONS num]])]

Load nodes

Dex G

rap

h D

ata

base


Script loaders

LOAD EDGES ‘file_name’ [LOCALE loc]COLUMNS attribute_name [alias_name], …INTO node_type_name[IGNORE (attribute_name|alias_name), …]WHERETAIL (attribute_name|alias_name) = node_type_name.attribute_nameHEAD (attribute_name|alias_name) = node_type_name.attribute_name[FIELDS

[TERMINATED char][ENCLOSED char][ALLOW_MULTILINE [max]]]

[FROM num][MAX num][MODE (ROWS|COLUMNS [SPLIT [PARTITIONS num]])]

Load edges

Dex G

rap

h D

ata

base


Script loaders

create gdb WIKIPEDIA into 'wikipedia.dex' create node TITLES (

ID int unique, 'TEXT' string, NLC string, TITLE string indexed

) create node IMAGES (

ID int unique, NLC string, FILENAME string indexed

)create edge REFS from TITLES to TITLES (

NLC string, "TEXT" string, TYPE string

) materialize neighborscreate undirected edge IMGS

Examples

Dex G

rap

h D

ata

base


Script loaders

use gdb WIKIPEDIA into 'wikipedia.dex' load nodes 'images.csv'

columns ID, NLC, FILENAME into IMAGES from 2 max 10000

load edges 'references.csv' columns NLC, 'TEXT', TYPE, FROM F, TO T into REFS ignore F, T where

tail F = TITLES.ID head T = TITLES.ID

fieldsterminated ‘|’enclosed ‘”’allow_multiline

mode columns split partitions 3

Examples

Dex G

rap

h D

ata

base


Index

Introduction

Basic Concepts


Query database

Loaders

Script loaders

Tips & tricks

Dex G

rap

h D

ata

base


Tips & tricks

Index or not? Attributes:

Attributes used at select operations must be indexed.

Optionally, index once the attribute has been created/loaded.

Neighbors: It is recommended to index those

edge types used at neighbors operations.

Dex G

rap

h D

ata

base


Tips & tricks

String attributes String

Maximum length = 2047. Indexed or not.

Select [==, !=, >, >=, <, <=, Like, LikeNoCase, RegExp]

Text (Character large object) Unlimited length. Not Indexed.

Just get and set. Streaming read and write operations.

Dex G

rap

h D

ata

base


Tips & tricks

Others: DB cross-platform format.

32 – 64 bits, OS independent. Just take into account platform endianness.

Read only mode. Configuration:

com.sparsity.dex.gdb.DexConfig Set the maximum memory usage.

0 means unlimited. License.

No license means evaluation version.

Dex G

rap

h D

ata

base


Thanks for yourattention

Any questions?

SPARSITY-TECHNOLOGIESJordi Girona, 1-3, Edifici K2M

08034 [email protected]


Dexjava Technical Seminar Dec 2011

Technology

Transcript of Dexjava Technical Seminar Dec 2011