OrigoDBBuild faster systems faster
Robert Friberg
• Design, build and maintain systems for clients at Devrex• Trainer – Microsoft NET, SQL Server, java, perl, python, linux• Machine learning, AI• Squash fanatic• @robertfriberg, [email protected]
Why?
ServiceLayer
DomainLayer
Data AccessLayer
RelationalModel
Views/SP’sXBuild faster systems faster
What is OrigoDB?
• In-memory database toolkit• Code and data in same process• Write-ahead command logging and snapshots• Open Source single DLL for NET/Mono• Commercial server with mirror replication
.NET Process Storage
Engine
2. Execute command
Handles queries and commands, guards model
1. AppendToLogPlaceOrderCommand
Snapshot
Client codepasses commandsand queries
PlaceOrderCommand
NewCustomerCommand
IncreaseInventoryLevelCommand
PlaceOrderCommand
PlaceOrderCommand
Snapshot
time
In-memoryModel
How does it work?
Demand drives change
• Performance• Data volume• Scalability• Availability• Modeling
• NoSQL
• Big data
• Graph
• Real time analytics
• In-memory computing
• Column stores
One size (RDBMS) no longer fits allPolyglot Persistence
B-trees and Transactions
LOG
DATA 64KB blocks w 8x8KB pagesLogical BTREE of 8kb data pagesIn the buffer pool (cache)
BufferManager
Transactions append inserted, deleted, original and modified pages to the LOG
• Fill factor• Page splits• Clustered index• Checkpoint
When?
• Whenever data fits in RAM• Alternative to general RDBMS OLAP/OLTP• Complex models and transactions• In-memory analytics• Traceability requirements (complete history of events)
Core framework types
Start your engines!
var engine = Engine.LoadOrCreate<SalesModel>();
// lambda queryvar customer = engine.Execute(db => db.Customers.GetById(42));
//Command objectvar cmd = new PlaceOrderCommand(customer.Id, newOrder);engine.Execute(cmd);engine.Close();
• 2 Core, 8GB RAM Cloud VM• IIS Web, SPA, ajax, ASP.NET MVC3• OrigoDB Server same host, currently 4GB process memory• 3k blogs, 500k posts, 500k keywords, 33 million links• Journal 1 GB, no snapshots, growth ~ 8MB/day
http://geekstream.devrexlabs.com
• Environmental news referral since 1997• 100 articles/day, fulltext search, summaries, categories, sources,
metadata• 5000 subscribers, clients, users, billing• Email history, weekly newsletters• < 4 GB Total RAM
Modeling
Creating a data model
• Domain specific vs. Generic vs. Hybrid• Rich vs. Anemic• Explicit vs. Implicit commands• References vs. Foreign keys
Computational model types
• Interpreter hosting – Javascript, Roslyn• Lucene index• Machine learning models (Accord.NET)
Object oriented domain modeling
• DDD?• Entities and collections• Choose collections wisely – space/time tradeoffs• Transaction script pattern or thin commands• Avoid CRUD
Silly Relational
[Serializable] public class RelationalModel : Model { private DataSet _dataset;
public RelationalModel() { _dataset = new DataSet(); } //... ExecuteQuery and ExecuteCommand omitted }
Generic Relational
[Serializable] public class RelationalModel : Model { Dictionary<string,SortedDictionary<object>> _tables; }
Domain specific relational
[Serializable] public class Model : Model { SortedDictionary<int,Customer> _customers; SortedDictionary<int,Order> _orders; SortedDictionary<int,Product> _products; SortedDictionary<string,Customer> _customersByName; }
[Serializable] public class Order { public int CustomerId; //foreign key vs. reference }
JS interpreter hosting (V8)
[Serializable] public class JurassicModel : Model { private ScriptEngine _scriptEngine;
public JurassicModel() { _scriptEngine = new ScriptEngine(); _scriptEngine.Execute("var model = {}"); } //... ExecuteQuery and ExecuteCommand omitted }
Commands
• Serial execution• Exclusive access to the model• Transition the model from one valid state to the next
s0 s1 s2t1 t2
Command guidelines
• No side effects or external actions• No external dependencies• Unhandled exceptions trigger rollback (full restore)• Call Command.Abort() to signal exception
The model is an object graph
TaskListTask
Task
Task
TaskList
Task
Task
Task
Task
Category
Category
Category
Category
TaskModel
Query alternatives
• Ad-hoc lambda: Engine.Execute(Func<M,R> query)• Derive from Query<M,R>• Compiled LINQ: Engine.Execute<R>(string, args)
Query guidelines
• Know thy object graphs• Don’t modify the model
Demo: HockeySkolan
• ASP.NET MVC 3 with backing OrigoDB• Domain specific model + CmsModel• Anemic model => fat commands, model is data
OrigoDB WorkshopModule 3 - Testing
Automated Test alternatives
• Transparent testing of domain behavior: entities, model• Test commands, queries, entities on model without engine• Test with in-memory storage• Full stack testing – slow, requires cleanup
In-memory storage
• Non persistent command journal and snapshots• Mimics FileStore using MemoryStreams• Tests serialization/identity issues
var config = EngineConfiguration.Create().ForIsolatedTest();
OR: config.SetCommandStoreFactory(cfg => new InMemoryCommandStore(cfg));
config.SetSnapshotStoreFactory(cfg => new InMemorySnapshotStore(cfg));
Demo: RedisModel
• Testing model behavior with NUnit
OrigoDB WorkshopModule 4 – Hosting
Direct in-process engine creation
• Static Engine methods• Create()• LoadOrCreate()• Load()
• Returns: Engine, Engine<M>
Engine<MyModel> engine = Engine.LoadOrCreate<MyModel>();
Engine.For<M>()
• Returns IEngine<M>() or derivative• Reuse based on EngineConfiguration.Location property• Remote or in-process• ILocalEngineClient<M>• IRemoteEngineClient<M>
• Running engines are tracked by Config.Engines
Db.For<M>()
• Returns a proxy for M• Remote or Local analogous to Engine.For<M>
x64 vs. x32
• Core Library compiled with AnyCPU• x32 = 32-bit pointers, max 4GB• x64 = 64-bit pointers• Server ships with x64 and x32 binaries
IIS Hosting
• Disable application pool recycling• Ensure single process, no farming or LB• Litter controllers with Db.For<M>() / Engine.For<M>()• Or put a static ref somewhere, eg Global.asax
Proxy
• Proxy has same interface as the Model• Method calls are intercepted• void methods interpreted as commands (and logged)• other methods interpreted as queries• Can be overriden with attributes• Local or remote, cluster
Demo: Let’s build a key/value store
• New project• Reference origodb.core• Define KeyValueStoreModel• void Put(key, value)• object Get(key,value)• bool Exists(key, value)• bool Remove(key,value)
• Add some unit tests
OrigoDB WorkshopModule 5 – Configuration
EngineConfiguration class – key properties• AsyncJournaling• EnsureSafeResults• Kernel• Location• PacketOptions• PersistenceMode• SnapshotBehavior
EngineConfiguration.Location property• File location for FileStorage• Connection string when SqlStorage• Defaults (when null)• Will look in app.config for connection string• Type name of model• Current working directory• App_Data in web context
• Magic• mode=remote;host=10.0.0.20;port=3001
Persistence modes
• Journaling (default)• SnapshotPerTransaction• ManualSnapshots
var config = EngineConfiguration.Create();
config.PersistenceMode = PersistenceMode.SnapshotPerTransaction;
var db = Engine.For<MyModel>(config);
Kernels
• OptimisticKernel (default)• RoyalFoodTaster
var config = EngineConfiguration.Create();
config.Kernel = Kernels.RoyalFoodTaster;
var db = Db.For<MyModel>(config);
Logging
• Logging disabled by default
//enable loggingConsoleLogger.MinimumLevel = LogLevel.Trace;
//Plugin custom loggerLogProvider.SetFactory(new Log4NetLoggerFactory());
Extensibility
• EngineConfiguration methods• SetAuthorizerFactory()• SetSynchronizerFactory()• SetCommandStoreFactory()• SetFormatterFactory()
ProtobufFormatter
• Protocol Buffers by Google• IFormatter wrapper around protobuf-net by @marcgravell• Contract based as opposed to embedded metadata• Compact, fast, loose coupling, cross platform• Configure with attributes or code • Use it!
Protobuf: Attribute based configuration [ProtoContract] public class Company { [ProtoMember(1)]
public string Name { get; set; }
[ProtoMember(2)]
public List<Employee> Employees { get; set; } }
Protobuf: Code-based configuration
var typeModel = TypeModel.Create();
typeModel.Add(typeof (Company), false)
.Add(1, ”Name")
.Add(2, ”Employees");
Inject formatter factory
//use helper methods
ProtoBufFormatter.ConfigureSnapshots<MyModel>(config, typeModel);
ProtoBufFormatter.Configure(config, FormatterUsage.Results, typeModel);
//or do it yourself
config.SetFormatterFactory((cfg,fu)
=> new ProtoBufFormatter<MyModel>(typeModel),
FormatterUsage.Snapshot);
OrigoDB WorkshopModule 6 – Immutability
Immutability and blockingwriter
reader
States share immutable objects
s0
s3s1s2
(Animated slide)
Exampleimmutablemodel
Exampleimmutableentity
Immutable command example
Configuration
OrigoDB WorkshopModule 6 – Server
OrigoDB Server
• Console Application or Windows Service• Process hosting single Engine / Model• Ad-hoc Linq / Razor queries• Javascript API• Primitive web based UI• Commercial License• Multiserver replication
Transparent client migration
• Configuration strings:• mode=embedded• mode=remote;host=10.10.10.10;port=3001
Thank you for listening!
• http://origodb.com• http://dev.origodb.com• http://github.com/devrexlabs• http://geekstream.devrexlabs.com• @robertfriberg, @devrexlabs
Top Related