© Ibuildings 2014/2015 - All rights reserved

#DrupalDaysEU

Deploying an Open Source DAM in SAAS Mode Michel van de Ven

#DrupalDaysEU

© Ibuildings 2014/2015 - All rights reserved

Gold Sponsors

#DrupalDaysEU

© Ibuildings 2014/2015 - All rights reserved

Media Sponsors

Silver Sponsors

Deploying an Open Source DAMin SAAS Mode

Michel van de VenInuits

European Drupal Days 2015 Milano

MediaMosa

● Drupal-based Digital Asset Management system

● Commissioned by SURFnet and Kennisnet in 2007

● Open Source (GPLv2), Open Standards

● Webservice oriented: REST

● Store & retrieve assets

● Manage metadata using open standards

– Dublin Core, Qualified DC, IEEE/LOM, W3C Ontology

– OAI-PMH, CQL (SRU/SRW), BagIt

● Transcode video / audio (FFmpeg), images, PDF &c, stream content

● Storage on disk, SAN/NAS, S3

● Drupal modules for connection mgmt, media and views (by Acquia)

● Users: Kennisnet, NIBG, UGent, Uni Leuven, UvA, TiU, RUG, Digipolis Antwerp, UOslo, Avans, Politie Academie, Acquia (NBC Sports), Berkeley, Cineca &c

● http://mediamosa.org & MediaMosa Foundation

=

MediaMosa as a Service

MediaSalsa infrastructure (simplified)

● For each DTAP environment

– Backends: Core MediaMosa services

– Frontends: Optional, Drupal, Java, Python, apps, ...

– Web servers

– Database servers

– Solr servers

– Transcoding servers

● Instance @ RUG DC for NL .edu

● Instances @ Commercial DCs for non-edu

CultureAutomation

MeasurementSharing

Inuits

● Inuits is an Open Source company

– We contribute back

● 50+ people in 4 locations (BE, NL, UA, CZ)

● One language: English

● We offer

– Consulting

– Development

– System Administration– and a nice SaaS Platform

Ideal world vs budget and reality→ pragmatic approach

Distributed team

Communication = HELL

How to fix it?

Distributed team

● Daily virtual stand-up over XMPP / Hangout

● Redmine project management

– Issues, tasks, sprints

– Repositories

– Documentation (wiki)

– Used by customers as well

● Internal mailing lists

● Notifications by mail and XMPP

● Internal training, all the time

Dev and Ops

● Dev

● Architecture

● Develop new features

● Write unit tests (!)

● Ops

● Infrastructure (Puppet)

● CD and CI (Jenkins)

● Monitoring

→ Ops teach dev about monitoring and distributed services, scale, redundancy, failover, high availability, systems engineering, ...→ Dev teach ops about required libraries and testing, dependencies, daemon tuning, software engineering & architecture, ...

CultureAutomation

MeasurementSharing

Puppet

Puppet automates all the things

mcollective orchestrates all the things

Continuous D.*

● Unit Tests + Continuous Integration →

● Continuous Delivery vs Continuous Deployment

– “Continuous Delivery doesn't mean every change is deployed to production ASAP. It means every change is proven to be deployable at any time” (@ccaum)

● Puppet code

– Deployed to dev environment

– Same puppet code for each environment

– User-triggered deployments to UAT & production

– Feature flags in Puppet code per environment (switchable architecture)

● Application code

– Continuous integration in dev

– User-triggered deployments to UAT

– Deployment to production is a business decision

Testing

● Developers test a lot, but

– The tests don’t work

– It works on my machine™

– Wrong platform

– Wrong PHP version

Fixed now, thanks to Jenkins!

and Vagrant / $container_tech of the week

Version Control

● Git

● Code is under revision control

– Prefer small commits

– Local feature branches

● Infrastructure as code → git / hiera

next: Software Defined Infrastructure

Use the dist native packaging system

● Consistency, security, dependencies

● Uniquely identify where files are coming from

● Source repo may not be reacheable

● Little overhead when you automate

● Configuration does not belong in a package

Pipelines

● A collection of jobs

● Run in sequence

● Start on checkout, end at deployment

● From the developers’ side:

→ Git push

← Mail with changes + link to deploy / promote

Our current pipelines

● Puppetized

● Puppet code and Application code (backend and frontends)

● 1 pipeline per project phase / sprint

Pipelines steps

● Checkout● Syntax: php -l● Style: Drupal Coder● Package: FPM● Deploy to dev environment: mcollective● Tests in dev environment: drush run-tests● Publish package and promote: mcollective

Promotion

● At the end of the pipeline● Send a simple email

● A link to the promotion page● The changelog

● Promotion page contains one button per environment● Must promote to UAT before Production

Tools used with Jenkins

● Pulp to manage RPM repositories

– Collect, process, serve / push out

● Mcollective to update packages, run drush

– Control and update marionettes

Shared responsibility, pain, and joy

●Same Pipelines, Tools, Patternsare used by both devs and ops

CultureAutomation

MeasurementSharing

Logstash

● Collect all the logs

– Drupal logs

– Apache logs

– Deployment logs

– System logs

● Interpret, filter and correlate them

● Logstash, ElasticSearch, Kibana, statsd, Graphite

Icinga

● Monitor everything

– vhosts

– databases

– cronjobs

– unit test suites

Graphite + gdash

● Collectd

● Monitor platform usage

● FFmpeg usage

● Number of accounts

● Pipelined !

CultureAutomation

MeasurementSharing

Open Source

● Mediamosa is Fully Open Source

● Lots of the PuppetCode to deploy it is Open Source

● Our passwords etc aren't

Experiences

We have just shared our experience...

But also at the NLUUG Fall Conference, DrupalDays Kyiv, DrupalCon Prague, FlossUK, ElastiCon SF, ...

ContactContactKris Buytaert <Kris Buytaert <Kris.Buytaert@inuits.beKris.Buytaert@inuits.be>>

Inuits Chief Trolling O�cerInuits Chief Trolling O�cer

Further ReadingFurther Reading@krisbuytaert @krisbuytaert

http://www.krisbuytaert.be/blog/http://www.krisbuytaert.be/blog/http://www.inuits.be/http://www.inuits.be/

Inuits HQ Inuits HQ Antwerp, BelgiumAntwerp, Belgium+32 475 961221+32 475 961221

Branches:Branches:Rotterdam NLRotterdam NLKyiv UAKyiv UAPrague CZPrague CZ