Training MongoDB - Monitoring and Operability

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Confi

dent

ial

RESTRICTED

MongoDB Monitoring & Operability

Training for App Mgmt and Operations

Nicolas Motte, 18/11/2014

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Link to this presentation

_ http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Mongos_and_MongoDB_OSAG• Section 10. Training

_ http://gmstst.muc.amadeus.net/trends/• NoSQL section• Select “Mongos and MongoDB OSAG”• Section 10. Training

http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Mongos_and_MongoDB_OSAG


http://gmstst.muc.amadeus.net/trends/

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Agenda_ Overview of MongoDB_ Amadeus Architecture around NoSQL_ MongoDB concepts_ Workflows_ Operability & Monitoring

414213741

© 2

014

Amad

eus I

T Gr

oup

SA

Confi

dent

ial

RESTRICTED

Overview of MongoDB1

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Document store

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Document store• Stores JSON-style documents

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Document store• Stores JSON-style documents• Uses BSON format for storage

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


_ Several binaries written in C++

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


_ Several binaries written in C++• Mongod : The actual database process, owning the data

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology• Agent : Monitoring process

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology• Agent : Monitoring process• MMS : Monitoring UI, connected to the Agents

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology• Agent : Monitoring process• MMS : Monitoring UI, connected to the Agents

_ Opensource and Enterprise versions

High-level

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

Mongod

Mongod

Mongod

Architecture

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

Mongod

Mongod

Mongod

Mongos

Architecture

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

Mongod

Mongod

Mongod

Mongos

Config

Architecture

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

Mongod

Mongod

Mongod

Mongos

ConfigClient

Client

Client

Architecture

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

Mongod

Mongod

Mongod

Mongos

ConfigClient

Client

Client

Agent

Architecture

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

Mongod

Mongod

Mongod

Mongos

ConfigClient

Client

Client

Agent

MMS

Architecture

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Why do we use MongoDB ?Couchbase

MemcachedMongoDB

Oracle

Comparison

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

Memcached Couchbase MongoDB OracleDatabase Model Key-Value Key-Value Document TableData scheme Schema-free Schema-free Schema-free Schema-fullIndexesPersistencyData IntegrityTransaction

Comparison

Confi

dent

ial

RESTRICTED

Overview of MongoDB

_ Mongod is the process storing the data

Mongod

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Mongod is the process storing the data_ 3 different types

Mongod

Confi

dent

ial

RESTRICTED

Overview of MongoDB

_ Mongod is the process storing the data_ 3 different types• PRIMARY

Mongod

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Mongod is the process storing the data_ 3 different types• PRIMARY• The primary is the only process that

receives write operations

Mongod

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


receives write operations• SECONDARY

Mongod

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


receives write operations• SECONDARY• A secondary maintains a copy of the

primary’s data set

Mongod

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB



primary’s data set• ARBITER

Mongod

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB



primary’s data set• ARBITER• An arbiter does not have a copy of

data set and cannot become a primary

Mongod

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB



primary’s data set• ARBITER• An arbiter does not have a copy of

data set and cannot become a primary_ Heartbeats are sent every 10s

between mongods

Mongod

Heartbeat

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Config servers are special mongod instances that store the metadata for a cluster (like topology)

Config Server

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Config servers are special mongod instances that store the metadata for a cluster (like topology)

_ A production cluster has exactly three config servers

Config Server

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Mongos is a connection multiplexer

_ Heartbeat are sent every 10s

_ Map/Reduce queries

Mongos

Hun

dred

s of

clie

nts

A single connection/server

Clients are not aware of the topology

Mongos hides all the complexity and standardizes

the configuration

MongosMongod

Mongod

Mongod

Config

Heartbeat

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


Mongos

Mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


Mongos

Hun

dred

s of

clie

nts

MongosMongod

Mongod

Mongod

Config

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


Mongos

Hun

dred

s of

clie

nts


MongosMongod

Mongod

Mongod

Config

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


Mongos

Hun

dred

s of

clie

nts



MongosMongod

Mongod

Mongod

Config

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


Mongos

Hun

dred

s of

clie

nts




the configuration

MongosMongod

Mongod

Mongod

Config

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB



Mongos

Hun

dred

s of

clie

nts




the configuration

MongosMongod

Mongod

Mongod

Config

Heartbeat

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB



_ Map/Reduce queries

Mongos

Hun

dred

s of

clie

nts




the configuration

MongosMongod

Mongod

Mongod

Config

Heartbeat

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Monitoring process

Agent

Agent

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB

_ Monitoring process_ 1 active/1 passive

per cluster

Agent

Agent

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB


per cluster_ Retrieves the

config servers from MMS

Agent

Agent

MMS

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB




_ Retrieves the list of processes from a config server

Agent

Config

Agent

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Overview of MongoDB




_ Retrieves the list of processes from a config server

_ Connects to all processes

Agent

Mongod

Mongod

Mongod

Mongos

Config

Agent

MMS

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

_ Datastore• Pros• Structured data• Flexible (Schema-less)• Indexes• Scaling is “easy”• Easy to deploy• Cons• Transactions are not yet supported• Each command is atomic but locking is at database level in 2.6 (moving to

document level in 2.8)_ Analytical search• MSX

Use CasesOverview of MongoDB

© 2

014

Amad

eus I

T Gr

oup

SA

Confi

dent

ial

RESTRICTED

Amadeus Architecture Around NoSQL2

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongos

AS OBE(GOS, App Support)

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongos ?

Amadeus Architecture Around NoSQL

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongos


AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongos ?_ Deployed on each AS

mongos


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongos


AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongos ?_ Deployed on each AS_ BEs are connected via the

standard Mongo C++ client

BE BE BEBE

mongos


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongos


AppTier

NoSQLTier MongoDB

(GOS, DZB)



_ Mongos is connected to all the MongoDB servers

BE BE BEBE

mongos


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongos


AppTier

NoSQLTier MongoDB

(GOS, DZB)




_ Duwrapped process monitored by the local Agent

SI Agent

BE BE BEBE

mongosdwi


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongos


AppTier

NoSQLTier MongoDB

(GOS, DZB)





_ Equivalent of ProxyD for Memcached and Couchbase

SI Agent

BE BE BEBE

mongosdwi


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongos


AppTier

NoSQLTier MongoDB

(GOS, DZB)





_ Equivalent of ProxyD for Memcached and Couchbase

_ Also running on NoSQL Tier for administration tasks

SI Agent

BE BE BEBE

mongosdwi

mongos


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongod


AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongod ?


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongod


AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongod ?_ Deployed on cluster

nodes


mongodmongod

mongodmongod

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongod


AppTier

NoSQLTier MongoDB

(GOS, DZB)


nodes_ Accepts connections

from Mongos on App Tier and NoSQL Tiermongod

mongodmongod

mongod

mongos


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Mongod


AppTier

NoSQLTier MongoDB

(GOS, DZB)


nodes_ Accepts connections

from Mongos on App Tier and NoSQL Tier

_ A mongod can be a primary, a secondary or an arbiter

mongodmongod

mongodmongod

mongos


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Agent


AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Agent?

mongodmongod

mongodmongod

mongos


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Agent


AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Agent?_ Deployed on 2

cluster nodes, 1 active / 1 passive

mongodmongod

mongodmongod

mongos

agent


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Agent


AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Agent?_ Deployed on 2

cluster nodes, 1 active / 1 passive

_ Connects to Mongos to monitor all processes

mongodmongod

mongodmongod

mongos

agent


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Config Server


AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Config servers?

mongodmongod

mongodmongod

mongos

agent


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Config Server


AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Config servers?_ Running on

dedicated nodes

mongodmongod

mongodmongod

mongos

agent

config


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Config Server


AppTier

NoSQLTier MongoDB

(GOS, DZB)


dedicated nodes_ Accepts connections

from mongosmongod

mongodmongod

mongod

mongos

agent

config

mongos


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Config Server


AppTier

NoSQLTier MongoDB

(GOS, DZB)


dedicated nodes_ Accepts connections

from mongos_ These nodes are

treated like any other cluster nodes in our tools

mongodmongod

mongodmongod

mongos

agent

config

mongos


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Amadeus Architecture Around NoSQLConfiguration


AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos

MongoS port retrieved from

the MAGdwi

Duwrapper Conf(listen port)

+Configuration files

Keyfile_xxxMongos_xxx.yml

xxx.pem

_ BE Configuration• http://rndwww.nce.amadeus.n

et/wiki/wikidoc/index.php/BE_Configuration_(MongoDB)

_ Mongos Configuration• http://rndwww.nce.amadeus.n

et/wiki/wikidoc/index.php/Mongos_Configuration

_ MongoDB Configuration• Handled entirely by DZB

http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/BE_Configuration_(MongoDB)



http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Mongos_Configuration



© 2

014

Amad

eus I

T Gr

oup

SA

Confi

dent

ial

RESTRICTED

MongoDB concepts3

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Replica Set or ShardMongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Replica Set or Shard• A replica set is a group of mongod

instances that host the same data set

MongoDB

Replica Set

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Shard Key

• Determines the distribution of the documents among the cluster’s shards

• The shard key is an indexed field that exists in every document

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Shard Map• MongoDB partitions data in

“chunks”, chunks are associated to a replica set

MongoDB

Shard Map

Shard 1 Shard 2 Shard 3

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


“chunks”, chunks are associated to a replica set• When a chunk grows over the

chunk size (64 MB), MongoDB splits the chunk in smaller chunks. The (arbitrary) limit has been chosen to avoid moving too much data during a balancing.

MongoDB

Shard Map


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


“chunks”, chunks are associated to a replica set• When a chunk grows over the

chunk size (64 MB), MongoDB splits the chunk in smaller chunks. The (arbitrary) limit has been chosen to avoid moving too much data during a balancing.• The config server provides a

shard map with at least one mongod process per shard

MongoDB

Shard Map


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ MongoDB functional architectureMongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ MongoDB functional architecture• A MongoDB cluster contains several databases

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ MongoDB functional architecture• A MongoDB cluster contains several databases• Each database contains several collections (equivalent of Oracle schemas)

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ MongoDB functional architecture• A MongoDB cluster contains several databases• Each database contains several collections (equivalent of Oracle schemas)• A collection contains a set of JSON documents (equivalent of Oracle tables)

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ MongoDB functional architecture• A MongoDB cluster contains several databases• Each database contains several collections (equivalent of Oracle schemas)• A collection contains a set of JSON documents (equivalent of Oracle tables)• Documents are distributed across all the shards

• Each sharded collection need a value to shard on (=shard key)

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

_ Example

MongoDB ConceptsMongoDB {

“name”: “Winky”}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

_ Example



mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

_ Example



mongos

SHARDSHARD SHARD SHARD

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

_ Example



mongos


INSERT in my_collection

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

_ Example



mongos



Chunk = HASH(doc.name)

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

_ Example



mongos



_ Remember, a shard is a replica set, it is spread over different machinesChunk = HASH(doc.name)

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Data ReplicationMongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Data Replication• A write goes first to the primary of a shard

MongoDB

NODE 1

Mongod(primary)

Replica Set/Shard

write

NODE 2

Mongod(secondary)

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Data Replication• A write goes first to the primary of a shard• The primary writes the operation in a special collection called oplog

MongoDB

NODE 1

Mongod(primary)

Replica Set/Shard

oplogwrite

NODE 2

Mongod(secondary)

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Data Replication• A write goes first to the primary of a shard• The primary writes the operation in a special collection called oplog• Secondaries periodically poll the oplog to get the latest updates

MongoDB

NODE 1

Mongod(primary)

Replica Set/Shard

oplogwrite

NODE 2

Mongod(secondary)

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Data Replication• Oplog window• Time difference between the first and the last entry in the oplog collection• Replication Lag• Time between the latest entry on the primary and the latest entry on a secondary• Replication headroom• Difference between Oplog Window and Replication Lag

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Journaling• When running with journaling, MongoDB stores and applies write

operations in memory and in the on-disk journal before the changes are present in the data files on disk• MongoDB journals operations by batch (called group commits), the size is

configurable• A group commit must block all writers during the commit• In case of a crash, mongod uses the journal to warm-up

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Chunk splitting• Each mongos knows how much it writes to each chunk• When a mongos thinks a chunk needs to be split (=reached the max chunk

size), it queries the related mongod and check the chunk size. If needed, it will start the balancer (if enabled)

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ When do we move a chunk ?MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ When do we move a chunk ?• A shards holds too many chunks

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

“I want to move this chunk”

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Global lock, no other chunk can be moved

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

“Please move chunk 4 to shard 2”

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Please poll this list of documents {..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Polling..

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Block read/writes

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Block read/writes

Update chunk map

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2Connects with outdated map

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Get new map2

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2 2

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Scalability• If we need more nodes, we add more shards (done through one of the

mongos running on NoSQL tier)• When you add a shard to a sharded cluster, you affect the balance of

chunks among the shards. The balancer will begin migrating chunks so that the cluster will achieve balance• The balancer has an impact on performance• It moves one chunk at a time

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Automatic RecoveryMongoDB

Primary

Secondary Arbiter

Replicationmongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Automatic Recovery• Mongod-mongod

heartbeat checks if the processes are up and contains the priority of the mongod

MongoDB

Primary

Secondary Arbiter

Heartbeat

Replicationmongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod

heartbeat checks the role of the processes

MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

PrimarySecondary

mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

PrimarySecondary

mongos

Write outage (read depends on configuration)

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

PrimarySecondary

mongos


Election

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

PrimarySecondary

mongos


Primary

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

PrimarySecondary

mongos

isMas

ter() Primary

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recoverymongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

Secondary

mongos 1. Read from journal

2. Pull oplog

Primary

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

Secondary

mongos

Primary

Secondary

Replication

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

PrimarySecondary

mongos Sends its priority in the heartbeat. In case it is the highest…

Secondary

Primary

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

Primary

Recovery

Primary

mongos

…self stepdown

Secondary

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

Primary

RecoveryPrimary

Secondary

mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts




MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

Primary

RecoveryPrimary

Secondary

mongos

isMaster()

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Rollback files• A rollback reverts write operations on a former primary when the member

rejoins its replica set after a failover. A rollback is necessary only if the primary had accepted write operations that the secondaries had not successfully replicated before the primary stepped down• When a rollback does occur, administrators must decide whether to apply

or ignore the rollback data. • Note: Rollback files can be avoided if the application uses replica

acknowledged write concern (=write successful only when written on replica)

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Full resyncMongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

2. Pull oplogmongos 1. Read from journal

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts

_ Full resync• In case the down time

is higher than the oplog window, a full resync is needed

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


is higher than the oplog window, a full resync is needed • 1. Stop process

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


is higher than the oplog window, a full resync is needed • 1. Stop process • 2. wipe-out data

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

MongoDB Concepts


is higher than the oplog window, a full resync is needed • 1. Stop process • 2. wipe-out data • 3. start process

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery


Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Training MongoDB

_ MongoDB processes• mongod (primary/secondary/arbiter), mongos, conf, agent, MMS

_ Replica Set / Shard, Shard Key, Shard Map_ Database, Collection, Document_ Replication (oplog)_ Journaling_ Capture/Restore_ Chunk Move_ Automatic Recovery / Election

Recap – What you should know by now

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Workflows – discover topologyMongoDB


AppTier

NoSQLTier MongoDB

_ Mongos startsAgent

BE BE BEBE

mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB

_ Mongos starts_ Reads Mongos.yml file

and connects to the config server

Agent

BE BE BEBE

mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB



_ Retrieves the shard map

Agent

BE BE BEBE

mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB



_ Retrieves the shard map_ Connects to all the

mongods

Agent

BE BE BEBE

mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB




mongods_ Retrieves the complete

list of processes for each shard from the mongods

Agent

BE BE BEBE

mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB




mongods_ Retrieves the complete

list of processes for each shard from the mongods

_ Retrieves the role of each process is the command isMaster()

Agent

BE BE BEBE

mongos

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Workflows - writeMongoDB


AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos

_ The BE wants to store a document

{..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos


_ The document is routed according to the shard key{..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos


_ The document is routed according to the shard key

_ Written only on primaryprimary

{..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos



_ Written only on primary_ And replicated to the

secondary asynchronouslyprimary secondary{..}{..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos



_ Written only on primary_ And replicated to the

secondary asynchronously_ The client can decide to

wait for a successful write in memory, on disk or on the secondaries

primary secondary{..}{..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Workflows – readMongoDB


AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos

_ The BE wants to read documents

{..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos


_ The query is routed according to the shard key{..}

primary

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos


_ The query is routed according to the shard key

_ Depending on the query, several mongods can be targeted primary primary

{..}{..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos



_ Depending on the query, several mongods can be targeted

_ Depending on the configuration, mongos could read on a primary or a secondary

primary primary secondary{..}{..} {..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA



AppTier

NoSQLTier MongoDB

Agent

BE BE BEBE

mongos



_ Depending on the query, several mongods can be targeted

_ Depending on the configuration, mongos could read on a primary or a secondary

_ Mongos will process the documents returned and reply to the BE

primary primary secondary

{..}

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA

Operability & Monitoring

_ Let’s check the new stuffs in the life cycle of a software in ADP

• Deployment• ACO• Logs• Monitoring• Automatic Alerts• Operations

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ New Pega flow defined for MongoDB clusters

_ Uses Puppet for installation automation

_ http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Pega_flow_Big_Data

Deployment

http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Pega_flow_Big_Data

http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Pega_flow_Big_Data

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ ULAM: mongos is deployed as a component of the OBE using it. _ Later, MongoDB upgrades will be done by ULAM too (not initial

installation)

Deployment

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ Called by ULAM during a load_ http://appcheckout.os.amadeus.net:8080/appcheckout/_ New check to be created for Mongos

Application Checkout (ACO)

http://appcheckout.os.amadeus.net:8080/appcheckout/

http://appcheckout.os.amadeus.net:8080/appcheckout/

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ Current design (might evolve)• Mongos sends its logs a binary using Tracer, this binary sends all the logs

(Error + Info) to the application logserver in “mongos_link”

Logs

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ http://sentinel.muc.amadeus.net/index.php?page=mongodbMonitoring_ http://sentinel.muc.amadeus.net/index.php?page=mongosMonitoring

Sentinel

http://sentinel.muc.amadeus.net/index.php?page=mongodbMonitoring



http://sentinel.muc.amadeus.net/index.php?page=mongosMonitoring

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ Uniping• New pinger for mongo (check the status of mongod, mongos and config

processes on database side)• http://uniping.muc.amadeus.net/watchdog/

_ Scheck• Nothing new, it checks nodes via ssh

_ Topx• Nothing new, it monitors memory

_ YALB • Monitors logs and raises specialized IRs for mongos on application side

MongoDB

http://uniping.muc.amadeus.net/watchdog/

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ Trends• Tracks IRs for Ops• http://gmstst.muc.amadeus.net/trends

MongoDB

http://gmstst.muc.amadeus.net/trends

http://gmstst.muc.amadeus.net/trends

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ OSAG• Operators and System Administrators Guide• It contains OPROCs, POCs, escalation procedures, Impact & Recovery of

configuration changes, Setup procedures and much more!• http

://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Mongos_and_MongoDB_OSAG

MongoDB




Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ For GOS• Escalation Procedure• http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Escalation_Proce

dure_for_MongoDB

• The link will be in each IR• We don’t expect any action from GOS on MongoDB side, recovery

procedures are fully automated. The actions available in Workbench can be used for test purposes

MongoDB

http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Escalation_Procedure_for_MongoDB

http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Escalation_Procedure_for_MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ Workbench• http://houston.muc.amadeus.net/index.php?page=workbench/launch

MongoDB

http://houston.muc.amadeus.net/index.php?page=workbench/launch

http://houston.muc.amadeus.net/index.php?page=workbench/launch

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ NoSQL tab_ Cluster status_ Node status_ Process Status

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ NoSQL tab_ Cluster status_ Node status_ Process Status_ Shard

distribution

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ Start/stop of a Secondary: Right click on the mongod process

_ Stepdown a primary: Right click on the mongod process

MongoDB

Confi

dent

ial

RESTRICTED

© 2

014

Amad

eus I

T Gr

oup

SA


_ Shard distribution

_ Highlight of the node hosting secondary processes for the selected node

MongoDB

Number of primaries

Primary Secondary

© 2

014

Amad

eus I

T Gr

oup

SA

Confi

dent

ial

RESTRICTED

You can follow us on:AmadeusITGroup

amadeus.com/blogamadeus.com

Thank you

Training MongoDB - Monitoring and Operability

Software

Transcript of Training MongoDB - Monitoring and Operability