Training MongoDB - Monitoring and Operability

RESTRICTED

MongoDB Monitoring & Operability

Training for App Mgmt and Operations

Nicolas Motte, 18/11/2014

RESTRICTED

Link to this presentation

_ http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Mongos_and_MongoDB_OSAG• Section 10. Training

_ http://gmstst.muc.amadeus.net/trends/• NoSQL section• Select “Mongos and MongoDB OSAG”• Section 10. Training

RESTRICTED

Agenda_ Overview of MongoDB_ Amadeus Architecture around NoSQL_ MongoDB concepts_ Workflows_ Operability & Monitoring

414213741

RESTRICTED

Overview of MongoDB1

RESTRICTED

Overview of MongoDB

_ Document store

High-level

RESTRICTED

Overview of MongoDB

_ Document store• Stores JSON-style documents

High-level

RESTRICTED

Overview of MongoDB

_ Document store• Stores JSON-style documents• Uses BSON format for storage

High-level

RESTRICTED

Overview of MongoDB

_ Several binaries written in C++

High-level

RESTRICTED

Overview of MongoDB

_ Several binaries written in C++• Mongod : The actual database process, owning the data

High-level

RESTRICTED

Overview of MongoDB

_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer

High-level

RESTRICTED

Overview of MongoDB

_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology

High-level

RESTRICTED

Overview of MongoDB

_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology• Agent : Monitoring process

High-level

RESTRICTED

Overview of MongoDB

_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology• Agent : Monitoring process• MMS : Monitoring UI, connected to the Agents

High-level

RESTRICTED

Overview of MongoDB

_ Several binaries written in C++• Mongod : The actual database process, owning the data• Mongos : A connection multiplexer• Config servers : Holds cluster topology• Agent : Monitoring process• MMS : Monitoring UI, connected to the Agents

_ Opensource and Enterprise versions

High-level

RESTRICTED

Overview of MongoDB

Mongod

Architecture

RESTRICTED

Overview of MongoDB

Mongod

Mongos

Architecture

RESTRICTED

Overview of MongoDB

Mongod

Mongos

Config

Architecture

RESTRICTED

Overview of MongoDB

Mongod

Mongos

ConfigClient

Client

Architecture

RESTRICTED

Overview of MongoDB

Mongod

Mongos

ConfigClient

Client

Architecture

RESTRICTED

Overview of MongoDB

Mongod

Mongos

ConfigClient

Client

Architecture

RESTRICTED

Overview of MongoDB

_ Why do we use MongoDB ?Couchbase

MemcachedMongoDB

Oracle

Comparison

RESTRICTED

Overview of MongoDB

Memcached Couchbase MongoDB OracleDatabase Model Key-Value Key-Value Document TableData scheme Schema-free Schema-free Schema-free Schema-fullIndexesPersistencyData IntegrityTransaction

Comparison

RESTRICTED

Overview of MongoDB

Comparison

RESTRICTED

Overview of MongoDB

Comparison

RESTRICTED

Overview of MongoDB

Comparison

RESTRICTED

Overview of MongoDB

Comparison

RESTRICTED

Overview of MongoDB

Comparison

RESTRICTED

Overview of MongoDB

Comparison

RESTRICTED

Overview of MongoDB

_ Mongod is the process storing the data

Mongod

RESTRICTED

Overview of MongoDB

_ Mongod is the process storing the data_ 3 different types

Mongod

RESTRICTED

Overview of MongoDB

_ Mongod is the process storing the data_ 3 different types• PRIMARY

Mongod

RESTRICTED

Overview of MongoDB

_ Mongod is the process storing the data_ 3 different types• PRIMARY• The primary is the only process that

receives write operations

Mongod

RESTRICTED

Overview of MongoDB

receives write operations• SECONDARY

Mongod

RESTRICTED

Overview of MongoDB

receives write operations• SECONDARY• A secondary maintains a copy of the

primary’s data set

Mongod

RESTRICTED

Overview of MongoDB

primary’s data set• ARBITER

Mongod

RESTRICTED

Overview of MongoDB

primary’s data set• ARBITER• An arbiter does not have a copy of

data set and cannot become a primary

Mongod

RESTRICTED

Overview of MongoDB

data set and cannot become a primary

Mongod

RESTRICTED

Overview of MongoDB

data set and cannot become a primary_ Heartbeats are sent every 10s

between mongods

Mongod

Heartbeat

RESTRICTED

Overview of MongoDB

_ Config servers are special mongod instances that store the metadata for a cluster (like topology)

Config Server

RESTRICTED

Overview of MongoDB

_ Config servers are special mongod instances that store the metadata for a cluster (like topology)

_ A production cluster has exactly three config servers

Config Server

RESTRICTED

Overview of MongoDB

_ Mongos is a connection multiplexer

_ Heartbeat are sent every 10s

_ Map/Reduce queries

Mongos

A single connection/server

Clients are not aware of the topology

Mongos hides all the complexity and standardizes

the configuration

MongosMongod

Mongod

Config

Heartbeat

RESTRICTED

Overview of MongoDB

Mongos

RESTRICTED

Overview of MongoDB

Mongos

MongosMongod

Mongod

Config

RESTRICTED

Overview of MongoDB

Mongos

MongosMongod

Mongod

Config

RESTRICTED

Overview of MongoDB

Mongos

MongosMongod

Mongod

Config

RESTRICTED

Overview of MongoDB

Mongos

the configuration

MongosMongod

Mongod

Config

RESTRICTED

Overview of MongoDB

Mongos

the configuration

MongosMongod

Mongod

Config

Heartbeat

RESTRICTED

Overview of MongoDB

_ Map/Reduce queries

Mongos

the configuration

MongosMongod

Mongod

Config

Heartbeat

RESTRICTED

Overview of MongoDB

_ Monitoring process

RESTRICTED

Overview of MongoDB

_ Monitoring process_ 1 active/1 passive

per cluster

RESTRICTED

Overview of MongoDB

per cluster_ Retrieves the

config servers from MMS

RESTRICTED

Overview of MongoDB

_ Retrieves the list of processes from a config server

Config

RESTRICTED

Overview of MongoDB

_ Retrieves the list of processes from a config server

_ Connects to all processes

Mongod

Mongos

Config

RESTRICTED

_ Datastore• Pros• Structured data• Flexible (Schema-less)• Indexes• Scaling is “easy”• Easy to deploy• Cons• Transactions are not yet supported• Each command is atomic but locking is at database level in 2.6 (moving to

document level in 2.8)_ Analytical search• MSX

Use CasesOverview of MongoDB

RESTRICTED

Amadeus Architecture Around NoSQL2

RESTRICTED

Mongos

AS OBE(GOS, App Support)

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongos ?

Amadeus Architecture Around NoSQL

RESTRICTED

Mongos

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongos ?

RESTRICTED

Mongos

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongos ?_ Deployed on each AS

mongos

RESTRICTED

Mongos

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongos ?_ Deployed on each AS_ BEs are connected via the

standard Mongo C++ client

BE BE BEBE

mongos

RESTRICTED

Mongos

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongos is connected to all the MongoDB servers

BE BE BEBE

mongos

RESTRICTED

Mongos

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Duwrapped process monitored by the local Agent

SI Agent

BE BE BEBE

mongosdwi

RESTRICTED

Mongos

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Equivalent of ProxyD for Memcached and Couchbase

SI Agent

BE BE BEBE

mongosdwi

RESTRICTED

Mongos

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Equivalent of ProxyD for Memcached and Couchbase

_ Also running on NoSQL Tier for administration tasks

SI Agent

BE BE BEBE

mongosdwi

mongos

RESTRICTED

Mongod

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongod ?

RESTRICTED

Mongod

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongod ?

RESTRICTED

Mongod

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Mongod ?_ Deployed on cluster

mongodmongod

RESTRICTED

Mongod

AppTier

NoSQLTier MongoDB

(GOS, DZB)

nodes_ Accepts connections

from Mongos on App Tier and NoSQL Tiermongod

mongodmongod

mongod

mongos

RESTRICTED

Mongod

AppTier

NoSQLTier MongoDB

(GOS, DZB)

nodes_ Accepts connections

from Mongos on App Tier and NoSQL Tier

_ A mongod can be a primary, a secondary or an arbiter

mongodmongod

mongos

RESTRICTED

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Agent?

mongodmongod

mongos

RESTRICTED

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Agent?_ Deployed on 2

cluster nodes, 1 active / 1 passive

mongodmongod

mongos

RESTRICTED

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Agent?_ Deployed on 2

cluster nodes, 1 active / 1 passive

_ Connects to Mongos to monitor all processes

mongodmongod

mongos

RESTRICTED

Config Server

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Config servers?

mongodmongod

mongos

RESTRICTED

Config Server

AppTier

NoSQLTier MongoDB

(GOS, DZB)

_ Config servers?_ Running on

dedicated nodes

mongodmongod

mongos

config

RESTRICTED

Config Server

AppTier

NoSQLTier MongoDB

(GOS, DZB)

dedicated nodes_ Accepts connections

from mongosmongod

mongodmongod

mongod

mongos

config

mongos

RESTRICTED

Config Server

AppTier

NoSQLTier MongoDB

(GOS, DZB)

dedicated nodes_ Accepts connections

from mongos_ These nodes are

treated like any other cluster nodes in our tools

mongodmongod

mongos

config

mongos

RESTRICTED

Amadeus Architecture Around NoSQLConfiguration

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

MongoS port retrieved from

the MAGdwi

Duwrapper Conf(listen port)

+Configuration files

Keyfile_xxxMongos_xxx.yml

xxx.pem

_ BE Configuration• http://rndwww.nce.amadeus.n

et/wiki/wikidoc/index.php/BE_Configuration_(MongoDB)

_ Mongos Configuration• http://rndwww.nce.amadeus.n

et/wiki/wikidoc/index.php/Mongos_Configuration

_ MongoDB Configuration• Handled entirely by DZB

RESTRICTED

MongoDB concepts3

RESTRICTED

MongoDB Concepts

_ Replica Set or ShardMongoDB

RESTRICTED

MongoDB Concepts

_ Replica Set or Shard• A replica set is a group of mongod

instances that host the same data set

MongoDB

Replica Set

RESTRICTED

MongoDB Concepts

_ Replica Set or Shard• A replica set is a group of mongod

instances that host the same data set

MongoDB

Replica Set

RESTRICTED

MongoDB Concepts

_ Shard Key

• Determines the distribution of the documents among the cluster’s shards

• The shard key is an indexed field that exists in every document

MongoDB

RESTRICTED

MongoDB Concepts

_ Shard Map• MongoDB partitions data in

“chunks”, chunks are associated to a replica set

MongoDB

Shard Map

Shard 1 Shard 2 Shard 3

RESTRICTED

MongoDB Concepts

“chunks”, chunks are associated to a replica set• When a chunk grows over the

chunk size (64 MB), MongoDB splits the chunk in smaller chunks. The (arbitrary) limit has been chosen to avoid moving too much data during a balancing.

MongoDB

Shard Map

RESTRICTED

MongoDB Concepts

“chunks”, chunks are associated to a replica set• When a chunk grows over the

chunk size (64 MB), MongoDB splits the chunk in smaller chunks. The (arbitrary) limit has been chosen to avoid moving too much data during a balancing.• The config server provides a

shard map with at least one mongod process per shard

MongoDB

Shard Map

RESTRICTED

MongoDB Concepts

_ MongoDB functional architectureMongoDB

RESTRICTED

MongoDB Concepts

_ MongoDB functional architecture• A MongoDB cluster contains several databases

MongoDB

RESTRICTED

MongoDB Concepts

_ MongoDB functional architecture• A MongoDB cluster contains several databases• Each database contains several collections (equivalent of Oracle schemas)

MongoDB

RESTRICTED

MongoDB Concepts

_ MongoDB functional architecture• A MongoDB cluster contains several databases• Each database contains several collections (equivalent of Oracle schemas)• A collection contains a set of JSON documents (equivalent of Oracle tables)

MongoDB

RESTRICTED

MongoDB Concepts

_ MongoDB functional architecture• A MongoDB cluster contains several databases• Each database contains several collections (equivalent of Oracle schemas)• A collection contains a set of JSON documents (equivalent of Oracle tables)• Documents are distributed across all the shards

• Each sharded collection need a value to shard on (=shard key)

MongoDB

RESTRICTED

_ Example

MongoDB ConceptsMongoDB {

“name”: “Winky”}

RESTRICTED

_ Example

mongos

RESTRICTED

_ Example

mongos

SHARDSHARD SHARD SHARD

RESTRICTED

_ Example

mongos

INSERT in my_collection

RESTRICTED

_ Example

mongos

Chunk = HASH(doc.name)

RESTRICTED

_ Example

mongos

RESTRICTED

_ Example

mongos

RESTRICTED

_ Example

mongos

RESTRICTED

_ Example

mongos

_ Remember, a shard is a replica set, it is spread over different machinesChunk = HASH(doc.name)

RESTRICTED

MongoDB Concepts

_ Data ReplicationMongoDB

RESTRICTED

MongoDB Concepts

_ Data Replication• A write goes first to the primary of a shard

MongoDB

NODE 1

Mongod(primary)

Replica Set/Shard

NODE 2

Mongod(secondary)

RESTRICTED

MongoDB Concepts

_ Data Replication• A write goes first to the primary of a shard• The primary writes the operation in a special collection called oplog

MongoDB

NODE 1

Mongod(primary)

Replica Set/Shard

oplogwrite

NODE 2

Mongod(secondary)

RESTRICTED

MongoDB Concepts

_ Data Replication• A write goes first to the primary of a shard• The primary writes the operation in a special collection called oplog• Secondaries periodically poll the oplog to get the latest updates

MongoDB

NODE 1

Mongod(primary)

Replica Set/Shard

oplogwrite

NODE 2

Mongod(secondary)

RESTRICTED

MongoDB Concepts

_ Data Replication• Oplog window• Time difference between the first and the last entry in the oplog collection• Replication Lag• Time between the latest entry on the primary and the latest entry on a secondary• Replication headroom• Difference between Oplog Window and Replication Lag

MongoDB

RESTRICTED

MongoDB Concepts

_ Journaling• When running with journaling, MongoDB stores and applies write

operations in memory and in the on-disk journal before the changes are present in the data files on disk• MongoDB journals operations by batch (called group commits), the size is

configurable• A group commit must block all writers during the commit• In case of a crash, mongod uses the journal to warm-up

MongoDB

RESTRICTED

MongoDB Concepts

_ Chunk splitting• Each mongos knows how much it writes to each chunk• When a mongos thinks a chunk needs to be split (=reached the max chunk

size), it queries the related mongod and check the chunk size. If needed, it will start the balancer (if enabled)

MongoDB

RESTRICTED

MongoDB Concepts

_ When do we move a chunk ?MongoDB

RESTRICTED

MongoDB Concepts

_ When do we move a chunk ?• A shards holds too many chunks

MongoDB

RESTRICTED

MongoDB Concepts

_ When do we move a chunk ?• A shards holds too many chunks• A new shard is added

MongoDB

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

“I want to move this chunk”

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Global lock, no other chunk can be moved

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

“Please move chunk 4 to shard 2”

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Please poll this list of documents {..}

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Polling..

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Block read/writes

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Block read/writes

Update chunk map

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2Connects with outdated map

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2

Get new map2

RESTRICTED

MongoDB Concepts

MongoDB

mongos

SHARD2

SHARD1config

mongos1 1 1 1 2 21 1 1 1 2 2 2

RESTRICTED

MongoDB Concepts

_ Scalability• If we need more nodes, we add more shards (done through one of the

mongos running on NoSQL tier)• When you add a shard to a sharded cluster, you affect the balance of

chunks among the shards. The balancer will begin migrating chunks so that the cluster will achieve balance• The balancer has an impact on performance• It moves one chunk at a time

MongoDB

RESTRICTED

MongoDB Concepts

_ Automatic RecoveryMongoDB

Primary

Secondary Arbiter

Replicationmongos

RESTRICTED

MongoDB Concepts

_ Automatic Recovery• Mongod-mongod

heartbeat checks if the processes are up and contains the priority of the mongod

MongoDB

Primary

Secondary Arbiter

Heartbeat

Replicationmongos

RESTRICTED

MongoDB Concepts

heartbeat checks if the processes are up and contains the priority of the mongod• Mongos-mongod

heartbeat checks the role of the processes

MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

PrimarySecondary

mongos

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

PrimarySecondary

mongos

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

PrimarySecondary

mongos

Write outage (read depends on configuration)

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

PrimarySecondary

mongos

Election

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

PrimarySecondary

mongos

Primary

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

PrimarySecondary

mongos

ter() Primary

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recoverymongos

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

Secondary

mongos 1. Read from journal

2. Pull oplog

Primary

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

Secondary

mongos

Primary

Secondary

Replication

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

PrimarySecondary

mongos Sends its priority in the heartbeat. In case it is the highest…

Secondary

Primary

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

Primary

Recovery

Primary

mongos

…self stepdown

Secondary

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

Primary

RecoveryPrimary

Secondary

mongos

RESTRICTED

MongoDB Concepts

MongoDB

Primary

Secondary Arbiter

Heartbeat

Replication

Primary

RecoveryPrimary

Secondary

mongos

isMaster()

RESTRICTED

MongoDB Concepts

_ Rollback files• A rollback reverts write operations on a former primary when the member

rejoins its replica set after a failover. A rollback is necessary only if the primary had accepted write operations that the secondaries had not successfully replicated before the primary stepped down• When a rollback does occur, administrators must decide whether to apply

or ignore the rollback data. • Note: Rollback files can be avoided if the application uses replica

acknowledged write concern (=write successful only when written on replica)

MongoDB

RESTRICTED

MongoDB Concepts

_ Full resyncMongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

2. Pull oplogmongos 1. Read from journal

RESTRICTED

MongoDB Concepts

_ Full resync• In case the down time

is higher than the oplog window, a full resync is needed

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

RESTRICTED

MongoDB Concepts

is higher than the oplog window, a full resync is needed • 1. Stop process

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

RESTRICTED

MongoDB Concepts

is higher than the oplog window, a full resync is needed • 1. Stop process • 2. wipe-out data

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

RESTRICTED

MongoDB Concepts

is higher than the oplog window, a full resync is needed • 1. Stop process • 2. wipe-out data • 3. start process

MongoDB

Primary

Secondary Arbiter

Heartbeat

Primary

Recovery

RESTRICTED

Training MongoDB

_ MongoDB processes• mongod (primary/secondary/arbiter), mongos, conf, agent, MMS

_ Replica Set / Shard, Shard Key, Shard Map_ Database, Collection, Document_ Replication (oplog)_ Journaling_ Capture/Restore_ Chunk Move_ Automatic Recovery / Election

Recap – What you should know by now

RESTRICTED

Workflows4

RESTRICTED

Workflows – discover topologyMongoDB

AppTier

NoSQLTier MongoDB

_ Mongos startsAgent

BE BE BEBE

mongos

RESTRICTED

AppTier

NoSQLTier MongoDB

_ Mongos starts_ Reads Mongos.yml file

and connects to the config server

BE BE BEBE

mongos

RESTRICTED

AppTier

NoSQLTier MongoDB

_ Retrieves the shard map

BE BE BEBE

mongos

RESTRICTED

AppTier

NoSQLTier MongoDB

_ Retrieves the shard map_ Connects to all the

mongods

BE BE BEBE

mongos

RESTRICTED

AppTier

NoSQLTier MongoDB

mongods_ Retrieves the complete

list of processes for each shard from the mongods

BE BE BEBE

mongos

RESTRICTED

AppTier

NoSQLTier MongoDB

mongods_ Retrieves the complete

list of processes for each shard from the mongods

_ Retrieves the role of each process is the command isMaster()

BE BE BEBE

mongos

RESTRICTED

Workflows - writeMongoDB

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ The BE wants to store a document

RESTRICTED

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ The document is routed according to the shard key{..}

RESTRICTED

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ The document is routed according to the shard key

_ Written only on primaryprimary

RESTRICTED

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ Written only on primary_ And replicated to the

secondary asynchronouslyprimary secondary{..}{..}

RESTRICTED

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ Written only on primary_ And replicated to the

secondary asynchronously_ The client can decide to

wait for a successful write in memory, on disk or on the secondaries

primary secondary{..}{..}

RESTRICTED

Workflows – readMongoDB

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ The BE wants to read documents

RESTRICTED

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ The query is routed according to the shard key{..}

primary

RESTRICTED

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ The query is routed according to the shard key

_ Depending on the query, several mongods can be targeted primary primary

{..}{..}

RESTRICTED

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ Depending on the query, several mongods can be targeted

_ Depending on the configuration, mongos could read on a primary or a secondary

primary primary secondary{..}{..} {..}

RESTRICTED

AppTier

NoSQLTier MongoDB

BE BE BEBE

mongos

_ Depending on the query, several mongods can be targeted

_ Depending on the configuration, mongos could read on a primary or a secondary

_ Mongos will process the documents returned and reply to the BE

primary primary secondary

RESTRICTED

Operability & Monitoring5

RESTRICTED

Operability & Monitoring

_ Let’s check the new stuffs in the life cycle of a software in ADP

• Deployment• ACO• Logs• Monitoring• Automatic Alerts• Operations

MongoDB

RESTRICTED

_ New Pega flow defined for MongoDB clusters

_ Uses Puppet for installation automation

_ http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Pega_flow_Big_Data

Deployment

RESTRICTED

_ ULAM: mongos is deployed as a component of the OBE using it. _ Later, MongoDB upgrades will be done by ULAM too (not initial

installation)

Deployment

RESTRICTED

_ Called by ULAM during a load_ http://appcheckout.os.amadeus.net:8080/appcheckout/_ New check to be created for Mongos

Application Checkout (ACO)

RESTRICTED

_ Current design (might evolve)• Mongos sends its logs a binary using Tracer, this binary sends all the logs

(Error + Info) to the application logserver in “mongos_link”

RESTRICTED

_ New section in InfrastructureSentinel

RESTRICTED

_ http://sentinel.muc.amadeus.net/index.php?page=mongodbMonitoring_ http://sentinel.muc.amadeus.net/index.php?page=mongosMonitoring

Sentinel

RESTRICTED

_ Uniping• New pinger for mongo (check the status of mongod, mongos and config

processes on database side)• http://uniping.muc.amadeus.net/watchdog/

_ Scheck• Nothing new, it checks nodes via ssh

_ Topx• Nothing new, it monitors memory

_ YALB • Monitors logs and raises specialized IRs for mongos on application side

MongoDB

RESTRICTED

_ Trends• Tracks IRs for Ops• http://gmstst.muc.amadeus.net/trends

MongoDB

RESTRICTED

_ New links to MongoDB OSAGs/OPROCsMongoDB

RESTRICTED

_ OSAG• Operators and System Administrators Guide• It contains OPROCs, POCs, escalation procedures, Impact & Recovery of

configuration changes, Setup procedures and much more!• http

://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Mongos_and_MongoDB_OSAG

MongoDB

RESTRICTED

_ For GOS• Escalation Procedure• http://rndwww.nce.amadeus.net/wiki/wikidoc/index.php/Escalation_Proce

dure_for_MongoDB

• The link will be in each IR• We don’t expect any action from GOS on MongoDB side, recovery

procedures are fully automated. The actions available in Workbench can be used for test purposes

MongoDB

RESTRICTED

_ Workbench• http://houston.muc.amadeus.net/index.php?page=workbench/launch

MongoDB

RESTRICTED

Operability & MonitoringMongoDB

RESTRICTED

_ NoSQL tab

RESTRICTED

_ NoSQL tab_ Cluster status

RESTRICTED

_ NoSQL tab_ Cluster status_ Node status

RESTRICTED

_ NoSQL tab_ Cluster status_ Node status_ Process Status

RESTRICTED

_ NoSQL tab_ Cluster status_ Node status_ Process Status_ Shard

distribution

RESTRICTED

_ Start/stop of a Secondary: Right click on the mongod process

_ Stepdown a primary: Right click on the mongod process

MongoDB

RESTRICTED

_ Shard distributionMongoDB

RESTRICTED

Number of primaries

RESTRICTED

Number of primaries

Primary

RESTRICTED

Number of primaries

Primary Secondary

RESTRICTED

_ Shard distribution

_ Highlight of the node hosting secondary processes for the selected node

MongoDB

Number of primaries

Primary Secondary

RESTRICTED

You can follow us on:AmadeusITGroup

amadeus.com/blogamadeus.com

Thank you

Training MongoDB - Monitoring and Operability

Software

Transcript of Training MongoDB - Monitoring and Operability

Benchmarking TimescaleDB vs. MongoDB for Time-Series Data · Benchmarking TimescaleDB vs. MongoDB 3 Overview Time-series data is popping up in many places: DevOps and monitoring,

Hazard & Operability {HAZOP}

OPerability Dynamics 2007

MongoDB Monitoring Become a MongoDB DBA - Monitoring ... Live... · Copyright 2017 Severalnines AB Shards make write scaling transparently Sharding can be solved with two methods:

Using the MongoDB Monitoring Service (MMS)

Mongo db monitoring 어떤 것이 좋을까(mongodb korea, 김인범)

MongoDB Performance Tuning and Monitoring

Hazard and Operability

Operability TroubleShooting

WiMAX Operability Solution

MongoDB Europe 2016 - MongoDB 3.4 preview and introduction to MongoDB Atlas

Automate MongoDB with MongoDB Management Service

Operability in Process Design: Achieving Safe, Profitable ... 2013/Operability book... · Operability in Process Design: Achieving Safe, Profitable, and Robust Process Operations

Operability WhitePaper Ver2.0

Percona Monitoring and Management Demonstration · •MySQL/MariaDB/MongoDB •Reduce costs and optimize performance Percona Monitoring and Management Demos •Point-in-time visibility

Operability fuel parameters

Run MongoDB with Confidence: Backing up and Monitoring with MMS

MongoDB - Monitoring & queueing

Operability in Process Design: Achieving Safe, Profitable ...pc-education.mcmaster.ca/Operability/Operability book/Chapter 5 Safety.pdf · F&EI Dow’s Fire and Explosion Index fI

Operability in Process Design: Achieving Safe, Profitable ...pc-education.mcmaster.ca/Operability/Operability book/Chapter 4... · Operability in Process Design: Achieving Safe, Profitable,