Sizing Your MongoDB Cluster

FEBRUARY 15, 2018 | BELL HARBOR

#MDBlocal

Sizing MongoDB Clusters

#MDBlocal

Master Solutions Architect

Jay Runkel

MongoDB @jayrunkel

#MDBlocal

MongoDB: 4 years

NoSQL: 9 years

Solution Architect: engineer assigned to sales

Significant experience sizing MongoDB clusters

About me

#MDBlocal

Objective

#MDBlocal

Do I need to shard?

What size servers should I use?

What will my monthly Atlas/AWS/Azure/Google costs be?

When will I need to add a new shard or upgrade my servers?

How much data can my servers support?

How many queries can my servers support?

Will we be able to meet our query latency requirements?

Sizing

#MDBlocal

• Large coffee chain: PlanetDollar

• Collect mobile app performance

• Every tap, click, gesture will generate an event

• 2 Year History

• Perform analytics • Historical• Near real-time (executive dashboards)

• Support usage

• 3000 – 5000 events per second

Your boss comes to you…

I need a budget for the monthly Atlas costs?

#MDBlocal

• Build a prototype

• Run performance tests using actual data and queries on hardware with specs similar to production servers

The only accurate way to size a cluster

• EVERY OTHER APPROACH IS A GUESS

• Including the one I am presenting today

#MDBlocal

• Early in project, but

• Need to order hardware

• Estimate costs to determine “Go/No Go” decision

• Schema design

• Compare the hardware requirements for different schemas

Sometimes, it is necessary to guess ☹

#MDBlocal

MongoDB Clusters Look Like This

Config

Config

Config

Application

Driver

Primary

Secondary

Secondary

#MDBlocal

• # of shards

• Specifications of each server

• CPU

• Storage

• Size

• Performance: IOPS

• Memory

• Network

Our Solution Will Consist Of

#MDBlocal

• Sizing Objective ✓

• IOPS, Query Processing, Working Set

• Sizing Methodology with an Example

Agenda

#MDBlocal

IOPS, Query Processing,

Working Set

#MDBlocal

• # of shards

• Specifications of each server

• CPU

• Storage

• Size

• Performance: IOPS

• Memory

• Network

Our Solution Will Consist Of

#MDBlocal

• IOPS – input output units per second

• Throughput

• Random access

• Most workloads “randomly” access documents

IOPS

collection

#MDBlocal

Storage Performance

Type IOPS

7200 rpm SATA ~ 75 – 100

15000 rpm SAS ~ 175 – 210

RAID-10 (24 x 7200 RPM SAS) 2000

Amazon EBS 250 – 500

Amazon EBS Provisioned IOPS 10000 - 20000

SSD 50000

Flash Storage 100K – 400K (or more)

http://en.wikipedia.org/wiki/IOPS

http://www.thecloudcalculator.com/calculators/disk-raid-and-iops.html

http://www.thecloudcalculator.com/calculators/disk-raid-and-iops.html

#MDBlocal

• How many IOPS do we need?

• Want the real answer, run a test

• How to estimate?

Hardest Part of Sizing is IOPS

#MDBlocal

Processing a Query

Select Index

Load relevant index entries from disk

Identify documents using index

Retrieve documents from disk

Filter documents

Return Documents

#MDBlocal

Processing a Query

IO

Select Index




Filter documents

Return Documents

#MDBlocal

But MongoDB Has a Cache

File System

indexes collections

CPU

Memory

indexes

documents

Disk access is only necessary if indexes

or documents are not in cache

Select Index




Filter documents

Return Documents

#MDBlocal

Working Set

Working Set = indexes plus frequently accessed

documents

If RAM greater than working set then reduced IO

Select Index




Filter documents

Return Documents

File System

indexes collections

CPU

Memory

indexes

documents

#MDBlocal

This is all great, but how do we estimate IOPS?

#MDBlocal

Assume

• Working Set < RAM < Data Size

• Memory contains indexes only

MongoDB Simplified Model

File System

indexes collections

CPU

Memory

indexes

documents

#MDBlocal

Assume appropriate indexes

To resolve find:

• Navigate in-memory indexes

• Retrieve document from disk

1 IOP per document returned

Find Queries With Simplified Model

File System

indexes collections

CPU

Memory

indexes

documents

#MDBlocal

To resolve insert:

• Write document to disk

• Update each index file

IOPS = 1 + # of indexes

Inserts With Simplified Model

File System

indexes collections

CPU

Memory

indexes

documents

#MDBlocal

To resolve delete:


• Mark document deleted



Deletes With Simplified Model

File System

indexes collections

CPU

Memory

indexes

documents

#MDBlocal

To resolve delete:


• Mark document deleted

• Insert new document version



Updates With Simplified Model

File System

indexes collections

CPU

Memory

indexes

documents

#MDBlocal

• Working Set

• Checkpoints

• Document size relative to block size

• Indexed Arrays

• Journal, Log

The Simplified Model is too simplistic

#MDBlocal

• WiredTiger write process:

1. Update document in RAM (cache)

2. Write to journal (disk)

3. Periodically, write dirty documents to disk (checkpoint)

• 60 seconds or 2 GB (whichever comes first)

Checkpoints

Checkpoint 1 Checkpoint 2 Checkpoint 3

B C A A C A

3 writes3 documents written

3 writes2 documents written

#MDBlocal

• Estimate total requirements (using simplified model):

• RAM

• CPU

• Disk Space

• IOPS

• Adjust based upon working set, checkpoints, etc.

• Design (sharded) cluster that provides these totals

How are we going to get there?

#MDBlocal

Sizing Process

#MDBlocal

Methodology

Application

Requirements

Cluster Sizing• Number of

shards

• Server specsMagic

Happens

#MDBlocal

1.Collection Size

2.Working Set

3.Queries -> IOPS

4.Adjust based upon working set, checkpoints, etc.

5.Calculate # of shards

6.Review, iterate, repeat

Methodology (cont.)

Build a spread

sheet

Multiple

iterations may

be required

#MDBlocal

1.Assumptions

1.Data Size

1.Working Set

• Index Size

• Frequently Accessed Documents

1.Queries – IOPS

1.Shard Calculations

Sizing Spreadsheet

#MDBlocal

1.Assumptions

1.Data Size

1.Working Set

• Index Size


2. Queries – IOPS

1. Shard Calculations

Sizing Spreadsheet

#MDBlocal

https://github.com/jayrunkel/mdbw2017Sizing

Sizing Spreadsheet Download

#MDBlocal

Assumptions

#MDBlocal

Sizing Spreadsheet

1.Assumptions

1.Data Size

1.Working Set

• Index Size


2. Queries – IOPS

1. Shard Calculations

#MDBlocal

• # of documents

• Data size

• Index size

• WT compression

Collection Analysis

#MDBlocal

Calculate The Number of Documents

Application Description # of Documents in Collection

There will be 20M documents in the

collection by the end of 2017

20,000,000

We expect to insert 10K documents

per day with 1 year retention period

365*10,000 = 3,655,000

We have 3000 devices each

producing 1 event per minute and we

need to keep a 90 day history

3000 * 60 * 24 * 90 = 388,800,000

#MDBlocal

Calculate The Number of Documents

Application Description # of Documents in Collection

There will be 20M documents in the collection

by the end of 2017

20,000,000

We expect to insert 10K documents per day

with 1 year retention period

365*10,000 = 3,655,000

We have 3000 devices each producing 1 event

per minute and we need to keep a 90 day

history

3000 * 60 * 24 * 90 = 388,800,000

PlanetDollar:

2 year history. Each day 5000 inserts per

second for 5 hours and 3000 inserts per second

for 19 hours

2*365*(5000*5*3600 + 3000*19*3600) = 215,496,000,000

#MDBlocal

• Data Size = # of documents * Average document size

• This information is available in db.stats(), Compass, Ops Manager, Cloud Manager, Atlas, etc.

Calculate the Data Size

#MDBlocal

• Write some code

• Programmatically generate a large data set

• 5-10% of expected size

• Measure

• Collection size

• Index size

• Compression

What if there aren’t any documents?

#MDBlocal

• Use db.collection.stats()

• Take data size, index size and extrapolate to production size

• Calculate compression ratio

db.collection.stats()

{

count: 10000

size: 70,388,956

avgObjSize: 7038

storageSize: 25341952

…

totalIndexSize: 147456

}

Determine collection and data size

Parameter Formula Value

# of documents 2.5B

avgObjSize 7038

Collection Size =2.5B * 7038 1.760E13 Bytes

WT Compression = 25341952/70388956 .36

Collection Storage =2.5B * 7038 * .36 6.33E12 Bytes

Index Size Per Doc = 147456 / 10000 15 Bytes

Collection Index Size =2.5B * 15 /1024^3 35 GB

#MDBlocal

PlanetDollar - Collection Size

#MDBlocal

Sizing Spreadsheet

1.Assumptions

1.Data Size

1.Working Set

• Index Size


1.Queries – IOPS


#MDBlocal

WorkSet = Indexes plus the set of documents accessed frequently

We know the index size from previous analysis

Estimate the frequently

accessed documents

Given the queries

What are the frequentlyaccessed docs?

Working Set

File System

collections indexes

CPU

Memory

indexes

documents

#MDBlocal

Query Analysis

Dashboards – last minute of data

Customer support - last hour of data

Reports (run once per day) inspect last years worth of data

Active Documents = 1 hours worth of data

PlanetDollar Working Set

#MDBlocal

PlanetDollar - Working Set

#MDBlocal

1.Assumptions

1.Data Size

1.Working Set

• Index Size


1.Queries – IOPS


Sizing Spreadsheet

#MDBlocal

+ # of documents returned per second

+ # of documents updated per second

+ # of indexes impacted by each update

+ # of inserts per second

+ # of indexes impacted by each insert

+ # of deletes per second (x2)

+ # of indexes impacted by each delete

- Multiple updates occurring within checkpoint

- % of find query results in cache

Total IOPS

IOPS Calculation

#MDBlocal

• 5000 inserts per second

• 5000 deletes per second

Dashboards (aggregations: 100 per minute)

• Total events per minute across all users (current minute)

• Total events per minute per region (current minute)

• Total events per store per minute (current minute)

Debugging Tool (ad hoc – 5 per second)

• Find all events for a user in last 60 minutes (100 events returned, on average)

PlanetDollar Queries

#MDBlocal

• Each insert:

• Update collection

• Update each index (3 indexes)

• Each Delete:

• Update collection

• Update each index (3 indexes)

• 5000 inserts/sec

• 5000 deletes/sec

IOPS for inserts and deletes

4 IOPS

5 IOPS

(4 * 5000) + (5 * 5000) = 45000 IOPS

#MDBlocal

• Example: Total events per minute across all users (current minute)

• How many documents will be read from disk?

IOPS for PlanetDollar Aggregations

05000 per second * 60 seconds = 300,000

Most data in cache

Some IOPS will likely be required

#MDBlocal

• Find all events for a user in last 60 minutes

• 5 per second

• 100 documents per query

• # IOPS = 5 * 100 = 500 IOPS

IOPS For Find

#MDBlocal

PlanetDollar - Queries/IOPS

#MDBlocal

• CPU utilized for:

• Compress/decompress

• Encrypt/Decrypt

• Aggregation queries

• General query processing

• In most cases, RAM requirements → large servers → many cores

• Possible exception: aggregation queries

• One core per query

• # cores >> # of simultaneous aggregation queries

How Many CPUs Do I Need?

#MDBlocal

1.Assumptions

1.Data Size

1.Working Set

• Index Size


1.Queries – IOPS


Sizing Spreadsheet

#MDBlocal

• At this point you have:

1. Required storage capacity

2. Working Set Size

3. IOPS Estimate

4. Some idea about class of server (or VM) the customer plans to deploy

• Determine number of required shards

Shard Calculations

#MDBlocal

• Sum of disk space across shards greater than required storage size

Disk Space: How Many Shards Do I Need?

Example

Data Size = 9 TB

WiredTiger Compression Ratio: .33

Storage size = 3 TB

Req Disk: 6 TB

Server disk capacity = 2 TB

3 Shards Required

Recommend providing 2X the compressed data size in disk

#MDBlocal

RAM: How Many Shards Do I Need?

Example

Working Set = 428 GB

Server RAM = 128 GB

428/128 = 3.34

4 Shards Required

#MDBlocal

IOPS: How Many Shards Do I Need?

Example

Require: 50K IOPS

AWS Instance: 20K IOPS

3 Shards Required

#MDBlocal

PlanetDollar - Shard Calculations

#MDBlocal

1. Calculate:

• Collection size

• Index size

2. Estimate Working Set

3. Use simplified model to estimate IOPS

4. Revise (working set coverage, checkpoints, etc.)

5. Calculate shards

Sizing Summary

#MDBlocal

Questions?

https://github.com/jayrunkel/mdbw2017Sizing

Sizing Your MongoDB Cluster

Technology

Transcript of Sizing Your MongoDB Cluster