Lessons Learned from Building and Operating Scuba

Lessons learned from building and operating Scuba

Ciprian GereaFacebook

events

metrics

livehistorical

Presto & Hive

Demo time

• Getting started– Writing to Scuba

What is Scuba

• Database– Real time ingestion & queries– Simple query model: rollups, no joins– Simple data model, flexible schema

• UI platform• Service– Runs its own ETL– Demand control

• Retention• Queries

Scribelogsfrom

serversScuba GUI

`scuba` CLI

Scuba gauge ScriptAlerts

combinedlogs

for each scribe

category

Tupperware

manage perfpipe

tailer

Tailer

Data storagerockfortexpress.wildcard

SMC tier in PRN1

Scuba backend

Root aggregator

adddirectly to

leaf servers

queries

results

SparkleTable insertion counts

Scuba system architecture

dataswarm

HiveToScuba

Scuba DB

• Data lives in tables• Columns can be: int/string/vector<string>• Can change schema on the fly.• Shared nothing storage in memory & flash• Data sharded at random• Only support rollup queries:

sum/avg/percentile.• Best effort queries: skip bad nodes.

Demo time

• Let’s run some queries• Customize the UI• ETL control

How we keep it running

• ODS metrics for everything• Scuba data sets for queries & subsystems we’re

actively debugging• Dashboards– Cubism is king!– Unidash for niche cases

• Active management of demand– Table size quotas– CPU load -> push to stream processing

Root cause for outages

• Other systems– Scribe– Hosting layer– Deployment mechanism

• Media failures: high on disk, low on flash.• Queries of doom• High load– DOS workloads– Load shedding bugs

Why it is successful

• Scuba’s niche: – Easy to get started– Fast <50ms P50 wall time– Smooth learning curve– UI is customizable (~1k custom presenters!!!)– Its flaws are acceptable• Not everyone needs transactions from the beginning• Users are OK with retrying queries

• Other tools don’t serve this niche well

What could be better

• Customers ask– More space– More consistent results– More expressive queries

• Sharding• Better persistent storage• Better support for time series

Cubism Intro

• Horizon charts

Lessons Learned from Building and Operating Scuba

Technology

Transcript of Lessons Learned from Building and Operating Scuba

Lessons&Learned&in&Deploying&Akka& Persistencedownloads.typesafe.com/website/presentations/lessons-learned-akk… · Lessons&Learned&in&Deploying&Akka& Persistence Duncan&K.&DeVore

Lessons learned from accident investigation Seminar/IHS - Day 2... · LESSONS LEARNED FROM OTHERS LESSONS LEARNED FROM UAE ... Lessons Learned . ... Dealing with survivors and bereaved

Lessons Learned and Lessons to Be Learned: Investment Law & … · 2014. 4. 30. · 1 Lessons Learned and Lessons to Be Learned: Investment Law & Development for Developed Countries

Lessons Learned from “Lessons Learned” · 2 LESSONS LEARNED FROM “LESSONS LEARNED” 2. The NRC set safety goals in its Safety Goal Policy Statement, initiated not long after

Lessons Learned:

Lessons learneD

Lessons learned & not learned at MSU

Lessons Learned Handbook · 2011-10-12 · • Lessons Learned, an adjective, describes anything related to a Lessons Learned procedure. E.g. Lessons Learned process, Lessons Learned

Lessons Learned

LESSONS LEARNED AND GOOD PRACTICES...Lessons Learned Lessons are learned for sharing and applying to similar projects. Project lessons are learned from experiences, both inspiring

ASSESSMENT & LESSONS LEARNED FROM AFRICAN DIASPORA MARKETPLACEpdf.usaid.gov/pdf_docs/PDACT256.pdf · ASSESSMENT & LESSONS LEARNED FROM AFRICAN ... ASSESSMENT & LESSONS LEARNED FROM

LEARNED LESSONS:

Scuba Diving Lessons

Lessons Learned Events at SLAC · 2012-09-14 · Lessons Learned Events: Summary Comments . Important to identify and communicate lessons learned! Lessons learned and near misses

Successes, Challenges, & Lessons Learned...Successes, Challenges, & Lessons Learned Overview Introduction Successes Challenges Lessons Learned New Approaches Highway 70 To Phoenix

Lean lessons learned - lean startup methodology and lessons learned from Blossom

Lessons Learned Globally - Campaign for Tobacco-Free Kids...Process lessons learned 2. Content lessons learned See next page for lists of these lessons learned. 7 | Lessons Learned

Lessons Learned Remote PM+ Cohort Integrated K+S · Lessons Learned Below are key lessons learned on providing Remote PM+ Training of Helpers. These lessons learned are based off