My mom told me that Git doesn’t scale by Vicent Martí

Saturday, May 11, 13

These are the things you don’t

care about


github


github

Git hosting:No longer a pain in the ass


github

Git hosting:No longer a pain in the ass

for you.Not for us.

Because, goddamnit,if I ever find the guy who invented

this thing I’m going to hang him froma fence by his underwear and.Saturday, May 11, 13

Let’s host some Git repos!

file.c src

file.h README.md COPYING.md

.git

Bare Repository

HEAD index objects refs

git-daemon


OK, now about the web...

grit

Ruby - Gitinterface


OK, now about the web...

grit

Bare Repository

Bare Repository

Bare Repository

Ruby - Gitinterface


1VMgrit

📽storage

rails app


nVM 📽storage


nVM 📽storage(GFS)


Rails was making us slow.


Literally.Saturday, May 11, 13

Time to move to

Real Hardware


fileservers

frontends

📸db


fileservers

frontends

📸db

?????????


smoke


bert(binary Erlang term)


bert(binary Erlang term)ernie

(not an acronym)


📸chimney(Redis)

frontend

fileserver

smoke

grit

erniegrit


Vertical Scaling#realtalk


bottleneck:

grit


bottleneck:

grit

solution:

git

shell out to


bottleneck:

git

shell out to


bottleneck:

git

solution:

git

shell out to

shell out to


bottleneck:

git

solution:

git

shell out to

shell out to

properly


GUISE


GUISEGUISE


GUISEGUISEGUISE


GUISEGUISEGUISE

...what?


Why don’t we take


Why don’t we take

the Git binary...


Why don’t we take

the Git binary...yeah?


Why don’t we take

the Git binary...yeah? and compile it as


Why don’t we take


a library


Why don’t we take


a libraryoh... go on...


Why don’t we take



and link that into


Why don’t we take



and link that intoour server


ScientificGraph™


Mem

ory

Usa

ge

Time

ScientificGraph™


Well, we didn’t think about

freeing memory, but...



freeing memory, but...THIS IS THE KIND

OF PROBLEMWE COULD SOLVE

WITH CGISaturday, May 11, 13


freeing memory, but...THIS IS THE KIND

OF PROBLEMWE COULD SOLVE

WITH CGIIN 1995


ScientificGraph™


Mem

ory

Usa

ge

Time

ScientificGraph™


What do you mean

the server died?


die("BUG: non-INDEX attr direction

in a bare repo");

die("a bad revision is needed");

die("'%s' is not a valid branch name.", name); die("Empty patc

h.

Aborted.");

die("unable to read index file");

What do you mean

the server died?


libgit


libgit2the “2” means this

one

frees memory


libgit2the “2” means this

one

frees memory

NOT ENOUGH

ABSTRACT

FACTORIES


JGitthe “J” means this oneis in Java

...not our thing.Saturday, May 11, 13

Javaa brief timeline

New companies don’t use Java

because it’snot like Unix

1995

New companies use Java

because it’snew and shiny

1997


because it’sooooooold

2005

New companies use the JVM

because WEBSCALE

2011


Javaa brief timeline


because it’snot like Unix

1995

New companies use Java

because it’snew and shiny

1997


because it’sooooooold

2005

New companies use the JVM

because WEBSCALE

2011

github


If you think you understandthe JVM, you are either:



a) Very smart



a) Very smartb) Very wrong


Some people think that github is a

Rails shopRuby shop.or even a


Some people think that github is a

Rails shopRuby shop.or even a

github is a

Unix shopand everything else is

just a detail.


libgit2So,


Good Heavens,just look at the time.

It’s NoSQL o’clock

NoSQL

NoSQL NoSQL

NoSQL

NoSQL NoSQL

NoSQLNoSQL


...do you even


...do you even mongo?


a b r i e f i n t r o d u c t i o nt o t h e G i t d a t a m o d e l


file.c src



file.c src


treesrc/

README.md

COPYING.md

treefile.c

file.h

blob

blob

blob

blob


commit

parent

tree Tmetadata


commitT

commitT

commitT

commitT

commitT

commitT

Behold,a graph.


Well that was easy.


master

Ohgod

killme


Li!le knowntorture methods:


warning:the rabbithole ispretty deep


Git doesn’t give a #!%$ about CAP


Number of hops on a complex query 1,000,000



Required hops for a successful query 1,000,000



Required hops for a successful query 1,000,000

Replica count to ensure 100% availability

a metric shittonSaturday, May 11, 13

We could fix it.


We could fix it.But we won’t.


libgit2


GitRPCSaturday, May 11, 13

GitRPCLess.


GitRPC

Rugged

libgit2

server

Ruby

Ruby

C


📸chimney(Redis)

frontend

fileserver

smoke

grit

ernie-corn

grit

GitRPC GitRPC


📸chimney(Redis)

frontend fileserver

GitRPC GitRPCserverclient


evolutionary(disappointing?)


We’ve had a lot of


We’ve had a lot of hard

engineering challenges


We tackled them by:


Using themost reliable

tools we know.


Challenging ourselves to build

the simplest thing.Not because it’s easy,but because it works.


Innovatingwhere it really ma!ers.


revolutionaryproduct

building a

revolutionarybackend.

not a


Having fun


My mom told me that Git doesn’t scale by Vicent Martí

Technology

Transcript of My mom told me that Git doesn’t scale by Vicent Martí