Dark Side of Git - We use this on my big data team

How GIT Works InternallySeongJae Park <[email protected]>

Nice To Meet You

SeongJae Park

[email protected]

Git

DVCS(Distributed Version Control System)http://git-scm.com/images/logos/downloads/Git-Logo-2Color.png

Git

DVCS(Distributed Version Control System)

Made-by Linus Torvalds For Linux

http://git-scm.com/images/logos/downloads/Git-Logo-2Color.png

http://cdn.memegenerator.net/instances/400x/37078331.jpg

Git

Many Projects Use Git Because It’s Awesome

http://blog.appliedis.com/wp-content/uploads/2013/11/android1.pnghttp://upload.wikimedia.org/wikipedia/en/4/40/Octocat,_a_Mascot_of_Github.jpghttp://upload.wikimedia.org/wikipedia/commons/thumb/3/35/Tux.svg/512px-Tux.svg.png


http://blog.appliedis.com/wp-content/uploads/2013/11/android1.png

http://blog.appliedis.com/wp-content/uploads/2013/11/android1.png

http://upload.wikimedia.org/wikipedia/en/4/40/Octocat,_a_Mascot_of_Github.jpg

http://upload.wikimedia.org/wikipedia/en/4/40/Octocat,_a_Mascot_of_Github.jpg

Git

Hard To LearnConfusing For CVCS UsersPush? Pull? Fetch? Rebase? HEAD???

http://www.quickmeme.com/img/fd/fd09e17b3393b2ea1cd7e52af1ad7c77f3c2d7a83e9f47d4b90ba3af52dde329.jpg


Git: The Information Manager From Hell

http://www.youblob.com/sites/default/files/styles/large/public/field/image/frontlego1.png?itok=XA5CXt84


$ git log e83c516

commit e83c5163316f89bfbde7d9ab23ca2e25604af290

Author: Linus Torvalds <[email protected]>

Date: Thu Apr 7 15:13:13 2005 -0700

Initial revision of "git", the information manager from hell



That’s Why So Confusing And Hard To Learn

$ git log e83c516

commit e83c5163316f89bfbde7d9ab23ca2e25604af290

Author: Linus Torvalds <[email protected]>

Date: Thu Apr 7 15:13:13 2005 -0700

Initial revision of "git", the information manager from hell


This Time, We Will...

See How Git Works From The Scratch

https://lh4.googleusercontent.com/gBpfuABUjSNi2RagtJrGi8TW-pmtgak_0qtGOGubihvKH-5-umreO9CwJgjX2kaA9E7RkLwtEwiDnoMtOgm4iMJ0IWhvXlzlKL1kNVUYWuNa-gLRtRoyNjkVYg



Just For Fun...Or To Be Friend Of Git




Just For Fun...Or To Be Friend Of Git

Forget About TheComplicated CommandsThis Time


In Short,

Git Is A Content-Addressable Storage System

http://www.juliagiff.com/wp-content/uploads/2014/03/tldr_trollcat.jpg

In Short,

Git Is A Content-Addressable Storage System

Blob, Tree, Commit, Reference. That’s It =3


Plumbers: Unsung Heroes Behind

● Git Looks Graceful Owing To Plumbing Commands Consisting Them

http://cfile4.uf.tistory.com/image/182FF7244CFDDFB33CC999http://cfile29.uf.tistory.com/image/18574F224CFDD89B163073

http://cfile4.uf.tistory.com/image/182FF7244CFDDFB33CC999


Plumbers: Unsung Heroes Behind

● Git Looks Graceful Owing To Plumbing Commands Consisting Them○ The Wounded Foots Are What We Interested In

http://cfile4.uf.tistory.com/image/182FF7244CFDDFB33CC999http://cfile29.uf.tistory.com/image/18574F224CFDD89B163073



Again, From The ScratchVCS? Why? How?

Why VCS?

Usual Life Of File

FileA ver 0 FileB ver 0

Why VCS?

Usual Life Of File

FileA ver 0 FileB ver 1FileB ver 0

Why VCS?

Usual Life Of File

FileA ver 0 FileB ver 1

Why VCS?

Usual Life Of File

FileB ver 1 FileA ver 1FileA ver 0

Why VCS?

Usual Life Of File

FileB ver 1 FileA ver 1

Why VCS?

Usual Life Of File

FileB ver 2FileA ver 1FileB ver 1

Why VCS?

Usual Life Of File

FileB ver 2FileA ver 1

We Need Version Control System

VCS Would...Record Every Changes Safely, Efficiently


VCS Would...Record Every Changes Safely, EfficientlyAble To Check Out Any Version


VCS Would...Record Every Changes Safely, EfficientlyAble To Check Out Any VersionEasy To Read History

Brute-force IdeaVersion Control Using File System

Brute-force Idea

Rename / Backup Every Files Whenever Change Made

Brute-force Idea


$ ls

foo.c

Brute-force Idea


$ ls

foo.c

foo_20140111.c

Brute-force Idea


$ ls

foo.c

foo_20140111.c

foo_final.c

Brute-force Idea


$ ls

foo.c

foo_20140111.c

foo_final.c

foo_realfinal.c

foo_planb.c

foo_finalfinal.c

Brute-force Idea + History Isolation

Keep Working / History Directory Seperately.

Brute-force Idea + History Isolation

Keep Working / History Directory Seperately.Better, But...

$ find . -type f

./working/foo.c

./history/foo_20140111.c

./history/foo_final.c

./history/foo_realfinal.c

./history/foo_planb.c

./history/foo_finalfinal.c

TODOs From Version Control Using FS

Use Storage Space-Efficiently


Use Storage Space-EfficientlyEasy History Searching

Mission #1:Store History Space-Efficiently

Basic Idea: Avoid Duplicated Objects


Content-Addressable Storage System


Content-Addressable Storage SystemKey: SHA-1 Hash Of Object’s Content

Value: Compressed Content


Content-Addressable Storage SystemKey: SHA-1 Hash Of Object’s Content

Value: Compressed Content

Same Content Never Saved Twice

Save / Load ‘homer’

$ mkdir simpsons; cd simpsons; git init

Initialized empty Git repository in simpsons/.git/

$ echo ‘homer’ | git hash-object -w --stdin

de7e45490c9a4a3b5d5fae106faa4235ec669e02

$






$ find .git/objects/ -type f.git/objects/de/7e45490c9a4a3b5d5fae106faa4235ec669e02

$






$ find .git/objects/ -type f.git/objects/de/7e45490c9a4a3b5d5fae106faa4235ec669e02

$ git cat-file -p de7e4

keep it simple, stupid

$ git cat-file -t de7e4

blob

What `hash-object -w` did

hash_object_w(‘homer\n’)



# Save compressed header + content at sha1 path

def hash_object_w(content):

header = ‘blob %d\0’ % len(content)

store = header + content

sha1 = sha.new(store).hexdigest()








dir = ‘.git/objects/’ + sha1[0:2] + ‘/’

filename = sha1[2:]








dir = ‘.git/objects/’ + sha1[0:2] + ‘/’

filename = sha1[2:]

open(dir + filename, ‘w’).write(

zlib.compress(store))

Version Control Using Hash Value

$ echo “bart” > son

$ git hash-object -w son

e00ddae83bdab443f4267426623aa34636c935f2

$




e00ddae83bdab443f4267426623aa34636c935f2

$ echo “hugo” > son


8e1e2f09585e021c9727585af72e10871d7be7ce

$




e00ddae83bdab443f4267426623aa34636c935f2

$ echo “hugo” > son


8e1e2f09585e021c9727585af72e10871d7be7ce

$

# Need former version, “bart”

$ git cat-file -p e00dd > son

$ cat son

bart


Use Storage Space-EfficientlyEasy History Searching


● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History

https://www.sciencenews.org/sites/default/files/main/articles/sad_opener.jpg



● TODO○ Support Directory Structure○ History Management○ Better Reference Than Hash Value


WAIT!

Q: What If Small Changes Inside A Big File?

WAIT!

Q: What If Small Changes Inside A Big File?

$ du -h bigfile.c188Kbigfile.c$ du -sh408K.$ echo ‘/* small change */’ >> bigfile.c$ git commit -as -m “small change, big difference”$ du -sh496K.$

WAIT!

Q: What If Small Change Inside A Big File?A: Git Pick up Diff-Only If Necessary

But, Don’t Forget To Keep It Small, Simple

$ du -sh496K.$ git gc

Counting objects: 6, done.

Delta compression using up to 4 threads.

Compressing objects: 100% (4/4), done.

Writing objects: 100% (6/6), done.

Total 6 (delta 1), reused 0 (delta 0)

$ du -sh

388K.

Mission #2:Store History Of Directories

tree Object

Point Other Objects(Using Hash) With Name

tree Object


tree

blob blob tree

blob

a113f2mommy b8934

son

c9240pets

d9b13cat

tree Object


“A Root tree Object Is A Snapshot”

tree

blob blob tree

blob

a113f2mommy b8934

son

c9240pets

d9b13cat

I’m a snapshot

tree object$ mkdir pets; echo ‘snowball’ > pets/cat

$ git update-index --add son pets/cat

$ git write-tree

15ee76ed3e744b6796950d07f26283d033ea3ea7$



$ git write-tree

15ee76ed3e744b6796950d07f26283d033ea3ea7$ git cat-file -p 15ee7

040000 tree 85ab72cf1946dc56392718a1aafb3c6f66c02072 pets

100644 blob 8e1e2f09585e021c9727585af72e10871d7be7ce son

$



$ git write-tree

15ee76ed3e744b6796950d07f26283d033ea3ea7$ git cat-file -p 15ee7

040000 tree 85ab72cf1946dc56392718a1aafb3c6f66c02072 pets

100644 blob 8e1e2f09585e021c9727585af72e10871d7be7ce son

$ git cat-file -p 85ab7

100644 blob 6a1f952e1baedcb3db93a3ea5e3389e5a87941e9 cat

$ git cat-file -p 6a1f9

snowball

$

Internal Data Structure

tree

blob tree

8e1e2son

85ab7pets


tree

blob tree

blob

6a1f9cat

8e1e2son

85ab7pets

Version Control Using tree Object


$ git update-index --add son

$ git write-tree

661e6ad514a7f05c46c2931280cb78a339d34ee2

$




$ git write-tree

661e6ad514a7f05c46c2931280cb78a339d34ee2

$ git cat-file -p 661e6040000 tree 85ab72cf1946dc56392718a1aafb3c6f66c02072 pets

100644 blob e00ddae83bdab443f4267426623aa34636c935f2 son

$




$ git write-tree

661e6ad514a7f05c46c2931280cb78a339d34ee2

$ git cat-file -p 661e6040000 tree 85ab72cf1946dc56392718a1aafb3c6f66c02072 pets

100644 blob e00ddae83bdab443f4267426623aa34636c935f2 son

$ git cat-file -p e00dd

bart

$


tree

blob tree

blob

8e1e2son

85ab7pets

6a1f9cat


tree

blob tree

blob

tree

blob

e00ddson85ab7

pets

8e1e2son

85ab7pets

6a1f9cat



● TODO○ Support Directory Structure○ History Management○ Better Reference Than Hash Value



● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure

● TODO○ History Management○ Better Reference Than Hash Value


Mission #3:Commit Message

commit Object

Describe Who / When / Why The Change Made

http://modthink.com/wp-content/uploads/2013/05/WhoWhatWhenWhereWHY.jpg

commit Object

Describe Who / When / Why The Change Made

Point A tree Object With Information Above

http://modthink.com/wp-content/uploads/2013/05/WhoWhatWhenWhereWHY.jpg

commit Object

$ echo '1st commit' | git commit-tree 661e6

0ca7304ad6f5a40f8a26ba05b10b514ff2d8d8a0

$

commit Object



$

$ git cat-file -p d075ctree 661e6ad514a7f05c46c2931280cb78a339d34ee2author SeongJae Park <s**@gmail.com> 1410527921 +0900

committer SeongJae Park <s**@gmail.com> 1410527921 +0900

1st commit

$

commit Object



$

$ git cat-file -p d075ctree 661e6ad514a7f05c46c2931280cb78a339d34ee2author SeongJae Park <s**@gmail.com> 1410527921 +0900


1st commit

$

Who WhenWhy

Version Control Using commit Object

$ echo '2nd commit' | git commit-tree 15ee7 -p 0ca73

003b5e66caa89a6228c7b4d91e0475e56bf1bdf6

$

$ git cat-file -p 003b5

tree 15ee76ed3e744b6796950d07f26283d033ea3ea7

parent 0ca7304ad6f5a40f8a26ba05b10b514ff2d8d8a0author SeongJae Park <s**@gmail.com> 1410528231 +0900


2nd commit

$


That’s Why People Says, “A Commit is a snapshot”

tree

blob tree

blob

tree

blob

commit commit

tree

parent

tree

85ab7pets

8e1e2son

85ab7pets

6a1f9cat

e00ddson


● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure

● TODO○ History Management○ Better Reference Than Hash Value



● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure○ Manage History Well

● TODO○ Better Reference Than Hash Value


Mission #4:Human Readable Name

Git References

File With Human-Readable Name

Git References


Storing SHA-1 Value Of commit Object

Git References


Storing SHA-1 Value Of commit Object

Resides In .git/refs/

Git References Using echo

$ echo "0ca7304ad6f5a40f8a26ba05b10b514ff2d8d8a0" > .git/refs/heads/first

$



$

$ git log --pretty=oneline first

0ca7304ad6f5a40f8a26ba05b10b514ff2d8d8a0 1st commit

$



$

$ git log --pretty=oneline first


$

$ find .git/refs/heads -type f

.git/refs/heads/first

.git/refs/heads/master

$

Git References Using update-ref

$ git update-ref refs/heads/master 003b5

$ git log --pretty=oneline master003b5e66caa89a6228c7b4d91e0475e56bf1bdf6 2nd commit


$





$




$





$




$

$ cat .git/refs/heads/master

003b5e66caa89a6228c7b4d91e0475e56bf1bdf6


tree

blob tree

blob

tree

blob

commit commit

tree

parent

tree

85ab7pets

8e1e2son

85ab7pets

e00ddson

6a1f9cat


tree

blob tree

blob

tree

blob

commit commit

tree

parent

tree

refs/heads/master

refs/heads/first

85ab7pets

8e1e2son

85ab7pets

e00ddson

6a1f9cat


● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure○ Manage History Well

● TODO○ Better Reference Than Hash Value


Version Control Using Reference

● DONE○ Efficient Space Usage○ Safe Record / Checkout Of History○ Support Directory Structure○ Manage History Well○ Easy To Remember Specific Snapshot

● TODO○ ...cooperation?


FAQ #1How Git Make-up Working Directory?

How Git Knows Current Commit?

Answer: HEAD


Answer: HEAD

HEAD Points reference Using ref format(Not SHA-1)


Answer: HEAD

HEAD Points reference Using ref format(Not SHA-1)

$ cat .git/HEADref: refs/heads/master

HEAD$ cat .git/HEAD

ref: refs/heads/master

$

HEAD$ cat .git/HEAD


$ git branch

first

* master

$

HEAD$ cat .git/HEAD


$ git branch

first

* master

$

$ git symbolic-ref HEAD refs/heads/first

$ cat .git/HEAD

ref: refs/heads/first

$ git branch

* first

master


tree

blob tree

blob

tree

blob

commit commit

tree

parent

tree

refs/heads/master

refs/heads/first

85ab7pets

8e1e2son

85ab7pets

e00ddson

6a1f9cat


tree

blob tree

blob

tree

blob

commit commit

tree

parent

tree

refs/heads/master

refs/heads/first .git/HEAD

85ab7pets

8e1e2son

85ab7pets

e00ddson

6a1f9cat

FAQ #2Cloned. Now Fetch Or Pull ?

Fetch / Pull

Fetch Or Pull To Get Latest Code?

Fetch

● Just Fetch Remote Repository’s Objects And References To Local Git Internal Storage

Fetch


● If You Need The Changes On Your Working Directory,

Fetch


● If You Need The Changes On Your Working Directory,○ Manually Merge Them Using git-merge Or,○ Checkout

Fetch

Refspec Describes Source / Destination

$ cat .git/config | grep remote -A3

[remote "origin"]

url = git://10.0.0.1/git/simpsons.git

fetch = +refs/heads/*:refs/remotes/origin/*

Source Destination

Fetch: Beforeurl = git://10.0.0.1/git/simpsons.git


tree

blob tree

blob

a134fson

799cfpets

7cc07cat

tree

blob

65464son

799cfpets

commit commit

tree

parent

tree

refs/heads/master

.git/HEAD

git://10.0.0.1/git/simpsons.git

tree

blob tree

blob

a134fson

799cfpets

7cc07cat

commit

tree

refs/heads/master

.git/HEAD

file:///home/sjpark/simpsons

Fetch: Afterurl = git://10.0.0.1/git/simpsons.git


tree

blob tree

blob

a134fson

799cfpets

7cc07cat

tree

blob

65464son

799cfpets

commit commit

tree

parent

tree

refs/heads/master

.git/HEAD

git://10.0.0.1/git/simpsons.git

tree

blob tree

blob

a134fson

799cfpets

7cc07cat

tree

blob

65464son

799cfpets

commit commit

tree

parent

tree

refs/remotes/

origin/master

refs/heads/master

.git/HEAD

file:///home/sjpark/simpsons

git merge origin/master

tree

blob tree

blob

a134fson

799cfpets

7cc07cat

tree

blob

65464son

799cfpets

commit commit

tree

parent

tree

refs/remotes/

origin/master

refs/heads/

first

.git/HEAD

tree

blob tree

blob

a134fson

799cfpets

7cc07cat

tree

blob

65464son

799cfpets

commit commit

tree

parent

tree

refs/remotes/

origin/master

refs/heads/

first

.git/HEAD

Pull

Pull Is Just An Abbrev Of Fetch && Merge

May Merge Conflict Occur…

Pull Is Sufficient For Simple Project

Wrap-up

In Short,

Git Is A Content-Addressable File System

Blob, Tree, Commit, Reference. That’s It =3


Thank you :)

http://jeancharpentier.files.wordpress.com/2012/02/capture-plein-c3a9cran-01022012-230955.jpg

Slide-share

http://www.slideshare.net/SeongJaePark1/deep-darkside-ofgit

Latest Version Of This Slide Would Be There





References

http://git-scm.com/bookhttp://www.youtube.com/watch?v=4XpnKHJAok8http://en.wikipedia.org/wiki/The_Simpsons

http://git-scm.com/book

http://git-scm.com/book

http://www.youtube.com/watch?v=4XpnKHJAok8



http://en.wikipedia.org/wiki/The_Simpsons

http://en.wikipedia.org/wiki/The_Simpsons

This work by SeongJae Park is licensed under the Creative Commons Attribution-ShareAlike 3.0 Unported

License. To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/3.0/.

http://creativecommons.org/licenses/by-sa/3.0/



Dark Side of Git - We use this on my big data team

Technology

Transcript of Dark Side of Git - We use this on my big data team