Advanced Git and Gitlab - NERSC · PDF fileAdvanced Git and Gitlab ... • Suppose you have...

Tony Wildish!Dan Udwary

Advanced Git and Gitlab

-1-

May30,2017

Advanced Gitlab

•  Prerequisites•  BranchingandTagging•  Buildingmul<plecontainers•  Pushingimagestomul1plerepositories•  Usingmetadataincontainers•  DeployingrunnersonNERSChosts•  Bestprac<ces&recommenda<ons

•  =>Getthecodeforthistutorial:–  Forkthetutorialrepository,thencloneyourforktoyourlaptop–  h@ps://gitlab.com/TonyWildish/gitlab-advanced/

-2-

Prerequisites

•  Familiaritywithgit,docker,gitlab–  Git,version2.11orhigher–  Docker,version1.12.3orhigher–  Anaccountongitlab.com

•  Earliertutorials:–  h@ps://www.nersc.gov/assets/Uploads/Git+Docker-Tutorial-Dec01-2016.pdf

•  Doexercises4and5–  h@ps://www.nersc.gov/assets/Uploads/2017-02-06-Gitlab-CI.pdf

•  Dothefirstexercise

-3-

h@ps://gitlab.com/TonyWildish/gitlab-advanced/

Bonus gitlab tip: Notification emails

-4-

GotoyouraccountseXngs,No1fica1ons,Custom

Bonus gitlab tip: Notification emails

-5-

Chooseany/allfields,cansetper-projecttoo

Branching and tagging

•  Branches–  Allowparalleldevelopmentinasinglerepository–  Createbranchesasneeded,deletewhenobsolete–  Canmergebranchesifyoulike,orkeepforever

•  Bugfixbranches:merge,deletethebranch•  Featurebranches:keepforever.

–  Canmergeback&forthtocontroldivergence

-6-

Master

Feature1

Bugfix1

“ProGit”,bySco@Chacon,Chapter3


•  Tags–  Sta1clabel,iden1fiesapar1cularcommit–  Easilyrecoverpar1cularversionatany1meinfuture–  Oncepushed,tagsshouldn’tbedeletedormoved!

-7-

Master

Feature1

Bugfix1

Tag1 Tag2

Tag3 Tag4


•  Tagsandbranchesingitlab–  Canbeusedtoiden1fybuildproducts,labelimagesetc

•  Ifthere’satag,usethat•  Ifnot,usethebranchname•  ‘master’branch->‘latest’dockerversion(byconven1on)

–  Let’sdoexercise01!

-8-


Working with forked repositories

•  Howdoyoukeepaforkedrepositoryuptodate?–  Addtheoriginalsourceasanother‘remote’repository

-9-

>[email protected]:TWildish/jgi-lapinpy.gitCloninginto'jgi-lapinpy'...[…]

>cdjgi-lapinpy/>[email protected]:berkeleylab/jgi-lapinpy.git>[email protected]:TWildish/jgi-lapinpy.git(fetch)[email protected]:TWildish/jgi-lapinpy.git(push)[email protected]:berkeleylab/jgi-lapinpy.git(fetch)[email protected]:berkeleylab/jgi-lapinpy.git(push)

>gitpullupstreammasterFrombitbucket.org:berkeleylab/jgi-lapinpy*branchmaster->FETCH_HEADUpda1nga3f5e1e..03943c8Fast-forward

“ProGit”,bySco@Chacon,Sec1on2.5

Working with forked repositories

•  Howdoyoukeepaforkedrepouptodate?–  Addtheoriginalsourceasanother‘remote’repository

-10-

>[email protected]:TWildish/jgi-lapinpy.gitCloninginto'jgi-lapinpy'...[…]

>cdjgi-lapinpy/>[email protected]:berkeleylab/jgi-lapinpy.git>[email protected]:TWildish/jgi-lapinpy.git(fetch)[email protected]:TWildish/jgi-lapinpy.git(push)[email protected]:berkeleylab/jgi-lapinpy.git(fetch)[email protected]:berkeleylab/jgi-lapinpy.git(push)

>gitpullupstreammasterFrombitbucket.org:berkeleylab/jgi-lapinpy*branchmaster->FETCH_HEADUpda1nga3f5e1e..03943c8Fast-forward

“ProGit”,bySco@Chacon,Sec1on2.5

Building multiple containers

•  Supposeyouhaveapar<cularpackagewith:–  Afewcoredependencies,verysmalltotal–  Severalop1onalextrasthataddhundredsofMB

•  Howdoyoubuildanop<malcontainer?–  Includeeverything->baggagethatnotallusersneed–  Leavestuffout->don’tsa1sfyallusers

•  Solu<on:–  Buildtwocontainers(ormore)inthesamerepository

-11-


Building multiple containers

•  GitlabsupportsbuildingDockerimageswithnamesotherthantherepositoryname–  DefaultDockernamestructure

•  $REGISTRY_USER/$APPLICATION:$RELEASE_TAG–  Extendedsyntax:

•  $REGISTRY_USER/$APPLICATION/real-name:$RELEASE_TAG

–  Useextendedsyntaxrepeatedlyin.gitlab-ci.yml,withdifferent‘real-name’s

–  “myapp-lite”&“myapp”,or“myapp”&“myapp-full”

–  Seeexercise02!

-12-


Pushing images to multiple repositories

-13-

GITLABandSHIFTERvariablespointto

differentregistryhosts



-14-

BuildandpushtoGITLAB

Re-tag,pushtoSHIFTER



•  Caveat:Security!–  Gitlabhandsyoualogin-tokenforeverybuild–  Forshiser,onceyou’reinsidethefirewall,there’snoauthen1ca1onneeded,sonotoken

–  Anywhereelse,youprobablyneedatokenorpassword,butwheredoyoustoreit?•  Can’tbeintherepository,istoovisible•  Hastobeintherunnerrun1meenvironmentsomehow•  CandothisinSPIN,thoughnotverysecurelyatthemoment•  Candoitonyourlaptops•  Wanttodoitelsewhere?comeforachat

–  Exercise03,inyourown1meJ

-15-


Using metadata in containers

•  Passinforma<onfromthebuildenvironment–  Totheimage,ortotheuseratrun1me

•  Telltheuseranythingtheymightwanttoknow:– Whatrun1meenvironmentthesoswareneeds– Whatleveloftes1ng,cer1fica1onhasbeenperformed–  Pointerstodocumenta1on,sourcecode,maintainers…–  Run1medetails:

•  wherethecontainerlooksforinput•  whereitexpectstobeabletoputoutput…

-16-

h@p://docs.master.dockerproject.org/v1.5/userguide/labels-custom-metadata/h@ps://speakerdeck.com/garethr/shipping-manifests-bill-of-lading-and-docker-metadata-and-container

ThanksMichael,Alex


-17-

Developmentenvironment

Buildcontext(dockerdaemon)(ARGXYZ=$XYZ)

Dockerimage(LABELXYZ=$XYZ)

Run1meenvironment(container)(ENVXYZ=$XYZ)

dockerinspect…|grepXYZ

dockerrun…echo$XYZ

dockerbuild…--build-argsXYZ=123

Howmetadatagoesfromthebuildenvironmenttotheimage,andtotherunningcontainer

SeeDockerfile.metadataintherepo



•  Howcanweusemetadata?–  E.g.definingaproperontology–  Automa1ngpipelines,tes1ng,discovery…

•  Workinggroup(?)toinves<gatethis–  Probablylaterintheyearaserthemigra1on–  Volunteers/sugges1onsgratefullyaccepted!

–  Exercise04!

-18-


Deploying runners on NERSC hosts

•  ArunneratNERSCwithwrite-accessto$HOMEetc?•  Youcandothis,butthereareseriousrisksinvolved!

–  Don’tsharetherunnerregistra1ontokenwithanyone•  ~=givingthemyourNERSCpassword

–  Don’tgiveotherusersmaster-levelaccesstoyourrepository–  Consideralterna1ves:

•  UseaDockerimage,withyourcustombuildenvironment,onSPIN•  UseaVMsomewhere…

–  Talktoaconsultantbeforea@emp1ngthis!

–  Someoftheserisksaregitlab-specific–  Someareinherentinrunninganyinternet-enabledservices

-19-


Deploying runners on NERSC hosts

•  Basicrecipe–  Downloadthebinaryforagitlabrunner–  Registerit,giveitahost-specificconfigfile–  Giveitspecifictagswhenregistering,toiden1fyit–  Usethosetagsinyour.gitlab-ci.ymlfile–  Yourpipelinecanroamovertheen1refilesystemifyouwant,butit’suptoyouthentoensurethedirectoriesyouuseareclean

–  Seeexercise05fordetails–wewon’tdothistoday!

-20-


Other gitlab features •  API,programmableinterfacetoGitlab

–  h@ps://docs.gitlab.com/ee/api/•  SeeJGI/gitlab-cli-toolsrepoforsomebasictools,contribu1onswelcome!

•  Buildhooks–  Triggerac1onsonexternalservicesotherthangitlab

•  Similarcapabili1esongithub,bitbucket–  Triggerac1onsingitlabfromexternalservice

•  E.g.nightlybuild,regardlessofcommits

•  Mirroringrepositories–  Masterrepositoryinbitbucket/github?–  Canmirrortogitlab,automa1cally,transparently

•  Issue-tracking,wiki…–  Othergoodiescomeforfreewithgitlab,aswithotherhos1ngservices

-21-

Best practices, recommendations

•  Git:–  Usethefork/pull-requestmodelinsteadofgran1ngpeopledirect-commitaccesstoyourrepository

–  Usebranchestoexperiment,tryoutbugfixesetc•  Mergelong-livedbranchesfrequentlytocontroldivergence

–  Usetagstoiden1fystableversions,releasesetc

–  Don’tdeleteormovetagsoncethey’repushedtothemasterrepository

-22-


Best practices, recommendations

•  Gitlab:–  Buildmul1pleDockerimagesifyouhavedifferentuse-casestoservefromthesamecode-base

–  Pushingtomul1pleregistriesletsusersaccessyourimagesfrommanyplaces,easily

–  Usemetadatainyourcontainers!•  HelpusestablishstandardsforJGIcontainermetadata

–  Controlaccesstoyourrepositories•  Don’tgiveouttherunner-registra1ontoken•  Avoidgivingothersadmin/developer-accesstotheproject•  ThinktwicebeforedeployingrunnersonNERSCresources

-23-


Finally…

•  You’reallexpertsnow,soupdateyourresumes!

–  “experiencebuildingandop1mizingDockerimagesforbioinforma1csosware”

–  “experienceconfiguringandusingcon1nuous-integra1onpla~orms,suchasgitlab,toautomatebuildinganddeployingsosware”

–  “in-depthunderstandingofbest-prac1cesforsoswaremanagement,suchasversioncontrolwithgitanduseofmetadatatodescribeDockerimages”

–  “understandingofgitworkflowmodelsforteams,includingtheuseofbranches,tags,anddeveloperaccess-control”

-24-


National Energy Research Scientific Computing Center

-25-

Advanced Git and Gitlab - NERSC · PDF fileAdvanced Git and Gitlab ... • Suppose you have...

Documents

Transcript of Advanced Git and Gitlab - NERSC · PDF fileAdvanced Git and Gitlab ... • Suppose you have...