computing Refresher (day 1) - Stanford Universitytym1/refresher/materials/computing_slides.pdf ·...
Transcript of computing Refresher (day 1) - Stanford Universitytym1/refresher/materials/computing_slides.pdf ·...
ComputingRefresher(Day1)TimMoonICMERefreshersSeptember19,2016
Introduction
AboutMe• Third-yearICMEPhD• Researchinterests:• High-performancecompuFng• Numericallinearalgebra
• Coursewebsite:hJps://stanford.edu/~tym1/refreshers• Personalwebsite:hJps://stanford.edu/~tym1• Email:[email protected]
Goals• AccessStanford’scompuFngresources• NavigatewiththeUnixcommandline• EdittextfileswithVim• UseMATLABasacalculator
AdditionalResources• hJps://uit.stanford.edu/service/sharedcompuFng/loggingin(Stanford’stutorialtologintoFarmshare)
• hJps://linuxjourney.com(verygoodonlinecourseonLinux)• hJps://danielmiessler.com/study/vim/(tutorialtobecomeaVimguru)
• hJp://tnerual.eriogerg.free.fr/vimqrc.pdf(Vimcheatsheet)• hJp://vim-adventures.com(silly,Legend-of-Zelda-styleVimtutorial)
• hJp://stanford.edu/class/cme192/(ICMEMATLABcourse)
WhatisUnix?
FarmShare• SharedcomputersopentoStanfordstudents• corn:generalpurpose• rye:GPUs• cardinal:long-term,low-intensityjobs• barley:high-intensityjobs(usingjobmanager)
• FilesstoredonAndrewFileSystem(AFS)• ManagefilesathJps://afs.stanford.edu• WebhosFngwithWWWdirectory
• Availableso`ware:• R,abinit,acml,allinea,ampl,amplgurobi,ansys,biopython,bowFe2,caffe,
celltool,cplex,cuda,cudasamples,cudnn,cufflinks,farmvnc,fiji,gams,gaussian,gurobi,hisat2,intel,java,lemon,lifelines,maple,mathemaFca,matlab,meep,meta,openfoam,parallel,pyroseJa,r,relion,roseJa,rstudio,sagemath,sas,seaborn,sentaurus,sicstus,spectra,sratoolkit,statamp,statase,staJransfer,terraform,tophat,torch,xmipp
SSH• “SecureShell”• Standardmethodtologintoaremotecomputer• GraphicaluserinterfacewithX11• XWindowSystem,version11
LoggingintoFarmShare• Linux• OpenTerminalandtype:
ssh -X <SUNetID>@corn.stanford.edu
• Mac• InstallXquartz(hJps://www.xquartz.org)• OpenTerminal(ApplicaFons/UFliFes/Terminal)andtype:
ssh -X <SUNetID>@corn.stanford.edu
• Windows• InstallXming(hJps://sourceforge.net/projects/xming/)• DownloadandopenPuTTY(hJp://www.puJy.org)
• Set“HostName(orIPAddress)”to“corn.stanford.edu”(inSession)• Select“EnableX11forwarding”(inConnecFon/SSH/X11)
• UseyourSUNetIDasyourusernameandfollowthestepsfortwo-factorlogin
CommandLineEssentials
PracticeFiles• Downloadfilesbytyping:
git clone https://github.com/timmoon10/icme_computing_refresher_2016
cd icme_computing_refresher_2016
GettingStarted
• AbortcommandsbypressingCtrl-c
Command Purposeecho <arg> Printargumentstodisplayexit Exitshell
Directories
• RelaFvepathvs.absolutepath• Pathshortcuts• .(currentdirectory)• ..(parentdirectory)• ~(homedirectory)
Command Purpose UsefulOp2onsls <dir> Listcontentsindirectory -l,-apwd Printcurrentdirectorycd <dir> Changedirectorymkdir<dir> Makedirectoryrmdir<dir> Removedirectory
ReadingFilesCommand Purpose Useful
Op2onstouch <file> Createemptyfilecat <file> Printfilehead <file> Printfirstfewlines -n <lines>tail <file> Printlastfewlines -n <lines>less <file> Readfilein“less”
• UseVimnavigaFoninless• lessisbeJerthanmore
ManipulatingFilesCommand Purpose UsefulOp2onscp <old> <new> Copyfile -r,-imv <old> <new> Movefile -i
rm <file> Removefile -r,-f,-i
• Becareful,Unixdoesnotforgive• Wildcards• *(anynumberofcharacters)• ?(exactlyonecharacter)
Help
• Manycommandscanalsoprintahelppage• Put“-h”or“--help”asacommand-lineargument
• Googleisyourfriend
Command Purposeman <command> Manualpage
AdvancedUnixTopics
UsefulCommandsCommand Purpose UsefulOp2onshistory Printpreviouscommandswc <file> Countwords -w,-l,-m,-csort <file> Sorttextlines -r,-n
grep
• Youcanusearegularexpression(regex)• UsefulforsearchingthroughmulFplefiles
Command Purpose UsefulOp2onsgrep <string> <file> Searchforstringinfile -i,-r
RedirectionandPiping• RedirecFon(>)putsstandardoutputinfile• Use >>toappendtofile• ls > file_list.txt
• Pipe(|)givesoutputtoanotherprogram• Unixphilosophyusesmodularprograms• ls | echo
ProjectTrees• Commonstructure:• README:helpfile• Makefile:Makefileformake• src:sourcefiledirectory• doc:documentaFondirectory• include:headerfiledirectory• ext:directoryforexternalprojects• bin:directoryforcompiledexecutables• lib:directoryforcompiledlibraries• build:directoryforintermediatebuildfiles
Make• ManyprogramsrequirecompilaFonfromsourcecode• MayalsorequireconfiguraFon,installingdependencies,andinstallaFon
• Makeisacommontoolforbuildingprojects• LooksatMakefileincurrentdirectory• ManyprojectsuseCMake,whichgeneratesMakefiles
• Usage:• Changetoprojectdirectoryandtype“make”• “make clean”iso`enusedtoresetdirectory• MayrequireconfiguraFonwith“./configure”• MayrequireinstallaFonwith“make install”
VersionControl• CollaboraFvecodingcanbeapain• Simultaneouschanges• Findingwhereabugwasadded• Seeingwhowrotewhat
• Commonprograms:Git,Subversion,Perforce,Mercurial• HighlyrecommendmakingaGitHubaccount
GitCommand Purposegit clone <repo> Downloadrepositorygit pull Updatefromremotegit status Getrepositorystatusgit add <file> Preparetoaddfilegit rm <file> Removefilegit commit –m “<message>” Commitlocallygit push Submitchangestoremote
SimpleGitWorkQlow
Makechangestocode
git pull
git add <file>
git commit -m “<message>”
git push
Tarballs
• “Tapearchive”• Compressionwithgzip• .tar.gzor.tgzfileextension• TrydownloadinganddecompressinghJps://stanford.edu/~tym1/refresher/materials/example_tarball.tgz
Command Purposewget <url> Downloadfiletar –czvf <tarfile> <files> Compresstotarballtar -xzvf <tarfile> Decompresstarball
ComputingRefresher(Day2)TimMoonICMERefreshersSeptember20,2016
LoggingintoFarmShare• Linux• OpenTerminalandtype:
ssh -X <SUNetID>@corn.stanford.edu
• Mac• InstallXquartz(hJps://www.xquartz.org)• OpenTerminal(ApplicaFons/UFliFes/Terminal)andtype:
ssh -X <SUNetID>@corn.stanford.edu
• Windows• InstallXming(hJps://sourceforge.net/projects/xming/)• DownloadandopenPuTTY(hJp://www.puJy.org)
• Set“HostName(orIPAddress)”to“corn.stanford.edu”(inSession)• Select“EnableX11forwarding”(inConnecFon/SSH/X11)
• UseyourSUNetIDasyourusernameandfollowthestepsfortwo-factorlogin
PracticeFiles• TryupdaFngtherepo:• cd icme_computing_refresher_2016• git pull
• Ifyougetanerror,deletetherepoanddownloadagain:• cd ~• rm –rf icme_computing_refresher_2016• git clone https://github.com/timmoon10/icme_computing_refresher_2016
Vim
WhatisVim?• “ViIMproved”• viisasimilar,oldertexteditor
• Mostpopularcommand-linetexteditor• FoundonalmostallUnixsystems
• ModalediFng• Powerful,butsteeplearningcurve
• Veryusefultoprintoutacheatsheet:• hJp://tnerual.eriogerg.free.fr/vimqrc.pdf
• StartVimbytyping:vim <file>
OtherTextEditors• Emacs• SublimeText• nano• Notepad++
Navigation
• DefaultVimmodeisNormalMode• Navigatewitharrowkeysorhjklkeys
Command PurposeEsc ReturntoNormalModeh Le`j Downk Upl Right
InsertMode
• Thisiswhereyouactuallytype• ReturntonormalmodewithEsc
Command Purposei EnterInsertModebeforecursor(“insert”)a EnterInsertModea`ercursor(“append”)
SavingandExitingCommand Purpose:w <file> Savetofile(“write”):e <file> Openanotherfile(“edit”):q Quit(requiresnochanges):wq Saveandquit:q! Quitwithoutsaving
SearchingCommand Purpose/<string> Forwardsearchforstring?<string> Backwardsearchforstringn NextsearchresultN Previoussearchresult
VisualModeCommand Purposev EnterVisualMode(character-based)V EnterVisualMode(line-based)y CopyselecFon(“yank”)x Cut/deleteselecFonp Pastea`ercursorP Pastebeforecursor
• VisualModeisusedtoselectregionsoftext• NavigationissameasNormalMode
MoreUsefulCommandsCommand Purpose0 Gotobeginningofline$ Gotoendoflinegg GotobeginningoffileG Gotoendoffile:<number> Gotospecificlinedd Deletelineu Undo. Repeatlastcommandorredoundo
ScreenSplittingCommand PurposeCtrl-w s SplitwindowCtrl-w n CreateemptywindowCtrl-w j MovetonextwindowCtrl-w k MovetopreviouswindowCtrl-w o Closeallbutcurrentwindow
• EachwindowislikeanindependentsessionofVim
MATLAB
WhatisMATLAB?• “MatrixLaboratory”• Proprietaryso`warebyMathWorks• VerygoodforexperimentaFon• High-levelprogramminglanguage• FastmatrixoperaFons• Feature-rich
• StarFngMATLABonFarmShare:module load matlabmatlab
• Towriteascript,putmulFplecommandsina.mfile• Torunmy_script.m,typemy_scriptinMATLAB
Alternatives• Octave• Python(withNumPyandSciPy)• Julia• R
Variables• a = 4• b = -5.3• third_variable = a * b• d = sin(a) + 1.1• a > b• Comparisons:<,<=,>,>=,==,~=
• Suppressoutputwithsemicolon:• f = 3.1 – 4.3 * i;
• Specialvalues:pi,i,Inf,NaN• FuncFons:sin,cos,exp,sqrt,round,floor,ceil
Matrices• A = [1 2 3; 4 5 6]• MathemaFcalresult:
• B = exp(A)• A + B• [1; 2; 3]• [4 5 6]
A = 1 2 34 5 6
⎡
⎣⎢
⎤
⎦⎥
MatrixOperations• A = [1 2; 3 4]• B = [5 6; 7 8]• [A; B]• [A B]• A * B(matrixmulFplicaFon)• A \ B(matrixinverse)• Entry-wiseoperaFons:.*or./or.\or.^
UsefulFunctions• zeros(2,3)• ones(2,3)• eye(2,3)• rand(2,3) (uniformdistribuFon)• randn(2,3) (normaldistribuFon)• 1:4• 5:-2:1• linspace(2,4,5)
Indexing• A = [1 2 3 4; 5 6 7 8; 9 10 11 12]• A(2,3)• A(1:2,2:4)• A(1:2,2)• A(2,2:end)• A(1:2,:)• A(:)• A(3:5)
Plotting• x = linspace(1,10,50);• y = sin(x) + 1.1;• z = sin(2*x) + 1.1;• plot(x,y)• plot(x,z)• hold on• close
Help• help<function>• doc <function>• Google
Scripting• JustputmulFplecommandsinafile• .mfileextension
• Runbytypingfilename,excluding.m• OpenMATLAB’sfileeditorwith“edit <funcFon>”• Commentwith%• Controlstatements:if,else,elseif,while,for• Closeblockwithend
Exercise• Let’ssolvetheheatequaFon:
• ForanapproximatesoluFonatthenthFmestep:
• IniFalizeuasarandomcolumnvectorandusec=0.25• ComputeuforseveralFmesteps.Plota`ereveryFmestep(uselinspace(0,1,5)forx-dimension).Whatishappening?
∂u∂t−∇2u = 0
un ≈ un−1 + cAun−1
A =
1 −1−1 2 −1
−1 2 −1−1 2 −1
−1 1
⎡
⎣
⎢⎢⎢⎢⎢⎢
⎤
⎦
⎥⎥⎥⎥⎥⎥
ComputingPotpourri
VirtualPrivateNetwork• FarmSharerequiresuserstobeinStanfordnetwork• VPNclientscreatesecureconnecFontoStanfordnetwork• Veryusefulwhenoff-campus
• SeeStanford’sVPNtutorial• hJps://uit.stanford.edu/service/vpn
LaTeX• LaTeXisadocumentpreparaFonsystem• LiJleeffortfornice-lookingdocuments
• Verygoodformathandtechnicaldocuments• MakeslideshowswithBeamer• LaTeXmarkuplanguage• .texfileextension
• ForaWYSIWYGeditor,tryLyX(hJps://www.lyx.org)
Jupyter• VeryniceenvironmentforJulia,Python,andR• Showsupinwebbrowser• Workbooksareconvenientwaytosharecodesnippets• SeeinstallaFoninstrucFons:• hJp://jupyter.readthedocs.io/en/latest/install.html
LinksCommand Purposeln -s <target><link> Createsymboliclinktotarget
• Symboliclinkpointstoatargetpath• Leavingoutthe“-s”opFoncreatesahardlink• Besttoavoidsincehardlinksarenotassafe
FilePermissions• “ls -l”showsfilepermissions• Filetype• Ownerpermissions(read,write,execute)• Grouppermissions(read,write,execute)• Otheruserpermissions(read,write,execute)
• Changefilepermissionswithchmod <number> <file>
• Numberhasthreedigits:owner,group,andotherusers• Eachdigit(inbinary)givesread,write,executepermissions• Commonnumbersare500(fortext)and700(forexecutables)
BashScripting• “BourneAgainShell”• Writescriptsbypusngcommandsasifincommand-line• Commentwith#• Setvariableswith=• MY_VARIABLE=<value>
• Referencevariableswith$• echo $MY_VARIABLE
• Runwithsourceorbysesngfilepermissionstoexecute• Commontoput“#!/bin/bash”infirstline
SuperUser
• Defaultstosuperuser• Alsoknownasroot,administrator,supervisor
• Usermustbeinsudoerlisttousesudo• Commonforinstallingpackagesandconfiguringsystem• Requiresmodifyingsystemfiles
Command Purposesu <users> Switchusersudo <command> Runcommandasanotheruser
StanfordComputingResources• FarmShare:compuFngclusterforstudents,faculty,staff• ICMEclusters• icme-sharedmem• icme-hadoop1• icme-mpi1• icme-gpu• VirtualdesktopsforICMEstudents• ContactBrianTempero([email protected])foraccess
• Sherlock:high-performancecompuFngcluster• Forresearch,requiressponsorshipfromfacultymember
• JobmanagementwithSLURM
ICMEComputingClasses• Fullcourses:• CME211-compuFngbasicswithPythonandC++• CME212-scienFficcompuFngwithC++• CME213-parallelcompuFng(mostlyGPUs)• CME214-Fortran• CME323-distributedcompuFngwithSpark• CME342-theoreFcalparallelcompuFng
• Shortcourses:• CME192-MATLAB• CME193-Python• CME195-R• CME292-AdvancedMATLAB