Using Cori - National Energy Research Scientific … · Using Cori. How to Compile & MCDRAM ......

Steve Leak, and Zhengji Zhao!NESAP Hack-a-thon!November 29, 2016, Berkeley CA

Using Cori

How to Compile & MCDRAM

-2-

SteveLeak

Building for Cori KNL nodes

• What’sdifferent?• Howtocompile– ..tousethenewwidevectorinstruc6ons

• Whattolink• MakinguseofMCDRAM– HighBandwidthMemory

Building for Cori KNL nodes

Don’tPanic(much)

KNL can run Haswell executables

But ...

HaswellExecutablescan’tfullyuseKNLhardware AVX2

(haswell)Opera6onon4DPwords

AVX-512(knl)Hardwarecancompute8DPwordsperinstruc6on

And ...

KNLreliesmoreonvectorizaKonforperformance

GFLOPS

ArithmeKcintensity

And ...

KNLmemoryhierarchyismorecomplicated

How to compile

Best:UsecompileropKonstobuildforKNLmodule swap craype-haswell craype-mic-knl

•  Theloadedcraype-*modulesetsthetargetthatthecompilerwrappers(cc,CC,Wn)buildfor–  Eg-mknl(GNUcompiler),

-hmic-knl(Craycompiler)•  craype-haswellisdefaulton

loginnodes•  craype-mic-knlisforKNLnodes

How to compile

Best:CompilerseXngstotargetKNLAlternate:CC -axMIC-AVX512,CORE-AVX2 <more-options> mycode.c++

•  OnlyvalidwhenusingIntelcompilers(cc,CCorWn)•  -ax<arch>addsan“alternateexecuKonpaths”opKmized

fordifferentarchitectures– Makes2(ormore)versionsofcodeinsameobjectfile

• NOTASGOODasthecraype-mic-knlmodule– (modulecausesversionsoflibrariesbuiltforthatarchitecturetobeused-egMKL)

How to compile

RecommendaKons:•  Forbestperformance,usethecraype-mic-knlmodulemodule swap craype-haswell craype-mic-knl CC -O3 -c myfile.c++

•  IfthesameexecutablemustrunonKNLandHaswellnodes,usecraype-haswellbutaddKNL-opKmizedexecuKonpathCC -axMIC-AVX512,CORE-AVX2 -O3 -c myfile.c++

What to link

UKlitylibraries• Notperformance-criKcal(bydefiniKon)– KNLcanrunXeonbinaries..canuseHaswell-targetedversions

•  I/Olibraries(HDF5,NetCDF,etc)shouldfitinthiscategorytoo– (forCray-providedlibraries,compilerwrapperwillusecraype-*toselectbestbuildanyway)

What to link

Performance-criKcallibraries• MKL:hasKNL-targetedopKmizaKons– Note:needtolinkwithwith-lmemkind(moresoon)

• PETsc,SLEPc,Caffe,MeKs,etc:– (soon)hasKNL-targetedbuilds

• Modulefileswillusecraype-{haswell,mic-knl}tofindappropriatelibrary• Keypoints:– SomeoneelsehasalreadypreparedlibrariesforKNL– Noneedtodo-it-yourself– Loadtherightcraype-module

What to link

• NERSCconvenKon:/usr/common/software/<name>/<version>/<arch>/[<PrgEnv>]

•  Eg:/usr/common/software/petsc/3.7.2/hsw/intel /usr/common/software/petsc/3.7.2/knl/intel

• KNLsubfoldermaybeasymlinktohsw– Librariescompiledwith-axMIC-AVX512,CORE-AVX2

•  ModulefilesshoulddotherightthingTM

– UsingCRAY_CPU_TARGET,setbycraype-{haswell,mic-knl}

Where to build

• Mostly:ontheloginnodes– KNLisdesignedforscalable,vectorizedworkloads– Compilingisneither!

•  WillprobablybemuchsloweronKNLnodethanXeonnode

• Cross-compiling– YouarecompilingforaXeonPhi(KNL)target,onaXeonhost•  Toolslikeautoconf(./configure)maytrytobuild-and-runsmallexecutablestotestavailabilityoflibraries,etc..whichmightnotwork

– CompileonKNLcomputenode?•  Slow(andcurrentlynotworking)

– craype-haswell+CFLAGS=-axMIC-AVX512,CORE-AVX2

Don’t Panic!

InSummary:• Buildonloginnodes(likeyoudonow)• Useprovidedlibraries(likeyouprobablydonow)

• Here’sthenewbit:•  module swap craype-haswell craype-mic-knl

– ForKNL-specificexecutables,or•  CC -axMIC-AVX512,CORE-AVX2 ...

– ForHaswell/KNLportability

What about MCDRAM?

• What’sdifferent?• Howtocompile– ..tousethenewwidevectorinstruc6ons

• Whattolink• MakinguseofMCDRAM– HighBandwidthMemory

MCDRAM in a nutshell

•  16GBon-chipmemory– cf96GBoff-chipDDR(Cori)

• Not(exactly)acache– LatencysimilartoDDR

• Butveryhighbandwidth– ~5xDDR

•  2waystouseit:– “Cache”mode:invisibletoOS,memorypagesarecachedinMCDRAM(cache-linegranularity)– “Flat”mode:appearstoOSasseparateNUMAnode,withnolocalCPUs.Accessiblevianumactl,libnuma(pagegranularity)

MCDRAM in a nutshell - cache mode

MCDRAM in a nutshell - flat mode

How to use MCDRAM

• OpKon1:Letthesystemfigureitout– Cachemode,nochangestocode,buildprocedureorrunprocedure– Mostofthebenefit,free,mostofthe6me

How to use MCDRAM

• OpKon2:Run-KmeseXngsonly– Flatmode,nochangestocodeorbuildprocedure– Doeswholejobfitwithin16GB/node?

•  srun<op6ons>numactl-m1./myexec.exe– Toobig?

•  srun<op6ons>numactl-p1./myexec.exe

How to use MCDRAM

• OpKon3:MakeyourapplicaKonNUMA-aware– Flatmode– UselibmemkindtoexplicitlyallocateselectedarraysinMCDRAM

memkind

jemalloc libnuma

APIforNUMAalloca6onpolicyinLinuxkernel

Mallocimplementa6onemphasizingfragmenta6onavoidanceandconcurrency

NUMA-awareextensibleheapmanager

#include <hbwmalloc.h> malloc(size) -> hbw_malloc(size)

Using libmemkind in code

• C/C++hbw_malloc()replacesmalloc()#include <hbwmalloc.h> // malloc(size) -> hbw_malloc(size)

•  Fortran!DIR$ MEMORY(bandwidth) a,b,c ! cray real, allocatable :: a(:,:), b(:,:), c(:) !DIR$ ATTRIBUTES FASTMEM :: a,b,c ! intel

• Caveat:onlyfordynamically-allocatedarrays– Notlocal(stack)variables– OrFortranpointers

Using libmemkind in code

• WhicharraystoputinMCDRAM?– Vtunememory-accessmeasurements:– amplxe-cl-collectmemory-access…

Building with libmemkind

•  module load memkind •  (ormodule load cray-memkind)

• Compilerwrapperswilladd

-lmemkind -ljemalloc -lnuma •  Fortrannote:NotallcompilerssupportFASTMEMdirecKve– CurrentlyIntelandmaybeCray

AutoHBW: Automatic memkind

• UsesarraysizetodeterminewhetheranarrayshouldbeallocatedtoMCDRAM• Nocodechangesnecessary!•  module load autohbw •  Linkwith-lautohbw

RunKmeenvironmentvariables:export AUTO_HBW_SIZE=4K # any allocation # >4KB will be placed in MCDRAM export AUTO_HBW_SIZE=4K:8K # allocations # between 4KB and 8KB will # be placed in MCDRAM

Don’t Panic!

InSummary:• Buildonloginnodes(likeyoudonow)• Useprovidedlibraries(likeyouprobablydonow)• Here’sthenewbit:•  module swap craype-haswell craype-mic-knl

– ForKNL-specificexecutables,or•  CC -axMIC-AVX512,CORE-AVX2 ...

– ForHaswell/KNLportabilityAnd:•  ThinkaboutMCDRAM– numactl,memkind,autohbm

A few final notes

•  Edisonexecutables(probably)won’tworkwithoutrecompile– ISA-compa6ble,but…– CorihasnewerOSversion,updatedlibraries– So:recompileforCori

• KNL-opKmizedMKLuseslibmemkind– Willneedtolinkwith-lmemkind -ljemalloc – Shouldbeinvisiblyintegratedinfutureversion

Running jobs on Cori KNL nodes

-30-

ZhengjiZhao

Agenda

•  What’snewonKNLnodes•  Process/thread/memoryaffinity•  Samplejobscripts•  Summary

-31-

KNL overview and legend Knights Landing Overview

Chip: 36 Tiles interconnected by 2D Mesh Tile: 2 Cores + 2 VPU/core + 1 MB L2 Memory: MCDRAM: 16 GB on-package; High BW DDR4: 6 channels @ 2400 up to 384GB IO: 36 lanes PCIe Gen3. 4 lanes of DMI for chipset Node: 1-Socket only Fabric: Omni-Path on-package (not shown) Vector Peak Perf: 3+TF DP and 6+TF SP Flops Scalar Perf: ~3x over Knights Corner Streams Triad (GB/s): MCDRAM : 400+; DDR: 90+

TILE

4

2 VPU

Core

2 VPU

Core

1MB L2

CHA

Package

Source Intel: All products, computer systems, dates and figures specified are preliminary based on current expectations, and are subject to change without notice. KNL data are preliminary based on current expectations and are subject to change without notice. 1Binary Compatible with Intel Xeon processors using Haswell Instruction Set (except TSX). 2Bandwidth numbers are based on STREAM-like memory access pattern when MCDRAM used as flat memory. Results have been estimated based on internal Intel analysis and are provided for informational purposes only. Any difference in system hardware or software design or configuration may affect actual performance. Omni-path not shown

EDC EDC PCIe Gen 3

EDC EDC

Tile

DDR MC DDR MC

EDC EDC misc EDC EDC

36 Tiles connected by

2D Mesh Interconnect

MCDRAM MCDRAM MCDRAM MCDRAM

3

DDR4

CHANNELS

3

DDR4

CHANNELS

MCDRAM MCDRAM MCDRAM MCDRAM

DMI

2 x161 x4

X4 DMI

-32-

CPU CPU

CPU CPU

CPU CPU

CPU CPU

Tile

Core Core

1Socket/Node68Cores(272CPUs)/Node36Tiles/Node(34ac6ve)2Cores/Tile;4CPUs/Core

1.4GB/CoreDDRmemory235MB/CoreMCDRAM

•  ACoriKNLnodehas68cores/272CPUsrunningat1.4GHz,96GBDDRmemory,16GBhigh(~5xDDR)bandwidthonpackagememory(MCDRAM)

•  Threeclustermodes,all-to-all,quadrant,sub-NUMAclustering,areavailableatbootKmetoconfiguretheKNLmeshinterconnect.

UseSlurm’sterminologyofcores,CPUs(hardwarethreads).

KNL overview – MCDRAM modes

10

Memory Modes

Hybrid Mode

DDR 4 or 8 GB MCDRAM

8 or 12GB MCDRAM

16GB MCDRAM

DDR

Flat Mode

Ph

ysic

al A

dd

ress

DDR 16GB

MCDRAM

Cache Mode

• SW-Transparent, Mem-side cache • Direct mapped. 64B lines. • Tags part of line • Covers whole DDR range

Three Modes. Selected at boot

Ph

ysic

al A

dd

ress

• MCDRAM as regular memory • SW-Managed • Same address space

• Part cache, Part memory • 25% or 50% cache • Benefits of both

-33-

96GB 96GB96GB

•  Nosourcecodechangesneeded

•  Missesareexpensive

•  Codechangesrequired•  ExposedasaNUMAnode•  Accessviamemkindlibrary,

joblaunchers,and/ornumactl

•  Combina6onofthecacheandflatmodes

MCDRAMcanbeconfiguredinthreedifferentmodesatbootKme-cache,flat,andhybridmodes

What’s new on KNL nodes (in comparison with Cori Haswell nodes from the perspective of running jobs)

1.   Alotmore(slower)coresonthenode2.   Muchreducedpercorememory3.   DynamicallyconfigurableNUMAandMCDRAM

modes…

-34-

A proper process/thread/memory affinity is the basis for optimal performance

•  Processaffinity(orCPUpinning):binda(MPI)processtoaCPUorarangeofCPUsonthenode,sothattheprocessexecuteswithinthedesignatedCPUsinsteadofdriWingaroundtootherCPUsonthenode.

•  Threadaffinity:finepineachthreadofaprocesstoaCPUorCPUswithintheCPUsthataredesignatedtotheprocess.–  Threadsliveintheprocessthatownsthem,sotheprocessandthreadaffinityarenotseparable.

•  Memoryaffinity:restrictprocessestoallocatememoriesfromthedesignatedNUMAnodesonlyratherthananyNUMAnodes.

-35-

The minimum goal of process/thread/memory affinity is to achieve best resource utilization and to avoid NUMA performance penalty

•  SpreadMPItasksandthreadsontothecoresandCPUsonthenodesasevenlyaspossiblesothatnocoresandCPUsareoversubscribedwhileothersstayidle.Thiscanensuretheresourcesavailableonthenode,suchascores,CPUs,NUMAnodes,memoryandnetworkbandwidths,etc.,canbebestuKlized.

•  AvoidaccessingremoteNUMAnodesasmuchaspossiblesotoavoidperformancepenalty.

•  IncontextofKNL,enableandcontroltheMCDRAMaccess.

-36-

Using srun’s --cpu_bind option and OpenMP environment variables to achieve desired process/thread affinity

•  Usesrun--cpu_bindtobindtaskstoCPUs–  Ozenneedstoworkwiththe–copKonofsruntoevenlyspreadMPI

tasksontheCPUsonthenodes–  Thesrun–c<n>(or--cpus-per-task=n)allocates(reserves)nnumber

ofCPUspertask(process)–  --cpu_bind=[{verbose,quiet},]type,type:cores,threads,map_cpu:<list

ofCPUs>,mask_cpu:<listofmasks>,none,…•  UseOpenMPenvs,OMP_PROC_BINDandOMP_PLACESto

finepineachthreadtoasubsetofCPUsallocatedtothehosttask–  Differentcompilersmayhavedifferentdefaultvaluesforthem.The

followingarerecommended,whichyieldamorecompa6blethreadaffinityamongIntel,GNUandCraycompilers:OMP_PROC_BIND=true#SpecifyingthreadsmaynotbemovedbetweenCPUsOMP_PLACES=threads#SpecifyingathreadshouldbeplacedinasingleCPU

–  UseOMP_DISPLAY_ENV=truetodisplaytheOpenMPenvironmentvariablesset(usefulwhencheckingthedefaultcompilerbehavior)

-37-

Using srun’s --mem_bind option and/or numactl to achieve desired memory affinity

•  Usesrun–mem_bindformemoryaffinity–  --mem_bind=[{verbose,quiet},]type:local,map_mem:<NUMAidlist>,mask_mem:<NUMAmasklist>,none,…

–  E.g.,--mem_bind=<MCDRAMNUMAid>whenalloca6onsfitintoMDCRAMinflatmode

•  UseNumactl–p<NUMAid>–  Srundoesnothavethisfunc6onalitycurrently(16.05.6),willbesupportedinSlurm17.02.

–  E.g.,numactl–p<MCDRAMNUMAid>./a.outsothatalloca6onsthatdon’tfitintoMCDRAMspillovertoDDRmemory

-38-

Default Slurm behavior with respect to process/thread/memory binding

•  BySlurmdefault,adecentCPUbindingissetonlywhentheMPItaskspernodexCPUspertask=thetotalnumberofCPUsallocatedpernode,e.g.,68x4=272

•  Otherwise,SlurmdoesnotdoanythingwithCPUbinding.Thesrun’s--cpu_bindand–copKonsmustbeusedexplicitlytoachieveopKmalprocess/threadaffinity.

•  NodefaultmemorybindingissetbySlurm.ProcessescanallocatememoryfromallNUMAnodes.The--mem_bind(ornumactl)shouldbeusedexplicitlytosetmemorybindings.

-39-

Default Slurm behavior with respect to process/thread/memory binding (continued)

•  ThedefaultdistribuKon,the–mopKonofsrun,isblock:cycliconCori.–  Thecyclicdistribu6onmethoddistributesallocatedCPUsforbindingtoa

giventaskconsecu6velyfromthesamesocket,andfromthenextconsecu6vesocketforthenexttask,inaround-robinfashionacrosssockets.

•  The–mblock:blockalsoworks.Youareencouragedtoexperimentwith–mblock:blockassomeapplicaKonsperformbeyerwiththeblockdistribuKon.–  Theblockdistribu6onmethoddistributesallocatedCPUsconsecu6vely

fromthesamesocketforbindingtotasks,beforeusingthenextconsecu6vesocket.

•  The–mopKonisrelevanttotheKNLnodeswhentheyareconfiguredinthesub-NUMAclustermodes,e.g.,SNC2,SNC4,etc.Slurmtreats“NUMAnodeswithCPUs”as“sockets”,althoughKNLisasinglesocketnode.

-40-

Available partitions and NUMA/MCDRAM modes on Cori KNL nodes (not finalized view yet)

•  SameparKKonsasHaswell–  #SBATCH–pregular–  #SBATCH–pdebug–  Typesinfo–sformoreinfoaboutpar66onsandnodes

•  Usingthe–Cknl,<NUMA>,<MCDRAM>opKonsofsbatchtorequestKNLnodeswithdesiredfeatures–  #SBATCH–Cknl,quad,flat

•  SupportscombinaKonofthefollowingNUMA/MCDRAMmodes:–  AllowNUMA=a2a,snc2,snc4,hemi,quad–  AllowMCDRAM=cache,split,equal,flat–  Quad,flatisthedefaultfornow(notfinalized)

•  NodescanberebootedautomaKcally–  Frequentrebootsarenotencouraged,astheycurrentlytakealong6me–  Wearetes6ngvariousmemorymodessotosetaproperdefaultmode

-41-

Example of running interactive batch job with KNL nodes in the quad,cache mode

zz217@gert01:~>salloc-N1–pdebug–t30:00-Cknl,quad,cachesalloc:Grantedjoballoca6on5545salloc:Wai6ngforresourceconfigura6onsalloc:Nodesnid00044arereadyforjobzz217@nid00044:~>numactl-Havailable:1nodes(0)node0cpus:0123456789101112131415161718192021222324252627282930313233343536373839404142434445464748495051525354555657585960616263646566676869707172…...238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271node0size:96757MBnode0free:94207MBnodedistances:node00:10

-42-

•  Runthenumactl–HcommandtocheckiftheactualNUMAconfigura6onmatchestherequestedNUMA,MCDRAMmode

•  Thequad,cachemodehasonly1NUMAnodewithallCPUsontheNUMAnode0(DDRmemory)

•  TheMCDRAMishiddenfromthenumactl–Hcommand(itisacache).

Sample job script to run under the quad,cache mode

SampleJobscript(PureMPI)#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH-Cknl,quad,cache

exportOMP_NUM_THREADS=1#op6onal*srun-n64-c4--cpu_bind=cores./a.out

-43-

Processaffinityoutcome0 68

136 204

1 69

137 205

3 70

138 206

4 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

Thisjobscriptrequests1KNLnodeinthequad,cachemode.Thesruncommandlaunches64MPItasksonthenode,alloca6ng4CPUspertask,andbindsprocessestocores.Theresul6ngtaskplacementisshownintherightfigure.TheRank0willbepinnedtoCore0,Rank1toCore1,…,Rank63willbepinnedtoCore63.EachMPItaskmaymovewithinthe4CPUsinthecores.

Each2x2boxaboveisacorewith4CPUs(hardwarethreads).ThenumbersshownineachCPUboxistheCPUids.Thelast4coresarenotusedinthisexample.Thecores4-59werenotbeshown.

*)Theuseof“exportOMP_NUM_THREADS=1”isop6onalbutrecommendedevenforpureMPIcodes.Thisistoavoidunexpectedthreadforking(compilerwrappersmaylinkyourcodetothemul6-threadedsystemprovidedlibrariesbydefault).

64 132

200 268

65 133

201 269

66 134

202 270

67 135

203 271

…

…

Rank0 Rank2 Rank3

Rank60 Rank61 Rank62 Rank63

Rank1

Core0

Core60

Core1 Core2 Core3

Core61 Core62 Core63

Core64 Core65 Core66 Core67


SampleJobscript(PureMPI)#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH-Cknl,quad,cache

exportOMP_NUM_THREADS=1#op6onal*srun–n16–c16--cpu_bind=cores./a.out

-44-

Processaffinityoutcome

0 68

136 204

1 69

137 205

3 70

138 206

4 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

Thisjobscriptrequests1KNLnodeinthequad,cachemode.Thesruncommandlaunches16MPItasksonthenode,alloca6ng16CPUspertask,andbindseachprocessto4cores/16CPUs.Theresul6ngtaskplacementisshownintherightfigure.TheRank0ispinnedtoCore0-3,andRank1toCore4-7,…,Rank15toCore60-63.TheMPItaskmaymovewithinthe16CPUsinthe4cores.

64 132

200 268

65 133

201 269

66 134

202 270

67 135

203 271

…

…

Rank0

Rank15

Core0

Core60

Core1 Core2 Core3




SampleJobscript(MPI+OpenMP)#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH-Cknl,quad,cache

exportOMP_NUM_THREADS=4srun-n64-c4--cpu_bind=cores./a.out

-45-


0 68

136 204

1 69

137 205

3 70

138 206

4 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

Thisjobscriptrequests1KNLnodeinthequad,cachemodetorun64MPItasksonthenode,alloca6ng4CPUspertask,andbindseachtasktothe4CPUsallocatedwithinthecores.EachMPItaskruns4OpenMPthreads.Theresul6ngtaskplacementisshownintherightfigure.TheRank0willbepinnedtoCore0,Rank1toCore1,…,Rank63toCore63.The4threadsofeachtaskarepinnedwithinthecore.Dependingonthecompilersusedtocompilethecode,the4threadsineachcoremayormaynotmovebetweenthe4CPUs.

Thread0

Thread1

Thread2

Thread3

64 132

200 268

65 133

201 269

66 134

202 270

67 135

203 271

…

…

Rank0 Rank2 Rank3


Rank1

Core0

Core60

Core1 Core2 Core3





exportOMP_PROC_BIND=trueexportOMP_PLACES=threadsexportOMP_NUM_THREADS=4srun-n64-c4--cpu_bind=cores./a.out

-46-

Process/threadaffinityoutcome0 68

136 204

1 69

137 205

2 70

138 206

3 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

WiththeabovetwoOpenMPenvs,eachthreadispinnedtoasingleCPUwithineachcore.Theresul6ngthreadaffinity(andtaskaffinity)isshownintherightfigure.E.g.,forRank0,Thread0ispinnedtoCPU0,Thread1toCPU68,Thread2toCPU136,andThread3ispinnedtoCPU204.

Thread0

Thread1

Thread2

Thread3

64 132

200 268

65 133

201 269

66 134

202 270

67 135

203 271

…

…

Rank0 Rank2 Rank3


Rank1

Core0

Core60

Core1 Core2 Core3





exportOMP_NUM_THREADS=8srun–n16–c16--cpu_bind=cores./a.out

-47-


0 68

136 204

1 69

137 205

2 70

138 206

3 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

Thread0

Thread1

Thread2

Thread3

Thread4

Thread5

Thread6

Thread7

Dependingonthecompilerimplementa6ons,the8threadsineachtaskmayormaynotmovebetween4cores/16CPUsallocatedtothehosttask.

64 132

200 268

65 133

201 269

66 134

202 270

67 135

203 271

…

…

Rank0

Rank15

Core0

Core60

Core1 Core2 Core3





exportOMP_PROC_BIND=trueexportOMP_PLACES=threadsexportOMP_NUM_THREADS=8srun–n16–c16--cpu_bind=cores./a.out

-48-


136 204

1 69

137 205

2 70

138 206

3 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

Core61

Thread0

Thread1

Thread2

Thread3

Thread4

Thread5

Thread6

Thread7

WiththeabovetwoOpenMPenvs,eachthreadispinnedtoasingleCPUonthecoresallocatedtothetask.Theresul6ngprocess/threadisshownintherightfigure.E.g.,forRank0,Thread0ispinnedtotheCPU0(onCore0),Thread1totheCPU1(onCore1),Threads2toCPU2(onCore2),andsoon.

64 132

200 268

65 133

201 269

66 134

202 270

67 135

203 271

…

…

Rank0

Rank15

Core0

Core60

Core1 Core2 Core3

Core62 Core63


Example of running under the quad,flat mode interactively

zz217@gert01:~>salloc-pdebug-t30:00-Cknl,quad,flatzz217@nid00037:~>numactl-Havailable:2nodes(0-1)node0cpus:01234567891011121314151617181920212223242526272829303132333435363738394041424344454647484950515253545556575859606162636465666768697071727374….262263264265266267268269270271node0size:96759MBnode0free:94208MBnode1cpus:node1size:16157MBnode1free:16091MBnodedistances:node010:10311:3110 -49-

zz217@cori10:~>scontrolshownodenid10388NodeName=nid10388Arch=x86_64CoresPerSocket=68CPUAlloc=0CPUErr=0CPUTot=272CPULoad=0.01AvailableFeatures=knl,flat,split,equal,cache,a2a,snc2,snc4,hemi,quadAcKveFeatures=knl,cache,quad…State=IDLEThreadsPerCore=4…BootTime=2016-10-31T13:43:12…

Sample job script to run under the quad,flat mode

SampleJobscript(MPI+OpenMP)#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH-Cknl,quad,flatexportOMP_NUM_THREADS=4srun-n64-c4--cpu_bind=cores./a.outexportOMP_NUM_THREADS=8srun–n16–c16--cpu_bind=cores./a.out

-50-


0 68

136 204

1 69

137 205

2 70

138 206

3 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

0 68

136 204

1 69

137 205

2 70

138 206

3 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

…

…

Rank0 Rank2 Rank3


Rank1

Core0

Core60

Core1 Core2 Core3


…

…

Rank0

Rank15

Core0

Core60

Core1 Core2 Core3


Sample job script to run under the quad,flat mode

SampleJobscript(MPI+OpenMP)#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH-Cknl,quad,flatexportOMP_PROC_BIND=trueexportOMP_PLACES=threadsexportOMP_NUM_THREADS=4srun-n64-c4--cpu_bind=cores./a.outexportOMP_NUM_THREADS=8srun–n16–c16--cpu_bind=cores./a.out

-51-


136 204

1 69

137 205

2 70

138 206

3 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

0 68

136 204

1 69

137 205

3 70

138 206

4 71

139 207

60 128

196 264

61 129

197 265

62 130

198 266

63 131

199 267

…

…

Rank0

Rank15

Core0

Core60

Core1 Core2 Core3


Thread0

Thread1

Thread2

Thread3

Thread4

Thread5

Thread6

Thread7

…

…

Rank0 Rank2 Rank3


Rank1

Core0

Core60

Core1 Core2 Core3


Sample job script to run under the quad,flat mode using MCDRAM

SampleJobscript(MPI+OpenMP)#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH-Cknl,quad,flat

#Whenthememoryfootprintfitsin16GBofMCDRAM(NUMAnode1),runsoutofMCDRAMexportOMP_NUM_THREADS=4exportOMP_PROC_BIND=trueexportOMP_PLACES=threads

srun-n64-c4--cpu_bind=cores--mem_bind=map_mem:1./a.out#orusingnumactl-msrun-n64-c4--cpu_bind=coresnumactl–m1./a.out

-52-

SampleJobscript(MPI+OpenMP)#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH-Cknl,quad,flat

#PrefersrunningonMCDRAM(NUMAnode1)ifmemoryfootprintdoesnotfitonMCDRAM,spillstoDDRexportOMP_NUM_THREADS=8exportOMP_PROC_BIND=trueexportOMP_PLACES=threads

srun–n16–c16--cpu_bind=coresnumactl–p1./a.out

How to check the process/thread affinity

•  Usesrunflag:--cpu_bind=verbose–  Needtoreadthecpumasksinhexidecimalformat

•  UseaCrayprovidedcodexthi.c(seebackupslides).•  Use--mem_bind=verbose,<type>tocheckmemoryaffinity

•  Usethenumastat–p<PID>commandtoconfirmwhileajobisrunning

•  Useenvironmentalvariables(Slurm,compilerspecific)–  SLURM_CPU_BIND_VERBOSE--cpu_bindverbosity(quiet,verbose).

-53-

A few useful commands •  sinfo–format=“%F%b”foravailablefeaturesofnodes,

orsinfo–format=“%C%b”–  A/I/O/T(allocated/idle/other/total)

•  scontrolshownode<nid>•  scontrolshowjob<jobid>-ddd•  sinfo–stoseeavailableparKKonsandnodes•  sbatch,srun,squeue,sinfoandotherSlurmcommand

manpages–  needtodis6nguishthejoballoca6on6me(#SBATCH)andjobstepcrea6on6me(srunwithinajobscript)

–  Someop6onsareonlyavailableatJoballoca6on6me,suchas–ntasks-per-core,someonlyworkwhencertainpluginsareenabled

-54-

Summary

•  Use–Cknl,<NUMA>,<MCDRAM>torequestKNLnodeswiththesameparKKonsasHaswellnodes(debug,orregular)

•  Alwaysexplicitlyusesrun’s--cpu_bindand-copKontospreadtheMPItasksevenlyoverthecores/CPUsonthenodes

•  UseOpenMPenvs,OMP_PROC_BINDandOMP_PLACEStofinepinthreadstotheCPUsallocatedtothetasks

•  Usesrun’s--mem_bindandnumactl–ptocontrolmemoryaffinityandaccessMCDRAM–  Usingmemkind/autoHBWlibrariescanbeusedtoallocateonlyselectedarrays/memoryalloca6onstoMCDRAM(Steve’stalk)

-55-

Summary (2)

•  Considerusing64coresoutof68inmostcases•  Moresamplejobscriptscanbefoundinourwebsite

–  h�p://www.nersc.gov/users/computa6onal-systems/cori/running-jobs/running-jobs-on-cori-knl-nodes/

•  WehaveprovidedajobscriptgeneratortohelpyoutogeneratebatchjobscriptsforKNL(andHaswell,Edison)

•  SlurmKNLfeaturesareinconKnuousdevelopmentandsomeinstrucKonsaresubjecttochange

------URLforthejobscriptgenerator:

h�ps://my.nersc.gov/script_generator.php

-56-

Backupslides

-57-

Cray provided a code to check process/thread affinity xthi.c (http://portal.nersc.gov/project/training/KNLUserTraing20161103/UsingCori/xthi.c/)

•  Tocompile,cc–qopenmp–oxthi.intelxthi.c#Intelcompilerscc–fopenmp–oxthi.gnuxthi.c#GNUcompilerscc–oxthi.gnuxthi.c#Craycompilers

•  Torun,salloc–N1–pdebug–Cknl,quad,flat#startaninterac6ve1nodejob…exportOMP_DISPLAY_ENV=true#todisplayenvsused/setbyopenmprun6meexportOMP_NUM_THREADS=4srun–n64–c4--cpu_bind=verbose,coresxthi.intel#run64taskswith4threadseach

Srun–n16–c16--cpu_bind=verbose,coresxthi.intel#run16tasks4threadseach

sinfo --format="%F %b” to show the number of nodes with active features

zz217@cori01:~>dateMonNov2815:10:36PST2016zz217@cori01:~>sinfo--format="%F%b"NODES(A/I/O/T)ACTIVE_FEATURES1805/157/42/2004haswell0/107/1/108knl,flat,a2a0/0/1/1knl0/225/6/231knl,flat,quad2500/1942/60/4502knl,cache,quad2676/995/6/3677quad,cache,knl768/0/0/768snc4,flat,knl0/8/0/8quad,flat,knl0/8/1/9snc2,flat,knl

-59-

Thiscommandshowsthattherethereare8179KNLnodesinquad,cachemode;239nodesinquad,flatmode;768nodesinsnc4,flatmode;9insnc2,flatmode

Theorderoffeaturesdoesnotmakedifference.

The --cpu_bind option of srun enables CPU bindings

-60-

srun–n4./a.out#noCPUbidings.Taskscanmovearoundwithin68cores/272CPUs

salloc–N1–pdebug–Cknl,quad,flat…

srun–n4–cpu_bind=cores./a.out

srun–n4--cpu_bind=threads./a.out

…...

Rank0

Core0

Core1

Core3

Core2

Core67 CPUCPUCPUCPU

…

Core65

Core66

…...

Rank0

Rank1

Rank2

Core0

Core1

Core3

Core2

Core67 CPUCPUCPUCPU

…

Core65

Core66

Rank3

…...

Core0

Core1

Core3

Core2

Core67 CPUCPUCPUCPU…

Core65

Core66

srun–n4--cpu_bind=map_cpu:0,204,67,271./a.out

Rank1

Rank2

Rank3

Rank0

Rank1

Rank3

Rank2

The --cpu_bind option: the -c option spreads tasks (evenly) on the CPUs on the node

-61-


srun–n16–c16–cpu_bind=cores./a.outsrun–n4–c8–cpu_bind=cores./a.out

…...

Rank0Core0

Core1

Core3

Core2

CPUCPUCPUCPU

…

Rank1

Rank3

Rank2Core4

Core5

Core7

Core6

First8cores/32CPUSareused;rest60cores/240CPUsstayidle

…...

Rank0

Core0

Core1

Core3

Core2

Core63 CPUCPUCPUCPU

…Core61

Core62

Rank1

Core60

Rank15

Last4cores/16CPUsareidle

The --cpu_bind option (continued): the –c option spread tasks (evenly) on the CPUs on the node

-62-


srun–n16–c16–cpu_bind=threads./a.outsrun–n4–c8–cpu_bind=threads./a.out

…...

Rank0Core0

Core1

Core3

Core2

CPUCPUCPUCPU

…

Rank1

Rank3

Rank2Core4

Core5

Core7

Core6


…...

Rank0

Core0

Core1

Core3

Core2

Core63 CPUCPUCPUCPU

…Core61

Core62

Rank1

Core60

Rank15

Last4cores/16CPUsareidle

The -c option: --cpu_bind=cores vs –cpu_bind=threads

-63-


srun–n4–c6–cpu_bind=threads./a.outsrun–n4–c6–cpu_bind=cores./a.out

…...

Rank0Core0

Core1

Core3

Core2

CPUCPUCPUCPU

…

Rank1

Rank3

Rank2Core4

Core5

Core7

Core6


…...

Core0

Core1

Core3

Core2

CPUCPUCPUCPU

…

Core4

Core5

Core7

Core6


Rank0

Rank1

Rank2

Rank3

Snc2,flat (salloc -N 1 -p regular -C knl,snc2,flat) zz217@nid11512:~>numactl-Havailable:4nodes(0-3)node0cpus:01234567891011121314151617181920212223242526272829303132336869707172737475767778798081828384858687888990919293949596979899100101136137138139140141142143144145146147148149150151152153154155156157158159160161162163164165166167168169204205206207208209210211212213214215216217218219220221222223224225226227228229230231232233234235236237node0size:48293MBnode0free:46047MBnode1cpus:34353637383940414243444546474849505152535455565758596061626364656667102103104105106107108109110111112113114115116117118119120121122123124125126127128129130131132133134135170171172173174175176177178179180181182183184185186187188189190191192193194195196197198199200201202203238239240241242243244245246247248249250251252253254255256257258259260261262263264265266267268269270271node1size:48466MBnode1free:44949MB

-64-

node2cpus:node2size:8079MBnode2free:7983MBnode3cpus:node3size:8077MBnode3free:7980MB

nodedistances:node01230:102131411:211041312:314110413:41314110

NUMAnode2

NUMAnode0

NUMAnode3

NUMAnode1

Core34-67

Core0-33

The default distribution on Cori is -m block:cyclic

-65-

salloc–N2–pdebug–Cknl,snc2,flat…srun–n11–c4–cpu_bind=cores./a.out#orsrun–n11–c4–cpu_bind=cores–mblock:cyclic./a.out

KNLNode1NUMAnode0

…...

NUMAnode1

…...

Rank0 Rank1Rank2 Rank3Rank4 Rank5

Core0

Core1

Core3

Core2

Core33 Core67

Core34

Core35

Core37

Core36

CPUCPUCPUCPU

… …

CPUCPUCPUCPU

KNLNode2NUMAnode0

…...

NUMAnode1

…...

Rank7Rank6Rank9Rank8

Core33

Core0

Core1

Core3

Core2

Core67

Core34

Core35

Core37

Core36

……

Rank10

CPUCPUCPUCPU CPUCPUCPUCPU

The default distribution on Cori is -m block:cyclic (continued)

-66-

salloc–N2–pdebug–Cknl,snc2,flat…srun–n11–c8–cpu_bind=cores./a.out#orsrun–n11–c8–cpu_bind=cores–mblock:cyclic./a.out

KNLNode1NUMAnode0

…...

NUMAnode1

…...

Core0

Core1

Core3

Core2

Core33 Core67

Core34

Core35

Core37

Core36

… …Rank2

Rank4 Rank5

Rank3

Rank1Rank0

KNLNode2NUMAnode0

…...

NUMAnode1

…...

Core33

Core0

Core1

Core3

Core2

Core67

Core34

Core35

Core37

Core36

……Rank10

Rank8 Rank9

Rank7Rank6

CPUCPUCPUCPU CPUCPUCPUCPUCPUCPUCPUCPU CPUCPUCPUCPU

Block distribution: -m block:block

-67-

Salloc–N2–pdebug–Cknl,snc2,flat…Srun–n11–c4–cpu_bind=cores–mblock:block./a.out

KNLNode1NUMAnode0

…...

NUMAnode1

…...

Rank0 Rank3Rank1 Rank4Rank2 Rank5

Core0

Core1

Core3

Core2

Core33 Core67

Core34

Core35

Core37

Core36

… …

KNLNode2NUMAnode0

…...

NUMAnode1

…...

Rank9Rank6Rank10Rank7

Core33

Core0

Core1

Core3

Core2

Core67

Core34

Core35

Core37

Core36

……

Rank8


Block distribution: -m block:block (continued)

-68-

Salloc–N2–pdebug–Cknl,snc2,flat…Srun–n11–c8–cpu_bind=cores–mblock:block./a.out

KNLNode1NUMAnode0

…...

NUMAnode1

…...

Rank0Core0

Core1

Core3

Core2

Core33 Core67

Core34

Core35

Core37

Core36

… …

KNLNode2NUMAnode0

…...

NUMAnode1

…...

Core33

Core0

Core1

Core3

Core2

Core67

Core34

Core35

Core37

Core36

……

Rank1

Rank2 Rank5

Rank4

Rank3

Rank7

Rank8

Rank6

Rank10

Rank9


Sample job script to run MPI+OpenMP under the snc2,flat mode

-69-

#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH–LSCRATCH#SBATCH-Cknl,snc2,flat

exportOMP_NUM_THREADS=8exportOMP_PROC_BIND=trueexportOMP_PLACES=threads

#using2CPUs(hardwarethreads)percoreusingDDRmemory(orusingMCDRAMvialibmemkind)srun–n16–c16--cpu_bind=cores–mem_bind=local./a.out

#using2CPUs(hardwarethreads)percoreusingMCDRAMsrun–n16–c16--cpu_bind=cores–mem_bind=map_mem:2,3./a.out#orsrun–n16–c16--cpu_bind=coresnumactl–m2,3./a.out

#using2CPUs(hardwarethreads)percorewithMCDRAMpreferredsrun–n16–c16--cpu_bind=coresnumactl–p2,3./a.out

#using4CPUs(hardwarethreads)percorewithMCDRAMpreferredsrun–n32–c8--cpu_bind=coresnumactl–p2,3./a.out

Sample job script to run large jobs (> 1500 MPI tasks) (quad,flat)

Samplejobscript(MPI+OpenMP)#!/bin/bash-l#SBATCH–N100#SBATCH–pregular#SBATCH–t1:00:00#SBATCH–LSCRATCH#SBATCH-Cknl,quad,flatexportOMP_NUM_THREADS=4exportOMP_PROC_BIND=trueexportOMP_PLACES=threads#using4CPUs(hardwarethreads)percoreusingDDRmemory(orusingMCDRAMvialibmemkind)srun--bcast=/tmp/a.out–n6400–c4--cpu_bind=cores–mem_bind=local./a.out#or#using4CPUs(hardwarethreads)percorewithMCDRAMpreferredsbcast./a.out/tmp/a.out#copya.outtothe/tmpofeachcomputenodeallocatedfirstsrun–n6400–c4--cpu_bind=coresnumactl–p1/tmp/a.out

-70-

Sample job script to use core specialization (quad,flat) Samplejobscript(MPI+OpenMP)#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH–LSCRATCH#SBATCH-Cknl,quad,flat#SBATCH–S1

exportOMP_NUM_THREADS=4exportOMP_PROC_BIND=trueexportOMP_PLACES=threads#using4CPUs(hardwarethreads)percoreusingDDRmemory(orusingMCDRAMvialibmemkind)Srun–n64–c4--cpu_bind=cores–mem_bind=local./a.out#or#using4CPUs(hardwarethreads)percorewithMCDRAMpreferredSrun–n64–c4--cpu_bind=coresnumactl–p1./a.out

-71-

Sample job script to run with Intel MPI (quad,flat) Samplejobscript(MPI+OpenMP)#!/bin/bash-l#SBATCH–N1#SBATCH–pregular#SBATCH–t1:00:00#SBATCH-Cknl,quad,flat

exportOMP_NUM_THREADS=4exportOMP_PROC_BIND=trueexportOMP_PLACES=threadsmoduleloadimpiexportI_MPI_PMI_LIBRARY=/usr/lib64/slurmpmi/libpmi.so#exportI_MPI_FABRICS=shm:tcp

#using4CPUs(hardwarethreads)percoreusingDDRmemory(orusingMCDRAMvialibmemkind)srun–n64–c4--cpu_bind=cores–mem_bind=local./a.out

#or#using4CPUs(hardwarethreads)percorewithMCDRAMpreferredsrun–n64–c4--cpu_bind=coresnumactl–p1./a.out

-72-

Thankyou!

-73-

Using Cori - National Energy Research Scientific … · Using Cori. How to Compile & MCDRAM ......

Documents

Transcript of Using Cori - National Energy Research Scientific … · Using Cori. How to Compile & MCDRAM ......