A Brief Introduction to Chroma

S. Collins, University of Regensburg

Outline

◮ What is chroma?

◮ What can it do?

◮ Design.

◮ XML input and running chroma.

◮ How to get it.

◮ How to compile it.

2

Other chroma lectures and tutorials

Several chroma lectures and tutorials have been given before (seechroma web site):

◮ HackLatt’09/08/07/06, Balint Joo, Dave Richards.

◮ Numerical Methods Lecture, Seattle Summer School, BalintJoo,

◮ Lattice Practices 2008, Carsten Urbach and Andreas Jüttner.

Some have a different emphasis - e.g. hmc.

3

What is chroma?

Chroma is a C++ software package for performing Lattice QCDcalculations.

Chroma is based on QDP++.

◮ QDP++ provides an interface where operations can beapplied on lattice wide objects.

- e.g. LatticeColorMatrix, LatticeFermion, LatticePropagatore.g. Gamma(G5) ∗ quark_propagator ∗ Gamma(G5)

◮ The level of abstraction is such that user code written usingthe interface can be run unchanged on a single processor ormultiprocessors with parallel communications.

◮ Architectural dependencies are hidden below the interface.

For chroma: storage, IO, manipulation and parallelizationdelegated to QDP++.

4

What is chroma?

QDP++ itself uses other packages

- QIO for data parallel IO + XpathReader (included in QDP++download), libxml2

- QMT for multi-threading,

- QMP for parallel communications.

Q-packages developed as part of the SciDAC initiative.Additional target architecture specific optimized packages,

Chroma - SSE Wilson Dslash (included in chroma download)- CG-DWF (included in chroma download)- BAGEL Wilson Dslash- BAGEL clover

QDP++ - BAGEL QDP

Also need BAGEL generator for BAGEL optimizations.

5

What is chroma?

There is a hierachy of software packages needed for chroma

- Scalar build very simple: chroma and qdp++(QIO/XpathReader included, need libxml2)

- Parallel build more complicated depending on target.

Chroma compiles on any target machine,

- Optimization packages mean good performance on sometargets, SSE2 based machines, BlueGene/L(P).

6

What can chroma do?

The chroma software package can generate configurations usingthe executables

◮ purgaug - for pure gauge configurations using heat bath andoverrelaxation algorithms,

◮ hmc - for hybrid Monte Carlo for dynamicalconfigurations (including “sea quarks”).

The executable chroma can be used to calculate manyobservables, using the configurations generated.

The configurations and observables can be calculated in manydifferent ways (different actions, gauges, using different algorithmsetc),

◮ naturally, orientated towards the programme of the JLablattice group + collaborators (who wrote the code),

◮ constantly expanding,- some measurements will not be refined or working properly.

7

What observables can be calculated?

An incomplete list:Eigenvalue/-vector measurements

◮ Eigenvalue/-vector calculation via Kalkreuter-Simmaalgorithm

Gluonic observables

◮ Plaquette, Wilson loops

Hadronic observables

◮ Meson and baryon 2pt + 3pt functions

◮ Meson-meson 4pt functions

◮ Static-light correlators

◮ Hybrid mesons

◮ Nucleon-nucleon 4pt functions

8

Some of the available actions

Gauge actions

◮ Wilson gauge action

◮ Lüscher-Weisz gauge action(tree level and 1-loop)

Fermion actions

◮ Staggered fermions (naïve and Asqtad)

◮ Wilson fermions

◮ Clover improved Wilson fermions(Standard, SLiNC)

◮ Domain wall fermions

◮ Overlap fermions (many approx. schemes in 4D and 5D anddeflation)

9

Some of the available BCs, inverters, sources and sinks

Boundary conditions◮ Periodic, antiperiodic BCs◮ Dirichlet BCs◮ Twisted BCs

Inverters◮ Conjugate gradient◮ BiCGStab, BiCGStab with “reliable” updates for mixed

precision◮ Minimal Residual◮ EigCG

Quark sources and sinks◮ Point, Shell (gauge invariant Gaussian)◮ Derivative (covariant derivative and displacements)◮ Momentum wall◮ Stochastic 10

Gauge fixing and link smearings

Gauge fixing

◮ Axial gauge

◮ Coulomb gauge

◮ Landau gauge

◮ and Random gauge transformation

Link smearings

◮ APE

◮ HYP

◮ Stout

11

Design

Here, only mention basics and what is relevant for understandinginput files.

◮ Chroma is an object-orientated code.◮ Makes use of design patterns - general solutions to commonly

occuring problems (there are books on this).- e.g. handle, factory function, factory, singleton.

◮ Implementation of design patterns comes from LOKI library(A. Alexandrescu).

How to achieve a code which

◮ has a high degree of flexibility at run time in terms of what iscalculated and how,

◮ where the calculations themselves may be complicatedinvolving several steps,

◮ (easily) extendable?

12

Design

Split the calculation into modules or tasks,

◮ a task may appear repeatedly in different calculations, e.g.make a source for the propagator, invert on the source toobtain the propagator.

◮ natural in object-oriented programming.

When calculating observables the tasks are referred to as inlinemeasurements in chroma.

At run time need to specify what tasks need to be performed,

◮ many tasks possible,

◮ many ways to perform the task (e.g. what inverter to use forgenerating the propagator).

13

Solution: use the Factory pattern

Digression: object-oriented programs are designed in terms ofobjects,

◮ a class is used to define the type of object,e.g. class InlinePropagator{}, class InlineMakeSource{},

◮ classes for objects with similar properties can be derived froma base class (inheritance),e.g. class InlineMeas{},

class InlinePropagator: public InlineMeas {},class InlineMakeSource: public InlineMeas {},

◮ One can create instances of the object using a creationfunction, (c.f. new InlinePropagator())e.g. InlinePropagator* createInlineProp(),

InlineMakeSource* createInlineMakeS(),

Can doInlineMeas *my_inline;

my_inline = createInlineProp();

my_inline = createInlineMakeS();14

The Factory pattern

This is essentially a map from an id (or key) to a creation functionfor the object.

Map:

“PROPAGATOR” −→ InlinePropagator* createInlineProp()

Can now create different inline measurements by querying the map.std::map<std::string, InlineMeas* (*)(void)> factory_map;

factory_map.insert(make_pair("PROPAGATOR",createInlineProp()));

factory_map.insert(make_pair("MAKE_SOURCE",createInlineMakeS()));

withInlineMeas *my_line = (factory_map["PROPAGATOR"])();

This pattern is used throughout chroma,

- e.g. smearing, inverters, boundary conditions, . . .,- in the input file factory keys are capitalized.

More complicated: chroma uses handles, maps are singletons etc.

15

IO

Output of observables calculated within an inline measurement isdealt with within the inline measurement.

Lattice objects in chroma (defined in QDP++),

- e.g. LatticeColorMatrix, LatticeFermion,LatticePropagator,

are needed as input for inline measurements or created by themeasurement.

These objects are assigned an ID (string) and stored in memory as“NamedObjects”,

- using a map, TheNamedObjMap (singleton).

- Input/Output dealt with using inline measurements (separatedfrom the calculation of observables).

16

Documentation

Chroma is a very large package - difficult to get an overview(QDP++ better documented).

Useful:◮ doxygen documentation - generated from the code, accessible

from chroma website or generate from source code yourself,◮ cd chroma/docs; make

◮ lectures/tutorials,◮ notes - Huey-Wen Lin’s compilation of notes at

http://www.jlab.org/~hwlin/Chroma_notes/chroma_note.html,also available as latex source files on chroma website or inchroma source code (chroma/docs/notes),

◮ chroma is changing all the time, look at comments in gitrepository.

Find out how to invoke different gauge actions/boundaryconditions/inline measurements by looking in the test files.

17

Test files

These are written for regression tests and can be found insub-directories of source code: chroma/tests

◮ chroma/tests/chroma (chroma inline measurement tests)

◮ chroma/tests/hmc (HMC tests)

◮ chroma/tests/purgaug (pure gauge configuration generationtests)

◮ . . .

For each test there is

◮ an input file for the test,

◮ the corresponding output file.

No guarantees - there can be/have been bugs in chroma. Lesslikely in older (well used) measurements.

18

Citations

If you use chroma in your simulations you must cite theappropriate people.

Always R. G. Edwards, B. Joo,arXiv:hep-lat/0409003, Nucl. Phys B140(Proc. Suppl) 882, 2005.SSE Optimised Dslash code C. McClendon,Jlab preprint JLAB-THY-01-29.BAGEL assembly generator P. A. Boyle,http://www.ph.ed.ac.uk/~paboyle/bagel/Bagel.html, 2005.QUDA GPU Library M. A. Clark, R. Babich, K. Barros, R. C.Brower, C. Rebbi, arXiv:0911.3191v1 [hep-lat]

Look at the chroma website for the most up-to-date references.

19

Running

For generating pure gauge configurations:

purgaug -i config.ini.xml -o config.out.xml

Input file: XML,

- what gauge action to use, lattice size, how many updates, anyobservables to calculate . . ..

Output:

- standard output (screen): can include status, performance,timings etc.

- XML file, config.out.xml: log of calc., can contain physics e.g.the plaquette or the values of any observables that werecalculated.

XML output may be lengthy.

20

Advantages and disadvantages of using chroma

Plus

◮ Very powerful, general software available for free!

- Many observables can be calculated with a large choice ofactions, inverters etc.

- All parallelization done behind the scenes.

◮ Many features chosen at runtime, lattice size, inverter, actionsetc.

- No need to recompile.

◮ Runs on basically every architecture, with target specificimprovements

- good performance on many machines.

◮ How long would it take to write the code that you want?

21

Advantages and disadvantages of using chroma

Minus

◮ Lack of documentation means it is painful to

- compile - what options on what platforms, many packages tocompile,

- compose the input XML file, what calculations are possible,what input parameters are needed?

- understand and extract the observables calculated from outputfiles.

◮ Compilation of chroma takes a long time.

◮ XML makes input and output lengthy.

◮ Need a reasonable level of C++ to make additions (even tocut and paste).

22

Overview of the input information

The program purgaug needs the following information in order togenerate configurations.

◮ The starting configuration, whether starting from scratch,before thermalisation, or continuuing from already thermalisedconfigurations.

◮ The seed for the random number generator and the number ofthermalisation sweeps and the number of sweeps in total.

◮ The gauge action information, include lattice size, boundaryconditions etc.

◮ The Monte Carlo algorithm, how many heat bath sweeps andoverrelaxation sweeps per update.

◮ Inline measurements, what observables to measure on theconfigurations as they are generated.

23

Input XML: general structure

General structure of an input XML file:

<?xml version="1.0"?> <!-- XML statement -->

<purgaug>

<Cfg>

<cfg_type>WEAK_FIELD</cfg_type> <!-- config. format -->

<cfg_file>dummy</cfg_file> <!-- file name -->

</Cfg>

<MCControl>..</MCControl> <!-- Seed and no. updates -->

<InlineMeasurements>..</InlineMeasurements>

<!-- observables to measure -->

<HBItr>..</HBItr>

<!-- gauge action and No. of HB and Ovr -->

</purgaug>

24

Input XML

XML:◮ content is wrapped up in tags <purgaug></purgaug>,◮ tags can be nested,◮ comments with <!-- -->.◮ best viewed in mozilla family of browsers

(firefox/iceape/iceweasel/icecat) or using an editor that canhandle XML.

Tags are mostly self-explanatory (more details are given in thetutorial)

◮ <CFG>< /CFG> - specifies the configuration to be read inat start-up. Has object id default_gauge_field.

◮ <cfg_type>< /cfg_type> - the type or format of theconfiguration,

- UNIT (free-field), WEAK_FIELD (small fluct.), DISORDERED(random), CLASSICAL_SF. All generated within chroma.

- CPPACS, KYU, MILC, NERSC, SCIDAC (also reads ILDGformat), SZIN, SZINQIO. Formats allowed for external files.

25

Additional information

For this tutorial chroma has already been installed, so you do notneed to download and install the software yourself.

However, the following slides give you some information of thenecessary steps involved.

26

Getting the software

Chroma website:http://usqcd.jlab.org/usqcd-docs/chroma/ & links

The latest stable versions of chroma and it’s dependencies(qdp++/qmp/bagel/. . .) are available as tarballs.

◮ e.g. chroma-3.38.0.tar.gz, qdp++-1.36.1.tar.gz,qmp-2.1.6.tar.gz

Chroma source management system is Git,◮ list of projects on gitweb, chroma.git, qdp++.git etc.

To clone chroma repository with submodules, (git version 1.6.5.2or better):

◮ git clone git://git.jlab.org/pub/lattice/usqcd/chroma.git◮ cd chroma◮ git submodule update --init --recursive

Older cvs repository.27

Compiling

Chroma and dependencies need to be compiled, “built”, insequence, - dependencies first then chroma.

What you need to compile depends on scalar/parallel and theplatform:

◮ scalar: qdp++, chroma

◮ multi-threading: qmt, qdp++, chroma

◮ parallel: qmp, qdp++, chroma (gmp, libxml already installed)

◮ parallel (BlueGeneL/P): bagel, bagel_qdp, qmp, qdp++,bagel_wilson_dslash, bagel_clover, chroma

Chroma takes the longest to compile > 30 minutes.

Each package built using: configure; make; make install,

◮ but configure has many options.

28

Compiling

Example of a few of the qdp++ configure options:

./configure

--prefix=QDP++_INST_DIR QDP++ installation dir

--enable-Nd=Nd # of spacetime dim., default=4

--enable-Nc=Nc # of colours, default=3

--enable-Ns=Ns # of spin comp., default=4

--enable-parallel-arch=arch arch = scalar, parscalar

--enable-precision=prec prec = single, double

--enable-sse optimize code with intel sse

--with-qmp=QMP_INST_DIR QMP installation directory

Don’t repeat the above flags for chroma.◮ additional flags: --enable-sse-wilson-dslash

For what configure options are available use configure --help.Examples can be found in jlab-standard-chroma-standard buildpackage in the git repository.

29

jlab-standard-chroma-standard-build

Used by JLab to automate, download, building and installing ofchroma & dependencies.

Scripts for different platforms & builds (scalar/parallel)

◮ scalar.sh

◮ bgp_build.sh

◮ ranger_build.sh

Look in these scripts to see what packages are needed.

Look in package/machine/build-type/build/configure.sh for usefulflags

- e.g. chroma/bgp/parscalar-bgp-double-sloppy/build/configure.sh

With a small amount of work, these scripts can also (almost) workfor you.

30

Compiling

Once chroma is installed then find executables in

◮ CHROMA_INST_DIR/bin/chroma

◮ in same directory, hmc, purgaug + others

◮ also chroma-config, usage: chroma-config --[options]

>chroma-config --prefix

/psi_devel/chroma_3.38.0_scalar-single64

>chroma-config --version

3.38.0

>chroma-config --Nd

4

>chroma-config --parallel-arch

scalar

31