pbench-docs Documentation

pbench-docs Documentation

Haowei

Jul 27, 2020

Contents:

1 Pbench Start Guide 1

2 Configuration 21

3 Getting started 25

4 Installation 31

5 Structure of results 33

6 Travis CI 37

7 Contributing 39

i

CHAPTER 1

Pbench Start Guide

1.1 WARNING

This document may describe some future capabilities of pbench. We are working as fast as we can to make the codecatch up to the documentation and vice-versa, but things may not be exactly as described here. If you find somethingnot working as described here, please let us know: it may be a bug in the documentation, a bug in the code or a featurethat we want to implement but we haven’t quite gotten to yet.

1.2 What is pbench?

Pbench is a harness that allows data collection from a variety of tools while running a benchmark. Pbench hassome built-in script that run some common benchmarks, but the data collection can be run separately as well witha benchmark that is not built-in to pbench, or a pbench script can be written for the benchmark. Such contributions aremore than welcome!

1.3 Quick links

Convenience links for some places of interest (N.B. The results directory has changed - see Accessing results on theweb for the details):

• Results directory.

• pbench RPM repo

• Release notes

[2015-01-23 Fri] In preparation for moving the incoming directory to bigger storage, older directories in the pbenchincoming area were archived (tarred/compressed/moved to new area). The list of these directories can be found here.

1

http://pbench.example.com/results/

http://pbench.example.com/repo

http://pbench.example.com/doc/RELEASE-NOTES.html

./archived-directories.html


1.4 TL;DR version

Here is the minimum set of commands to get from 0 to a crawl:

wget -O /etc/yum.repos.d/pbench.repo http://repohost.example.com/repo/yum.repos.d/→˓pbench.repoyum install pbench-agent -y. /etc/profile.d/pbench-agent.sh # or log out and log back inregister-tool-setuser-benchmark -C test1 -- ./your_cmd.shmove-results

Visit the Results directory to see the results: assuming you ran the above on a host named “myhost”, the results willbe in its “myhost/user_benchmark_test1_<yyyy.mm.ddTHH.MM.SS>” subidrectory.

For explanations, and if crawling is not enough, see subsequent sections.

1.5 How to install

Visit

http://repohost.example.com/repo/yum.repos.d

and copy the pbench.repo file to /etc/yum.repos.d on the SUT1. If the SUT consists of multiple hosts, thefile (and pbench) should be installed on all of them. Here is a command that you can execute (as root) on the SUT toaccomplish this:

wget -O /etc/yum.repos.d/pbench.repo http://repohost.example.com/repo/yum.repos.d/→˓pbench.repo

Then

yum install pbench-agent

will install the pbench agent in /opt/pbench-agent. Before you do anything else, you need to sourcethe file /etc/profile.d/pbench-agent.sh. It sets PATH appropriately and defines an env variable_PBENCH_AGENT_CONFIG to point to the default pbench configuration file (see Customizing below). Alterna-tively, logging out and logging back in will source the script automatically.

1.5.1 Updating pbench

Since the package (as well as the benchmark and tools RPMS that are in the repo) gets updated fairly often, I havefound it necessary to clean the yum cache in order for yum to see the new package, although you might want to tryupdating without cleaning the cache first and if yum reports no packages to update, then try again after cleaning thecache:

yum clean expire-cacheyum update pbench-agent

If you try the above update and encounter problems (e.g. the pbench scripts, or the config file, are not found even afteryou log out and log back in), then try the following workaround:

1 “System under test”.

2 Chapter 1. Pbench Start Guide

http://pbench.example.com/results/

http://repohost.example.com/repo/yum.repos.d


yum reinstall pbench-agent

That should reestablish the missing symlink.

The workaround should not be necessary if your currently installed release is 0.31-95 or later.

In desperate situations, removing the pbench-agent package and reinstalling after cleaning the cache should clearup any problems:

yum erase pbench-agentyum clean expire-cacheyum install pbench-agent

If you upgrade to a release later than -102, please also ensure that you clear out your tools and reregister them afterthe upgrade: the label handling has changed. For example:

clear-toolsregister-tool-set --label=ceph-server --remote=my-ceph-serverregister-tool-set --label=kvmhost --remote=my-kvmhostregister-tool-set --label=kvmguest --remote=my-kvmguest

1.6 First steps

All of the commands take a --help option and produce a terse usage message.

The default set of tools for data collection can be enabled with

register-tool-set

You can then run a built-in benchmark by invoking its pbench script - pbench will install the benchmark if necessary2:

pbench_fio

When the benchmark finishes, the tools will be stopped as well. The results can be collected and shipped to thestandard storage location3 with

move-results

or

copy-results

1.6.1 First steps with user-benchmark

If you want to run something that is not already packaged up as a benchmark script, you may be able to use theuser-benchmark script: it takes a command as argument, starts the collection tools, invokes the command, stopsthe collection tools and postprocesses the results. So the workflow becomes:

2 The current version of pbench-agent yum installs prebuilt RPMs of various common benchmarks: dbench, fio, iozone, linpack, smallfile anduperf, as well as the most recent version of the sysstat tools. We are planning to add more benchmarks to the list: iperf, netperf, streams, maybe thephoronix benchmarks. If you want some other benchmark (AIM7?), let us know.

3 The standard storage location currently is http://resultshost.example.com/incoming but it is subject to change without notice.

1.6. First steps 3

http://resultshost.example.com/incoming


register-tool-setuser-benchmark --config=foo -- myscript.shmove-results

See for host in $hosts ;do register-tool-set –remote=$host done user-benchmark –config=foo – myscript.sh move-results

Apart from having to register the collection tools on all the hosts, the rest is the same: user-benchmark will startthe collection tools on all the hosts, run myscript.sh, stop the tools and run the postprocessing phase, gathering upall the remote results to the local host (the local host may be just a controller, not running any collection tools itself,or it may be part of the set of hosts where the benchmark is run, with collection tools running).

The underlying assumption is that myscript.sh will run your benchmark on all the relevant hosts and will copy allthe results into the standard directory which postprocessing will copy over to the controller host. user-benchmarkcalls the script in its command-line arguments (everything after the – is just execed by user-benchmark) and redirectsits stdout to a file in that directory: $benchmark_run_dir/result.txt.

1.7 Defaults

The benchmark scripts source the base script (/opt/pbench-agent/base) which sets a bunch of defaults:

pbench_run=/var/lib/pbenchpbench_log=/var/lib/pbench/pbench.logdate=`date --utc "+%Y.%m.%dT%H.%M.%S"`hostname=`hostname -s`[email protected]_repo_dir=/pbench/public_html/incomingssh_opts='-o StrictHostKeyChecking=no'

These are now specified in the config file /opt/pbench-agent/config/pbench.conf.

1.8 Available tools

The configured default set of tools (what you would get by running register-tool-set) is:

• sar, iostat, mpstat, pidstat, proc-vmstat, proc-interrupts, perf

In addition, there are tools that can be added to the default set with register-tool:

• blktrace, cpuacct, dm-cache, docker, kvmstat, kvmtrace, lockstat, numastat, perf, porc-sched_debug, proc-vmstat, qemu-migrate, rabbit, strace, sysfs, systemtap, tcpdump, turbostat, virsh-migrate, vmstat

There is a default group of tools (that’s what register-tool-set uses), but tools can be registered inother groups using the --group option of register-tool. The group can then be started and stopped usingstart-tools and stop-tools using their --group option.

Additional tools can be registered:

register-tool --name blktrace

or unregistered (e.g. some people prefer to run without perf):

unregister-tool --name perf



Note that perf is run in a “low overhead” mode with options “record -a –freq=100”, but if you want to run it differently,you can always unregister it and register it again with different options:

unregister --name=perfregister-tool --name=perf -- --record-opts="record -a --freq=200"

Tools can be also be registered, started and stopped on remote hosts (see the --remote option described in Runningpbench collection tools with an arbitrary benchmark below for more on this)

Note that in many of these scripts the default tool group is hard-wired: if you want them to run a different tool group,you need to edit the script5.

1.9 Utility scripts

This section is needed as preparation for the Second steps section below.

Pbench uses a bunch of utility scripts to do common operations. There is a common set of options for some of these:--name to specify a tool, --group to specify a tool group, --with-options to list or pass options to a tool,--remote to operate on a remote host (see entries in the FAQ section below for more details on these options).

The first set is for registering and unregistering tools and getting some information about them:

list-tools list the tools in the default group or in the specified group; with the –name option, list the groups thatthe named tool is in. TBD: how do you list all available tools whether in a group or not?

register-tool-set call register-tool on each tool in the default list.

register-tool add a tool to a tool group (possibly remotely).

unregister-tool remove a tool from a tool group (possibly remotely).

clear-tools remove a tool or all tools from a specified tool group (including remotely).

The second set is for controlling the running of tools – start-tools and stop-tools, as well aspostprocess-tools below, take --group, --dir and --iteration options: which group of tools tostart/stop/postprocess, which directory to use to stash results and a label to apply to this set of results. kill-toolsis used to make sure that all running tools are stopped: having a bunch of tools from earlier runs still running has beenknow to happen and is the cause of many problems (slowdowns in particular):

start-tools start a group of tools, stashing the results in the directory specified by --dir.

stop-tools stop a group of tools.

kill-tools make sure that no tools are running to pollute the environment.

The third set is for handling the results and doing cleanup:

postprocess-tools run all the relevant postprocessing scripts on the tool output - this step also gathers up tooloutput from remote hosts to the local host in preparation for copying it to the results repository.

clear-results start with a clean slate.

copy-results copy results to the results repo.

move-results move the results to the results repo and delete them from the local host.

edit-prefix change the directory structure of the results (see the Accessing results on the web section below fordetails).

cleanup clean up the pbench run directory - after this step, you will need to register any tools again.

5 That will be handled by a configuration file in the future.

1.9. Utility scripts 5


register-tool-set, register-tool and unregister-tool can also take a --remote option (seeBenchmark scripts options).

• Check that the necessary prerequisites are installed and if not, install them.

• Iterate over some set of benchmark characteristics (e.g. pbench_fio iterates over a couple test types: read,randread and a bunch of block sizes), with each iteration doing the following:

– create a benchmark_results directory

– start the collection tools

– run the benchmark

– stop the collection tools

– postprocess the collection tools data

The tools are started with an invocation of start-tools like this:

start-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dir

where the group is usually “default” but can be changed to taste as described above, iteration is a benchmark-specifictag that disambiguates the separate iterations in a run (e.g. for pbench_fio it is a combination of a count, the testtype, the block size and a device name), and the benchmark_tools_dir specifies where the collection results are goingto end up (see the Results structure section for much more detail on this).

The stop invocation is exactly parallel, as is the postprocessing invocation:

stop-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dirpostprocess-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dir

1.9.1 Benchmark scripts options

Generally speaking, benchmark scripts do not take any pbench-specific options except --config (see register-tool–name=blktrace [–remote=foo] – –devices=/dev/sda,/dev/sdb

There is no default and leaving it empty causes errors in postprocessing (this should be flagged).

1.9.2 Utility script options

Note that move-results, copy-results and clear-results always assume that the run directory is thedefault /var/lib/pbench.

move-results and copy-results now (starting with pbench version 0.31-108gf016ed6) take a --prefixoption. This is explained in the Accessing results on the web section below.

Note also that start/stop/postprocess-tools must be called with exactly the same arguments. The built-inbenchmark scripts do that already, but if you go your own way, make sure to follow this dictum.

--dir specify the run directory for all the collections tools. This argument must be used by start/stop/postrprocess-tools, so that all the results files are in known places:

start-tools --dir=/var/lib/pbench/foostop-tools --dir=/var/lib/pbench/foopostprocess-tools --dir=/var/lib/pbench/foo

--remote specify a remote host on which a collection tools (or set of collection tools) is to be registered:



register-tool --name=<tool> --remote=<host>

1.10 Running pbench collection tools with an arbitrary benchmark

If you want to take advantage of pbench’s data collection and other goodies, but your benchmark is not part of theset above (see [[Available benchmark scripts][Available benchmark scripts]]), or you want to run it differently so thatthe pre-packaged script does not work for you, that’s no problem (but, if possible, heed the *WARNING above). Thevarious pbench phases can be run separately and you can fit your benchmark into the appropriate slot:

group=defaultbenchmark_tools_dir=TBD

register-tool-set --group=$groupstart-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dir<run your benchmark>stop-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dirpostprocess-tools --group=$group --iteration=$iteration --dir=$benchmark_tools_dircopy-results

Often, multiple experiments (or “iterations”) are run as part of a single run. The modified flow then looks like this:

group=defaultexperiments="exp1 exp2 exp3"benchmark_tools_dir=TBD

register-tool-set --group=$groupfor exp in $experiments ;do

start-tools --group=$group --iteration=$exp<run the experiment>stop-tools --group=$group --iteration=$exppostprocess-tools --group=$group --iteration=$exp

donecopy-results

Alternatively, you may be able to use the user-benchmark script as follows:

user-benchmark --config="specjbb2005-4-JVMs" -- my_benchmark.sh

which is going to run my_benchmark.sh in the <run your benchmark> slot above. Iterations and such areyour responsibility.

user-benchmark can also be used for a somewhat more specialized scenario: sometimes you just want to run thecollection tools for a short time while your benchmark is running to get an idea of how the system looks. The ideahere is to use user-benchmark to run a sleep of the appropriate duration in parallel with your benchmark:

user-benchmark --config="specjbb2005-4-JVMs" -- sleep 10

will start data collection, sleep for 10 seconds, then stop data collection and gather up the results. The config argumentis a tag to distinguish this data collection from any other: you will probably want to make sure it’s unique.

This works well for one-off scenarios, but for repeated usage on well defined phase changes you might want toinvestigate Triggers.

1.10. Running pbench collection tools with an arbitrary benchmark 7


1.11 Remote hosts

Note that from latest version onwards, we would like to have a file at http://pbench.example.com/pbench-archive-hostwhere the FQDN of the pbench web-server lies and the results would be pushed here. Currently it is archivehost.example.com. This would mean, if in future, we would like to change the central server settings, we wouldn’t wantthe users to upgrade to a latest version of pbench. Rather, just change the FQDN in this hosted file and then new resultswould automatically be pushed to the updated location.

1.11.1 Multihost benchmarks

Usually, a multihost benchmark is run using a host that acts as the “controller” of the run. There is a set of hosts onwhich data collection is to be performed while the benchmark is running. The controller may or may not be itself partof that set. In what follows, we assume that the controller has password-less ssh access to the relevant hosts.

The recommended way to run your workload is to use the generic user-benchmark script. The workflow in thatcase is:

• Register the collection tools on each host in the set:

for host in $hosts ;doregister-tool-set --remote=$host

• Invoke user-benchmark with your workload generator as argument: that will start the collection tools onall the hosts and then run your workload generator; when that finished, it will stop the collection tools on allthe hosts and then run the postprocessing phase which will gather the data from all the remote hosts and run thepostprocessing tools on everything.

• Run copy-results or move-results to upload the data to the results server.

If you cannot use the user-benchmark script, then the process becomes more manual. The workflow is:

• Register the collection tools on each host as above.

• Invoke start-tools on the controller: that will start data collection on all of the remote hosts.

• Run the workload generator.

• Invoke stop-tools on the controller: that will stop data collection on all of the remote hosts.

• Invoke postprocess-tools on the controller: that will gather all the data from the remotes and run thepostprocessing tools on all the data.

• Run copy-results or move-results to upload the data to the results server.

1.12 Customizing

Some characteristics4 of pbench are specified in config files and can be customized by adding your own config file tooverride the default settings.

TBD4 Only a few such characteristics exist today, but the plan is to move more hardwired things into the config files from the scripts. If you need to

override some setting and have to modify scripts in order to do so, let us know: that’s a good candidate for the config file.


http://pbench.example.com/pbench-archive-host


1.13 Best practices

1.13.1 Clear results

The move-results script removes the results directory (assumed to be within the /var/lib/pbench hierarchy)after copying it the results repo. But if there are previous results present (perhaps because move-results was neverinvoked, or perhaps because copy-results was invoked instead), move-results will copy all of them: youprobably do not want that.

It’s a good idea in general to invoke clear-results, which cleans /var/lib/pbench, before running yourbenchmark.

1.13.2 Kill tools

If you interrupt a built-in benchmark script (or your own script perhaps), the collection tools are not going to bestopped. If you don’t stop them explicitly, they can severely affect subsequent runs that you make. So it is stronglyrecommended that you invoke kill-tools before you start your run:

kill-tools --group=$group

1.13.3 Clear tools

This tool will delete the tools.$group file on the local host as well as on all the remote hosts specified therein. Afterdoing that, you will need to re-register all the tools that you want to use. In combination with clear-results, thistool creates a blank slate where you can start from scratch. You probably don’t want to call this much, but it may beuseful in certain isolated cases.

1.13.4 Register tools

Some tools have required options9 and you have to specify them when you register the tool. One example isthe blktrace tool which requires a --devices=/dev/sda,dev/sdb= argument. register-tool-setknows about such options for the default set of tools, but with other tools, you are on your own.

The trouble is that registration does not invoke the tool and does not know what options are required. So the best thingto do is invoke the tool with --help: the --help option may or may not be recognized by any particular tool, buteither way you should get a usage message that labels required options. You can then register the tool by using aninvocation similar to:

register-tool --name=blktrace -- --devices=/dev/sda,/dev/sdb

1.13.5 Using --dir

If you use the tool scripts explicitly, specify --dir=/var/lib/pbench/<run-id> so that all the data are col-lected in the specified directory. Also, save any data that your benchmark produces inside that directory: that way,move-results can move everything to the results warehouse.

Make the <run-id> as detailed as possible to disambiguate results. The built-in benchmark scripts use the followingform: <benchmark>_<config>_<ts>, e.g

9 Yes, I know: it’s an oxymoron.

1.13. Best practices 9


fio_bagl-16-4-ceph_2014.12.15T15.58.51

where the <config> part (bagl-16-4-ceph) comes from the --config option and can be as detailed as youwant to make it.

1.13.6 Using --remote

If you are running multihost benchmarks, we strongly encourage you to set up the tool collections using --remote.Choose a driver host (which might or might not participate in the tool data collection: in the first case, you registertools locally as well as remotely; in the second, you just register them remotely) and run everything from it. Duringthe data collection phase, everything will be pulled off the remotes and copied to the driver host, so it can be movedto the results repo as a single unit. Consider also using --label to label sets of hosts - see Using ‘‘–label‘‘ for moreinformation.

1.13.7 Using --label

When you register remotes, --label can be used to give a meaningful label to the results subdirectories that comefrom remote hosts. For example, use =–label=server” (or client, or vm, or capsule or whatever else is appropriate foryour use case).

1.14 Results handling

1.14.1 Accessing results on the web

This section describes how to get to your results using a web browser. It describes how move-results moves theresults from your local controller to a centralized location and what happens there. It also describes the --prefixoption to move-results (and copy-results) and a utility script, edit-prefix, that allows you to changehow the results are viewed.

N.B. This section applies to the pbench RPM version 0.31-108gf016ed6 and later. If you are using an earlier version,please upgrade at your earliest convenience.

Where to go to see results

The canonical place is

http://resultshost.example.com/results/

There are subdirectories there for each controller host (the host on which move-results was executed) and under-neath those, there are subdirectories for each pbench run.

The leaves of the hierarchy are actually symlinks that point to the corresponding results directory in the old, flatincoming/ hierarchy. Direct access to incoming/ is now deprecated (and will eventually go away).

The advantage is that the results/ hierarchy can be manipulated to change one’s view of the results10, while leavingthe incoming/ hierarchy intact, so that tools manipulating it can assume a fixed structure.

10 E.g. A performance engineer was NFS-mounting the incoming/ hierarchy, grouping his results under separate subdirectories for fio, iozoneand smallfile, and grouping them further under thematically created subdirectories (“baremetal results for this configuration”, “virtual host resultsunder that configuration” etc.), primarily because having them all in a single directory was slow, as well as confusing. There were two problemswith this approach which motivated the prefix approach described above. One was that the NFS export of the FUSE mount of the gluster volumethat houses the result is extremetly flakey. The other is that the incoming/ hierarchy is modified, which makes the writing of tools to extract dataharder: they have to figure out arbitrary changes, instead of being able to assume a fixed structure.


http://resultshost.example.com/results/


In the interim, a simple script is running once an hour creating any missing links from results/ to incoming/.It will be turned off eventually after everybody has upgraded to this or a later version of pbench.

move-results and its --prefix option

In order to make move-results more robust, it now packages up the results in a tarball, computes an MD5 check-sum, copies the tarball to an archive area, checks that the MD5 checksum is still correct and then deletes the resultsfrom one’s local host.

The tarball is unpacked into the incoming/ hierarchy by a cron script which runs every minute (so there might be ashort delay before the results are available), and plants a symlink to the results directory in the results/ hierarchy.

Using the --prefix= option to move-results affects where that symlink is planted (and that’s the only thing itaffects). For example, if your controller host is alphaville and the results name is fio__2015.03.30T13.33.15, normally move-resultswould unpack the tarball under incoming/alphaville/fio__2015.03.30T13.33.15 and plant a symlink pointing to that at results/alphaville/fio__2015.03.30T13.33.15. But if you wanted to group all your fio results under results/alphaville/fio, you could instead say

move-results --prefix=fio

which would plant the link at results/alphaville/fio/fio__2015.03.30T13.33.15 instead of plant-ing it at results/alphaville/fio__2015.03.30T13.33.15.

edit-prefix

What if you forget to use --prefix when calling move-prefix? Or you want to reorganize further, perhapspushing a set of results further down in the results/ hierarchy?

You can do that with edit-prefix. For example, continuing the example above, suppose you want to push a bunchof results from fio/ down another level, perhaps to group all the fio results on a particular platform together:

edit-prefix --prefix=fio/dl980 fio/fio__2015.03.30T13.33.15 ...

would do that. The arguments must exist in the appropriate place in the results hierarchy and the symlink at theleaf must point to an existing result in the incoming/ hierarchy. The links are then moved, using the new prefix, toa different place in the results/ hierarchy.

edit-prefix works similarly to move-results: it sends instruction to the centralized results repository whichare executed by a cron script running once a minute; so it may take a bit before the change takes effect.

1.14.2 Normalized directory structure

Andrew writes:

• All of the benchmark scripts use var/lib/pbench/$benchmark-$config-$date/$iteration/reference-result/tools-$tool_group

• This allows for 1-N iterations and 1-N samples per iteration. For example, user-benchmark usesvar/lib/pbench/user-benchmark-$config-$date/1/reference-result

• A self-explanatory example of the above mentioned hierarchical pattern is as follows:

fio__2015.01.15T19.45.10/ --> $benchmark-$config-$date1-read-4KiB --> $iteration

reference-result --> reference-result/

(continues on next page)

1.14. Results handling 11


(continued from previous page)

tools-default --> tools-$tool_group/cmdsiostatmpstatperfpidstatproc-interruptsproc-vmstatsarturbostat

reference-results This is calculated (based on standard deviation) as the best result from all the iterations,after the tests have ended. This is just a sym-link to one of the iterations, so as to make it easier for the user takea quick look at the results.

1.14.3 CSV

Postprocessing now produces CSV files of the results. Each row consists of a timestamp and a series of measures. Thefirst row is a header row with the labels.

The CSV files are directly used by the Javascript library that allows users to view graphs. The library runs in the clientbrowser and pulls the CSV file from the server. If that file is large, there might be a substantial delay in the renderingof the graphs. In certain cases, large files have caused browsers to explode. The only known method to avoid thatcurrently is to reduce the sampling frequency and therefore make the files smaller. This is unsatisfactory and we areworking to mitigate this problem

1.14.4 Results structure

Local results structure

Andrew writes:

To understand how data is arranged, you have to understand the different requirements users & bench-marks might have:

The simplest use case is when a user just wants to get tool data for a single measurement. For example, auser may run:

register-tool-setdir=/var/lib/pbench/myteststart-tools --dir=$dirmy-benchmark-script.shstop-tools --dir=$dirpostprocess-tools --dir=$dirmove-results

(the “my-benchmark-script.sh” above could be substituted by simply waiting until whatever thing is hap-pening is done, or a “sleep <x>”, etc)

The hierarchy is then pretty simple: /var/lib/pbench/my-test is the base directory for this test,and the tool data is in tools-$tool_group. Since they used the default tool group (they did not spec-ify an alternative), it’s “tools-default”. The base directory is where a user should put any data regardingthe workload (benchmark result). So, in general, when processing a test result, the benchmark data is



in ./mytest, and the tool data for this benchmark is in ./mytest/tools-$tool_group/. These two are alwaystightly coupled to ensure the tool data is always included in the benchmark result.

In the case above, the user has total control over the –dir name. The “tools-default” is a fixed name,which originates from “tools-$tool_group”. This should not change. “./<dir>/tools-*” should always berecognizable by other postprocessing scripts as the tools data for test <dir>. If a user wants to identify thisresult uniquely, the upper directory should be used, for example:

a first test:

dir=/var/lib/pbench/mytest-using-containersstart-tools --dir=$dirmy-benchmark-script.sh --use-dockerstop-tools --dir=$dirpostprocess-tools --dir=$dirmove-results

and then a second test:

dir=/var/lib/pbench/mytest-using-VMsstart-tools --dir=$dirmy-benchmark-script.sh --use-vmsstop-tools --dir=$dirpostprocess-tools --dir=$dirmove-results

When a user uses a built-in pbench benchmark, the directory hierarchy is maintained [and optionallyexpanded], but some of the directory names (or rather a portion of the name) is under the control of thepbench benchmark script. This is to maintain consistency across the pbench benchmark scripts. Thepbench benchmark scripts should include a date in the base directory name and include contents from the–config option.

Since many benchmarks actually have several measurements, an extra level of directory is addedto accommodate this. Instead of /var/lib/pbench/<mytest>/tools-default, we usually end up with/var/lib/pbench/<mytest>/<test-iteration[s]>/tools-default.

There are actually multiple reasons for the ./<test-iteration[s]>/ addition, as there are many reasons tohave more than one test execution for any given benchmark. These include (but are not limited to):

1. a benchmark simply has multiple different tests.

2. a pbench benchmark script often tries to execute several benchmark configurations, varying thingslike load levels & different benchmark options, so the user does not have script these themselves.

3. benchmarks may need multiple samples of the exact same benchmark configuration to computestandard deviation.

An example of (1) is SPECcpu, where there are several completely different tests, and they each shouldget their own result sub-directory (./<test-iteration-X/), with its own tools-$tool_group subdirectory. The“main” directory (/var/lib/pbench/<mytest>) includes the overall result, and generally where any reportgenerated would reside.

An example of (2) is uperf, where by default this script runs several configurations, varying message size,number of instances, and protocol type. This can produce dozens of different results, all of which needto be organized properly. Each unique configuration uses a unique ./<iteration>/ directory under the maindirectory, each with their own tools-$tool_group subdir.

An example of (3) is dbench, where by default 5 samples of the same test are taken, Each of these resultsare kept in a ./<iteration>/ subdir. After the end of the tests, the dbench script computes the standard

1.14. Results handling 13


deviation and even creates a symlink, “reference-result”, to the 1 iteration-dir that it’s result closest to thesdtdev.

More than one of these uses for iterations can also be used. In fact, uperf, uses iterations for both varyingbenchmark options (like message sizes), but for each of those unique configurations, multiple samples aretaken to compute a standard deviation. This then involves two levels of subdirs for the iterations. So, inthis case, we have a hierarchy like:

/var/lib/pbench/<my-test>/var/lib/pbench/<my-test>/1-tcp-stream-1024k-1instance//var/lib/pbench/<my-test>/1-tcp-stream-1024k-1instance/sample1/

So, in summary:

1. tool data is always in a subdir of where the benchmark result is kept. The tool subdir starts with“tools-“

2. A benchmark result dir can be as high up as var/lib/pbench/<mytest>, or it can be 1 or two levelsdeeper, depending on the need for multiple test runs. Some kind of benchmark summary shouldalways be in /var/lib/pbench/<mytest>.

I will cover remote tools in another comment section.

Remote results structure

When pbench tools are registered remotely, the structure described above is followed on each host

Post-processing collects all the remote results locally. The results from each remote host are pushed down one level inthe hierarchy, with the name of the host (augmented by the value of the --label option if applicable) providing theextra directory level at the top.

In addition, if local results are present, they are also pushed down one level in the hierarchy with the name of the localhost providing the extra directory level at the top (this happens in the purely local case as well, for uniformity’s sake).Again, the value of the --label option is used to augment the name if applicable.

1.15 Advanced topics

1.15.1 Triggers

Triggers are groups of tools that are started and stopped on specific events. They are registered withregister-tool-trigger using the --start-trigger and --stop-trigger options. The output of thebenchmark is piped into the tool-trigger tool which detects the conditions for starting and stopping the specifiedgroup of tools.

There are some commands specifically for triggers:

register-tool-trigger register start and stop triggers for a tool group.

list-triggers list triggers and their start/stop criteria.

tool-trigger this is a Perl script that looks for the start-trigger and end-trigger markers in the benchmark’s output,starting and stopping the appropriate group of tools when it finds the corresponding marker.

As an example, pbench_dbench uses three groups of tools: warmup, measurement and cleanup. It registers thesegroups as triggers using



register-tool-trigger --group=warmup --start-trigger="warmup" --stop-trigger="execute"register-tool-trigger --group=measurement --start-trigger="execute" --stop-trigger=→˓"cleanup"register-tool-trigger --group=cleanup --start-trigger="cleanup" --stop-trigger=→˓"Operation"

It then pipes the output of the benchmark into tool-trigger:

$benchmark_bin --machine-readable --directory=$dir --timelimit=$runtime--warmup=$warmup --loadfile $loadfile $client |tee $benchmark_results_dir/result.txt |tool-trigger "$iteration" "$benchmark_results_dir" no

tool-trigger will then start the warmup group when it encounters the string “warmup” in the benchmark’s outputand stop it when it encounters “execute”. It will also start the measurement group when it encounters “execute” andstop it when it encounters “cleanup” - and so on.

Obviously, the start/stop conditions will have to be chosen with some care to ensure correct actions.

1.16 FAQ

1.16.1 What does --name do?

This option is recognized by register-tool and unregister-tool: it specifies the name of the tool that is tobe (un)registered. list-tools with the --name option list all the groups that contain the named tool8.

1.16.2 What does --config do?

This option is recognized by the benchmark scripts (see Available benchmark scripts above) which use it as a tag forthe directory where the benchmark is going to run. The default value is empty. The run directory for the benchmark isconstructed this way:

${pbench_run}/${benchmark}_${config}_${date}

where $pbench_run and $date are set by the /opt/pbench-agent/base script and $benchmark is set tothe obvious value by the benchmark script; e.g. a fio run with config=foo would run in the directory /var/lib/pbench/fio_foo_2014.11.10T15.47.04.

1.16.3 What does --dir do?

This option is recognized by start-tools, stop-tools, tool-trigger and postprocess-tools. Itspecifies the directory where the tools are going to stash their data. The default value is /tmp. Each group thenuses it as a prefix for its own stash, which has the form $dir/tools-$group. Part of the stash is the set ofcmds to start and stop the tools - they are stored in $dir/tools-$group/cmds. The output of the tool is in$dir/tools-$group/$tool.txt.

This option has to be specified identically for each command when invoking these commands (actually, each of thecommands should be invoked with the identical set of all options, not just --dir.)

8 A list of available tools in a specific group can be obtained with the --group option of list-tools; unfortunately, there is no option tolist all available tools - the current workaround is to check the contents of /opt/pbench-agent/tool-scripts.

1.16. FAQ 15


If you use these tools explicitly (i.e. you don’t use one of the benchmark scripts), it is highly recommended that youspecify this option explicitly and not rely on the /tmp default. For one, you should make sure that different iterationsof your benchmark use a different value for this option, otherwise later results will overwrite earlier ones.

N.B. If you want to run move-results or copy-results after the end of the run, your resuls should be under/var/lib/pbench: move/copy-results does not know anything about your choice for this option; it onlylooks in /var/lib/pbench for results to upload. So if you are planning to use move/copy-results, makesure that the specified directory is a subdirectory of /var/lib/pbench.

1.16.4 What does --remote do?

pbench can register tools on remote hosts, start them and stop them remotely and gather up the results from the remotehosts for post-processing. The model is that one has a controller or orchestrator and a bunch of remote hosts thatparticipate in the benchmark run.

The pbench setup is as follows: register-tool-set or register-tool is called on the controller with the--remote option, once for each remote host:

for remote in $remotes ;doregister-tool-set --remote=$remote --label=foo --group=$group

done

That has two effects: it adds a stanza for the tool to the appropriate tools.$group file on the remote host and italso adds a stanza like this to the controller tools.$group file:

remote@<host>:<label>

The label is optionally specified with --label and is empty by default.

When start-tools is called on the controller, it starts the local collection (if any), but it also inter-prets the above stanzas and starts the appropriate tools on the remote hosts. Similarly for stop-tools andpostprocess-tools.

1.16.5 What does --label do?

TBD

1.16.6 How to add a collection tool

Tool scripts are mostly boilerplate: they need to take a standard set of commands (–install, –start, –stop, –postprocess)and a standard set of options (–iteration, –group, –dir, –interval, –options). Consequently, the easiest thing to do is totake an existing script and modify it slightly to call the tool of your choice. I describe here the case of turbostat.

There are some tools that timestamp each output stanza; there are others that do not. In the former case, make sure touse whatever option the tool requires to include such timestamps (e.g. vmstat -t on RHEL6 or RHEL7 - but strangelynot on Fedora 20 - will produce such timestamps).

There are some tools that are included in the default installation - others need to be installed separately. Turbostatis not always installed by default, so the tool script installs the package (which is named differently on RHEL6 andRHEL7) if necessary. In some cases (e.g. the sysstat tools), we provide an RPM in the pbench repo and the tool scriptmakes sure to install that if necessary.

The only other knowledge required is where the tool executable resides (usually /usr/bin/<tool> or /usr/local/bin/<tool>- /usr/bin/turbostat in this case) and the default options to pass to the tool (which can be modified by passing –optionsto the tool script).



So here are the non-boilerplate portions of the turbostat tool script. The first interesting part is to set tool_bin topoint to the binary:

# Defaultstool=$script_nametool_bin=/usr/bin/$tool

This only works if the script is named the same as the tool, which is encouraged. If the installed location of your toolis not /usr/bin, then adjust accordingly.

Since turbostat does not provide a timestamp option, we define a datalog script to add timestamps (no need for that forvmstat e.g.) and use that as the tool command:

case "$script_name" inturbostat)tool_cmd="$script_path/datalog/$tool-datalog $interval $tool_output_file";;

esac

The datalog script uses the log-timestamp pbench utility to timestamp the output. It will then be up to thepostprocessing script to tease out the data appropriately.

The last interesting part dispatches on the command - the install is turbostat-specific, but the rest is boilerplate:--start just executes the tool_cmd as defined above and stashes away the pid, so that --stop can kill thecommand later; --postprocess calls the separate post-processing script (see below):

release=$(awk '{x=$7; split(x, a, "."); print a[1];}' /etc/redhat-release)case $release in

6)pkg=cpupowerutils;;

7)pkg=kernel-tools;;

*)# better be installed already;;

esac

case "$mode" ininstall)if [ ! -e $tool_bin ]; then

yum install $pkgecho $script_name is installed

elseecho $script_name is installed

fistart)mkdir -p $tool_output_direcho "$tool_cmd" >$tool_cmd_filedebug_log "$script_name: running $tool_cmd"$tool_cmd >>"$tool_output_file" & echo $! >$tool_pid_filewait;;stop)pid=`cat "$tool_pid_file"`debug_log "stopping $script_name"kill $pid && /bin/rm "$tool_pid_file"


1.16. FAQ 17

https://github.com/distributed-system-analysis/pbench/tree/tool-scripts/turbostat

https://github.com/distributed-system-analysis/pbench/tree/tool-scripts/datalog/turbostat-datalog



;;postprocess)debug_log "postprocessing $script_name"$script_path/postprocess/$script_name-postprocess $tool_output_dir;;

esac

Finally, there is the post-processing tool: the simplest thing to do is nothing. That’s currently the case for the turbostatpost-processing tool, but ideally it should produce a JSON file with the data points and an HTML file that uses thenv3 library to plot the data graphically in a browser. See the postprocess directory for examples, e.g. the iostatpostprocessing tool.

1.16.7 How to add a benchmark

TBD

1.16.8 How do I collect data for a short time while my benchmark is running?

Running

user_benchmark -- sleep 60

will start whatever data collections are specified in the default tool group, then sleep for 60 seconds. At the end of thatperiod, it will stop the running collections tools and postprocess the collected data. Running move-results afterwardswill move the results to the results server as usual.

1.16.9 I have a script to run my benchmark - how do I use it with pbench?

pbench is a set of building blocks, so it allows you to use it in many different ways, but it also makes certain assump-tions which if not satisfied, lead to problems.

Let’s assume that you want to run a number of iozone experiments, each with different parameters. Your scriptprobably contains a loop, running one experiment each time around. If you can change your script so that it executesone experiment specified by an argument, then the best way is to use the user-benchmark script:

register-tool-setfor exp in experiment1 experiment2 experiment3 ;do

user-benchmark --config $exp -- my-script.sh $expdonemove-results

The results are going to end up in directories named /var/lib/pbench/user-benchmark_$exp_$ts foreach experiment (unfortunately, the timestamp will be recalculated at the beginning of each user-benchmarkinvocation), before being uploaded to the results server.

Alternatively, you can modify your script so that each experiment is wrapped with start/stop/postprocess-tools andthen call move-results at the end:

register-tool-setfor exp in experiment1 experiment2 experiment3 ;do

start-tools<run the experiment>



https://github.com/distributed-system-analysis/pbench/tree/tool-scripts/postprocess/turbostat-postprocess

https://github.com/distributed-system-analysis/pbench/tree/tool-scripts/postprocess

https://github.com/distributed-system-analysis/pbench/tree/tool-scripts/postprocess/iostat-postprocess

https://github.com/distributed-system-analysis/pbench/tree/tool-scripts/postprocess/iostat-postprocess



stop-toolspostprocess-tools

donemove-results

1.17 Footnotes

1.17. Footnotes 19

CHAPTER 2

Configuration

2.1 Getting Started

By default, pbench benchmarks collect configuration data of a system, stored in the sysinfo top level directory of apbench result, collected at the beginning and end of a benchmark (with one exception, pbench-user-benchmark onlycollects at the end).

The structure of the sysinfo directory in a pbench result, for example, /var/lib/pbench-agent/pbench_userbenchmark_example_2019.07.18T12.00.00/ is given below followed by a brief expla-nation of each different type of configuration data.

• sysinfo

• end

– hostname

– block-params.log

– config-5.0.17-300.fc30.x86_64

– libvirt/

– lstopo.txt

– security-mitigation-data.txt

– sosreport-localhost-localhost-pbench-2019-06-10-mrqgzbh.tar.xz

– sosreport-localhost-localhost-pbench-2019-06-10-mrqgzbh.tar.xz.md5

– stockpile.json

– stockpile.log

2.1.1 config-[kernel_version]

The file contains kernel configuration data.

21


The data is collected using pbench-sysinfo-dump#L38. The script uses uname system utility (systemcall is a termused for all the APIs provided by the kernel) to collect kernel release information and then checks if a correspondingkernel configuration file exists on the system. If it does, the script simply copies the file, located in /boot directory,to the sysinfo directory.

The file contains data in a key value format where the key is a metric name and the value can be a string literal or anumber. The keys and the values are separated by an equality sign.

2.1.2 security-mitigation-data.txt

The file contains CPU vulnerabilities data and RHEL-specific flag settings.

The data is collected using pbench-sysinfo-dump#L44. The script checks if /sys/devices/system/cpu/vulnerabilities directory exists. If it does, the script prints the filenames and the contents of all the fileslocated in the directory. After that, it repeats the same steps for the /sys/kernel/debug/x86 directory.

The file contains data in a key value format where the key is a file name and the value is the content of the file.

2.1.3 libvirt/

The directory provides information about libvirt, an open-source API, daemon and management tool for managingplatform virtualization.

The data is collected using pbench-sysinfo-dump#L60. The script copies libvirt files located at /var/log/libvirtand /etc/libvirt directories to the sysinfo/libvirt/log and sysinfo/libvirt/etc directories re-spectively. Only the files whose name follows the regex *.log are copied from the /var/log/libvirt directory.

2.1.4 lstopo.txt

The file provides information about the topology of the system.

The data is collected using pbench-sysinfo-dump#L71. The script executes the command lstopo --of txt anddumps its output into a text file only if /usr/bin/lstopo file exists on the system.

2.1.5 block-params.log

The file provides information about block devices.

The data is collected using pbench-sysinfo-dump#L79. The script loops over all the files which satisfy the regex:/sys/block/[s,h,v]d\*[a-z]/ and prints each file name along with the contents of the file.

The file contains data in a key value format where the key is the file name and the value is the content of the file..

2.1.6 sosreport tarball (e.g. sosreport-localhost-localhost-pbench-2019-05-29-rtvzlke.tar.xz)

The tarball contains system configuration and diagnostic information collected by invoking the sosreport com-mand.

The data is collected using pbench-sysinfo-dump#L87. The script uses sosreport command with different pluginsto get the required system information. The resulting tarball contains a number of files copied from the system as wellas the output of several commands executed on the system.

22 Chapter 2. Configuration

https://github.com/distributed-system-analysis/pbench/blob/master/agent/util-scripts/pbench-sysinfo-dump#L38







2.1.7 ara

This specific file is not in the scope of my internship because ara works with python2 and Fedora 30, which isinstalled on my system, works with python3 only.

2.1.8 stockpile.json

The file contains system information gathered by the stockpile tool using Ansible.

The data is collected using pbench-sysinfo-dump#L153. The script runs a specified stockpile playbook with the givenstockpile options. The stockpile playbook has a number of roles associated with it, for example, ceph, cpu, etc. foreach of which there is a specific ansible playbook called main.yml, which contains the rules to collect informationrelated to that role.

The file contains data in a json file format.

2.1.9 insights tarball

The tarball contains system information gathered by the insights-client.

The data is collected using pbench-sysinfo-dump#L189. The script uses insights-client command with differ-ent options to get the system information. The resulting tarball contains a number of files copied from the system aswell as the output of several commands executed on the system.

2.1. Getting Started 23

https://github.com/redhat-performance/stockpile


https://github.com/RedHatInsights/insights-client



24 Chapter 2. Configuration

CHAPTER 3

Getting started

3.1 Getting Started

The following is an introduction on how to use the pbench agent.

Pbench can be used to either automate tool execution and postprocessing for you, or also run any of its built-inbenchmark scripts. This first test will run the fio benchmark.

3.1.1 Installation

If you have not done so, install pbench-agent (via RPM or other Linux distribution supported method, documentedin INSTALL file).

After pbench-agent is installed, verify that your path includes:

/opt/pbench-agent:/opt/pbench-agent/util-scripts:/opt/pbench-agent/bench-scripts

If you do not have this, you may need to source your .bashrc, re-log in, or just run, . /opt/pbench-agent/profile to have the path updated.

3.1.2 Tool Registration

After you are certain the path is updated, register the default set of tools:

register-tool-set

This command will register the default tool set, which consists of sar, mpstat, iostat, pidstat,proc-vmstat, proc-interrupts, and perf.

When registering these tools, pbench-agent checks if they are installed and may install some of them if they arenot present. Some of these tools are built from source, so you may see output from fetching the source and compiling.Following any installation, you should have this output:

25


sar tool is now registered in group default[debug]tool_opts: default --interval="3"[debug]checking to see if tool is installed...iostat is installediostat tool is now registered in group default[debug]tool_opts: default --interval="3"[debug]checking to see if tool is installed...mpstat is installedmpstat tool is now registered in group default[debug]tool_opts: default --interval="3"[debug]checking to see if tool is installed...pidstat is installedpidstat tool is now registered in group default[debug]tool_opts: default --interval="3"[debug]checking to see if tool is installed...proc-vmstat tool is now registered in group default[debug]tool_opts: default --interval="3"[debug]checking to see if tool is installed...proc-interrupts tool is now registered in group default[debug]tool_opts: default --record-opts="record -a --freq=100"[debug]checking to see if tool is installed...perf tool is now registered in group default

If at any time you are unsure which tools are registered, you can run:

# list-toolsdefault: perf,proc-interrupts,proc-vmstat,pidstat,mpstat,iostat,sar

The output above shows which tools are in the “default” tool group. And by specifying the --with-optionsswitch, you get the options used for these tools:

# list-tools --with-optionsdefault: perf --record-opts="record -a --freq=100",proc-interrupts --interval="3",proc-vmstat --interval="3",pidstat --interval="3",mpstat --interval="3",iostat --→˓interval="3",sar --interval="3"

In the above example, the --interval option is set for all tools but perf. Optioonally, you can change theseindividually with the register-tool command:

# register-tool --name=pidstat -- --interval=10[debug]tool_opts: --interval="10"[debug]checking to see if tool is installed...pidstat is installedpidstat tool is now registered in group default

Then run list-tools --with-options again to confirm:

# list-tools --with-optionsdefault: pidstat --interval="10",perf --record-opts="record -a --freq=100",proc-interrupts --interval="3",proc-vmstat --interval="3",mpstat --interval="3",→˓iostat --interval="3",sar --interval="3"

And the interval for pidstat is now 10.

26 Chapter 3. Getting started


3.1.3 Running a Benchmark

OK, now that the tools are registered, it’s time the run the benchmark. We’ll use the fio benchmark for this exmaple.To run, simply type ‘pbench_fio’, the wrapper script pbench-agent provides for the fio benchmark.

If this is the first time running fio via the pbench-agent, pbench-agent will attempt to download and compilefio. You may see quite a bit of output from this. Once fio is installed, pbench-agent will run several tests bydefault. Output for each will look something like this:

about to run fio read with 4 block size on /tmp/fio--------fio will use this job file:--------[global]bs=4kioengine=libaioiodepth=32direct=1time_based=1runtime=30[job1]rw=readfilename=/tmp/fiosize=896M-------------------------------------------

Right before the pbench_fio script starts a fio job, it will call start-tools, which will produce output likethis:

[debug][start-tools]/opt/pbench-agent/tool-scripts/sar --start --iteration=1-read-→˓4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/→˓tools-default default --interval="3"[debug][start-tools]/opt/pbench-agent/tool-scripts/iostat --start --iteration=1-read-→˓4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/→˓tools-default default --interval="3"[debug][start-tools]/opt/pbench-agent/tool-scripts/mpstat --start --iteration=1-read-→˓4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/→˓tools-default default --interval="3"[debug][start-tools]/opt/pbench-agent/tool-scripts/pidstat --start --iteration=1-read-→˓4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/→˓tools-default default --interval="3"[debug][start-tools]/opt/pbench-agent/tool-scripts/proc-vmstat --start --iteration=1-→˓read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-→˓4KiB/tools-default default --interval="3"[debug][start-tools]/opt/pbench-agent/tool-scripts/proc-interrupts --start --→˓iteration=1-read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_→˓12:54:42/1-read-4KiB/tools-default default --interval="3"[debug][start-tools]/opt/pbench-agent/tool-scripts/perf --start --iteration=1-read-→˓4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/→˓tools-default default --record-opts="record -a --freq=100"

That is output from start-tools starting all of the tools that were registered.

Next is the output from the actual fio job:

fio: Going to run [/usr/local/bin/fio /var/lib/pbench/fio__2014-09-11_12:54:42/1-read-→˓4KiB/fio.job]job1: (g=0): rw=read, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=32fio-2.1.7Starting 1 process





job1: Laying out IO file(s) (1 file(s) / 896MB)

job1: (groupid=0, jobs=1): err= 0: pid=12961: Thu Sep 11 12:55:47 2014read : io=1967.4MB, bw=67147KB/s, iops=16786, runt= 30003msecslat (usec): min=3, max=77, avg= 7.95, stdev= 2.45clat (msec): min=1, max=192, avg= 1.90, stdev= 1.48lat (msec): min=1, max=192, avg= 1.90, stdev= 1.48

clat percentiles (usec):| 1.00th=[ 1736], 5.00th=[ 1736], 10.00th=[ 1752], 20.00th=[ 1752],| 30.00th=[ 1768], 40.00th=[ 1768], 50.00th=[ 1768], 60.00th=[ 1912],| 70.00th=[ 1912], 80.00th=[ 2064], 90.00th=[ 2096], 95.00th=[ 2224],| 99.00th=[ 2256], 99.50th=[ 2256], 99.90th=[10304], 99.95th=[10816],| 99.99th=[44800]

bw (KB /s): min=34373, max=70176, per=100.00%, avg=67211.32, stdev=5212.44lat (msec) : 2=78.09%, 4=21.73%, 10=0.05%, 20=0.10%, 50=0.01%lat (msec) : 100=0.01%, 250=0.01%

cpu : usr=5.97%, sys=22.23%, ctx=501089, majf=0, minf=332IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=100.0%, >=64=0.0%

submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.1%, 64=0.0%, >=64=0.0%issued : total=r=503651/w=0/d=0, short=r=0/w=0/d=0latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):READ: io=1967.4MB, aggrb=67146KB/s, minb=67146KB/s, maxb=67146KB/s, mint=30003msec,

→˓ maxt=30003msec

Disk stats (read/write):dm-1: ios=501328/154, merge=0/0, ticks=947625/12780, in_queue=960429, util=99.53%,

→˓ aggrios=503626/101, aggrmerge=25/55, aggrticks=949096/9541, aggrin_queue=958491,→˓aggrutil=99.49%sda: ios=503626/101, merge=25/55, ticks=949096/9541, in_queue=958491, util=99.49%

Now that this fio job is complete, the pbench_fio script calls stop-tools:

[debug][stop-tools]/opt/pbench-agent/tool-scripts/sar --stop --iteration=1-read-4KiB -→˓-group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/tools-→˓default default --interval="3"[debug][stop-tools]/opt/pbench-agent/tool-scripts/iostat --stop --iteration=1-read-→˓4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/→˓tools-default default --interval="3"[debug]stopping sar[debug][stop-tools]/opt/pbench-agent/tool-scripts/mpstat --stop --iteration=1-read-→˓4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/→˓tools-default default --interval="3"[debug]stopping iostat[debug][stop-tools]/opt/pbench-agent/tool-scripts/pidstat --stop --iteration=1-read-→˓4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/→˓tools-default default --interval="3"[debug]stopping mpstat[debug][stop-tools]/opt/pbench-agent/tool-scripts/proc-vmstat --stop --iteration=1-→˓read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-→˓4KiB/tools-default default --interval="3"[debug]stopping pidstat[debug][stop-tools]/opt/pbench-agent/tool-scripts/proc-interrupts --stop --→˓iteration=1-read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_→˓12:54:42/1-read-4KiB/tools-default default --interval="3"





[debug]stopping proc-vmstat[debug][stop-tools]/opt/pbench-agent/tool-scripts/perf --stop --iteration=1-read-4KiB→˓--group=default --dir=/var/lib/pbench/fio__2014-09-11_12:54:42/1-read-4KiB/tools-→˓default default --record-opts="record -a --freq=100"[debug]stopping proc-interruptswaiting for PID 12934 (perf) to finish

Next, pbench_fio calls postprocess-tools. This is what generates the .csv files and renders the .htmlfile containing the NVD3 graphs for the tool data.

collecting /proccollecting /sys[debug][postprocess-tools]/opt/pbench-agent/tool-scripts/sar --postprocess --→˓iteration=1-read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_→˓12:54:42/1-read-4KiB/tools-default default --interval="3"[debug][postprocess-tools]/opt/pbench-agent/tool-scripts/iostat --postprocess --→˓iteration=1-read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_→˓12:54:42/1-read-4KiB/tools-default default --interval="3"[debug][postprocess-tools]/opt/pbench-agent/tool-scripts/mpstat --postprocess --→˓iteration=1-read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_→˓12:54:42/1-read-4KiB/tools-default default --interval="3"[debug]postprocessing iostat[debug][postprocess-tools]/opt/pbench-agent/tool-scripts/pidstat --postprocess --→˓iteration=1-read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_→˓12:54:42/1-read-4KiB/tools-default default --interval="3"[debug]postprocessing mpstat[debug][postprocess-tools]/opt/pbench-agent/tool-scripts/proc-vmstat --postprocess --→˓iteration=1-read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_→˓12:54:42/1-read-4KiB/tools-default default --interval="3"[debug]postprocessing pidstat[debug][postprocess-tools]/opt/pbench-agent/tool-scripts/proc-interrupts --→˓postprocess --iteration=1-read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-→˓09-11_12:54:42/1-read-4KiB/tools-default default --interval="3"[debug]postprocessing proc-vmstat[debug][postprocess-tools]/opt/pbench-agent/tool-scripts/perf --postprocess --→˓iteration=1-read-4KiB --group=default --dir=/var/lib/pbench/fio__2014-09-11_→˓12:54:42/1-read-4KiB/tools-default default --record-opts="record -a --freq=100"[debug]postprocessing proc-interrupts

This will repeat for a total of 6 different fio jobs, then the fio benchmark will be complete. Now that the job iscomplete, we want to move the results to the archive host. The results are currently in /var/lib/pbench/fio-. To movethese results, simply run:

# move-results

Once that command completes, the data should be moved to the configured archive host. To view your results, usea link like this in your browser (replacing the “resultshost.example.com” with your pbench deployed web server,and replacing the “your-HOSTNAME” with the $(hostname -s) of the machine where you issued the “move-results”above):

http://resultshost.example.com/results//?C=M;O=D

Towards the top of the list, there should be a directory like “fio__2014-09-11_12:54:42”. That is your pbenchfio job. Click on that directory to see the results.

There should be a file, fio-summary.txt, which will contain the results for all of the fio jobs that were run.

In this same directory, there should be more sub-directories, one for each fio job. They should have a format like


http://resultshost.example.com/results//?C=M;O=D


“N-[read|write]-MKiB”. In pbench-speak, these are called an “iteration” and usually start with “1-“. Undereach of these you will find the details of that job/iteration:

• fio.cmd: the actual fio command used

• fio.job: the job file pbench_fio created

• result.txt: the output from the fio job

• tool-default: all of the collected tool data

• sysinfo: data pbench_fio collected from /sys & /proc

Under the tools-default directory, there should be text output for each tool as well as .html files, and a csvsub-directory containing all of the raw tool data.


CHAPTER 4

Installation

NOTE: to install pbench-agent, use the published instructions at, http://distributed-system-analysis.github.io/pbench/pbench-agent.html#sec-5

31

http://distributed-system-analysis.github.io/pbench/pbench-agent.html#sec-5

http://distributed-system-analysis.github.io/pbench/pbench-agent.html#sec-5


32 Chapter 4. Installation


5.1.2 Multiple hosts

One registers tools on each remote by hand. Assuming that you are sitting on the host that acts as the orchestrator forthe benchmark run (where one might or might not run tools - in the following I’m assuming that tools are run on theorchestrator, but skipping the local register-tool-set would only run tools on the remotes), then one mightproceed as follows:

register-tool-setfor remote in $remotes ;do

register-tool-set --remote $remotedone

for iter in $iterations ;dots=$(date +....)dir=/var/lib/pbench/...

start-tools --dir=$dir

<run the benchmark>

stop-tools --dir= $dirpostprocess-tools --dir=$dir

done

where the run directory may be more than a single level below /var/lib/pbench and may have various compo-nents specified through –config, the “iteration” name (which may also be thought of as the name of the experiment- that makes more sense in some cases), a timestamp and anything else that one might think of to disambiguate thisexperiment from the next one.

The results structure on the orchestrator and on each remote is exactly the same as in the single host case. But in thiscase, postprocess-tools pulls the remote data and creates subdirs for each remote under the sysinfo branch ofthe local hosts tree and also under the tools-$group branch. That way, each remote’s results are spread over varioussubdirs of the local host and the local host is treated specially. It might make more sense to have a structure like thisinstead (exp1 == iter1 etc. if you prefer to think of them as iterations of a single experiment, rather than as separateexperiments):

benchmark__TS/|-- benchmark.txt|-- exp1| |-- exp1.txt| |-- host1| | |-- bench.cmd| | |-- bench.job| | |-- result.txt| | |-- sysinfo| | `-- tools-default| |-- host2| `-- host3|-- exp2| |-- exp2.txt| |-- host1| | |-- bench.cmd| | |-- bench.job| | |-- result.txt| | |-- sysinfo| | `-- tools-default| |-- host2


5.1. Structure of results 35



| `-- host3`-- exp3

|-- exp3.txt|-- host1| |-- bench.cmd| |-- bench.job| |-- result.txt| |-- sysinfo| `-- tools-default|-- host2`-- host3

18 directories, 13 files

where the hosts are treated symmetrically. Any benchmark data that are gathered on each host (remote or local)are under the subdirectory for that host. Benchmark data that are “global” in some sense are under the “expN”subdirectory, and there might be a summary describing the set of experiments in the top-level directory.

I think this structure accommodates Archit’s and Peter’s concerns and is fairly easy to implement: it only requiressimple changes to postprocess-tools. A thornier problem is the already existing base of results, but it could be fixed uponce and for all with a bit of scripting (although that remains to be proved).

The question is whether it imposes artificial limits that are going to get in our way later, but I cannot think of any(although that may be a lack of imagination on my part).

36 Chapter 5. Structure of results

CHAPTER 6

Travis CI

6.1 Running unit tests using the travis-ci infrastructure at github.

The unit tests require bash, perl (for the postprocessing scripts) and python for the indexing unit tests.

The travis-ci hosts are set up with Ubunty 12.04 LTS, with some additions. That works fine for bash and perl - the.travis.yml file was taking care of different perl versions like this:

language: perlperl:- "5.20"- "5.18"- "5.16"- "5.10"

install:- sudo apt-get install python python-pip bc libjson-perl realpath- sudo pip install configtools

script:- ./agent/bench-scripts/unittests- ./agent/tool-scripts/postprocess/unittests- ./bgtasks/pbench/bin/unittests

That causes travis-ci to run the same unit tests four times, once for each version of perl, but that accommodates all theperl versions running on RHEL6/7 or recent Fedora versions.

6.2 Python dependencies

The indexing tests run the index-pbench script which requires python >= 3.3. The main reason for that is that thetarfile.py module does not handle xz-compressed tarballs before that version. There is also a minor dependencyon FileNotFoundError which was introduced in python 3.3, but that can be easily worked around. The tarfile.pydependency can also be worked around of course, at the price of introducing a compatibility layer (something whichwe did for other projects), but it seems wasteful to do that in this case and in the future. We hope that at some point

37


the travis-ci infrastructure will use a more modern distro, one with a built-in (or easily available) python >= 3.3 (at thetime of writing, this looks like python 3.5).

For the time being, we go through a bunch of hoops and install python 3.4 on Ubuntu 12.04 LTS:

language: perlperl:- "5.20"- "5.18"- "5.16"- "5.10"

install:- sudo apt-get install python python-pip bc libjson-perl realpath- sudo pip install configtools elasticsearch- sudo apt-get install python-software-properties- sudo add-apt-repository ppa:fkrull/deadsnakes -y- sudo apt-get update- sudo apt-get install python3.4- sudo apt-get install python3-setuptools- sudo easy_install3 pip- sudo pip3 install configtools elasticsearch- sudo cp -r /usr/local/lib/python3.2/dist-packages/* /usr/local/lib/python3.4/dist-→˓packages- sudo ln -sf python3.4 /usr/bin/python3

script:- ./agent/bench-scripts/unittests- ./agent/tool-scripts/postprocess/unittests- ./bgtasks/pbench/bin/unittests

python-software-properties provides the add-apt-repository command that is used to add the fkrull/deadsnakes ppa.

There are no official repos for python > 3.2 for Ubuntu 12.04. The best I could find was the fkrull/deadsnakes PPAwhich provides unofficial packages of more recent python versions.

To install configtools and elasticsearch in the python3.4 package directory, we have to install pip3, using easy_install3,which in turn is installed using python3-setuptools. The packages are installed under python3.2 however, so we copythem to the python3.4 directory by hand. We also symlink /usr/bin/python3 to the python3.4 version.

With these additions, python3.4 is installed and the indexing unit tests are now passing.

38 Chapter 6. Travis CI

CHAPTER 7

Contributing

7.1 Setup

To create a development environment, one just clones what they need

7.1.1 Getting the code

Grab the code from git

$ git clone https://github.com/distributed-system-analysis/pbench $ cd pbench

7.1.2 Run unit tests

The first step is to install the necessary dependencies that are required to run on the system:

sudo yum install -y perl-JSON sudo yum install -y python3-pip sudo pip3 install tox

Once tox is installed you can run the unittests:

tox -e agent (for pbench agent) tox -e server (for pbench server) tox -e pytest_agent (for pytest pbench agent) tox -epytest_server (for pytest pbench server)

7.2 Python formatting

This project uses the flake8 method of code style enforcement, linting and checking, flake8

All python code contributed to pbench must match the style requirements. These requirements are enforced by [pre-commit] (https://pre-commit.com).

39

https://github.com/distributed-system-analysis/pbench

http://flake8.pycqa.org/en/latest

https://pre-commit.com


7.2.1 Use pre-commit to set automatic commit requirements

This project makes use of pre-commit to do automatic lint and style checking on every commit containing Pythonfiles.

To install the pre-commit hook, run the executable from your Python 3 framework while in your current pbench gitcheckout:

$ cd ~/pbench $ pip3 install pre-commit $ pre-commit install –install-hooks

Once installed, all commits will run the test hooks. If your commit fails any of the tests, the commit will be rejected.

40 Chapter 7. Contributing

https://pre-commit.com/

pbench-docs Documentation

Documents

Transcript of pbench-docs Documentation