LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance...
Transcript of LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance...
![Page 1: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/1.jpg)
LKP-testsLinux kernel performance test tool
![Page 2: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/2.jpg)
What is LKP-tests● Framework to run benchmarks
○ Fetch/install the benchmarks○ Prepare test environment
■ For example, format disk○ Run benchmark with various parameters
■ Collect performance statistics○ Benchmark/performance statistics parsing
● Framework for performance analysis○ Result compare
● Used for benchmark running and reproducing regression captured by 0-Day
![Page 3: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/3.jpg)
Highlights● Rich benchmarks support● Capture performance statistics (monitors) in addition to benchmark score
○ For example, perf stat, perf profile, vmstat, etc.
● Compare/analyze all performance statistics data (benchmark and monitors)○ Unified data model
![Page 4: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/4.jpg)
Benchmarks● Integrated ~50 benchmarks
○ Fetch/install benchmarks○ Run benchmarks with various parameters○ Parse result of benchmarks
● Mostly micro-benchmarks, several macro-benchmarks● Mostly server workload
![Page 5: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/5.jpg)
Benchmark List - 1● Scheduler
○ Hackbench, perf-bench-sched-pipe, etc.
● File system/IO○ Fio, fsmark, iozone, dbench, dd, fileio, postmark, etc.
● Network○ Netperf, iperf, apachebench, ku-latency, nepim, netpipe, nuttcp, qperf, siege, sockperf, stutter,
tbench, thrulay, etc.
● Scalability○ Reaim, aim7, will-it-scale, etc.
● Memory management○ vm-scalability, chromeswap, exit_free, pft, perf-bench-numa-mem, pmbench, swapin, tlbflush,
etc.
![Page 6: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/6.jpg)
Benchmark List - 2● Database
○ Pgbench, oltp, etc.
● Noise○ Ftq, fwq, etc.
● HPC○ Hpcc, linpack
● Workload emulation○ Blogbench, ebizzy (web application server), kbuild, pbzip2, pigz, pixz, plzip, pxz, etc.
● Others○ Tcrypt, unixbench
![Page 7: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/7.jpg)
Functionality Test● Memory Management
○ Bust_shm_exit, libhugetlbfs, ndctl, nvml
● File system○ Xfstests, ext4-frags, ocfs2test
● Virtualization○ Kvm-unit-tests
● Network○ packetdrill
● Others○ Cpu-hotplug, ftrace_onoff, hwsim(wifi), kernel_selftests, locktorture, ltp, mce-log, mce-test,
perf_event_tests, perf-sanity-tests, pm-qa, rcutorture, stress-ng, suspend, test_bpf, trinity, piglit, etc.
![Page 8: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/8.jpg)
Test Coverage
![Page 9: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/9.jpg)
Monitors● Capture performance statistics when benchmark running
○ Important for performance analysis, regression root causing
● About 38 monitors now● Examples:
○ Perf profile: find the hot spot○ Vmstat: CPU usage, IO bandwidth, memory usage○ Iostat: IO bandwidth, requests○ Latencytop: measure latency○ Turbostat: idle, frequency, and power of the CPU○ /proc files: meminfo, sched-debug, interrupts, etc.
![Page 10: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/10.jpg)
Setups● Scripts to prepare the test environment● About 20 setup scripts● Example setup scripts
○ disk: wait/check test disk to be ready○ fs: format disk with file system, mount it○ cpufreq_governor: CPU frequency governor○ iosched: choose IO scheduler○ dirty_thresh: writeback dirty threshold
![Page 11: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/11.jpg)
Jobs● Define the performance tests● Benchmark + monitors + setups
○ Each with parameters
● In yaml format○ Key -> value with array
● Combination of various test parameters○ One jobfile can define a set of tests○ For example: ext4+performance, ext4+powersave, xfs+performance, xfs+powersave
fs: - ext4 - xfscpufreq_governor - performance - powersave
![Page 12: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/12.jpg)
Job file example# swap-test.yamltestcase: vm-scalabilityswap:vmstat:perf-profile: delay: 20# ...vm-scalability test: swap-w-seq nr_task: - 1 - 2 - 4 - 8
![Page 13: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/13.jpg)
Supported distribution● Debian sid● Ubuntu 14.04● Centos 7● Archlinux
![Page 14: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/14.jpg)
Getting started$ git clone https://github.com/01org/lkp-tests $ cd lkp-tests$ make install$ lkp help
![Page 15: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/15.jpg)
Prepare host description● lkp-tests/hosts/<hostname>
○ Yaml format○ Setup disks for test
● Example
memory: 128Gssd_partitions: /dev/sda1
![Page 16: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/16.jpg)
Install packages for a job# browse and select a job you want to run, for example, job/hackbench.yaml$ ls lkp-tests/jobs$ lkp install $LKP_SRC/jobs/hackbench.yaml
![Page 17: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/17.jpg)
Split job# Generate atomic job from one job file$ lkp split-job lkp-tests/jobs/hackbench.yamljobs/hackbench.yaml => ./hackbench-1600%-process-pipe.yamljobs/hackbench.yaml => ./hackbench-1600%-process-socket.yamljobs/hackbench.yaml => ./hackbench-1600%-threads-pipe.yamljobs/hackbench.yaml => ./hackbench-1600%-threads-socket.yamljobs/hackbench.yaml => ./hackbench-50%-process-pipe.yamljobs/hackbench.yaml => ./hackbench-50%-process-socket.yamljobs/hackbench.yaml => ./hackbench-50%-threads-pipe.yamljobs/hackbench.yaml => ./hackbench-50%-threads-socket.yaml
![Page 18: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/18.jpg)
Run atomic job$ lkp run hackbench-50%-process-pipe.yaml...2016-11-18 07:55:12 /usr/bin/hackbench -g 1 --process --pipe -l 60000Running in process mode with 1 groups using 40 file descriptors each (== 40 tasks)Each sender will pass 60000 messages of 100 bytesTime: 7.388...
![Page 19: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/19.jpg)
Check result and analyze# link to result directory of latest run is created automatically$ ls result...$ lkp result hackbench/result/hackbench/50%-process-pipe-performance/lkp-kvm/debian/x86_64-rhel-7.2/gcc-6/4.5.0-2-amd64/1//result/hackbench/50%-process-pipe-performance/lkp-kvm/debian/x86_64-rhel-7.2/gcc-6/4.8.0-1-amd64/0/$ lkp ncompare -s commit=<kernel1> -o commit=<kernel2>
![Page 20: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/20.jpg)
Linux kernel source● Optional● Put it in /c/repo/linux or $GIT_ROOT_DIR/linux
![Page 21: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/21.jpg)
Run user supplied workload● Use “mytest” test
![Page 22: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/22.jpg)
Gaps● Bug fixes● Feedback from more users
![Page 23: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/23.jpg)
References● Git repo: https://github.com/01org/lkp-tests● Documents in source code: find lkp-tests -name README.md● Intel internal mailing list: [email protected]● External mailing list: [email protected]
![Page 24: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/24.jpg)
Thanks!
![Page 25: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/25.jpg)
Backup
![Page 26: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/26.jpg)
Cluster: Multiple machine test● One server and multiple clients● Steps
○ Queue job for server○ When schedule server job, queue job for client○ Sync among server and clients
■ Server and client will all run the cluster job○ Server run server side○ Clients run client side
![Page 27: LKP-tests - 01.org | Intel Open Source Technology Center Rich benchmarks support Capture performance statistics (monitors) in addition to benchmark score For example, perf stat, perf](https://reader031.fdocuments.net/reader031/viewer/2022021704/5b4086d37f8b9a51528d4808/html5/thumbnails/27.jpg)
Cluster: Example job filesuite: netperftestcase: netperfcategory: benchmark
ip: ipv4
runtime: 300snr_threads: 200%
cluster: cs-localhost
if role server: netserver:
if role client: netperf:
test: TCP_RR