Organisingbenchmarking LLVM-based compiler: Arm...
Transcript of Organisingbenchmarking LLVM-based compiler: Arm...
![Page 1: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/1.jpg)
© 2018 Arm Limited
Organising benchmarking LLVM-based compiler:
Arm experience• Evgeny Astigeevich
• LLVM Dev Meeting April 2018
![Page 2: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/2.jpg)
2 © 2018 Arm Limited
Terminology
• Upstream: everything on llvm.org side. • Downstream: everything on your side.• Benchmarking a compiler: part of QA process where compiler quality requirements,
such as generated code performance, code size, compilation time and others, are verified.
• Bisecting a regression: a process of identifying commits caused the regression.• Bare-metal application: an application which runs without OS supporting it.• OS-hosted application: an application which needs OS to run.
![Page 3: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/3.jpg)
3 © 2018 Arm Limited
Benchmarking a compiler: get answers to
• Do my changes affect the compiler?• Is the compiler improving?• What caused regressions/improvements?
![Page 4: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/4.jpg)
4 © 2018 Arm Limited
What is ARM Compiler 6?
• Toolchain for development of bare-metal applications• C/C++ and GNU assembly compiler based on Clang/LLVM (armclang)• Assembler for legacy Arm-syntax assembly• Linker• C++ libraries based on LLVM libc++• C libraries• ARM librarian (armar)• ARM image conversion utility (fromelf)
![Page 5: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/5.jpg)
5 © 2018 Arm Limited
Why did we base our compiler on LLVM?
![Page 6: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/6.jpg)
6 © 2018 Arm Limited
Why did we base our compiler on LLVM?
![Page 7: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/7.jpg)
7 © 2018 Arm Limited
Cortex-A vs Cortex-R vs Cortex-M
![Page 8: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/8.jpg)
8 © 2018 Arm Limited
ARM Compiler product requirements
• Good quality of Cortex-A/R/M code.• No significant regressions in releases.
![Page 9: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/9.jpg)
9 © 2018 Arm Limited
The benchmarking process highly depends on how an interaction between upstream and downstream is organized.
![Page 10: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/10.jpg)
10 © 2018 Arm Limited
Avoiding merge conflicts
We do development upstream as much as possible.The rough difference is ~20-50K SLOC.
![Page 11: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/11.jpg)
11 © 2018 Arm Limited
Benchmarking Cortex-A code
• Cortex-A can run Linux => More benchmarks can be run
• Benchmarks are CPU-oriented => OS-hosted benchmarking can be used
• Llvm.org already has a working solution: BuildBot + LNT client/server tools
![Page 12: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/12.jpg)
12 © 2018 Arm Limited
lnt.llvm.org
![Page 13: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/13.jpg)
13 © 2018 Arm Limited
Internal LNT
• The infrastructure is similar to llvm.org infrastructure: BuildBot, LNT client/server tools.
• LNT provides all needed benchmarking functionality out-of-the-box.
• The internal LNT works with upstream Clang/LLVM repositories to get bisecting working.
BuildBot
Benchmarking
bots
Build
cache
LNT server
BinariesCI bot
Results
![Page 14: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/14.jpg)
14 © 2018 Arm Limited
Benchmarking Cortex-A code
• We use the internal and upstream LNTs to analyze significance of regressions.
![Page 15: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/15.jpg)
15 © 2018 Arm Limited
Benchmarking Cortex-R/M bare-metal code
![Page 16: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/16.jpg)
16 © 2018 Arm Limited
Daily Upstream ó Downstream synchronization
At 6:30am update upstream copy, build and test
Manually resolve issues
All Passed
No
Merge to downstream
Yes
NoPassed
YesEnd
6:30am was chosen because of the lowest development activity in upstream.
![Page 17: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/17.jpg)
17 © 2018 Arm Limited
Repositories status
• Toolchain builds: b01, b02, b03.
Upstream
Downstream
Commits
Merge
b01 b02 b03
![Page 18: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/18.jpg)
18 © 2018 Arm Limited
Product building
Build testable toolchain
Run benchmarks
Submit results to database
Compare results
Nightly downstream benchmarking
![Page 19: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/19.jpg)
19 © 2018 Arm Limited
“What caused regressions/improvements?”
• Manual bisecting:• An upstream commit needed to be merged to downstream. Not always possible.• Compiler binaries needed to be built per a merge. Not always possible.
Upstream
Downstream
Commits
Merge
b01 b02 b03
![Page 20: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/20.jpg)
20 © 2018 Arm Limited
The first solution: summary
• Pros:• Very simple to implement.• Upstream CI guards you from “bad” commits.• Merge conflicts are resolved when upstream is less active.• Nightly toolchain builds are based on a “stable” upstream trunk revision.
• Cons• No CI. Testing and benchmarking is started after the full toolchain is built.• Downstream benchmarking results are always outdated.• Complex merge conflicts can take more than one day and block synchronization.• Bisecting is very difficult.
![Page 21: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/21.jpg)
21 © 2018 Arm Limited
The first solution worked well enough
• Not many commits into Arm related areas => Not many merge conflicts • Not many optimization works => No need to automate manual tasks• Not many embedded benchmarks => Not many regressions
![Page 22: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/22.jpg)
22 © 2018 Arm Limited
But…
![Page 23: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/23.jpg)
23 © 2018 Arm Limited
Increased upstream development activity (100+ commits per day) => More merge conflicts
![Page 24: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/24.jpg)
24 © 2018 Arm Limited
Complex merge conflicts => Merges were blocked for days => Delayed benchmarking => A snowball effect
![Page 25: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/25.jpg)
25 © 2018 Arm Limited
Any building infrastructure instabilities => No toolchain => Delayed benchmarking
![Page 26: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/26.jpg)
26 © 2018 Arm Limited
More benchmarking configurations => More regressions
![Page 27: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/27.jpg)
27 © 2018 Arm Limited
At the end of 2016 our solution stopped working…
Engineers might spend a week on bisecting regression. Then it was too late to report.
This resulted a lot of internal regression reports (50+) to be created but nothing was investigated and reported upstream.
![Page 28: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/28.jpg)
28 © 2018 Arm Limited
We wanted to have fun but the benchmarking was a real pain in a the neck.
![Page 29: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/29.jpg)
29 © 2018 Arm Limited
The Optimization Team
• The team responsible for benchmarking and for implementing optimizations.• 2 engineers (inc. a team lead): only benchmarking related tasks, no optimization tasks.• 3 engineers (inc. a team lead): some optimization tasks.• 4 engineers (inc. a team lead): capable to deliver great results.
![Page 30: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/30.jpg)
30 © 2018 Arm Limited
Problem #1: regressions
• Solution: Continuous Integration
![Page 31: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/31.jpg)
31 © 2018 Arm Limited
Continuous Integration
• In software engineering, continuous integration (CI) is the practice of merging all developer working copies to a shared mainline several times a day.
• https://en.wikipedia.org/wiki/Continuous_integration
![Page 32: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/32.jpg)
32 © 2018 Arm Limited
New Upstream ó Downstream schema
Create a list of commits
Any unresolved conflicts?
End
Merge a commit
Yes
No
Yes
End
Is the list empty?
NoIs it failed?
Push the failed to Gerrit for manual
resolution
No
Push the merge result to
downstream
Yes
![Page 33: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/33.jpg)
33 © 2018 Arm Limited
New Upstream ó Downstream schema
Create a list of commits
Any unresolved conflicts?
End
Merge a commit
Yes
No
Yes
End
Is the list empty?
NoIs it failed?
Push the failed to Gerrit for manual
resolution
No
Push the merge result to
downstream
Yes
It is implemented as Jenkins’ job and is run every 30 minutes.
![Page 34: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/34.jpg)
34 © 2018 Arm Limited
New Upstream ó Downstream schema
Create a list of commits
Any unresolved conflicts?
End
Merge a commit
Yes
No
Yes
End
Is the list empty?
NoIs it failed?
Push the failed to Gerrit for manual
resolution
No
Push the merge result to
downstream
Yes
Commits are sorted by date to solve a multirepo problem
![Page 35: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/35.jpg)
35 © 2018 Arm Limited
New Upstream ó Downstream schema
Create a list
of commits
Any
unresolved
conflicts?
End
Merge a
commit
Yes
No
Yes
End
Is the list
empty?
NoIs it failed?
Push the failed to
Gerrit for manual
resolution
No
Push the
merge result to
downstream
Yes
If the result is not empty, the
push triggers building armclang
in Jenkins.
![Page 36: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/36.jpg)
36 © 2018 Arm Limited
New Upstream ó Downstream schema
Create a list
of commits
Any
unresolved
conflicts?
End
Merge a
commit
Yes
No
Yes
End
Is the list
empty?
NoIs it failed?
Push the failed to
Gerrit for manual
resolution
No
Push the
merge result to
downstream
Yes
A conflict is annotated with 3-way conflict markers to help an engineer to resolve it.
![Page 37: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/37.jpg)
37 © 2018 Arm Limited
Results
• On average, a merge contains 2-3 upstream commits.• On average, bisecting time reduced from a day to a few hours. We still need to build
armclang per commit.• Most of merge conflicts are easy to resolve.
Upstream Commits
Downstream
m01 m02 m03 Downstream only commits
![Page 38: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/38.jpg)
38 © 2018 Arm Limited
But we still…
• Did a lot of manual building.• Did manual bisecting.• Found that more hardware needed for regression analysis and benchmarking.• Found hardware dependent regressions.
![Page 39: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/39.jpg)
39 © 2018 Arm Limited
Build cache
• Our build cache is built on Artifactory.
![Page 40: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/40.jpg)
40 © 2018 Arm Limited
Jenkins
Benchmarkingjobs
Buildcache
LNT server
Armclangbinaries
CI job
Results
Auto-bisecting job
Regression tracking systemAuto-merge job
![Page 41: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/41.jpg)
41 © 2018 Arm Limited
Jenkins
Benchmarkingjobs
Build
cache
LNT server
Armclangbinaries
CI job
Results
Auto-bisecting job
Regression tracking systemAuto-merge job On Cortex-R/M benchmarks
are noiseless.
![Page 42: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/42.jpg)
42 © 2018 Arm Limited
Hardware (bare-metal boards)
• The process of initialization can take more time than an actual benchmark run.
![Page 43: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/43.jpg)
43 © 2018 Arm Limited
Hardware (bare-metal boards)
• We use performance simulators where it is possible.• We moved from vendor-specific boards to FPGA boards.
![Page 44: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/44.jpg)
44 © 2018 Arm Limited
Benchmarking system
Cortex-A benchmarking
Cortex-R/M benchmarking
CI(BuildBot) + manual bisecting
upstream sources
CI(Jenkins) + auto-bisecting
Toolchain full overnight
benchmarking
downstream sources
![Page 45: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/45.jpg)
45 © 2018 Arm Limited
Dealing regressions
• Time is your enemy. L• A good report is the key. Focus on creating a reproducer.• Can be a workaround/downstream patch on a branch but not on the trunk.
![Page 46: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/46.jpg)
46 © 2018 Arm Limited
Preventing regressions
• Be part of the community.• Monitor llvm mailing lists• Help with assessing impact
– But we always don’t have time L.
• Open question: how to automate?
![Page 47: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/47.jpg)
47 © 2018 Arm Limited
Future works
• Unify our systems• Public build cache
![Page 48: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/48.jpg)
48 © 2018 Arm Limited
Future works
• Unify our systems• Public build cache
![Page 49: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/49.jpg)
49 © 2018 Arm Limited
Public build cache
![Page 50: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/50.jpg)
50 © 2018 Arm Limited
Questions
![Page 51: Organisingbenchmarking LLVM-based compiler: Arm experiencellvm.org/devmtg/2018-04/slides/Astigeevich... · Organisingbenchmarking LLVM-based compiler: Arm experience ... Nightly downstream](https://reader030.fdocuments.net/reader030/viewer/2022041021/5ed10b23b092f05fef744943/html5/thumbnails/51.jpg)
5151
The Arm trademarks featured in this presentation are registered trademarks or trademarks of Arm Limited (or its subsidiaries) in the US and/or elsewhere. All rights reserved. All other marks featured may be trademarks of their respective owners.
www.arm.com/company/policies/trademarks
© 2018 Arm Limited