Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka...
Transcript of Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka...
![Page 1: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/1.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �1
Migrating large codebases to C++ Modules
Yuka Takahashi - The Univers i ty of Tokyo Pr inceton Univers i ty
Oksana Shadura - UNL Vassi l Vass i lev
![Page 2: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/2.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
1. Motivation of C++ Modules 2. C++ Modules in ROOT 3. C++ Modules in CMSSW 4. CMS Performance Results 5. Conclusion
Agenda
�2
![Page 3: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/3.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
Motivation of C++ Modules
�3
![Page 4: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/4.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
Motivation of C++ Modules
�4
C++ Modules technology: - Cache parsed header file information - Avoid header re-parsing - Avoid runtime header parsing (In ROOT) - Part of C++20
![Page 5: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/5.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �5
#include <vector>
Motivation of C++ Modules
![Page 6: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/6.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �6
Textual Include Precompiled Headers (PCH) Modules
#include <vector>
Expensive Fragile
Inseparable
Motivation of C++ Modules
![Page 7: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/7.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
#include "TVirtualPad.h"#include <vector>#include <set>
int main() {…
�7
…………
# 286 "/usr/include/c++/v1/vector" 2 3namespace std { inline namespace __1 {template <bool> class __vector_base_common { __attribute__ ((__visibility__("hidden"), __always_inline__)) __vector_base_common() {}
……# 394 "/usr/include/c++/v1/set" 3namespace std {inline namespace __1 {template <…> class set {public: typedef _Key key_type;
……
int main {……
Parse.o
vector
set
TVirtualPad.h
Compile
original code
one big file!
Motivation of C++ Modules
Preprocess
Textual Include
![Page 8: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/8.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �8
#define PI 3.14…
Rcpp library
Users’ code#include <header.h>
…double PI = 3.14;// => double 3.14 = 3.14;
Textual Include
1.Expensive Reparse the same header
2.Fragile Name collisions
.h
.c
.c
.c .o
.o
.o
Motivation of C++ Modules
.h
![Page 9: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/9.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
1.Storing precompiled header information (same as modules) 2. Stored in one big file - Monolithic
�9
PCH (Precompiled Headers)
allDict.cxx.pch
Interpreter
Motivation of C++ Modules
PCH
.c
.c
.c .o
.o
.o
In ROOT
![Page 10: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/10.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
Each PCM file (a.pcm) corresponds to a library (liba.so)
Interpreter
b.pcma.pcm d.pcmc.pcm e.pcm
�10
- Module files contain parsed header information - PCMs are separated
ModulesMotivation of C++ Modules a.pcm
In ROOT
b.pcm
c.pcm
.c
.c
.c .o
.o
.o
![Page 11: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/11.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �11
Compile-time scalability No Fragility Separable
ModulesMotivation of C++ Modules
- Modules files contain parsed header information - PCMs are separated
![Page 12: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/12.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
C++ Modules in ROOT
�12
Technology Preview released in ROOT 6.16
![Page 13: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/13.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
C++ Modules in ROOT
�13
ROOT
rootcling
Clang Cling
ROOT Dictionaries
binaries
files
Overview - Dependency Graph
![Page 14: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/14.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �14
Cling calls Clang API
ROOT
rootcling
Clang Cling
ROOT Dictionaries
binaries
files
C++ Modules in ROOT Overview - Dependency Graph
![Page 15: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/15.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �15
rootcling generates dictionaries (rootmap, rdict)
ROOT
rootcling
Clang Cling
ROOT Dictionaries
binaries
files
C++ Modules in ROOT Overview - Dependency Graph
![Page 16: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/16.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �16
ROOT
rootcling
Clang Cling
ROOT Dictionaries
binaries
files
ROOTMAP Used to map symbols and identifiers to libraries
C++ Modules in ROOT Overview - Dependency Graph
![Page 17: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/17.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �17
ROOT
rootcling
Clang Cling
ROOT Dictionaries
binaries
files
RDICT Efficiently store
information needed for serialization
C++ Modules in ROOT Overview - Dependency Graph
![Page 18: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/18.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �18
Dictionaries are used at ROOT runtime
ROOT
rootcling
Clang Cling
ROOT Dictionaries
binaries
files
C++ Modules in ROOT Overview - Dependency Graph
![Page 19: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/19.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �19
ROOT PCMs
ROOT
rootcling
Clang Cling
ROOT Dictionaries
binaries
files
C++ Modules in ROOT Overview - Dependency Graph
![Page 20: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/20.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �20
C++ Modules in ROOT
Preloading of all moduleshttp://root-bench.cern.ch (login required)
Mechanism of loading modulesPreloading of modules
- Replace some functionality of RDICT and ROOTMAP with a more stable implementation - Load all ROOT modules at the startup time
RSS Memory HSimple (4 months)665MB
Production Development
![Page 21: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/21.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �21
Global Modules Index
- Remove further overhead in ROOT - Mechanism to create the table of symbols and PCM names - ROOT will be able to load corresponding library when a symbol lookup failed - The prototype shows promising results
C++ Modules in ROOT Mechanism of loading modules
![Page 22: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/22.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �22
C++ Modules in ROOT
Bloom filter
Mechanism of loading librariesBloom filter- Hash tables of symbols in .gnu.hash section in shared object files (further read) - ROOT can skip unnecessary libraries by reading it
232MB
Production Developmenthttp://root-bench.cern.ch (login required)
RSS Memory HSimple (4 months)
![Page 23: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/23.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
C++ Modules in CMSSW
�23
Available in CMS CXXMODULE IB
![Page 24: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/24.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �24
Genreflex (rootcling)
SCRAM Build system
CMS Runtime
C++ Compiler
CMS Dictionaries
CMS Libraries
C++ Modules in CMSSW
binaries
files
Overview - Dependency Graph
![Page 25: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/25.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �25
Genreflex and GCC, executed by SCRAM
Genreflex (rootcling)
SCRAM Build system
CMS Runtime
CMS Dictionaries
CMS Libraries
C++ Compiler binaries
files
C++ Modules in CMSSW Overview - Dependency Graph
![Page 26: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/26.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �26
Dictionaries generated by Genreflex
Genreflex (rootcling)
SCRAM Build system
CMS Runtime
CMS Dictionaries
CMS Libraries
C++ Compiler binaries
files
C++ Modules in CMSSW Overview - Dependency Graph
![Page 27: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/27.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �27
Libraries compiled by gcc
Genreflex (rootcling)
SCRAM Build system
CMS Runtime
CMS Dictionaries
CMS Libraries
C++ Compiler binaries
files
C++ Modules in CMSSW Overview - Dependency Graph
![Page 28: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/28.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �28
Genreflex (rootcling)
SCRAM Build system
CMS Runtime
CMS C++ Modules PCMs
CMS Dictionaries
CMS Libraries
C++ Compiler binaries
files
C++ Modules in CMSSW Overview - Dependency Graph
![Page 29: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/29.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �29
Genreflex (rootcling)
SCRAM Build system
CMS Runtime
CMS C++ Modules PCMs
CMS Dictionaries
CMS Libraries
C++ Compiler binaries
files
C++ Modules in CMSSW Overview - Dependency Graph
- Not all CMS libraries were modularized - Modules can co-exist with the old infrastructure
![Page 30: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/30.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �30
Genreflex (rootcling)
SCRAM Build system
CMS Runtime
CMS C++ Modules PCMs
CMS Dictionaries
CMS Libraries
C++ Compiler binaries
files
C++ Modules in CMSSW Overview - Dependency Graph
Modulemap
![Page 31: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/31.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �31
module.modulemap
- Definition file of headers to build a PCM in Clang - Contain all “interface” headers, which are used by librariesmodule "MathCore" { module “TComplex name” { header "TComplex.h" export * } module <name of the file> {
header <relative path to the header file location> }}
C++ Modules in CMSSW Explicit PCMs in CMSSW
modulemap will contain all interface header filesAutogeneration of modulemap
![Page 32: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/32.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �32
- CMSSW has “interface” headers - Exposed to libraries outside - Automatically generate the modulemap by adding interface headers - Modulemap needs to be generated before the execution of genreflex - Build system is responsible for the autogeneration
C++ Modules in CMSSW Explicit PCMs in CMSSWAutogeneration of modulemap
![Page 33: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/33.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �33
Genreflex (rootcling)
SCRAM Build system
CMS Runtime
CMS C++ Modules PCMs
CMS Dictionaries
CMS Libraries
C++ Compiler binaries
files
C++ Modules in CMSSW
Modulemap
Explicit PCMs in CMSSW
![Page 34: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/34.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �34
modulemap, modulemap overlay file, virtual modulemap overlayC++ Modules in CMSSW
/
$build_dir
include/
usr/
module.modulemap
module “header.h”{ header “some/directory/
headers.h” }
Mechanism of the modulemap
*.h (headers)directories
files
some/ directory/
![Page 35: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/35.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �35
C++ Modules in CMSSW Mechanism of the modulemapmodulemap, modulemap overlay file, virtual modulemap overlay
Modulemap for system headers - Modulemap needs to know the location of headers - System headers’ location cannot be hardcoded
Modulemap Overlay File
![Page 36: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/36.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �36
C++ Modules in CMSSW
/
$build_dir
include/
usr/
Modulemap Overlay File
name : “/usr/include” contents : [ { name : “module.modulemap” external-contents :
“path/to/stl.modulemap” }
Mechanism of the modulemap
*.h (headers)directories
files
some directory/
stl.modulemap
module.modulemap
Location of system headers (Generated from CMake)
Relative path to stl.modulemap
modulemap, modulemap overlay file, virtual modulemap overlay
![Page 37: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/37.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �37
C++ Modules in CMSSW
/
$build_dir
include/
usr/
Modulemap Overlay File
name : “/usr/include” contents : [ { name : “module.modulemap” external-contents :
“path/to/stl.modulemap” }
Mechanism of the modulemap
*.h (headers)directories
files
some directory/
stl.modulemap
module.modulemap
Location of system headers (Generated from CMake)
Relative path to stl.modulemap
module “stl” { module “algorithm” { header “algorithm” }}
modulemap, modulemap overlay file, virtual modulemap overlay
![Page 38: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/38.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �38
C++ Modules in CMSSW Mechanism of the modulemapmodulemap, modulemap overlay file, virtual modulemap overlay
Clang interprets those information as - module.modulemap exists in the location of system headers (/usr/include, in this example) - module.modulemap has the contents of stl.modulemap
Modulemap Overlay File can handle system headers, but it needs to be generated at configuration time
![Page 39: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/39.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �39
C++ Modules in CMSSW Mechanism of the modulemapmodulemap, modulemap overlay file, virtual modulemap overlay
Solution: Virtual Modulemap Overlay File
The location of system headers needs to be generated at CMake (configuration) time - Not binary distributable - Not relocatable - CMS builds and distributes binary to other locations
![Page 40: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/40.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �40
C++ Modules in CMSSW
/
$build_dir
include/
usr/
Mechanism of the modulemap
*.h (headers)
some directory/
module.modulemap
Virtual Modulemap Overlay File (VMOF)
name : <Location of system header at runtime> contents : [ { name : “module.modulemap” external-contents : “path/to/stl.modulemap” ]
Not an actual file (virtual file on memory)
modulemap, modulemap overlay file, virtual modulemap overlay
stl.modulemap
Dynamically determined
![Page 41: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/41.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �41
Summary
- C++ Modules integration in CMSSW - Genreflex generates pcm files - Autogeneration of modulemap from build system - Modulemap for system headers (libc, stl) - Virtual Modulemap overlay file
C++ Modules in CMSSW
![Page 42: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/42.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
CMS Performance Results
�42
![Page 43: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/43.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �43
Standard error of mean
22.5 seconds better than ROOT Master
96 pcms 121 pcms0 pcms
“Fast simulation test” ESetup Lock (seconds)
random 100 events 15 times execution
![Page 44: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/44.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �44
96 pcms 121 pcms0 pcms
“Fast simulation test” ESetup Get (seconds)
15.2 seconds better than ROOT Master
Standard error of mean
random 100 events 15 times execution
![Page 45: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/45.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �45
96 pcms 121 pcms0 pcms
“Digitization test” CPU Total Loop (seconds)
331 seconds better than ROOT Master
Standard error of mean
random 100 events 15 times execution
![Page 46: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/46.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �46
96 pcms 121 pcms0 pcms
“Digitization test” RSS Memory (MBytes)
143 Mbytes worse than ROOT Master
Standard error of mean
random 100 events 15 times execution
![Page 47: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/47.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �47
Summary
CMS Performance Results
- Results suggests the performance benefits at runtime - Especially at the initialization time - ~150 MBytes RSS overhead - Investigation is ongoing
![Page 48: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/48.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
Conclusion
�48
![Page 49: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/49.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
Conclusion
�49
- C++ Modules were implemented and tested in ROOT and in CMSSW - Improves the header modularity of libraries - Preliminary performance study suggested the performance improvement at runtime - Work on performance improvement is ongoing
![Page 50: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/50.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �50
Thank you for your attention!
![Page 51: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/51.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019
Backup slides
�51
![Page 52: Migrating large codebases to C++ Modules - GitHub Pages+Modules.pdf · 2020-06-17 · Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 1.Motivation of](https://reader035.fdocuments.net/reader035/viewer/2022063004/5f96f9c8116cc32d6444aed9/html5/thumbnails/52.jpg)
Yuka Takahashi 13.03.2019 Migrating large codebases to C++ Modules, ACAT 2019 �52
Implicit pcmsImplicitly generated without modulemaps - Add all possible header files needed for the generation of the dictionary - Huge header duplication
C++ Modules in CMSSW Implicit PCMs in CMSSW
Explicit pcmsExplicitly generated with modulemaps - Only add defined headers to the PCM - Reduce header duplication