CCSM Testing Status Tony Craig Lawrence Buja Wei Yu CCSM SEWG Meeting Feb 5, 2003.
-
Upload
dominic-blair -
Category
Documents
-
view
214 -
download
0
Transcript of CCSM Testing Status Tony Craig Lawrence Buja Wei Yu CCSM SEWG Meeting Feb 5, 2003.
Test StrategyWhen-Who-Why
WHEN WHO WHY
pre-commit developer validate changes
post-commit test engineer, automated?
verify commits
pre-release release team verify model is ready for release
post-release test engineer, automated?
verify platform is not changing
Test StrategyWhat
• Exact restart• Bit-for-bit, round-off, other• Different Platforms• Different Resolutions• Serial, multitasked, threaded, hybrid• Physics packages, dynamical cores• Scientific Validation - long climate runs• Performance• I/O, Data
Testing Process for each Component
ccsm atm lnd ocn ice cpl datapre-commit comp dev dev gate-
keepergate-keeper
? ?
post-commit test engr test engr gate-keeper
gate-keeper
gate-keeper
? ?
pre-release release team
release team
release team
N/A N/A N/A ?
post-release
test engr ? ? N/A N/A N/A ?
What do we do now
• CAM test-model– used by developers
• CAM dev branch testing– automated testing after each commit
• CCSM beta tag tests– manual testing of entire system periodically
• CCSM release tests– automated testing of releases versions regularly
CAM test-model
• Script that runs many CAM cases automatically• Runs on many platforms (SGI, IBM, CPQ, PC/Linux)• Many automated tests
– dynamical cores– tasks and threads– error growth
CAM dev branch testing
• Automatic testing of CAM every night after a commit on the primary development branch
• Uses test-model• Runs on chinook, blackforest, anchorage
CCSM beta tag tests
• Exact restart testing– Various configurations– Various resolutions
• Comparison with previous beta tag
CCSM Release Tests
• Weekly testing of CCSM2.0, CCSM2.0.1 releases on – chinook– blackforest– bluesky– seaborg
• Includes some patches
CCSM performance testing
• Carry out a large suite of timing tests• Comparison with previous versions• Determine appropriate load balance
What next
• Develop a formal test plan document• Determine Who, When, Why, What• Establish formal processes for each component and
for the CCSM overall• Consider resources required (both people and
computer time)• Unit tests
Notes
• Test test-model• Test Configuration thoroughly, what’s supported,
– configuration = dycores, large components, physics, machine dependencies
• chunking• testing <-> requirements• test-model lite• add tests - cost?• difference between dev tests, nightly tests, release testing, test-
model is same for all• community testing
Notes - 2
• pop - coarse and high resolution tests cover most of physics space, also different pe configurations
• test-model compares to previous version, very useful• test-model for land exists• make consistency between stand-alone and ccsm
versions of the model, unify compiler options?• ccsm requirements vs component requirements• specs for makefile, specs in general?• unit tests - physics wrapper
Notes - 3
• Decide where tests take place, who is responsible for what parts
• recommendation that working groups test coupling aspects, also verification of make in coupled system,
• debug flags testing• gap in test process wrt make• track make fixes/changes through bug tracker• go to wg, get test requirements
Notes - 4
• performance requirement• science requirement• is it just the “control run”• get developers input on what should be in the test suite• library issues, internal libraries (ESMF, MCT), mass• test cost - test-model, chinook (1 hours, 16 pes), blackforest(30-
40 minutes, 32 pes), babyblue (30 minutes), anchorage (30 minutes), lots of time spent building
• regular testing, automated testing, different frequencies
Recommendations
• component working groups test coupling, cseg liaisons coordinate
• develop specs for makefile• go to working groups, find out test requirements, then
decide who does what• develop ccsm test requirements• revisit benchmarking requirements
Open Discussion
• Community has a difficult time keeping up with what’s happening in NCAR
• Could have more forward planning, IPCC, resolutions, chemistry, bgc,
• SE and science status• Should we make component development work more visible• CRBs, adaptation, software practices• Need more background on decisions made• Code review - need? time?• Design walkthroughs and code reviews are working for the
ocean model.• Code reviews help educate others on code
open discussion - 2
• SEWG encourages the use of code reviews. SEWG will explain what we encourage. Recommend brown bag.
• recommend a code walkthrough of cpl6.• Discuss status of code review plans at june workshop, internal
SEWG• ccsm-progs, ccsm-sci, wg notes to be distributed to ?• get mailing lists under control• who do we make recommendations too? SSC, SE dev guide,
word of mouth