© 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers...
-
Upload
nelson-lambert -
Category
Documents
-
view
218 -
download
0
Transcript of © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers...
![Page 1: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/1.jpg)
© 2012 Whamcloud, Inc.
Whamcloud and Quality
• Chris Gearing & Mike StokSoftware EngineersWhamcloud, Inc
Lustre User GroupAustin TXApril 2012
Version 1.0
![Page 2: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/2.jpg)
© 2012 Whamcloud, Inc.2
• Whamcloud’s View Of Quality
• A Year’s Progress
• The Year Ahead
• Maloo ‘Your Window onto Test’ – Mike Stok
Agenda
![Page 3: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/3.jpg)
© 2012 Whamcloud, Inc.3
Whamcloud’s View Of Quality
![Page 4: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/4.jpg)
© 2012 Whamcloud, Inc.4
Eric vs. Non Delivery
‟ The only thing I’m non-Linear about is
Whamcloud not delivering what it says
it will deliver ”
Whamcloud’s View Of Quality
Eric BartonCTO Whamcloud
![Page 5: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/5.jpg)
© 2012 Whamcloud, Inc.5
• Whamcloud’s view of quality development
Whamcloud’s View Of Quality
StabilityQualityFeat
ures
Perf
orm
ance
Time
![Page 6: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/6.jpg)
© 2012 Whamcloud, Inc.6
• Whamcloud is investing money, time and expertise in Quality
• …is continuing to develop tools and invest in infrastructure to enhance the Lustre community
• The whole of its engineering team is orientated around quality principles
Whamcloud’s View Of Quality
![Page 7: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/7.jpg)
© 2012 Whamcloud, Inc.7
A Year’s Progress
![Page 8: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/8.jpg)
© 2012 Whamcloud, Inc.8
• For landing testing we have carried out;– 687 days of landing testing– 2381 landing sessions– 2092758 individual tests– 2087607 passed
• Statistically– 1425 passes – 956 failures– 59.85% Session passed
Landing Test Performance
![Page 9: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/9.jpg)
© 2012 Whamcloud, Inc.9
• This chart shows the percentage of landings that passed all tests since LUG 2011
Landing Test Performance
05/2
011
06/2
011
07/2
011
08/2
011
09/2
011
10/2
011
11/2
011
12/2
011
01/2
012
02/2
012
03/2
012
30.00%
35.00%
40.00%
45.00%
50.00%
55.00%
60.00%
65.00%
70.00%
75.00%
80.00%
Pass RateLinear (Pass Rate)
![Page 10: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/10.jpg)
© 2012 Whamcloud, Inc.10
Distributed Test
Source code
repository
PublicSector
ISVs
Resellers
![Page 11: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/11.jpg)
© 2012 Whamcloud, Inc.11
Distributed Test
Source code
repository
PublicSector
ISVs
PublicSectorResellers
Resellers
But!
![Page 12: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/12.jpg)
© 2012 Whamcloud, Inc.12
• Juelich financed a fully equipped cluster• Used for testing all head releases• Specialises in failover testing
– 36TB of multi attached storage
• Good for performance regression tests– Private network with no contention so results are repeatable
• Also used for manual large Lun testing for 2.2 release
• I’d like to thank Juelich and in particular Frank Heckes for making this happen
Juelich Supercomputer Centre
![Page 13: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/13.jpg)
© 2012 Whamcloud, Inc.13
• Indiana provided a 36 node cluster for the development and rollout of a backup test system
• Used for secondary 2.2 release testing• Enabled the transfer of the primary test cluster
from the West Coast to Colorado without any break in the Lustre test and landing processes
• Again I’d like to thank Indiana and Steve Simms for enabling this
Indiana University
![Page 14: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/14.jpg)
© 2012 Whamcloud, Inc.14
The Year Ahead
![Page 15: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/15.jpg)
© 2012 Whamcloud, Inc.15
Development vs. Landing Test• Which is landing test?• Which is development test?
05/2
011
06/2
011
07/2
011
08/2
011
09/2
011
10/2
011
11/2
011
12/2
011
01/2
012
02/2
012
03/2
012
0.00%
10.00%
20.00%
30.00%
40.00%
50.00%
60.00%
70.00%
80.00%
90.00%
100.00%
Pass RateLinear (Pass Rate)
![Page 16: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/16.jpg)
© 2012 Whamcloud, Inc.16
• Distinct paths for development vs. landing– Similar process but distinct purpose
• Provide for maximum flexibility in development testing– Encourage systematic test to be part of the development process– Test during development does improve product quality
• Encourage peer review as the code is written– Earlier review leads to better code and more opportunity for
education
• Be auditable– Improvement requires knowledge of the past– Development test part of the landing collateral
• 100% pass rate for landing test– Developers should push tried and tested code for landing
Development vs. Landing Test
![Page 17: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/17.jpg)
© 2012 Whamcloud, Inc.17
Development Test Cloud
![Page 18: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/18.jpg)
© 2012 Whamcloud, Inc.18
The Goal
0%
20%
40%
60%
80%
100%
Development Test
FailPass
0%
20%
40%
60%
80%
100%
Landing Results
FailPass
![Page 19: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/19.jpg)
© 2012 Whamcloud, Inc.19
• Whamcloud’s Quality Approach
• A Review Of The Last Year
• Our Plans For The Coming Year
Summary
![Page 20: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/20.jpg)
© 2012 Whamcloud, Inc.20
Maloo ‘Your Window onto Test’
![Page 21: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/21.jpg)
© 2012 Whamcloud, Inc.21 LUG Austin, TX - April 2012
• What is Maloo?• Recent changes to Maloo
– High points since last LUG
• Development priorities– Tool quality– Easy access to timely, accurate data
• Some planned features– User preferences– Automated scanning of incoming log files for “interesting” data
Maloo Agenda
![Page 22: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/22.jpg)
© 2012 Whamcloud, Inc.22 LUG Austin, TX - April 2012
• A repository for lustre test result data– Collects the test results and the logs generated– Allows users to query the database– Contains about 1TB of log files
• https://maloo.whamcloud.com
What is Maloo?
![Page 23: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/23.jpg)
© 2012 Whamcloud, Inc.LUG Austin, TX - April 201223
Recent changes to Maloo
![Page 24: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/24.jpg)
© 2012 Whamcloud, Inc.24
Release report
LUG Austin, TX - April 2012
https://maloo.whamcloud.com/reports
![Page 25: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/25.jpg)
© 2012 Whamcloud, Inc.25
Node utilization report
LUG Austin, TX - April 2012
https://maloo.whamcloud.com/reports/show_node_utilization_report
![Page 26: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/26.jpg)
© 2012 Whamcloud, Inc.26
Internal changes
• Invisible work– Development practices– Testing– Packaging– Deployment– …
LUG Austin, TX - April 2012
![Page 27: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/27.jpg)
© 2012 Whamcloud, Inc.LUG Austin, TX - April 201227
Development priorities
![Page 28: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/28.jpg)
© 2012 Whamcloud, Inc.28 LUG Austin, TX - April 2012
• Tool quality– Mechanics of Maloo and its development– Visibility into the tool– Usability
Development priorities
![Page 29: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/29.jpg)
© 2012 Whamcloud, Inc.29 LUG Austin, TX - April 2012
• Data quality– Accurate– Timely– Accessible
Development priorities
![Page 30: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/30.jpg)
© 2012 Whamcloud, Inc.LUG Austin, TX - April 201230
New features
![Page 31: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/31.jpg)
© 2012 Whamcloud, Inc.31 LUG Austin, TX - April 2012
Log file scanning
• Automates a tedious task
New features
![Page 32: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/32.jpg)
© 2012 Whamcloud, Inc.
Maloo Footprints
![Page 33: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/33.jpg)
© 2012 Whamcloud, Inc.33
Maloo Footprints
![Page 34: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/34.jpg)
© 2012 Whamcloud, Inc.34
Maloo Footprints
![Page 35: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/35.jpg)
© 2012 Whamcloud, Inc.35
Maloo Footprints
![Page 36: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/36.jpg)
© 2012 Whamcloud, Inc.36 LUG Austin, TX - April 2012
User profiles
• One size needn’t fit all
New features
![Page 37: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/37.jpg)
© 2012 Whamcloud, Inc.37 LUG Austin, TX - April 2012
What was covered• Changes in Maloo since last year• Our development priorities• A couple of the planned changes
Wrap up and questions
![Page 38: © 2012 Whamcloud, Inc. Whamcloud and Quality Chris Gearing & Mike Stok Software Engineers Whamcloud, Inc Lustre User Group Austin TX April 2012 Version.](https://reader035.fdocuments.net/reader035/viewer/2022062422/56649ef25503460f94c0469d/html5/thumbnails/38.jpg)
© 2012 Whamcloud, Inc.
Thank you very much
38