March 26, 2018 Sam Siewert

SE420 Software Quality Assurance

Lecture 11 – Using Test Automation for Process Improvement [SEI Level 3,4,5 – Defined,

Repeatable, Measureable, Continuously Improved]

http://dilbert.com/strips/comic/2007-07-30/

Reminders Assignment #5

Remaining Assignments [Top Down / Bottom-Up] – #6 – Complete Code, Refine and Run all V&V Tests and Deliver

Track Bugs with Bugzilla - http://prclab.pr.erau.edu/, or GitHub Issues or Spreadsheet (backup) Update your Project Code on GitHub - https://github.com/ Assignment #5 & #6 Posted – Questions?

Sam Siewert 2

Regression Testing and Test Automation

Re-test Units After Defects are Fixed Re-run I&T and System Tests Automation from Checkout, Build, Test Sets, Verification & Validation of Expected Results

Sam Siewert 3

Reg

ress

ion

Test

ing

(Tes

t Aut

omat

ion)

Common Languages Used for SQA Automation

Historically – BASH and CSH Shell Scripting – Used for quick and simple test automation – TCL/TK - Tool Command Language, Expect – Larry Wall’s Perl language (found in older test suites)

In Addition and/or More Often Today, [PL specific] … – Python and numerous libraries such as robotframework – MS PowerShell, Autoit for Windows and BASH for Linux – DejaGnu for most any compiled PL – Java and JUnit, JCov – Ruby using cucumber – Team #2 - JavaScript - https://coveralls.io/, Jasmine, AVA, Mocha

Examples to Automate C/C++ with BASH (Expect) - Example SRS – gcov, gprof METRICS – assert (why?) TEST CONTROL – Expect TEST AUTOMATION – Makefile, BASH TEST SCRIPTING

Sam Siewert 4

Test Case, Test Set Automation Compose Drivers, C main(char * argv[], int argc) to run Test Cases based on Argument Vector – Each Driver is a Major Test Case with Numerous Sub-test Cases – Theme for Each, such as Performance – Data Compare and Verification – Error Handling with Fault Injection

Run Each Driver from C-shell, Perl, or Python Script to Automate Execution of Test Cases [Vectors] in Test Set Save Results to a Log File Process Log File to Generate [PASS/FAIL] Report

Sam Siewert 5

Simple Automation Example https://github.com/siewertserau/Examples-RAID-Unit-Test https://github.com/siewertserau/Examples-RAID-Unit-Test/blob/master/raid_regression.sh

Sam Siewert 6

Always Do “git pull” and Build First

Automated Test Run for RAID

TEST SET 2 RAID5 Encode Testing Rebuild Erased Sector from XOR Parity Dump Both Data Compare Visually Verify or Automate with Expect / DejaGNU Also Dumps Binary Files for diff compare

Sam Siewert 7

TCL / Expect / DejaGNU Expect (NIST Tool – Don Libes) built on TCL http://www.gnu.org/software/dejagnu/ - Extension of TCL and Expect, To Automate Build for Test Concept – Automate Verification of Expected Results and Convert Test Output o PASS / FAIL Useful for Regression Testing Test Sets or Suites – Composed of Many Test Cases DejaGnu Manual Howto: Using DejaGnu for Testing

Sam Siewert 8

TEST SET 2 Test 1000 Unique (or more) Cases of Rebuild with Assert to Catch Data miscompare External Binary File Diff – Expected differences

Sam Siewert 9

TEST SET 3 Performance – Adapted from Earlier Driver Improve to Add Cases for Encode Case for Rebuild of N Sectors Read-Modify Write Encode Cases Fill-out Test Set with More Cases and Verify with DejaGnu

Sam Siewert 10

Expect PASS/FAIL Automation http://www.thegeekstuff.com/2010/10/expect-examples/ List all Expect patterns for FAIL and timeout and exit if FAIL matches On timeout expiration, note and go on to next test On PASS match, note and go on

Sam Siewert 11

No Input to Expect Script

Expected PASS Input to Expect Script

General V&V Automation

Many Tools Exist DejaGnu – a short introduction Go Beyond Batch Test and Visual Inspection Expect / DejaGnu – Automates V&V Reports in Addition to Running Drivers

Sam Siewert 12

Batch is Execution Automation Only To Run Drivers in A Sequence

Test Automation and SEI CMM Coverage Criteria, Bug Open/Close Rate Metrics, Pass/Fail Statistics Provide Quantitative Management [Level 4] Automation is a Step Toward Optimization – Necessary, but Not Sufficient (Metrics for Feedback Process Improvement)

Sam Siewert 13

Ad Hoc, “Code & Fix” – Driven by Individual Styles, Un-coordinated, More of an Art than Practice

Clear Phases with Key Deliverables – Analysis, Design, Units, Test Plans

E.g. Boehm’s “V” and Spiral Model, Applied to All Projects - Repeatable

Entry/Exit Criteria for Phases, Metrics Drive Decisions in Process [Gates]

Feedback Used to Improve Process with Automation and Innovation

SEI CMM Levels

http://www.tutorialspoint.com/cmmi/cmmi-maturity-levels.htm

SEI CMM Levels Described Level 3 and Above Requires IPPD – Customers – Managers – Engineers – Users – SQA & Audit

To Contribute to Improvement of Process over Lifecycle and Products Sam Siewert 14

http://www.tutorialspoint.com/cmmi/cmmi-maturity-levels.htm

SEI CMM 4 & 5 Most On-Going Software and Systems Organizations are SEI CMM 3 to Survive [Required for Business Continuity] Levels 4 & 5 Speed Time to Market, Improve Customer Experience, Grow Market Share, Reduces Lifecycle Cost

Sam Siewert 15

http://www.tutorialspoint.com/cmmi/cmmi-maturity-levels.htm

SEI Rating Examples Certification for Organizations and Professionals – http://www.sei.cmu.edu/ – http://www.sei.cmu.edu/certifi

cation/ – https://www.sei.cmu.edu/train

ing/certificates/holders – http://www.sei.cmu.edu/meas

urement/ (methods for Level-4, 5)

Organizations Claiming SEI CMM Level 4 or 5 – More Today – Based on

Automation, Life-long Learning

Sam Siewert 16

Note on Validation and Verification

Metrics, Repeatability and Optimization Layer on Good Base of V&V – Did We build the right thing – Validation – Did We build the thing right – Verification

For tests, did we build the test right? – Test verification – Test your tests – Follow good practices in Test driver code

Did we build the right test? – Test validation – Are we running the test on correct Architecture and HW

configuration? – Are we running the test with the correct and most current build of

the software? Sam Siewert 17

Discussion and Q&A – Does SEI CMM Rating Matter?

System Testing – Entrance Criteria [When is I&T Complete?] – Of All Recognized System Test Types, Which are Critical to My Organization?

Wikipedia on System Test Types Select Focus in System Testing Based on System Requirements and Product

– Exit Criteria

Regression Test Automation – Checkout, Build, Test Set (Cases in Drivers) Scripted, Expected Results Verified

& Validated – Unit, I&T, System Test Sets Re-run – Daily, Weekly, Prior to “Push”

SQA Process

– Documentation, Audits, Improvement – Certification – E.g. ISO 9000, SEI CMM Levels 3 … 5, Etc. – Is Process Documented? Repeatable? Can it Be Improved? Automated?

Verifiable Criteria? [E.g. Code Coverage Tool Records] – Does Certification Matter? – Who Does Audits? – Current Issue – How are Auditors Accountable?

Sam Siewert 18