© Vector Software

The Industrial Internet of Things (IIoT)Why Software Quality Matters

John J. Paliotta CTO, Vector Software

© Vector Software

Industrial

Consumer

The (Industrial) Internet of Things IIoT

Transportation

Factory Automation

Smart Buildings

© Vector Software

Where Does Software Fit?

Internet of Things

© Vector Software

What Changes will IoT Bring to Software

Lots of Consumer Grade Software will Suddenly be the Critical Path• Re-use of legacy code bases, cameras for autonomous driving

• Cellular/Internet connectivity to cars for SOS

The Quantity of Embedded Software will Increase Exponentially• Autonomous software that integrates unobtrusively

• Heart Function, Blood Sugar and Blood Pressure Monitoring

Intelligence will be Deployed to the End Points• IoT will not be a collection of dumb sensors

• Complexity and Data Volumes are too large for central control

© Vector Software

Is Industry Prepared for IoT

The Software Quality Challenge

Improving Software Quality

© Vector Software

We Know how to Build Software Efficiently … Right?

Cambridge University research estimates the global cost of debugging software to be $312 Billion annually Source: Cambridge University, January 8, 2013

Short Term Maintenance costs are generally double the original development and testing costs

According to Gartner, after 15 years the original development cost is 8% of the total lifetime cost of ownership. 92% is maintenance.Source: Gartner, Inc., "The Four Laws of Application Total Cost of Ownership," Andy Kyte, April 3, 2012

© Vector Software

So maybe there’s SOME Room for Improvement

© Vector Software

Is Industry Prepared for IoT

The Software Quality Challenge

Improving Software Quality

© Vector Software

Software is Different

Best Feature of Software: It’s Easy to Create and Change

Worst Feature of Software: It’s Easy to Create and Change

Building a Physical Product

Building Software

Prototype Ship

Design PrototypeDesign Factory

Manufacture Test Ship

© Vector Software

Market Pressure

Release cycles have shrunk from years to days to hours• We have to get these new features to customers

There is more competition than Ever• The barriers to entry for the software business are low

Trade-offs made during development• We Can Fix That in the Next Release …

GOOD

FAST CHEAP

NOTPOSSIBLE

EXPENSIVEAND GOOD

SLOWAND GOOD

LOWQUALITY

© Vector Software

Common Software Development Workflow

Independent Process• Designers, Developers and Testers work in isolation

Periodic Testing • Performed after Development

• We attempt to “bolt-on” quality at the end of the cycle

Long Feedback Loops• Bugs are Fixed weeks after they are Introduced

• Defective products are reworked

© Vector Software

The Cost of Independent Processes

© Vector Software

The Cost of Periodic Testing

Bugs get found late in the release cycle• When many tests are run for the first time

Bug fixing is costly• Context switching by Developers

• Long Feedback Loops before they know about failures

New features get released slowly• Features get Stuck in QA

• Hard to solve regressions

© Vector Software

The Cost of Bug Fixing

Cambridge University research estimates the global cost of debugging software to be $312 Billion annually Source: Cambridge University, January 8, 2013

© Vector Software

Is Industry Prepared for IoT

The Software Quality Challenge

Improving Software Quality

© Vector Software

Improved Software Development Workflow

Independent Process Integrated Processes• All team Members contribute to Quality

Periodic Testing Continuous Testing• Quality is built-in at each stage of development

Long Feedback Loops Immediate Feedback• Bugs are Fixed when they are introduced

© Vector Software

Make Testing Easy: Test Collaboration

Make Tests Sharable• There cannot be flavors of tests owned by separate groups

• Everyone has something to contribute

Remove Infrastructure Limitations• Virtualized Test Environments

• Labor is your most expensive resource

Tests Must be Small and Easy to Run• Anyone can run any test at any time

Make Results Simple to Understand • Test pass or fail, no thinking needed

• Failed tests must be debuggable

© Vector Software

Build In Quality

Test Each Component Before Integration• Test the published API of all components

Build Tests for APIs and Requirements• Test for Correctness

• Verify that all requirements have tests

Ensure Tests are Complete• Use Code Coverage to verify adequate testing

© Vector Software

Test Each Component in Isolation

Component Testing, not Unit Testing• Define a component logically

• Might be a function, a file, or a whole subsystem

Expose Tests to all stakeholders• Test are not Developer, Integration, or System Tests

• All tests should be run by all stakeholders

Refactor Tests over Time• Get in the habit of throwing old tests away

• Tests that fail a lot imply fragile code / tests

ComponentUnder Test

StubbedDependents

API

© Vector Software

Build Tests For APIs and Requirements

Where do you find more bugs• Component Testing or System Testing?

Tests written against the Implementation don’t find bugs• The tests prove that the code does what it does

• Same omissions get made in the tests as are made in code

Benefit of API testing• How can I call this API versus, how should I call it

Benefit of Requirements Traceability• Code Change Implies Requirement Change

• Requirement Change Implies Code Change

Requirement

CodeTest

int algorithm (int p1, int p2) ;

© Vector Software

Ensure Testing is Complete

Code Coverage has an extremely high Return on Investment (ROI)• 100% Coverage is not the goal, complete testing is the goal

© Vector Software

Implement Continuous Testing

Easy• Easy to build new tests

• Easy to understand existing tests

• Easy to run and interpret results

• Easy to debug failing tests

Fast• Shorten the feedback loop

Automated • Push button testing with no manual setup or steps

• Anyone can run any test at any time

© Vector Software

Make Testing Fast: Change Based Testing

What is Change Based Testing• Intelligently choose tests to run based on code

changes

Why Change Based Testing• Running all tests takes too long

• We can’t run 3 days of testing for a one line change

• Testing multiple source changes together makes it hard to find the source of a regression

Benefits• Shorter feedback loops on test breakages

• Faster release cycles for new features

Software TestsSourceCode

Source Change Test Cases to Re-Run

© Vector Software

Make Testing Automated

Eliminate Manual Testing• 100% of testing should be automatable

• Design for testability

Remove Infrastructure Limitations• Virtualized Test Environments

• Labor is your most expensive resource

Push Button Test Execution• Pre-conditions, tests steps and Post-conditions

• Hardware / Virtualization Setup

• Tests must support debugging

TestMaster

Developers Developers

TestNode

TestNode

TestNode

TestNode

© Vector Software

Prevent Bugs

Identify Quality Metrics• Logic Complexity, Interface Complexity,

Comment Density, Testing Completeness

Analyze Data• We have plenty of data, but what does it tell us

• Listen to the story

Use Data to Improve the Process• Without a defined process, there is no chance

of improvement

• Every process needs a quality gate

© Vector Software

Identify Quality Metrics

Logic Complexity• How many unique paths exist through each function

Interface Complexity• How many data item “inputs” exist for each function

Comment Density• Comment Lines as a percentage of Total Lines

Testing Completeness• What level of code coverage have we achieved

• How many tests are linked to requirements

© Vector Software

Analyze Quality Data

Improvement is not possible without Goals• Metrics have the Ability to Motivate

• Choose achievable goals

Progress is not measurable without Data• Data must correlate to goals

• Average Complexity

• Comment density

• Gathering and publishing must be automatic

Trends are the key, Incremental Improvement• The data values are not important

• Incremental Improvement

© Vector Software

Implement Quality Gates

Without gates on behavior code bases become buggy• Software development should be engineering not art

With too many gates nothing gets done.

Build Documented workflows for each Activity• SCM commit, requires a clean status analysis report

• Branch cannot go to QA until all existing tests pass

Quality Gates Must be Automated• Except for peer review – which still has great value

© Vector Software

Conclusion

The Quantity and Criticality of Software is Increasing

Traditional Workflows are not Sufficient to Meet the Challenge

Quality must Replace Time to Market as the Goal

© Vector Software

Questions and Answers

!?

© Vector Software