Levers for changing software delivery - v1.2

Development model Improvements Process Concurrency Challenges Summary

Levers for changing software deliveryv1.2.0

Michiel Kalkman


System development scaling

Figure 1: Solo efforts can redefine space asthey go along

Figure 2: Team efforts need a sharedunderstanding of space

Image sources: wikimedia.org a, b

https://commons.wikimedia.org/wiki/File:Dump_truck_-_panoramio.jpg

https://en.wikipedia.org/wiki/File:Eastern_Freeway_Belford_St.jpg


Development model


Our reference phases

Concept Inception Construction Release Production Disposal

Feedback

IdeationVisionFeasibility

DesignFundingResourcing

AnalysisDevelopmentTesting

TestingAcceptanceDeployment

OperationsSupportAssessment

Data conversionUser migrationRetirement

Figure 3: Reference Lifecycle - Typical fixed-resource project


General Observations

Phases are ordered sequentiallyWith feedback loops to register defects, learnings

Phases have distinct functionsTransitions between phases require a context switch or handover

Work in earlier phases affects later phasesLater phases affect earlier phases via feedback


General Goals

We want to,

Minimise feedback volume, rate and processing timeMinimise the cost of phase transitionsMinimise the volume of non-feature work passing through the cycleMaximise the speed of work passing through the cycleMaximise the rate of work passing through the cycle


Improvements


Project Phases

Delivery

Construction Release Production

Feedback

Figure 4: The DevOps Cycle


Identified issue example

ScenarioAnalytics show us that an increasing number of customer journeys end inFAILED or ABORTED states. Root cause analysis has determined that themajority of these cases are caused by valid, but less common, sequences ofinteractions that are not handled correctly, are missing code to handle themor are using obsoleted code that cannot execute.

ActionRaise test coverage requirements at unit and integration levelBlock a release unless it meets the coverage requirements


Add Quality Gates

Delivery

Construction ReleaseQuality Gate ProductionQuality Gate

Feedback

Figure 5: Add quality gates

BenefitsLower feedback volume and rate

CostRaise the cost of phase transitions


Quality Gate examples

Unit test coverageLoad or soak tests of Release before ProductionAutomated static or dynamic analysis of code policies. Example,

Security patternsMissing codeCyclomatic complexity


Identified issue exampleScenario

Our support team is unable to keep up with the backlog of issues, whichkeeps growing along with the mean time to issue resolution. Analysis thatmost of the time spent by the support team goes to collecting event logsfrom different systems based on timestamp information and other attributesto reconstruct transactions and individually engaging engineers to providecontext.

ActionAll log events will be accessible from a single systemAll log events will be structuredAll systems will implement correlation identifier generation and propagation forservice transactionsCorrelation identifiers will be added to log events


Increase Defect Traceability

Delivery


Feedback

Traceability TraceabilityTraceability

Figure 6: Tracing

BenefitsLower feedback processing time (support and incident handling)Faster remediation (lower lead time)


Traceability examples

Centralised logging with propagated correlation IDsDistributed tracing implementationUser activity measurement connected to backend tracingSelf-service dashboarding and alert managementChange visibilityBusiness metric generation and reporting


Option : Improve Transition Throughput

Delivery


Feedback

Figure 7: Concurrent streams of work

CostIncreases phase transition cost (handover, interface, gates)

BenefitsIncrease the volume of work


Option : Improve Phase Output

Delivery


Feedback

Figure 8: Higher quality output

BenefitsLower feedback volumeLower the volume of non-feature work


Combined model

Delivery

Construction ReleaseQuality Gate ProductionQuality Gate

Feedback

Traceability TraceabilityTraceability

Figure 9: Combined improvements


Process


Cost of defects

1

10

100

Preven�on

Correc�on

Failure

Figure 10: 1-10-100 Rule


Example : Cost scenario of feature rollout

Outcome Value CostSuccess Positive Planning time

+ Execution time

Rollback Zero Cost of success+ Recovery time+ Opportunity cost

Failure Negative Cost of rollback+ Emergency time+ Lost function cost+ Reputation cost


Early detection

Defects found early are less costly to remediate

If we want to lower the cost of defect remediation, there are two key drivers in thisobservation,

1 Early detection of defects2 Fast remediation of detected defects


Right Shifted

Design Implemention Test

Figure 11: Testing happens last


Right Shifted - Step details

Feature Test

Design Implemention Design Implementation Execution

Figure 12: Examination of testing steps - these also contain design and implementation


Right Shifted - Step contexts

Feature Test


Business contextVariable time

Technical contextVariable time

Fixed timeCan be automated

Business contextVariable time

Technical contextVariable time

Figure 13: Steps with context


Context mapping

Feature Test


Business Context Technical Context

Figure 14: Right Shifted - late duplication of contexts


Flow independent use cases

Business Context Technical ContextValidation Context

Design FeatureDesign QA Criteria Implement FeatureImplement TestsExecution

Tester DeveloperAnalyst

Figure 15: Feature development actors, cases, contexts


Ideal development

Construction Phase

Quality Gate

Business Context Technical Context

Feature Design Feature Implementation

Test Design Test Implementation Execution

Figure 16: Left Shifted - early alignment and no late duplication


Concurrency


Minimise external dependencies

Feature team

Load testing team

Release team

Infra team

Monitoring team

Security team

Feature team

Feature team

Feature team

Figure 17: One team per non-featurecapability

DevDev

Tester

SecurityPlatform

DevDev

Tester

SecurityPlatform

DevDev

Tester

SecurityPlatform

Figure 18: Capabilities embedded in teams


Goals for concurrency

Reduce idle time, maximise productive timeReduce delivery risk

Break one large project into small featuresFail fast - detect problems early

Avoid single points of blockage/stallingConcurrent, isolated streams of work

Capture value early with fast exposure of features to production


Independent Feature Development

Work Stream 1 - Feature A

Work Stream 2 - Feature B

Itera�on 1

Figure 19: Testing V-Model


Independent Feature Development

Requirements,

Up-front planningCoordination activitiesContinuous monitoring and adjustmentDedicated investment in process management


Multiple Development Streams

Test execu�on

Test defini�on

Defect raised

Feature delivered

Stre

ams

of w

ork

Time

Figure 20: Parallelised, Left-Shifted Delivery


Challenges


Technical Debt

"Ship now,deal with it later"

"What is layering?"

"Now we know howwe should have done it"

"We don't have �mefor design"

Figure 21: Martin Fowler’s quadrants of tech debt acquisition


Technical Debt Types

1 Eventually paid off2 Sustainable3 Compounding


Technical Debt

CausesTime-to-market drive - Delivery pressure to meet value pointTeam ramp up, learning curve - Immature engineering practicesProgressive learnings - Mismatch between technology and applicationProduct repositioning - Mismatch between platform and application

EffectsOpportunity costCompound interest - debt load increases over timeDevelopment overhead - Any change is more complex than necessaryTesting overhead - Complexity doesn’t increase linearlyStill has to be maintained - Any maintenance is more complex than necessary

Industry average 15 to 50 errors per 1000 delivered LoC(Code Complete, Microsoft Press)


Technical Debt

Key PointsTech debt is taken on for a reason - usually related to creating valueTech debt itself is not good, but having it can be a good signalTech debt itself is not costly, the cost is in carrying the debt

ActionsProvide a decision making framework for taking on debt

And paying it off (or not)Make debt visibleTrack the cost of carrying debt


Pressure of re-work and technical debt

Figure 22: Cumulative flow of re-work andtechnical debt pressures

Re-work is lost opportunityInterest on technical debtcompoundsTrack both new debt and its interestpressureTrack re-work and cycle times


Pre-project feedback

Pre-Project Project

Post-Project

Concept Inception

Feedback


Feedback

Disposal

Figure 23: Concept feedback

No well-defined feedback mechanismsNo clear outcomesPoliticsMove more decisions to Construction phase pre-project


Non-feature budget expenditure

Process changes require significant expenditure of resources. Find secondary benefitsthat address pain points or have a financial impact to lower the cost.

Improving Traceability for Defects can also be used to lower to cost of SupportAutomation and parallelisation can be used to scale DevelopmentIf a form of compliance is on the roadmap, implementing automation for AccessControl, Change Management and Control Monitoring will reduce that cost


ROI of Manual vs Automated Testing

Figure 24: By stage

There is always an upfrontinvestment with automationUnderstand where the crossover withmanual testing isMake this visible to stakeholdersKeep track of and report on thevalue of automation as the projectprogresses


Value of work by stage

Figure 25: By stage

Get completed features toproduction as soon as possible, noexcusesUse feature flags to gate activationif necessary


Shift Left = Process change + automation

What HowDefine test requirements early Engage test team at each step

Make testing everyone’s responsibilityExecute tests as soon as possible Gate progress at each step on testingRapidly act on test defects Implement consistent traceability, reportingMinimise test execution barriers Automate tests, accessible to allMinimise test execution time Automate tests, make test runs cheapHave test execution consistency Automate tests, make test runs predictable


Summary


Summary

Figure 26: Effect of changes across concerns


Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)

You are free to:

Share — copy and redistribute the material in any medium or formatAdapt — remix, transform, and build upon the material

The licensor cannot revoke these freedoms as long as you follow the license terms.

Under the following terms:

Attribution — You must give appropriate credit, provide a link to the license, andindicate if changes were made. You may do so in any reasonable manner, but notin any way that suggests the licensor endorses you or your use.

NonCommercial — You may not use the material for commercial purposes.

https://creativecommons.org/licenses/by-nc/4.0/

https://creativecommons.org/licenses/by-nc/4.0/

Levers for changing software delivery - v1.2

Documents

Transcript of Levers for changing software delivery - v1.2