1 Staffless SQA An Effective SQA Alternative Unmesh Gundewar E-Mail: [email protected]...

1

Staffless SQAAn Effective SQA Alternative

Unmesh GundewarE-Mail: [email protected] 25, 2003

EMC and the EMC logo are registered trademarks and "where information lives" is a trademark of EMC Corporation.

John Brackett

Changed the line break on the title and centered it in the area

2December 9, 2002 Copyright © 2002 EMC Corporation. All Rights Reserved.

Agenda Introduction

SQA Purpose, Goals, Roles and Responsibilities

Typical SQA Approach and Issues

EMC’s Strategic Approach for Staffless SQA SQA Coverage (Level 2 and Level3)

Challenges

Lessons Learned

Summary


To meet the needs of today’s tumultuous marketplace, delivery of products and services with superior quality is the key to corporate success and survival. As a result, to improve quality, many corporations are using CMM framework for process improvement.

“Software Quality Assurance is the least frequently satisfied Level 2 Key Process Area”

— August 2002, Process Maturity Profile published by Software Engineering Institute (SEI)

Introduction


SEI Capability Maturity Model

Level Focus Key Process Areas Result

5Optimizing

4Managed

3Defined

2Repeatable

1Initial

Continuous processimprovement

Product and processquality

Engineering process

Project management

Heroes

Defect preventionTechnology innovationProcess change management

Process measurement andanalysisQuality management

Organization process focusOrganization process definitionPeer reviewsTraining programIntergroup coordinationSoftware product engineeringIntegrated software management

Project planningProject trackingSubcontract managementQuality assuranceConfiguration managementRequirements management

Productivity& Quality

Risk

Carnegie Mellon University - Software Engineering Institute


The CMM Structure

Maturity Levels

Key Process Areas

Common Features

Key Practices

ProcessCapability

Goals

Implementation orInstitutionalization

Infrastructureor Activities

indicate

achieve

address

describe

contain

organized by

contain


Expanding CMM Structure

5 CMM maturitylevels

0-7 KPAs per maturity level

5 common features per KPA with 1-n practices each

2-4 goals per KPA

= 18 total KPAs

= 52 goals

= 316 total key practices


Software Quality Assurance GoalsPurpose Provide management with appropriate visibility into the

processes being used by the project, and of the products being built.

Goal 1 Quality Assurance activities are planned.

Goal 2 Adherence of products and activities to the applicable standards, procedures, and requirements is verified objectively.

Goal 3 Affected groups and individuals are informed of quality assurance activities and results.

Goal 4 Noncompliance issues that cannot be resolved within the project are addressed by senior management.


Ensure that the organizational standards, practices, and procedures have been selected and followed.

Identify all deviations from the agreed upon processes as early as possible

Assist project managers in developing quality assurance Software Quality Assurance Plan

Promote best practices

Conduct audits and check all software lifecycle work products

Escalate noncompliance and non-resolvable issues

Periodic reporting to affected groups

SQA Roles and Responsibilities

John Brackett

I added software project management to the first sub bullet, since I regard that as one of our key goals.

John Brackett

I tried to make things consistent in saying that a team would develop both a Program plan and a software project management plan.


Approach:

Establish Group to Focus on SQA

Issues:

Low Morale

Funding Issues

Policing/Enforcement

Lack of Respect

Typical SQA Approach and Issues


Program managers lead cross-functional teams and use corporate processes to drive consistency, product quality, business readiness, and speed time to market for product and service solutions.

EMC’s Strategic Approach to Staffless SQA


Leads one or more cross-functional teams to drive end-to-end product and service solutions for EMC customers.

Defines scope, develops cross-functional program plans, drives program to completion, and ensures that the product is delivered on time and meets the EMC quality goals.

Leads the cross-functional team to resolve program issues. Escalates issues to the appropriate senior management team when needed.

Role of a Program Manager

EMC’s Strategic Approach to Staffless SQA


Organizational Software Development (SDP) Process– Corporate Lifecycle (LC 300)

– Software Development Lifecycle

– Software Development Process and Checkpoints

– Professional Development Process and Checkpoints

Readiness Management Team (RMT) Process

Escalation Process

Compliance Report

Quality Metrics

EMC’s Strategic Approach Implementation


EMC’s Product Lifecycle

A Corporately-Accepted Process

Guideline for: Product Creation Product Testing Delivery and Release Field Support Services Replacement/End-Of-

Life

A Flexible Model for Program Management Teams (PMT’s)

Phase 0Feasibilty

Phase 1Design

Phase 2Qualification

Phase 3AvailabilityReadiness

Phase 4Availability

Phase 5End of Life

New Startor

Major Revision

Minor FeatureEnhancements

GA


Software Development ProcessActivities and Checkpoints

0 2 3 4 5 7

SDS

EFS

PDS PEL

PSARelease SpecificPRD

Kic

koff

GA1

CodingPrototypeQA Tracking

Unit Testing QA TestingWaterfallExample

Int

STP Test Cases

6Time

SPPp SPPc

SRS

0 Kickoff1 Preliminary Requirements Review2 Critical Requirements Review3 Preliminary Planning Review

4 Critical Planning Review5 Functionally Complete6 Beta Ready7 Release to Operations


Service Offering Development Overlay and Checkpoints

31 4Publish!

PDS

Release SpecificPRD

NPC0

RTO

Time

1 (TSG) PS Positioning & Impact Review2 Service -Dev Validation of SDR/SCR3 Final Approval to build Offering4 Field Readiness Review

PS Positioning Impact

SDRDevelop Build

Monitor EngagementsValidate

t0 t1

BetaReadyProduct

2Positioning & Bundling

PEL

65

SCR

LCR

Validate

LC300Phase 3

LC300Phase 2

LC300Phase 1

LC300Phase 0

5 Publication Review

6 Monitoring initial release of Servicet – Times will vary depending upon size of product

and type of Service

t2


Functional Group Lead Deliverables CommentsProg/Prod Mgmt {Name} Program Schedule

{Name} PRD{Name} Packaging {physical, market bundles, Models etc}{Name} Positioning{Name} 3rd Party Vendors{Name} PDS

Development {Name} Product {includes Beta, Demos, all things product related}{Name} Knowledge Transfer {Includes customer docs, white papers, web casts, best practices}{Name} Train the Trainer

Marketing {Name} Marketing Collateral{Name} Pricing{Name} Reference/Testimonial

Training {Name} Training Plans {Include PS, Customer, Service, Support, SE & Sales}{Name} Course Development{Name}

Global Services {Name} SVT{Name} Co-operative Support {If needed}{Name} PS Offering

Sales {Name} Readiness {includes all things required to be ready to make this happen}{name} Channels {any new/specific channel activity}{Name} Volume {post GA tracking}

Manufacturing {Name} Supply Base Mgmt{Name} Manufacturing Plan

Readiness Management Process{Full Product Name V#} Readiness Dashboard {Date}

GAVarious Milestones Beta Starts SVT

DateDateDate Date

Program Manager: {Name} Current LC300 Phase(s): {Phase #(s)}

Steady State

Post GA


Milestone

Phase 0

Tentative Dates

{date}

Phase 1

PDS Dates

{date}

Phase 2

{date}

Phase 3

{date}

Estimate of Next Phase

PRD Sign off

NPC Phase 0 Presentation

PDS Sign off

BRT Product Overview

Beta Start

NPC Go to Market Presentation

PS Offering

Marketing Collateral

Field, Help Desk, Sales trained

SVT begins

SVT With MASTER begins

RTM

GAQuality, Service, Unit Tracking begins

Readiness Management Process (Continued)

{Product Name V#} Schedule Delta {Date}

Explanation:

{What are the factors that contributed}

Impact:

{What does this mean}

{How has the plan been modified?}


Readiness Management Process (Continued) {Product Name V#} Program Overview {Date}

Highlights:

{what are the items to be excited about}

{what is going very well}

{nine bullets max}

Lowlights:

{Where are we NOT performing up to our capability – past events}

{These may map directly to an ASK}

{nine bullets max}

Risks/Issues:

{Facts - Not a value judgment}

{Acknowledgement of the Risks – Future items}

{These may map directly to an ASK

{nine bullets max}

Ask:

{what does the Cross Functional Team Require from Management?}

{nine bullets max}


ProductManager Tech Pubs

G/SCustomerServices

CorporateQuality

Corp MktgMarkets &Products

PMTPMT

Corp MktgAlliance &ChannelPartners

G/SProfessional

Services

Leadership Meetings WorkflowEscalation Process

Product Management andProgram Management

RMT

ProgramManager

ProgramManager

PMT

Sr. LeadersCross-Functional

Organization

FunctionalRepresentatives

LeadersCross-Functional

Organization

Weekly Executive Report

Weekly Executive Report

Cross-Functional Representatives


Late/ Escalation6%

At Risk8%

Missing6%

Future15%

On Schedule21%

Done or NA44%

Compliance MetricsSW Development Checkpoint Status – MM/DD/YY

Total: 34 Product Releases

CheckPoint Breakdown:

Exec Summary Reds:(Products Below with Arrow)

7 Programs with Missing Checkpoint Items:• 2 Program Schedules • 5 Not Scheduled for Reviews

5 Programs have a ‘Red’ Checkpoint Item: Program A•Program B• Program C Program D• Program E

Program Managers’RMT Dashboards


Daily Triage for Product A GA Issues

147

86

48

12 120 00

100

200

300

5-S

ep6-

Sep

7-S

ep8-

Sep

9-S

ep10

-Sep

11-S

ep12

-Sep

13-S

ep14

-Sep

15-S

ep16

-Sep

17-S

ep18

-Sep

19-S

ep20

-Sep

21-S

ep22

-Sep

23-S

ep24

-Sep

25-S

ep26

-Sep

27-S

ep28

-Sep

29-S

ep30

-Sep

1-O

ct2-

Oct

3-O

ct4-

Oct

5-O

ct6-

Oct

7-O

ct8-

Oct

9-O

ct10

-Oct

11-O

ct12

-Oct

13-O

ct14

-Oct

15-O

ct16

-Oct

17-O

ct18

-Oct

19-O

ct20

-Oct

21-O

ct

New Resolved A.1 Open A.0 Open


Product B Open Problem Tracking (OPT) and Backlog

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

7-Ju

n

14-J

un

24-J

un

1-Ju

l

8-Ju

l

15-J

ul

22-J

ul

29-J

ul

5-A

ug

12-A

ug

19-A

ug

26-A

ug

2-S

ep

9-S

ep

16-S

ep

23-S

ep

30-S

ep

7-O

ct

14-O

ct

21-O

ct

28-O

ct

Severity 1 Severity 2 & 3 New Sev 1 New Sev 1 (Moving average)


Product A.1 Test Trends vs A.0

25%

18%

42%

22%

45%49%

44% 44%

70%

27%

63%

71%

79%

88%

98% 99% 99% 100% 100% 100%

75%

86%

81%78%

83% 83%

92% 92%89% 88%

92% 92% 93% 94% 95%93%

98% 98% 98% 99%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

7-Ju

n

14-J

un

24-J

un

1-Ju

l

8-Ju

l

15-J

ul

22-J

ul

29-J

ul

5-A

ug

12-A

ug

19-A

ug

26-A

ug

2-S

ep

9-S

ep

16-S

ep

23-S

ep

30-S

ep

7-O

ct

14-O

ct

21-O

ct

28-O

ct

Test Coverage % Tests Passed A.0 Test Coverage A.0 % Tests Passed


Software Quality Assurance CoverageRequirements Management Key Process Area

KPA Verification 3 ComplianceRM

Ve 3

The activities and work product reviews and/or audits are conducted to manage requirements and results are reported.

Product requirements (PRD) are approved and signed by affected groups (Checkpoint 1).

Changes to requirements are approved by affected groups (Requirements Management Process).

Commitments are negotiated and agreed to by the affected groups.

Italic compliance is in progress


Software Quality Assurance CoverageProject Planning Key Process Area

KPA Verification 3 ComplianceSPP

Ve 3

The activities and work product reviews and/or audits are conducted for project planning and results are reported.

Preliminary project plan is approved (Checkpoint 3).

Standards are used for preparing the project plan.

The content of the software development plan is reviewed.


Software Quality Assurance CoverageProject Tracking and Oversight Key Process Area

KPA Verification 3 CompliancePTO The activities and work

product reviews and/or audits are conducted for project tracking and oversight and results are reported.

Revised project plan is approved [Critical Planning

Review] (Checkpoint 4).

The activities for revising the software development plan is reviewed with affected groups.

The content of the revised software development plan is reviewed.


Software Quality Assurance CoverageConfiguration Management Key Process Area

KPA Verification 4 ComplianceSCM

Ve 4

The activities and work product reviews and/or audits are conducted for configuration management and results are reported.

CM responsibilities are assigned and approved. A build plan exists.

SCM standards are applied for changes to the software.

Different versions of the software exist.


Software Quality Assurance coverageSoftware Quality Assurance Key Process Area

No. Commitment ComplianceCo 1 The project follows

organizational policy.Quality assurance function is in place.

Program manager, engineering group, and other affected groups have a reporting channel to senior management.

Senior management reviews the compliance activities weekly.


Software Quality Assurance CoverageSoftware Quality Assurance Key Process Area

No. Ability ComplianceAB 1 SQA Group exists. Program manager owns SQA

function.AB 2 Adequate resources and

funding are provided to conduct reviews.

Reviews are an integral part of the engineering organization’s processes. Program managers are chartered to ensure that checkpoint reviews are planned and conducted.

AB 3 Individuals are trained to perform quality assurance activities.

Program managers are trained as well as coached through RMT process.



No. Activities Compliance

Act 1 SQA activities are planned.

Checkpoint reviews are planned in the software project plan.

Act 2 SQA activities are performed in accordance with the plan.

Checkpoints reviews are conducted and affected groups are involved for the reviews.

Act 3 Software Development Plan is reviewed.

Program manager works closely with the software development project manager.




No. Activities Compliance

Act 4 Software Engineering activities are reviewed for compliance.

Covered in the verification of other Key Process Areas.

Act 5 Software work products verified for compliance.

Covered in the compliance metrics.

Act 6 The results of review activities are periodically reported to affected groups.

Communicated weekly in the leadership management meetings.



No. Measurement Compliance

Me 1 Measurements are made and used to determine the cost and schedule status of the SQA activities.

No measurements are made because the roles and responsibilities of program managers include SQA.



No. Verification Compliance

Ve 1 SQA activities are reviewed periodically with senior management.

Weekly leadership meetings cover compliance activities.

Ve 2 SQA activities are reviewed with the project manager.

Program manager reviews work products with the project manager.


Software Quality Assurance CoverageSoftware Product Engineering Key Process Area

KPA Verification 3 Compliance

SPE

Ve 3

The activities and work product reviews and/or audits are conducted for software product engineering and results are reported.

Readiness and completion criteria for each software engineering task are satisfied.

Required testing is performed. Tests satisfy acceptance criteria, as documented in the software test plan.


Software Quality Assurance CoverageSoftware Product Engineering Key Process Area

KPA Verification 3 ComplianceSPE

Ve 3

The activities and work product reviews and/or audits are conducted for software product engineering and results are reported.

Problems and defects detected are documented, tracked, and addressed.

Tracing of the allocated requirements through the software requirements, design, code, and test cases is performed.



Software Quality Assurance CoverageIntegrated Software Management Key Process Area

KPA Verification 3 ComplianceISM

Ve 3

The activities and work product reviews and/or audits are conducted for integrated software management and results are reported.

Software development plan is developed using organizational standards.

Organizational process database to support for planning, estimating, and tracking is not available.



Software Quality Assurance CoveragePeer Review Key Process Area

KPA Verification 1 CompliancePR

Ve 3

The activities and work products reviews and/or audits are conducted for Peer Reviews, and results are reported.

Peer review process is developed and peer reviews are planned.

Peer review process is being implemented.



Resistance to change

Need for individual group processes

Compliance of individual group processes

Tracking and oversight

Too many processes

Impact on business

Improving processes

Challenges

John Brackett

I reworded the bullet on "tracking and planning more effectively" and moved it up on the slide.


Program managers are key part of the solution

Executive forums helped program managers to achieve EMC goals Faster escalation, resolve issues, assign resources

Best practices shared among program managers

Quicker feedback to program management

Sponsorship and coaching

Lessons Learned

John Brackett



“Using organizational and program management standards, program managers drive end-to-end product and service solutions by breaking down organizational barriers, anticipating problems before they occur, and speeding the delivery of products and service solutions.”

Summary

— Peter Conway, V.P. Software Product Management, EMC

John Brackett



Closing Slide

“The best way to predictpredict the

futurefuture is to crecreaatete it.”

Peter DruckerPeter DruckerPeter DruckerPeter Drucker

1 Staffless SQA An Effective SQA Alternative Unmesh Gundewar E-Mail: [email protected]...

Documents

Transcript of 1 Staffless SQA An Effective SQA Alternative Unmesh Gundewar E-Mail: [email protected]...