NASA Langley Research Center
Software Process Improvement Initiative
(SPII)
Findings Presentation
October 27, 1997
NASA Langley Research Center
CornerStone Findings Presentation
CornerStone Overview Findings Next Steps
NASA Langley Research Center
CornerStone Overview
NASA Langley Research Center
CornerStone Goals The CornerStone Phase is the foundation of LaRC’s
overall Software Process Improvement Initiative The goals of the CornerStone Phase are:
Develop a plan to improve LaRC’s software development practices Identify current state of software development at
LaRC Identify current best practices used in software
development at LaRC Develop a High Performance Model for LaRC’s
software development activities (incorporates the appropriate elements of the Capability Maturity Model, ISO 9000-3, Strategic and Quality Framework, and Baldrige Award Criteria)
Obtain management’s support, complete with resources, to implement LaRC’s Software Process Improvement Plan
NASA Langley Research Center
Software Process Improvement Initiative (SPII) Goals
Improve the work environment for LaRC’s software community, leading to higher morale and increased productivity
Develop sustainable mechanisms for continuous improvement in the productivity and quality of software developed across LaRC
Increase customer satisfaction with LaRC software products
NASA Langley Research Center
CornerStone Activities
Lay the Foundation Establish Infrastructure
Plan for the Assessment CornerStone Planning and Validation
Baseline Workshop Training Customer Workshops Supplier Workshops Follow-Up Interviews (as needed) Best Practices Documentation Analyze Workshop Results Prepare and Present Findings
Plan Define LaRC SEPG Develop SPI Plan Review SPI Plan with Sponsors Present SPI Plan
Implement Improvements …
NASA Langley Research Center
CornerStone Team Members
Norma Campbell, RTG/FDCD Victoria Chung, IOG/ISSD Michael Holloway, RTG/FETD Chuck Niles, IOG/FSED Pam Rinsland, IOG/AESDPat Schuler, IOG/ISSDJim Townsend, RTG/FMADJim Watson, OSEMA/OMASue Voigt, SASPG/SSCD
Consultant: Cindy Torpey, ChangeBridge Inc.
NASA Langley Research Center
CornerStone Scope
Centerwide involvement Organizations involved in software
management, development, and maintenance
Customers of software projects 101 civil servants and contractors
interviewed
NASA Langley Research Center
CornerStone Principles
Start with a process framework Baseline current state (not audit)
Conduct interviews and discussions Observe strict confidentiality
Involve senior management Work as a Centerwide team Focus on LaRC needs
NASA Langley Research Center
Capability Maturity Model
Initial
Repeatable
Defined
Process is informal and unpredictable
Project management system in place; performance is repeatable
Software engineering and management processes defined and integrated
Product and process are quantitatively controlled
Time/$/...
Time/$/...
Time/$/...
Optimising Process improvement is institutionalised
Time/$/...
Time/$/...
Level Process Characteristics Predicted Performance
Managed
Pro
bab
ilit
y
Ta
rge
t N
-2
Pro
bab
ilit
y
Ta
rge
t N
-y
Pro
bab
ilit
y
Ta
rge
t N
-x
Pro
bab
ilit
y
Ta
rge
t N
-1-a
Pro
bab
ilit
y
Ta
rge
t N
NASA Langley Research Center
Capability Maturity Model Repeatable / Level 2 (All Key Process Areas covered in
interviews ) Requirements Management Software Project Planning Software Project Tracking & Oversight Software Quality Assurance Software Subcontractor Management Software Configuration Management
Defined Level (Selected Key Process covered in interviews) Training Program Software Product Engineering Intergroup Coordination Peer Reviews
NASA Langley Research Center
CornerStone Approach….
Baseline current state (snapshot in time) …some improvements are already initiated
Baseline included all areas which impact the software development group … some are under our control, others will require a
coordinated effort Not all points are at the same detail
…some are very specific, other are more general Processes were assessed, not specific groups
…especially when we discuss SQA and Training Program
Identify areas for improvement …not specify how to
NASA Langley Research Center
CornerStone Findings
Key Process Area
Purpose and Description
Best Practices
Improvement Opportunities
NASA Langley Research Center
Requirements Management
Purpose Establish a common understanding of the customer’s
requirements that will be addressed by the software project Description
Establish and maintain an agreement with the customer on the requirements for the software project
The "customer" is a system engineering group, TAG, a project office, another internal organization, or an external customer
Agreement covers both the technical and nontechnical (e.g., delivery dates) requirements
Agreement forms the basis for estimating, planning, performing, and tracking the software project's activities throughout the software life cycle
NASA Langley Research Center
Requirements Management
Best Practices Requirements are prioritized and managed
according to priority Developer works with customer to
interactively draw out requirements Operational Concepts Documents help in
understanding the requirements (some projects put on the web for increased visibility)
Test cases traceable back to requirements Explicit contractor task to capture and
manage requirements Rapid prototyping to validate requirements
with customer
NASA Langley Research Center
Requirements Management
Improvement Opportunities Importance of requirements is not realized Requirements are neither defined early enough, nor refined far enough Inadequate resources dedicated to requirements management Software requirements do not track to system requirements Don’t know how to adequate document requirements Customers are not trained in requirements generation Changes in requirements handled by overtime masks over-commitment Requirements creep is not managed nor controlled (cost is not
understood) No established policy to guide projects in requirements management Role and responsibility of systems analysis is not clearly understood Requirements are not consistently documented and reviewed with the
customers Segmentation of responsibilities between researchers and software
developers leads to uncertainty in product functionality
NASA Langley Research Center
Requirements Management
Consequences Project schedule and cost are significantly
increased when requirements are inadequately defined and documented
Rework is common due to changing requirements
Customer satisfaction is decreased when their requirements are not met
NASA Langley Research Center
Software Project Planning
Purpose Establish reasonable plans for performing
software engineering and managing the software project
Description Develop estimates for the work to be
performed, establish the necessary commitments, and define the plan to perform the work
NASA Langley Research Center
Software Project Planning
Best Practices Project cost, effort and duration estimates are
developed and tracked Software Work Breakdown Structure is generated
at the same level as hardware WBS Scheduling tools (MS Project, REVIC) are used Statements of Work specifies creation of project
plans Software staged release at planned milestones Software manager designated to oversee a
software development project Software personnel participate in project plans SQA involved in project planning
NASA Langley Research Center
Software Project Planning Improvement Opportunities
Schedules are driven by externally set milestones, instead of actual work required
Software Management Plans are not consistently documented, if at all
No planning metrics are captured across the organization on which to base realistic future estimates
Commitments are not honestly negotiated Inadequate project planning is compensated for by overtime Over-commitment No LaRC project policy for managing software projects - the
approach to software project management is project dependent
The value of software project plans is not fully understood by developers, managers, or researchers
Personnel are not aware of available procedures and guidelines for software estimating
No guidelines are available for document software development plans
NASA Langley Research Center
Software Project Planning
Consequences Software projects do not meet schedules or
budgets (could face cancelation) Burn out civil servants and contractors
NASA Langley Research Center
Software Project Tracking and Oversight
Purpose Provide adequate visibility into actual progress
so that management can take effective actions when the software project's performance deviates significantly from the software plans
Description Track and review the software accomplishments
and results against documented estimates, commitments, and plans
Adjust plans based on the actual accomplishments and results
NASA Langley Research Center
Software Project Tracking and Oversight
Best Practices Informal interchange meetings are held
frequently for tracking Automated tools (MS Project, PC Artemis) are
used to track project progress Milestones used as checklist Formal reviews held at selected milestones Dedicated business manager to track
software project status
NASA Langley Research Center
Software Project Tracking and Oversight Improvement Opportunities
Corrective actions are not taken in a timely manner Software costs are not accurately reflected in project
charge structures No measurements or lessons learned are captured and
used for future projects Software size is not estimated or tracked Managers are not trained in software project
management and tracking Technical issues are not managed and communicated Software Development Plans are not used to manage
the project Software project reviews are ad hoc and ineffective;
results are not elevated to management (guidelines needed)
NASA Langley Research Center
Software Project Tracking and Oversight
Consequences Products not completed on expected dates The real software project status is not known
until it is too late to do anything about it
NASA Langley Research Center
Software Subcontractor Management
Purpose Select qualified software subcontractors and
manage them effectively Description
Select a software subcontractor Establish commitments with the
subcontractor Track and review the subcontractor's
performance and results
NASA Langley Research Center
Software Subcontractor Management Best Practices
Periodic reviews are conducted with contractors to review progress and communicate status
COTRs and Technical Monitors are designated to establish and manage the software contract
Source Evaluation Boards select contractors based on their ability to perform the work
Success has been demonstrated using Performance Based Contracting
Formal quarterly evaluations are required Software services are acquired using standard
NASA contracting regulations
NASA Langley Research Center
Software Subcontractor Management Improvement Opportunities
No accountability for poor contractor performance COTRs and Technical Monitors are not trained in software
engineering or managing software development efforts Performance Based Contracting is misunderstood, and
due to poor training it is ineffectively applied Source Evaluation Board frequently recommends lowest
bidder rather than best qualified Required reviews are not consistently conducted Contractor turnover is high and requires frequent re-
orientation resulting in increased costs No written LaRC policy or guidelines for managing
software contracts Contractors over-commit and rely on “free” overtime Skill level and training on the contract does not match
what is required
NASA Langley Research Center
Software Subcontractor Management
Consequences Frequent dissatisfaction with contractor’s
products PBC task generation is more time-consuming
NASA Langley Research Center
Software Quality Assurance
Purpose Provide management with appropriate visibility
into the process being used by the software project and of the products being built
Description Review and audit the software products and
activities to verify that they comply with the applicable procedures and standards
Provide the software project and other appropriate managers with the results of these reviews and audits
NASA Langley Research Center
Software Quality Assurance
Best Practices
SQA contractors advise and give consultation to projects on software engineering when requested (on configuration management and software management plan)
Infractions are reported to participating project teams
NASA Langley Research Center
Software Quality Assurance
Improvement Opportunities Need sufficient staff for uniform coverage of SQA
on Langley software projects Increase awareness of added value of SQA
practices Appropriate guidelines for SQA activities (current
LHB not widely accepted) Review SQA activities on a regular basis Track cost and associated return on investments
on SQA tasks
NASA Langley Research Center
Software Configuration Management
Purpose Establish and maintain the integrity of the products of the
software project throughout the project's software life cycle Description
Identify the configuration of the software (i.e., selected software work products and their descriptions) at given points in time
Systematically control changes to the configuration Maintaining the integrity and traceability of the
configuration throughout the software life cycle Placed under software configuration management both the
software products delivered to the customer (e.g., the software requirements document and the code) and the items identified with or required to create these software products (e.g., the compiler)
NASA Langley Research Center
Software Configuration Management
Best Practices Configuration Control Boards are established and
used to control changes, assess impacts and communicate status
Automated tools (ClearCase, PVCS, RCS, LibLink, CVS, LabView) are used to control software work products (requirements, change rationale, code and supporting documentation)
Intent of SCM can be met without the use of automated tools for projects
Web-based change request system established (ClearDDTS)
SCM process documented and followed on some projects
NASA Langley Research Center
Software Configuration Management
Improvement Opportunities Rigor of formal SCM is not always appropriate for project’s
size, phase and purpose Change request status is not adequately communicated across
all project personnel Lack of awareness of existing guidelines and templates results
in significant duplication of effort Insufficient funding for SCM staff, tools and training SCM typically applied too late in project phase SCM plans are not documented and/or followed and changes
are not controlled Lack of understanding of the benefits of SCM Contracts do not always require the appropriate level of SCM
and there is inadequate CM on deliverables
NASA Langley Research Center
Software Configuration Management
Consequences Lack of SCM results in compromised mission
certainty and validity of data and products
NASA Langley Research Center
Training Program
Purpose Develop the skills and knowledge of individuals so
they can perform their roles effectively and efficiently Description
Identify the training needed by the organization, projects, and individuals
Develop or procure training to address the identified needs
Evaluate current and future skill needs and determines how these skills will be obtained
Use informal vehicles when appropriate (e.g., on-the-job training and informal mentoring)
NASA Langley Research Center
Training Program
Best Practices Training Office exists to meet the needs of the
LaRC including the software development community
Some projects and organizations successfully leverage the capabilities of the Training Office to meet their needs
Training Office annually surveys LaRC to establish training needs
NASA Langley Research Center
Training Program
Improvement Opportunities Increase is needed in training budget due to cross
training and retraining of LaRC personnel Inconsistent management support for training Insufficient time is allocated for attending training Training is not considered a priority Need full implementation of Individual Development
Plan (IDP) which ties to the SQF and the agency strategic plan
No effective mechanism for consistently training contractors and civil servants working on the same project
No project training plan Travel funds shortage limits training opportunities
and technical exchange
NASA Langley Research Center
Software Product Engineering
Purpose Consistently perform a well-defined
engineering process that integrates all the software engineering activities to produce correct, consistent software products effectively and efficiently
Description Perform the engineering tasks to build and
maintain the software using the project's defined software process and appropriate methods and tools
NASA Langley Research Center
Software Product Engineering
Best Practices Operational Concept, Users Manual and Test Plans/Cases
developed prior to code Apply techniques such as Object Oriented Design (OOD),
reverse engineering, structured programming, and modularity to reduce code complexity, increase reusability and improve maintainability
Automated software development tools (Purify, Quantify, PureCoverage, Rational Rose, ClearCase, ClearDDTS, Object Manual, LabView GUI) are used to design, test, CM, and document software
Provide realistic operational testing scenario for software Office automation tools used in innovative ways for
software development, testing and documentation (databases, spreadsheets)
NASA Langley Research Center
Software Product Engineering
Best Practices (cont.) A project has demonstrated LaRC’s ability to meet
FAA standards for airworthy flight software On line Web sites exist that describe software
engineering guidebooks, formal methods and metrics collection database
Third party tests against requirements agreed to by users
Continuous rapid prototyping used to demonstrate feasibility and early progress
Requirements gathering techniques resulted in improved software product
Centralized facility provides access, guidance and expertise for domain-specific tools
NASA Langley Research Center
Software Product Engineering
Improvement Opportunities Insufficient time allocated for software engineering
tasks (requirements definition, design, testing, integration, configuration management, and documentation)
Software engineering activities often hidden because their value is not recognized, perceived as overhead by projects, and researchers/projects do not want to pay for reuse (no corporate view for long-term investment)
No rewards for good software engineering Single point failures on many projects created by one
person and insufficient documentation Testing address physics only, not software quality No LaRC dissemination guidelines for software exist
NASA Langley Research Center
Software Product Engineering
Improvement Opportunities (cont.) Lack of in-house software product engineering
expertise Need point of contact where software engineering
expertise and guidance can be obtained Ineffective tool utilization due to poor communication
and awareness Web sites not advertised Software engineering techniques not appropriately
tailored for small projects No incentive to take research code to production
quality or make it reusable by other projects Lack of expertise in project planning and scheduling
leads to insufficient time for fundamental software engineering tasks
NASA Langley Research Center
Software Product Maintenance
Best Practices Use portable computer to set up, perform experimental
test, and analyze results Improvement Opportunities
Insufficient funding for sustaining and maintenance of software
Ineffective implementation of CM prohibits quick minor changes to software for experimental use
Insufficient verification of software prior to delivery leads to high maintenance cost
Due to insufficient manpower and maintenance procedures, staff is not adequately notified on software changes
Constant changes in COTS upgrade decrease productivity due to perpetual learning curve
NASA Langley Research Center
Intergroup Coordination
Purpose Establish means for the software engineering group to
participate actively with the other engineering groups so the project is better able to satisfy the customer's needs
Description Software engineering group participates with other project
engineering groups to address system-level requirements, objectives, and issues
Representatives of the project's engineering groups participate in establishing the system-level requirements, objectives, and plans by working with the customer and end users, as appropriate
Requirements, objectives, and plans become the basis for all engineering activities
NASA Langley Research Center
Intergroup Coordination
Best Practices Workshops and shared facilities bring different groups
together for improved information exchange Interface Control Documents, frequent meetings and
timely meeting notes ensure effective exchange of information
Web-based information and email exchange facilitates technical coordination
Physical co-location of discipline specialists promotes intergroup coordination during a project
Teamwork training is available Communication between software programs used by
different groups achieved by in-house developed interfaces (scripts, GUIs, wrappers, standard features, filters)
NASA Langley Research Center
Intergroup Coordination Improvement Opportunities
Need to have representatives of all technical disciplines (including software) involved from the start of a project (requirements, cost estimates, operational concepts, requirements allocation)
Due to the lack of systems engineering performed at LaRC, software and hardware staff are forced to fill that role in an ad-hoc manner
Poor communication among groups doing similar work within LaRC results in duplication of work
No effective mechanism to ensure all disciplines are represented on project teams
Software specialists not aware of overall project concept Perception that software can fix anything, including poor hardware
selection, leads wasted funds and staff hours Lack of documented commitments between engineering groups Changes are not effectively communicated across the engineering
groups Contractors resist communication due to proprietary fear
NASA Langley Research Center
Peer Reviews
Purpose Remove defects from software products early in
the development process where it is more cost effective to remove them
Develop better understanding of software products and defects that might be prevented
Description Peers methodically examine software work
products to identify defects and areas where changes are needed
Identify and schedule specific products that will undergo peer review as part of the defined software process
NASA Langley Research Center
Peer Reviews
Best Practices Peer reviews identify problems early in the lifecycle resulting
in saved time and decreased costs Post-mortem review is effective mechanism to capture
lessons learned Informal review by an in-group peer works well on small
projects Peer reviews captures problems not otherwise identified Completion of peer reviews is a phase exit requirement and
indicates readiness to proceed Peer reviews are effective mechanism to train staff and
indoctrinate new hires Peer review process is documented (NASA Formal Inspection
Handbook, project documentation) User requirements are basis for code reviews Email and Key Activities are used to document peer review
results
NASA Langley Research Center
Peer Reviews Improvement Opportunities
Peer reviews are not commonly used (even unknown by some projects) in spite of positive return on investment experienced at LaRC
Current review process frequently omits software issues
Current formal design review system (PDR, CDR) often misses large problems
Post-mortem reviews hardly ever held Limited time, training, and staff are provided for
performing peer reviews No data collected on peer review results and
lessons learned (dormant data base)
NASA Langley Research Center
Non-CMM Related Concerns
Improper ISO 9000 implementation could impose restrictive procedures on research and significantly impact resources to perform future research
Numerous management issues were identified such as: The lack of management awareness of the pervasiveness and
magnitude of work required to develop software at LaRC Need investment, encouragement, and reward system for
improvements, best practices, positive technical transfer Lack of adequate resources for performing software
development
There is a lack of a central pool of software developers to service LaRC
Relation of LaRC’s Strategic Quality Framework to real work is poorly defined
NASA Langley Research Center
Next Steps
NASA Langley Research Center
Critical Point in LaRC’s SPI Program
CornerStone phase is a critical milestone in the software process improvement program
CornerStone phase baselines our current state and develops a plan for future process improvement
Proactive management is required to go forward with successful implementation of the Improvement Plan
NASA Langley Research Center
Upcoming Activities
Receive feedback on Findings Briefing Develop draft Software Process
Improvement Plan Review SPI Plan with sponsors Establish Senior Management Steering
Group (SMSG) Establish Software Engineering Process
Group (SEPG) Finalize SPI Plan Conduct SPI Plan Briefing Implement Improvement Plan
NASA Langley Research Center
Management Steering Group Description Comprised of the LaRC CIO, ISO Project Manager, and
Division Chiefs who have a stake in the successful achievement of the Software Process Improvement (SPI) goals
Monitors and evaluates SPI efforts from the perspective of the total organization to ensure that the overall improvement efforts are in concert with LaRC’s mission and goals
Meets regularly with the Software Engineering Process Group (SEPG) to discuss the progress of the SPI program, problems, issues and concerns
Works together with the SEPG to identify and provide the long-term commitments and resources required to coordinate the development and maintenance of software engineering processes for use by current and future software projects
NASA Langley Research Center
Suggested Management Steering Group Membership
Doug Arbuckle/Associate Director, LaRC Fay Collier/ISO 9000 Project Manager, LaRC Pat Dunnington/CIO, LaRC Luat Nguyen/FDCD Bill Wessel/OSEMA Carl Gray/FSED Milt Holt/FETD Rob Kudlinski/ISSD Lenny McMaster/AESD Bill Smith/SSCD David Stephens/FMAD
NASA Langley Research Center
Responsibilities of the Management Steering Group (MSG) Ensure alignment of Software Process Improvement (SPI)
initiatives with LaRC mission and goals Approve strategic plan for SPI Provide sponsorship, pro-active commitment, and visible
management support Allocate resources Monitor the progress of the SPI initiative Provide guidance and direction to the Software
Engineering Process Group (SEPG) Conduct regular meetings with the SPI Group Promote cooperation and cross-functional
communications Cultivate an enabling environment for continuous
process improvement Obtain and sustain the support for the SPI initiative
NASA Langley Research Center
Software Engineering Process Group (SEPG) Description Chartered by the MSG, as the focal point for software
process improvement Coordinate and plan the SPI program activities in
concert with the guidance and direction of the MSG Report the progress of SPI related activities to the MSG
Provide technical support and consultation Staffed by experienced software development
practitioners who facilitate the definition, maintenance, and improvement of the software engineering processes used within the organization
Work collaboratively with the projects to resolve process related problems and assist in the development, training and implementation of processes and procedures
NASA Langley Research Center
Suggested Software Engineering Process Group (SEPG) Membership Members of the CornerStone team
Volunteers from interview workshops
Appointees from participating divisions
NASA Langley Research Center
Responsibilities of the Software Engineering Process Group (SEPG)
Define and manage the plan for development and implementation of software process improvements across the LaRC
Provide a pool of software engineering expertise and corporate knowledge (Formal part of LaRC organization with assigned resources and management commitment)
Provide consultation and guidance on appropriate level of software engineering implementation and future directions
Provide and facilitate education on software engineering to management and staff via workshops, seminars, symposiums, and setting up news/user groups and maintain web sight
Provide a repository for reuse code, documents, tool knowledge, procedures, processes, LaRC best practices, templates, lessons learned, metrics, and examples
Facilitate sharing of tools and maintenance of COTS across projects Build and reinforce sponsorship for the SPI program management
support at all levels
NASA Langley Research Center
Next Steps Establish membership of MSG & SEPG (by Nov. 12)
Post Best Practices on Web
Prioritize Improvement Opportunities
Develop SPI Project Plan
Review and Obtain Approval of Plan (by Dec. 19)
Complete Final Report
Implement Improvement Plan
NASA Langley Research Center
Closing Thought
“Creative thinking may simply be the realization that there is no particular value in doing things the way we’ve always done them.”
- R. Flesch, Educator
NASA Langley Research Center
Odd and ends follow:
Global Improvement Opportunities Seminars (maybe during lunch) on SW topics of interest Establish Web site to capture and publicize best practices Establish basic training in software engineering Short-course training on widely-used COTS. Need to provide guidelines and expertise for appropriate level of software
engineeringprocedures based on project size, application and criticality.
NASA Langley Research Center
Odds and ends cont.:
Some analysis was started on the Closing Q1 data that is captured below(not to be used in findings briefing):
Global Improvement Opportunities>Provide education on requirement gathering and documentation>Do pilot/demonstration projects on selected software engineering activities>Conduct lesson learned review on project complete and post results on the web
Management Related Improvement Opportunities>Need to overcome resistance to positive change>Segmentation of responsibilities across organizations has led to lack of single point of accountability for technical issues>Improve management awareness of the pervasiveness?? importance and magnitude of work required to develop software
Global Concerns>Improper ISO 9000 implementation could impose restrictive procedures on research and significantly impact resources to perform future research>Civil servants should be made technically knowledgeable in software>Sponsor research in appropriate software engineering practices for the research environment
NASA software put in commercial system with no profit to NASA (companies make profits on NASA software)
NASA Langley Research Center
Responsibilities of the Software Engineering Process Group (SEPG)
Establish a repository for SPI products (best practices, code, and lessons learned)
Establish appropriate organizational metrics and mechanisms for the collection of metrics
Provide desk-side mentoring to project teams Track, monitor and report the status of SPI initiatives to the MSG Promote technical awareness and coordinate training for software
engineering with the LaRC training office Provide consultation and CMM expertise to the MSG and the technical
working groups which implement improvements Compare current practices against the goals and key practices of the
CMM Keep LaRC apprised of SPI activities and progress
NASA Langley Research Center
Center Sponsors
Kristin Hessenius/Associate Director, LaRC
Fay Collier/ISO 9000 Project Manager, LaRC
Pat Dunnington/CIO, LaRC
Rob Kudlinski/ISSD
Top Related