8/4/2019 Nasa Gb a301

1/18of 18

SOFTWARE QUALITY ASSURANCE AUDITS GUIDEBOOK

NOVEMBER 1990

PREFACEThe growth in cost and importance of software to NASA has causedNASA to address the improvement of software development acrossthe agency. One of the products of th i s program i s a ser ies ofguidebooks t h a t def ine a NASA concep t o f the assurance p rocessestha t are used in software development.Th e Sof tware Assurance Guidebook, NASA-GB-A201, i s sued inSeptember , 1989, prov ides an ove r a l l p i c tu r e o f th e NASA concept sand pr ac t i ce s i n sof tware assurance . Second l eve l guidebooksfocus on specif ic ac t iv i t i e s tha t fa l l within the softwareassurance disc ip l ine , and provide more deta i led information forthe manager and/or prac t i t ioner .This i s th e second l eve l Software Qual i ty Assurance AuditsGuidebook t h a t descr ibes sof tware qua l i ty assurance aud i t s i n awa y t h a t i s compat ible with pr ac t i ce s a t NASA Center s . For amore genera l i zed view o f ho w sof tware qua l i ty assurance aud i t sr e l a t e to Sof tware Assurance, r e f e r to the Sof tware AssuranceGuidebook, document number NASA-G8-A201.I . GENERALThe NASA Sof tware Assurance Guidebook c l a s s i f i e s th e sof twareq u a l i t y assurance (SQA) aud i t as a fundamental qua l i ty assurancetechnique . I t i s the i n t e n t of t h i s guidebook to f u r t h e r def ineaud i t s , descr ibe the aud i t process , and provide a samplec h e c k l i s t t h a t can be t a i l o r e d fo r use in an aud i t . Theguidebook i s wri t ten fo r q u a l i t y assurance p r a c t i t i o n e r s who w i l lperform aud i t s , sof tware developer s who wi l l be aud i ted , and fo rsof tware p r o j e c t managers and acq u i r e r s who have to decide theex t en t of aud i t ing to be done.In t h i s guidebook, th e term "audit!! s p e c i f i c a l r e f e r s to an SQA

t h a t i s used to examine th e conformance of aprocess to and th e conformance o f

to An SQA examine th e conformance ofth e ac t u a l s t a t u s of the to the repor tedstatuso Th e term HauditH i s used to describe a number ofadd i t iona l sof tware a c t i v i t i e s ; however due to t h e i r d i f f e r e n t

http://satc.gsfc.nasa.gov/auditlaudg

\

7/30/20029:19

8/4/2019 Nasa Gb a301

2/18of 18

purpose and focus , they are not addressed in th i s guidebook. Forexample, the Func t iona l Configurat ion Audit (FCA) and Physica lConfigurat ion Audit (PCA) are conf igura t ion management (CM)a c t i v i t i e s . Qual i ty (Engineering) Audits and Safe ty Audits aret echnica l ac t i v i t i e s t h a t evalua te a sof tware p roduc t aga ins tQual i ty Engineer ing and Safe ty r equ i rements . These types ofaud i t s are not covered in th i s guidebook.I I . CONCEPTS AND DEFINITIONSAn SQA audi t i s an a c t i v i t y t h a t i s performed to determine theadherence to , and adequacy of , a p r o j e c t ' s es t ab l i shed sof twaredevelopment s tandards and procedures and th e e f f ec t iveness oft h e i r implementat ion. As used in t h i s guidebook, the mainobjec t ive o f an SQA aud i t i s to determine the adherence toe s t ab l i shed s tandards and procedures; checking t h e i r adequacy o re f f ec t iveness i s a secondary objec t ive t h a t usua l ly i s notrequested of an aud i to r .In the NASA Software Assurance Guidebook, s tandards are def inedas nthe es t ab l i shed c r i t e r i a to which sof tware products arecompared.!! Software s tandards inc lude documentat ion s tandards ,des ign s tandards , and coding s tandards . In t h a t guidebook,procedures are def ined as the "es tabl i shed c r i t e r i a to which th edevelopment and con t r o l processes are compared. I! Procedures ,then , are the s t ep- by- s t ep d i rec t ions t h a t are to be followed toaccomplish some development o r con t r o l process ; fo r example, eMo r nonconformance r epor t ing and cor rec t ive ac t ion (NRCA). Inothe r words, s tandards and procedures are requirements fo rsof tware management, engineering, and assurance; SQA aud i t sver i fy t h e i r exis tence and assess a p r o j e c t ' s compliance withthem.SQA audi t s a l so can compare the ac tua l s t a tus o f a product withrepor ted s t a t u s . Sta tus aUdit ing i s most e f f e c t i v e i f there a reob jec t ive and cons i s ten t c r i t e r i a for eva lua t ing the l eve l ofproduc t completeness . For example, u n i t Development Folders(UDFs) have a cover shee t for recording th e progress of a un i tthrough i t s development s tages i the f o lde r con ta ins the ac tua lproduct . I f a pro jec t uses UDFs, then an aud i t can compare theac tua l product to th e cover shee t and to th e progress r epor t .Th e ac tua l processes and products examined by an audi t w i l l varydepending on th e objec t ive o f the aud i t . Th e objec t ive of theaud i t can vary, and i s determined by th e organiza t ion tha t ca l l edfo r the aud i t . A genera l aud i t prov ides a comprehensiveoverview, whi le a l im i ted aud i t might be an examinat ion ofc e r t a i n procedures , such as eM, o r a check on a cer ta inrequirement , such as nAre coding s tandards be ing followed?!!An audi t may be descr ibed as in te rna l o r ex te rna l , depending onthe organiza t ion of or ig in o f the aud i to r ( s ) . An in te rna l aud i ti s an audi t conducted by the SQA s t a f f of the sof tware developer .I n t e r na l aud i t s a r e in tended to be preventa t ive in nature ; tode tec t prOblems before they become major .An ex terna l aud i t i s one performed by an independent aud i to r whoi s outs ide of th e developing organ iza t ion . External audi t s a remost of t en reques ted by the acqu i r ing organiza t ion f as a means o fob ta in ing an independent about the work in progress .Exte rna l aud i t s t end to be more in nature thanin te rna l aud i t s , and usua l encompass a broad area of the

a c t Such aud i t s a re becauseth e i s uncer ta in of the ef fec iveness o f the in te rna lprogram o r because of l ack of in fonna t on and fears about the

i ty of on the p a r t of he .;[1

http://satc.gsfc.nasa.gov/audit/aud

7/30/20029: 19

8/4/2019 Nasa Gb a301

3/18

of 18

advantage of an ex te r na l aud i t i s t h a t th e aud i to r may be moreobjec t ive about a p r o j e c t than an i n t e r n a l aud i to r ; however, anex te r na l aud i to r must spend more t ime l e a rn i n g about th e p r o j e c tand i t s development process .I I I . CONDUCTING AN SQA AUDITAn SQA au d i t has four phases : p lann ing and p rep a ra t i o n , th e s i t ev i s i t , repor t ing, and fol low-up. During th e plann ing andp rep a ra t i o n phase , th e aud i to r gains an unders tand ing of thep r o j e c t . Based on th e scope of th e a u d i t , th e aud i to r determinesth e spec i f ic ques t ions t h a t need to be answered, as wel l as th epersons to be in terv iewed and the r ecords and products to beexamined to answer th e ques t ions . Th e in te rv iews a re conducted,and r ecords and products a re examined dur ing th e s i t e v i s i t . Th er ep o r t i n g phase cons i s t s of the e x i t debr ie f ing of the aud i tedp r o j e c t , th e pr epa r a t ion o f a wr i t t en r epor t on th e aud i t , andc l a r i f y i n g i s s u es and prov id ing r e l a t e d in format ion as needed.Follow-up i s done by the pr o jec t , as th e problems andde f i c i enc ie s found in th e aud i t a re remedied . Fol low-up mayinc lude r eaud i t i ng to as s es s th e adequacy of th e remedies .Th e a c t i v i t i e s conducted dur ing the phases vary depending on th el i f e cycle phase of th e p r o j e c t be ing au d i t ed and th e sccpe ofthe aud i t . Th e a c t i v i t i e s a l so vary depending on whether th eaud i t i s ex te r na l o r i n t e r n a l i an ex te r na l aud i t r equ i r e s moreex tens ive p rep a ra t i o n and should examine a more comprehens ivesample o f ma t e r i a l than an in te rna l a u d i t .Each of the four phases of an aud i t i s descr ibed in the fol lowingsec t ions . Th e a c t i v i t i e s of each phase a re descr ibed as i f agenera l , ex te r na l aud i t i s to be done s i n ce t h i s r e su l t s in th eg r e a t e s t d e t a i l . Some of th e a c t i v i t i e s may be super f luous to anin te rna l SQA aud i t and may be omit ted .A. Audi t Planning and Prepara t ionA genera l SQA a u d i t should be planned c a r e f u l l y to examine a l l o fthe sof tware engineer ing , management and assurance p rocesses anda l l of t h e i r product s . Software management processes inc ludes t a t u s r ep o r t i n g and eM. Engineering processes inc lude ana lys i s ,des ign , and code. Assurance processes inc lude ver i f i ca t ion andva l ida t ion (V&V) and NRCA. Products include documents and code.I f th e scope of th e aud i t i s more l imi t ed , then p lann ing w i l l bewith in the def ined l i m i t s . A l imi t ed aud i t might examine onlyone o f th e p r o ces s es o r a l imi ted s e t of products . A c t i v i t i e sdur ing th e plann ing and p rep a ra t i o n phase a re s imi l a r fo r a l laud i t s , bu t pr epa r a t ion fo r a l imi t ed aud i t i s focused on th ei d e n t i f i e d process o r product .AS a f i r s t s t ep , th e aud i to r should unders tand th e objec t ive ofth e sof tware development p r o j e c t and what products a re to beproduced. Th e aud i to r needs to know what t he con t r ac t r eq u i r e sin th e way o f de l ive r ab le sof tware and documentat ion , and what,i f any, requirements e x i s t fo r management, engineer ing , andassurance On e source of t h i s in format ion may be th es ta tement o f work and othe r cont rac t documents . Once it i s c l e a rwhat i s be ing developed and what th e con t r ac t r equ i r e s , th eaud i to r should review management documentat ion , such as thesof tware management, development, and assurance p lan s tounderstand th e processes t h a t wi l l be used to and con t ro lth e Then th e I s s tandardsmanual should be reviewed to determine the s tandards andthe de ta i l ed to be ied to the sof tware andthe process From t h i saud i to r should be able to understand th e in format ion , th eU t ' V ' L L ' J p e r I s sof tware

http://sate.gsfc.nasa.gOY Iaudit!aud

7/30/20029:19

8/4/2019 Nasa Gb a301

4/18

of 18

development process .Th e aud i to r a l so should review some r ecen t s t a t u s r epor t s fromth e d ev e lo p e r . These r ep o r t s wi l l f u r n i sh in format ion on thes tage o f completeness of products and may con ta in in format ion asto problem a r ea s .A f t e r background f ami l ia r iza t ion and a look a t p r o j e c t s t a t u s ,th e aud i to r should def ine the areas t h a t wi l l r eq u i r e t h e mostca r e f u l and d e t a i l e d a t t e n t i o n , i.e., th e processes o r productst h a t seem to be in some d i f f i c u l t y o r whose s t a t u s i s in doubt .These a reas may be i d e n t i f i e d by th e s t a t u s r epor t s , discuss ionswith th e acq u i r e r o f the sof tware ( i f it i s th e acq u i r e r who hasr eques ted the a u d i t ) , review of nonconformance r epor t s , and th er e su l t s of p rev ious aud i t s .Once th e a u d i t o r understands th e p r o j e c t and has i d e n t i f i e d th eareas o f concen t ra t ion , he/she should develop a c h e c k l i s t . Ac h e c k l i s t i s a list of i t ems to be examined and ques t ions to beasked. Each c h e c k l i s t should be t a i lo r ed fo r th e spec i f icp r o j e c t being aud i ted and i t s l i f e cycle phase and should r e f l e c tthe scope of the aud i t . A more comprehensive and l e s s d e t a i l e dc h e c k l i s t i s r eq u i r ed fo r a genera l a u d i t i a l imi t ed aud i tr eq u i r e s a c h e c k l i s t t h a t i s more d e t a i l e d in s p e c i f i c a r ea s .Guidance on prepar ing a check l i s t i s given in Chapter VI. Ac h e c k l i s t i s in tended to provide th e aud i to r with a '*road map"dur ing the s i t e v i s i t . It must be complete , so t h a t th e aud i to rcan know t h a t s u f f i c i e n t in format ion has been gathered if a l l ofthe check l i s t i tems a re completed. The check l i s t ques t ions helpdef ine th e i nd iv idua l s with whom th e aud i to r wishes an in te rv iewand th e types o f r ecords t h a t th e aud i to r w i l l examine.The aud i to r should schedule th e s i t e v i s i t to th e p r o j e c t throughi t s assurance s t a f f o r o t h e r s u i t a b l e co n t ac t a f t e r th ep rep a ra t i o n i s done and the c h e c k l i s t prepared . During t h i scon tac t with th e p r o j e c t , th e aud i to r should s p ec i fy th e i n t e n tof the a u d i t , the r ecords to be examined, and which people th eaud i to r wishes to in terv iew. People to be in terv iewed w i l linc lude managers , se lec ted developer s , CM s t a f f , assurance s t a f f ,and t e s t e r s . Copies o f the c h e c k l i s t may be fu rn ished toincrease the p r o j e c t ' s unders tand ing . Th e pr o jec t should beprepared to provide the aud i to r with a convenient working areat h a t inc ludes normal o f f i c e f a c i l i t i e s , access to a l l productsand records , and in terv iews with th e i d e n t i f i e d i nd iv idua l s .B. Th e Si te V i s i tTh e purpose of the aud i t s i t e v i s i t i s to c o l l e c t th e datanecessary to assess t h a t th e r e ~ ~ i r e d products a re be ingproduced, th e degree to which they conform to app l icab les tandards , how wel l procedures a re being fo l lowed, and t h a t th erepor ted s t a t u s corresponds to th e ac tua l s t a t u s . Th e aud i t i sin tended to uncover any s i g n i f i c a n t dev ia t ion from s tandards ,procedures , o r r ep c r t ed s t a t u s so t h a t cor r ec t ive ac t ion can bet aken . Th e aud i to r uses tw o bas ic t echn iques : in terv iews wi thpr o jec t s t a f f and examinat ion of documentat ion and records .The s i t e v i s i t should begin wi th an en t rance br i e f ing , invo lv ingth e aud i to r and key ee t s t a f f . During t h i s b r i e f i n g , th eaud i to r should descr ibe th e focus of th e aud i t , and iden t i fy th ein terv iews to be conducted aE d the r ecords be examir1l2d> Th een t rance b r i e f i n g may a l so be used th e ee t to b r i e f th eaud i to r on i t s processes , s t a f f mewbers, and cu r r en t s t a t u s .Time fo r and answers should be inc luded . The aud i to ra l so should assure the ec t t h a t an e x i t in terview wi l l be

http://satc.gsfc.nasa.gov/auditlaudg

7/30/2002 9: 19

8/4/2019 Nasa Gb a301

5/1818

held where th e aud i to r w i l l presen t p re l iminary f ind ings to th ep r o j e c t and the p r o j e c t may provide any add i t iona l in format ion toth e aud i to r . This pre l iminary exchange of in format ion cans i g n i f i c a n t l y help to a l l ay th e fears of the p r o j e c t and tosmooth th e course of the s i t e v i s i t .A f t e r th e en t rance br ie f ing , th e aud i to r should proceed with thegather ing of in format ion . I t i s usefu l to begin th e in format iongather ing process with in te rv iews , dur ing which th e aud i to r t r i e sto unders tand th e r e a l i t i e s behind th e documented p lan s andprocedures . Th e aud i to r should l e a r n which i n d i v i d u a l s car ry o uta procedure , approve a change or f ix , keep p r o j e c t records , e t c .Each ind iv idual should be asked to descr ibe h i s / h e r percep t ionsof and i n t e r a c t i o n s with the p rocess . The aud i to r should t akeno tes , an n o ta t e o r develop procedura l flow diagrams, askques t ions to c l a r i f y , and make it h i s / h e r objec t ive to c l e a r l yunders tand the p rocess . In p a r t i c u l a r , the aud i to r should bea l e r t f o r i nd ica t ions of shor t cu t s o r abbrev ia t ions to th eprocedure. During in te rv iews , th e aud i to r must remember t h a td a t a a re being gathered , and t h a t conclus ions should wai t u n t i la l l of the f ac t s a r e i n . This prov ides a c l e a r e r unders tand ingof the ac tua l processes used on th e p r o j e c t and eas escommunications with the s t a f f . Th e c h e c k l i s t developed dur ingt h e p rep a ra t i o n phase i s used to guide th e discuss ions dur ing thein te rv iew.Once th e aud i to r i s sure t h a t the p rocesses and procedures a reunders tood as they r e a l l y e x i s t , he/she should begin examiningth e t angible p a r t s of th e p r o j e c t : i t s products and records .Produc t s cons i s t o f requirements and des ign documentat ion,inc lud ing u n i t development folders , user manuals , code, e t c .Records c o n s i s t of memoranda and forms t h a t document the even tsin th e l i f e of a product , They come from CM, NRCA, and V&V,among othe r s .1. Records ExaminationThe aud i to r examines r ecords to see i f a procedure i s beingc o r r e c t l y fol lowed. Record examinat ion i s descr ibed below int erms of the pr inc ipa l p r o ces s es t h a t SQA aud i t s examine: eM,NRCA, and V&V. Simi lar a c t i v i t i e s would be used in th eexaminat ion of o t h e r se t s of records . CM Audi tDuring an au d i t of CM, th e aud i to r should look a t th e completechange con t ro l cyc le , beginning with the i n i t i a l proceSSing o f achange reques t ; through an a l y s i s of impact and di spos i t i on ing ;des ign , code; and t e s t i n g ; updat ing of documentat ion; submiss ionof th e modif ied products to th e l ib ra ry ; and c losure of th echange reques t . Records to be examined inc lude the changereques t s as processed by th e Change Cont ro l Board, th e workau t h o r i z i n g documents i ssued as a r e su l t of approved changes , th ecode and documentat ion products t h a t a re in tended to r e f l e c t th eapproved changes , and th e program l i b r a r y r ecords t h a t cap tureth e changes to code and d a t a . Throughout the aud i t , th e aud i to rshould be a l e r t fo r and document any evidence o f unauthor izedchanges .The r ecords should show the au thor i za t ion o f each F th eD r ' O ( j u , ~ t ( B ; to be and the vers ion numbers of the

Much of the aud i to r ! s a t t e n t i o n should be devoted toth e Program o r , s ince t h i s i s where thevar ious Vers ions of documents con t ra I lthose vers ions a re s to r ed . Th e aud i to r should check th e

http;/ satc,gsfc.nasa.gOY!audit!aud g

7130/2002 9: 19

8/4/2019 Nasa Gb a301

6/18of 18

in th e l i b r a r y to ensure t h a t documentat ion i s u p - t o -d a t e withcode changes . The aud i to r should check th e v e r s i o n numbering andi d e n t i f i c a t i o n schemes, and the con t ro l documents . The r ecordsshould demonst rate t h a t t he r e a r e adequate secur i ty measures inp l ace to preven t l os s and unauthor ized changes . The a u d i t o rshould ve r i f y t h a t every i tem o f code and documentat ion in th eprogram l i b r a r y was p ro p e r l y r ece iv ed . NRCA Audi tWhen au d i t i n g th e NRCA system, the au d i t o r should look a t th ecomplete cy c l e . The a u d i t o r should review th e nonconformancer epor t s t h a t a re f i l e d , to assure t h a t they a re comple te ly andc o r r e c t l y f i l l e d out . The d i s p o s i t i o n process and board ac t i o n sshould be recorded , u s u a l l y on th e same form. Thenonconformances t h a t r e su l t in produc t changes should be t r ack edto th e product , and evidence should be gathered t h a t changes a remade, t e s ted o r reviewed, and approva l s fo r i s suance a re g r an t ed .The NRCA procedures w i l l p a r a l l e l t hose used in eM, and can beaud i ted in much th e same way, espec ia l ly when it comes to th eprogram l i b r a r y . In bo th cas es (CM and NRCA) , th e aud i to r shouldpay p a r t i c u l a r a t t e n t i o n to co r r ec t ed products to as s u re t h a tthey still sa t i s f y requ i rement s and s t an d a r d s . V&V Audi tAn a u d i t of V&V procedures should inc lude a check of th ev e r i f i c a t i o n matr ix o r equiValen t , to assure t h a t everyrequ i rement has a t e s t and every t e s t checks a r equ i rement . T e s tplans should be adequa te , spec i fy ing the t e s t envi ronment , t e s tprocedures , and th e expec ted r e s u l t s fo r each t e s t . T e s tprocedures should be c l e a r and d e t a i l e d . Test plans andprocedures should be reviewed and approved.The aud i to r should v e r i f y from SQA records t h a t t e s t procedureswere fo l lowed and t h a t a l l nonconformances observed dur ingt e s t i n g a re recorded in the NRCA system. In add i t ion to t e s t i n g ,the aud i to r should assess o t h e r methods o f V&V, i f used . Forexample , if inspec t ions o r another form of peer reviews a re usedto f ind problems, the a u d i t o r should ve r i f y t h a t th e r ecords o fth e review show t h a t they were done and t h a t cor r ec t ions andchanges agreed to in the review a re made in th e p r o d u c t .2 . produc t Examinat ionThe i n t e n t of examinat ion o f products i s two- fo ld : to see i fs t an d a r d s a re being fo l lowed, and to see i f s t a t u s i s accu ra t e l yrep o r t ed . Documents a re measured ag a i n s t documentat ionrequ i rement s to make sure t h a t a l l r equ i red documents e x i s t , andagainst documentat ion s tandards to ensure t h a t they have th ecor r ec t co n ten t and s t y l e . The au d i t o r must read enough of thedocuments to form an opin ion on th e above; t h a t i s , th e au d i t o rmust be ab le to determine t h a t a document p r es en t ed as showingth e indeed con ta ins des ign in format ion . On t h e o t h e rhand, t h e au d i t o r i s no t fo r th e t e ch n i ca lco r r ec t n es s o f the documents and should no t spend t ime t ry ing toa s c e r t a i n if th e documents a re cor r ec t .Code a l so i s examined to determine i f it meets s tandards . Codes t an d a r d s a re 1 to fo r i n t e r n a l documentat ion ,s ize o f modules; formats , and o t h e r such i tems t h a t th ea u d i t o r can v e r i fo r cons t ruc t s or va r i ab le

convent ions are more d i f f i c u l t to I f the e c thas a code s tandards checker , th e au d i t o r may run it on somecode, I f the standardS checker i s to be run a t a c e r t a i n s tep in

http;J/satc.gsknasa.gov/auditiaudg

7/30/2002 9: 19

8/4/2019 Nasa Gb a301

7/18of 18

the development process , o r i f peer reviews a re used to v e r i f ycoding s tandards , the aud i to r must have access to those records .Produc t s a l so a re examined to compare t h e i r s t a t u s with t h a trepor ted . Documents repor ted as complete , fo r example, shouldcon ta in a l l of the sec t ions given in th e t ab le of con ten t s (whichmay be prescr ibed by a documentat ion s tandard) , should be s ignedby th e approving au thor i t i e s , and should con ta in few, i f any, ToBe-Determined (TBDs) i t ems . Code implementat ion usua l ly goesthrough the s t eps of d e t a i l e d des ign , code, peer review, and u n i tt e s t . A module t h a t i s repor ted as complete should have gonethrough a l l of the above s t eps , should meet th e coding s tandards ,and should have whatever approvals are requ i red , Th e UnitDevelopment Folde r o r equ iva len t should con ta in a l l of theevidence to look a t s t a t u s of coding.3. SamplingDuring th e process o f checking r ecords and products , the aud i to rusua l ly cannot examine each and every i tem; therefore , somesampl ing process must be used. Th e aud i to r must decide on samples i z e s t h a t can be accommodated in the s i t e v i s i t . Th e samples izes must be ba lanced between completeness of coverage (somei tems from each product o r se t of records) and depth of coverage(number of i tems from a spec i f ic produc t o r s e t of r eco rd s ) . I fth e focus of th e a u d i t i s l imi ted, the sample s ize can be l a rgerfo r the spec i f ic product o r processes t h a t are to be covered. Indecid ing on sample s izes , th e aud i to r must al low t ime to fol lowup in more depth in areas where the i n i t i a l sample i nd ica t e sproblems. Th e s p e c i f i c products o r r ecords to be inc luded in th esample should be chosen by some "randomizing ll method and th epro jec t s t a f f should not be informed in advance which i tems wil lbe examined and which wil l not .C. Audi t Repor t ingOnce the in terviews and record examinat ion have been completed ri n i t i a l r e su l t s should be shared with the s t a f f of the aud i tedpro jec t dur ing an e x i t in te rv iew. Th e e x i t in te rv iew provides anoppor tun i ty t o c l e a r up misunders tandings and al lows pro jec ts t a f f to p resen t any in format ion t h a t they fee l the aud i to rf a i l e d to cons ider . In addi t ion, pro jec t s t a f f l e a r n immediatelyabout the problems t h a t have been found and can begin makingplans to cor r ec t them.After adjus t ing the i n i t i a l r esu l t s to r e f l e c t the in format iongathered in the e x i t in terview, th e aud i to r prepares a wr i t t enf i n a l r epor t . The repor t should be organ ized to high l igh t themost s i g n i f i c a n t r e su l t s , address ing both problems andcommendations, and should inc lude a genera l nar ra t ive of theaud i t . An example t a b l e of con ten t s fo r an audi t r epor t i s shownin Appendix A. Th e aud i t r epor t should be addressed to themanagement o f f i c i a l who ar ranged for the aud i t , i f the aud i t i sex te r na l ; o r di r ec t ed as requ i red by procedures , i f i n t e r n a l .Th e of the aud i t r epor t i s to p resen t a c l e a r p ic tu reo f the s t a tus of a development a c t i v i t y or a face t of th ea c t i v i t y to pr o jec t management. Th e r epor t must be c lear ,objec t ive , and f ac tua l . In some cases , the aud i to r w i l l f indt h a t , while a re fol lowed o r s tandards a remet r the o r e f f e c t i v e in a

S P ~ ~ : : " ~ ~ ~ ~ : ' ~ ' l O ~ ; f th e audi to r to notethe caused t and lor s tandard andinc lude them in the r epor t . however! t h a tthe audi tor i d e n t i f i e s should be r e l a t e d to

http:// sate. gsfc. nasa.gOY audit!aud g

7/30/20029: 19

8/4/2019 Nasa Gb a301

8/18of 18

con t r ac tua l ly - r equ i r ed procedures and s tandards ; the aud i to r ' sopinion of t h e i r d e s i r a b i l i t y should not a f f e c t h i s / h e revalua t ion of the adherence to them.D. Follow-upWhile the aud i to r ! s ro le i s e s se n t i a l l y f in ished a f t e r producingth e au d i t r epor t , a c t ions to reso lve def ic ienc ies i d e n t i f i e d int h a t repor t must be t aken by pro jec t management. Problems t h a ta re f ea s ib l e and r easonab le to cor r ec t should be conver ted toac t ion i tems and ass igned to appropr ia te i nd iv idua l s . Ar a t iona le should be developed fo r those t h a t a re not to becorrec ted . I t i s th e r e s p o n s i b i l i t y of the developers to improvet h e i r processes in response to def ic ienc ies i d e n t i f i e d by th eaud i t , The changes should be t racked to ensure they occur anda re ef fec t ive and the c losure of ac t ion i tems should bedocumented. In many cas es th e bes t wa y to determine i f theproblems have been solved i s through a fol low-up aud i t .IV . SQA AUDIT SCHEDULINGA. Rout ine Schedul ingI n t e r na l SQA aud i t s should be scheduled f requen t ly enough toiden t i fy po ten t ia l problems so t h a t no su rp r i s es develop fo rpr o jec t management. They should be scheduled r ou t ine ly dur ingth e l i f e cycle , p a r t i c u l a r l y around l i f e cycle phase t r ans i t ions .Th e most e f f e c t i v e i n t e r n a l aud i t programs schedule f r equen taud i t s of smal l areas o f p r o j e c t a c t i v i t y . Frequent audi t ing,combined with othe r SQA moni tor ing a c t i v i t i e s , would assurepr o jec t management t h a t th e ac tua l s t a tus of the pr o jec t i sknown, v i s - a -v i s s tandards , procedures , and schedules .External audi t s requi re more plann ing and in te rv iew t ime, but a rescheduled much l e s s f requen t ly . Th e most impor tant t ime fo r anex te r na l SQA au d i t i s a t th e s t a r t of th e implementat ion phase .This aud i t as sures t h a t th e deve lope r 1 s s tandards and proceduresa re implemented in a manner appropr ia te fo r th e pro jec t and t h a tthey a re being fol lowed. A second impor tant t ime in a p r o j e c t ' sl i f e cycle i s the beginning of system in teg ra t ion . An externa laud i t helps to assure t h a t the sof tware i s ready fo r i n t egr a t ion ,t h a t t e s t plans and procedures a re in place , and tha t proceduresfo r con t ro l of the sof tware a re not s h o r t - c i r c u i t e d . Projec tst h a t a re in t roub le o r have no i n t e r n a l a u d i t funct ion shouldhave more f r equen t ex te r na l aud i t s .Another f ac to r to cons ider in the schedul ing of aud i t s , e i t h e rin te rna l or ex te r na l , i s the r e su l t s of prev ious aud i t s . EachSQA au d i t should inc lude a review of the r e su l t s and ac t ion i temsfrom any prev ious aud i t s to confi rm c losure . I f there were anumber of problems and ac t ion i tems, aud i t s should be scheduledmore f requen t ly . Projec ts t h a t fol low t h e i r procedures , meett h e i r s tandards , and a re accura te in r epor t ing schedule ands t a t u s need l e s s f r equen t audi t ing,B, SQA Audits in Response to Warning SignsSome pr o jec t s may show i nd ica t ions of problems in the developmentprocess . When warning s igns appear , th e acql .l i rer should cons ideran externa l aud i t as p a r t of i t s Th e samecan be used the sof tware to s tep up o r

e f f ec t iveness of i t s i n t e r n a l aud i t program.Th e au d i t program should be i n t e n s i f i e d i f the exh ib i t sany of the fo1

http://sate,gsfc,nasa.gov!audit'aud

7/30/20029:19

8/4/2019 Nasa Gb a301

9/18f 18

Frequent schedule/miles tone changes . Incons i s tency of the developer l s organiza t iona l s t ruc tu rewith o r i g i n a l plans o r apparent incons is tency with the s t ruc tu reor funct ional i ty of the products to be produced. Unexplained f luc tua t ion o f pro jec t s t a f f l eve l o r under - o rover - s ta f f ing compared to es t imates . Increases in th e number of TBD i tems and ac t ion i temswithout adequate progress in so lu t ions . Th e i n a b i l i t y or unwil l ingness of the developer to prov ideadequate and accura te in forma t ion on p r o j e c t s t a tus , schedules ,and plans . Continual delay of scheduled software system c a p a b i l i t i e s tol a t e r r e leases /ver s ions . Unreasonable numbers of nonconformances or change reques ts ;fo r example, a l a rge number unreso lved , o r a sudden increase innumbers . An "unreasonable number" might be a suspic ious ly smallamount o f nonconformances fo r a complex system.There may be othe r i nd ica t ions t h a t a re apparent to p ro jec tmanagement in s p e c i f i c cases . An exper ienced p r o j e c t managerlsi n t u i t i o n t h a t something may be wrong i s a warning s ign t h a tshould be heeded. An ex te r na l aud i t i s a cos t e f f e c t i v e way fo ran acqui re r to ascer ta in the r ea l produc t s t a tus and r ea lprocesses be ing used by a deve lope r ; deve lope r management shouldhave an ongoing aud i t program to assure t h a t no su rp r i s es are ins to re fo r them.C. Announcing AuditsAdequate no t i f i ca t ion of aud i t s should be provided to thedevelopers fo r a number o f reasons . Unannounced ( su rpr i se)audi t s a re d i s r up t ive and demoral iz ing to the development s t a f fand should be avoided. Th e i n t e n t o f an aud i t program should beto help promote conformance with s tandards and procedures and th erepor t ing of accura te s t a t u s , not to "ca tch in the act!! thosel lguiltyll of v i o l a t i o n s . An announced schedule of aud i t s al lowsproper prepara t ion in t erms o f having requi red documentationava i l ab le and being prepared to answer the aud i to r1s ques t ions .v. SQA AUDITS DURING THE SOFTWARE LIFE CYCLEA. Software Concept and I n i t i a t i o n PhaseDuring th e concept and i n i t i a t i o n phase , th e sof tware concept i sdeveloped, th e f e a s ib i l i t y of th e sof tware system i s evalua ted ,th e acqu is i t ion s t r a tegy i s developed, and, i f a con t r ac t i s tobe used to acquire the sof tware , procurement i s i n i t i a ~ e d and acon t r ac t i s awarded. Before se lec t ing an organiza t ion to performa pro jec t , the acqu i r ing organiza t ion can reques t a pre-award SQAaud i t . Th e in ten t o f th i s t y ~ e of aud i t i s s l igh t ly d i f f e r en tfrom aud i t s performed l a t e r in the l i f e cycle . Since there a reno a c t i v i t i e s underway on the sof tware t h a t i s to be developed,the aud i to r can only review the prov ider ' s lIcorporatel1 or gener ics tandards and t and pas t ec t s . I f , theseshould be examined in the context of the e c t f sotha t t h e i r e f f ec t iveness can This type o f aud i taud i to r ,

and s tandards fo r th e e c t a re formula ted

http://satc. gsfc. nasa.gOY lauditfaudg

7/30/20029: 19

8/4/2019 Nasa Gb a301

10/18

during th i s phase. The SQA s t a f f of the acquirer should ensuret ha t standards and procedures adopted are appropriate for thepro jec t and are audi table , i . e . , have a c lea r documentationt r a i l , with easy- ta- fo l low s teps . They also should make suret ha t the contract allows external audi ts and requires in te rna laudi ts .8. Software Requirements PhaseDuring the software requirements phase, the software concept andal located system requirements are analyzed and documented assoftware requirements . Tes t planning is begun, with a method forver i fy ing each requirement iden t i f ied and included in aprel iminary t e s t plan. Risks are iden t i f ied and r i sk managementcontrol mechanisms are es tab l i shed . The s ize and scope of theremainder of the pro jec t is reevaluated , and changes in resourcesand schedules are made. Methods, s tandards , and procedures arede ta i l ed and put in place . The phase ends with a requirementsreview, a t which the requirements are agreed to between theacquirer and developer and put under CM.In ternal audi ts during th i s phase concentrate on the process ofdeveloping, documenting, and contro l l ing the requirements . Someprocess should be in place to control the requirements and dra f tdocuments as they a re developed. This process probably wil l ber e la t ive ly informal , an d may include NRCA and an act ion itemt racking system. There may be procedures for repor t ing onprogress, es t imat ing system and pro jec t resources, and r i skassessment. Al l of these can be audi ted to the extent tha tcontro l led processes are in p lace. In addi t ion to procedures,audi tors should ver i fy tha t requirements documents follow theformat speci f ied in the documentation standard.An external audi t , i f one i s performed during th i s phase, maylook a t the same i tems t ha t are covered by an i n t e rna l audi t . Inaddi t ion , an external audi t can cover the same i tems as l i s t edfo r a pre-award audi t .C. Software Archi tec tura l Design PhaseThe object ive of the arch i t ec tu ra l design phase i s to develop anoveral l design fo r the software , al locat ing a l l of therequirements to software components. Th e software requirementsare contro l led and managed, and documents baselined fol lowing therequirements phase are changed only by a formal process . Thephase ends with the pre l iminary design review, during which theacquirer and developer agree on the archi tecture of the systemtha t i s to be produced. Rework and act ion i tems resul t ing fromthe review are t racked and completed.In ternal and external audi ts during th i s phase should include thedesign documentation, ver i fy ing tha t format standards are met.The aud i to r should assure that a l l requirements are beingal located to software components. I t i s especia l ly important toaudi t the configuration control mechanisms for the requirementsto make sure t ha t unauthorized and uncont ro l led requirementchange and growth is not occurring. In addi t ion , i tems such asthose mentioned in the previous phase, i . e . , s ta tus repor t ing,act ion i tem t racking, and nonconformance repor t ing should beaudited.D. Software Detai led Design Phase

th e deta i led ,,'''''Me I th e archi t ec tu ra l i sto the uni t level In te r face cont ro l documents a re

http:Hsatc,gsknasa.gov/audit/audg

7/30/20029:19

8/4/2019 Nasa Gb a301

11/18

of 18

completed and t e s t p lan s rev i sed . Cons t ra in t s and objec t systemr esource l i m i t s are rees t imated and analyzed, and s t a f f i n g andt e s t r esources a re va l ida t ed . Th e phase ends with the c r i t i c a ld es ig n review, and th e d e t a i l e d des ign i s base l ined .Audi ts dur ing t h i s phase should focus on th e p r o g r es s anddocumentat ion of the d e t a i l e d des ign . I f u n i t developmentf o lde r s (o r othe r s imi l a r documentation} a re used, they should bes t a r t e d dur ing t h i s phase , and can be aud i ted . As au d i t i n g i sdone, repor ted s t a t u s should be compared with th e ac tua l s t a t u s .An y discrepancies should be noted . Both th e requ i rement s and th ea r c h i t e c t u r a l des ign should be under CM and th e CM p r o ces s shouldbe aud i ted . Other i t ems l i s t e d in th e desc r ip t ions of theprev ious phases a re still ap p l i cab l e .E. Sof tware Implementat ion PhaseDuring th e implementat ion phase , the sof tware i s coded and u n i tt e s t e d . A ll documentat ion i s produced i n q u a s i - fi n a l form,i n c lu d in g i n t e r n a l code documentat ion . At th e end of the phase ,a l l requ i red products should be ready fo r d e l i v e ry , s u b j ec t tomodi f ica t ion dur ing i n t e g r a t i o n and t e s t i n g . Audi t s dur ing t h i sphase check the r e s u l t s o f des ign and coding, eM a c t i v i t i e s andprogram l i b r a r y , NRCA p r o ces s , and schedule and s t a t u s of thep r o j e c t .I n t e r n a l aud i t s should be f r equen t dur ing t h i s phase . Thep r o j e c t s t a f f i s usua l ly a t its maximum, and t h e re a r e a gr ea tnumber of simul taneous a c t i v i t i e s . SQA aud i t ing i s one of themore impor tan t ways fo r management to keep the p rocess underco n t ro l , assure t h a t q u a l i t y products a re be ing developed, andt h a t s t a t u s i s a c t u a l l y as repor ted . Completed products a rebe ing s en t to t e s t as they a re r eady , and the p roducts and t h e i rco n t ro l process should be aud i ted . Audi ts should inc lude codeaud i t s to make sure coding s tandards a re being fol lowed and t h a ti n t e r n a l code documentat ion s tandards a re met. I f i n s p ec t i o n s o rsome othe r form of peer reviews a re done, th e aud i to r shouldcheck t h a t they a re completed on a l l products and t h a t ac t ioni tems r esu l t ing from them a re ca r r i ed out .An ex t e rn a l au d i t i s most e f f e c t i v e i f done ea r ly in th eimplementat ion phase . At t h i s p o i n t in th e l i f e cycle , a l lco n t ro l procedures are in opera t ion and a l l s tandards a re in use .This ex t e rn a l SQA aud i t as sures t h a t they a re being fo l lowedc o r r e c t l y and t h a t s t a t u s i s c o r r e c t l y r ep o r t ed . I f any problemsa re noted , it i s ea r ly enough fo r meaningful change andcor r ec t ive ac t i o n .

Sof tware I n t egr a t ion and T e s t PhaseThe o b j ec t i v es of the i n t egr a t ion and t e s t phase a re to i n t egr a t eth e sof tware u n i t s in to a completed system, d i s co v er and cor r ec tany nanconformances, and demons t r a t e t h a t the system meets itsrequ i rement s . Th e phase ending review i s the t e s t read inessreview, dur ing which the p rov ides to t h e acq u i r e revidence t h a t th e sof tware system i s r eady fo r accep tancet e s t i n g . During t h i s phase , th e t e s t plan i s executed , th edocumentat ion i s updated and completed , and the p roducts a ref ina l i zed fo r d e l i v e ry .

t h i s phase! i n t e r n a l aud i t s inc lude any and a l l o f th ei terns However, i n t e r na l aud i t s shouldconcen t ra te on t h a t made to cor r ec tnonconformances discovered the a re con t r o l l ed ,

and documented. Audi ts of the eM and NRCA processes ,

http://satc.gsfc.nasa.gov/auditiaudg

7/30/20029:19

8/4/2019 Nasa Gb a301

12/18

and computer program l ib rary are highly important . The SQA audi tshould include a check of the formal t e s t procedures and the t e s tr esu l t s . In tegrat ion and t e s t is often the most confusing andt ime-pressured par t of a project , and there i s a tendency todiscard standards an d procedures due to th i s pressure .External audits during th i s phase should concentrate on the samei tems as in ternal audi ts , with addi t ional emphasis on assur ingcompletenessi tha t i s , tha t t e s t ing has not been shortchanged inorder to meet schedules.G. Software Acceptance and Delivery PhaseDuring the acceptance and del ivery phase, the formal acceptanceprocedure is car r ied O l i t . As a minimum, there i s a requirementsdriven demonstration of the software to show tha t it meets thoserequirements . The process also may include acquirer t es t s , f i e ldusage, or other arrangements tha t are intended to assure tha t thesoftware wil l function correc t ly in i t s intended environment.This phase is very much l ike the end of the previous phase, withsystem t e s t s being run, nonconformances noted, and correct ionsbeing made to the software , documentation, and data bases. Thei tems to be audi ted are simi lar , especia l ly the CM and NRCAprocesses.H. Software Sustaining Engineering an d Operat ions PhaseDuring th i s phase of the software l i f e cycle, the software i sused to achieve the object ives for which i t was acquired.Corrections and modifications are made to the software to susta ini t s opera t ional capab i l i t i e s an d to upgrade i t s capaci ty tosupport i t s users . Software changes may range in scope fromsimple correct ive act ion up to major modificat ions tha t require afu l l l i f e cycle process.In ternal audi ts should respond to the ex ten t and type of changesbeing made to the system. I f there i s only a low l eve l ofcorrect ive ac t ion, then audi ts may be l imited to the eM and NRCAprocedures an d to ver i fy ing tha t qual i ty i s being maintained inthe products . I f subs tan t ia l modificat ions are being made,however, then a fu l l or min i - l i f e cycle should be in place andaudi ts should be performed as described fo r the appropr ia tes tage .When long term susta in ing engineering i s being performed, anexterna l audi t should be done per iodical ly to assure the acqui rertha t product qual i ty i s maintained an d susta ined . A minimum ofone externa l audi t p er year i s recommended; more i f the l eve l ofchange act iv i ty i s high.VI. PREPARING A CHECKLISTAn audi t checkl is t is a l i s t of i tems tha t the audi tor intends toexamine and quest ions the audi tor intends to ask during the s i tev i s i t port ion of the audi t . While a generic check l i s t may beused as a basis fo r a l l audi ts , be t te r re su l t s wil l be achievedi f the generic checkl is t i s ta i lored for each audi t . Tailoringconsis ts of choosing appropr ia te i tems or questions from thecheckl is t , expanding the level of deta i l , addingaddi t ional quest ions an d , an d the of thef i t the ec t ! s nomenclature. Information forcorne from the cont rac tpract ices , and re su l t saudits Addit ional information to be forshould include the s t ructure of the

http://satc.gsfc.nasa.gov/auditiaudg

7/30/20029:19

8/4/2019 Nasa Gb a301

13/18

organ iza t ion and p r o j e c t , l i f e cycle phase , and a u d i t focus .In developing th e c h e c k l i s t , th e aud i to r should be ca re fu l not toover look impor tan t in format ion t h a t appears to be obvious . Forexample , assuming th e p r o j e c t has a produc t s p e c i f i c a t i o n may bee r roneous ; adding t h a t i tem to th e c h e c k l i s t w i l l help to assuret h a t the in format ion i s conf i rmed.A sample g en er i c c h e c k l i s t , div ided by t o p i c , i s provided inAppendix B. Under each top ic i s a s e r i e s of t yp ica l ques t ionst h a t should be addressed i f t ha t top ic i s going to be p a r t of th ea u d i t . To t a i l o r t h i s check l i s t , th e a u d i t o r should determinewhich top ics apply to t h e au d i t and whether ques t ions should beanswered by in te rv iews , examinat ion of the sof tware p roducts anddocuments , examinat ion of records , o r a combinat ion of methods.Th e au d i t o r then should s o r t th e q u es t i o n s by th e method t h a t i sin tended to be used to answer them, and f u r the r , by t h e p rec i s esource to be used . For example , ques t ions about how CM o p er a t e smight be asked of th e CM manager dur ing an i n t e r v i ew , bu t some o fthose same q u es t i o n s migh t be di r ec t ed a t th e per son who o p er a t e sth e p r o j e c t ! s computer program l i b r a r y . Answers t o o t h e r eMq u es t i o n s might be found through an examinat ion of the r ecords o fth e CM process i still othe r s by an examinat ion o f code anddocumentat ion p r o d u c t s .As much as p o s s i b l e , q u es t i o n s should be phrased in t erms of thespec i f i c p r o j ec t and organ iza t ion be ing aud i ted , and should usethe names and t erms t h a t th e pr o jec t u s es . This t a i lo r ing w i l lt ake some work on th e p a r t o f th e aud i to r , but t h i s e f f o r t w i l lbe r epaid by th e f a c t t h a t e f f e c t i v e communication w i l l bees t ab l i shed e a r l i e r .The p a r t s of the t a i l o r e d check l i s t t h a t w i l l be answered by anexaminat ion of records o r products should be put on a form fo ruse on-s i t e . The form can be simple , b u t should al low space fo ranswers to each ques t ion and ad d i t i o n a l comments. The formshould , i f p o s s i b l e , al low th e checking o f boxes o r s imple en t ryof information.As the aud i to r proceeds wi th th e s i t e v i s i t , th e check l i s t s andforms can be completed with th e in format ion ob ta ined . Th eau d i t o r must r e t a i n th e f l e x ib i l i t y to modify th e forms o rques t ions as in format ion i s gathered . Addi t ional ques t ions a rel i k e l y to be sugges ted by answers given , and forms ma y not havebeen proper ly made in advance to r ecord th e r e a l s i t u a t i o n . Iti s important to remember t h a t th e c h e c k l i s t and forms der ivedfrom it a re guides , and t h a t th e o b j ec t i v e of t h e au d i t i s tounders t and and r ep o r t on the ac tua l s t a t e o f a f f a i r s in th edeveloping o rg an i za t i o n .VII . AUDITING IN THE ABSENCE OF STANDARDS AND PROCEDURESAn au d i t o r ma y be asked to "audit!! a pr o jec t t h a t l acksdocumented s tandards and procedures , perhaps because of warningi n d i ca t ed in Chapte r IV. Most of t en , t h i s t ype of au d i tw i l l be ex t e rn a l to th e pr o jec t , even i f th e a u d i t o r i s employedby th e developing organ iza t ion , because a developer t h a t does n o thave documented s t an d a r d s and procedures i s unl ike ly to have ani n t e r n a l au d i t program.

ind iv idual t echn ica l o r t h e i r managers . e c t shandle changes and and t e s t t h e i r softWare"methods may be somewhat act-hoc and on th eind iv idual s involved in a f ic ease l but do e x i s t ,

http://satc.gsfc. nasa. gOY !auditlaudg

713()j2002 9:19

8/4/2019 Nasa Gb a301

14/18

of 18

documented o r not . Th e r o l e of th e a u d i t o r i s to discover anddocument th e "s tandards n and nprocedures!! t h a t a re ac tua l lyfol lowed.After th e aud i to r has determined from in te rv iews what "standards!!and nprocedures!l a re fol lowed, th e r e s t of th e aud i t can proceedl i k e any othe r aud i t . That i s , th e a u d i t o r can fol low th eprogress o f co n t ro l p a th s and de te rmine th e ex ten t to which th eprocedures a re fol lowed versus th e number of excep t ions t h a t a real lowed. Th e a u d i t o r can sample th e products and r a te t h e i rconformance to th e (unwri t ten) s tandards .The aud i to r must g a t h e r enough in format ion to evalua te th es u i t a b i l i t y and cons i s tency of th e unwri t t en s tandards andprocedures . Th e a u d i t o r may be exper ienced enough to do th eevalua t ion , o r th e aud i to r may wish to l eave the evalua t ion tothe management to which he/she wi l l r epor t . In e i t h e r case , th eaud i to r has t o ga the r in format ion on product q u a l i t y , cons i s tencyof app l i ca t ion of the unwri t t en r u l e s , t he adequacy of t e s t i n gand reviews, and i n s t an ces of confusion and /or e r r o r t h a t mayhave r e su l t ed from unce r t a in ty . This in format ion i s then usedfo r eva lua t ion .

VIII . QUALITIES OF AN AUDITORTh e major con t r ibu t ion of an i n t e r n a l or ex te r na l aud i to r topr o jec t success i s th e co l l ec t ion and pr e sen ta t ion of in format iont h a t al lows p r o j e c t management a c l e a r view of the product ! sac t u a l s t a t u s and th e ac tua l compliance with s tandards andprocedures . This requi res an i mp ar t i a l aud i to r . In p a r t i c u l a r ,an in te rna l aud i to r must remember t h a t cover ing up problems, dueto fee l ings of empathy with the pr o jec t s t a f f o r a des i re topresen t the developer ! s organ iza t ion in a good l i g h t / i scounterproduct ive . Problems t h a t a re not brought to l i g h t wi l lnot be so lved , and may r e s u l t in much l a r g e r problems l a t e r inth e l i f e cycle .A good aud i to r should have a bas ic unders tand ing o f the sof twaredevelopment l i f e cycle and the products and processes involved ineach of i t s phases . I f an aud i to r i s expected to eva lua te th es tandards and procedures used by th e developer and to judge t h e i rimpact on product q u a l i t y and p r o j e c t schedule , then he/she needss i g n i f i c a n t exper ience and background in sof tware development andsof tware management. I t h e lp s i f th e a u d i t o r i s knowledgeableabout the type of sof tware being aud i ted , and i s aware of th espec i f ic sof tware development procedures used in th e pr o jec t . Iti s usefu l i f th e aud i to r i s exper ienced or t r a ined in aud i t ingt echn iques .IX . TECHNIQUES AND TOOLSThe most f requen t ly used t oo l fo r an SQA a u d i t i s an au d i tcheck l i s t . Th e check l i s t must be t a i lo r ed to the pr o jec t to beaud i ted , as it prov ides a list of ques t ions t h a t must be answeredabout t h a t p a r t i c u l a r p r o j e c t .Automated t o o l s , e i t h e r brought by th e aud i to r or provided by th epr o jec t , may be used i f compat ib le with th e p r o j e c t ' s s tandardsand procedures . For example, th e p r o j e c t may have a s tandardschecker fe r code. Th e aud i to r can run th e checker on a sample ofth e code, o r can t h a t the runs th e checker .Th eas s i s tedprocessor .transferred o r

precess a l so may bec h e c k l i s t in a database o r werdbe au tomat ica l

to the a u d i t on a

http://satc.gsfc.nasa.goviaudit/aud

7/30/20029: 19

8/4/2019 Nasa Gb a301

15/18of 18

computer.APPENDIX A, SQA AUDIT REPORTThe following i s the minimum content fo r an SQA audi t repor t .1 . Backgrounda. Ident i ty of audi tb. Date of audi tc. Audit team membersd. Current phase of development2. Findingsa. Version of products audi tedb. Anomalous condit ions encounteredc. Recommendation for each anomalous condit ion ( ifapplicable)3 . Summarya. Summary of f indingsb. Statusc. Date of follow-up or next scheduled audi t

APPENDIX B, SQA AUDIT CHECKLIST QUESTIONSThe fol lowing i s a sample master l i s t of questions tha t can bet a i lo red fo r an SQA audi t . Questions appropriate to a specif icaudi t should be selected an d then modified to r ef l ec t loca lterminology or procedures. The questions should be placed on aform tha t allows space for recording answers.Questions shown in i t a l i c s are mainly for use in the s t a f finterviews.Software AssuranceHas an SQA plan been prepared? Is it maintained current withprogram requirements?Ha s the SQA plan been submitted fo r approval?Does the SQA plan include or def ine , SQA requirements and ac t iv i t ies to be implemented? Schedule showing when each of the ac t iv i t i e s wil l beimplemented? Budget for act iv i t i es? Speci f ic organizat ional assignments? In teract ion between SQA and the overal l development e f fo r t? SQA par t ic ipa t ion in the overal l change management process? SQA par t ic ipat ion in the overal l t e s t process?Is there evidence that SQA planned ac t iv i t i e s are implementedthroughout the l i f e cycle?Development DocumentationAre standards for prepara t ion of del iverable documentationestabl ished?Does the documentation meet the s tandards?Are procedures establ i shed and documented to assure thatstandards are followed?Do thetha t a re underchanges reviewed in theAre methods es tabl ished

to software documentationmanagement control? Are thesame manner as the base document?for t r aceab i l of documentation,Ar e the contents of del iverable documents c lea r , concise ,

h t t p : / / s a t c , g s f c . n a s a ~ g o v / a u d i t l a u d g

7/30120029: 19

8/4/2019 Nasa Gb a301

16/18of 18

complete , and unders t andable?Are procedures es t ab l i shed to enforce cons is tency i n w r i t i n g ?Are review t eams f ami l i a r with th e mat e r i a l be ing reviewed tode tec t incons is tency?I s approva l au thor i ty fo r de l ive r ab le documentat ion c l e a r l ys t a t ed?I s r eq u i r ed documentat ion provided to th e acq u i r e r in a t imely ,r espons ive manner?Are s u f f i c i e n t copies fu rn ished?Are es t ab l i shed procedures fo l lowed in the p roduct ion of bothde l ive r ab le and nondel iverab le documents?Does th e documentat ion i n th e development fo l d e r match th e phaseo f th e life cyc le?Does th e l ev e l o f d e t a i l in documentat ion look r easonab le?CodeDo code , pro log , and Program Design Language (PDL) adhere to a l lpr eva i l i ng s tandards and convent ions?Are n eces s a r y elements o f th e pro log comple te ; e . g . , a re a l l d a taelements descr ibed , a l l s u b r o u t in es def ined?I s i n t e r n a l code documentat ion p res en t in amounts requ i red bys tandards?I s th e code co n s i s t en t wi th its des ign , as p r es en t ed in itspro log and PDL?Does th e code appear to be cor r ec t fo r t e s t cas es t h a t can bev e r i f i e d by a quick , vi sua l i n s p ec t io n ?I s a l l debug code c l e a r l y i den t i f i ed?Are all s tu b s and t e s t f i l e s i den t i f i ed?Do t e s t cas es appear adequate based on the PDL?Conf igura t ion ManagementHas a sof tware conf igura t ion management (CM) plan been developed?Has t h e p l an been basel ined? Provided to t h e acq u i r e r ?Are CM i ns t r uc t ions fo r i d e n t i f i c a t i o n o f b as e l i n e i tems andsubsequent r ev i s i o n o r v e r s i o n s be ing fo l lowed?Are CM procedures i n p l ace which r eq u i r e approval a u t h o r i t y fo radding and removing i t ems in th e program l i b r a r y?I s th e CM o r g an iza t i o n adequate ly s t a f f ed , f u l l y funded, andre spons ive? Are r e s p o n s i b i l i t i e s c l e a r l y under s tood?Do b as e l i n e documents comply wi th co n t r ac t requ i rement s?Do th e approved s p e c i f i c a t i o n s serve as a b as e l i n e fo r co n t ro l o fchanges?I s a list o f approved s p e c i f i c a t i o n s mainta ined? Curren t?Changes posted?Are procedures es t ab l i s h ed fo r th e product ion o f sof twaredocumentat ion adequate and r i g i d l y enforced?Are procedures fo r handl ing problem r ep o r t s adequa te ande f f i c i e n t ?Has a Conf igura t ion Cont ro l Board (CCB) been es t ab l i s h ed ? Whoa re th e members? Is SQA r epresen ted? Do a l l merrbers a t t en dregu lar ly? Are CCB ac t i o n s handled in a t imely manner? Areagenda and minutes publ i shed? Are CCB ac t i o n i tems fo l lowed up?Are CM s t a t u s account ing documents mainta ined? Are cu r r en t ?Does th e eM p lan address conf igura t ion au d i t s ?Have formal co n f ig u r a t i o n aud i t s been conducted o r planned( inc lud ing FCA and peA)?computer Program LibraryHas a Computerl i b r a r i a n alJP,ointcetl?Have

been es tabl i shed? A programbeen i den t i f i ed fo r : cont rols?item con t ro l s? Problem r ep o r t handl

http://satc.gsk nasa. gov! auditJaudg

7/30/20029:19

8/4/2019 Nasa Gb a301

17/18of 18

I s th e program l i b r a r i a n complying with es t ab l i shed procedures?Are problem r epor t s implemented i n to appropr ia te developmentfo lders?Are computer program vers ions accura te ly i d e n t i f i e d , con t r o l l ed ,and documented through th e l i f e cycle? Is an automated sourcecon t r o l system used? Is it adequate ly maintained?How i s th e l i b r a ry con t r o l l ed ( e r ro r r epor t , change reques t ,e tc . ) ?Are only au thor ized /approved modif ica t ions made to source andobjec t programs r e l ea s ed to th e l i b r a r y? How i s it con t r o l l ed( e r r o r r epor t , change reques t , e t c . ) ?What measures a re be ing taken to assure a l l approvedmodi f ica t ions a re p ro p e r l y i n t egr a t ed and t h a t sof tware submit tedfo r t e s t ing i s th e cor r ec t vers ion?I s nondel iverab le sof tware moni tored and con t r o l l ed to th e ex ten tspec i f i ed in th e development plan?Are development folders r egu la r ly submit ted to th e programl i b r a r i a n ?Does a l i b r a ry documentat ion index ex is t? Is it curren t?Does a log e x i s t showing what mat e r i a l has been checked in andou t of the l i b r a r y? Does it appear accura te?Does a l l submit ted code inc lude proper t r ansmi t ta l information?I s t h i s ava i l ab le fo r review?I s documentat ion updated to correspond with newly submit ted code?Are a l l i tems placed in th e program l i b r a ry as s ig n ed ani den t i f i ca t ion number r e l a t e d to the vers ion number? Does t h i snumber r e l a t e to th e assoc ia ted documentat ion?Is th e flow through a change cycle c lear , e f f i c i e n t , documented,and cor rec t? (Test s ev e ra l samples . )Nonconformance Repor t ing and Correc t ive ActionHave procedures assur ing prompt de tec t ion and cor r ec t ion ofde f i c i enc ie s been es t ab l i s h ed ?Are data ana lyzed and problem and def ic iency r epor t s examined tode te rmine ex t en t and causes?Are t rends in performance o f work analyzed to preven t developmentof nonconforming products?Has cor r ec t ive ac t ion been documented accu ra t e l y on problemrepor t s?Has cor r ec t ive ac t ion been reviewed and moni tored to determineadequacy, e f f ec t iveness , and whether cont rac t requi rements a rebe ing met?Are a l l cor r ec t ive ac t ion r epor t s and analyses on f i l e ?I s t he r e management suppor t fo r th e cor r ec t ive ac t ion sys tem?I s the program l i b r a r i a n fol lowing procedures fo r main ta in ingco n t ro l and s t a t u s of problem repor t s?Are discrepancies genera ted by nondel iverab le computer programst r e a t e d th e same as those fo r d e l i v e rab l e s ?Are problem r epor t s pe r t a in ing to a u n i t conta ined with in th edevelopment fo l d e r fo r t h a t uni t?Are th e sof tware developers complying with th e requi rement togenera te problem r epor t s dur ing in tegra t ion?I s there documented approval fo r a l l changes to i t ems underconf igura t ion con t ro l? Do a l l forms have requ i red s ignatu res?Ver i f i ca t ion and Val ida t ionHave th e sof tware reqUirements been analyzed to determinet e s t a b i lAre t e s t o b j ec t i v esdemonstra te sof tware

? Aref ea s ib l e , and s u f f i c i e n t toperformance to meet co n t r ac t u a lunders tood e c t personnel?Are th e t e s t phi losophy and based on t ha t

http://sate. gs fc. nasa.gOYlaud itlaudg

7/30;20029;19

8/4/2019 Nasa Gb a301

18/18

are accep tab le to SQA? Is there a procedure to monitorassumptions and a way to a l e r t the t e s t d i r e c t o r i f an assumptioni s unaccep tab le?Do t e s t plans and procedures comply with spec i f ied s tandards andcont rac tua l r equ i rements?Are the t e s t plans and procedures approved by th e acqui re r , whererequired?Are a l l t e s t t o o l s and equipment i den t i f i ed , def ined , ca l ib ra ted ,and con t r o l l ed p r i o r to t e s t ing the sof tware? I s a l l necessaryt e s t hardware ce r t i f i ed (both computer and a n c i l l a r y ) ?I s sof tware base l ined p r i o r to tes t ing?Are th e cor r ec t ve r s ion o f sof tware and assoc ia t ed documentat ionc e r t i f i e d p r i o r to tes t ing?Are acceptance t e s t s monitored by an SQA r epr e sen ta t ive? By th eacqui re r , when requ i red? I f not , then who monitored th e t e s t s?Are t e s t s conducted accord ing to t e s t plans and procedures?Have t e s t r esu l t s been c e r t i f i e d by par t i c ipa t ing members tor e f l e c t th e ac tua l t e s t f indings?Have t e s t r epor t s been reviewed and c e r t i f i e d ? By whom? Aredef ic ienc ies documented in problem repor ts?Has t e s t - r e l a t e d documentat ion been mainta ined and con t r o l l ed toal low r epea tab i l i t y of t e s t s?I s there a t e s t v e r i f i c a t i o n matr ix to assure a l l requirementsare tes ted? Does it look reasonable?Are t e s t procedures c l e a r and repea tab le?Do ac tua l and expec ted t e s t r esu l t s match? I f not , has a problemr epor t been f i l ed?Projec t S ta tusDo complet ion dates in development fo lde r s / s t a tus shee ts agreewit-h s t a tus r epor t to management? I f not , ho w grea t i s thedi f fe rence?According to th e development/management plan, th e pro jec t whereit -should be? What ac t i v i t i e s should be cur rent? How should th epro jec t be s taf fed? What in termediate pro jec t s should bedel ivered? What reviews o r miles tones should have occurred?Where does th e p r o j e c t a c t u a l l y s tand now? Determine:

Current phase Act iv i t i es l eve l s Staf f composi t ion Documents del ivered Milestones reached Resul t s o f reviews.

http://satc.gsfc. nasa.gOYfaudit!audg