Apollo Experience Report Reliability and Quality Assurance

8/8/2019 Apollo Experience Report Reliability and Quality Assurance

1/37

E / 7 3 - 3 / 7 6N A S A T E C H N I C A L N O T E NASA TN D-7438

a0Mw

APOLLO EXPERIENCE REPORT -RELIABILITY AND QUALITY ASSURANCEby K . P . SperberLyndon B . Johnson Space CenterHouston, Texus 77058N A T I O N A L A E R O N A U T I C S A N D S PA C E A D M I N I S T R A T I ON W A S H I N G T O N , D. C. SEPTEMBER 1973


2/37

.... Repor t No. 2. Governm ent Accession No.NASA TN D-7438

1 Tit le and Subt i t leAPOLLO EXPERIENCE REPORTRELIABILITY AND QUALITY ASSURANCE

I. Author(s)K . P. Sperber, JSC

3. Performin g Organization Name and AddressLyndon B. Johnson Space CenterHouston, Texas 77058

13. T y p e o f Report and Period Covered

3. Recipient's Catalog No.

5. Repor t Date6. Performing Organizat ion Code

September 1973

8. Performing Organizat ion Repor t No.JSC S-371

10. Work Unit No.951-18-00-00-72

11 . Contract or Grant No.

2. Sponsoring Agency Name and AddressNational Aeronautics and Space AdministrationWashington, D. C. 20546

I Technical Note14. Sponso ring Agency Code

7. Key Words (Suggested by Author(s))Reliability Assurance* Quality ControlApollo Spacecraft'Production Management'Reliability Te sts

18. Distr ibut ion Statement

19. Security Classif. (of this repor t) 20. Security Classif. (of this page) 21 . NO. of PagesNone None 37

22 . Rice fDomestic, 53.00Foreign,35.50 I


3/37

CONTENTS

Section PageSUMMARY. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1MANAGEMENT OF RELIABILITY AND QUALITY ASSURANCE . . . . . . . . 2

Reliability and Quality Assurance Requirements for Space Systems 3ontractors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Government Source Inspec ion . . . . . . . . . . . . . . . . . . . . . . . . 5Lyndon B. Johnson Space Center Procurement of Space-Flight Equipment . . . 5

6uality Assurance Functions of Government Agencies . . . . . . . . . . . .6ocumentation Requirements . . . . . . . . . . . . . . . . . . . . . . . . .

Reliability and Quality Information . . . . . . . . . . . . . . . . . . . . . . 8Assessment of Reliability and Quality Assurance FunctionsAssessments of Space-Flight Equipment . . . . . . . . . . . . . . . . . . .

. . . . . . . . . 99

Training 10. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11esign and Procedural Standards . . . . . . . . . . . . . . . . . . . . . . .

ENGINEERING FUNCTIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 21 2eliability Predictions . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Fai lure Mode and Effects Analysis 1 214

Nonmetallic Materials . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15. . . . . . . . . . . . . . . . . . . . . .

Electrical, Electronics, and Electromechanical Parts . . . . . . . . . . .

Fluid Systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17Problem Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19

MANUFACTURING FUNCTIONS . . . . . . . . . . . . . . . . . . . . . . . . . 2 12 1General . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

iii


4/37

Section PageSpacecraft Wiring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21Electr ical Connectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25Contamination Control . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26COMPONENT TESTS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .Background 27Certification of Space-Flight Equipment . . . . . . . . . . . . . . . . . . . 28Acceptance Test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29Nondestructive Tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29

CONCLUDING REMARKS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30REFERENCES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

iv


5/37

~

TABLE

Table PageI SUMMARY OF ACCEPTANCE TEST FAILURES FOR THE LM ANDCSM BY INDIVIDUAL SUBSYSTEM. . . . . . . . . . . . . . . . . . 29

FIGURES

Figure Page1 Format for fai lure mode and effects analysis . . . . . . . . . . . . . . 132 Format for single fai lure point summary . . . . . . . . . . . . . . . . 143 Service propulsion system tanks installed in the Apollo service

m o d u l e . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174 Crew compartment electri cal harness . . . . . . . . . . . . . . . . . 2 15 Special fixture for assembly of the Apollo command module crewcompartment wire harnes s . . . . . . . . . . . . . . . . . . . . . . 226 Special Teflon-lined trough fo r installation of the Apollo crewcompartment wire har ness . . . . . . . . . . . . . . . . . . . . . . 237 Lunar module cabin wiring cri mp and solder spl ices and end ca ps . . . 248 Command module cable tr ay s . . . . . . . . . . . . . . . . . . . . . . 249 Apollo spacecraft fa ilu res . . . . . . . . . . . . . . . . . . . . . . . . 2810 Distribution of Apollo spacecraf t certif ication fa ilures byenvironment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

V


6/37

APOLLO EXPER I NCE REPORTREL IAB IL ITY AND QUAL ITY ASSURANCEB y K . P . Sperber

L y n d o n B. J o h n s o n Space Cen te rSUMMARY

The reliability and quality of the Apollo spacecraf t and associa ted equipment re-sulted fro m the application of reliability and quality assuran ce prac tice s and from thetechniques developed and adopted during the engineering and manufacturing phases.Two reliability and quality prac tice s adjusted as the program developed included: (1) theestablishment of NASA Llyndon B. Johnson space Center ( formerly the Manned Space-craf t Center) and contractor reliability and quality assura nce management review meet-ings, and (2) specifying in the contract, the scope, format, and content of reliabi lity andquality plans.

A contractor p art s program was implemented to increase the integrity of theequipment. This program included requirements fo r pa rt s derating and screening,control of th e use of nonmetallic mater ial s in the spacecraft, procedur es and controlsfo r assu ring cleanliness of both the equipment and fluid systems, and a comprehensivetest ing program. These engineering and manufacturing techniques were developed asthe program progressed.

The mor e significant experiences a r e presented in this r eport as guidance forfuture space-flight prog rams . The experiences described indicate in many specificways deficiencies that were detected and corrected to improve the reliability and qualityof the equipment.

INTRODUCTIONA summary of the reliability and quality assuranc e (R&QA) experiences in theApollo Program is presented. The rep ort was written prim ari ly to be used by Gov-ernment and industry program manag ers who contribute to the design, fabrication,testing, and operation of saf e and rel iable equipment for spa ce applications. Reliabil-ity and quality are presented as product attributes, not as organizations.


7/37

MANAGEMENT OF R E L I A B I L I T Y A N D Q U A L I T Y A S SU R A N C EThe basic policy fo r the R&QA act ivi ties in NASA prog ra ms establ ishes the re-sponsibilities fo r NASA progra m direc to rs and managers. This policy provides forthe planning, organizing, conducting, and evaluating of the act ivi ties to ensure the re-

quired levels of reliability and quality. The R&QA activ ities a r e intended to en surethat the equipment produced fo r the Government is in accordance with the req uirementsof the contract; however, the R&QA functions performed by the Government do not inany way reli eve con trac tors of the respons ibili ty fo r the integrity of equipment o rservices.During the e ar ly developmental phase of the Apollo Pr ogram, some R&QA pe r-sonnel were organizationally a pa rt of the Apollo Spacecraft Pro gram Office (ASPO)while the res t of the R&QA personnel were decentralized. These personnel that werenot a part of ASPO and their associated R&QA effort were a part of ot her Lyndon B.Johnson Space Cente r (JSC), fo rmerly the Manned Spacecraf t Cente r (MSC), organi-zational elements such as the Flight Safety Office, technical divis ions of the Engi-

neering and Development Directorate, o r the Resident Apollo Spacecraft ProgramOffice (RASPO) that w a s established at the prim e con tractor plants.TheMSC Flight Safety Office (with specialized safety, reliability, and qualitypersonnel) provided assi stance to the ASPO and RASPO elements during program de-velopment. This office also perfo rmed the inspection function fo r all equipment fab-ricated, assembled, and tested at JSC. The R&QA Dersonnel associa ted with thetechnical divisions ensured the implementation of R&QA requi rements in the con tractsfo r which the divisions were responsible. The RASPO R&QA elements per formed re-quired functions associated with the contractors' R&QA effor t.This decentra lization of R&QA functions and responsibili ties between the se JSCelements resulted in differences regarding the establishment and interpretat ion of re -quirements , the degree of implementation, and the monitoring of contractor R&QAactivities. These differences occur red because of the varia nces of opinions betweenthe personnel in the organizational elements. Program-oriented personnel have adifferent philosophy about R&QA disciplines than the R&QA management and policypersonnel have.In 1968, all but two groups of the R&QA elements were reorganized into onecentra l R&QA off ice responsible f or all R&QA acti vit ies associa ted with all spacecrafthardware and providing appropriate support to all program offices and JSC organiza-tional elements. The two groups not included in the reorganization were the Mission

. Control Center of the Flight Operations Direc torate and the Airc raf t Operations Officeof the Flight Crew Opera tions Directorate. This centralization aided in establishingcoordinated requirements and provided fo r the uniform interpretatio n and implementa-tion of the R&QA ta sks including the monitoring ac tiv ities.To achieve the most uniform and useful implementation of R&QA policies, theR&QA should be an integrated, centralized organizational element reporting to theCente r Director. This organizational element should have defined functions stating

2


8/37

its responsibility for a ll R&QA activiti es. To be effective in contributing t o theoverall goals of a program, the organization should be both hardwa re and problemoriented; have a clos e working relationship with pro gra m e lement s such as engineer-ing, design, and procurement; provide rapid response capability in all technical skillar ea s; and have dire ct ac ces s to top management.The management of R&QA functions cons ist s of the establishment of R&QArequiremen ts and assessment of the R&QA functions that implement the requ irem entsand of the deliverab le equipment to ensur e continued effectiveness. To be effective,the functions must be planned, continually monitored, and adjusted as the programadvances.

R e l i a b i l i t y a n d Q u a l i t y A s s u r a n ce R e q u i r e m e n t s f o rSpace Systems C o n t r a c t o r sBackground. - The R&QA requirements are disciplines that, if properly imple-mented by appropriate elements of a contractor organization, will enhance the proba-bility of producing and deliver ing rel iable equipment. The reliability re qui rem ent sare defined in NASA Handbook 5300.4(1A), "Reliability Provis ions for Aeronauticaland Space System Contractor s, " April 1970. The quality require men ts were publishedin NASA Handbook 5300.4(1B), '' ma li ty Program Provis ions for Aeronautical andSpace System Contractors , '' April 1969. These requ iremen ts include the activit iesthat should be implemented for major aerospace s yste ms such as the Apollo Pro gra mand a r e the re su lt of inputs solicited fr om NASA cente rs, other Government agencies,and industry. Much experience gained from the Apollo Prog ra m and from .other ae ro-space programs is reflec ted in the publications. In addition, the NASA R&QA publica-tions include associated efforts to be performed by the engineering, manufacturing,purchasing, logistic s, and program-management elements of a contractor.Experience. - The JSC experience indicates that the R&QA requirements shouldbe se lected or "tailored" to meet the needs of the procurement and to ens ure achieve-ment of the desi red discipl ine at the lowest possible cost. This tailoring is accom-plished by using the basic requirement document as a guideline. The requ irem entsselected are included in purchase or de rs and contract statements of work only to theextent necessa ry as determined by an evaluation of the equipment crit ica lity , cha rac -ter ist ics , and end use. The evaluation is performed jointly by the technical monitorsand R&QA personnel. This tailoring concept w a s developed as the Apollo Pro gra mprogressed.Reliability requirements. - The requirements for reliability are grouped intoreliabi lity management, reliab ility engineering, and tes ting and reliabi lity evaluation.The management of reliabi lity functions is most effective when re sou rc es a r e directedto the following tasks.

~

1. Review of design specif ications2. Review of f ai lu re mode and effects analyses (FMEA)3. Review of single fa ilu re point (SFP) summ ar ies

3


9/37

4. Review of maintainability analyses and maintenance plans5. Participation in design reviews6. Participation in the investigation and ana lys is of f ailures and other proble ms7. Review of proposed corr ect ive action r esulting from participation in the in-vestigation and analysis of fai lure s and other prob lems8. Assessment of the us e of electrical, electronic, and electromechanical(EEE) parts9. Control of materials specified by designers

10. Assessmen t of test plans, procedures, and res ultsQuality assurance requirements. - The quality assurance requirements are

grouped into the following major activi ties.1.2.3.4.5.

6.7.8.9.

10.

Quality as surance management and planningDesign and development controlsIdentification and data retrievalProcurement controlsFabrication controlsInspections and testsNonconforming ar ti cl e and materi al controlMetrology control sStamp controlsHandling, storage, preservation, marking, labeling, packaging, packing,and shipping

11. Sampling plans, sta tist ica l planning, and analysi s12. Government property contro l

The JSC experience in the management of contractor R&QA functions indicates that,in most procurements, some of all the preceding requi remen ts are needed fo r a well-managed program. These requirements a r e not concentrated in a single R&QA orga-nization and a r e intended fo r use by those contractor elements responsib le fo r thedesign, development, fabrication, assembly, and testing of the equipment. By par-ticipating i n the evaluation of contractor propos als before contract award, the R&QA

4


10/37

Office can advise program managers of any excessive o r unnecessary cost ite ms andof the R&QA effort to be implemented by the contractor .

number and began reaching the hardware stages, it became apparent that the limitedJSC R&QA staff could not monitor and support each contract to the extent applied to

A f t e r a contract award, contrac t changes that affect the deliverable equipmentare reviewed by the R&QA Office to ensure that the original R&QA requirem ents ne-gotiated in the contract ar e still valid. In addition, the cont ractor R&QA ta sk s arecontinually monitored to ensure that the effort is maintained at the proper level.G o v e r n m e n t S o u r c e I n s p e c t i o n

Background. - The NASA requi re s the contractor quality organizations to reviewprocurement documents before release to ensure t h a t appropriate quality ass uran cerequi remen ts have been included. The NASA quality as surance represen tative is re-quired to review the procurement documents to determine i f Government source in-spection is necessary and to prepare a let ter to the Government agency at the supplierfacility delegating cpality ass uranc e functions to that agency.

This review at the pr ime cont ract or plant slowed down the release of the pro-curement documents to an unacceptable flow. Th is reduction in flow w a s caused bylimited NASA R&QA staffing fo r the review. The procurement document releaseprocess w a s enhanced by having the pr ime contractor R&QA personnel review and re-lease the documents using the guidelines prepare d by the JSC R&QA. The Govern-ment review of the documents w a s made after thei r release. The JSC experience maybe summarized as follows.The co nt ractor and NASA R&QA organization must have adequate staffing forreview of a la rg e number of p rocurement documents earl y in the procurement cycle

fo r la rg e contract s. In conjunction with this, the program management must allo t areasonable time for this procurement review or ri sk hardware delivery and qualitycontrol problems late r.L y n d o n B . J o h n s o n S pa ce C e n t e r P r o c u r e m e n t of Space-F l i gh t Equ ipment

Background. - The Government-furnished equipment (GFE) fo r the Apollo Pro -gr am consisted initially of the guidance and navigation system, the flightcrew sui ts,the portable life-support sys tem, and miscellaneous other items. Fo r each of theseprocurements, the JSC R&QA Office elements adequately supported the JSC technicalpersonnel. A s additional G F E and ne w experiments were authorized fo r the Apollomissions, the number and the rate of initiation of GFE pro curements inc rea sedsignificantly.

-

5


11/37

for significant problems. The hardware cont rac tors were thus required to providethe maximum available R&QA functions with only a relatively sm al l amount of Govern-ment direc t support.

Quality Assurance Functions of Government AgenciesBackground. - Quality as sura nce functions are performed by other age ncies ofthe Government under delegations issued by the appropri ate NASA o r JSC contractingofficer. These agencies, which include the Air Force, the Army, the Navy, theAtomic Energy Commission, and the Defense Contract Administration Service, arereferred to as other Government agencies.Contract administration ser vice s c osts (including quality assura nce) a r e reim-bur sable under NASA/Government-agency agreeme nts. When the NASA work is dom-inant o r extensive, NASA or JSC R&QA personnel are placed in the contractorfacilities to assist and provide technical direction regarding quality mat ter s to theGovernment agency.Ekperience in the selection of Government quality as su ra nc e functions. - Forreasons of cost effectiveness, the prac tice of tailoring the R&QA requirements to theprocurement was established at JSC . The procurement documents a r e reviewed byreliability and quality engineering personnel fo r equipment complexity, cri ticalit y,special processes, test requirements, and value. Fr om this review, the quality func-tions to be performed by the Government agency are selected or tailored from thebasic requi rement document (NASA Handbook 5300.4(2B), "Quality Assurance Prov i-sions to r Government Agencies, '' November 1971 edition) and forwarded to the contract-ing officer for transmittal to the delegated agency.Typical functions performed by the Government agencies f or JSC a r e inspectionof equipment and incoming material, witness tests, fai lure notification to JSC, reviewof nonconforming equipment, and surveys of suppl iers quality assura nce operations.A t the beginning of the Apollo Prog ram, the delegation of quality assu rance functionsto the Government agencies caused some concern because of the di fference betweenthe JSC requirements and the procedur es normally implemented on Department ofDefense (DOD) and other Government cont racts. The JSC quality assurance r ep re -sentat ives were designated to assist the Government agencies to ensure that the re-quirements were understood and implemented properly. The JSC experience indicatesthat the services of Government agencies can be used most effectively i f timely qualitydirect ion and assis tanc e a r e given and points of contact are provided. This direction,which is provided by the JSC quality assurance rep resenta tive , includes interpreting

R&QA requirements , evaluating the adequacy of fabrica tion and inspection pro cesses ,providing guidance in the preparation of procedures, assi st ing in the disposition ofnonconforming equipment, and resolving any prob lem or question the Governmentagency may have in performing the delegated functions.

Docu mentation R eq u rement sGeneral. - The planning and asse ss me nt of R&QA functions requ ires the r ece ivaland review of several types of documents. Initially, documentation as required in the

6


12/37

NASA R&QA publications w a s requested from the cont rac tors without any additionaldocumentation cri te ri a such as format, scope, and content. It soon became evidentthat the documentation requ irements must be adjusted and tailo red for each procure-ment in a simila r manner as the R&QA requirements, and t h a t the scope, format, andcontent of documentation must be specif ied in the procurement documents. Examplesof documents used by the JSC R&QA organization include re liabil ity and quality plans ,EEE par ts specifications, pa rt s list, certification and acceptan ce test plans, failurereport s, and failur e analyses. Except for the R&QA plans, the other documents li st edar e required by JSC engineering and program office elements.

Background - R&QA plans. - The Apollo spacecr aft contrac tors w ere requiredto pr epare R&QA plans and submit them to JSC for approval. This requirement w a snew to t h e aeronautics industry. Quality control sys tem requi rements imposed onDOD contra ctors requi re them to document the system including proce dures subjectto disapproval by the Government. Submittal of thi s document is not requir ed but itmust be available to the Government.Experience - R&QA plans. - In the judgment of the JSC R&QA personnel whoass is ted the contractor i n the development of the R&QA plans, the plans constitute agood medium for contractor and Government management to understand the objectives,procedures , and milesto nes of th e R&QA functions. For maximum benefits, thi s sa mediscipline should be used by the pri me contractor in the review and approval of sub-contractor plans.Experience in the area of documentation has shown t h a t the scope, format, andcontent of R&QA plans must be specified in the contract to preclude arb it ra ry changesin acceptance cr it er ia . To be fully effective, these plans must be established in thecontract as documents requi ring customer approval. The JSC R&QA personnel havefound t ha t R&QA plans a r e mo st beneficial on complex o r costly procurements. Tech-nical and financial ri sk s a r e the greate st for equipment the design of which is new o runproved, especially fo r costly and very complex hardware. For these types of pro-curements, the submittal and implementation of R&QA plans help provide confidenceand assura nce that technical problems and costs associated with these r is ks a r e mini-mized. It w a s found that normally no benefits were obtained by requesting the develop-ment and submittal of plans fo r short- term, less costly, off-the-shelf procurements.R&QA sta tus rep ort s. - In the ea rly stages of the Apollo space craf t development,major contractors were required to submit periodic re por ts on the prog re ss of thereliability and quality ass ura nce activities. Scope, forma t, and content of these re-ports were determined by the contractors. These status re po rt s proved to be ineffec-

tive because the program w a s moving ahead rapidly and the data in the re por ts werenot cu rr en t and not in sufficient detail to permit effective asses smen t of the R&QAactivi ties. The Apollo pr ime con tracto rs were directed to discontinue the statu s re-po rt s and to prep are monthly briefings fo r JSC R&QA management personnel . Thesebriefings were given by the cont rac tor R&QA managers to the JSC R&QA manager anddivision chiefs. The meetings were conducted in accordance with an established agendaand were documented.vided for the participants. Topics normally discussed were budget control (includingmanpower and cos ts) , suppl ier hardw are problems, quality trend data showing defects,rework, mater ial review actions, fai lur e information, and corrective actions taken toresolve problems.

Action it ems were established, and handout mat erial was pro-

7


13/37

These management review meetings are effective, especially in the ear ly phasesof a pro r a m .Centerkc),SC, the White Sands Test Facility, and the quality as sur ance officesresident at prime contractor plants participated in the briefings, providing valuableviewpoints from sev eral perspectives . Many brief ing topic problems, which wouldhave taken considerable time to resolve if correspondence or telephone conversationmedia had been used, were di scussed at t h e briefings. Action w a s assigned and, inmost cases, the problem w a s resolved before the next regul ar meeting. These meet-ings proved to be very effective fo r bringing to focus quickly many of the di fficultproblems that could only be resolved with the proper representat ion of both contractorand NASA personnel.

The NASA personnel from Washington, the John F. Kennedy Space

A s the Apollo Pro gram prog ress ed and assu med some measure of routine, manyof the meetings were conducted by te leconference . It would have been desirable tohave simila r briefings with the GFE contractors , but it w a s considered im practicalbecause of the la rg e number of GFE procurements. In these case s, ass ess ment of theR&QA act ivi ties continued to depend on stat us r ep or ts supplemented by cont rac torplant visits and frequent teleconferences.Re1 a b i l i y a n d Q u a l i t y I n f o r m a t i o n

Background. - Procedures for t h e sto rage and ret ri eval of R&QA informationwere developed for the Apollo Pro gra m. Requirements for automated storag e and re -tri eva l of failure information were established with Apollo con tracto rs. These fi le swere supplemented by a library of inspection record s, te st reports , and other docu-ments required by R&QA, engineering, and prog ra m management personnel.Experience in failu re information. - Apollo spacecraft contrac tors were r equir edto provide for the. storage and retrieval of f ai lu re informat ion on magnetic tape at theirfacilities and to transmit a duplicate tape to JSC. This procedure did not adequatelysupport the spacecraft milestone reviews because the elapsed time f rom date of fail-ure to firstprintout at JSC frequently w a s sev era l weeks, which w a s too late to permitthe review and approval of the proposed corr ecti ve action by the responsible JSC pe r-sonnel. Therefore, a method fo r obtaining mo re c ur rent information w a s needed. TheJSC procedure was changed, and the magnetic tape file w a s retained as a historicaldata fil e on failu res. Requirements were revise d fo r the submittal of fa ilure informa-tion to JSC. The contractors were required to notify JSC within 24 hours after r e-ceipt of failure notification by rel iab ili ty i f fai lur es occurred during certification,acceptance vibration and acceptance thermal tests, catastrophic fail ure s during de-velopment and acceptance testing, o r fai lu res that had a dir ect impact on vehicles in

prelaunch checkout at KSC. Similar notification had to be provided fo r significantunsatisfactory conditions (fire, explosion) o r ground support equipment (GSE) safetyfailures. A failure-information focal point (clearinghouse) w a s established by R&QAto receive and distr ibute fai lu re report s at JSC (Houston) to the progra m office and toengineering. A s a pa rt of th is clear inghouse activity, R&QA kept a current list of allunresolved (open) fa ilures and related problems . Special emphasis was placed onea rly resolution of the prob lems affecting the vehicles at the launch site, using spe-cially prepared li st s fo r frequent reviews with pro gram offic e management.

8


14/37

Assessment of R e l i a b i l i t y a nd Q u a l i t y As s u r a n c e F u n c t i o n sBackground. - The management of R&QA functions consis ted of two principalactivities. These activities were the establ ishment of R&QA requirements and, th er e-after, as sessme nt of the functions and of the del iverable equipment to en sure that the

act ivi ties we re effective. The JSC ass ess ment of R&QA functions w a s not a one-time,one-type action; rather, it w a s a continual se ri es of planned and integrated actions.These actions include the following.1. Audits of contractor and subcontractor R&QA functions and as se ssme nt ofcorrec tive action2. Management review meetings between JSC and contractor R&QA managersand supervis ors and asse ssme nt of contractor resp onses to action ite ms3. Weekly rep or ts of significant events fr om the residen t JSC R&QA personnelat prime contractor facilities4. Monthly r ep or ts fr om the JSC representative at subcontractor facilitie s5. Analyses of test and inspection rec ords6. Reports generated by JSC R&QA i n the performance of R&QA tasksBackground - R&QA audits. - Audits of con tracto r and supplier R&QA act ivi tie sto d etermine compliance to contractual R&QA req uireme nts were pe rfo rme d by JSCas required by NASA policy and program di rectives.Experience. - Many contr actors and sup pliers providing space-flight equipment

to NASA cen te rs and to DOD agencies were audited seve ral tim es each year. Theseindependent audits by the centers and agencies creat ed a problem with the contr act orsbecause many R&QA manpower r esou rc es had to be used to par tic ipa te in th e audit.This, in turn , affec ted the functions of the R&QA organization because key personnelsupporting the audits were unable to concentrate on thei r day-to-day operations. Toensure that the se audits would not be performed independently at least within NASA,the R&QA Office establ ished a centra l focal point to coordinate audits with NASAHeadquarters and its centers. This coordination included the establ ishment of sched-ul es of the aud its to be performed for the next 6 months and invited the other ce nt er sto par tic ipa te and become memb ers of the audit team. Before the audit, a determina-tion w a s made about which R&QA act ivi ties were to be audited, the type of checkl ist sto be used, and how the audit would be conducted. Audits that a r e planned, scheduled,and coordinated have been beneficial to NASA and t he c en ter s in t e rms of cost effec-tiven ess because of le s s travel expenditures and in te rm s of better contractor andsupplier R&QA management because the re was less disturbance in their operations.

Assessments of S p a c k F l i g h t E q u ip m e n tBackground. - Two major a sse ssm ents o r reviews of the individual Apollo sp ace-cr af t were made by the prog ram office before each launch. The first assessment, a

9


15/37

Customer Acceptance Readiness Review (CARR), was performed to assess the readi-ness for the initiation of individual subsys tem, integrated testing, and Governmentacceptance of the spacecraft before shipment to KSC. The second asse ssme nt, aFlight Readiness Review (FRR), w a s to evaluate the s pace cra ft and related GSE beforelaunch and to accomplish the mission. The R&QA personnel ass is ted in these as se ss -me nts by being members of the data review teams and also as membe rs of the formalCARR and FRR Boards. The purpose, composition of the review teams and fo rmalboards , and procedures a r e defined in the progr am office CARR Plan and FRR Plan.The R&QA representatives documented the re sul ts of these reviews in the for m ofasses sment s tatements signed by the Manager of t he R&QA Office. These statementswere presented to the CARR Board stating the re were no cons traint s to shipment of thevehicle o r to the FRR Board indicating that the spacecraft was ready for launch. Inthe event the re was a constraint (such as open items) , the ass essm ent statement wouldindicate that at sati sfactory completion of open ite ms th er e would be no constrain t.

Experience. - Initially, forma l reviews of spacecraft reco rds were not perfor medbefore shipment of the spacecraft to the launch site. The reviews of vehicle rec ordsmade by JSC were performed on a noninterference basis.re co rd s and documentation wer e reviewed immediately before the FRR. The reviewof these reco rds w a s concentrated mainly on nonconformance records and evaluationof the action taken to cor re ct discrepanc ies; certif ication test reports; parts replace-ment records ; and other inspection and te st data, including nonmetallic ma ter ia ls(NMM) usage, EEE pa rt s application, pr ess ure vessel his tories , and contaminationand cleanliness records. As the program progressed , the amount of documentationincreased significantly, and it w a s not possible to do this data and document review inthe time allotted before the FRR. With the establishment of a n additional programmilestone review such as the CARR, the assessm ent procedures were re vised to in-clude incremental rev iews by R&QA at the milestone reviews, supplemented by con-tinual R&QA assessme nt of f ai lu re s and problems. This procedure w a s much moreeffective in ensuring earl y detection and correction of problems that oc cur red af terthe incremental reviews, thereby permitting program management to be awar e of thelatest readiness st atus of the space-flight equipment and allowing adequate time fo ractions and corrective mea sures in the event of problems.

Fo r the ear ly missions, the

TrainingBackground. - Early in the Apollo Program, the fabrication and operations r e-lated to special pro cesses, particularly the soldering operations, were of concern toNASA Headquarters. Because of the unknown environments that the cr it ical space -flight equipment may be subjected to, NASA considered it necessary to ensure that

uniform solder ing operations and techniques be used by industry fabricating th is equip-ment. The NASA published a soldering requirements document (ref. 1)defining theseoperations and techniques. Because NASA as well as contractor personnel requiredtrain ing for these operations and techniques, soldering schools were established byNASA fo r the training and certification of contractor and supplier operato rs, inspec-tors, and ins tructors. In addition, NASA personnel were tra ined and certif ied, therebyenabling them t o evaluate the contractor operat ions and techniques effectively. A s therequired skills were developed and the personnel were trained, the schools were dis-continued and the co ntracto rs were responsible for the train ing and certification of thesoldering personnel.1 0


16/37

Experience. - The NASA publication r equirement s included detailed "how to"techniques for solder operations, and the training of pe rsonnel i n these techniqueswas extensive. A s the Apollo Program progressed, many contractors requestedwaiver s and deviations to the NASA document. These we re normally granted when thecont rac tor soldering operations and techniques proved to be equally reliable . Thesereques ted deviations revealed that the NASA how-to techniques we re not flexible enoughto permit the use of equally good techniques developed by con tractor s, thus causing anunnecessary in cre ase in cost s to ret rai n personnel. Review of the NASA publicationby the NASA centers considered the recommendation that industry be permitted t o us ethe ir pr oc es s specifications and techniques. A revised NASA soldering document de-leting the rigid how-to techniques w a s published, and the qual ity and workmanship ofequipment we re maintained with reduced training cos ts not only fo r the cont ractors butalso f or NASA and delegated Government agencies.

Contractors are required to document the solder ing prog ram including informa-tion regarding qualification of ins truc tors , procedu res fo r training, les son plans, in-struction hours, and procedur es for t h e certification and recerti fication of sold erpersonne l. Early development and implementation of the tra ining pro gra m and pro-cedures are requ isit es to e nsu re the availability and maintainability of reliableequipment.

D e s i g n a n d Procedural StandardsSignificant space cra ft design and operational problems identified during Pro je ctMercury and the Gemini P ro gr am wer e investigated to develop a clear and completeunderstanding of each problem and to determine i f preventive requir ements weretechnically feasible for spacec raft in future programs. Requirements consideredfeasib le were developed into design standards, procedu ral st andards , and engineeringcr ite ria bulletins. A Criteria and Standards Board w a s established with repres enta-tives fro m key JSC management and technical disciplines to review and recommendapproval of the bulletins. After concurrence by the board, the bulletins were sub-mitted to the JSC Director fo r approval signature. After publication of the bulletins,the appropriate JSC elements and prog ram office technical mon itors were responsiblefor including the bulletin requirements in hardware procuremen t contracts.A s the Apollo Pr og ra m progressed, other significant prob lems we re identifiedand investigated, and design and procedu ral standards ( former ly bulletins) were pub-lished. When issued, standards wer e reviewed by the technical monitors to determinethei r applicability to active contracts. In many case s, the implementation cost s pro-hibited re troac tive application. When it w a s considered nec ess ary to make retroa ctiveapplication, the s tanda rds we re imposed by Configuration Change Board action. Pro-ce du re s we re developed whereby R&QA personnel in thei r review of new purchaserequests verified i f standar ds were included as requirements . If stan dard s were notcited, the technical monitor w a s contacted to verify that the standar ds were not ap-plicable to the procurement.Experience at JS C indicated that, for new programs and contracts, it w a s neces-sa ry to have the cu rr en t standards imposed in the statement of work, requiring thecontrac tor to review and determine the degree of compliance with each standard. Ap-proval must be obtained fro m JSC for those standards not complied with by thecontractor.

11


17/37

ENG l NEERl NG FUNCTIONSThe reliability of space-flight equipment is a function of the emphasis appliedduring equipment development by equipment des igne rs, stress analysts, and otherengineering personnel. The most effective contributions to product integrity ar e madeby the engineer who appropriately consid ers the environments involved, se lect s thepart s and materials, establishes the tolerances, and pre par es the documents fro mwhich the product is purchased, fabricated, assembled, tested, and handled. Many ofthe NASA provisions for reliability and quality assurance, which are cited i n the pub-li shed NASA handbooks, can be implemented only by the equipment designer s andengineers.Unreliable parts, us e of incompatible materi als , and insufficient attention toenvironments were a few examples of problems encountered during the program thatclearly indicated a need for emphasizing the role of the des igner in product integrity.

Re1 abil i y Pred ic t ionsBackground. - Overall reliability numbers were generated as goals for the safetyand success of the Apollo missions and wer e used as the basis for apportioning sub-sys tem reliability goals. Initially, con tracto rs we re required to continue to developreliability predictions and asse ssm ent s as the Apollo design matured.Experience. - Rigorous numerical reliability predictions and asse ssm ents canbe calculated only by using failure -ra te information developed from ac tual hardware.Because of (1)t h e relatively limited amount of hardwa re developed fo r the Apollo Pro-gram, (2) th e deletion of reliability testing, and (3) the relatively limited operatingtim e accumulated on the hardware i n actual environments, the requirement f or the

generation of rigorous reliability predic tions and as sess me nts was deleted. There-fore, the use of reliability numer ical techniques has been relega ted to us e only ininit ial trade-off studies during the conceptual design pha ses of ha rdware development.

Fa i lur e Mode and Ef fec ts Ana lys isBackground. - A s an integral par t of the ea rly sp ace cra ft design phase, the flighthardware design should be analyzed to determine poss ible modes of fai lure and the ef-fect s of failu res on mission objectives and crew safety. Ideally, ana lyse s should beconducted at the system, subsystem, and component levels including individualswitches, relays, and so forth. The pri ma ry objective of these analyses is to identify

the failures that could affect mission su cce ss o r cr ew safety, so that redundant sys-te ms can be added to the design where possible.Experience. - The spac ecraf t FMEA was an effective tool f o r design evaluation.

Each potential failure was categorized for worst-case effect on mission succe ss. Fo reach case i n which a single failure could cause a risk to the mission succe ss or crewsafety, the hardware in question was liste d se parate ly in a n SFP list. Each portion ofthe design that had a n SF P then got specia l attention, and the S FP w a s eliminatedwhe re possible. In many cas es , the complete elimination of the bad effec ts of an SF Pmalfunction was not desi rable because such action would make the no rma l function12


18/37

mor e difficult to obtain. Fo r example, the use of single switches to initiate parall elsequences of events with redundant crossove rs that require precis e timing and the useof s er ies-para lle l contacts fo r safety could have jeopardized normal sequencing. Insuch cases , procedures were devised to minimize vulnerability to inadvertent switchclosure, such as keeping the s ystem disarmed and using circuit b rea ker s at all t imesexcept when arming w a s required.

In other cas es, any backup for an esse ntial function was simply impractical,such as the cas e of t he lunar module (LM) ascent st age engine used to leave the luna rsurface. In such cases , extensive testing programs fo r qualification and flight engineswere used, together with rigid configuration control and quality as sur ance surveillanceof each flight article to minimize the risk.Although the FMEA and SFP information provided by con tracto rs w a s usually i ndifferent formats and the data were presented in vari ous manners, an analysis wasaccomplished. To overcome this problem and to ensure a uniformity of data on thespacecraft FMEA and SFP sum marie s, a recommended format wa s established fordocumenting the information (figs. 1 and 2).

FAILURE-MODE AND EFFECTS ANALYSIS

PREPARED BY PAGE OAPPROVED DATE

SYSTEMSUBSYSTEMASSEMBLY

ITEM I'ENTIFI-CATIONIUMBER

- -STEMLOGICIAGRAMUMBER :AILUR EODE ANDCAUSE FAILURE EFFECT ON ZORRECTIVEICTION TIMEAVAl LA BL VT IMEREQUIRED FAILURE-R lT lCAL lTYODECATEGORYITlFlCATlONDRAWINGZEFERENCEESIGNATIONIC)

1ISSIONPHASENCTION \SSEMBLY iU BSY STEM 8YSTEM

le )

-

(f) (k)l)

Figure 1.- Forma t fo r failure mode and effects analysis.13


19/37

SINGLE-FAILURE-POINT SUMMARYSYS TEM CR IT1 CALI TY CATEGORY REFE REN CE FME ASUBSYSTEM PREPARED BYASSEMBLY APPROVED DATE

SUPERSEDINGITEMIDENTIFICATION

( a )

FAILURE MODEAND CAUSE(b)

FAILURE EFFECT(C )

FAILURE-DETECTION METHOD RATIONALE FORACCEPTABILITY(e )

R EVI S I ON(f)

Figure 2. - Format for single failure point summary.The FMEA and the SFP summarie s were not required fo r all flight hardware.In some case s in which the hardware was not essential to the completion of the mis sion(for example, a simple experiment) and a malfunction could not be hazardous, theFMEA and the SFP summ arie s were omitted.The JSC experience with fa ilur e mode and effec ts analyses and SFP analysesindicates that these analyses a r e most effective when used in conjunction with designreviews. The Apollo spacecraft engineers performed comprehensive fail ure mode andeffects analyses and SFP summari es and, through a se ri es of iter ative design reviews,eliminated or minimized the number of potential fa ilure points (ref. 2).

Electr ical, Electronics, and Electromechanical Pa rt sBackground. - The original NASA provisions for control of EEE pa rt s includedthe use of par ts specialists to act as adv ise rs to the design groups and to conduct thepa rt s activities. The specific pa rt s provisions included selection of pa rt s, preparat ionof specifications, qualification te st s, preparat ion of approved pa rt s li st s, and review ofpa rt s applications.

14


20/37

Experience. - These provisions were only parti ally effective. Design reviewsconducted at the part s level disclosed sev era l problems that requir ed changes in theprovisions to ensure the rel iabil ity of the Apollo equipment. A par ts reliability require-ment document w a s developed by JSC that required that a pa rt s program plan be devel-oped by con tra cto rs. This plan included requi rements fo r development of specifications,one-way traceabilit y, derating of pa rt s, screening, and burn-in procedures andtechniques.

Review of the st r e ss analyses for electronic assemb lies revealed se veral cas esin which the pa rt s wer e s tr ess ed beyond the values established in the part s specifica-tions. Several fa ilures of Apollo equipment were attributed to pa rt s over str ess . Onthe bas is of th is experien ce, the current goal is to have any new pa rt s design str es se s(voltage and s o for th) no higher than 75 percent of the rate d capability. This under-stressing is usually r efe rre d to as derating.The pra cti ces of screening par ts for defects before use and burn-in (to eliminateinfant mortality p ar ts ) were not implemented f o r some of the ear ly Apollo equipment,

and seve ra l fa ilures of thi s equipment were traced to defective pa rt s. In many cases,analysis disclosed that the fai lures could have been avert ed i f screen ing and burn-intechniques had been used. A s a result of these lessons, a dedicated effort w a s madeto secu re scre ened and burned-in p ar ts for each critical flight application.Background - parts information. - An automated file of informat ion on EEE par tsw a s established at JSC. Parts usage information was obtained from contractors andsuppli ers and processed into the file, and repor ts were made and distributed to con-tra ctor s. This allowed the contractors to compare pa rt s test plans with the repo rt in-formation to determine i f the same or similar p a r t s had already been tested by anothercontractor. More than $2 million in planned testing was avoided in th is manner.Experience. - Initially, it w a s thought that the p ar ts usage information would beextremely valuable to the des igners of the next generation of space syst ems. However,i t w a s soon rea lized that the rapid technological advances in the EEE pa rt s field makesmuch of the info rmation obsolete within a short time un less updated part s usage and ap-plication information is continuously inserted into the system. Therefore , the useful-ne ss of the file may be limited primarily to curren t pro gra ms. In addition, the file isuseful for ass essi ng the reliability of space-flight equipment at milestone reviews. Fo rexample, when equipment fa ilures occur and the fa ilur e cause is isolated to an EEE pa rt ,the part s m ast er fi le can be used to determine where the part type is used other thanthe failed equipment. The file is effective to a limited degree, but, like all automatedfiles, it may not be curr en t. In those ca se s when curr en t data are missing, it becomes

necessa ry to have the contractor searc h their record s (such as mater ial and par ts lists)fo r the usage and application of suspec t pa rt s. However, the JSC file does pe rmit thein it ia l rapid location and investigation of the equipment.N onm e t a lI c M a t e r i a ls

Background. - From a materials standpoint, a significant exper ience of the ApolloProgram is the use of nonmetallic ma te rial s i n the oxygen-rich spacecraft cabin atmos-phere. The rel iabili ty ta sk s contained in the NASA handbook "Reliability Pr og ra m15


21/37

Provisions fo? Aeronautical and Space Contractors" included general requ irements re-garding mater ials but did not emphasize the importance of nonmetallic materials.Experience. - At the s ta rt of the spacecraft design effort, the amount of data onthe flammability of NM M in pure oxygen at 5 psi a and the toxicity of NMM was limited.Extensive testing of a variety of nonmetallic mater ials wa s required to find the leastflammable and toxic mate ria l fo r each application. As the pro gram progre ssed , manychanges in materi als we re made as data were developed and requirements were includedin cont rac ts. The control of the NMM used in the Apollo spacecraf t contributed large lyto the enhancement of crew safety in the spacecraft. However, other importan t fa ct or sthat must be considered for crew safety include control of ignition sou rce s, control ofthe environment, fire-detection capability, and fire-extinguishment provisions. A s y s -te ms engineering approach to firesa fety ( re f. 3) must be established e arly i n the devel-opment of space-flight programs.Offgassing, t h e te rm used to describe the evolution of gaseous products from liq-uid o r solid material, represe nt s one of the haza rds associated with the use of NMM.

In som e cases, the offgassed products are toxic. In other cases , the offgassed productscan coat sensitive equipment and degrade its performance. Although most organic ma-terials offgas to some extent at normal atmospheric conditions, many mate rial s do notoffgas significantly unless exposed to elevated temperatures, reduced pre ssur es, or acombination of elevated temperature and reduced press ure. In the Apollo spacecraft ,offgassing char acte ris tics of materials must be evaluated during the ma ter ial selectionprocess. Material selection cri teri a ar e presented in reference 4 .Background - materi als information. - Materials tests were conducted at contrac-tor and JSC facili ties. However, JSC soon realized that performing the te st s at sev-eral different locations could re sul t in duplicate testing and unnecessary expense. Dataon the tests performed were accumulated and reproduced a t JSC and distributed to con-

tractors fo r review and analysis with the objective that c ontract ors would not have totest materi als already tested by other contr actors .Experience. - The automated file of NMM information was designed to providetest data to contr actors , Government organizations, and other organizations that wantedthe dat a and also to yield NMM usage information for each Apollo spacecraf t ( materialtype, location in the spacecraf t, weight, and so forth). Data from standardized te st son flame propagation, ignition temperature, and toxicity of offgassed products wereaccumulated and published periodically in JSC document MSC-0268 1, "Nonmetallic M a-terials Design Guidelines and Test Data Handbook. '* The cur rent issue of t his documentconta ins information on approximately 15 OUO standard t es ts (conducted at six test facil-it ie s) that have been made on approximately 3700 different nonmetallic materi als that

are used or were considered fo r application on the Apollo spacecraft. Result s fr omapproximately 2600 special test s are also included in the document. The document pro-vided Government and industry personnel with test data and made possible the develop-ment of a cooperative program of data exchange between Government agencies andindustry fo r the avoidance of duplicate testing.


22/37

FIuid SystemsBackground. - On a typical Apollo mission, 71 tanks are installed in the commandand service module (CSM) and L M . These tanks contain it ems such as water, gaseousand liquid oxygen, liquid hydrogen, gaseous helium, gaseous nitrogen, nitrogen tetrox-

ide , and Aerozine-50. The tanks have relatively thin w a l l s and are highly stressed.Fo r this reason, smal l surface imperfections can be significant stress ri se r s and po-tential failure points.Experience. - Some of the more significant prob lems encountered in the fabrica-tion, installation, and testing of fluid syst ems ar e discussed in the following paragrap hs.Pressu re vessels. - The servi ce pro-pulsion system (SPS) tanks ins talled in theApollo service module are shown in f ig-u re 3. The two helium tanks are spherical,are more than 40 inches in diame ter, anda r e fabricated of titanium alloy. The oxi-dizer storag e tank is 45 inches in diam eter,is more than 153 inches long, and has aw a l l thickness of 0.047 inch. The la rgesi ze and thin w a l l s make these tanks awk-ward to handle and thus extremely suscep-tible to inadvertent damage.Special handling procedures were de-veloped and used during the manufacturing,installation, and testing phases of these

tanks, making all personnel aware of thecri tica l operations associated with the han-dling of the pr ess ure ves sels . Experiencerelated to pre ssur e vessels used in theApollo Program is discussed in d etail inrefer ence 5. Eighteen tank failur es werereported during operations before flight.Some fai lur es attributed to stress corro-sion, weld cr ack s, and mat eri als defectsare described as follows.

Oxidizer sump

tank sector\ Fuel storagef ank sector 6

nitrogentanks

Figure 3. - Service propulsion syst emtanks installed in the Apollo s ervicemodule.Str ess corrosion: The failures caused by stress corrosion were caused princi-

pally by the use of fluids that contained substances that were incompatible with the tankmaterials.and testing in this ca se disclosed that a parti cular methanol and the titanium used areincompatible mate ria ls (ref. 6). In another case, impur e water w a s used as a testfluid and failur e resulted fro m incompatible materials. Experience with these problemsshowed that controls must be implemented to a ssu re that fluids used i n pressur e vesselsare compatible with the tank material.

In one case , methanol w a s used as a te st fluid fo r the SPS tanks. Analysis

Weld crack s: An SPS propellant tank failed during an acceptance proof te st ; thefai lure was a ttributed to contamination in the weld joint between the cylindrical portionof the tank and the domed ends. Although the source of the contamination w a s not17


23/37

definitely established, i t most likely occurred during the welding operation. Specialcontrols were instituted to preclude r ecurr enc e of thi s problem including vapor blastcleaning before welding, control of the working and wetting environments, and the useof clean gloves during the welding pro cess . These procedures apparently have beeneffective because no fur ther problems have occurr ed with this type of weldment.Material defects: An LM descent propulsion system pre ssure ves se l failed duringa hydrostatic acceptance proof test. Investigation revealed that the f ailu re was causedby localized micro str ucture abnormality, consisting of "massive" alpha-phase st ruc-tu re in the upper dome. Alpha inclusions of th is so rt a r e rare and cannot be detec tedin the raw materia l with existing nondestructive testing techniques. Proof testing isthe only practical approach for screening finished pr es su re ve sse ls.The major nondestructive tes t methods implemented on the Apollo pr es su re ves-sels are ultrasonic inspection, X-ray photography, dye-penetrant inspection, and mag-netic particle inspection (steel tanks only).limitations. For example, X-ray photography can reveal surface and subsurface flaws

but is sensitive to flaw orientation. Penetrant inspection is limited to the detection ofsurf ace flaws. These techniques must be used despite the limitations, and experienceindicates that acceptance proof testing is the only way to ensure the integ rity of thecompleted tanks.

Nondestructive test methods have

Background - pressure vessel historical records. - Every pres sure v ess el in theApollo spacecraft was pre ssur ized and vented many times before flight. The vesse lswere pressurized during acceptance tests by the suppliers and seve ral times afte r in-stallation in the spacecraft. These pressurization cycles vary in pre ssure levels at-tained, elapsed time at pres sur e, fluids used, and tempera ture while under pre ssure .In addition, the tarks were frequently exposed to fluids for cleaning and purging.Experience. - Investigation of tank fai lures early in the program disclosed thatdetailed historical records had not been maintained for the individual tanks. Withoutsuch recor ds, the flightworthiness of each flight tank could not be determined becauseof lack of knowledge about pr es su re levels and cycles attained; duration at pre ss ur eand tempera ture; fluids used fo r fabrication, cleaning, and test ing; and any noncon-formances recorded against the tank. Historical recor d ca rds we re developed by JSCand requirements were imposed on the tank manufacturers and spacecraft contractorsto maintain the car ds cu rren t and to record the information and data jus t mentioned.Fluids. - The quality of the fluids used in the Apollo spacecraft syst ems w a s vitalto the reliability of the syst ems; many fa ilures of equipment were at tributed to impuri-tie s and contaminants in the fluids. In some cas es , fluid syst em qualification te st swere conducted with fluids that were of a higher purity (less contaminants) than thefluids used in acceptance te st s o r sys tem operation. Many of the problems associa tedwith fluids occurred from the lack of adequate standards or specifications for fluidcleanliness and fluid sampling. Guideline documents have been prepared to ensu reproper control of fluids used in the spacecraft. Whenever changes are made to manu-facturing processes, handling procedur es, o r procurement sour ces for any fluids(liquids or gases) , consideration should be given to requalifying a system that ha s beenpreviously qualified using a particula r fluid.

18


24/37

Contamination. - Contamination was one of the major problem categorie s on theApollo spacec raf t. The equipment contamination problems were attributable dire ctlyto inadequate knowledge in the disciplines required to maintain the level of cleanl inessestablished during the equipment design phase.Basically, contamination in fluid systems can be categorized as follows.1. Fluids used fo r testing or operating the sys tem; specifications o r samplingprocedures inadequate2 . Fluid transmission sys tems (GSE and launch site facilities) not maintained tosam e level of cleanl iness as the spacecraft3 . Fluid system components not properly cleaned or sealed before installation4 . Contaminants introduced into the system during s ystem installation (brazingand solde ring fluid lines, environment not controlled while fluid lines and components

were being installed, and s o forth)5. Fluid syste m components, such as quick disconnects, generating par tic lesupon assembly or disassemblyTo prevent the occurr ence of these problems, it was essential that control s in-cluding proper fil tering be established at the sta rt of fluid sys tem design. The levelsof cleanl iness specif ied fo r the sy stem by the designer must be maintained in the fluids,the s ystem components, and the ground support equipment, at all locations. Controlledenvironments must be maintained during component assembly , installation, and removal.Plans and proced ures w ere developed to ensure that the required disciplines were main-tained by all participants, including suppliers.

Problem ControlBackground. - A system w a s developed by JSC for managing the dispositionof p rob lems (failure s and anomalous performance) occur ring on hardware dur ing theApollo Pr og ra m. The sys tem included all program participants (contr actors , subcon-tr ac to rs , NASA test si te s, and KSC) and provided fo r the recording and reporting offailu res and other deficiencies, asse ssment of problems fo r cause and responsibility,corre ctive action to preclude recurr ence, an d screen ing operations that allowed prob-lems to be solved at the lowest management level. Only the most significant problems

(high ri sk and unresolved) were presen ted to program management for review. Thesyst em was designed to support the spacecraft prelaunch reviews by providing summa-r ie s of subsyst em problems occu rring on the spacecraft in review as well as otherproblems that may affect the vehicle in review. The problem control system also wasintegrated with the NASA-wide ALERT system fo r dissemination of par ts and ma te ri al sproblems.Experience - recording and reporting of problems. - A reasonable degree ofawareness of problems and the impact of these pro blems on the program was difficultto maintain because of the large number of different problem reporting f orms used bymany cont rac tors. Another difficulty developed concerning the report of fa ilur es to

19


25/37

JSC. The spacecraft contractor s expressed concern about the definition of the te rm"failure" and the feasibility of repor ting fai lures before acceptance test. To assurethat all problems were properly documented and ass ess ed , a fai lure was defined b yJSC as "the inability of any pa rt , component, o r subsystem to per form i ts intendedfunction under specified conditions and for a specified period of time" rega rd les s ofoutside influence such as procedural or human er ro r. All such problems were report-able to JSC after the beginning of hardware acceptance testing and required JSC ap-prova l of the fixes by subsystem manag ers and R&QA personnel before the problemscould be closed out.

Problem assessment. - Some problems had the ir origin in the design and had tobe solved by the designers. Other problems, which originated in the manufacturing ac-tivit ies, had to be co rrected by the department responsible fo r the problem. Manyproblems of manufacturing origin were re por ted to JSC, where i t was recognized thatthe investigation and closeout of th is type of problem should be performed by contrac torpersonnel at the cont racto r facility, with JSC personnel participating and concurringin the actions.This experience led to the establishment of problem as ses sment groups at theCSM and LM contractor facilities and at JSC. The pr im e function of the problem as-sess men t groups was to review and scre en the problem re po rt s for manufacturing ordesign origin. All problems of design origin and significant manufacturing problemswere transmitted to the JSC problem as sess men t group, where they were redistribu tedto the JSC subsyst ems managers, the reliability engineers, and the pro gram office.Experience indicates that the problem ass ess me nt function was effective in promptresolu tion and closeout of p rob lems.Corrective action and closeout. - The ideal procedure was to review each problemand implement correc tive action immediately. However, experience shows that thisideal condition could not be achieved. Discipline was requ ired to a ss ur e that the mostcri tica l problems were addressed first and that the remaining problems w ere attackedon a priority basis. The contractor problem asse ssment groups were charged with theresponsibility of assigning and approving correct ive action. Final approval for close-out of problems of design origin was provided by the JSC subsystem manager and re-liability engineer. Closeout of problems of manufacturing origin was provided by thelocal quality pe rsonnel at the contractor plant o r by JSC (Houston) quality assuranceoffice, together with the appropriate subsystem manager.The NASA ALERT system. - Pa rt s and materi als application, failur e, malfunc-tion, o r problems of common int ere st were reported in the ALERT system. ThisNASA-wide system, established in 1964, provides for distribution of problem notices

to NASA and cont ractors through coordinators a t the various ce nt er s and to DOD throughthe Interservice Data Exchange Program (presen tly called Government-Industry DataExchange Pro gram (GIDEP)). The Apollo contr actors were requi red to respond to JSCon each ALERT, stating whether o r not the con tracto r hardware was affected. If thehardware was affected, a resolution to the problem was worked out.

20


26/37

MANUFACTU RING FUNCTIONSGenera l

Ana lys is of the Apollo spacec ra f t f a i lu re r eco rds d i s c losed that a l a r g e p e r c e nt a g eof t h e f a i l u r e s were caused by workm anship (10 .6 percen t ) , con taminat ion of equipment( 8 . 2 p e r c e n t ) , and o ther manufac tu r ing cau ses (8.0 perc ent) . Some of the mo re s ign i f-i c a n t e x p e r i e n c e s are discu ssed in the following para grap hs .

S p a c e cr a ft W i r i n gBackground. - Because of weight limitations, much of the wiri ng in the Apollo com-mand module w a s insu lated with thin-walled Teflon cov ered with a thin polyimide coating.T h i s w i r e w a s ex t rem ely suscep t ib le to damage, and many sp ec ia l p roc edu res were de -veloped to ensure the integ rity of the wir ing a f te r in sta l l a tion in the spac ecra f t , du r ing

equipment ins ta l la t ion and checkout, and during the miss ion . The complexi ty of thec r e w c o m p a r t m e n t wire h a r n e s s is shown i n figure 4 .

Figure 4 . - C r e w c o m p a r tm e n t e l e c t r i c a l h a r n e s s .

21


27/37


28/37


29/37

F i gu r e 7. - Lunar module cabin wir ingc r i m p (C ) and s o l d e r (S) s p l i c e s a ndend caps (EC).

F i gu r e 8. - Command module cablet r a y s .

24

Background - protec t ion of wi r ing in-s t a l l a t i on . - Ini t ia l ly, the wiring in the c r e wc om pa r t m e n t w a s exposed and su bjec t toposs ib l e damage dur ing ground-based ope r -a t i ons as wel l as during f l ight .Expe r i ence . - To pro t ec t the c a b l e sf r om phys i ca l damage and to reduce ther i sk of f l am e propaga tion , spec i a l p ro t ec -t i ve t r a y s w e r e de s igne d ( fi g . 8). In addi-t i on , these t r a y s a ls o are removab le t ope r m i t changes i n t he w i r ing ins t a ll a t ion .Chang es in wi r ing ins ta l la t ions usua l ly re-q u i r e the ma ting and dem ating of con nec tors.T o e n s u r e t hat the connec tors are not dam-aged , p ro t ec t ive cove rs were needed andu s e d . Metal s c r e w c a ps were us e d first, ,

but thread ga l li ng and me ta l pa r t i c l e con-t aminat ion cau sed cons ide rab l e conc e rn .As a r e s u l t , the following d esig n and pr o-c e du r a l s ta nda r d w a s p r e pa r e d by JSC.Elec t r i ca l p lugs and recep tac l e s offl ight equipment and ground equipment thatconnec t w i th fl ight equipment must be pro-tec ted at all t i m e s . P r o t ec t iv e c o v e r s o rc a p s are r e q u i r e d o v e r electrical plugs andrece p tac l e s wheneve r t hey a r e no t connec ted

to the mat ing pa r t . The pro t ec t ive cov e rso r c a p s m u s t p e r fo r m the followingfunct ions .1. P r o t e c t the plugs and rece p tac l e sf r o m m o i s tu r e2 . Pro tec t aga ins t damage to sea l ings u r f a c e s , t h r e a d s , o r p i ns3. Be re s i s t an t t o abra s ion , ch ipp ing ,o r f l a k ing4 . Be br ight ly co lored so as t o beeas i l y d i sce rn ib l e and command a t t en tion5. Be maintained at a le ve l of cle an -l ine ss equiva lent to the p l ugs o r r e c e p t a c l e son w hich they are used6. Be ma de of m ate ria l that is c om -pat ible with the c onne c to r m a t e r i a l


30/37

Electrical ConnectorsBackground. - Many electr ica l connectors a r e used in the spacecraft . Many ofthese connectors have a large number of closely spaced, sma ll-diameter pins. Pro b-lem s encountered included crimping of smal l connector pins, moistur e proofing the

connectors, bent pins, and verification that connectors were mated properly. A s theseproblems occu rred, solutions were found and techniques and procedu res were imple-mented to prevent recurr ence . In some instances, design and procedu ral standa rdswere developed and contractually imposed on contracto rs; such w a s the case regardingmoistur e proofing of connectors. The standard requir ed that elec tr ica l connectors andwiring junctions to connectors be sealed from moisture to prevent open and shor t cir-cu it s and that the composition and quality of the seal depend on the environment to whichthe seal will be subjected.Background - crimping. - The l arge incidence of cri mp failu res in the ear ly daysof Apollo manufacturing precipitated an intensive review of the crimping tools and thecrimping techniques. One type of crimping tool then in use , even though proper ly cal-

ibrated , frequently produced overcrimped pins, with the result that the w i r e would failthe pull test at the point of o vercr imp. In other cases, wires were pulled f ro m crimpedpins because of improper calib rat ion of the tool.Experience. - The crimping operations must be controlled rigidly to en sur e thatthe proper tools, proper ly calibrated, a r e used by trained personnel. The followingcontrols were established by the contract ors.1. Each cri mp tool was ser ialized .2. A calibration reca ll date was established fo r each tool.3 . Each tool was pre set and seal ed fo r each crimp size .Despite these controls, a problem developed where 24-gage wires pulled out ofconnectors. Investigation disclosed that the crimps were made with the wrong tools .Th is condition occurr ed because the technician had se ve ral c rimp tools at hand andselected the wrong one. The tool contro l procedures were changed to pe rmi t techni-cians to check out only one tool at a time. To verify that a cr imp had been made prop-erly, test samp les were made before and after each crimping operation and subjectedto pull tests to verify that the proper tool w a s used and that the operat or performed thecrimping properly.Background - bent connector pins. - In many cases, the us e of high-density con-nec to rs res ulte d in bent pins. Added to the basic problem of extremely delicate andeasily bent pins w a s the frequent demating and mating o r reconnect ion of connectorsfo r equipment removal o r installation.Experience. - The problem of bent connector pins became so seve re that specialtraining w a s provided for selected personnel, and, thereaf ter , all connector handlingwas performed by these trained personnel. Proced ures for straightening pins wereprepar ed, and requirements were established for recording all instanc es of bent pins.Experience indicates that constant vigilance must be maintained to en sur e that connec-to r handling is a closely controlled function a t all locations.

25


31/37

A technique used on the Apollo spacecra ft to provide the capability of connectorintegrity control is to apply paint st ri pes on connectors that have been properly matedand ver ifi ed to provide an indication when unauthorized demating of the connectors oc-curs . Special paint was requ ired to ensure that the paint did not chip or flake and be-come a contaminant in the crew comp artment.Radiographic inspection of ele ctr ica l connectors. - Radiographic inspection wasused to validate the integrity of c ert ain connectors and termin al board s. While wir eharnesses were still in the assembly fixture, elect rical connectors were examined byX-ray photography. This procedure proved effective in detecting int ernal defects suchas improper component seating, broken wi res, bent pins, birdcaging, open join ts, andconductive inclusions . Radiographic inspection of connectors and te rm inal board s als owas used for troubleshooting after the wire har nes ses wer e installed in the spa cecraf t.Small portable X-ray equipment made this procedure entirely practicable. In thi s man-ne r, the connectors that could be X-rayed without having to remove other equipmentcould be inspected for bent pins and other d efect s. Experience with X-ray photographyof electr ic connectors indicates that this pr actice should be used mor e extensively. Theprocedure was relatively inexpensive, and many defects that could not otherw ise be de-tected were found with this method.

Contamination ControlBackground. - The advent of manned spa ce flight introduced stringent r equ ire -ments for cleanliness in the spac ecraft , spacecraft subsy stems, and ground supportequipment. In the zero-g environment of spac e, loose objec ts in the spacecraft (nut s,bolts, washers) can be a nuisance and, i n some cases , a haz ard because they floatfreel y about the int erior of the spacecraft and can provide an unwanted conducting path

in electrical circuitry or impede mechanica l actuation of equipment. Exper ience withglass-column .manometers and droplights used in the c rew compartment indicated thatbreakage of these gl as s devices would const itute a threat to the safe ty of the crew ifgl as s dust were left in the cabin and eventually inhaled by a crewmember. The us e ofall glass item s in the spacecraft is res tri cte d seve rely and controlled to prevent thishazard. Other for ms of contamination included the spill age and leakage of flu ids usedin the spacecraft system.Experience - contaminants in fluid sy ste ms. - The fabrication and assembly offluid syst ems and the associated components required p rec ision cleaning techniques andcontrolled environments. Many of the experien ces from the space prog rams are pre-sented in ref erences 7 and 8. Many cases of fluid s yst em contamination were caused

by the introduction of contaminants during welding, solder ing , brazing, or mechanicalcoupling of f luid system lines during installation of equipment. This experience led tothe establishment of special procedures to minimize the chances of contaminating thesys tem s during these operations.Experience - spacecraft cleanliness. - The following specia l controls were imple-mented t o minimize the problem of loose m ateri als i n the crew compartment.1. The spacecraft w a s "tumbled" or rotated severa l times to recover loose arti-cles that otherwise were inaccessible.

26


32/37

2 . The crew compartment w a s vacuum cleaned frequently .3 . Ultraviolet light inspection w a s used for detection of resi dua l solvents andother potential contaminants.4. Filtered air w a s supplied to the crew compartment when the compartmentw a s open.5. Hatch monitors were used to control entry into the crew compartment. Tools,mat eri als, and equipment were recorded in a logbook during in gr es s and egr es s.6. Mate ria ls (cleaning fluids and s o forth) that were used to se rvi ce the space-cr af t were controlled and had to be approved before they could be near or in thespacecraft.Experience - fluid spillage. - Several incidences of fluid spillage occur red in theearly phases of spacecraft assembly and test . For example, mercury w a s found in the

crew compartment after a glass-column manometer shat tered. The toxic effects ofmercu ry on the crew o r to spacecra ft metals caused concern and result ed in the estab-lishment of design and procedural s tandards to preclude the us e of mercury in or aroundthe spacecr af t. Another example of fluid spillage w a s the spi llag e of th e coolant solu-tion (water/glycol) used in the environmental control system. Spillage of th is ma terialon spacec raf t equipment and wir e har nesse s frequently necessi tated removal and re-placement of the equipment and costly and time-consuming cleanup operat ions. Reworkof the removed equipment to r end er i t usable is also expensive.Te st s were conducted to determine the effects of water/glycol solution on elec-trical wiring because the chemica l reaction between the water/glycol and the si lver -coated wire with broken insulation can provide a conductive path. Special procedures

were developed to as su re that water/glycol spills were cleaned proper ly with chemicalte st s to prove the adequacy of each cleaning.COMPONENT TESTS

B ackground??The ingle most important fac tor leading to the high degree of rel iabili ty of theApollo spacecraft w a s the tremendous depth and breadth of the te st activi ty. " The pre-ceding quotation is from reference 9.Test ing of the Apollo spacec raf t equipment re presen ts a substan tia l portion of thepro gram r eso urces and consists of two basic categories: certification testing and ac-ceptance testing. These test activities are discussed in the following paragraphs.Experience indicates that unl ess a comprehensive and well-integrated test plan

is prepared, much unnecessary testing can be performed and some may be missed.formed at the wrong assembly level. This experience led to the development of spec ial

Ofgre ater importance from a reliability standpoint is the fact that testing may be per -ground ru le s for testing Apollo spacecraft and equipment. These ground ru le s are dis-cussed in referen ces 10 and 1 1 .

27


33/37

C e r t i f i c a t i on o f S pac e - F li gh t E qu i pm en tBackground. - The Apollo spacecraf t equipment design is certif ied by t es t, byanalysis, o r by simil arity to other certified equipment. This certification methodminimizes the amount of hardware required for t est purpo ses and shows the capabil-ity of the equipment to withstand mission environments and durat ions. More than

700 test s were requi red before the f ir st manned Apollo flight.Experience. - The certi fication testing activity was guided by the following rules.1. U s e production equipment fo r test.2 . Test at highest practicable level of assembly.3 . Test all redundant paths.4. Test at or above maximum expected natural and induced environments.5. Perform acceptance test before certification test.6. Analyze to supplement testing.7. Understand and resolve all failures thoroughly.

The distribution of fa ilures detected during certif ication te st s, acceptance tests, andflight is shown in figure 9. Reference 12 contains additional information about the JSCcertification test p rogra m.Many design deficiencies were detected and correc ted as a resu lt of the certifica-

tion test activities. A distribution of certif ication te st fa ilur es by te st environment isshown in figure 10. The dynamic environments yielded a substantial number of the totalfailures experienced.Test environment*r *r

Acceptance Acceptance

Temperature cycling

0 10 20 M 40 5oFaiIu es. percentFigure 9. - Apollo spacecraft failures. Figure 10. - Distribution of Apollospacecraft certification failur esby environment.

28


34/37

Acceptance Test

Vibration testsonstruction type,percentS u b s y s t e m Electronic Elec tro- Component Units Number of F a i l u r e s ,mech an i ca l typ es te s ted fa i l u res p ercen tS eq u en cers 9 0 1 0 10 169 11 6 . 5En vi ron men ta l 10 90 14 339 32 9. 4E l e c t r i c a l 80 20 5 3 847 103 12.2Stabi l ization 80 20 11 184 28 15 .2C ommu n i ca t i on 9 0 10 23 441 105 23.8Instrumentation 9 0 10 7 118 30 25.4R eact i on con tro l 20 80 1 543 7 1 .3Prop u l s i on 15 85 9 235 16 6. 8Abort guidance 80 20 9 257 1 7 30 .0Mechanical 0 10 0 1 7 4 7 10 .6Radar 80 20 4 52 32 6.2i Display components 9 0 10 3 2 7 9 2 8 3 53 4 . 5D i sp l ay p an e l s 9 0 10 0 0 0 0

Tota l - - -- 174 11 187 741 6 .7

Acceptance tests were conducted on completed equipment and were intended to de-tect manufacturing defects. The acceptance test requirements for Apollo spacecraftequipment included therm al and vibration tes ts for those equipment it ems that were sus-ceptible to failure caused by temperatu re o r vibration. Electronic equipment containingsoldered connections is one example of this category. Defective solder connections canbe detected during the rma l and vibration testing, especially i f instrumentation is used tomonitor the circu itry for intermittency during the test.

T h e r m a l testsComponent Units Number of F a i l u r e s ,typ es te s ted fa i l u res p ercen t

8 159 13 6.213 197 22 11.230 449 28 6.211 388 71 1 8 . 321 273 73 2 6 . 0

5 469 26 5.63 757 7 1 9 . 33 22 1 4 .55 88 100 1 1 4 . 01 5 1 6 11 .70 0 0 0

18 848 177 2 1 . 011 106 77 7 .3

129 3807 665 17 .4

Many manufacturing and design deficiencies can be detected during acceptancetests. A summary of acceptance test fa ilu res that occur red during vibration and the r-mal testing is contained in table I. The testing of spacecraft equipment is described inreference 11.

T A B L E I .- SUMMARY OF ACC EPT ANC E TEST FA ILURES FOR ?HE LM AND CSM BY INDIVIDUAL SUBSYSTEM

Nondestruct ive TestsBackground. - The conventional nondestructive methods (radiography, ultrasonics,dye penetrant, magnetic particle, and so for th) of tes ting equipment were used on Apollospacecraft equipment. In some ca se s, combinations of these methods were used be-cause of the limi tations of a parti cular method in a parti cular application.

29


35/37

Experience. - Multiple inspections using nondestructive test methods can be usedeffectively i f there is doubt rega rding the reliability of the equipment. For example,LM descent stage gaseous-oxygen tanks that had been proof p re ss ur e tested were re-called and subjected to additional nondestructive testing. These tanks were reinspectedby using X - r a y photography, ultrasonic pulse-echo, and dye-penetrant nondestructivemethods and were found to be acceptable for flight use . Neutron radiography providedgood results i n detecting hydrogen embrittlement in weldments and for inspection ofpyrotechnic devices.

One innovation in radiographic inspection was very effective. Polaroid film wasused with conventional portable X-ray equipment to "rough in" shots of it em s o r objec tsin difficult ar ea s. The us e of thi s film sav es much time because of the sho rt develop-ment time compared to that of conventional X-ray fi lm.Background - receiving inspection at KSC. - When the spacecraft ar rived at KSC,an inspection w a s perfo rmed by the vehicle contractor and NASA, pri mar ily as a checkfor damage in tra nsi t. Many deficiencies were found, some of which were not record edat the time of vehicle shipment and ot he rs which had been recorded and accepted "as

is. '' Thus, i t was found that the inspecto rs at the two sites, at KSC and at the contrac-to r facility, were inspecting without benefit of written acceptance instructions o r cr it e-ria that were compatible with each other .Experience. - Inspection procedures were p repared, and a set of deficiency dia-grams was shipped with each vehicle to indicate where deficiencies had been noted andrecord ed at time of shipment. This condition is unusual because the spacecraft isshipped with many ins tal lat ions and operat ions that can be per formed only at KSC. Ex-perien ce indicates that much unnecessary effort and paperwork can be averted by a co-ordinated planning activity of the inspection operat ions re ga rd le ss of where they areperformed.

C O N C L U D I N G REMARKSTo be effective, reliability and quality assurance requ iremen ts and functions mustbe planned, monitored continually, and adjusted as the program develops and advances.The reliability and quality assurance activit ies are, in reality, disciplines that, i f prop-e r l y implemented, will enhance the probability of producing and delivering reliableequipment. These disciplines have to be selected o r tailored for each procurement.Some experiences related to the disciplines that we re monitored and adjusted as thepro gram developed are as follows.1. Specifying in contracts the scope, format, and content of reliabi lity and qualityassura nce plans2. Establishment of customer/contrac tor reliability and quality assu rance management review meetings of reliab ility and quality ass urance prog ram sta tus

30


36/37

3 . Development of incremental data a n d documentation reviews associated withprog ram milestone reviews for assessment of space-flight equipment4. Application of broad specifications for soldering, welding, painting, and si mi la rprocesses, instead of detailed step-by- step requirementsEffective contributions to product integrity are made by engineering personnelwho select pa rt s and materia ls, establish tolerances, consider environments involved,and prepa re the documentation fr om which the Droduct is purchased, fabricated, andhandled. Some experiences encountered in association with engineering functions that

contributed to this product integrity are as follows.1. Documentation of fai lur e mode and effects ana lys is and single fail

Apollo Experience Report Reliability and Quality Assurance

Documents

Transcript of Apollo Experience Report Reliability and Quality Assurance