THE NATIONAL SCIENCE FOUNDATION'S MANAGEMENT …

AD-AO 753 GENERAL ACCOUNTING OFFICE WASHINGTON DC PROGRAM ANAL-ETC F/6 5/1THE NATIONAL SCIENCE FOUNDATION'S MANAGEMENT INFORMATION SYSTEM-ETCIU)

APR 80

UNCLASSIFIEn GAO/PAO-80-7 ML

E"IEEIIEEIIEEEEllEEEEEElllEEEIIEEEEEEEEEEEEEEEEIIEIIEEEE

MBY THE COMPTROLLER GENERALI :, Report To The Chairman, Committee,On Science And Technoloy

oUnited States House Of RepresentativesQ F THE UNITED STATES r

The National Science Foundation'sManagement Information System:

__kA Status Report.The review of the current status of the Na-tional Science Foundation's managementinformation system and its plans for makingneeded improvements focuses on identifying-t/-- -changes planned or in process, quality con-

trols, user satisfaction, and cost.

Reliability problems, including inaccurate andincomplete data and slow response time atcomputer terminals, seriously impair thesystem's effectiveness and frustrate its users. cRecommendations for management and tech-nical improvements made by a Foundationconsultant 3 years ago have not been carriedout. A September 1979 study of system per-formance also pointed out the need for manysystem improvements. i

The Foundation can increase system reliabil-ity by improving quality control, system re-

>- sponse time, long-range planning, perform-ance evaluation, and administrative manage-

C ment. The Foundation also should determine( users' needs, and establish specific, quantified

goals and target dates for correcting defi-ciencies.(

This

80 5 2 7 15r .,,0

-wJ

Single copies of GAO reports are availablefree of charge. Requests (except by Membersof Congress) for additional quantities shouldbe accompanied by payment of $1.00 percopy.

Requests for single copies (without charge)should be sent to:

2U.S. General Accounting OfficeDistribution Section, Room 1518441 G Street, NW.Washington, DC 20548

Requests for multiple copies should be sentwith checks or money orders to:

U.S. General Accounting OfficeDistribution SectionP.O. Box 1020Washington, DC 20013

Checks or money orders should be madepayable to the U.S. General Accounting Of-fice. NOTE: Stamps or Superintendent ofDocuments coupons will not be accepted.

PLEASE DO NOT SEND CASH

To expedite filling your order, use the re-port number and date in the lower rightcorner of the front cover.

GAO reports are now available on micro-fiche. If such copies will meet your needs,be sure to specify that you want microfichecopies.

(~ COMPTROLLER GENERAL OF THE UNITED STATMS

WASHINGTON. o.C. OW

MAY 2 8 mo8

B-198136

The Honorable Don FuquaChairman, Committee on Science

and TechnologyHouse of Representatives

Dear Mr. Chairman:

In accordance with your Committee's request andsubsequent discussions with our representatives, we reviewedthe current status of the National Science Foundation's man-agement information system (MIS) and the Foundation's plansfor making needed improvements. The review focused on identi-fying major changes planned or in process, quality controls,user satisfaction, and cost. As agreed, we did not conducta major audit of MIS data guality because we found earlyindications of system and management problems that wouldlikely result in unreliable data. Nor did we include in ouraudit scope a determination of how well actual MIS operationsconformed tc stated procedures. Such audit work would havebeen unproductive, in our view, because system documentationand procedures were not always available or current.

We reviewed a 1977 management consultant's study of MISto determine how the system functioned then, and what im-provements had been recommended. We then determined howthe Foundation had acted on these recommendations. We useda General Accounting Office standardized format to assessreliability, determined user satisfaction from user question-naires based on several MIS routine reports, interviewed asampling of Foundation personnel who manage the system oruse it regularly, ani made test checks of data from severalsubsystems by comparing automated data with original docu-ments. We also gathered system cost data from Foundationpersonnel. Finally, we reviewed a report on a system per-formance study conducted during 1979 by the Federal ComputerPerformance Evaluation and Simulation Center. At the requestof ycur office, we delayed issuance of this report until wecould evaluate the Center's study which was published aftermost of our work was done.

We obtained Fodnda'ion officials' views on our findingsand recommendations, and their comments are considered inthe report. Our findings are summarized below and discussedin more detail in the appendix.

- - -- -- - . -

-'B-198136

SYSTEM DESCRIPTION AND MANAGEMENT

The goal of MIS is to automate all the Foundation's* administrative and program functions. MIS is made up of 5

major systems which are divided into 20 subsystems with specif-ic Foundation functions. All the subsystems use data from acommon data base (integrated data base) so that data must beentered only once into the system. A Honeywell computer systemprovides access to the data from 284 terminals throughout theFoundation's headquarters and annexes. Users of the system'sdata include Foundation management, the Congress, the Officeof Management and Budget, and grantee institutions.

A staff of. 128 Foundation and contractor personneladminister'the system, which is managed by the Directorate forAdministration through the Division of Information Systems(DIS). The Foundation began developing MIS in 1971 andhas spent over $15 million on it through fiscal year 1979,excluding personnel costs.

MANAGEMENT CONSULTANT'S EVALUATION

A management consultant assessed the system in 1976 andfound that user objectives had not been met and that signif-icant improvements in the system were needed. The consul-tant recommended changes and provided an action plan forfuture work. Since the consultant's study, systems and man-agement changes made to improve the system include establish-ment of a steering committee, user representatives, director-ate task groups, and a management controls branch; develop-ment of a financial accounting system implementation plan;and redesign and enhancements to several subsystems. TheFoundation has .not studied the impact of these changes, butsystem managers and users find these changes useful. How-ever, some of the consultant's recommendations for changesin important areas such as data quality assessment, long-range planning, and system performance evaluation have notbeen implemented.

THE SYSTEM AND ITS MANAGEMENTSTILL NEED IAPROVEMEdT

While MIS is not yet fully implemented, it is being usedto supply data important to manajing the Foundation. However,users still cite system reliability problems, particularlyinaccurate and incomplete data, and slow response time at thecomputer terminals. To solve these and other problems, improve-ments are needed in five areas--quality control, system re-sponse time, long-range planning, performance evaluation, and

4 administrative management. Each of these areas is discussedbriefly below.

2

B- 19 8136

The Foundation does not have procedures to monitor dataquality or identify system problems which cause poor dataquality. The consultant recommended in his January 1977 reportthat the Foundation immediately conduct a data quality studyto identify system weaknesses and develop data quality controlprocedures. However, the Foundation deferred the study be-cause of higher priority work and because system managers be-lieved the data was accurate despite the absence of formalquality control procedures.

Our reliability assessment revealed some system weaknessesthat reduce reliability. The weaknesses include inadequateerror correction procedures to insure the timely and ae curatecorrection of data errors that are found in the data base,system design problems which make identifying and correctingdata errors difficult, and inadequate system documentationwhich make identifying and correcting system design problemsdifficult.

System users complain that response time at MIS computerterminals is frequently slow. Reported delays of severalminutes when numerous terminals are concurrently active indi-cate a probable system overload. Foundation officials areaware that this problem has existed for at least 2 years.

In February 1979, the Federal Computer Performance Eval-uation and Simulation Center (FEDSIM) began assessing thedesign and performance of the Foundation's computer systemincluding system response time. The study showed that oc-currences of slow system response time were not primarilytraceable to system hardware, but rather to the managementof available computer resources. FEDSIM stated that improv-ing operating procedures, workload scheduling, and softwaredesign could minimize the impact of heavy prime time userdemand for computer resources and improve system responsetime. Also, the Foundation needs better system accounta-bility data to improve its use of available computer re-sources. Meanwhile, the Foundation added 46 more terminalsto the system in fiscal year 1979.

The Foundation's MIS management program lacks both along-range plan and a planning process. The consultant citedthe absence of effective long-range planning as adverselyaffecting system management and recommended that the Founda-tion pursue immediately a long-range plan and a planning, pro-cess. The Foundation still has no firm plan for developinga long-range system plan, and consequently has no firm targetdate or projected cost for completing the system.

The system is now managed on a short-range basis, es-sentially reacting to daily problems. While numerous work

3

B- 198136

projects for system improvements have been identified, therewere often no detailed documented plans for the improve-ments nor funds to put them into effect.

The Foundation has no procedures for systematicallyassessing user needs and evaluating system performance andcost in meeting those need'-. Since the MIS initial require-muents study in 1971, the Division of Information Systemshas assumed the task of determining the current needs ofprogram users. In so doing, system managers have used feed-back from program personnel through user representatives andtask groups. The consultant's recommendations for performanceeval'iation and cost accounting programs have not been imple-mented.

We found that other aspects of overall system managementneed improving, such as system security, documentation, andtraining for users and managers. Also, the Foundation'sinternal audit staff, which should monitor the MIS operation,lacks the technical capability to evaluate the system ef-fectively.

CONCLUSIONS

Significant improvements in MIS and its management areneeded if the system is to fulfill its intended purpose andachieve a degree of reliability satisfactory to its users.Although 3 years have passed since the consultant's evaluationstudy, the Foundation has not yet implemented major recom-mendations that the consultant said were needed immediately.As a result, serious technical and management problemsthreaten the successful operation of the system. Systemmanagers cite lack of resources as the main reason for notimplementing important recommendations. However, we believethat the absence of long-range planning and a knowledge ofsystem effectiveness has caused the Foundation to give lowpriority to areas such as data quality and system reliabilitywhich should have had the highest priority.

The Foundation is adding additional equipment to thesystem, which is already experiencing response time delays,without knowing what adverse impacts this equipment willhave on the system's overall performance. Considering theabsence of good system accountability data and FEDSIM'sconclusion that system hardware is not a constraint to im-proved system performance, we believe the Foundation shouldcarefully assess any further plans for adding terminals andassociated equipment to the system. Better management ofexisting computer resources, as recommended by the consultanit

4 and FEDSIM, would seem to be a more desirable alternative atthis time.

4

-m 1.- IF ,

B-198136

The value of 14IS to the Foundation, and user ac-ceptance of it, depends on high system reliability. ourreliability assessment shows the system to be of mediumreliability, which means that users of system data shouldverify the accuracy of the data before using it for impor-tant decisions. In our view, system users within theFoundation, the Congress, the Office of Management and Budgetand others will not, nor should they, be satisfied with datafrom a medium-reliability system.

While MIS is not yet fully implemented, it is alreadybeing depended upon to provide many essential servicesand data to Foundation management. Yet the system will beunable to provide highly reliable data until the technicaland management problems that we identified in this reportare addressed and remedied. In addressing these problems,we believe the Foundation should re-examine the purposeof MIS, clearly define its users and their needs, andestablish specific, quantified goals and target dates forcorrecting deficiencies and completing the system.

AGENCY COMMENTS AND OUR EVALUATION

We proposed that the Foundation take several actionsto improve MIS including actions to resolve the slow sys-tem response time problem, provide data quality control,and assess user needs. In a letter dated June 15, 1979,the Foundation advised that although it had concerns aboutour reliance on the use of questionnaires and interviewsto assess user satisfaction with MIS, it generally agreedwith most of our proposed recommendations. However, theFoundation did not agree that it had a slow system responsetime problem of the magnitude we described in the draft re-port. The Foundation referred to FEDSIf4's study of MIS'suser response time which, the Foundation said, showedresponse time to be in an acceptable range.

fining regnadin twogewid used rortuse produfcbtiSThdngegoudin suggwietdl htyu usep r satisfctbI

could not be attributed solely to the quality of MIS'sintegrated data base, but rather imay have been the result Iof differences in user perception of the reports. We be-lieve our questionnaires and interviews identified a legi-timate system probleim--errors in the data base--which ap-pears to adversely affect the accuracy of a widely used re-port produced from the data base. We selected two widelyused reports produced by MIS, each of which is used by adifferent Foundation management group. Program managers usethe Workload Status Report, whereas division directors use

4 the Proposal Aging Report. We asked only program managers

5

EEL

B-198136

about the Workload Status Report, while only division directorswere asked about the Proposal Aging Report. Our questionnaireshowed that users of the Proposal kging Report (divisiondirectors) were having problems with the report because oferrors, and the Foundation confirmed this by noting that ofthe two reports we tested, more errors occur in the ProposalAging Report.

The Foundation suggested- in its letter that althougherrors are present in the Workload Status Report, the errorsare not as easily recognized as in the Proposal Aging Reportbecause of the larger volume of information present in theworkload report. 'vie believe ezrors in the data base aristill errors, not just perceptions, no matter how unrecog-nizable they might be in some reports. The Foundation alsosuggested that the use of the Proposal Aging Report by manage-ment as an oversight tool over program managers may havecaused the managers to resent the-aging report, and judgethe report more harshly in our user survey. However, wedid not obtain program managers' views on their satisfactionwith the Proposal Aging Report--program, managers were notasked to judge this report. We asked only division directors--who are the primary users of the aging report--how satis-I fied they were with it.

The Foundation stated in its letter that the FEDSI!4tests show system response time to be in an acceptablerange and that the tests, as well as the agency's recentexperience, do not support the implication in the GAO re-port that only 20 terminals can be concurrently active onthe system. The Foundation's letter was dated June 15, 1979,prior to FEDSIM's final report. The final report, datedSeptember, 1979', acknowledges the occurrence of slow systemresponse times and does not characterize current systemperformance as being either acceptable or unacceptable.PEDSIM's report shows that hardware performance was generallygood during the controlled testing and was not a constraintto improved system performance, but that better managementof available computer resources would improve system perform-ance.

We conclude from the FEDSIM report that there are timeswhen system response time is poor. The study indicates thatunder controlled testing it is possible to have more than20 terminals concurrently active. How~ever, the day to dayoperations of the Foundation are a real world environmentwhere the computer system must respond to uncontrolled de-mands which we believe can yield situations where only 20

4 terminals could be concurrently active, or slow system re-sponse times could occur.

6

B-198136

Regardless of the cause, users will react negatively toslow response time at the terminals even if they are apprisedof possible causes. The Foundation's task, as we see it, isto eliminate or minimize what Foundation personnel perceive*as a system response time problem regardless of the exactnumber of terminals involved. We believe FEDSIM's recommenda-tions should be helpful in doing so.

Although the Foundation states in its letter that it hastaken steps to address user needs by establishing an advisorygroup, we believe the Foundation should insure that thegroup's actions will be adequate to provide for user needsassessment. As we have recommended, a performance evaluationprogram is.alsoneeded in addressing user needs.

RECOMMENDATIONS

We recommend that the Director of the National ScienceFoundation improve MIS by taking actions in the followingareas:

--System Response Time--Resolve expeditiously the slowsystem response time problem which is frustratingsystem users, and reassess the need for acquiringadditional hardware.

--Data Quality Control--Assess MIS's data qualityand establish procedures for periodically testingand maintaining data quality to insure high re-liability.

--Long-Range Planning--Develo' and maintain a long-range system and hardware planning process to insureorderly and systematic development of MIS. The plan-ning process should include top management partici-pation; provide a reasonable 5-year projection ofsystem requirements and costs, which would iicide.life-cycle cost analysis; establish quantified goaj4-and priorities for work projects; and set Mileston -a..completion dates.

--Performance Evaluation--Develop and implement perform-ance evaluation procedures to determine and controlthe efficiency of the system in meeting its goals.-- -Such procedures should assess user needs and insuresystem cost effectiveness.

--Overall Syste:n Management--Assess MIS short- and mid-range management efforts to insure that areas such as.

7

-.-'-- . -. ,, . ., ' .

B-198136

general system security, system documentation, andtraining receive adequate attention. Also, theDirector should strengthen the internal audit staffcapability to monitor MIS's performance.

As agreed with your representatives, our office will limitinitial distribution of this report to House Science and Tech-nology Committee and National Science Foundation recipients.Subsequent distribution will be made to interested parties 1week from the date of the report.

y your

Comptroller Generalof the United States

ts'

4i

8

I.

Contents

Page

APPENDIX

I THE NATIONAL SCIENCE FOUNDATION'SMANAGEMENT INFORMATION SYSTEM:A STATUS REPORT 1

Introduction IData quality control procedures

and improved system reliabilityneeded 5

Improvement in system responsetime needed 12

Long-range planning needed 15Improvement in performanceevaluation needed 15

Status of overall system manage-ment 17

System costs 21Conclusions 22Agency comments and our

evaluation 23Recommendations 27

ATTACHMENT

I Organization of the NSF MIS 29

II National Science Foundationorganization chart datedJuly 27, 1979 30

III Summary of NSF/MIS changes since 1/77 31

IV Analysis of data processing costsFY 1971-FY 1979 35

V Letter dated June 15, 1979, to theDirector, Program AnalysisDivision, GAO, from the Director,National Science Foundation trans-mitting comments on draft report 36

VI Excerpts from the Federal ComputerPerformance Evaluation and Simu-lation Center's September 1979report on the Foundation's MISentitled "Honeywell 6060 Measure-ment and Evaluation Study" 40.

_ ., . _ 4 .... ... .....

ABBREVIATIONS

MIS Management Information System

FEDSIM Federal Computer Performance Evaluation andSimulation Center

NSF National Science Foundation

DIS Division of Information Systems

ADP Automatic Data Processing

PI/PD Principal Investigator/Project Director

GAO General Accounting Office

FY Fiscal Year

I!

4,

APPENDIX I APPENDIX I

THE NATIONAL SCIENCE FOUNDATION'SMANAGEMENT INFORMATION SYSTEM: A STATUS REPORT

INTRODUCTION

On July 20, 1978, the Chairman of the House Committeeon Science and Technology requested that we review the Nation-al Science Foundation's management information system (MIS).The Chairman was concerned with the system's data quality andreports from Foundation staff that the system had not beenentirely satisfactory. He stated that an assessment of thecurrent status of MIS and the Foundation's plans for makingneeded improvements would be useful to the Committee in carry-ing out its oversight of the Foundation. The Chairman askedthat the assessment consider:

--identifying major changes planned and in process;

--identifying and testing quality controls for assuringdata accuracy;

--determining the Foundation's efforts to identify theneeds of the system's users and its plans to meetthose needs; and

--identifying the cost of MIS and of any plannedmajor purchases of hardware or software.

Pursuant to the Chairman's request and later agreementswith our representatives, we:

--reviewed a 1977 consultant's study of MIS to under-stand how the system functioned then, what problemswere identified and recommendations made; and whatactions the Foundation had taken on these recommenda-ations;

--interviewed a cross section of Foundation personnelwho manage the system and use it regularly to determinehow the system functions today and how it is managed;

--performed a system reliability assessment using our"Audit Guide for Reliability Assessment of ControlsIn Computerized Systems";

--made limited test checks of data accuracy in three keysubsystems by comparing automated data with originaldocuments;

41

APPENDIX I APPEN~DIX I

--determined user satisfaction from questionnaires basedon several -MIS routine reports;

--gathered system cost data; and

--reviewed a report on a system performance study con-ducted during 1979 by the Federal Computer PerformanceEvaluation and Simulation Center (FEDSIt4).

In assessing the status of MIS we limited our audit workto assessing the system through interviews, questionnaires,and spot testing of data. As agreed with the Hiouse Committee,we did not attempt the time-consuming effort to conduct a majoraudit of MIS data quality because we found early indicationsof system and management problems that would likely result inunreliable data. Nor did we include in our audit scope adetermination of how well actual MIS operations conformed tostated procedures. Such audit work would have been unpro-ductive, in our view, because system documentation and pro-cedures were not always available or current.

System development

The Foundation, under the authority of the N~ationalScience Foundation Act of 1950, as amended (42 U.S.C. 1861 etseq.) supports scientific research in various disciplines(such as chemistry, physics, biology, and engineering) pri-marily by awarding grants to colleges and universities. TheFoundation processes about 28,000 research proposals annually. i

of these, approximately 24,000 are acted on and eventuallyabout $900 million is paid out in grants.

To help process proposals and to provide current datafor program management, the Foundation decided in 1970 toautomate all administrative and program functions by usingan MIS. The project began in 1971 with a major requirementsstudy to identify all the functions the system would perform.This study was followed by a system design phase and theinstallation of the system. By 1975, all major subsystemshad been installed although some were running parallel witholder manual systems and some subsystems, such as the financialaccounting subsystem, had serious operational problems. In1977, an office automation project extended the system toinclude automated word processing.

WJhile the Foundation had automated parts of its programbefore 1971, the parts had not been organized into a compre-hensive information system. The new system design, on theother hand, used the concept of a single integrated data base

4 --that is, data would be entered only once into the system for

2


use in all subsystems, thus eliminating the need for dupli-cating data in numerous nonintegrated files. This datawould be accessible directly from video terminals.

The Foundation chose to put all of the functional partsof the system into operation at the same time rather thanimplementing them in a series of time-phased steps. Thelatter approach would have provided better control of system

Jo development and quality. Since then, the system has exper-ienced numerous technical and management problems as wellas resistance from agency personnel.

In 1976, a management consultant firm, D. P. ManagementCorporation of Lexington, Massachusetts, was hired to assessthe MIS project. The consultant compared MIS's performancewith the Foundation's goals and concluded in January 1977that the:

--MIS goals were ambitious but achievable over time;

--MIS had not yet met user objectives, althoughprogress had been made;

--efforts to maintain the system's current programand to achieve its growth objectives concurrentlyhad caused an imbalance;

--ADP administrative organization needed significantstrengthening; and

--many management programs and procedures needed forthe successful operation and administration of theMIS system did not exist.

The consultant recommended controlled, incremental developmentof MIS and provided an action plan with priorities for systemdevelopment. Some of these recommendations have been fully orpartially implemented; others have been deferred for lack ofresources.

System description

MIS uses a Honeywell computer system which providesan integrated data base (JASON) accessible from 284 terminalsthroughout the Foundation's headquarters and annexes. MISis made up of 5 major systems divided into 20 subsystemswith specific Foundation functions (see attachment I).

3


The najor MIS systems are the:

--Proposal/Award Administration System which sup-ports the receipt, control and approval or rejectionof grant and contract proposals, and the adminis-tration of active awards;

--Peer Review Administration System, which con-tains the names of 80,000 potential reviewers usedby the Foundation to decide on grant proposals;

--Grantee/Contractor Information System, which main-tains business and historical information about insti-tutions and researchers having Foundation grants;

--Administrative Management System, which supportsthe Foundation's management and administration; and

--Local Management System, which gives services suchas individualized data searches, specialized reportformats, query of data bases outside the Foundation,and development of individual computer programs.

Users of the system include Foundation management, theCongress, the Office of Management and Budget, and granteeinstitutions. The primary user is the Foundation itself,where upper level managers use administrative and financialdata for overall Foundation management. Increasingly, MISis being used by other Foundation personnel, such as programmanagers and administrative staff, for monitoring grant fund-ing levels and expenditures, tracking proposals through pro-cessing, and expediting letter writing.

System management

MIS is managed under the Directorate for Administrationthrough the Division of Information Systems (DIS) which iscomprised of five branches:

--Application Systems Branch--responsible for all MISsubsystems, software development and maintenance,requirements analysis, design specifications, andsoftware.

--Co iputer Facility Management Branch--responsible forthe computer facility including the terminals, oper-ating system software, executive control, and manage-ment of the integrated data base.

41

L I

77.


-- Customer Service Branch--responsible for office auto-mation, directorate user representatives, user train-ing, and technical writing.

-- Systems Support Services Branch--responsible for inputservices, processing support, fund controls, and thedocumentation library.

-- fManagement Controls Branch--responsiole for data qual-ity, system standards, security, system project manage-ment, and certification of new software.

The system is operated by a staff of 128, of which 80 areFoundation management and administrative personnel and 48are contractor personnel who operate the computer hardwareand develop system software.

Our audit found that MIS users continue to cite techni-cal and management problems affecting system reliability in-cluding inaccurate and incomplete data, slow response time atthe computer terminals, and lack of attention to the user'sneeds. To solve these and other problems, improvements areneeded in five areas: quality control, system response time,long-range planning, performance evaluation, and system manage-

ment. These areas are discussed in the following text.

DATA QUALITY CONTROL PROCEDURES ANDIMPROVED SYSTEM RELIABILITY NEEDED

The Foundation does not have procedures to monitor dataquality and identify system problems that cause poor dataquality. Furthermore, our assessment of system reliabiltyidentified other areas which need strengthening.

Data quality is not monitored

The Foundation has no systematic process for monitoringdata quality or statistically measuring the reliabilty of thesystem. Several one-time efforts have been made to correctselected portions of the data base where known errors existed.However, no formal and regular effort has been made to iden-tify the causes and solutions of the problems to prevent recur-rences. In addition, the system lacks documented error-cor-rection procedures to insure timely and appropriate correctionsto errors that are found.

The Foundation's consultant did not study data qualitybecause he found problems in the system that could contributeto poor data quality. However, he recommended 3 years ago

5


that the Foundation undertake such a study within 6 months.Nevertheless, the Foundation has not performed the recommendeddata quality study and has given it low priority becauseofficials believe the data quality is generally good. Insupport of their view, system managers cite a 1976 internalstudy of the Proposal and Application Information Subsystemwhich examined that subsystem during its first full year ofoperation. A~ total of 9,539 observations were made on 41data elements in a sample of 947 proposals from all Founda-tion directorates. 1/ The result was an overall accuracyof 95.1 percent, and 98.9 percent for critical data elements.

As part of our reliability assessment, we made spotchecks of 152 critical data elements from six proposals andfour awards to see if computerized data and original recordsagreed. Almost no errors were found. We also made a spotcheck of duplicate names in the principal investigatorsubsystem. Out of 2,176 entries in the alphabetical listof names for the letter J, 5 percent possible duplicationswere found. Further limited testing of seven data elementsin the financial accounting subsystem data from four awardsand three proposals showed no errors.

These spot checks do not represent the quality of datain the 28,000 proposals or 24,000 awards in process annually,but were done to determine the kinds and quantity of errorswhich might be anticipated in a full-scale quality audit.Both the Foundation's study of 1976 and our spot checkswere concerned only with the value of basic data elementsat a certain time, not with what happens to data duringthe running of individual programs or with the operationof all subsystems over time.

Through a user questionnaire we solicited commentsfrom a selection of 27 Foundation division directors andprogram officers on the quality of data in two widely-used routine MIS reports and the usefulness of MIS incarrying out their work. Program officers commented on theWorkload Status Report and division directors on the Pro-posal Aging Report. Respondents also provided numerousnarrative comments on MIS. The results, which follow, arethe opinions of a smaller selection of users, but are notnecessarily representative of the opinions of users as awhole.

4 I/The Foundation is organized around the Office of theDirector, six program directorates, and one administrationdirectorate. (See attachment II.)

6


USER REACTION TO MIS REPORTS--RESPONSES TO KEY QUESTIONS

Workload status Proposal agingreport report

Is the reported data: Percent Yes Percent Yes

--accurate and reliable? 82 29--complete? 73 86--available early enough? 67 79--current? 67 38--useful? 67 92--understandable? 100 100

Can the report be used as iswithout further correcting,explanation, or analysis? 69 57

Do you maintain manualrecords to supplement com-puter produced information? 1/ 69 71

Does the report duplicateany information you nowreceive? 2/ 25 71

On a scale of 1 to 10:

In the work of your office ordivision, the report is:(not important at all = 1,very important = 10) 5 5

How useful is the MIS to youin carrying out your dutiesand responsibilities: (notuseful at all = 1, very use-ful = 10) 5 6

I/Some respondents said manual records were kept because auto-mated records were unreliable, but others indicated thatmanual records might be kept anyway as a backup system.

2/Some respondents said duplication provided a useful cross-check of data and aided in error identification.

74


Representative respondent comments

-- MIS seems to be a closed system and used primarilyby DIS and upper managers. The needs of the programofficials are secondary.

-- MI3 data are too unreliable to be of use, necessitatingtne maintenance of manual records to check the computer.

-- The system is improving over time, but still has along way to go.

-- DIS should find out what programs personnel need be-fore generating software.

-- Too much time is required locating errors in the finan-cial accounting system and identifying such for the

Division of Financial and Administrative Management.

-- The system will never be better than the accuracyand coinleteness of its data.

-- Resoonse time at the terminals is often unacceptable.

-- Down time on the system is much too frequent and oftencomes at the end of the month when it is importantto get actions processed.

-- DIS staff are most helpful and courteous in respond-ing to any questions we have.

The results of the questionnaire survey including the nar-rative comments strongly indicate the need for improved systemreliability.

System reliability needs strengthening

Our reliability assessment is a judgment based on astructured analysis of organizational and computer operationcontrols and procedures including access controls, file con-trols, disaster recovery, teleprocessing, input and outputcontrols, and user opinions. We obtained information neces-sary for our evaluation from system managers and users throughinterviews and questionnaires. (Time constraints did notallow verification of this information with actual Foundationprocedures or system operations.) Reliability is evaluated interms of risk (potential for error). High reliability indi-cates low risk, low reliability indicates high risk.

4

L !L'


Our assessment revealed system strengths and weaknessesand resulted in a rating of medium reliability, which meansthat a user should audit any system data significant tocritical decisionmaking. A discussion of these strengthsand weaknesses follows:

Strengths

--Steering committee--Top management involvement, inthe form of a steering committee, is essential to thedevelopment and operation of an information system.In July 1977, the Foundation established the MISsteering committee to provide top management parti-cipation in all phases of MIS planning and resourceallocation. However, the committee, composed ofassistant directors, was chaired by an assistantdirector until January 1979, when the Deputy Direc-tor was made chairman. The committee lacks full-time staff support needed to function effectivelyand has spent much of its effort thus far definingits role.

--Separation of duties--Separation of duties enhancessystem security and, by providing an effective check

ulation of data, promotes accuracy and reliability

in dta.Within DIS branches, we found differentFoundation and contractor employees performing suchduties as maintenance of the operating system anddata management system, system design and programming,hardware operations, file maintenance, and data input.

--Security--Physical equipment, production programs andprocedures, and data files should be adequatelysecured to prevent unauthorized access. We foundaccess to the computer area and hardware is limitedto necessary personnel through the use of key locksand visitor sign-in logs and badges. Account codes,authorization codes, and passwords are controlledto prevent unauthorized use of data files and pro-cedures. Other DIS security measures prevent theunauthorized entry of program changes and data throughthe operator's console, detect attempts at unauthor-ized system intervention, and prevent unauthorizedmodifications to production programs. In addition,the MIS tape library is accessible only to authorizedlibrarians.

--Disaster recovery--Disaster recovery controls are pre-4 ventive processes that help protect critical data

9


files, software, and systems documentation from fireand other hazards, allowing a continuation of dataprocessing activities if such hazards occurred. DIShas established such procedures describing the actionto be taken in the event of a disaster involving thedata center, data files, and computer programs. Amongthese are provisions Zor copying and retaining masterfiles. Sufficient generations of files are main-tained to facilitate reconstruction of data records,

and copies of operating and production programs arealso maintained. These critical data files and soft-ware are stored in a vault outside the data centerand at an offsite storage facility.

-- Other controls present--Traditional control conceptsare augmented by controls written into computer pro-grams and other system software as well as controlsbuilt into the computer hardware. The MIS executivecontrol is a collection of software modules whichcontrol access to the integrated data base andjournalize all transactions which alter the database. Journalization, the storage of before andafter images of all altered pages in the data base,provides an audit trail for analysis of data baseintegrity and system restoration. In addition,computer programs include various types of editsand routines which are helpful in assuring that datais properly processed through the entire run.

Weaknesses

-- Internal audit oversight--Internal audit oversight ofdata processing activities helps to insure that com-puter-produced information is accurate and reliable.However, the Foundation's audit office has had littleinvolvement in DIS activities. Internal audit moni-toring has consisted of minimal participation in MISdesign and implementation, membership in DIS studyteam efforts, and verification of information oncomputer reports. Although an operational audit ofMIS has been a part of the yearly audit plan sinceFY 1974, the internal audit staff has not been ableto perform a detailed review due to limited funds,time, and technical expertise.

-- System documentation--Internal control in an auto-matic data processing department requires adequatedocumentation describing both the system (includingthe application systems and programs) and the proce-dures used. However, DIS management has not encouraged

10

( APPENDIX I APPENDIX I

the systems analysts/programmers to maintain and up-date systems documentation, feeling it would diverttheir limited resources from system/program main-tenance and other more important functions. Theamount of documentation varies for each system (sub-system). Original system documentation which wasdeveloped by the contractor and turned over to theFoundation during implementation has not been updatedor, in some cases, retained. Although these systems/programs have undergone considerable testing, testplans and their results are not available. Systemdescriptions and flowcharts are developed when neededfor documents and briefings. In addition to the lackof systems documentation, we observed limited proce-dural documentation. For example, DIS could not pro-vide documentation for ADP personnel duties and respon-sibilities, other than position descriptions. Althoughtape library procedures, disaster recovery procedures,and security procedures are documented, error correc-tion procedures are not. This can result in standardprocedures not being followed and errors not beingcorrected promptly and appropriately.

-- System utilization data--Utilization data is essentialto management for measuring the efficiency and effec-tiveness of the data-processing system. Availableutilization data includes the names of individualsaccessing MIS, the number of times they were online per period, and total lapsed time for each indi-vidual per period; for batch processing, the numberof jobs run daily. No data is gathered which could

be used to determine the number of times each proce-dure was accessed per period, the number of job reruns,or total processing time for each program. Failureto obtain and analyze these types of information couldresult in inefficient utilization of '13.

-- Internal controls--Internal controls safeguard re-sources from waste, fraud, and inefficiency and pro-mote accuracy and reliability in accounting andoperating data. Although MIS functions appear tobe secure, several procedures need to be improved.For example, tighter controls are needed over systemand program documentation kept at the system analysts'and programmers' desks to prevent unauthorized access tosuch material. Also, sensitive output, such as personalinformation, is discarded along with nonsensitiveinformation instead of being shredded or burned.There is no nonprinting/nondisplaying or obliteration

4i


facility incorporated in the terminals to preventunauthorized persons from having access to accountingand authorization codes and passwords. These addi-tional control procedures would help produce a moresecure climate for data processing.

System design problems

In addition to the security and system control weaknesses,we learned about system design problems in the Principal in-vestigator/Project Director (PI/PD) Subsystem. The MIS inte-grated data base allows data to be strung out in chains, allow-ing records to be linked by pointers within each record andpermitting the linkage of related files so that data needbe entered only once. However, this chain linkage is absentbetween the P1/PD Subsystem and the integrated data base.Instead, this subsystem is linked by common data elementswithin the integrated data base. Any changes to the P1/PDdata records must be made individually to each common dataelement. Errors could be made while making these changes orsome common data elements could be overlooked, creatingunsynchronized data within the data base. Another problem,the absence of a standard identifier within P1/PD records,can lead to duplicate entries in the data base. DIS is cur-rently seeking solutions to both of these problems.

Our data quality and reliability assessments were per-formed from October through November 1978 and apply only tothat time frame. The results are not necessarily typical ofthe system over other time frames because the system ischanging with respect to design, procedures, and managementcontrols, and there is no ongoing data quality assessmentprocess. The Foundation has system and computer programchange control procedures that provide some control over, anda chronological history for, the modifications made to thesystem.

IMPROV1EMENT IN SYSTEM RESPONSE TIME NEEDED

System users and managers reported experiencing exces-sive response times at the terminals, sometimes as long as5 minutes. System managers want a response time of under5 seconds. The problem was generally described as an over-load problem arising when 20 or more of the Foundation's238 (at the time of our study) MIS terminals were concur-rently active. The Foundation's consultant noted in his re-port that response times of several minutes were common. We,too, experienced long delays in getting on the system andsubsequently retrieving data during information gathering

4 for our audit.

12


While system managers agreed that a response timeproblem had existed for at least 2 years, they had not de-fined the exact degree of the problem or identified itscauses. Instead, they have been trying to work around theproblem by requesting program personnel to use MIS at timeswhen the computer center was not performing high volume dataentry. System managers were looking to a study by the FederalComputer Performance Evaluation and Simulation Center (FEDSIM)to assess the problem further.

In February, 1979, FEDSIM began studying the Foundation'scomputer system analyzing workload characteristics, systemresponse, system design, system utilization levels, andoperational procedures. The study included two system load-ing experiments to determine the number of users that couldbe adequately serviced on the present computer, and to iden-tify potential system bottlenecks during peak use periods.The principal conclusion of the FEDSIM study was that signi-ficant improvements in slow response times were not to befound by adding central system herdware. Rather, they would

come from better management of existing computer resources.

The FEOSIM findings were qualified by the instrumentation Iavailable and the study methodology used. For example, JASON,one of the two major systems tested, lacked the softwareinstrumentation necessary to collect terminal response times.Since response time delays had been observed using JASON,this limited the study's ability to identify and suggestspecific solutions to response time problems. Other qualifi-cations include a limited workload and operational environ-ment measured under controlled conditions; no batch pro-cessing load applied during the system loading experiments;and Foundation staff and representative workloads used inthe experiments were selected and coordinated by Foundationpersonnel, not FEDSIM. (See Agency Comments and Our Evalu-ation on page 23 for additional comments.)

FEDSIM reported that those system response times thatwere measured were generally good during the testing. (mostwere in a range of 1 to 10 seconds.) However, some delaysranging from 6 seconds to 6 minutes and 15 seconds occurred.The tests also demonstrated that up to 71 users could accessthe system concurrently under the controlled conditions ofthe test.

The study reported that the Foundation does not haveadequate system accountability data for assessing and manag-ing the performance of its computer system under varyingworkloads. Improving this condition was FEDSIM's first

13

{APPENDIX I APPENDIX I

recommendation to the Foundation. Additional FEDSIM findingsaffecting system performance and responsiveness include:

--Competition exists between data entry personnel andsystem users for computer resources.

--Formal response and turnaround time requirements forsystem users do not exist, together with lack ofcommunication with users over what constitutes rea-sonable expectations of computer performance.

--Most batch workloads are processed during the dayshift when users' demands are at a peak.

--Central processing unit and memory resources are al-most fully used during the day shift.

--System resources are not a constraint to additionalworkload during evening, night, and weekend processingperiods.

PEDSIM recommended many changes to the system includingthe following:

--Enhance the policies and procedures for system account-ing to provide Foundation personnel with better in-formation by which to manage computer resources.

--Establish performance goals for the system by identify-ing formal response and turnaround requirements forsystem users.

--Reevaluate the design of the Office (Local) ManagementSystem to meet user needs while reducing load on thesystem, and

--Improve guidance to users concerning th, optimum useof the system.

Even though the Foundation has no continuing performanceevaluation program, it has continued to expand the number ofterminals accessing the system without knowing whether thisexpansion will further delay system response time. In 1977 theFoundation added a second communications control unit with 16communications lines to the MIS computer, which already had64 lines. The Foundation added 46 additional terminals infiscal year 1979 to the 238 it already had.

14


LONG-RANGE PLANNING NEEDED

*rhe Foundation's MIS management program lacks both along-range plan and a planning process to establish quanti-fied goals, priorities, and target dates. As early as 1972,Foundation management recognized that past efforts to auto-mate Foundation files had been too narrow in scope and hadled to disappointing results. While development of MISwas based on a 5-year plan, the plan ended in 1976 with nocontinuing long-range planning process in existence. TheFoundation then contracted for a system evaluation. The con-sultant cited the absence of effective long-range planning asadversely affecting system mnanagement and recommended thit theFoundation immnediately develop a long-range plan and a plan-ning process. He emphasized the need for long-range hardwareplanning to control costs and efficiency of operations andrecommended that a long-range hardware plan and a computeroptimization study be done immediately. He also recommendedthe immediate implementation of a system development manage-ment program which would provide for the orderly developmentof system software.

After 3 years, the Foundation still has not developedthese long-range plans, performed the optimization study,

L or implemented a system development management program. DISofficials said these projects have received low prioritybecause daily operation and maintenance of the system wasconsuming the available resources. Furthermore, no firmplans exist for pursuing these projects.

Currently MIS is managed on a short-range basis,essentially reacting to daily demands and problems. dJhilenumerous work projects for system improvements have beenidentified, there were often no detailed documented plansfor the improvements nor funds to put them into effect.

IMPROVEMENT III PERFORMANCE EVALUATION NEEDED

Forty-thiee Foundation personnel including upper-levelmanagers, progjram officers, and administrative and financialpersonnel answered questionnaires. The users stated thatMIS should perform better than it does and that greaterattention is needed to determine and meet user informationneeds. These issues are discussed below.

Users expect MIS to perform better

Most users criticized MIS for unreliability in dataaccuracy and completeness and, as previously discussed, forunsatisfactory delays in getting on the system and long waits

15

{APPENDIX I APPENDIX I

at the terminals for data access. 3ecause of such problemsmany users have kept in their offices manual records which areused for computerized data verification and for program manage-ment. Several users noted that DIS had asked program officersto review long computerized lists of proposals and awards fortheir divisions at the end of fiscal year 1978 to help correcterrors, reconcile financial data, and find proposals andawards which had become lost in the system.. One user crit-icized the system for not having established error correctionprocedures so users could clearly describe the error discoveredand recommend how it should be corrected. Another user stated,however, that the computer was not to blame for all errors be-cause some were caused by the program office staff's failingto identify and report errors.

A better assessment of user needs is required

vie found no adequate process for evaluating and address-ing user needs. Except for the initial requirements study of1971, the Foundation has not systematically assessed user needs.In answering our questionnaires (see p?. 7) users said thatMIS is only moderately important and useful in helping themcarry out their duties. They identified problems which indi-cate that MIS5 is not adequately addressing their needs.For example, users said that MIS reports required further cor-I rection, explanation, or analysis, that manual records wereneeded to supplement computer-produced information, and thatsome reports duplicated information already received.

Some program officers complained that some routine MISreoorts were not useful and therefore not used. Instead, per-sonnel used their own computer programs to develop special-format reports tailored to their specific needs. Other criti-cisms included heavy use of numerical codes for classifyingdata, which required frequent use of a code dictionary, theundesirable format of data, lack of effective training for useof the system, and inadequate communication on user needs be-tween system users and system managers.

The Division of Information Systems has assumed the taskof determining the current needs of program users. In doingso, system managers have used feedback from program personnelthrough user representatives (DIS personnel located withinFoundation directorates) and the directorate task groups(committees that monitor MIS performance). These channelsare still the main sources for identifying user needs, sincethe Foundation has not established a performance evaluationprogram to assess user needs and how well they are served.The consultant recommended such a program 3 years ago.

16

L APPENDIX I APPENDIX I

STATUS OF OVERALL SYSTEM MANAGEMENT

The consultant made 23 recommendations for system improve-ments in applications organization, and systems and resourcesmanagement programs. Of these the Foundation has addressed13 recommendations, partially addressed 2, and not addressed8.

For those addressed we did not assess the extent to whichthe changes have effectively improved MIS performance. How-ever, a description of the recommendations and the status ofimplementation is presented below. (See attachment III for

additional details.)

Recommendations addressed

1. Establish a hierarchy of steering committees. Twocommittees were recommended: one, composed ofupper level managers, would determine overallpolicy, funding, goals, and objectives; thesecond, composed of operational staff, wouldmonitor policy and program implementation.

The first committee was established (July 1977),but not the second. Instead directorate taskgroups were established in early 1978 tomonitor MIS implementation.

2. Provide user representatives. Each directoratenow has a program analyst from DIS located on-site to help program personnel use MIS. Userreaction was favorable in GAO's questionnaireresponses.

3. Establish a "quick response" service group. Astaff of four has been established in DIS to ex-pedite MIS response to external requests for data.

4. Develop an implementation plan for a financialaccounting system. The consultant developed aplan which the Foundation is now implementingon a 5-year basis. Implementation is now inits third year.

5. Establish a DIS administrative staff. Qualitycontrol of MIS products and activities has beenfragmented among numerous parties with none hav-ing authority to establish standards. As a reme-dy, DIS established a management controls branch(April 1977) responsible for quality control, systemdevelopment, cost accounting, training, and security.

17

d&-


Thus far the branch's activities have been limitedto organizing itself and performing ad hoc tasks.Resource limitations reportedly have preventedgreater achievements.

6. Enhance award processing. The consultant statedthat the Foundation could enhance its award process-ing by automating award letters and institutionalindirect cost rates. Both capabilities have beenadded. Also, the financial accounting system nowautomatically reflects award actions.

7. Enhance award management. The consultant statedthat control and monitoring of active awards wasinefficient because of numerous manual operationswhich could be automated; for example, ticklerfile- used to identify delinquent reports. TheFoundation has made the recommended enhancementand is currently reviewing and redesigning, wherenecessary, the Awards Mlanagement System.

8. Conduct requirements study. The consultant per-formed a requirements study for the Foundation inearly 1977 which assessed detailed work projects,priorities, and resources required. The study wasI to be the initial step in developing a long-rangeplanning process. The study concluded that the MISworkload was well in excess of available resources,

thus requiring critical management decisions.

9. Centralize ADP expenditures. Several directoratescontract for supplementary data processing servicestailored to specific needs (cost is approximately$500,000 annually). Previously these contracts werenot coordinated with DIS. Now the steering commit-tee reviews and approves these expenditures.

10. Implement certain office automation modules. Officeautomation is the use of computers to perform officeprocesses otherwise done manually. MIS providesprograms (modules) for such processes as letterwriting, text editing, data base searches, and dic-tation services. Twenty percent of the Foundation'sdivisions have some modules installed.

11. Defer automation of general office functions. Con-ceivably office automation can result in usingelectronic media totally, instead of paper. TheFoundation has limited office automation to word-processing applications and deferred automating

18


processes that would eliminate altogether the useof paper.

12. Develop a skills improvement program. The consultantnoted that DIS needed to develop better managerialand ADP technical skills in its staff. Meanwhile,it recommended contracting for software until in-house skills could be improved. Since then, DIS hascontracted for all major software. Formal trainingcourses have been identified and made available topersonnel.

13. Make miscellaneous enhancements. Numerous detailedand technical recommendations were ,made for strength-ening major subsystems by 1980. The Foundation hasfollowed up by identifying numerous subsystem pro-jects, many of which were completed in FY '78. Anadditional 80 projects remain, 38 of which werescheduled for FY '79. The remaining 42 projects areunfunded and include such tasks as user documentationof system procedures, evaluation and enhancement ofsubsystems, providing report numbers on recurringreports, testing annual closing procedures for thefinancial accounting subsystem, and rewriting reportsfor optimization.

Recommendations partially addressed

1. Conduct a workload process study. The consultantsaid a study of workload processing proceduresshould be done within 6 months to streamline exist-ing procedures. Subsequently the Foundation hadthe consultant perform such a study in one direc-torate (science education). Studies in the otherdirectorates have been deferred.

2. Adopt a project management organization structure.MIS development efforts have been accomplished bynumerous people working on specific tasks with noone individual responsible for overall coordination.The consultant said this approach was hindering ef-fective management and recommended immediate adop-tion of a project manager approach where one personwas responsible for specifically identified projects.DIS adopted the project management principle withinits existing organizational units without changingits organizational structure, on the premise thatmost MIS work is now ,maintenance and not develop-mental. Also, limited resources did not permit

4 a staff of the size Project management requires.

19


Recommendations not addressed

1. Conduct data quality study. The consultant did notperform a data quality study but recommended thatthe Foundation perform one within 6 months. DISanalyzed and corrected errors in selected portionsof ~.he data base but did not perform a major studybecause data quality was considered good. In ad-dition, resources were lacking. Aks we began ourreview DIS officials said they were reinstitutingthis study with newly acquired resources.

2. Develop and maintain a long-range systems plan.The consultant cited the absence of effective long-range planning as adversely affecting system manage-iment, and recommended that the Foundation pursueimmediately a long-range plan and a planning process.No long-range planning process or plan has beendeveloped. Currently DIS relies on the MIS steer-ing committee and the Management Controls Branch forplanning.

3. Prepare a long-range hardware plan. The consultantnoted the necessity for effective long-range hard-ware planning in controlling costs and efficiencyof operations and noted that the Foundation wasnot doing such planning. The consultant recommended

.1 that a long-range hardware plan and a computer opti-mization study be done immediately. To date, lackof resources and low priority have prevented suchstudies, according to agency officials.

4. Develo2 a performance management program. The con-sultant found need for a better ability to determineand control how well MIS is performing its missionAn intermediate-level study was recommended usingreadily available data. DIS has not decided how itwill act on this recommendation.

5. Develop a program-budgeting and cost-accountingsystem. The Foundation's efforts at ADP budgetingand cost accounting were judged as very rudimentary,and a more formal systematic approach was recom-mended. The Foundation has not pursued this recom-mendation and is undecided about its ultimate dis-pos it ion.

46. Develop and install a systems development manage-ment program. DIS has lacked a systems development

20

IAPPENDIX I APPENDIX I

management program and has relied on contractors todevelop its software. The consultant recommendedimmediate development and installation of a formalprogram for this purpose. Subsequently, theFoundation had the consultant develop a program,but it has not been implemented and is not scheduledfor implementation.

7. Reposition system analysts into the developmentprocess. The consultant noted the desirability ofpositioning system analysts between users and pro-grammers to help insure that user needs were ef-fectively met. DIS has not found it possible to dothis because of manpower limitations. Therefore, itis seeking to develop system analyst skills withinits programmer ranks.

8. Develop a new personnel system. The consultant foundthe agency's personnel system needed major improve-ments. The Foundation made several enhancements ofthe system pending a system requirements study.

SYSTEM COSTS

According to Foundation figures, MIS has cost over $15million from fiscal year 1971 through fiscal year 1979,excluding personnel costs. Items of cost include equipmentrentals and purchases, supplies, and software and system de-velopment contracts. Costs for fiscal years 1976-1979 in-cluding personnel are shown below. (See attachment IV forgreater cost detail.)

Fiscal Year ADP Costs* Personnel Total

--------------------------------- (millions)-------------

1976 $2.4 $1.2 $3.6

1977 2.0 1.4 3.4

1978 2.3 1.6 3.9

1979 3.8 1.8 5.6

*ADP costs include equipment rentals and purchases, supplies,and support contracts.

During fiscal year 1979, the Foundation purchased the4 second processor it had been renting, but shelved plans to

21

APPENDIX I NPPENODI I

add a third communications control unit. The Foundation

added 46 more computer terminals.

CONCLUSIONIS

Significant improvements in M4IS and its management areneeded if the system is to fulfill its intended purpose andachieve a degree of reliability satisfactory to its users.Although 3 years have passed since the consultant's evaluationstudy, the Foundation has not yet implemented major recom-mendations which the consultant said were needed imneliately.These recommendations addressed data quality assessment, long-range planning, performance management, and system mana-leaent.

Serious technical and management problems threaten thesuccessful operation of the system. While system managerscite lack of resources as the main reason for not implementingimportant recommendations, we believe that decisions madewithout long-range planning or knowledge of system effective-ness are also major causes. We believe the Foundation hasgiven low priority to areas which should have had the highestpriority. We found, for example, that the Foundation isgiving higher priority to implementing office automationthan to achieving a highly reliable system. Nlso, the Foun-dation has added equipment to the system's operating hardware,which is already experiencing response time delays, withoutknowing what adverse impact this equipment will have onthe system's overall performance.

Additional equipment would serve to increase potentialdemand on the system during peak daytime hours. Consider-ing the absence of good system accountability data andFEDSIM's conclusion that system hardware is not a constraintto improved system performance, we believe the Foundationshould carefully assess any further plans for adding termi-nals and associated equipment to the system. Better manage-ment of existing computer resources, as recommended by theconsultant and FEDSIM, would seem to be a more desirablealternative at this time. Any additional equipment shouldbroaden the scope of services to users, yet the Foundationhas not yet assessed how effective existing services are,or how such services can be improved.

In our view, MIS's value to the Foundation and user ac-ceptance depend on high system reliabilty. Our reliabilityassessment shows the system to be of medium reliability,which means that users of system data should verify the ac-curacy of the data before using it for important decisions.System users within the Foundation, the Congress, the Office

4 of Management and Budget, and others will not, nor shouldthey, be satisfied with data from a medium-reliability system.

22

49--

(APPENDIX I APPENDIX I

Some changes have been made to MIS as a result of theconsultant's recommendations but the Foundation has notexamined the impact of these changes on the system. SoIfar, improvements are not obvious since users continue to citesystem unreliability and slow response time at the terminals.They also report that manual records must be kept to verifyand correct computerized data and that MIS reports are noteffectively meeting their needs. In addition they report theabsence of good training and documentation on using thesystem and correcting data errors.

While MIS is not yet fully implemented, Foundationmanagement already depends on it to provide many essentialservices and data. Yet the system will be unable to providehighly reliable data until the technical and management prob-lems identified in this report are addressed and remedied.In addressing these problems, we believe the Foundation shouldreexamine the purpose of MIS, clearly define its users andtheir needs, and establish specific, quantified goals and tar-get dates to correct deficiencies and complete the system.

AGENCY COMMENTS AND OUR EVALUATION

We proposed that the Foundation take the followingactions to improve MIS: (1) resolve the slow system responseI time problem, (2) assess MIS's data quality and establishprocedures to periodically test data quality, (3) develop andimplement a long range system and hardware planning process,(4) conduct MIS performance evaluation to include assessinguser needs, and (5) improve overall MIS management in areassuch as security and training.

In a letter dated June 15, 1979 (see attachment V), theFoundation advised us that although it had concerns aboutour reliance on the use of questionnaires and interviews toassess user satisfaction with MIS, it generally agreed withmost of our proposed recommendations. The Foundation statedthat it was or will be taking actions to implement most ofour recommendations to improve MIS. However, the Founda-tion did not agree that it had a slow system response timeproblem of the magnitude we described in the draft report.The Foundation referred to FEDSIM's study of MIS's user re-sponse time which, the Foundation said, showed response timeto be in an acceptable range. The Foundation agreed, how-ever, to conduct computer performance evaluations on aregular basis to maintain an optimized system. Our evalua-tion of the specific points the Foundation raised in itsletter follows.

23

APPEN~DIX I APPENDIX I

Audit methodology to assess user satisfaction

The Foundation stated it had concerns regarding ourreport's usefulness because of our reliance on interviewsand questionnaire data to help assess user satisfaction withMIS. The Foundation noted that the use of such data couldmake it difficult to distinguish between legitimate systemproblems and other factors that influence user satisfaction.The Foundetion suggested that our user satisfaction findingsregarding two widely used reports produced by MIS could notbe attributed solely to the quality of MIS's integrated database, but rather may have been the result of "differences inuser perception" of the reports.

our questionnaires and interviews identified a legiti-mate system problem--errors in the data base--which appearsto adversely affect the accuracy of a widely used report pro-duced from the data base. Sqe selected two widely used reportsproduced by AIS, each of which is used by a different Founda-tion management group. Program managers use the WorkloadStatus Report, whereas division directors use the ProposalAging Report. Wqe asked only program managers about the Work-load Status Report, while only division directors were askedabout the Proposal Aging Report. The differences in userI perception appear very much dependent on the data base errorssince the Foundation states that, of the two reports we tested,errors tend to be concentrated in the report (Proposal AgingReport) our work showed had the lowest user satisfaction rate(29 percent). our questionnaire showed that users of theProposal Aging Report (division directors) were having problemswith the report because of errors, and the Foundation confirmedthis by rnoting that of the two reports we tested, errors areconcentrated in the Proposal Aging Report.

The Foundation suggested in its letter that althougherrors are present in the Workload Status Report, the errorsare not as easily recognized as in the Proposal Aging Reportbecause of the larger volume of information present in theworkload report. However, errors in the data base are stillerrors no imatter how unrecognizable they might be in somereports. Errors are present in some data in the integrateddata base that can adversely affect the accuracy, and thusthe reliability, of a report produced from the data base. Theextent to which the errors adversely impact report qualitywill depend on the specific data used to develop the reportand the intended use of the report. Mlerely suggesting thaterrors in the data base are just perceptions because theerrors are more recogniz-able in one report than anotherattempts to obscure the real point--that there are errors

42

(APPENDIX I APPENDIX I

in the data base which have an adverse impact on usersatisfaction with MIS-produced reports.

The Foundation also suggested that the use of the Pro-posal Aging Report by management as an oversight tool overprogram managers may have caused the managers to resent theaging report, and therefore judge the report more harshlyin our user survey. However, we did not obtain program man-agers' views on their satisfaction with the Proposal AgingReport--program managers were not asked to judge this report.WJe asked only division directors--the primary users of theaging report--how satisfied they were with the report. Asa result, our questionnaire could not have measured the re-sentment program managers might have regarding the agingreport since they were not asked.

Foundation suggestion forGAO work scope qualification

The Foundation suggested that we include in the lettertransmitting our report to the Congress a notation regardingthe limitations that were placed on our work. We had alreadyincluded a paragraph qualifying our work scope in the intro-ductory section of the report (see paragraph 1 on page 2 ofappendix I). We have also included this work scope qualifi-cation in the introductory section of the basic transmittalletter to the chairman.

Analysis of FEDSIM reporton MIS response time

The Foundation stated that the FEDSIM tests show systemresponse time to be in an acceptable range and that the tests,as well as the agency's recent experience, do not supportthe implication in the GAO report that only 20 terminals canbe concurrently active on the system. The Foundation's letterwas dated June 15, 1979, prior to FEDSIM's final report. Thefinal report, dated September 1979, acknowledges the occur-rence of slow system response times and makes many recommenda-tions for improving overall system performance. The studydoes not characterize current system performance as beingeither acceptable or unacceptable. The study reports thathardware performance was generally good during the controlledtesting and was not a constraint to improved system perfor-mance, but that better management of available computerresources would improve system performance.

GAO's reference to diminished system response time capa-bility when 20 or more terminals are in use was based oninformation received by interviewing Foundation personnel.

4 our inquiry showed that system managers did not know the cause

25


of the apparent slow response time problem and may have, intheir speculation, attributed the problem to numerous relatedor unrelated causes. We conclude from the FEDSIM study thatthe number of terminals that can be concurrently active andthe response time at those terminals is dependent on manyvariables, including the number of users and the difficultyof their procedures, the competition between data entry per-sonnel and other users for computer resources, system sched-uling procedures, and others. 4Je also conclude from FEDSIMthat there are times when system response time is poor. Thestudy indicates that under controlled testing it is possibleto have more than 20 terminals concurrently active. H1owever,the day to day operations of the Foundation are a real worldenvironment where the computer system must respond to uncon-trolled demands which we believe can yield situations whereonly 20 terminals could be concurrently active. Improvedsystem account-ability data would be needed to show how fre-quently this condition actually occurs.

41e believe it is important to note the FEDSIM study'squalification that its test results deoended strongly on theworkload and operational environment measured under quali-fied conditions. W1e observed that some of the qualifica-tions imay limit the study's conclusions with respect to theI Foundation's actual daily operation. For example, JASON,one of the major systems tested, lacked the software instru-mentation necessary to collect terminal response times. Also,the test methodology used in the system loading experimentsexcluded oatch processing from the workload applied to thecomputer system. The combined effect of batch processingand normal user demands on limiting the nunber of terminalsconcurrently active is unknown.

In its discussion of available computer resources, theFEDSIfI study generally assumes 24-hour system availability.If all the Foundation's procedures could be distributed overa 24-hour period, undoubtedly system overloads could beminimized. But in practice, the Foundation has a prime day-time peak loading period which is a period of great user de-mand to which the system must respond successfully. As thestudy notes, the central processing unit resources and inemoryresources are almost fully utilized during this Jay-timeperiod.

4e also observed that the system loading experimentsused participants and representative workloads selected byFoundation personnel. Such an approach could introduce agencybias into the testing in that all aspects of a typical work-load environment mnay not have been represented.

26


We believe that tne FEDSIM study will be useful to theFoundation in strengthening the management of its system. Be-cause of the limitations of the study, however, we believe itwould not be surprising to find system users experiencing thesame response time problems we reported, despite FEDSIM'sconclusions that response time is generally good and thatsystem hardware is adequate to handle the current workload.The difference lies in assessing system components undercontrolled conditions as FEDSIM did, and the results of thesystem operating in a normal work environment.

Regardless of the cause, users will react negativelyto slow response time at the terminals even if they areapprised of possible causes. The Foundation's task, as wesee it, is to eliminate or minimize what Foundation personnelperceive as a system response time problem regardless of theexact number of terminals involved. 4Je believe FEDSIM'srecommendations should be helpful in doing so. (See attach-ment VI for excerpts from the FEDSIM report showing reportabstract and FEDSIM's conclusions and recommendations re-garding the Foundation's MIS.)

Assessing user needs

Although the Foundation states in its letter that itI has taken steps to address user needs by establishing an ad-visory group, we believe the Foundation should insure thatthe group's actions will be adequate to provide for user needsassessment. As we have recommended, a performance evaluationprogram is also needed in addressing user needs. The impor-tance of conducting user needs assessment cannot be over-emphasized. It helps insure that the system is providing theservices intended and that the services are accurate andreliable.

RECOMMENDATIONS

We recommend that the Director of the National ScienceFoundation improve MIS by taking actions in the followingareas:

--System Response Time--Resolve expeditiously the slowsystem response time problem which is frustrating sys-tem users, and reassess the need for acquiring addi-tional hardware.

--Data Quality Control--Assess MIS's data qualityand establish procedures for periodically testingand maintaining data quality to insure high reli-

4 ability.

27


-- Long-Range Planning--Develop and maintain a long-rangesystem and hardware planning process to insure orderlyand systematic development of MIS. The planningprocess shoul] i.nclude top management participation;provide a reasonable 5-year projection of system re-quirements and costs, which would include life-cyclecost analysis; establish quantified goals and priori-ties for work projects; and set milestone completiondates.

-- Performance Evaluation--Develop and implement perform-ance evaluation procedures to determine and controlthe efficiency of the system in meeting its goals.

Such procedures should assess user needs and insuresystem cost effectiveness.

-- Overall System Maintenance--Assess MIS short- and mid-range management efforts to insure that areas such asgeneral system security, system documentation, andtraining receive adequate attention. Also, the Directorshould strengthen the internal audit staff capability

to monitor IIS performance.

4

28

ATTACHMENT I ATTACHMENT I

I I

2

0z40

LL ow (fz.>

20

Iz0

o L c

0

4.

292

00. 4 0 aJ

ATTACHMENT II ATTACHMENT II

rtt

L - I-T I I LT - 1En I-1R

IcI0 W

-j

0 '

z s.

-sw al-

a ax.

30

ATTACHMENT III ATTACHMENT III

0 n

n, Co 0, 0'0) (D C)c )

.33 :3 c: .- a .V. . .3. C).

a) 0 0 c) *-3C- 0)400 04- En

In4433 4, 34 0, 0 4411 m4) wr 0 z CF.0 m U,4' 0 T3 3 'a a) 3 4(n C ) 0.m-3 04 1, " 34u i'

>43 In3- In . - C4)

(fl 0 a)4) a)' 0n a a4 0 0

In 4J 1 ) )t

04. 0U) 4)c

-3 CL0 c 1 f! 3c c), ' q w 0 (1a z

04 . ). 0 ~ )4) U 3

w0 4 3C)S3 U) 0 .40 UC.a

c4) 0C -30 r-43C' C( 33C .5 C

t-f ) 0 . 3 ) ) 3 3 -

. 3 0 4

Ez 4)U (D U ~334U -0 p 43, 0 .U e- M. 3~.U.3-.3 U 0)4

'34 n

-3 u- c- a, C.cII n

In3- 04~ u0 a0 ' '

u. (00 >34 4) 4)U)

.n 040 In c 0' 0

tO c3 > c .1a)3 4 1Wa0 3cmT c 1u C . -:,0

0- v 41 (a10 4)0 Cm CJ . 3 CtoCU

Ai 41Z 4)4 El fa C 'a ml -1u E - 0.34 LEI - ) E-'4)wIn(a -0 0 ) )-

>34 ) o ) C = 0 (1 '0ZK 0 W I 3 C U) 3 j t L1

:3 E -' 0) 3 w 4C Uco C 3:) .4) 0. I I.3 '03 3

0O 4) 0) Ci3 0U) 0(1) C ED

(D 34 a 0 .. 0.4 0 >4)- 0 cw) 0 - 4 3." . 'o-3 C'34C U) . . 4 .. - ..(a 4))0 .30 41 00')0- 0 0 qjI 0 0 0 4

U) Cn IS (n o 1 .) . c 3 33-30,. Ia3 O C0 -3 w0, r=.3C.- F ,3 0,. .- t4) J

S43) cx, 43 0 0 OuC0 E3.0C 0,0U . 0a) -D -C. 3-. C. 4) 4 In c w 3CIn- w c d)343 .003 .30'0 .3> 0 043w4

4 ~ ~ ~ ~ ~ ~ i C4 FYI 4 C-3030 43.- -34~r U)30 -3) m)3~.3 m440 3 .)43

w3 '.." -~ 440. 4 -0 0)04

34 1

C-7C


c I >r- 4)

0~> 0' .a) C w 4jI~ - .. 0

c c 44 w0 4)4 0~4) 4444w 10 01 >40 0n l w _

m 4JC 0>' 4-V .4)w 4

C a0 44 C.- 4 ) Q.~4 c) 0

m4 4C 4 a 3 C4 =. 4 '. 4J

-j r-l4J)~ or ~ a V0 Vco VC0. m'~ 04L4 4j 0

0~ T5 4j4 Cm 4)4) 0'(J4 4)0

4) 4), 4 >

V C 4) r--4a.0 00c 04 440 4J 440

0 w4 4; w0 4V

4J r- 0. WC) >.>0C c a 44. e 4j 0, C0 4l

Cl 00 0 c- C44 0 4j. OW = 4 m n 0

0o -4 LaJ. C 04 wa w) C 0co4 0) ) )) ,4.4 44

0 r- C -444. -0

Cs.~ ~~~~~ Cr4 )4 )4 --- 4J44 04W !(0 r 444 Ca 4j~ 'a . 0j (a4La w'Lw4

0j

Cs0a 4J V V V

4J0 4) 4 w ) w) to 0) 4)w:j4 4) f ) 4) 4) 4) 4) E4)a o .

4) 0~ to in to La r= 4L0a Lma W-AjA

wL0 0' 0' 0' 0 ' '

.1-.. ' 4 'a w4) t n t

444 4) .CW 0 4 a C0 CL 4) cm l V.W :44 r.4 44 .4w *4.c l - c m

-4 .> e., -. c - 0 cu 4

V to 0 0 >Lw .4a ) A .4) , c- ~ 44j 00IJ r 0, V 0 c-u0

44 o> c3wc c 44 4)J4L fa0 c4 04 '4) La03 4 0 w 4004 0.-4 ,4) C44 W4 m M- m4L Va4e44.t4'Cm4m4:

0n 0 0 4)0 4 0 044 0 44 c 1 0 >-')40 0) x4w 0' 444 4J fn4) 4 0 t I

w) wC 0 - 44) I 0 4 C m'4 4)JmCw)040a

w m.4a .10 4a4. La .0 m >, W 4J 4404 V C 4 0to44 w c w j 3t - = 4J C44. m4 to4. toLa 0 cu 44 L

u u C A4 ~ 4 t m4j t 0)Cc 4) 4) 40.0 aJ4 c- 0)4u e4 44a. c 4 tn -C44w 1ft 0 C 0 , La 44444 . t r

C C44 4)0-4 #a 4) 444 a. Ca. CV00t UA4 m.) " 4)44) 4) 4 "A C, V. 0.4 44 Ci Q. u.44C4

a.) 04-4 w EA w 0 ac r 0 44 Oc n 4V 0 3 U'l*w0 W =44 0O44L

0 CIO444 ms40-4 (.l4 00. u m 04 4 0. u 0. 0 w .44w

a.14

44 4) c;

Ir~' I 4) V2


0

4-)

01

A4 W 0

4) r_ WC J04 )cM0r nA0C . 4ua i C '4

Ur 0 .

W 0, 4j 0

41e m .:.ro a m McC 1

40- 0.-' wWw ma ,aA 4w i W 0 . , 4 Cr-W 0t Z0 W0

4c 0C u 0 C4.4~~~( 4) c )044C 'E4'4 4. 4)4) 4 ) 0444 'a M

'0 ~ ~ 4-)0.400 '4 4'4*'0 4-m a)) W4mC.C4 ~ 404

.4 ~ ~ ~. W04~ M W.C C.mfc.0

'4 -- 414 4>V 44) .4 )f'' C 4 4

44~~.4 44) EEr

0~~~ 4) '44W'

4 '_ 0 CL ct 0 0 4J44 -4 0a 0. 0 a4 ... 4) 0' 0 '44)to 4 4) -

COO 4) r_) '4 c0 .4 )4

WO o0' 0 ' 40d4C4>' 0' O 0=

..r .. M4 w ~ -4 0 0 -m4to 41 c) a, >4)4 '-4 nIn Il'4.-Ij ( w4 m4 .- c 41 4-) a

m44 ) -) 4) 4) 4)4 o0m ac0

4 D,,W0 4 u4 -4,-4ML -41

4) W. t 4C' '4m 0' ., ~ C J U4 4) 0

to*00 CL rD. W -10 4a>c0Z! w4. goa w 0 4) 44) -4 C.0

* a .0 40.)C C0 '4 '0Aj 4'U r-344' -W' CCD' 4 '0 4

W )0C'0 O I4' 0, Cl400 to'44-4 mI- 0 4) >.04) ONO 0 W 0 Mg 0 . r0 m0 rcto j

'4.4- C 4 QC' >4. 0).4-U 04) U '0 4J 0M U-0 0 C.4 Ml 4)0. -A . C 0) 4 C4) > 4)L

fG W C D m U)C 44J 10'4) 44 to ' C 0- o4 )4)a

E4 *4 c 4) 0 * - 0. 4 4 ,W0. t-V a W u Ac 4 041 wu- 0t.. ' r C6ClE 'a u.' 0 .4c )-4>4' WO : W A 4.

>04 0-44 43) c 0 4 m ' 'a -40' x .-.41 c6z 0 ' Amm' W-4 a .4W 04) ,0)t aU U44)0 V )m o0 0' oWt

'4 " 0 0) m4 4) 0. -C 0 Cw- W 0H 0 ,c F n0W 00

0 X )a 4I -.0444-~ 00 '44 C..40'0 coZ 0.-I , C0 4 JR0 - .U

CO OX33

>4 O~ 0


41 4) w)

0 W 0 w C 0 4 -4

C 0 c 10 r: a -!7c1jf 10 'a U. w. c:) w~ 0(40) 0( 0( -4) .4j

a- a- w- 0-"I 0 m 04m

44 CW. C) C ) 0 V

4 .0) (n 0 4J ) 4J.- AJM4

V 0V mm" w () c: r, :3 c 64(44)

W4) 00 4) -( 41 ~ 0 a) 4)4)0b..C

0) wI 4)u > c .410 Oo 4) C 0 0 .014

0 3 V4 4. w. ) 0CC a)- IOIal wV UWW U C_.4 W4)0 4U) tnC,4) '4) 0-4 > 4)4ow) MM

z CO J00 00 v )0 ) 4 _ww ) 0..)C 0.W C44W.T3

0 4J~ 4J ) 4) w) 4.VU)-40 I 4 J 40T4)WU.4J "

Uc 04) CU Cw 44~(-0 .l~4 >114

Z 0~~~~00~ 0 0 4) o.ew ,4C 444W V

C. ) 4)u ) 4 Z4)0~' 0 a) *4 'V4 a tn d) V u)W

C.4 :) 0'a4 04 0 M

Z14 1. V (44)(4.4

a- 4 J) 01 Cj 4C * 44 ) !7

4 c H C4 .4 4)0CO W)4) W.~ C)4 () 4) 0)

(D4)4)c 4j0a m c a 4) T3.4) 4) 4.)4)41 4) 4 0 4)1 jIna

r4.4C41 -fl.( AJ

4) 4J 4) w 'a I ;

C 71 44) 0144 Cn I" r 0 C to 04)4) >I c- V= >) 0 0 0- ) 4J 0 w 44) to4.)

:I r44 0C.4 C 14 .0 tn 9I d) 4j.V 1r- toC1 c ta 4J ti '-)41.-) 4 ) r r= 4C 4J .1*C )-I0 4)4a)-w 4.) 0u (ti44.414 w) 0 w to A144-4(a1mlto 0 4110

a)U 2 Mo fa(7 4)3 4)oC0 CCL 4)144r4( tn4) U- .0 - 0)0V4c4) 0 wa 0 40 M-1 010C0)4 (44 7-4 14 r 0 fC

.4 M-O M710 4) (a (47 .- 40 m (n 4 44)0 E .c m 0 e-

440 c44 >4 w4a-4 cc4) 0 ME 4 A-e40 c.). -1 4C 4)1 L .44w- 4 alla4 to '(v 00 CL (a11 4)4d) Mu 0 0 CZt 00M 4) .4ln1VG44004)4

-0 do 'j1 MC )U E41 to Cr 0)0. CC m)C) . 44 0

C 9 4) 0) a4 a.14 C mwu -4(tn4) -4 > C C00L-I 44 01 4 0 4 ) O Q .4 U * ) 0 w4m 4 ) . 1 : 3 w '. t n 4 ) 0 . . . 4 0 4 4 ) 4 ) r= C r - 03 t w4

w4) ( 0(4 ) 00 4) CM 0) m 10 M. 'a0 w -0 1-4>CO(aw a 0.4 .c w4) 4)u'-4 >4)0 to >0n4) >-404 14 c r 04Z., (4.- LOm.a0 c~ 44m41ac 4)4)14 4) 4) > 4) w w O'44( 0V 'w4 0- c4.. U 0E 4)-ICIW44)00

4.)r- 9( I 4JM4) M1 41 T)1

4) a,4) 0 c ( C C m4-I c WC4r) 44 fa. 00 t. .4V >.4) 444 4)-

to4 t >0 44(4 0 1 C > 4J0.-I 4) 0 > CillC l 1400 0. m0 (a1 W .4)o) - 1.-Ia 5. < _4 0 m w 44) -A4-.4 ~4-P) 41 a to0 a7) w 3 w01Oc )

10 u 0 C a4-m(4 u4~ .)C4 c 0' (4 ) 4) c 4) 0) u). * UC :N .- iW U) tw14 0 0) 4 4 C 171( r04)34tnr_04) 41( Cw .' efl5On4m)(tnm14

x44) 00 -IVOVO cIi w U)04 4 400-.r=4 W O4. c :3 41-44) w r 4) -4 ))014) WV-w = 44 .04 -m V C0M4 444) .c '" 4.4 300u

4- m 0 -40V V

41(0)4) 4)14V 4)4)V Cw(. 4)0410 (.4C OV4 1* 444)R.0 c " 0 --14 *--I-10 :3 CW 44 -4) C v () 14w >,heC W -C 41 W ) 44 W0 -I4 r=>1 e .401.0 a-Ito40 m z 4.4)4w4mC60t to4-0 M1(.-(.0 0 (DL.

43

(4) - (0 e

ATTACHMENT IV ATTACHMENT IV

W Wr nr nC 0qmvc'3 LCn

"W-4(n-41n or rD m dcOZ Li C c~i C~t'c.Ii - lm It

-4 1- -C ID r r 0 %D C

00)

.1.4 1 - - -

(/3- WI-

.)

0. C) 000 0D4-41. 4 CD c" - )

rn~~r rqI). t.0~0

ro wa)) 0

F= - -C DC

m~~ 0 (a C C ) )Cm ~~~ ~ 0)t 4J -r -;c

E-D- 0I 11 -4 - -4 ~(/3- U>

r- .o 00fnC

aI - -n C -4-' Od'D 0m .0-4 C 0 m~ 1>

' c O k. IT - - , r-4 r.4r.- 4 -4

ry % ). tf- v.

00%)

- -4 r--4%o0 0- 0r-r-

41 rn m 4v -4 0 -4o (C) 1-1r0n - -4 t 1 - 1 r- :0

04 3 (1)- % % ;> *354

-44.C) LIn) I- C4 . o nr 04o c'Jm0( a ko o CA C)

~~~'4~~~ ?I .0 t- O.

.,I ~ ~ ~ -9- '.0- - 0 . cDL

4-1 %l0 t.0 Lr c0% ( 0 041 r v wr- m C m - C1

rn -- r-4 r-35

4J£

IATTACHMENT V ATTACHMENT V

NATIONAL SCIENCE FOUNDATIONWASHINGTON. D C 20550

June 15, 1979

OFFICE OF THEDIRECTOR

Mr. Harry S. HavensDirectorProgram Analysis DivisionU. S. Government Accounting officeWashington, D.C. 20548

Dear Mr. Havens:

I appreciate the opportunity to comment on your office'sproposed report on the National Science Foundation'sManagement Information System (MIS). We appreciate theimportance of a responsive, accurate MIS and are deter-mined to continue improving our system's effectivenessfor both the NSF staff and the Congress. Your reportwill be useful to us as we develop plans for futureimprovements in this important resource.

However, I am concerned that the report's usefulness maybe diminished by the audit team's frequent reliance onanecdotal evidence rather than empirical data. Interviewsand questionnaires help assess user satisfaction, of course,but they also can make it difficult for an audit team todistinguish between legitimate system problems and otherfactors that influence user satisfaction. An example ofthis effect is the user survey conducted by your auditteam. The survey revealed that 82% of the respondentsare "satisfied" with the accuracy and reliability of the"Workload Status Report," while only 29% are "satisfied"with the "Proposal Aging Report." In fact, both reportsare derived simultaneously from the same integrated database. Thus, the differences in user perception for thesetwo reports cannot be attributed solely to the quality ofour integrated data base. The "Workload Status Report"contains information on all proposals received by theFoundation during the current fiscal year. The "ProposalAging Report" contains proposals received, less those onwhich an action has been entered correctly in the MIS.Thus, overall data base errors tend to be concentrated in

4this report simply as a result of th* Ifefte og "a report.

(See GAO response in Appendix I, pp. 24, 25.)

36

(ATTACHMENT V ATTACHMENT V

( These same proposals and errors also are present in the"Workload Status Report" but, because of the larger volumeof information present, they are not as easily recognized.Moreover, the aging report is used by top management toensure the prompt processing of all proposals submittedto the Foundation. The report serves as a valuable toolfor identifying "old" proposals and bottlenecks in theproposal processing stream. This use of the report asa management oversight tool may be resented on occasionas an intrusion into the responsibility of programmanagers, and therefore may be judged more harshly inuser attitude surveys than less sensitive reports.(See GAO response in Appendix I, pp. 24, 25.)I recognize that the audit team did not have time for anexhaustive study of the NSF system and had to rely on lesstime-consuming techniques for data gathering. I do believe,however, that this limitation should be acknowledged explic-itly in the Letter transmitting the report to Congress andin the conclusion of the repo2 (See GAO note below.)

I also would like to comment briefly on the report'sspecific recommendations.

The managers of the NSF system are aware of the importanceof system response time as a measure of system reliability.'I The users of our MIS, as noted in the report, would likea response time of under 5 seconds, and our system manage-ment agrees that this is a desirable goal. During thepast three years, the Federal Simulation Center (FEDSIM)has conducted two computer performance evaluations at NSF,the latest commencing in January, 1979. A major portionof the most recent study was designed to assess the inter-action between our MIS executive and the timesharing sub-system. Controlled tests consisting of adding sequentiallyup to 75 user terminals were conducted several times toassess the impact on the system, including response time.Response time during these tests was considered by FEDSIMto he in an acceptable range. These quantitative tests,as well as our own recent experience, do not support theimplication in the GAO report that only 20 terminals canbe concurrently active on the NSF system. The finalFEDSIM report will be published in June, 1979 and willprovide detailed data on system response capabilities.(See GAO response in Appendix I, pp. 25, 26, and 27.)

GAO Note: Qualification statement included on page 1 of theletter to the Chairman. our conclusions are basedon our audit work which is qualified.

37

(ATTACHMENT V ATTACHMENT V

We intend in the future to conduct computer performanceevaluations on a regular basis so that we maintain anoptimized system.

Data quality is of continual concern to us, since ourMIS is founded on the premise of supplying accurate andtimely information to all levels of users. We willinitiate a data quality assessment program in the nearfuture. This program will identify problem areas, andalso may involve adding controls to individual applicationsoftware as warranted.

We recognize long range planning as an area needing moreattention in the future. We also recognize the need forclose coordination of ADP plans with overall NSF plans.Indeed, in order to determine user's needs, an advisorygroup was established in January, 1979 to assist inbringing user's requirements to the attention of MISmanagement. This advisory group has recommended thecreation of a planning team comprised of computer spec-ialists, user representatives and management. Thisrecommendation is being adopted, and I expect it to leadto greatly improved planning, both for the next fiscalyear as well as the longer term.

Performance evaluation is a difficult art. It has rarelybeen successfully implemented in federal agencies. Never-theless, we recognize the need to evaluate system perform-ance and assess the costs of meeting user needs. Thus,beginning in July, 1979 we will begin experimenting with anew project management software system. This software toolshould help us collect data and produce reports on thecosts of data processing and use of our resources. Inaddition, we plan to implement a computer accounting system.Thus, in the future, cost data should be available forsoftware development, maintenance, and computer time.

Computer security is an important issue, and we havesubmitted our security plans in response to 0MB directives.moreover, in response to a previous GAO report, we recentlyappointed a computer security administrator in the officeof the Assistant Director for Administration. Theadministrator is responsible for developing a securityplan that will involve users, computer technicians, and

38

ATTACHMENT V ATTACHMENT V

internal audit staff. In addition, the NSF Audit Officerecently hired a computer specialist who will work on ADPaudits. A software certification program also has beenestablished to improve the quality of computer products.This program will define technical documentation standards,and then analyze all software products to ensure compliancewith these standards.

Again, I wish to express my appreciation for the effortsof GAO and to assure you that they will assist us inimproving the Foundation's computer system. I look forwardto seeing the final report.

Sincerely yours,

Richard C. AtkinsonDirector

39

ATTACHMENT VI ATTACHMENT VI

AY-0 88-072-NSF

HONEYWELL 6060 MEASUREMENTAND EVALUATION STUDY

SEPTEMBER 1979

PREPARED FOR:

NATIONAL SCIENCE FOUNDATIONWASHINGTON, DC 20230

PROJECT PERSONNEL ROVED:*

Capt Gary W. Richard RICHARD L. BLAIRr Colonel, USAF

Commander

FEDERAL COMPUTER PERFORMANCEEVALUATION AND SIMULATION CENTER

WASHINGTON, DC 20330

44

AI

{ATTACHMENT VI ATTACHMENT VI

PREFACE

This report is based on the analysis of a large volumeof data. The results address the measurement of the NationalScience Foundation's Honeywell 6060 computer system.Because the results depend strongly on the workload andoperational environment measured, generalizing the recom-mendations beyond the system described or extracting con-clusions without their respective qualifying conditions isnot possible. Questions related to this report or to thepossibility of extending the stated conclusions or recoin-

Ir mendations should be addressed to the study's author at theFederal Computer Performance Evaluation and SimulationCenter (FEDSIM).

41

ATTACHMENT NII ATTACHMENT VI

ABS TRACT

From 12 February to 13 April 1979, the Federal ComputerPerformance Evaluation and Simulation Center (FEDSIM)conducted a computer performance evaluation study of theHoneywell 6060 computer system operated by the NationalScience Foundation (NSF). This project report presents theobjectives, methodology, findings, conclusions, and recoin-mnendations of the study. This Abstract contains the mostimportant conclusions and recommendations based on NSF's (1)workload characteristics, (2) system response, (3) systemdesign, (4) system utilization levels, and (5) operationalprocedures.

WORKLOAD CHARACTER IST ICS

(1) NSF should establish formal response and turnaroundtime requirements for system users.

(2) Much batch workload is processed during the dayshift when on-line systems are also demanding systemI resources. NSF should evaluate the requirement formost batch activities to process during the dayshift.

(3) Programmers should make more use of system loadablefiles and libraries to reduce the overhead requiredto reload programs each time they are used.

(4) The use of BASIC programs on the TSS system shouldbe re-evaluated to see if these programs could bemore efficiently coded into loadable TSS subsystems.

SYSTEM RESPONSE

(1) TSS response times are good. Longer response timesexperienced by TSS users are a result of non-trivialrequests to TSS and problems (disk errors, etc.)that stop TSS from responding to all users.

(2) JASON response times are more a function of JASONuser activity (or the interaction of JASON4 sub-systems with the database) that a function ofhardware resource availability.

4?

ATTACHMENT VI1 ATTACHMENT VI

SYSTEM DESIGN

(1) NSF should re-evaluate the design of the OfficeManagement System to more closely meet the needs ofsystem users and to reduce the system's load on TSS.

(2) NSF should remove JASON data entry functions fromcompetition with other JASON users to provide data

* entry personnel with more consistent response.

(3) JASON should be instrumented to provide the necessaryinformation to better evaluate JASON response timeand the interaction between user-written JASONsubsystems.

SYSTEM UTILIZATION LEVELS

(1) Few additional CPU resources exist during the dayshift. However, CPU resources are not considered tobe a constraint to system performance because (1)excess CPU resources exist during non-prime processingperiods which may be used by jobs now executingduring the day shift and (2) response times aregenerally good or if poor, they are not a direct

function of CPU availability.

(2) Memory is almost fully used during the day shiftcausing some jobs to move in anc6 out of memoryiepeatedly. However, memory is not considered to bean overall constraint to system performance becauseexcess memory resources exist during non-primeprocessing periods.

(3) Disk and tape channel utilization are not a constraintto improved system performance.

(4) The Datanet 355 front processor and memory utilizationare not constraints to improved system performance.

OPERATIONAL PROCEDURES

(1) NSF should improve the collection and retention ofsystem accounting data. Improved reporting proceduresusing these data should be implemented to provideNSF management with information to better managecomputer resources.

44

f ATTACHMENT VI ATTACHMENT VI

(2) NSF should examine disk configuration, use of NOFMSdisk options, and system file placement to improvedisk efficiency.

(3) BASIC users should use the "RUN - filename' optionwhenever possible to reduce the resources requiredby the BASIC compiler.

(4) NSF should improve communications with system usersto (1) notify them of system malfunctions, etc. and(2) provide guidance to improve awareness of how auser's activity impacts response times and systemresource use.

4

44


E. SUMMARY OF CONCLUSIONS

The following summary of conclusions restate conclusionspresented throughout the Findings (Section IV) and containinformation based on the methodology, findings, and con-clusions of the system loading experiments. FEDSIM suggeststhat Appendix D be read before proceeding with this Section.Each conclusion is a result of the analysis of NSFBs (1)operational procedures, (2) workload characteristics, or (3)system component utilization levels.

1. More accurate accounting data and improved methods fortheir collection and retention would benefit management andevaluation of the NSF system.

2. Although system scheduling procedures were good, somefactors and procedures in the current scheduling designcould constrain system performance.

3. System availability was good. The TSS system wasavailable for most of its scheduled periods. However, theJASON on-line system was unavailable to users a greaterpercentage of the time which intensified JASON workload whenthe JASON system was available.

4. More detailed information should be collected andreported concerning system, subsystem, and component avail-ability and the reason for periods of unavailability.

5. The office Automation System does not meet the needs ofall divisions that use it. Centralized planning and controlfor this system does not exist, resulting in operationalproblems and potential constraints to future system per-formance.

6. Tape management procedures were generally adequate, butcertain occurrences of data losses due to the inadvertentscratching of tapes indicated that some small changes areneeded in this area.

7. Most batch activities are executed during day shiftprocessing periods when resource consumption of on-linesystems is greatest.

8. Most activities executed require small amounts ofprocessor, memory, and I/O resources.

4 45


9. Activities requiring two tapes imposed a significantlyhigher load on the tape subsystem than-did activitiesrequesting other amounts of tape drive resources.

10. The number of aborted activities is rnot a major problemon the H-6060 system. However, almost 6% of processingresources and over 10% of memory resources are lost to

aborted activities.

11. Most NSF users load (link-edit) production programs eachtime they are executed, instead of using load modules (H* or** files) and production libraries.

12. TSS response loads were moderate. Most NSF usersexecuted TSS subsystems that were fewer than 21K words insize.

13. The largest amount of TSS processor use was consumed byusers of the BASIC subsystem. Users compiled programs eachtime they were used instead of using more efficient processingtechniques.

14. Text editor functions of the TSS system are largely usedfor document processing. The Text Editor subsystem and theRUNY subsystem (under which Office Automation programs wererun) accounted for almost 45% of all characters transmittedby the TSS system.

15. Peak demand for system resources by the JASON on-linesystem and the TSS system occur concurrently.

16. System resources are not a constraint to additionalworkload during evening, night, and weekend processingperiods.

17. CPU resources are almost fully used during day shiftprocessing periods. These levels do not pose a significantconstraint to respons. levels of on-line systems at currentuser activity levels.

18. Memory resources are almost fully used during day shiftprocessing periods. This level of utilization is not aconstraint to system throughput for most batch jobs. However,some jobs exhibit significant swap activity which adds to jobelongation and system overhead use of processing and diskresources.

19. Disk channel availability is not a constraint to systemperformance.

46

(ATTACHMENT VI ATTAChMENT VI

20. Disk controller utilization is not a constraint tosystem performance.

21. System file placement was not a major constraint tosystem performance. However, some small changes in the waysystem files are allocated to disk devices could yield someperformance improvements.

22. Seek activity on some JASON disk packs may vary enoughto degrade response times of the JASON system.

23. System overhead can be reduced by placing system modules.MFSO3 and .MFS09 permanently in memory and implementing SSACache Memory to red _ce system module loads from disk.

24. Tape channel configuration is not a constraint to systemperformance.

25. The present tape controller is not a constraint tosystem performance.

26. The number of tape drives configured on the system isI generally adequate for processing the current NSF workload.However, a constant demand exists for additional taperesources. This demand may be reduced somewh&L. by changesin memory swap rates and job control language procedures toreduce the number of tapes allocated but idle.

27. The Datanet-355 front-end processor is not a constraintHto on-line response times.

28. JASON on-line response times, at current activitylevels, are more a function of the interaction of JASONprocedures than hardware resource availability.

29. TSS response times are good. Comments about slow TSSresponse seem to be more a factor of the type of functionithe user is doing than overall TSS performance. Occasionswhen system processing is interrupted (disk errors, accountingfile problems, line problems, etc.) are being interpreted byusers as poor TSS or JASON response because these users arenot made aware of the occurrance of system problems.Conversely, users experiencing problems are not alwaysmaking computer operations personnel aware of these problems.

44


V. RECOMMENDATIONS

A. OPERATIONAL PROCEDURES

FEDSIM recommends that the policies and procedures forsystem accounting be enhanced to provide NSF personnel withbetter information with which to manage computer and man-power resources. FEDSIM identified three main areas wheresystem accounting could be improved:

a. Accounting for User Activity. FEDSIM recommendsthat the $IDENT card format be modified and expanded toinclude detailed information on the submitting agency,project code, test or production status, and type ofsystem (JASON, TSS, or normal batch, etc). Figure V-1outlines a sample $IDENT card format.

$ IDENT AAAAAABBBB/CD/EEEEEE/,FFFFFFFFF,GGG...

AAAAAA = Office IdentificationBBBB = Employee NumberC = System Code

JzJASONL=TSSB=BATCH, etc.

D Production CodeP=ProductionT=TestS=Special (one time runs, etc.)

EEEEEE = Program Identification (JASON MODULE, etc.)FFFFFFFFF = Batch Distribution BannerGGG... = Free Field for Remainder of Card

SAMPLE $ IDENT CARD FORMAT

FIGURE V-1

FEDSIM also recommends that the "ACCOUNT?" questionbe implemented for TSS users and Fields A through E ofFigure V-1 be required of all workload submitted underTSS. The .MGNAT module (system input validity check),the .MSCAN (user scheduling), and .MSCN1 (control

48

'4


card examination) modules should be used in conjunctionwith each other to cross-reference the office identi-fication, program identification, and user identificationof each batch program and TSS user to ensure valid dataare being collected by the accounting system. Thiscross-reference check with the user identification(which is used in conjunction with a password) willprovide security for codes used on the $IDENT card yetwill not require important data on the $IDENT card to beoverlayed and lost for later accounting. FEDSIM recommendsthat, once enhanced system accounting has been implemented,users and their management be made aware of their ownactivity levels and resource consumption on a monthlybasis. The knowledge gained from monthly reportsprovides management with the ability to track workloadgrowth, aid capacity planning, identify candidates forprogram optimization, and "police" the use of accountingcodes and parameters.

b. Accounting for System Availability. FEDSIM recommendsthat the current method of tracking system, TSS, andJASON availability be expanded to include specific timeswhen outages are experienced, to account for componentavailability (front-end processors, etc.), and to tracethe causes for these outages. FEDSIM recommends thatNSF personnel design and implement a form (preferablywith sequence numbers for accountability) for computeroperations personnel to use when problems arise. Thisform will provide a record of system problems and serveas a tracer for subsequent disposition of problems.Figure V-2 is a sample of the type of form recommended.

Expanded accountability will ensure that eachproblem is noted and will allow NSF management to commitpersonnel and computer resources more effectively tosolve those problems identified as causing significantinconvenience to users.

FEDSIM also recommends that NSF personnel closelyevaluate data provided by test and diagnostic softwareto trace problems on tape and disk devices and monitorif and when these devices are becoming significantconstraints to system performance.

FEDSIM also recommends that information concerningsystem problems be made available to users so that theymay possibly relate their experiences (bad response, badturnaround, etc.) to system malfunctions.

49


DATE TIME UP TIME DOWN

OPERATOR LOG NUMBER

MAJOR PROBLEM AREA

/- H-6060 / D-355 /-7 MCS-30

.. DISK /___7 TAPE /-7 MEMORY

P7 JASON /-7 TSS -7 ACCOUNTING

/-7 OTHER

DISK PACK OR TAPE REEL #

SYMPTOMS:

DUMP? /-7YES /_NO TAPE #

PRINTED? /-7 YES /7 NO

ANALYSIS:

DISPOSITION:

SAMPLE SYSTEM PROBLEM TRACKING FORM

FIGURE V-2

50


%

c. Collection and Retention of Accounting Data. FEDSIM

recommends that NSF personnel implement a procedure toensure that accounting data are properly collected andmaintained. Table V-1 shows the accounting record typesnormally produced by GCOS. Figure V-3 presents asample accounting merge procedure designed to collect

and retain accounting data. This procedure will retaincertain record types only as long as necessary to meetthe information needs of NSF computer operations. Forexample, accounting records such as Type 6 (system dumprecords) are important for the analysis of TSS failures,etc., but are not needed for workload purposes. These

RECCRD TYPE RECCXD DESCRIPTION

1 Activity Accounting Record2 SYSOUT Report Writer Accounting Record3 I/O Error Accounting Record4 TcLTS Statistical Record5 System Input Accounting Record6 Program Durp or Operator SNAP Accounting Record7 Time Sharing System Aconting Record8 Activity Accounting Record overflow9 Restart Accounting Record

10 Transaction Processing Syse Statistical Record11 Reserved

12 Fperator-initiated Record13 Reserved14 Time Sharing Resource Data Recordis File Management Supervisor Accounting Record16 FMS Master Function Audit Record17 FMS User Function Audit Record18 FMS Allocate/Deallocate of Secured File Audit Record19 Time Sharing Test and Measurement Record20-100 Reserved for GCOS

101-OVER User-Defined Record Types

GCOS ACCOUNTING RECORDS

TABLE V-1

51

4--~


records are not merged into the accounting data but areprocessed at the same time the daily merge is done.Data such as FMS audit records (Types 16, 17, and 18)are merged into daily accounting tapes so that questionsconcerning file security and access may be answered;yet, these records are deleted by the weekly accountingmerge when they are no longer needed. Retention cyclesoutlined in Figure V-3 are designed to provide adequatebackup capability and ensure that data are available toanswer ad hoc inquiries for a reasonable time after datahave been merged into more generalized formats. Theweekly and monthly accounting merge tapes contain onlythose data needed for user accounting and capacitymanagement functions, thereby condensing the volume ofdata on these tapes. FEDSIM has provided NSF personnelwith an accounting merge/extract program that may betailored to accomplish this recommendation.

FEDSIM recommends that the current system schedulingphilosophy on the NSF computer system be altered to reflectpriorities in batch workload and to eliminate possibleperformance problems.

a. Establishment of Operations Class. FEDSIM recom-mends that NSF establish an operatio;ns scheduler queuefor all jobs spawned by computer operations personnel(VIDEO, HEALS, etc.). This class will provide a separatepathway into the system for operations jobs, thuseliminating them from competing with normal batchworkload in the system scheduler queues. Also, theoperations queue will ensure that jobs required by theoperator can become candidates for execution quickly.Disk accounting purge programs should also be scheduledthrough this queue to ensure their prompt execution.

b. Use of JORDER, JCLASS. FEDSIM recommends thatsystem operators alter their procedures for extractingjobs from scheduler queues. The computer operator oftenused the console verb 3SCHED to manually start workloadfrom other scheduler queues. Jobs that required signif-icant amounts of resources were rescheduled into theexpress queue and occupied program numbers normallyreserved for short, quick jobs. FEDSIM recommends thatcomputer operators use the JCLASS console verb to raiseand lower the number of jobs allowed from each class.This procedure will ensure that jobs will run from theclass to which they were assigned. if it is necessary

53

( ATTACHMENT VI ATTACHMENT VI

to run a particular job first, the computer operatorshould use the JORDER verb to reorder the slstem schedulerfile class before allowing a job to execute from thatclass.

c. Design of System Scheduler Classes. FEDSIM recom-mends that NSF personnel modify the system schedulingphilosophy to more closely reflect workload character-istics as outlined in the findings section (Section IV)of this report. Those findings have shown that most NSFbatch activities were of short duration and requiredsmall amounts of memory. Additionally, jobs requestingone tape made little demand on tape channel resources,whereas jobs requesting two tape drives showed signif-icantly more tape channel activity. FEDSIM recommendsthat a differentiation be made between jobs that requestone tape drive and those that request more tape resources.

FEDSIM recommends that the current configuration of theH-6060 be modifed slightly to balance MPC utilization and toreduce the chance of disk errors incapacitating the entiredisk subsystem. Currently, all 26 disk drives are definedon Channel 8 of the IOM with Channels 10, 12, and 14 actingas alternate paths to the disk drives. Although thisconfiguration is not a significant performance contraint,

GCOS builds one linked list for all I/O activity to diskdevices. When disk errors occur, they are linked to thefront of this list and constrain I/O activity for all otherprograms and disk devices. Figure V-4 shows a suggestedconfiguration to reduce this contention. Since most diskactivity is evenly balanced, NSF personnel should determinewhat devices should be on each disk string. One possiblitywould be to segregate TSS swap files from JASON removablepacks to reduce the possibility that these two on-linesystems could conflict with each other if disk errorsoccurred. The configuration change outlined in Figure V-4is a logical change to GCOS and does not require hardwaremodifications.

FEDSIM recommends that the use of the NOFMS option bere-evaluated at NSF. Disk packs DP5 and DP8 were defined tonot contain permanent files. Interviews with NSF personnelindicated that this NOFMS option was used to identify moretemporary file space for system users. GCOS system loadingalgorithims do not take NOFMS options into considerationwhen spreading files among disk devices. The NOFMS optionmay restrict the number of disk packs available to permanent

54


DoISx*191 DISK*191~..NIT-1 3 ITS-1 3

08 10 12 14 PFYSICAL 0W'NES09 11 13 15 LOG3ICAL Cawzj.S

CIOFIGURATIM,

$ ICt+-0 PUB-08,DISK*191,UNITS-13 ...$ ICt4-O PUB-12,D1SK*191,UNITS-13 ...

$ XBAR Icl4-0,PUB-8, PUB-14, PUB-9, PtB-15$ XBAR Iat-0,PB12,PJB-10,PB13,PUB-11

SAMPLE DISK CONFIGURATION

FIGURE V-4

55

ATTACHMENT VI ATrTACHMENT VI

disk files, thus increasing permanent file contention, butwithout guaranteeing that space will be effectively used onthe NOFMS packs. NSF personnel should evaluate whether asignificant number of jobs require the large contiguous diskareas provided by NOFMS packs (for sorting, etc.). If largetemporary disk storage areas are needed infrequently, NSFpersonnel should consider using removable packs for thesejobs. If these areas are not needed, the NOFMS optionsshould be removed to further balance permanent and temporaryfile placement.

FEDSIM recommends that NSF personnel periodically checkto ensure that four-way interlacing is being used for memoryaccesses on the H-6060 computer system. FEDSIM noted thatafter new hardware had been installed on the system, inter-lacing had been turned off. FEDSIM has found in variouscomputer studies, including the previous study at NSF, thatfour-way interlacing makes a significant improvement in theCPU's capability to quickly access memory locations.

FEDSIM recommends that the placement of TSS program andswap files be examined with each software release to ensurethat these files reside on separate disk devices.

FEDSIM noted that NSF, because of its relationships witheducational institutions, has a rapid turnover of personnel.FEDSIM recommends that seminars concerning the proper use of |the system be given to new personnel to ensure that they arecognizant of how their activity will affect system resourceconsumption and availability. Instruction should includeoperational procedures, proper use of JASON and/or TSS,reasonable programming standards, and where to seek assistancefor programming and/or procedural problems.

FEDSIM noted that NSF operational procedures require aSAVE SINCE of the permanent file system to be run threetimes a day. This save processing copies to tape all filesthat have been created or altered since the last SAVE SINCE.FEDSIM regarded this procedure as a possible constraint toon-line performance since, during SAVE processing, the FileManagement Supervisor (FMS) hash bits are closed for theuser master catalog being saved. With only 31 FMS bits,many different USERIDS may use the same hash bit. Whenthese bits are "closed," the ability for users to sign-on toTSS or access files is delayed. FEDSIM recommends that NSFpersonnel track how often a SAVE SINCE is actually used to

56


restore files and consider SAVE SINCE processing once a dayafter prime shift processing periods. If additional SAVESINCE processing is necessary, NSF personnel should evaluatethe need to save user files on an individual user basis.

FEDSIM recommends that tape menagement procedures bemodified to reduce the possibility of inadvertent misuse oftape reels. FEDSIM noted that some tape reels were accidentallyscratched before their release date, although this is not amajor problem. This problem may be reduced by expressingthe retention period on the external tape label as a daterather than the number of days in the retention cycle. Thischange would reduce the possibility of mathematical errorwhen deciding whether or not a tape volume should be scratched.As an additional precaution for JASON journal tapes, abrightly colored label, identifying current JASON journals,could reduce the possibility of inadvertently deletingimportant JASON data. After the retention period on JASONvolumes expires, these colored indicators should be removedbefore releasing JASON tapes. Additionally, FEDSIM recommendsthat tapes should be periodically cleaned and recertified.The use rates of tapes may be found from SCF accountingdata. Each time a tape is used an entry is created in theactivity accounting record. These data may be reduced torecord the frequency of use and the u:er of each tape.

Cleaning and recertification of tapes minimizes tape errors.

FEDSIM recommends that NSF management establish aconiputer management reporting and performance evaluationsection. This section should be responsible for (1) thecollection and maintenance of accounting and software monitordata and (2) management reporting concerning system use and

availability. Additional functions that could be includedwithin t: is section are JASON data base administrationfunctions and approval of software design of batch and on-line applications. Prior approval of application designwill ensure that these applications do not needlessly impactsystem turnaround, respose times, and the availability ofsystem resources.

FEDSIM noted that NSF management expressed great interestin enhanced information with which to manage personnel andcomputer resources. Much of this information currentlyexists in accounting data and software monitor output.Personnel in the management reporting section, with appropriateguidance from NSF management, should use these data tocreate a management reporting system tailored to NSF needs.

57


B. WORKLOAD

FEDSIM recommends that NSF personnel establish responseand turnaround requirements for production users of theH-6060 computer system. These response and turnaroundrequirements can be used to measure the performance andcapacity of the system and to select certain workloads forexecution during non-peak processing periods. Findingsindicated that much batch workload ran during the day shiftwhen on-line systems were active. Identification of batchworkloads that may be deferred to non-prime processingperiods would free day shift processing resources. Findingshave shown that the NSF computer system has excess capacityduring evening and night shift processing periods to processadditional workloads. FEDSIM does not recommend that NSFstop processing batch workloads during the day shift.Findings have shown that peaks and valleys exist in both CPUand memory utilization. Background batch activity serves to"fill in" the valleys in machine utilization, resulting ingreater throughput with little or no degredation in responsetimes. Turnaround and response requirements serve tomeasure the amount and type of batch workload that could beapplied during the day shift before degradation occurs. NSFpersonnel should design day shift system scheduling aroundthe primary philosophy of running priority workloads. Sincemost workload required small amounts of resources, NSFpersonnel should only consider scheduling based on resourceutilization for activities that require unusual resourcerequirements. Long-running, low priority workloads thatrequire little operating system support may prove to bebetter suited to run with on-line systems then many small,short activities that require more operating system activityfor scheduling, allocation, and termination.

FEDSIM recommends that the Office Management System beplaced under centralized control and be modified to moreclosely reflect the activity, response, and turnaroundrequirements of system users. Four main points were iden-tified in interviews with office Management System users:(1) users were required to group Office Management Systeminput because of limitations imposed on the number ofproposals that could be processed by the system, (2) someusers were limited in Office Management System applicationsbecause the number of reviewers required per proposal wasmore than that allowed by the system, (3) users were appre-hensive about using the Office Management System for a large

58


number of proposals since no restart capability was provided,and (4) most Office Management System users favored theability to quickly enter and validate proposal and reviewernumbers, but noted that the letter and form output printedby the system was not needed by users until the next day.

FEDS IM noted from workload findings that the OfficeManagement System also contributed significantly to thenumber of characters transmitted by the TSS system.

FEDSIM recommends that the Office Management System beredesigned into two subsystems. One subsystem would operateon-line under TSS in much the same way as the present systemand would provide on-line entry and validation of proposalnumbers and reviewers. .The other subsystem would replacethe current on-line function of printing large volume outputat low-speed terminals with a batch oriented subsystem.This batch subsystem would remove much Office Managemen tSystem workload from TSS and direct it to high-speed printing

devices with full restart capability.

Figure V-5 shows a suggested implementation of theIOffice Management System. Users interface with a TSS

subsystem to enter proposal and reviewer information anddirect the type of forms to be produced. This TSS subsystemthen generates batch jobs for printing each type of form andnotifies the user of the identification of each job. Thebatch system input program examines these jobs (via .MSCAN iand .MSCN1 to identify special $IDENT card identifiers oraspecial $ FORMS card) and places them on system schedulerqueues corresponding to the type of output desired (labels,special forms, etc.) .

During non-prime processing periods, the computeroperator can mount special printer forms and then run jobsfrom these special scheduler queues. Alternately, since theJob Control Language for these jobs is pre-determined withthe TSS subsystem, output from these programs may be assignedto fictitious terminal identification codes corresponding tothe type of output desired. The operator may then (1) re-direct output for these station codes to on-line printersfor labels and forms and (2) purge reviewer letter output totape for printing on the XEROX high-speed printer. Undersoftware control, this printer is capable of supplyingletterheads, selectable printing fonts, and extremely high-quality output. This design of the office Management Systemplaces responsibility for system output within computer

59


miSE 1:

rn~

SAMPLEOFFIC MANAEMENSYSTMMPEENAM

4 FIGURESE-M

60S

SCREDULD

.[.-XE1*'


operations instead of with the computer user. Also, thisdesign significantly reduces the printing load on the TSSsystem. FEDSIM feels that implementation of this recom-mendation would allow significant increases in OfficeManagement System use, as requested by NSF personnel, with-out serious response and turnaround degradations to on-linesystem users.

FEDSIM recommends that batch users maintain productionprograms on system loadable user libraries instead of onobject form. Findings have shown that almost all programexecutions are accomplished through GELOAD, the systemloader. Loading requires extra processing resources eachtime programs are run to link-edit each program. Additionally,new operating system releases may cause problems with programexecutions, since a source module compiled under one release(kept in object form) and then link-edited with a systemlibrary from a new release may yield unpredictable results.Use of user libraries would also enhance system accountingsince the activity identification would not be GELOAD butrather the actual name of the production program. Care inassigning unique production program names could allow theresource use and run frequency of individual programs to bemonitored.

FEDSIM recommends that NSF personnel investigate thelarge number of BASIC programs run under the TSS system. Ifthese programs are used repeatedly, they should be re-coded(e.g., into FORTRAN), and be made into loadable TSS sub-systems. Currently, FEDSIM feels that much CPU and I/O timeis being used by the TSS system to compile BASIC programseach time they are run. As a minimum, BASIC users shoulduse RUN=filename to save the object module generated fromBASIC compilations. Subsequent executions of the programwill require only loading.

FEDSIM recommends that NSF personnel investigate the useof word processing equipment in lieu of using the TSS TextEditor for the creation and maintenance of large documentfiles. Text Editor functions such as "Find String,* whichrequire character-by-character searches of large data files,impose a significant load on TSS processing and I/O resources.When large data files are edited on the H-6060, FEDSIMrecommends that TSS users use "Line" mode to position filepointers close to the page or paragraph to be searched, thenuse "Stringo mode to find the required word or phrase.

61

f4{ ATTACHMENT VI ATTACHMENT VI

FEDSIM recommends that NSF personnel examine the possi-bility of removing large-volume data-entry functions fromcompeting with other JASON users. Although NSF has imple-mented a form of this recommendation by placing certain datain staging areas within the JASON database, data entry usersmust still compete for memory, CPU, and 1/O resources withinthe JASON executive. Through user interviews, PEDSIM foundthat most data are not needed for processing the same daythey are entered into the system. FEDSIM recommends thatdata entry functions be implemented under TSS (requiringlittle or no change to user interface) and be stored in anon-JASON file for later rapid update of the JASON databaseafter the day shift. The JASON data base would then beavailable for processing by batch programs later in theevening and for on-line inquiries the following day.

* Implementation of this recommendation would provide aquicker and more consistent response time for data-entryfunctions.

FEDSIM recommends that the JASON system be modified toprovide better accountability for resources used and systemresponse times. FEDSIM feels that the current informationreported by the JASON system concerning user activity (useridentification, procedure name, terminal identification,date, start time, and stop time) is an excellent start forI tracking JASON activity but should be expanded to includethe accumulated processor time, number and type of data basecalls, number of terminal characters sent or received,

amount of memory used, and, if possible, keywords used

within the procedure. The information can be used to identifyprocedures that use large amounts of system resources forpossible re-evaluation and optimization. FEDSIM also recommendsthat a special accounting record be implemented within JASONto track system response time. This accounting record wouldbe written each time a response request was received by theJASON Executive and each time a response was transmittedback to the user. Figure V-6 presents a sample accountingrecord format. Since this accountability generates extraoverhead, the record should be optional so that it may beAturned on or off as needed to trace JASON response. Datareduction software should be written to sort records basedon transaction numbers and to determine response time on atransaction-by-transaction basis. Additionally, when responsedegradations do occur, this record provides detailed dataabout what transactions were in process at the time of thedegradation and can help NSF personnel identify functionsthat adversely affect JASON response.

62

(ATTACHMENT VI ATTACHMENT VI

0 RECORD LENGTH RECORD TYPE

1 DATE

2 TIME

3 TRANSACTION NUMBER

4 AdCOUNT CODE

s PROCEDURE TERMINAL

USER IDENTIFICATION

3 KEYWORDSo ETC.

9ACCUMULATED DATA BASE CALLS,

RESOURCES# ETC.10

SAMPLE JASON RESPONSE TIME ACCOUNTING RECORD

FIGURE V-6

63

art r.


C. EFFECTS OF WORKL~OAD ON SYSTEM RESOURCES

FEDSIM recommends that NSF personnel periodically addthe MCOUNT patch to the dispatcher to count module loads.Highly used re-entrant modules should be placed in the HardCore Monitor, thus eliminating large amounts of disk activity.FEDSIM also recommends that the SSA Cache Memory be maintainedon the H-6 060 computer system to further reduce the need foracquiring system modules from disk.

FEDSIM recommends that the current patches concerningPriority B dispatching be maintained. These patches givethe JASON on-line system first priority with a Priority Bdispatch interval of 224 milliseconds and TSS second prioritywith a dispatch interval of 96 milliseconds. FEDSIM notedthat, at current user activity levels, JASON response ismore a function of the interaction between JASON proceduresthan of CPU resource availability. System loading experiment#2 showed that overall JASON response could be adverselyaffected by dispatching parameters. FEDSI4 identifiescertain changes within the dispatcher that would ensurepriority dispatching to JASON when needed. The followingI patch to the GCOS dispatcher will allow Priority B programsto accumulate more time in Priority B status than is usuallyallowed through normal system parameter settings:

10132 OCTAL 000013735200 ALS 11 .MDISP

This patch will reduce the time quantum subtracted from theamount of Priority B dispatch time alloted to Priority Bprograms. NSF personnel may want to alter the contents ofthis patch. Each bit shifted left less than 12 bits dividesthe current quantum used by one-half. This patch willensure that the JASON on-line system receives adequate-Priority B status before relinquishing to non-Priority Bprograms.

FEDSIM identified that under release 3/1, TSS subsystemsare dispatched ahead of Priority B programs. Depending onthe savings realized by recommendations concerning TSSsubsystem activity, the following patch to the TSS executivecan increase the amount of CPU resources available to theJASON system, yet not seriously degrade TSS response:

2626 OCTAL 000001000000 1 PROC SUBDISPATCH .MTIMS

This patch restricts TSS subsystem execution to only one4 processor instead of allowing TSS subsystems to consume the

resources of both processors simultaneously. With some

64

( ATTACHMIENT VI ATTACHMENT VI

minor differences imposed by system release 3/1, this patchwill cause the GCOS dispatcher to allocate processingresources to programs in a similar manner as release 2/H.

These patches were developed for the GCOS 3/1 operatingsystem in use at NSF during the measurement study and assuch are considered to be correct. NSF personnel shouldfollow good patch implementation procedures when installingthese patches. Use of these patches on subsequent operatingsystem releases should be verified by NSF personnel.

FEDSI11 recommends that the system trace routine beturned off, except when these traces are necessary toanalyze recurring system failures. FEDSIM has found throughprevious Honeywell studies that running with full tracecapability subtracts significant processing resources fromuser workloads. (Each trace requires an average of 200microseconds to process.) System traces may be turned onwhen system problems occur, although examination of thefirst failure will lack necessary system traces. Alternately,FEDS IM has identified certain traces that are normally usedwhen system problems occur yet result in minimum impact onCPU resources. These trace tapes are 0, 1, 2, 4, 6, 7, 10,11, 13, 14, 15, 16, 20, 21, 22, 23, 32, 33, 37, 46, 50, 51,52, 61, 65, 70, 74, and 77.

FEDSIM recommends that the activity during test versionsof JASON be monitored to see if the *4.me periods requiredfor JASON testing can be consolidated. Currently, testversions of JASON execute twice daily. FEDSIM observed onnumerous occasions that very little activity occured duringthese test periods, yet the test versions of JASON imposed asignificant bottleneck on memory resources, sometimes forperiods of over one hour.

Consolidation of test times would make more productiveuse of test versions of JASON and free significant systemresources at other times for normal batch workload.

(116640)

65

* U.S. GOVERNMENT PRINTIN~G OFFICE :1980-620-38f6/122

THE NATIONAL SCIENCE FOUNDATION'S MANAGEMENT …

Documents

Transcript of THE NATIONAL SCIENCE FOUNDATION'S MANAGEMENT …