Social Security: A-02-00-10019%20Final

8/14/2019 Social Security: A-02-00-10019%20Final

http://slidepdf.com/reader/full/social-security-a-02-00-1001920final 1/30

OFFICE OFTHE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE MEASURE REVIEW:RELIABILITY OF THE DATA

USED TO REPORT800-NUMBER PERFORMANCE

September 2001 A-02-00-10019

AUDIT REPORT



��

��

��

��

��

��

��

��

Mission

We improve SSA programs and operations and protect them against fraud, waste,and abuse by conducting independent and objective audits, evaluations, andinvestigations. We provide timely, useful, and reliable information and advice toAdministration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units,called the Office of Inspector General (OIG). The mission of the OIG, as spelledout in the Act, is to:

� Conduct and supervise independent and objective audits andinvestigations relating to agency programs and operations.

� Promote economy, effectiveness, and efficiency within the agency.� Prevent and detect fraud, waste, and abuse in agency programs and

operations.� Review and make recommendations regarding existing and proposed

legislation and regulations relating to agency programs and operations.� Keep the agency head and the Congress fully and currently informed of

problems in agency programs and operations.

To ensure objectivity, the IG Act empowers the IG with:

� Independence to determine what reviews to perform.� Access to all information necessary for the reviews.� Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations,we are agents of positive change striving for continuous improvement in theSocial Security Administration's programs, operations, and management and in

our own office.



.".\. SECuOu~W/'US~~

\ 1II1II1~"""IST1\.t'

SOOAL SECURITY

Office of the Inspector General

1 8 2001

Refer To:

Larry G. Massanari

Acting Commissioner

of Social Security

Inspector General

Performance Measure Review: Reliability of the Data Used to Report 800-Number

Performance (A-02-00-10019)

The Government Performance and Results Act (GPRA) of 19931 requires the Social

Security Administration (SSA) to develop performance indicators that assess therelevant service levels and outcomes of each program activity .GPRA also requires

disclosure of the processes used to verify and validate the measured values used to

report on program performance. SSA is committed to ensuring the importance of

verifying and validating performance measures, and the Office of the Inspector General

audits of the performance measures are a means to achieve this. The objective of this

audit was to determine the reliability of the data used by SSA in Fiscal Year (FY) 1999

for the following GPRA performance indicators and to assess how the indicators will

assist SSA to achieve world class service:

FY 1999 Performance

Percent of callers who successfully

access the BOO-number within

5 minutes of their first call 95.0 percent 95.8 percent

Percent of callers who get through to

the 800-number on their first attempt 90.0 percent 92.9 percent

800-number telephone calls handled 55.5 million 58.8 million

The within "5-minute" access measures the number of call attempts from a specified

telephone number, referred to as individual callers, that select either the automated or

live agent option. This does not jnclude callers who reach the 800-number but

terminate the call prior to making a selection.

The "first attempt" measure refers to the number of call attempts that reach either the

automated or live agent option on the first attempted call of the day, or any subsequent

attempt after a previously successful call.

1 Public Law No.103-62.



The “calls handled” measure refers to the number of call attempts that select either theautomated menu or live agent option and complete their business prior to terminatingthe call. This does not include callers who reach the 800-number but terminate the calleither prior to selecting an option or after waiting in queue for an agent.

BACKGROUND

SSA implemented a national toll-free 800-number telephone service (1-800-SSA-1213)in 1989 as an alternative to field office contact for customers to use when conductingbasic Social Security business. Today, there are 3,000 teleservice representatives, alsoknown as live agents, who staff the 800-number system, which is comprised of37 teleservice centers (TSC). In FY 1999, the 800-number received over 78 millioncalls, with an average of 1.5 million calls per week.

After accessing the 800-number, callers are prompted to select an option, in English or

Spanish and to either select from an automated menu or to select to speak with a liveagent. The live agent option is available on business days between the hours of 7 AMand 7 PM local time, and automated service is available daily 24 hours a day. Callersusing the automated service encounter multiple menus. During FY 1999, approximately70 percent of the callers chose to speak with a live agent, while 30 percent chose theautomated menu.

A call to the 800-number is routed from the caller’s local network to the AT&T network,2

that then delivers it to SSA's 800-number network. If a caller selects the live agentoption, the call is routed to one of SSA’s 22 Automatic Call Distributors (ACDs). TheACDs operate like computerized switchboards to steer and deliver incoming calls

among the 37 TSCs, depending on the day of the week, time of day, and distribution ofcall volume among the various TSCs. When a live agent is not available to take the callat a TSC, the ACD puts the call in queue.

RESULTS OF REVIEW

While we replicated the "5-minute" and "first attempt" access rates, we cannot opine ontheir reliability because limitations in the source data do not allow us to determinewhether SSA is measuring the performance it intends. Specifically, the data does notidentify all individual callers, which is how SSA measures the access rates. While wereplicated the “calls handled” measure, we found errors in the measurement process

that could affect its reliability. We found SSA lacks formal documentation of theprocesses used to collect, analyze, and report performance information. Further, wefound weaknesses in the access controls applicable to processing and maintaining the800-number performance measures. We also believe that the definitions used for the

2 AT&T was the contract carrier for all 800-number service through March 2000. In April 2000, SSAbegan a conversion to MCI as the carrier.

2



800-number performance measures tend to emphasize access, as opposed to service,which would better address SSA’s goal to achieve world class service.

DATA LIMITATIONS HAVE AN UNKNOWN EFFECT UPON 800-NUMBER ACCESS MEASURES

SSA relies upon the Automatic Number Identification (ANI) file, which is the detaileddaily record of all calls made to SSA’s 800-number provided by AT&T, to calculate the"5-minute" and "first attempt" access performance measures. As shown in Appendix D,SSA receives this file daily, and applies daily and monthly data processing applicationsto develop the access rate measures. Separate systems subsequently capture andreport data on calls handled in the automated and live agent options.

The Office of Information Management (OIM) receives the AT&T ANI file of rawtelephone call data, and forwards a copy to the Office of Telecommunications andSystems Operations (OTSO). OTSO performs daily and monthly data processing

routines to produce the performance measurements. In its daily operations, OTSOedits out two types of call records: (1) calls that hang up after reaching SSA’s800-number up-front network prompt and (2) calls from non-digital local exchangecompanies in which the ANI system can not read or record the entire originatingtelephone number. These calls are edited out because they do not allow SSA to trackthe experience of individual callers, a condition necessary to the calculation of two ofSSA’s 800-number performance measures. After completing the edit checks, OTSOcounts and sorts the remaining call records according to various call dispositioncategories and calculates the daily "5-minute" and "first attempt" performancemeasures. OTSO also calculates the "5-minute" and "first attempt" performancemeasures twice in a monthly process: once for the current month and again

cumulatively for the FY to date. OTSO maintains summary-level files of the dailyperformance measure counts and access rates and deletes the original AT&T ANIrecord.

Applying the automated processing routines used by SSA to calculate the "5-minute"and "first-time" access measures, we replicated the reported FY 1999 measures.However, we found that limitations in the AT&T file prevent accurately measuring the"first attempt" and "5-minute" access rate performance measures. 3 Specifically, whileSSA has defined the access measures to be the experience of unique call attempts,referred to as individual callers, telecommunications architecture limits the ability toidentify all individual callers. These limitations occur with calls made from certain PBX4

systems and from nonequal5

access areas.

3 These data limitations continue to exist with MCI.4 PBX refers to a type of telephone equipment used in many commercial establishments that directsoutgoing calls.5 Nonequal access exchanges cannot report the originating number on the AT&T record, but rather reportit as an area code and seven zeroes. This prevents identification of an individual caller. AT&T and SSAofficials estimate the number of calls affected to be small and advised that they will be eliminated astechnology improves.

3



PBX Systems Prevent Identification of Individual Calls

Calls made from locations with a PBX system can be recorded in one of three ways onthe AT&T record. All outgoing calls can be assigned:

• one general number,

• random numbers, or

• the individual number from which the call was made.

The first two scenarios prevent the capturing and recording of a caller’s individualextension, and thus the identification of an individual caller. Since SSA’smeasurements require identification of individual callers to track "first attempt" and"5-minute" access, these limitations in the recording of originating numbers can affect

the reliability of the reported measures. As shown in Figure 1, the inability to identify allindividual callers could either overstate or understate actual reported performance.

Figure 1Potential Effect of Calls from PBX-Based Telephone Systems

upon the Access MeasurementsMethod of Call

Routing Through the PBXPotential Effect Measures

Affected

Single Lead Number:

Different calls from differenttelephone lines served by thePBX system are recorded asone unique number on theAT&T file.

� Over reports the number of successful

calls because a caller within 5-minutesof a previously unsuccessful callercould be classified as successful.

� Underreports the number ofsuccessful calls to the 800-numberbecause one successful first attemptwill prevent any subsequent call fromthe same number within a 24-hourperiod being classified successful.

"5-minute"

access

"first attempt"access

Multiple Numbers:

Different calls from the sametelephone served by the PBXsystem can be recorded asdifferent unique telephonenumbers on the AT&T file.

� Can either over or underreport the

number of successful calls to the 800-number depending upon the numberof attempts made by callers beforesuccessfully reaching SSA.

"5-minute" and

"first attempt"access

As Figure 1 shows, the “5-minute” access rate can be overstated if a call is routedthrough a single lead number because a caller who successfully accesses the number

4



within five minutes of another caller’s unsuccessful access would be counted as asuccessful call. At the same time, if calls are made from a location that assigns thesame number to all calls, the first caller from that location will be identified in order todetermine “first attempt” access, and subsequent callers from that location will not berecognized as individual callers. Consequently, if the first caller successfully reaches

the 800-number on the first attempt, any other callers assigned that number for that daywill not be categorized as a successful “first attempt” caller. The effect of this would beto understate the “first attempt” access rate.

As Figure 1 also shows, when calls are assigned random numbers, the effect could beto inflate or deflate reported performance. For instance, a caller may unsuccessfullyaccess the 800-number on the first nine attempts, but place a successful call on thetenth attempt 10 minutes later. If each attempt were assigned a different originatingnumber, the tenth call would be over reported as a successful “5 minute” access call.Then, if a second caller successfully accessed the 800-number on the caller’s firstattempt, assuming no other calls for the day, the successful “first attempt” rate for the

day would be underreported. The understatement occurs since SSA would calculatethe “first attempt” rate as 2 successful first attempts out of 11 unique callers, or18 percent, rather than correctly measuring 1 successful first attempt out of 2 uniquecallers, or 50 percent.

SSA officials did not believe that such calls would materially affect the performancemeasures, but neither SSA nor AT&T could estimate the extent to which calls originatefrom PBX systems. Consequently, we could not determine the effect upon theperformance measure.

Nonequal Access Exchange Areas Prevent Identification of Individual Calls

Calls made from nonequal access exchanges are not identified on the ANI file as acomplete originating telephone number. Therefore, these calls cannot be associatedwith an individual caller and are excluded from the access performance measurecalculations. While the source data does not permit identifying originating numbers orindividual callers to be included in the calculations of the access measures, these callsdo represent callers that are served. Consequently, excluding them could cause anunderreporting of actual performance. SSA staff estimated that these types of callsrepresent about five percent of all calls. We were unable to determine the effect ofincluding these calls in the measurement calculation.

“Calls Handled” Calculation Errors Could AffectReliability of the Performance Measure

“Calls handled” represents a workload measure that is derived monthly through a seriesof manual and computerized calculations using multiple data sources. We tested theprocessing routines that produce the “calls handled” measure, and replicated theprocess SSA used to report the FY 1999 measure. However, we found errors in the

5



processing routine used to calculate the “calls handled” measure that could affect itsreliability.The ANI file records all calls to the 800-number, and identifies whether the callerselected either the automated or live agent menu, or terminated the call prior toselecting an option. Callers choosing the automated menu are tracked daily through the

Automated Service Report system, and monthly through the Plasa Reports.6Customers who choose the live agent menu are tracked through the Automated CallDistributor system and these calls are processed daily in the Handled Report andmonthly in the FTS 2000 Usage Cost Reports. Multiple automated and manualprocesses that use various data sources are used to develop calls handled by live agents and in the automated menu system, which are combined to report the callshandled measure. As shown in Figure 2, we found errors in these processing routines:

Figure 2Effect of Processing Errors on “Calls Handled” Performance Measure

Processing Error Effect ofthe Processing

ErrorCalls received in the automated menusystem as reported in the Plasa Reportinclude calls that encountered a busy signal.

Overstates the “calls handled”measure

Calls abandoned in the automated menusystem as reported in the Handled Reportreflect callers that may have selected morethan one option within the automated menu.

This inflated number is subtracted fromactual calls received on the Plasa Report.

Understates the “calls handled”measure

We determined that the cumulative effect of the latter two errors understated theFY 1999 “calls handled” measure by approximately 100,000 calls, or less than onepercent. While the effect of these processing errors appears to be minimal during theperiod audited, the effect of the errors on future periods cannot be determined.SSA LACKS SUFFICIENT DOCUMENTATION OF THE PERFORMANCEMEASURE PROCESS

We found that SSA lacked sufficient documentation of the processes used to collect,analyze, and report performance indicator data. Further, complete documentationnecessary to recreate the performance measure was not available. Office ofManagement and Budget (OMB) Circular A-123, Management Accountability and

6The Plasa Report is generated directly from AT&T and gives SSA data on call volume from each of its

reporting sites on a daily, monthly, and quarterly basis. Information is provided by the hour and includesoriginating calls, successful calls, and call duration.

6



Control , states that documentation for transactions, management controls, and othersignificant events must be clear and readily available for examination. Consequently,agencies should establish a clear methodology for verifying the underlying data andretain the appropriate documentation to enable an audit of the methodology thatsupports the performance measures.

SSA did not have a comprehensive process map documenting the flow of performancemeasure data from the receipt of original, unedited 800-number data from an externalsource, through SSA edit checks and sorts, to the final calculation and reporting ofperformance measures. At our request, SSA documented the processes by which theperformance measure data is collected, analyzed, and reported. However,documenting the process was not only time consuming, it also disclosed that therationale for the steps in the process was not always known or understood. Theseprocesses are depicted in Appendices D and E.

We also found that two different offices within SSA develop 800-number performance

information. OTSO produces the GPRA performance measures, and OIM uses thesame call data to produce performance data for internal reports. OTSO and OIMperform different edit checks to the data to develop their statistics, which could result ininconsistent information. For instance, OIM deletes calls recorded as complete with azero call duration on the basis that they represent an illogical record. However, OTSOincludes these records in its performance measure calculations. We also found thatOTSO had not always accounted for all automated menu calls when developing the“calls handled” measure. For instance, during the month of January 2000, over60 percent of the calls were erroneously attributed to the live agent option rather thanthe automated menu. While this did not affect the overall “calls handled” performancemeasure, which relies upon the monthly Plasa Report, it did distort daily managementinformation on the distribution of 800-number workload between live agent andautomated menu options. Additionally, the number of calls attributed daily to theautomated menu is overstated because OTSO performs manual adjustments toincrease the number of calls received by certain submenu options rather than simplyreporting the number of calls. This has the effect of counting a call more than once.

We also found that OTSO erroneously calculates calls abandoned in the live agentmenu when reporting monthly data on the Commissioner's Tracking Report. The callsabandoned in the live agent option is taken from the FTS 2000 Usage Cost Report,which includes calls that are abandoned in the automated menu. Consequently, callsabandoned in the live agent menu are overstated. We determined that calls abandonedin the live agent menu were overstated by 298,333 calls, or 6 percent, in FY 1999.

After OTSO calculates the daily performance measures, the original data of individualcalls to the 800-number is deleted, and only summary counts are maintained fordeveloping subsequent year-to-date statistics. OIM maintains original call files for themost recent 30 to 60-day period, after which previous files are deleted. Consequently, acomplete audit trail is not maintained to allow recalculation of the yearly performance

7



measures. SSA staff did not know why the two units were producing similarperformance information, nor why different edit checks were applied.

WEAKNESSES EXIST IN ACCESS CONTROLS OVER PERFORMANCEMEASURE PROCESSING

OMB Circular A-127, Financial Management Systems, requires that a system of internalcontrols be established to ensure that data used to produce reports is reliable. Wefound that controls over access to 800-number program files and report listings wereweak. SSA stores the files under a dollar sign user identification number that allowsunlimited access to the files and sufficient authority for the average user to modify ordelete key programs or reported measures without detection. While our field work didnot disclose that any unauthorized changes or deletions had occurred, we believe SSAshould restrict access to the files to reduce the risk of accidental modification ordeletion, and help ensure the validity and completeness of the performance measure.

PERFORMANCE MEASURES EMPHASIZE ACCESS RATHER THANSERVICE

Figure 3 shows that during FY 1999, SSA reported that 78.7 million calls were placed tothe 800-number, of which 6.9 million callers encountered a busy signal prior to reachingthe 800-number prompt. Of those making a selection, 13 million terminated the callprior to being served, and 58.8 million completed their business.

Figure 3Disposition of Calls to the 800-Number

Disposition of CallsFY 1999

Total calls(million)

Percent

Reached the 800-number 78.7 100.0Encountered a busy signal 6.9 8.8Discontinued subsequent to beingconnected to either the live agent orautomated option

13.0 16.5

Handled to completion 58.8 74.7

The two access performance measures focus on whether the customer is connected to

one of the two menu options, which may or may not result in actually being served. Acall meets the performance measure if the caller chooses a live agent or the automatedmenu, not whether business was conducted or even completed to the customer'ssatisfaction. No consideration is given to how long the caller must wait to be giveninformation via the automated menu or to speak with a live agent, or whether the callersubsequently hangs up prior to being served. The performance measure for “callshandled,” attempts to measure service by only counting callers that actually completetheir business through either the automated or live agent option. However, it does not

8



report on the extent of customer service, such as the length of time that a caller is onhold prior to being served.

The February 1999 Office of Quality Assurance and Assessment’s (OQA) semiannual800-Number Customer Survey7 (OQA Survey) found that 59 percent of callers who

selected a live agent were placed on hold, and that 9 percent of those hung up prior tospeaking to a live agent. While the caller is the one who opts to terminate a call prior tocompletion, the reasons callers terminate a call are not known. Nevertheless, callsterminated prior to completion may negatively impact caller satisfaction. OQA releaseda report in August 2000 on how well the 800-number automated services met customerneeds. Limited information was obtained that showed automated service callers hangup before the message was completed primarily because of a desire to speak with arepresentative.

The OQA Survey also found that caller satisfaction is greatly affected by access and theextent to which a caller’s issue is handled completely. The OQA Survey reported that

overall satisfaction declines as the access rating goes down, and that complete callhandling has a profound effect on the overall satisfaction rating. The Survey found that98 percent of the callers who got through right away and whose call issue was handledcompletely, either by an automated service or a representative, were satisfied withservice overall. However, the Survey further found that callers' satisfaction ratingdropped to less than 60 percent when callers remembered having to call more thanonce and being unable to complete their business.

The September 1999 Social Security Advisory Board report, How the Social Security Administration Can Improve Its Service to the Public , raised the question as to whetherthe public regards the “5-minute” access rate as an acceptable level of service. Thereport further suggested that performance measurement should emphasize the percentof calls that conduct business, as well as the percent that achieve access.

CONCLUSIONS AND RECOMMENDATIONS

While we were able to replicate the measures reported by SSA to report on its800-number performance, limitations in the data do not allow us to conclude as towhether the measures reliably report the performance SSA intends to measure. SSAattempts to measure the experience of each caller, however, SSA does not disclosethat data limitations prevent identification of an individual caller in certain scenarios.The lack of formal documentation of the processes used to collect, analyze, and report

performance information does not comply with OMB guidance in Circular A-123.Weaknesses in access controls applicable to the data processing used to reportperformance do not ensure the integrity of the performance data. We believe that thedefinitions used by SSA for its 800-number performance measures tend to emphasizeaccess, as opposed to service. To ensure greater compliance with GPRA reportingrequirements, we recommend that SSA:

7 OQA surveys 800-number callers semiannually to assess the level of satisfaction based upon callers’recollection of service received on the day calls were selected for survey.

9



1. Disclose in the Accountability Report and other appropriate documents the datalimitation that prevents identifying individual callers from PBXs and nonequal accessexchange areas in the calculation of the access measures;

2. Work with the new contractor for managing the 800-number to quantify the effect

upon the access performance measures of the inability to identify individual callers;

3. Revise the process for calculating calls handled in the automated menu system toexclude calls that receive a busy signal, and to reflect only individual calls, asopposed to options selected;

4. Formally document the processes by which the 800-number performancemeasurements are derived;

5. Ensure that controls exist to prevent unauthorized access to the 800-numberperformance measure data files;

6. Consolidate responsibility for 800-number performance reporting within one office,and ensure that appropriate edit checks and processing routines are applied prior toperformance measure calculation;

7. Revise the process for calculating calls abandoned in automation to reflect individualcalls; and

8. Develop indicators to effectively measure the quality of service received bycustomers served by the 800-number that would more accurately reflect SSA’sperformance goal of achieving world class service.

AGENCY COMMENTS

SSA agreed with five (recommendations 1, 4, 5, 6, and 8) of our eightrecommendations, and reported that action had already been taken as a result of thetransition to a new 800-number contractor to address two others (recommendations 3and 7). In response to recommendation 2, “Work with the new contractor for managingthe 800-number to quantify the effect upon the access performance measures of theinability to identify individual callers,” SSA advised that present technology does notpermit determining the effect, but that SSA will revisit the issue periodically.

OIG RESPONSE

We are pleased that SSA agreed with our recommendations, and has, or plans to,implement all our suggested improvements. Regarding recommendation 2, “…toquantify the effect upon the access performance measures of the inability to identifyindividual callers,” we agree that SSA should periodically assess the extent to whichtechnology would allow identification of individual callers. Until such time as SSA’s

10



technology is able to do so, SSA should disclose the inability to identify individual callers

when reporting the performance measure.

In responding to recommendation 3, "revise the process for calculating calls handled in

th-e automated menu system to exclude calls that receive a busy signal, and to reflect

only individual calls, as opposed to options selected," SSA stated that the reportsproduced by the new BOO-number contractor used to calculate the performance

measures more accurately reflect the number of calls handled. These reports will

identify calls as opposed to options selected. Additionally, the revised method of

calculating calls handled will exclude calls that encountered a busy signal, which were

previously included .

SSA noted that the planned OQA survey on 800-number callers involved only callers

that selected the automated service, and that the report was issued in August 2000.

Our report has been clarified to note this.

/

11



Appendices APPENDIX A - Scope and Methodology

APPENDIX B - Acronyms

APPENDIX C - Agency Comments

APPENDIX D - Flowchart of the 800-Number First-Attempt and 5-MinuteAccess Performance Measurements

APPENDIX E- Flowchart of the 800-Number Calls Handled Performance Measure

APPENDIX F- OIG Contacts and Staff Acknowledgements



Appendix A

Scope and Methodology

As part of the Office of the Inspector General’s (OIG) on-going role to verify andvalidate the Social Security Administration’s (SSA) performance measure reporting,we conducted this audit to assess the reliability of the data used to report thefollowing SSA 800-number performance measurements:

• percent of callers who successfully access the 800-number within 5-minutes oftheir first call;

• percent of callers who get through to the 800-number on their "first attempt", and

•

number of 800-number "calls handled".

To meet our objectives, we documented, based upon analysis of SSA documentsand discussion with officials from the Office of Telephone Services (OTS) and theOffice of Telecommunications and Systems Operations (OTSO), the infrastructure ofthe 800-number system, definitions of the 800-number measurements, and theprocesses for capturing, analyzing, and reporting the data used in themeasurements. To obtain an understanding of the 800-number call routing process,we conducted a walk-through of OTS’ Voice Network Control Center, which isresponsible for overseeing and routing call traffic. To identify the key internalcontrols in the data collection and reporting process, we flow-charted the processthat collects, analyzes, and reports 800-number data from receipt of AT&TAutomatic Number Identification (ANI) data to the reporting of the performancemeasurements. We also reviewed OIG, the U.S. General Accounting Office, and theSocial Security Advisory Board reviews of SSA and other 800-number providers todocument any previously reported weaknesses in 800-number operations andperformance reporting.

We reviewed and confirmed the program language used by SSA to perform the editchecks on the AT&T ANI file, and calculations of the performance measure accessrates. We were unable to identify any prior audits of the AT&T ANI.1 To confirm therecording of originating telephone numbers on the AT&T ANI record, we placed300 calls on one day from 20 different numbers around the country. We then tracedthese calls to that day’s ANI record and confirmed that the call date, call time,originating number, and call disposition had been properly recorded.

Because OTSO does not retain the daily detail call records, but rather maintainssummary level data of its daily processing, we performed tests of OTSO’sprocessing in order to rely upon its summary files. We entered “dummy” records into

1 AT&T is not required to provide audits to establish the reliability of the 800-number ANI data.

A-1



each of the programs on 2 days to ensure that these records were processedcorrectly. Using data from the AT&T ANI file, we performed parallel simulation of thecontrol tests and the edit checks applied in the daily access rate calculationprograms for the period January 1, 2000 through January 31, 2000, and wereplicated the daily measures. These tests provided assurance for us to rely upon

the functioning of the edit routines and access rate calculations. After establishingthe reliability of OTSO’s daily record calculations, we recalculated the “first attempt”and “5-minute” access rates for Fiscal Year (FY) 1999 using OTSO’s files.

We reviewed and confirmed the program language used by SSA to perform both thedaily and the monthly calculation of the calls handled workload. This measurerepresents the monthly sum of calls handled at both the automated and live agentoptions. We performed parallel simulation of the automated and manual edit checksand calculations applied to derive the daily consolidated calls handled number forJanuary 2000. We traced the monthly calls handled number for FY 1999 to thesource documents. To obtain assurance as to the reasonableness of the breakout

between calls handled in each option, we reconciled calls offered in both live agentand automated options to the number of ANI complete calls per the handled report.

Our work was conducted at the OIG New York Field Office and SSA Headquarters inBaltimore, Maryland from November 1999 through October 2000. The entitiesaudited were OTS and OTSO. We conducted our audit in accordance with generallyaccepted government auditing standards, as applicable to a performance audit.

A-2



Appendix B Acronyms

ACD Automatic Call DistributorANI Automatic Number IdentificationAPP Annual Performance PlanFY Fiscal YearGPRA Government Performance and Results ActOIG Office of the Inspector GeneralOIM Office of Information ManagementOMB Office of Management and BudgetOQA Office of Quality Assurance and Performance AssessmentOTS Office of Telephone ServicesOTSO Office of Telecommunications and Systems OperationsSSA Social Security AdministrationTSC Teleservice Center



Appendix C Agency Comments



~" SEC-V.vV- ~-"

~~.?.'/1'usA'\\:

~ IIIIII1~~...NISTV

SOCIAL SECURITY

29300-24-573

Refer To: SlJ-3ate: August 29, 2001

Office of the Inspector General (OIG) Draft Report, "Perfonnance Measure Review: Reliability

of the Data Used to Report 800-Number Perfonnance" (A-O2-00-I0019)-INFORMATION

Our commentsto the subject draft report were releasedto you on August 3, 200I. Subsequently,we received clarifying information from your staff concerning recommendation 5. Therefore,the

attachedcommentsinclude a revision to our commentson that recommendation.

Pleaselet us know if we may be of further assistance.Staff questionsmay be referred to

Dan Sweeneyon extension 51957.

Attachment

SSA Comments



COMMENTS ON THE OFFICE OF INSPECTOR GENERAL (OIG) DRAFT REPORT,

“PERFORMANCE MEASURE REVIEW: RELIABILITY OF THE DATA USED TOREPORT 800-NUMBER PERFORMANCE” (A-02-00-10019)

Recommendation 1

Disclose in the Accountability Report and other appropriate documents the data limitation that

prevents identifying individual callers from PBXs and nonequal access exchange areas in thecalculation of the access measures.

Comment

We agree and will include the recommended disclosure in future reports.

Recommendation 2

Work with the new contractor for managing the 800-number to quantify the effect upon theaccess performance measures of the inability to identify individual callers.

Comment

The suggested technology does not presently exist to put this recommendation in place.

However, we will keep abreast of available technology and will revisit the issue periodically to

determine if technology is available that could assist us in identifying the origin of all calls to the

800 number.

Recommendation 3

Revise the process for calculating calls handled in the automated menu system to exclude calls

that receive a busy signal, and to reflect only individual calls, as opposed to options selected.

Comment

This recommendation was addressed with the transition from carrier AT&T to WorldCom. The

revised Automated Service Report (ASR) counts individual calls to accurately reflect theHandled Calls count. It should be noted that busy calls have never been a part of the automated

system; calls that encounter a busy signal do not enter the automated process.

Recommendation 4

Formally document the processes by which the 800-number performance measurements arederived.

C-1



Comment

We concur. The Division of Integrated Telecommunications Management (DITM) will write

formal documentation after implementation of the Call Center Network Solution. The target date

for completing the performance measurement documentation is late October 2001.

Recommendation 5

Ensure that controls exist to prevent unauthorized access to the 800-number performance

measure data files.

Comment

We concur. We will change the file names for the performance measure data set names fromdollar sign names to Top Secret protected “AIS” data set names. The data set name change will

ensure that the data sets cannot be accessed by any real time or batch request that does not havethe correct Top Secret Profile. We expect to institute the new data set names with the first dailyrun of fiscal year 2002.

Recommendation 6

Consolidate responsibility for 800-number performance reporting within one office, and ensure

that appropriate edit checks and processing routines are applied prior to performance measure

calculation.

Comment

We agree. The Office of Systems will assess the appropriate placement of the responsibility for

800 number performance reporting. The responsibility for reporting will be in place in one

office by the end of the year.

Recommendation 7

Revise the process for calculating calls abandoned in automation to reflect individual calls.

Comment

As with recommendation 3 above, this recommendation was addressed with the transition to our

new carrier, WorldCom. DITM no longer makes manual calculations in the automated

abandonment count, as was the case with AT&T. All calls are counted as an individual call inthe ASR Primary Report, while additional selections are counted in the ASR Subsequent Report.

Each call is counted once.

C-2



Recommendation 8

Develop indicators to effectively measure the quality of service received by customers served by

the 800-number that would more accurately reflect SSA's performance goal of achieving world

class service.

Comment

We agree that indicators should be developed to more accurately reflect our performance goal of

providing world class service. The Office of Operations (DCO) is in the process of developingnew performance metrics that DCO expects to have in place at the beginning of fiscal year 2003.

These new metrics will replace the current performance indicators (i.e., the 5-minute and first

time access goals) and will focus on our responsiveness to answering all calls.

Changes to our automated services based on customer input and our own analysis is a fluid

process, and we are constantly looking for ways to improve customer access and service over thenational 800 number. DCO has conducted focus groups to obtain input on the service provided.On an ongoing basis, DCO and the Office of Quality Assurance and Performance Assessment

(OQA) assess the level of satisfaction with service provided over the national 800 number.

When appropriate, changes are made to address issues/concerns raised by our callers. Forexample, we are in the process of making several major enhancements to the national 800

number automated service scripts.

• On April 13, 2001, Phase I was implemented to separate touch-tone callers from voiceresponse callers and provide distinct touch-tone or voice response call paths. Callers will

hear an English and Spanish touch-tone screening prompt during the main greeting. This

change will help ensure that callers reach the option selected.

• Phase II is scheduled to be implemented by the end of July 2001. This phase will restructurethe current 800 number menus to make them user-friendlier for our callers and add a new

automated option to provide information about the Password/Password Request Codeapplications currently in development. The enhancements planned will provide callers with

the ability to hear the list of options or message again, plus the option to return to the main

menu. The return to main menu feature will provide callers who complete an automated

service application with the opportunity to return to the main menu, where they can selectanother automated service or, during business hours, choose to speak with an agent.

•

Phase III is scheduled to be implemented on October 1, 2001. During this phase, we will addseveral new Password/Password Request Code automated service options which will allowcallers to request, change, replace or block their password, hear informational messages

about the password process and obtain monthly benefit amount information.

C-3



Appendix D

Flowchart of the 800-Number First Attempt

and

“5-Minute” Access Performance Measurements



F l o w C h a r t o f 8 0 0 N u m b e r F i r s t A t t e m p t a n d 5 - M i n u t e P e r f o r m a n c e M e a s u r e m e n t s

D a i l y , O T S O

a c c e s s e s f r o m

m a i n f r a m e u n e d i t e d

A T & T A N I f i l e o f c a l l s

m a d e t o S S A ' s 8 0 0

n u m b e r a n d u p l o a d s

f i l e t o m a i n f r a m e

D a i l y , O T S O a p p l i e s S A S - b a s e d

p r o g r a m s t o p e r f o r m e d i t c h e c k

r o u t i n e s a n

d c a l c u l a t e t h e d a i l y f i r s t

a t t e m p t a

n d f i v e m i n u t e a c c e s s

r a t e s , w h i c

h a r e p r o d u c e d a s o u t p u t

f i l e s . T h e d a i l y p e r f o r m a n c e

m e a s u r e s

a r e s e n t t o t h e O f f i c e o f

T e l e p h o n e

S e r v i c e s a n d p o s t e d o n

t h e 8 0 0 n u m b e r d a i l y M A P r e p o r t o n

S S A ' s i n t r a n e t w e b s i t e

M o n t h l y a n d a t y e a r - e n d O T S O

S A S p r o g r a m , F I R S T 2 D , i s r u n

a g a i n s t o u t p u t f i l e t o c a l c u l a t e t h e

f i r s t a t t e m p t p e r f o r m a n c e m e a s u r e

f o r f i s c a l y e a r t o d a t e .

M o n t h l y a n d a t y e a r - e n d , O T S

O

S A S p r o g r a m , C M O N 2 , i s r u n

a g a i n s t o u t p u t f i l e t o c a l c u l a t e

t h e

5 - m i n u t e p e r f o r m a n c e m e s u r e

f o r

f i s c a l y e a r t o d a t e .

M o n t h l y , F i r s t a t t e m p t a n d

5 - m i n u t e p e r f o r m a n c e m e a s u r e s

a r e p o s t e d f o r t h e c u r r e n t m o n t h

a n d f o r t h e f i s c a l y e a r t o d a t e i n

t h e C o m m i s s i o n e r ' s T r a c k i n g

R e p o r t

F i r s t a t t e m p t a n d

5 - m i n u t e p

e r f o r m a n c e

m e a s u r e s a r e r e p o r t e d i n

S S A ' s

A n n u a l

A c c o u n t a b i l i t y R e p o r t



Appendix E

Flowchart for the Processing of 800-Number

Calls Handled Performance Measure



F L O W C

H A R T O F T H E P R O C E S

S I N G O F T H E 8 0 0 - N U M B E R " C A L

L S H A N D L E D "

P E R F O R M A N C E M E A S U R E

D a i l y , O T S O r e c e i v e s 2 3 A C D

o f c a l l s

r e c e i v e d a n d c a l l s a b a n d o n e d b

y

l i v e

a g e n t s a t 2 3 T S C ' s .

D a i l y , O T S O

p e r f o r m s

m a n u a l e d i t s o n

t h e A C D f i l e s

D a i l y , O T S O r u n s S A S - b a s e d

p r o g r a m s a g a i n s t A C D f i l e s

t o c r e a t e

o u t p u t f i l e s o f c a l l s o f f e r e d , c a l l s

a b a o n d o n e d i n q u e u e , a n

d c a l l s

h a n d l e d a t T S C s .

D a i l y , O T S O r e c e i v e s A u t o m a t e d

S e r v i c e s R e p o r t ( A S R ) f i l e s o f c a l l s

o f f e r e d , c o m p l e t e d , a n d a b a n d o n e d

i n a u t o m a t e d m e n u o p t i o n s .

O T S O p e r f o r m s m a n u a l

c a l c u l a t i o n s t o r e v i s e c a l l s

o f f e r e d a n d c a l l s a b a n d o n e d

i n a u t o m a t i o n .

O T S O m a n u a l l y i n p u t s r e v i s e d

A S R c a l l c o u n t d a t a i n t o

a d a i l y

H a n d l e d p r o g r a m

.

D a i l y , O T S O g e n e r a t e s

a f i l e o f c o m

b i n e d A C D

a n d A S R d

a t a o f c a l l s

o f f e r e d a n d c a l l s

a b a n d o n e d .

M o n t h l y , v a r i o u s p r o c e s s e s a r e

p e r f o r m e d t o p r o d u c e s t a t i s t i c s r e p o r t e d

i n t h e C o m m i s s i o n e r ' s T r a c k i n g

R e p o r t . .

T e l e s e r v i c e C e n t e r s :

M o n t h l y , O T S O r u n s S A S - b a s e d p r o g r a m

t h a t c a l c u l a t e s c a l l s h a n d l e d a n d c a l l s

a b a n d o n e d a t T S C s

, w h i c h i s p r o d u c e d i n

t h e F T S 2 0 0 0

U s e a g e R e p o r t .

A u t o m a t i o n :

:

M o n t h l y , O T S O d e r i v e s c a l l s h a n d l e d b y s u b s t r a c t i n g

c a l l s a b a n d o n e d i n a u t o m a t i o n a s r e p o r t e d i n t h e H a n d l e d

R e p o r t , f r o m t h e c a l l s o f f e r e d , a s r e p o r t e d i n t h e P l a s a

R e p o

r t .

C a l l s h a n d l e

d c o u n t s a t

T e l e s e r v i c e C

e n t e r s a n d i n

a u t o m a t i o n a

r e m a n u a l l y

i n p u t t e d

i n t o t h e

C o m m i s s i o n

e r ' s M o n t h l y

T r a c k i n g

R e p o r t .

C a l l s h a n d l e d c o u n t s a r e

r e p o r t e d i n S S A ' s A n n u a l

A c c o u n t a b i l i t y R e p o r t

O T S O r u n s a d a i l y S

A S - b a s e d p r o g r a m

a g a i n s t c o m b i n e d A S R a n d A C D f i l e t o

d e r i v e a d a i l y c a l l s h a n d l e d c o u n t t h a t i s

p o s t e d o n t h e 8 0 0 - n

u m b e r d a i l y M A P

r e p o r t o n S S A ; s i n t r a n e t w e b s i t e . T h i s

S A S p r o g r a m a l s o s t o r e s t h e d a i l y c a l l s

o f f e r e d a n d a b a n d o n e

d c o u n t s f o r m o n t h l y

p r o c e s s i n g



Appendix F

OIG Contacts and Staff Acknowledgments

OIG Contacts

Frederick C. Nordhoff, Director Financial Management and PerformanceMonitoring Audit Division (410) 966-6676

Timothy Nee, Deputy Director (212) 264-5295

Acknowledgments

In addition to those named above:

John P. Harrison, Senior Auditor

Jeffrey Shapiro, Senior Analyst

Carol Ann Frost, EDP Auditor

Annette DeRito, Program Analyst

For additional copies of this report, please visit our web site at www.ssa.gov/oig orcontact Office of the Inspector General’s Public Affairs Specialist at (410) 966-1375.Refer to Common Identification Number A-02-00-10019.



DISTRIBUTION SCHEDULE

No. of

CaRies

1

10

1

1

3

1

1

Commissioner of Social Security

Management Analysis and Audit Program Support Staff, OFAM

Inspector General

Assistant Inspector General for Investigations

Assistant Inspector General for Executive Operations

Assistant Inspector General for AuditI

Deputy Assistant Inspector General for Audit

Director, Systems Audit Division

Director, Financial Management and Performance Monitoring Audit Division 1

Director, Operational Audit Division 1

Director, Disability Program Audit Division 1

1jrector, Program Benefits Audit Division

Director, General Management Audit Division

25

1

1

1

1

2

1

2

2

1

Issue Area Team Leaders

Income Maintenance Branch, Office of Management and Budget

Chairman, Committee on Ways and Means

Ranking Minority Member, Committee on Ways and Means

Chief of Staff, Committee on Ways and Means

Chairman, Subcommittee on Social Security

Ranking Minority Member, Subcommittee on Social Security

Majority Staff Director, Subcommittee on Social Security

Minority Staff Director, Subcommittee on Social Security

Chairman, Subcommittee on Human Resources

Ranking Minority Member, Subcommittee on Human Resources

Chairman, Committee on Budget, House of Representatives

Ranking Minority Member, Committee on Budget, House of Representatives

Chairman, Committee on Government Reform and Oversight

Ranking Minority Member, Committee on Government Reform and Oversight

Chairman, Committee on Governmental Affairs

Ranking Minority Member, Committee on Governmental Affairs



Page 2

Chairman, Committee on Appropriations, House of Representatives 1

Ranking Minority Member, Committee on Appropriations,

House of Representatives 1

Chairman, Subcommittee on Labor, Health and Human Services, Education

and Related Agencies, Committee on Appropriations,

House of Representatives 1

Ranking Minority Member, Subcommittee on Labor, Health and Human

Services, Education and Related Agencies, Committee on Appropriations,

House of Representatives

Chairman, Committee on Appropriations, U.S. Senate

1

1

Ranking Minority Member, Committee on Appropriations, U.S. Senate 1

Chairman, Subcommittee on Labor, Health and Human Services, Education

and Related Agencies, Committee on Appropriations, U.S. Senate 1

Ranking Minority Member, Subcommittee on Labor, Health and HumanServices, Education and Related Agencies, Committee on Appropriations,

U.S. Senate

Chairman, Committee on Finance

1

1

Ranking Minority Member, Committee on Finance 1

Chairman, Subcommittee on Social Security and Family Policy 1

Ranking Minority Member, Subcommittee on Social Security and Family Policy 1

Chairman, Senate Special Committee on Aging 1

Ranking Minority Member, Senate Special Committee on Aging1

Vice Chairman, Subcommittee on Government Management Information

and Technology 1

President, National Council of Social Security Management Associations,

Incorporated

Treasurer, National Council of Social Security Management Associations,

Incorporated

Social Security Advisory Board

1

1

1

AFGE General Committee

President, Federal Managers Association

9

1

Regional Public Affairs Officer 1

Total 97



Overview of the Office of the Inspector General

Office of Audit

The Office of Audit (OA) conducts comprehensive financial and performance audits of the

Social Security Administration’s (SSA) programs and makes recommendations to ensure thatprogram objectives are achieved effectively and efficiently. Financial audits, required by the

Chief Financial Officers Act of 1990, assess whether SSA’s financial statements fairly present

the Agency’s financial position, results of operations, and cash flow. Performance audits review

the economy, efficiency, and effectiveness of SSA’s programs. OA also conducts short-termmanagement and program evaluations focused on issues of concern to SSA, Congress, and the

general public. Evaluations often focus on identifying and recommending ways to prevent and

minimize program fraud and inefficiency.

Office of Executive Operations

The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG)by providing information resource management; systems security; and the coordination of

budget, procurement, telecommunications, facilities and equipment, and human resources. In

addition, this office is the focal point for the OIG’s strategic planning function and thedevelopment and implementation of performance measures required by the Government

Performance and Results Act. OEO is also responsible for performing internal reviews to ensure

that OIG offices nationwide hold themselves to the same rigorous standards that we expect fromthe Agency, as well as conducting employee investigations within OIG. Finally, OEO

administers OIG’s public affairs, media, and interagency activities and also communicates OIG’s

planned and current activities and their results to the Commissioner and Congress.

Office of Investigations

The Office of Investigations (OI) conducts and coordinates investigative activity related to fraud,waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoing

by applicants, beneficiaries, contractors, physicians, interpreters, representative payees, third

parties, and by SSA employees in the performance of their duties. OI also conducts joint

investigations with other Federal, State, and local law enforcement agencies.

Counsel to the Inspector General

The Counsel to the Inspector General provides legal advice and counsel to the Inspector Generalon various matters, including: 1) statutes, regulations, legislation, and policy directives

governing the administration of SSA’s programs; 2) investigative procedures and techniques;

and 3) legal implications and conclusions to be drawn from audit and investigative materialproduced by the OIG. The Counsel’s office also administers the civil monetary penalty program.

Social Security: A-02-00-10019%20Final

Documents

Transcript of Social Security: A-02-00-10019%20Final