Social Security: A-02-01-11011

8/14/2019 Social Security: A-02-01-11011

http://slidepdf.com/reader/full/social-security-a-02-01-11011 1/41

SOCIAL SECURITY

MEMORANDUM

Date: March 8, 2002 Refer To:

To: Jo Anne B. BarnhartCommissioner

From: Inspector General

Subject: Performance Measure Review: Reliability of the Data Used to Measure the Quality of the Social Security Administration’s Research (A-02-01-11011)

Following consultations with congressional committees, the Office of the Inspector Generalagreed to review the Social Security Administration’s (SSA) performance indicators over acontinuous 3-year cycle. We recently completed our first 3-year cycle. In conducting thiswork, we used the services of an outside contractor, PricewaterhouseCoopers, LLP, (PwC),to assist us in our efforts.

For this report, we used PwC to conduct the review of two of the Agency’s performanceindicators related to the quality of its research publications. The objective of the reviewwas to assess the reliability of the data used to measure the quality of SSA’s research.

Please comment within 60 days from the date of this memorandum on corrective actiontaken or planned on each recommendation. If you wish to discuss the final report,please call me or have your staff contact Steven L. Schaeffer, Assistant Inspector General for Audit, at (410) 965-9700.

James G. Huse, Jr.

Attachment



OFFICE OFTHE INSPECTOR GENERAL

SOCIAL SECURITY ADMINISTRATION

PERFORMANCE MEASURE REVIEW:RELIABILITY OF THE DATA USED TO

MEASURE THE QUALITY OF THESOCIAL SECURITY ADMINISTRATION’S

RESEARCH

March 2002 A-02-01-11011

EVALUATION REPORT

.



Mission

We improve SSA programs and operations and protect them against fraud, waste,and abuse by conducting independent and objective audits, evaluations, andinvestigations. We provide timely, useful, and reliable information and advice toAdministration officials, the Congress, and the public.

Authority

The Inspector General Act created independent audit and investigative units,

called the Office of Inspector General (OIG). The mission of the OIG, as spelledout in the Act, is to:

0

o0

Conduct and supervise independent and objective audits andinvestigations relating to agency programs and operations.Promote economy, effectiveness, and efficiency within the agency.Prevent and detect fraud, waste, and abuse in agency programs and

operations.Review and make recommendations regarding existing and proposedlegislation and regulations relating to agency programs and operations.Keep the agency head and the Congress fully and currently informed of

problems in agency programs and operations.

0

0

To ensure objectivity, the IG Act empowers the IG with:

o00

Independence to determine what reviews to perform.Access to all information necessary for the reviews.Authority to publish findings and recommendations based on the reviews.

Vision

By conducting independent and objective audits, investigations, and evaluations,we are agents of positive change striving for continuous improvement in theSocial Security Administration's programs, operations, and management and inour own office.



Evaluation of Selected PerformanceMeasures of the Social SecurityAdministration:

Reliability of the Data Used to Measure the Qualityof SSA’s Research

Office of the Inspector GeneralSocial Security Administration



1

INTRODUCTION

This report is one of five separate, stand-alone reports, corresponding to the following SocialSecurity Administration’s (SSA) process and performance measures (PM):

q Percent of customers assigning a high rating to the quality of SSA’s research andanalysis products in terms of accuracy, reliability, comprehensiveness, andresponsiveness. (PM #1)

Fiscal Year (FY) 2000 Goal: Develop customer survey and data collectionmechanism.

q Percent of Major Statistical Products that are timely. (PM #2)

FY 2001 Goal: Establish a baseline for percent of major statistical projects that areproduced on schedule.

This report reflects our understanding and evaluation of the process related to PMs #1

and #2. To achieve its strategic goal, “To promote valued, strong, and responsive socialsecurity programs and conduct efficient policy development, research, and programevaluation,” SSA has developed several strategic objectives. One of these objectives is“…to provide information for decision makers and others on the Social Security andSupplemental Security Income programs through objective and responsive research,evaluation, and policy development.” SSA’s FY 2001 Annual Performance Plan (APP)contains two performance indicators developed to meet this objective as follows:

q Percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness, and responsiveness - This indicator will be considered achieved if SSA develops a customer survey and data collection mechanism.

q Percent of major statistical products that are timely - This second indicator will beconsidered achieved if SSA establishes a baseline for percent of major statisticalprojects that are produced on schedule.

We performed our testing from September 21, 2000 through February 15, 2001. Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland. Theprocedures that we performed were in accordance with the American Institute of CertifiedPublic Accountants’ Statement on Standards for Consulting Services, and are consistentwith appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did notconduct an audit, the objective of which would be the expression of an opinion on the

reliability or accuracy of the reported results of the performance measures evaluated.Accordingly, we do not express such an opinion.



2

Performance Measure # 1 - Percent of customers assigning a high rating to the quality of SSA’s research and analysis products in terms of accuracy, reliability, comprehensiveness,and responsiveness.

BACKGROUND

This indicator has been created to measure the percent of customers assigning a high ratingto the quality of SSA’s research and analysis products in terms of accuracy, reliability,comprehensiveness, and responsiveness. The goal during FY 2000 was to develop acustomer survey and data collection mechanism. After the first survey is administered SSAplans to make updates based on the results, as appropriate. Below is an overview of SSA’scustomer satisfaction survey and data collection mechanism.

Survey Objectives

This initiative seeks to better understand the information needs of decision-makers andother customers of SSA research, statistics, and policy analysis products and services.Moreover, SSA seeks to determine how effectively it is meeting those needs. Specifically,

this study is to determine:q To what extent SSA’s work is reaching a broad audience in the research and policy

community.

q To what degree this work is perceived to be of high quality.

q How satisfied the SSA customers are with the information they receive.It also aims to better understand the needs of active participants in the research andpolicy community who are currently non-users of SSA informational services.

Sample Design

The survey is expected to employ a stratified probability sampling design, where the strataare defined based on the following customer characteristics:

I. Decision-maker;

II. Subscriber;

III. Non-subscribers who are active in the research and policy community; and

IV. Non-subscribers who are interested.

The first stratum, Decision-makers, consists of 59 individuals selected by SSA based on theassumption that they have significant potential for influencing policy decisions. Those in the

second stratum, nearly 1,000, are individual subscribers to various SSA publications. Thethird stratum is comprised of different individuals, including attendees of various SSAconferences and members of the National Academy of Social Insurance (NASI). Thisstratum includes about 1,500 individuals. The fourth stratum includes approximately 800non-subscribers who are interested in receiving SSA statistical publications. Whileindividuals from the third and fourth strata originate from separate sources, they will becombined to create an “active non-subscriber” group. The following table provides asummary of the sampling design for this survey.



3

Table 1. Sampling Design and Universe Counts

Strata Source Population Sample

Decision-makers Individually selected by SSA Office of Policy 59 59

Subscribers Subscribers to various SSA publications 978 978

Active, Non-Subscribers

Non-subscribers, attendees of RetirementResearch Consortium and other conferences, as well as members of NASI

1,422 485

Non-subscribersNon-subscribers, generated fromdatabase of stakeholders, maintained bySSA’s Office of Communication

814 278

Total 3,273 1,800

Because the list of decision-makers is relatively small, the sample design calls for a censusof all these individuals. Further, because the subscriber group is expected to contain thelargest concentration of “users,” the entire list will also be included in the sample. For theremaining two strata, however, it is not clear at this point what methods will be used to selectthe needed samples. It appears that all lists have been examined to remove individuals withmultiple entries, though additional processing is required to identify addresses and or telephone numbers where needed information is missing.

SSA personnel stated that the target response rate to this survey is expected to be at least 80percent. Consequently, SSA anticipates needing a sample of about 1,250 to secure the required1,000 responses. It is expected that this sample size will be sufficient for various statisticalanalyses. It should be noted that SSA’s contractor deems the 80 percent response rateunachievable and is designing the samples based on a lower rate of 50 percent. However, thetotal sample of 1,800 will not be adequate to obtain the desired 1,000 responses should theresponse rate be as low as 50 percent.

Questionnaire Design

The questionnaire for this survey is designed with the intent that respondents can completeit within 15 minutes. Considering that the length of a survey can impact the response rate, itis important to keep the response time within the specified time limit, even in a survey likethis where some respondents might have a vested interest.

Since there will be one single questionnaire for all sample types, a few skip patterns arerequired to administer appropriate questions to corresponding respondents. For instance,respondents who identify themselves in the survey as recent users will be asked about thetype, frequency, topics, quality, usefulness, and satisfaction with SSA materials. Non-userswill be surveyed about their Social Security related informational needs. Lastly, users andnon-users will also be asked about the most important research/policy issues that SSAshould focus on during the next few years.

Currently, the questionnaire is being tested and fine-tuned via focus groups and cognitiveinterviews. The focus groups were conducted in December 2000 with a sample of userswith various informational needs. SSA personnel stated that, consistent with Office of



4

Management Budget (OMB) restrictions, fewer than 10 respondents will participate in eachfocus group. Following the focus groups, cognitive interviews will be conducted to further test questionnaire content. These interviews are planned to be conducted over thetelephone with a combination of users and non-users once the questionnaire is finalized; thedate of which had not been determined at the time of our evaluation.

Administration/Data Collection (Anticipated)

Since the survey had not been administered at the time of our evaluation, this evaluation willbe limited to those procedures that have been proposed. According to the proposed plan,the survey administration will consist of the following steps:

q All sampled individuals will be mailed a pre-notification letter. This letter will provide abrief description of the purpose of the study and inform the individual that thequestionnaire will be arriving shortly.

q One week later, the survey packet will be mailed. This packet will consist of a copy of the questionnaire, a return envelope, and a letter regarding the purpose of the study.This letter will also assure confidentiality and specify the web-site address whererespondents can visit to complete the survey on-line.

q One week later, a reminder postcard will be sent to all sampled individuals.

q Two weeks later, non-responder telephone calls will be initiated, at which pointindividuals will be given the option to complete the questionnaire over the phone, as wellas on the web, or by mail.

It is anticipated that multiple data collection modes will help secure the required 80 percentresponse rate. Meanwhile, inactive non-subscribers will be dropped from the survey andexcluded from the number of completes and the response rate.

Analysis and Report Generation (Anticipated)

SSA’s contractor anticipates the first step in the data analysis plan to be combining the datagathered via the three modes of data collection into one file. This data file will then becleansed of duplicate entries. Next, the contractor will remove responses where over half of the items are missing. Checks will also be performed to clean inconsistent data.With the “cleansed” data, the contractor will conduct an analysis within the self-identifiedusers group. The users will be divided into two sub-groups – subscriber or non-subscriber,based on their sample frame origin. Comparisons between the subscriber and non-subscriber sub-groups are expected to be evaluated to determine any differences in the waythese groups perceive the quality of SSA’s work.

The primary analysis will focus around the subscriber and non-subscriber groups. Inaddition to simple descriptive statistics, SSA is expecting analyses that will effectivelymeasure respondents’ evaluations and levels of satisfaction with the services they receive.SSA’s contractor will also perform a leverage analysis to determine the most criticalcharacteristics in driving overall product and service satisfaction. This method usesregression techniques to identify the most important factors that can predict the outcomemeasure of interest. In the final report, SSA is expecting documentation regarding themethodology used, and conclusions and recommendations for further study.



5

RESULTS OF EVALUATION

During the period of September 21, 2000 to February 15, 2001, we evaluated the currentprocesses and controls, which support the FY 2000 SSA performance measurementprocess. Our evaluation of the information provided to us by SSA management and itscontractor as of February 15, 2001, allowed us to determine that SSA has made significantprogress to date in the implementation process, and it has the capacity to effectivelycomplete the implementation process.

Performance Measure Reported Result

1. Percent of customers assigning a high rating tothe quality of SSA’s research and analysisproducts in terms of accuracy, reliability,comprehensiveness, and responsiveness.

In the process of developing customer survey and data collection mechanism.

Although not directly related to GPRA compliance, we noted the following

three opportunities for improvement:

1. Sampling frames do not provide a comprehensive listing of all users and non-users of SSA research products.

2. Limited information has been collected by SSA from non-users of SSA researchproducts.

3. The questionnaire design could be improved by SSA.

These items were noted as a result of our testing of the SSA survey instrument, as itcurrently exists, as well as the focus group documentation, sampling parameters, and

discussions with SSA regarding future plans.

This survey has not been conducted yet. As a result, our evaluation will be limited to thesteps that have been completed to date and the planning that has been done for futuresteps. After the survey has been administered and the corresponding data has beenanalyzed, it is critical to apply a similar evaluation to these future components of research.The following is a listing of potential issues that deserve attention:

1. Sampling frames do not provide a comprehensive listing of all users and non-users of SSA research products.

Based on the very limited information provided by SSA’s contractor it appears that the

sampling frames that support the nonsubscriber stratum may not provide a complete listing.In addition, these lists include ineligible (non-active) individuals, as well as inaccuratecontact information for a significant number of listed individuals. 1 Consequently, the targetuniverse for this survey is not determinable at this time, and therefore the results of thesurvey may not be generalized.

1 Proposed sampling plan, data collection plan, and initial data analysis plans. SSA’s contractor, December 21, 2000.



6

The Paperwork Reduction Act of 1995, Implementing Guide, Chapter 6, E1 states thatstatistical laws that permit inference from a sample to a population assume completecoverage, complete response, and random selection. If any of these conditions are not met,then inferences cannot be demonstrated to be valid (i.e. cannot produce results that can begeneralized to the universe of study). The OMB Resource Manual for Customer Surveystates, “The ability to make general statements about your agency’s customers based onthe survey results is linked to how complete and accurate your frame is.”

Furthermore, based on the contractor’s report from the focus groups, we also are aware of inconsistencies in SSA’s current lists. Therefore, we suggest that SSA carefully reexaminethe sampling frame used to support the survey.

2. Limited information has been collected by SSA from non-users of SSA researchproducts.

Limited information has and will be collected from the active non-user group. Focus groupshave been conducted to improve the effectiveness and quality of the questionnaire.However, it appears that only the users of SSA services were included in the process andfocus groups, leaving out non-users, a group that can provide invaluable insight regardingthe available services. For example, users cannot comment on why they do not use SSA’sservices.

The OMB Resource Manual for Customer Survey, Chapter 3 Defining Customer Survey ,states, “It is important to talk to as diverse a group of customers as possible, so that nomajor perspective or point of view is omitted.”

3. The questionnaire design could be improved by SSA.

The most recent draft of the questionnaire we have been provided, dated January 29, 2001,asks non-users via an open-ended question, if there is any particular reason why they have

not used SSA’s information. The content of this question is a valuable addition to thequestionnaire design. However, because it is an open-ended question, fewer respondentsare likely to answer. Closed-ended questions receive better response rates than open-ended questions, because they are less burdensome for respondents than open-endedquestions. The use of closed-ended questions will secure a higher response rate and willimprove the quality of the resulting data. 2

CONCLUSIONS AND RECOMMENDATIONS

SSA is making progress in preparing for the implementation of this survey. Per conversations with SSA, the contractor has revised the questionnaire to eliminate some of the earlier issues. This process of questionnaire refinement is still in progress. It appearsthat SSA, with its contractor support, has the capacity to implement the process.

It is important to note that even if everything proceeds according to the plans, this survey willcontinue to have issues related to the extrapolation of results. Since the target universe for this survey is not determinable, the results of the survey cannot be generalized to a knownuniverse of potential users and non-users, with any degree of validity at this time. 2 Sudman, Seymour & Bradburn, Norman M. Asking Questions: A Practical Guide to Questionnaire Design. (1st ed.)Jossey-Bass Inc., Publishers, 1982, pp.148 - 152.



7

We recommend:

1) SSA and its contractor should investigate alternative sources for a sampling frame, sothey can better target the universe. For the current year, steps should be taken to refinethe universe by eliminating “ineligible” individuals.

2) Since users cannot comment on why they do not use SSA’s services, SSA shouldconduct focus groups that also include non-users of SSA’s defined statistical products.

3) In order to increase response rate to this question and improve the quality of theresulting data, SSA should rephrase the question to non-users of SSA’s statisticalproducts, regarding the reason why they have not used SSA’s information. The questioncould be a “check all” type question, with reasons why non-users have not used SSAinformation as response categories. In addition, SSA should include “Other:_________”as a response category.



8

Performance Measure # 2 - Percent of Major Statistical Products that are timely.

BACKGROUND

The indicator created by SSA is to measure the “Percent of major statistical products thatare timely,” with the goal during FY 2000 to establish a baseline for timeliness. Thisindicator is linked to SSA’s strategic objective to “Provide information for decision makersand others on the Social Security and Supplemental Security Income programs throughobjective and responsive research, evaluation, and policy development.”

The SSA APP for FY 2001 states that “Responsive research, evaluation, and policydevelopment means providing relevant information in the clearest possible manner with anefficient use of resources.” The Office of Research, Evaluation and Statistics (ORES) hasselected five of their publications to measure.

I. Fast Facts (annual)On Internet—10 weeks from release of Trustees ReportIn print—11 weeks from release of Trustees Report

II. Income of the Population 55 or Older (biannual, in even years)On Internet—January 31In print-March 15

III. Income of the Aged Chartbook (biannual, in even years)On Internet—March 31In print—April 7

IV. Annual Statistical Supplement (annual)On Internet—in phases between August and December (still finalizing dates)In print—December 31

V. SSI Annual Statistical Report (annual)On Internet—July 10In print—July 3

The publications selected by ORES were chosen based on their volume of usage. ORESarrived at the above dates through: 1) detailed discussions with the division directors or analysts involved in developing the publications and with staff of the Division of InformationResources (DIR) in ORES, and 2) historical data from DIR for the last several years ondates when documents were sent to the Office of Publications Management (OPUM), whenOPUM cleared the documents to be sent to the printer, and when documents were receivedfrom the printer. These dates have been established as the measure of a report being

timely.

ORES receives the information to be included in the publications from several sources bothwithin and outside SSA. The measure begins when ORES receives the data and endswhen the information is published on the Internet and in hard copy. The ORES milestonesin the publication process are as follows:



9

q Data availability;

q ORES analysts/programmer production of all text, tables and charts;

q DIR completion of editing and production of charts;

q For 3 publications, receipt of blue lines (a copy of the print format) from the printer;

q DIR posting of document on Internet in PDF format; and

q Receipt of publication from printer.

RESULTS OF EVALUATION

During the period of September 21, 2000 to February 15, 2001, we evaluated the currentprocesses, systems and controls, which support the FY 2000 SSA performancemeasurement process. Our evaluation of the information provided by SSA management asof February 15, 2001, allowed us to determine that SSA has made significant progress todate in the implementation process, and it has the capacity to effectively complete theimplementation process.

Performance Measure Reported Result

2. Percent of major statistical products that are timely Established baseline schedule

However, we did note the following two opportunities for improvement:

1. SSA should incorporate other publications as part of this measure.

2. SSA is not fully evaluating the currency of the data.

These items were noted as a result of our discussions with SSA regarding current and futureplans.

After evaluating SSA’s documentation and speaking with the Deputy AssociateCommissioner of ORES it is apparent that SSA has put significant effort into deciding whatpublications are to be measured and in creating a timetable for publication. The timelineswere established based on when ORES has historically received the information. Since partof the information included in the publications comes from outside SSA, the establisheddeadlines may need to be adjusted in the future due to circumstances beyond SSA’scontrol.

The following is a listing of potential issues that deserve attention.

1. SSA should incorporate other publications as part of this measure.

The strategic objective linked to this measure is for the entire agency while only one part of the agency is being measured. Currently SSA is only measuring five statistical publicationsthat ORES produces. Other divisions of SSA produce material “…for decision makers andothers…” about the Social Security and Supplemental Security Income programs.



10

Furthermore, this indicator only measures statistical publications, instead of statistical andnon-statistical publications about these programs. By including major publications producedby other SSA divisions and major non-statistical products, the performance measure wouldbetter support the strategic objective and allow management to see the timelines throughoutthe agency rather than within one division. Thus, this would provide management with atool to measure the performance of divisions throughout the agency.

2. SSA is not fully evaluating the currency of the data.

The strategic objective linked to this measure requires the information to be relevant.Currently SSA only measures the production cycle of the publication process. Thecollection cycle is not included in the measure. By not including the collection process, themetric does not yield a complete or full measurement of the time involved. If informationentering the production cycle is already outdated, it will not be relevant to users, andtherefore the measure does not evaluate whether the data was collected in a timely manner.By measuring the entire publishing cycle from data collection through release to users,management will be better able to evaluate if the information provided is relevant pertainingto timeliness.

CONCLUSIONS AND RECOMMENDATIONS

SSA has started the process to develop a good measure to support it strategic objective;however, the measure could be improved to better support the strategic objective. SSA isonly measuring the timeliness of publication of one division and only five publications, whichrepresent only a portion of the information provided to decision-makers for these programs.The strategic goal is for all information used by SSA, not just that published by ORES.

While not measuring inputs versus outputs, the measure does fall within the intent of GPRA.The measure, which is being used to establish whether information is being provided in atimely manner, will provide management the ability to evaluate their processes and improveinternal management.

We recommend:

4) SSA should expand PM #2 to include additional types of reports and include publicationsproduced by divisions other than ORES.

5) For the timeliness to be measured and to ensure that data being published is current,SSA should evaluate the entire publication cycle, not just the production cycle.



11

OTHER MATTERS

SSA has not quantified the timeliness goal.

While SSA has established a timeline and cycle for the production of the publications, theactual goal of how many of the publications have to be produced within their timeline inorder for SSA to meet its goal has not been set. SSA has not established the target goal,i.e., what would be considered a success for this indicator. Do all of the publications have tobe produced by their established publication date or only 60 percent for the Agency to besuccessful? GPRA states in section 1115 (a)(2), “…express such goals in an objective,quantifiable, and measurable form unless authorized to be in an alternative form under subsection (b).” Subsection (b) does not apply to this goal because it is feasible to meet thecriteria stated in section 1115 (a)(2).

Recommendation

6) Starting in FY 2002, SSA should consider establishing a quantitative goal (e.g., 95percent of all publications should be produced within their established timetable) in order to evaluate whether the timelines measure has been met.

By establishing what percentage of publication cycles has to be completed within their established timeframe, management will be establishing target goals for the performanceindicator. This would allow management to assert whether they have met their goal andallow it to be evaluated by outside observers.

APPROPRIATENESS OF PERFORMANCE MEASURES

As part of this engagement, we evaluated the appropriateness of each of the performancemeasures with respect to GPRA compliance and SSA’s APP. We determined whether the

specific indicators and goals corresponded to the strategic goals identified in SSA’s APP,determined whether each of these indicators accurately measures performance, anddetermined their compliance with GPRA requirements.

Performance Measure #1 aligns logically with the SSA Strategic Plan but still needssome improvement.

The relationship between PM #1 and the applicable SSA Strategic Goal is depicted in thefollowing figure:



12

SSA MissionTo promote the economic security of the nation's people

through compassionate and vigilant leadership in shapingand managing America's social security programs.

Strategic "Goal #1"To promote valued, strong and responsive social

security programs and conduct effective policydevelopment, research and program evaluation.

Strategic ObjectiveProvide information for decision-makers and others on the

Social Security and Supplemental Security Income programsthrough objective and responsive research, evaluation, and

policy development.

Performance Indicator Percent of customers assigning a high rating to thequality of SSA's research and analysis products interms of accuracy, reliability, comprehensiveness,

and responsiveness.

The SSA mission is supported by five strategic goals, including Goal #1, “to promote valued,

strong and responsive social security programs and conduct effective policy development,research and program evaluation.” Goal #1, in turn, is supported by several strategicobjectives, including the relevant objective that deals with the provision of SSA informationto decision makers, “Provide information for decision-makers and others on the SocialSecurity and Supplemental Security Income programs through objective and responsiveresearch, evaluation, and policy development.” Performance Measure #1 depicts thecustomer satisfaction with SSA research and analysis products. Assuming that the metrichas strong performance measurement attributes, the diagram indicates that PM #1 logicallyaligns with SSA’s strategic planning process.

Based on the taxonomy of performance measures included in Appendix F, PM #1 is ameasure of accomplishment because it reports on a result (customer satisfaction) achieved

with SSA resources. It is further categorized as an outcome measure because it indicatesthe accomplishments or results (customer satisfaction) that occur because of the services(SSA research products) provided. As shown in Appendix F, outcome measures includepublic perceptions of outcomes, such as a customer satisfaction index.

Within the framework of GPRA, Performance Measure #1 fits the intent of an outcomemeasure because it is “…a description of the intended result, effect, or consequence that



13

will occur from carrying out a program or activity.” 3 The intent of this performancemeasurement is to gauge customer satisfaction (i.e., the effect) for the activity of providingresearch and analysis products. Assuming it is accurate, it can be useful to bothmanagement and external stakeholders, as encouraged by OMB Circular A-11 . However, wehave identified a few underlying deficiencies that indicate it may not be a reliable GPRAperformance indicator.

First, the metric may not promote an adequate and correct understanding of the subjectarea. Outcome measures such as customer satisfaction indices are complex multivariatemetrics that depend on a host of attributes. Defining a critical performance indicator simplybased on a single percentage will fail to provide a comprehensive metric. Such simplisticmeasures may not provide sufficient intelligence for taking appropriate action andsubsequently improving performance. Due to the sampling frame issue, there is noguarantee the metric will provide a true picture of the value of SSA research data.

The data is partially subjective because it reflects the opinion of the respondents at the timethey answer the survey. Furthermore, use of the data to measure progress from year-to-year is reliable only for the specific sample. Since the employed sampling method is notprobability based, the extrapolative power of the resulting survey data will remainundeterminable. In addition, a performance metric should ideally allow SSA to trackprogress over time. In this case, consistency may suffer due to the nature of the samplingframe. If the survey is repeated from year to year, statistical comparisons between thevarious years are likely to be unreliable.

At the time of this evaluation, SSA and their contractor had not yet determined a method for analyzing and processing the survey data in order to obtain the desired baseline metric.Furthermore, they had not yet established a method for evaluating progress over a multi-year horizon. These shortcomings pose legitimate concerns, since such analyses areessential for the metric to be useful.

Based on their assigned task, Office of Policy and their contractor have done well inestablishing the activities required for obtaining PM #1. Furthermore, the customer satisfaction survey will be a useful product because SSA can use the data to gain valuableinsight into stakeholders’ needs with respect to the research and analysis products.However, as a GPRA performance metric, PM #1 has underlying deficiencies and itsreliability is in question.

Recommendation

7) SSA should put this metric on hold until the contractor develops a method for analyzingthe survey data, developing the baseline metric, and using the metric to manageperformance over a multi-year horizon. If these objectives can not be satisfactorily

completed, the metric should not be included in the GPRA measurement system.

Combined survey results can be misleading because the utility of the SSA researchand analysis products varies for each user.

At the time of this evaluation, neither ORES nor their contractor had established a plan for computing the customer satisfaction results from the survey data. When this is determined, 3 OMB Circular A-11 Preparation and Submission of Budget Estimates , section 200.2.



14

ORES should carefully consider the manner in which they combine the responses from thevarious user types. The utility of SSA research products is not equal for each user. A fewsignificant respondents may rely extensively on the SSA research and analysis products ascrucial inputs to their own analytical and decision making products, which, in turn, areimportant national policy tools in their own right. In such cases, research information usedby one user may have a significant benefit to society. On the other hand, many respondentsmay utilize these products to a far lesser degree and may not rely on their timeliness andaccuracy to as great an extent. Unfortunately, these two types of customers carry the sameweight in the customer satisfaction survey results. This is a ramification of the survey’ssampling frame. For example, assume that all of the significant users have responded andthat any additional responses will come from “occasional” users. As additional responsesare incrementally added to survey, the results may become skewed. The occasional usersmay have either a higher or a lower opinion of the products relative to the significant users.Regardless, they can change the apparent outcome of the customer satisfaction survey,even though the overall value of the SSA research products to the public have changed verylittle.

Recommendation

8) ORES should pay careful attention to the manner in which survey results are combined.

Performance Measure #2 aligns logically with the SSA Strategic Plan and canpotentially provide numerous management benefits.

The relationship between PM #2 and the applicable SSA Strategic Goal is depicted in thefollowing figure:



15

SSA MissionTo promote the economic security of the nation's peoplethrough compassionate and vigilant leadership in shapingand managing America's social security programs.

Strategic "Goal #1"To promote valued, strong and responsivesocia l secur i ty programs and conducteffective policy development, research andprogram evaluation.

Strategic ObjectiveProvide information for decision-makers and

o t h e r s o n t h e S o c i a l S e c u r i t y a n dSupplemental Security Income programsthrough objective and responsive research,evaluation, and policy development.

Performance IndicatorsPercent of major statistical products thatare timely.

The SSA mission is supported by five strategic goals, including Goal #1, “to promote valued,strong and responsive social security programs and conduct effective policy development,research and program evaluation.” Goal #1, in turn, is supported by several strategicobjectives, including the relevant objective that deals with the provision of SSA informationto decision makers, “Provide information for decision-makers and others on the SocialSecurity and Supplemental Security Income programs through objective and responsiveresearch, evaluation, and policy development.” Performance Measure #2 characterizes thetimeliness of the SSA statistical products. Assuming that the metric has strong performancemeasurement attributes, the diagram indicates that PM #2 logically aligns with SSA’s

strategic planning process.Based on the taxonomy of performance measures included in Appendix F, PM #2 is ameasure of accomplishment because it reports on a result (timeliness) achieved with SSAresources. It is further categorized as an output measure because it indicates the quantityof services provided. As shown in Appendix F, output measures include indications of thequantity of a service provided that meets a certain quality requirement, such as on-timestandards of achievement.



16

Within the framework of GPRA, PM #2 fits the intent of an output measure because it is“…a description of the level of activity or effort that will be produced or provided over aperiod of time or by a specified date, including a description of the characteristics andattributes (e.g., timeliness) established as standards in the course of conducting the activityor effort.” While this performance measure may, in theory, represent a good measure of timeliness, it also has some areas for improvement.

The metric is narrow in focus. Specifically, it counts the timeliness of publications from onlyORES, thus representing only one office within the agency. However, by its very name anddescription, stakeholders may be led to believe that it measures the timeliness of all of SSA’s major statistical products. Furthermore, the metric only includes five products in itscalculation. In addition, the measure does not distinguish between large and small-scaleefforts. A publication that theoretically requires 80 percent of the resources will impact themeasure in the same way as a publication that requires negligible resources. Finally, thereis no policy as of yet for determining the circumstances that allow for a change in the targetdate without being penalized for the lack of timeliness.

Despite these issues, the metric has strong performance measure attributes and can beused as an excellent project management tool. In fact, SSA is encouraged to leverage theadditional benefits that can be obtained with this type of project management tool. For example, they could focus beyond the literal description of the metric and utilize the dataitself as a management tool. As a result, real time decisions and corrections can be madeduring the publication process to help improve the on-time delivery of the products. It is alsonoteworthy that the measure addresses only the steps of the publishing process that arecompleted after ORES receives the relevant information.

Recommendation

9) To further leverage the project management characteristics of this metric, it isrecommended that ORES develop methods for obtaining status throughout the

publishing process. Finally, it would be relatively easy to compare the timeliness of SSAstatistical products with the on-time delivery of publications from other agencies. It isrecommended that ORES decompose their publication process into major subtasks, asnecessary, so that they can benchmark against other government and commercialpublishing operations. In addition, ORES should expand the metric to include other documents. They should also develop a policy to address the change of target dates.

Agency Comments

The Agency agreed with eight of the nine recommendations within this report. It disagreedwith the recommendation to “…include additional types of reports and include publicationsproduced by divisions other than the Office of Research, Evaluations, and Statistics” in the

performance measure, “Percent of major statistical products that are timely.” SSAdetermined that there are no other statistical publications that are produced regularly, havea statutory deadline, or are considered major in terms of usage.

In agreeing with the recommendation that SSA should evaluate the entire publication cycle,not just the production cycle, when measuring the timeliness of its statistical products, SSAstated that it was already doing what was recommended.



17

OIG Response

We appreciate the Agency’s response to this report. The implementation of therecommendations will help to ensure for the better measurement and reporting of the qualityof SSA’s research. We continue to recommend that SSA consider the inclusion of additionaltypes of reports and publications in its measurement of the timeliness of its research. Whilethe Agency may have determined that it currently includes all publications that are regularlyproduced, have a statutory deadline, or are considered major in terms of usage, we believethere are other research related publications produced by the Agency that are used bydecisionmakers. While these publications may not be regularly produced or mandated bystatute, their timely delivery to decisionmakers is nonetheless important. The measurementof the timeliness of all research products produced in a given year is in line with the strategicobjective related to the timeliness performance measure--Provide information for decision-makers and others on the Social Security and Supplemental Security Income programsthrough objective and responsive research, evaluation, and policy development.

We also continue to recommend that SSA evaluate the entire publication cycle, and not justthe production cycle, when measuring the timeliness of its statistical products. SSAcurrently does not measure the collection cycle of the process for all of the publicationsbeing measured. By measuring the entire publishing cycle, from data collection throughrelease to users, SSA will be better able to evaluate if the information provided is relevant toits users.



APPENDICES

APPENDIX A – Scope and Methodology

APPENDIX B – AcronymsAPPENDIX C – Agency Comments

APPENDIX D – Performance Measure Summary Sheets

APPENDIX E – Performance Measure Process Maps

APPENDIX F – Performance Measure Taxonomy



Appendix A

A-1

SCOPE AND METHODOLOGY

The Social Security Administration (SSA) Office of the Inspector General contractedPricewaterhouseCoopers (PwC) to evaluate 11 SSA performance indicators identified in itsFY 2001 APP. We performed our testing from September 21, 2000 through February 15,2001.

Our engagement was limited to testing at SSA’s headquarters in Woodlawn, Maryland. Theprocedures that we performed were in accordance with the American Institute of CertifiedPublic Accountants’ Statement on Standards for Consulting Services, and are consistentwith appropriate standards for performance audit engagements in Government Auditing Standards (Yellow Book, 1994 version). However, we were not engaged to and did notconduct an audit, the objective of which would be the expression of an opinion on thereliability or accuracy of the reported results of the performance measures evaluated.Accordingly, we do not express such an opinion.

For these two performance indicators, PwC was asked to assess SSA’s progress to date inthe implementation process, and evaluate if SSA has the capacity to effectively completethe implementation process.

1. Percent of customers assigning a high rating to the quality of SSA’s research andanalysis products in terms of accuracy, reliability, comprehensiveness, andresponsiveness. (PM #1)

We performed an evaluation of the survey methodology used for this performance measure,to assess SSA’s progress in implementation of the survey and ability to complete theimplementation process. Specifically, we assessed completed steps associated with thesampling frame, the sample design, and the draft of the questionnaire. This included

evaluating survey objectives, sample and questionnaire designs, data collection, andanalysis and report generation.

We obtained an understanding of the underlying process and procedures surrounding theimplementation of the measure through interviews and meetings with the appropriate SSAand contractor personnel and by reviewing and evaluating the following documentation:

q SSA’s Statement of Work, dated July 2000;

q Most recent draft of the questionnaire, dated January 29, 2001;

q Contractor’s status report to SSA, Office of Policy, dated October 27, 2000, regarding:“Sample frame, initial recommendations for data collection, and plans for pretest and

focus group;”

q Focus Group report, dated December 21, 2000;



Appendix A

A-2

q Contractor’s memo to SSA, Office of Policy, dated January 12, 2001, regarding“Proposed sampling plan, data collection plan, and initial data analysis plan.” Thismemo provided a summary of recent developments and modifications to the originalplan

q General Accounting Office (GAO) reports on SSA FY 2001 Performance Plan.

2. Percent of Major Statistical Products that are timely. (PM #2)

In order to assess SSA’s progress to date in the implementation process and evaluate if SSA has the capacity to effectively complete the implementation, we obtained anunderstanding of the underlying process and procedures surrounding the implementation of the measure through interviews and meetings with the appropriate SSA personnel. Inaddition, we evaluated the following:

q Memo from ORES regarding publication process and method used to establishedtimelines for publication;

q Process/cycle memos for the publications to obtain an understanding of the publicationprocess; and

q GAO reports on SSA FY 2001 Performance Plan.

3. Determined whether both performance measures were meaningful and incompliance with GPRA.

As part of this engagement, we evaluated the appropriateness of each of the performancemeasures with respect to GPRA compliance and SSA’s APP. To do so, we determinedwhether the specific indicators and goals corresponded to the strategic goals identified inSSA’s APP, determined whether each of these indicators accurately measure performance,

and determined their compliance with GPRA requirements.



Appendix B

ACRONYMS

APP Annual Performance Plan

DIR Division of Information ResourcesFY Fiscal Year

GAO General Accounting Office

GASB Government Accounting Standards Board

GPRA Government Performance Results Act

NASI National Academy of Social Insurance

OASDI Old Age, Survivors, and Disability Insurance

OIG Office of the Inspector General

OMB Office of Management and BudgetOP Office Policy

ORES Office of Research, Evaluation and Statistics

PM Performance Measure

PwC PricewaterhouseCoopers LLP

SSA Social Security Administration

SSI Supplemental Security Income

OPUM Office of Publications Management



Appendix C

AGENCY COMMENTS



SOCIAL SECURIlY

MEMORANDUM

December 10, 2001 SlJ-3efer To:

To James G. Ruse, Jr.Inspector General

Larry Dye --/9-rChief of Staff ~~ r-

Office of the Inspector General (OIG) Draft Report, "Perfonnance Measure Review: Reliability

of the Data Used to Measure the Quality of the Social Security Administration's Research"(A-M-Ol-llOI1 )-INFORMATION

O:J-We appreciate OIO's efforts in conducting this review. Our comments on the report contentand recommendations are attached.

Please et us know if we may be of further assistance. Staff questions may be referred toTrudy Williams on extension 50380.

Attachment:SSA Response



Appendix C

C-1

COMMENTS ON THE OFFICE OF THE INSPECTOR GENERAL (OIG) DRAFTREPORT, “PERFORMANCE MEASURE REVIEW: RELIABILITY OF THE DATAUSED TO MEASURE THE QUALITY OF THE SOCIAL SECURITYADMINISTRATION’S RESEARCH” (A-02-01-11011)

Thank you for the opportunity to review and comment on this draft report. We are pleased tonote that OIG acknowledges that SSA has developed useful performance indicators in the spiritof the Government Performance and Results Act (GPRA) and has discussed them in detail inthe Fiscal Year (FY) 2001 Performance Plan.

Our comments on the report recommendations are detailed below.

Performance Measure Number 1 - Percent of Customers assigning a high rating to thequality of SSA’s research and analysis products in terms of accuracy, reliability,comprehensiveness, and responsiveness.

Recommendation 1

SSA and its contractor should investigate alternative sources for a sampling frame, so theycan better target the universe. For the current year, steps should be taken to refine theuniverse by eliminating “ineligible” individuals.

Comment

We agree. SSA has long recognized that measuring satisfaction with policy analysis products is inherently difficult, especially in identifying actual and potential customers.SSA is the only agency that is attempting to include such a measure in its performance plan.In order to meet our goal of conducting an initial survey in FY 2001, we used the bestinformation that could be developed, while recognizing that we did not have a fullysatisfactory sampling frame. However, OIG has offered no possible solutions. We believethat attempting to measure customer satisfaction is important, even if it cannot be done withcomplete precision, and we will continue to do so periodically. Consequently, if we cannotdevelop a measure that is statistically representative and can be tracked over time, we willdrop this item from the performance plan.

Recommendation 2

Since users cannot comment on why they do not use SSA’s services, SSA should conduct

focus groups that also include non-users of SSA’s defined statistical products.Comment

We agree. We will implement this recommendation before conducting the next survey in FY2004.



Appendix C

C-2

Recommendation 3

In order to increase response rate to this question and improve the quality of the resultingdata, SSA should rephrase the question to non-users of SSA’s statistical products, regardingthe reason why they have not used SSA’s information. The question could be a “check all”type question, with reasons why non-users have not used SSA information as responsecategories. In addition, SSA should include “Other:_________” as a response category.

Comment

We agree and will implement this recommendation before conducting the next survey in FY2004.

Performance Measure Number 2 (PM #2) -Percent of Major Statistical Products thatare timely.

Recommendation 4

SSA should expand PM #2 to include additional types of reports and include publications produced by divisions other than the Office of Research, Evaluation, and Statistics (ORES).

Comment

In the report, OIG acknowledges the Agency made a significant effort in identifying publications for this measure. We have determined that no other SSA statistical publicationsare produced regularly, have a statutory deadline, or are considered major in terms of usage. Therefore, we disagree with this recommendation.

Recommendation 5

For the timeliness to be measured and to ensure that data being published is current, SSAshould evaluate the entire publication cycle, not just the production cycle.

Comment

We agree and believe that we are already doing what is recommended, with the exception of benchmarking against other publishing operations. The current timetable lists the reference period of the data, when the data will become available, when analysts will complete a draft,when editing and production will be completed, when the document will be posted on theWeb and when we will receive the published document from the printer. Most of these

publications are based on SSA program data and appear within months of the end of thereference year. Two of the publications are based on a Census Bureau survey that becomesavailable within six months of data collection. The schedule also includes target dates for each major function in the production process, so that we can track the source of any

problems in meeting the ultimate deadline. Finally, the schedule is updated annually to allowfor changes in target dates. In FY 2002, target dates were modified to reflect the need for all

publications to be section 508 compliant. Some dates are specified in terms of elapsed time



Appendix C

C-3

from the reference period of the data, others are fixed dates. However, no meaningfulcomparison can be made between our schedules and those of other agencies withoutconducting a detailed analysis of the complexity of the information being released and theresources available to produce it. Comparisons with commercial publishing operationswould need to consider that commercial entities are not required to use the Government

Printing Office or to be section 508 compliant. This performance measure is alreadyresulting in shorter timeframes and reliable schedules for users. We will continue to look for ways to improve our processes.

Other Matters

Recommendation 6

Starting in FY 2002, SSA should consider establishing a quantitative goal (e.g.,95 percent of all publications should be produced within their established timetable) in order to evaluate whether the timeliness measure has been met.

Comment

This has now been done. SSA’s goal for FY 2001 was to develop a baseline for thismeasure. SSA’s goal for FY 2002 is to produce all publications on time.

Recommendation 7

SSA should put this metric on hold until the contractor develops a method for analyzing the

survey data, developing the baseline metric, and using the metric to manage performanceover a multi-year horizon. If these objectives cannot be satisfactorily completed, the metricshould not be included in the GPRA measurement system.

Comment

See response to recommendation 1.

Recommendation 8

The Office of Research, Evaluation and Statistics (ORES) should pay careful attention to themanner in which survey results are combined.

Comment

We agree with the intent of this recommendation but not with trying to differentiate“significant” users from others. The contractor’s report on the results of the customer satisfaction survey provides not only overall results but also results by four groups of



Appendix C

C-4

respondents (decisionmakers, subscribers, nonsubscribers, and stakeholders). The resultsshow very little difference in outcomes across the four categories of respondents.

Recommendation 9

To further leverage the project management characteristics of this metric, it is recommendedthat ORES develop methods for obtaining status throughout the publishing process. Finally,it would be relatively easy to compare the timeliness of SSA statistical products with the on-time delivery of publications from other agencies. It is recommended that ORES decomposetheir publication process into major subtasks, as necessary, so that they can benchmark against other government and commercial publishing operations. In addition, ORES shouldexpand the metric to include other documents. They should also develop a policy to addressthe change of target dates.

Comment

We have taken steps to address this recommendation. See our response torecommendation 5.



A p p e n

d i x D

P E R F O R M A N C E M E A S U R E S U M M A R Y

S H E E T S

N a m e o

f M e a s u r e

M e a s u r e

T y p e

S t r a t e g i c G o a l / O

b j e c t i v e

1 ) P e r c e n t o f C u s t o m e r s

a s s i g n i n g a

h i g h r a t i n g

t o t h e

q u a l i t y o f S

S A ’ s r e s e a r c

h a n d a n a l y s i s p r o d u c t s

i n t e r m s

o f a c c u r a c y , r e l i a b i l i t y , c o m p r e h e n s i v e n e s s a n d

r e s p o n s i v e n e s s .

P e r c e n t a g e

G o a

l : T o p r o m o t e v a l u e d , s t r o n g , a n d r e s p o n s i v e

s o c i a l s e c u r i t y p r o g r a m s a n d c o n d u c t e

f f e c t i v e p o l i c y

d e v e l o p m e n t , r e s e a r c h , a n d p r o g r a m e v a l u a t i o n .

O b j e c t i v e :

P r o v i d e

i n f o r m a t i o n

f o r d e c i s i o n m a k e r s

a n d o t h e r o n

t h e S o c i a l S e c u r i t y a n d S u p p l e m e n t a l

S e c u r i t y

I n c o m e p r o g r a m s

t h r o u g h o b j e c t i v e a n d

r e s p o n s i v e r e s e a r c h , e v a l u a t i o n a n d p o

l i c y

d e v e l o p m e n t .

P u r p o s e

S u r v e y

F r e q u e n c y

T o a s s e s s

t h e p e r c e n t o

f c u s t o m e r s a s s i g n i n g a

h i g h r a t i n g

t o t h e q u a l i t y o f S

S A ’ s

r e s e a r c h a n d a n a l y s i s p r o d u c t s

i n t e r m s o f a c c u r a c y , r e l i a b

i l i t y , c o m p r e h e n s i v e n e s s

a n d r e s p o n s i v e n e s s ; S S

A w

i l l d e v e l o p a p r e l i m

i n a r y q u e s t i o n n a i r e

t o g a t h e r


f r o m c u s t o m e r s a b o u t t h e q u a l i t y o f r e s e a r c h , a n a l y s i s a n d e v a l u a t i o n

p r o d u c t s . S S A w

i l l a l s o i d e n t i f y c u s t o m e r s

f r o m w

h o m f e e d b a c k w

i l l b e e l i c i t e d .

T h e s u r v e y

h a s n o t b e e n

d e v e l o p e d a t

t h i s p o i n t .

S S A

h a s n o t y e t d e t e r m

i n e d t h e f r e q u e n c y i n w

h i c h t h e

s u r v e y w

i l l b e a d m

i n i s t e r e d .

T a r g e t

G o a l

H o w

C o m p u

t e d

D a t a

S o u r c e

T o d e v e l o p a c u s t o m e r s u r v e y a n d

d a t a c o l l e c t i o n m e c h a n i s m .

T h i s g o a l w

i l l b e c o n s i d e r e d a c h i e v e d

i f S S A :

a ) D e v e l o p s a s u r v e y

t h a t g a t h e r s i n f o r m a t i o n a b o u t w

h e t h e r r e s e a r c h a n d a n a l y s i s

i d e n t i f i e s n e w

/ e m

e r g i n g i s s u e s a n d p r o v i d e s a c c u r a

t e , r e l i a b l e c o m p r e h e n s i v e a n d

r e s p o n s i v e r e s e a r c h , a n a l y s i s a n d e v a l u a t i o n p r o d u c t s .

b ) I d e n t i f i e s c u s t o m e r s

f r o m w

h o m f e e d b a c k w

i l l b e e l i c i t e d , a n d

c ) C r e a t e s a m e c h a n i s m

b y w

h i c h t o a d m

i n i s t e r t h e s u r v e y a n d p r o c e s s

t h e d a t a .

S S A ’ s

C o n t r a c t o r



A p p e n

d i x D

D e s i g n a t e d

S t a f f M e m

b e r s

D i v i s i o n

M i c h a e l A

. C r o n i n

S u s a n

G r a d

J o h n W o o d s

O f f i c e P o l i c y

( O P )

T e s t i n g a n

d R e s u

l t s

W e o b t a i n e d a n u n d e r s t a n d i n g o f t h e u n d e r l y i n g p r o c e s s a n d p r o c e d u r e s s u r r o u n d i n g

t h e i m p l e m e n t a t i o n o f t h e m e a s u r e

t h r o u g h i n t e r v i e w s

a n d m e e t i n g s w

i t h t h e a p p r o p r i a t e

S S A a n d

i t s c o n t r a c t o r p e r s o n n e l a n d

b y e v a l u a t i n g

t h e f o l l o w i n g d o c u m e n t a t i o n :

·

C o n t r a c t o r S

t a t e m e n t o f W o r k ,

d a t e d J u l y 2 0 0 0 .

·

M o s t r e c e n t d r a f t o f

t h e q u e s t i o n n a i r e ,

d a t e d J a n u a r y 2 9 , 2 0 0 1

·

C o n t r a c t o r ’ s s t a t u s r e p o r t t o

S S A

, O f f i c e o f P o l i c y ,

d a t e d O c t o b e r 2 7 ,

2 0 0 0 , r e g a r d i n g : “ S a m p l e

f r a m e ,

i n i t i a l r e c o m m e n d a t i o n s

f o r d a t a

c o l l e c t i o n , a n d p l a n s

f o r p r e t e s t a n d f o c u s g r o u p .

”

·

T h e F o c u s G r o u p r e p o r t ,

d a t e d D e c e m

b e r 2 1 , 2 0 0 0 .

·

M e m o

f r o m S S A ’ s C o n t r a c t o r t o

S S A

, O f f i c e o f P o l i c y ,

d a t e d J a n u a r y 1 2 , 2 0 0 1 , r e g a r d i n g

“ P r o p o s e d s a m p l i n g p l a n ,

d a t a c o

l l e c t i o n p l a n ,

a n d i n i t i a l d a t a a n a l y s i s p l a n .

” T h i s m e m o p r o v i d e d a s u m m a r y o f r e c e n t d e v e l o p m e n t s a n d m o d i f i c a t i o n s

t o t h e o r i g i n a l p l a n .

R e f e r t o

“ R e s u l t s o f E v a

l u a t i o n ” f o r a d e s c r i p t i o n o f t h e

f i n d i n g s .



A p p e n

d i x D

N a m e o

f M e a s u r e

M e a s u r e

T y p e

S t r a t e g

i c G o a l / O

b j e c t i v e

2 ) P e r c e n t o f m a j o r s t a t i s

t i c a l p r o d u c t s t h a t a r e t i m e l y .

P e r c e n t a g e

G o a l : T o p r o m o t e v a l u e d , s t r o n g , a n d r e s p o n s i v e s o c i a l

s e c u r i t y p r o g r a m s a n d c o n d u c t e f f e c t i v e p o l i c y

d e v e l o p m e n t , r e s e a r c h , a n d p r o g r a m e v a l u a t i o n .

O b j e c t i v e :

P r o v i d e


f o r d e c i s i o n m a k e r s a n d

o t h e r s o n

t h e S o c i a l S e c u r i t y a n d

S u p p l e m e n t a l S e c u r i t y

I n c o m e p r o g r a m s

t h r o u g h o b j e c t i v e a n d r e s p o n s i v e

r e s e a r c h , e v a l u a t i o n a n d p o l i c y

d e v e l o p m e n t .

P u r p o s e

F r e q u e n c y

M e a s u r e o f t h e e x t e n t t o w

h i c h S S A ’ s m a j o r s t a t i s t i c a l p r o d u c t s a r e p r o d u c e d

i n a

t i m e l y m a n n e r .

A n n u a l I n d i c a t o r

T a r g e t

G o a l

D a t a

S o u r c e

T h i s g o a l w

i l l b e c o n s i d e r e d a c h i e v e d

i f S S A e s t a b l i s h e s a b a s e l i n e

f o r p e r c e n t o f m a j o r s t a t i s t

i c a l p r o j e c t s

t h a t a r e p r o d u c e d o n s c h e d u l e .

B u r e a u o f t h e

C e n s u s

S S A O f f i c e o f S y s t e m s ,

D i v i s i o n o f P r o g r a m

S t u d i e s

D e s i g n a

t e d S t a f f M e m

b e r s

D i v i s i o n

S u s a n

G r a d

A l e x a n d e r E s t r i n

O P

O R E S

T e s

t i n g a n

d R e s u

l t s

W e o b t a i n e d a n u n d e r s t a n d i n g o f t h e u n d e r l y i n g p r o c e s s a n

d p r o c e d u r e s s u r r o u n d i n g

t h e i m p l e m e n t a t i o n o f t h e m e a s u r e

t h r o u g h

i n t e r v i e w s

a n d m e e t i n g s w

i t h t h e a p p r o p r i a t e

S S A p e r s o n n e l .

I n a d d i t i o n , w e e v a l u a t e d

t h e f o l l o w i n g :

·

M e m o

f r o m O R E S r e g a r d i n g p u b l i c a t i o n p r o c e s s a n d m e t h o d u s e d

t o e s t a b l i s h e d

t i m e l i n e s

f o r p u b l i c a t i o n

·

P r o c e s s / c y c l e m e m o s

f o r t h e p u b l i c a t i o n s

t o o b t a i n a n u n d e r s t a n d i n g o f t h e p u b l i c a t i o n p r o c e s s

R e f e r t o

“ R e s u l t s o f E v a l u a t i o n ” f o r a

d e s c r i p t i o n o f t h e

f i n d i n g s .



A p p e n

d i x E

P E R F O R M A N C E M E A S U R E P R O C E S S M A P S

1



A p p e n

d i x E

P r o v i d e c o n t r a c t o r m u l t i p l e

s a m p l e

f r a m e w

i t h

a p p r o x i m a t e l y

3 , 0 0 0 n a m e s

c r e a t e d f r o m m u l t i p l e

l i s t s o f

t a r g e t e d s u b p o p u l a t i o n s

R e w o r k

G o a l # 1

o f P e r f o r m a n c e

P l a n

P r e p a r e

S t a t e m e n t o f

W o r k p r e p a r e d

( J u l y 2 0 0 0 )

A w a r d

C o n t r a c t

C o n d u c t f o c u s g r o u p s w

i t h

u s e r s

D e v e l o p s a m p l i n g p l a n

A d m i n i s t e r s u r v e y

A n a l y z e r e s u l t s a n d p r e p a r e

r e p o r t

D e v e l o p a s t r a t i f i e d

p r o b a b i l i t y s a m p l e

d e s i g n

y i e l d i n g a

t a r g e t o f 1 , 0 0 0

c o m p l e t e d q u e s t i o n n a i r e s o r

i n t e r v i e w s

S p e c

i f y i n t e n d e d m e t h o d s o f

d a t a c o l l e c t i o n , a n d

d e v e l o p

p l a n s f o r m a x i m

i z i n g

r e s p o n s e r a t e s

D e v e l o p p l a n s

f o r d a t a

a n a l y s i s , a n d p r e p a r e

r e q u i r e d f o r m s a n d

s u p p l e m e n t a r y m a t e r i a l f o r

O M B c l e a r a n c e o f t h e s u r v e y

P r e t e s t t h e

i n s t r u m e n t

t h r o u g h c o g n i t i v e

i n t e r v i e w s w

i t h u s e r s

P e r f o r m a n c e

M e a s u r e

# 1

C u s t o m e r

S a

t i s f a c

t i o n

S u r v e y

C o m p l e t e a l l

p r e p a r a t i o n s n e e d e d

t o c o n d u c t t h e s u r v e y

C o n d u c t t h e s u r v e y ,

i n c l u d i n g f o l l o w - u p

c o n t a c t s t o a c h i e v e

t a r g e t e d r e s p o n s e

r a t e s

C o d e

t h e d a t a , a n d

p r e p a r e d a t a f i l e s

a n d c o r r e s p o n d i n g

d o c u m e n t a t i o n

C o m p l e t e a p r e l i m

i n a r y

a n a l y s i s o f t h e

d a t a

C o m p l e t e a

f i n a l r e p o r t o n

t h e p r o j e c t ,

i n c l u d i n g f u l l

d i s c u s s i o n o f m e t h o d o l o g y ,

f u r t h e r d a t a a n a l y s i s ,

c o n c l u s i o n s , a n d

r e c o m m e n d a t i o n s

f o r f u r t h e r

s t u d y

F i n a l i z e t h e s u r v e y

i n s t r u m e n t

P r o v i d e c o n t r a c t o r w

i t h a

d r a f t o f t h e q u e s t i o n n a i r e

D e v e l o p q u e s t i o n n a i r e

d e s i g n

D e v e l o p s u r v e y o b j e c t i v e s

D e t e r m

i n e t o w

h a t d e g r e e

t h i s w o r k

i s p e r c e i v e d

t o

b e o f h i g h q u a l t i y a n d

e l i c i t s h i g h l e v e l s o f

c u s t o m e r s a t i s f a c t i o n

B e t t e r u n d e r s t a n d n e e d s

o f p o t e n t i a l u s e r s - t h o s e

w h o a r e a c t i v e

i n t h i s

r e s e a r c h / p o l i c y a r e a ,

b u t

c u r r e n t l y n o n - u s e r s o f

S S A i n f o r m a t i o n

D e t e r m

i n e t o w

h a t e x t e n t

w o r k

i s r e a c h i n g a

b r o a d

a u d i e n c e i n t h e r e s e a r c h /

p o l i c y c o m m u n i t y a n d

i s

p e r c e i v e d t o b e m e e t i n g

t h e i r n e e d s

G o a l #

1 i s t o p r o m o t e v a l u e d , s t r o n g , a n d

r e s p o n s i v e s o c i a l s e c u r i t y p r o g r a m s a n d

c o n d u c t e f f e c t i v e p o l i c y

d e v e l o p m e n t ,

r e s e a r c h , a n d p r o g r a m e v a l u a t i o n .

S h a d e d s t e p s

h a v e n o t b e e n c o m

p l e t e d

a s o f 2 / 1 5 / 2 0 0 1

D e t a i l s :



A p p e n

d i x E

F a s

t F a c

t s

( A n n u a l )

O n

I n t e r n e t - M a r c h

3 1

I n P r i n t - D e c e m

b e r 3 1

I n c o m e o

f t h e

A g e

d

C h a r t b o o

k

( B i a n n u a l e v e n y r s )

O n

I n t e r n e t - i n p h a s e s

( A u g u s t - D e c e m

b e r )

I n P r i n t - D e c e m

b e r 3 1

A n n u a

l S t a t i s t i c a

l

S u p p

l e m e n

t

( A n n u a l )

O n

I n t e r n e t - J u l y 1 0

I n P r i n t - J u l y

3 1

S S I A n n u a

l S t a t i s t i c a

l

R e p o r t

( A n n u a l )

O n

I n t e r n e t - J a n u a r y 3 1

I n P r i n t - M a r c h

1 5

I n c o m e o

f t h e

P o p u

l a t i o n

5 5 a n

d O l d e r

( B i a n n u a l e v e n y r s )

O n

I n t e r n e t - 1 0 w e e k s

I n P r i n t - 1 1 w e e k s

F r o m r e l e a s e o f T r u s t e e s

R e p o r t

G a t h e r d a t a a n d p e r f o r m

R e s e a r c h

P r e p a r e p u b l i c a t i o n s

( t e x t , t a b l e s a n d c h a r t s )

P e r f o r m e d i t i n g a n d c o m p l e t e

p r o d u c t i o n o f c h a r t s

R e c e i v e

b l u e l i n e s f r o m

p r i n t e r f o r 3 p u b l i c a t i o n s

P e r f o r m a n c e

M e a s u r e

# 2

P u

b l i c a

t i o n s

T o

b e

P r o

d u c e d

G o a l # 1

i s t o p r o m o t e v a l u e d , s t r o n g , a n d r e s p o n s i v e s o c i a l

s e c u r i t y p r o g r a m s a n d c o n d u c t e f f e c t i v e p o l i c y

d e v e l o p m e n t ,

r e s e a r c h , a n d p r o g r a m e v a l u a t i o n .

P r o

c e s s

t o P u

b l i c i z e I n f o r m a

t i o n

P o s t d o c u m e n t o n

I n t e r n e t

i n P D F f o r m a t

R e c e i v e p u b l i c a t i o n s

f r o m p r i n t e r



DISTRIBUTION SCHEDULE

No. of Copies

Commissioner of Social Security 1

Management Analysis and Audit Program Support Staff, OFAM 10

Inspector General 1

Assistant Inspector General for Investigations 1

Assistant Inspector General for Executive Operations 3

Assistant Inspector General for Audit 1

Deputy Assistant Inspector General for Audit 1

Director, Data Analysis and Technology Audit Division 1

Director, Financial Audit Division 1

Director, Western Audit Division 1

Director, Southern Audit Division 1

Director, Northern Audit Division 1

Director, General Management Audit Division 1

Issue Area Team Leaders 25

Income Maintenance Branch, Office of Management and Budget 1

Chairman, Committee on Ways and Means 1

Ranking Minority Member, Committee on Ways and Means 1

Chief of Staff, Committee on Ways and Means 1

Chairman, Subcommittee on Social Security 2

Ranking Minority Member, Subcommittee on Social Security 1

Majority Staff Director, Subcommittee on Social Security 2

Minority Staff Director, Subcommittee on Social Security 2

Chairman, Subcommittee on Human Resources 1

Ranking Minority Member, Subcommittee on Human Resources 1

Chairman, Committee on Budget, House of Representatives 1Ranking Minority Member, Committee on Budget, House of Representatives 1

Chairman, Committee on Government Reform and Oversight 1

Ranking Minority Member, Committee on Government Reform and Oversight 1

Chairman, Committee on Governmental Affairs 1

Ranking Minority Member, Committee on Governmental Affairs 1



Page 2

Chairman, Committee on Appropriations, House of Representatives 1

Ranking Minority Member, Committee on Appropriations,House of Representatives 1

Chairman, Subcommittee on Labor, Health and Human Services, Educationand Related Agencies, Committee on Appropriations,House of Representatives 1

Ranking Minority Member, Subcommittee on Labor, Health and HumanServices, Education and Related Agencies, Committee on Appropriations,House of Representatives 1

Chairman, Committee on Appropriations, U.S. Senate 1

Ranking Minority Member, Committee on Appropriations, U.S. Senate 1

Chairman, Subcommittee on Labor, Health and Human Services, Educationand Related Agencies, Committee on Appropriations, U.S. Senate 1

Ranking Minority Member, Subcommittee on Labor, Health and HumanServices, Education and Related Agencies, Committee on Appropriations,U.S. Senate 1

Chairman, Committee on Finance 1

Ranking Minority Member, Committee on Finance 1

Chairman, Subcommittee on Social Security and Family Policy 1

Ranking Minority Member, Subcommittee on Social Security and Family Policy 1

Chairman, Senate Special Committee on Aging 1

Ranking Minority Member, Senate Special Committee on Aging 1

Vice Chairman, Subcommittee on Government Management Informationand Technology 1

President, National Council of Social Security Management Associations,Incorporated 1

Treasurer, National Council of Social Security Management Associations,Incorporated 1

Social Security Advisory Board 1

AFGE General Committee 9

President, Federal Managers Association 1

Regional Public Affairs Officer 1

Total 97



Overview of the Office of the Inspector General

Office of Audit

The Office of Audit (OA) conducts comprehensive financial and performance audits of the

Social Security Administration's (SSA) programs and makes recommendations to ensure hatprogram objectives are achieved effectively and efficiently. Financial audits, required by theChief Financial Officers Act of 1990, assess whether SSA' s financial statements airly presentthe Agency's financial position, results of operations, and cash flow. Performance audits reviewthe economy, efficiency, and effectiveness of SSA s programs. OA also conducts short-term

management and program evaluations focused on issues of concern to SSA, Congress, and thegeneral public. Evaluations often focus on identifying and recommending ways to prevent andminimize program fraud and inefficiency.

Office of Executive Operations

The Office of Executive Operations (OEO) supports the Office of the Inspector General (OIG) byproviding information resource management; systems security; and the coordination of budget,procurement, telecommunications, facilities and equipment, and human resources. In addition,this office is the focal point for the OIG's strategic planning function and the development andimplementation of performance measures equired by the Government Performance and ResultsAct. OEO is also responsible for performing internal reviews to ensure hat OIG officesnationwide hold themselves to the same rigorous standards hat we expect from the Agency, aswell as conducting employee investigations within OIG. Finally, OEO administers OIG's publicaffairs, media, and interagency activities and also communicates OIG's planned and currentactivities and their results to the Commissioner and Congress.

Office of InvestigationsThe Office of Investigations (01) conducts and coordinates investigative activity related to fraud.waste, abuse, and mismanagement of SSA programs and operations. This includes wrongdoingby applicants, beneficiaries, contractors, physicians, interpreters, representative payees, hirdparties, and by SSA employees n the performance of their duties. Or also conducts ointinvestigations with other Federal, State, and local law enforcement agencies.

Counsel to the Inspector General

The Counsel to the Inspector General provides legal advice and counsel to the Inspector Generalon various matters, including: l) statutes, egulations, legislation, and policy directives

governing the administration of SSA s programs; 2) investigative procedures and techniques; and3) legal implications and conclusions to be drawn from audit and investigative material producedby the DIG. The Counsel's office also administers the civil monetary penalty program.

Social Security: A-02-01-11011

Documents

Transcript of Social Security: A-02-01-11011