Advancing Return on Investment Analysis for Government IT: A Public Value Framework

Click here to load reader

  • date post

  • Category


  • view

  • download


Embed Size (px)


Review the analysis of government IT investment, from broad levels to specific methods that document public value, which can be tailored to particular investment decisions. The scope applies to simple Web sites and government-wide information systems and architectures.

Transcript of Advancing Return on Investment Analysis for Government IT: A Public Value Framework

  • 1. Advancing Return on Investment Analysis for Government IT A Public Value Framework Anthony M. CresswellG. Brian BurkeTheresa A. PardoCenter for Technology in Government University at Albany, SUNY 187 Wolf Road, Suite 301 Albany, NY 12205Phone: (518) 442-3892 Fax: (518) 442-3886 E-mail: [email protected] 2006 This material is based upon work supported by SAP Any opinions, findings, conclusions, or recommendations . expressed in this material are those of the authors and do not necessarily reflect the views of SAP. 2006 Center for Technology in GovernmentThe Center grants permission to reprint this document provided this cover page is included.

2. CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 3. Acknowledgments T his paper and the project it is based onPublic ROI: Advancing Return on Investment Analysis for Government ITwas made possible with funding and guidance from SAP a, provider of enterprise software for the public sector. In particular, we would like to thank Russ LeFevre and Peg Kates, who have been partners from the very beginning of this work, and special thanks to Tom Shirk, Ingo Hoffman, Rod Massey, Ian Swann, Mor Sagmon, Carsten Friedland, Alex Bratzler, and Bonnie Rothenstein.Several members of the Center for Technology in Government staff made many important contributions to this white paper and the overall project: Sharon Dawes, Jane Krumm- Schwan, Alison Heaphy, Paula Hauser, and Lucy Dadayan.Finally, this white paper and the case study reports depended on the enthusiastic participation of many government officials from the five case study sites. In particular, we would like to thank Christian Ihle from the Ministry of Finance in Austria, Adam Jansen from the Washington State Digital Archives, Ronny Jacobowitz and Yitshak Cohen from the Merkava Project in Israel, Andy McIntyre from the Integrated Enterprise System in the Commonwealth of Pennsylvania, and Darrell Fowler from Service New Brunswick in Canada. We thank them for their generous hospitality and commitment in helping us conduct our site visits and creating this resource for improving government IT investments. CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 4. CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 5. Table of Contents Executive summary 1Introduction4Section I 5 Why Assess Public ROI for Government IT Section II6 Research Summary of Public Value Case Studies Section III 8 A Public Value Framework for Government IT Assessment10 A. The Framework Strategy.11 B. How Does Government IT Investment Link to the Public?13 C. What Kinds of Impacts Matter for Public Value.14 D. What is the Investment? Linking IT to Goals and Business Processes.16 E. What Kinds of Public Value are Produced?18 F. Who Receives Value? The Stakeholder Analysis.20 G. How to Demonstrate the Value? Identifying Variables and Methods.26 H. What Can Interfere With or Prevent Public Returns? The Risk Analysis.28 I. Overview of Using the Framework.32 J. How to Summarize and Present Results?Section IV 37 Reflections on the Framework and the Value of Public ValueAppendix 38 The Research Basis for the New Framework 38 A. Consultative Workshop40 B. Case Study SummariesEndnotes43 CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 6. CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 7. Executive Summary T he range and complexity of government information technology (IT) investments makes assessing their returns a daunting Weak or absent methods for tailoring a public ROI assessment to the specific context and goals of a government IT prospect. Projects can range from investment. systemwide transformations, to improving financial transparency, to more efficient dog Existing methods also deal with risks to licensing. The returns may be large or small, government IT investment, but primarily from obvious or obscure, and can run from a fewthe point of view of technology development minutes saved in a routine transaction to and implementation. The risks that involve improving the trust and legitimacy of anthe public beneficiaries of the investments entire government. In spite of the difficulty inmerit more attention. assessment, however, knowledge about public returns can be vital to fully informed This white paper presents a public ROI and justified IT investment decisions.assessment framework that addresses these shortcomings. We call it a public value Assessing these returns remains a coreframework to emphasize the point of view of problem in IT planning and decision making. the public, not the government, as the basis That problem results from shortcomings in for the assessment. The framework the available methods and models fordescribes how to identify and assess public assessing public returns, what we callvalue through the kinds of activities shown public return on investment (ROI). In looking below in Figure 1. over these existing methods and models we saw three significant shortcomings: The frameworks strategy is simple in concept, but complex in application: connect Incomplete analysis of public value,what happens in the government (on the resulting in too narrow a scope of what left) with the impacts on stakeholders in the can be considered returns to the public.public domain (to the right), then report and apply the results. The activities on the left Lack of systematic attention to how identify the potential value mechanisms and government IT investments generateoutcome goals. Those are linked to results of value from the point of view ofstakeholder interests, impacts, and risks to the public. the right. The curved arrows indicate that inFigure 1. Public Value Framework CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 1 8. practice the process would seldom be linear, Ideological impacts on beliefs, moral requiring reflection and backtracking to or ethical commitments, alignment of adjust for learning and new information. The government actions or policies or social full paper presents a detailed version of this outcomes with beliefs, or moral or schematic, showing the links among these ethical positions. activities.Stewardship impacts on the publics The public value proposition takes centerview of government officials as faithful stage. This value proposition must bestewards or guardians of the value of broadly conceived to do justice to the scope the government itself in terms of public of government and how it affects trust, integrity, and legitimacy. individuals, groups, and both public and private organizations. This frameworkExpanding the view of stakeholder interests presents a new and more comprehensivein this way brings into focus two distinct but way of describing public value, based on six equally impor tant types of public value: the kinds of impacts government IT can have on delivery of benefits directly to citizens and the interests of public stakeholders:enhancing the value of government itself asa public asset. An IT investment that makes Financial impacts on current orgovernment more transparent, more just, anticipated income, asset values,and a better steward has added public liabilities, entitlements, and other value, a non-financial but nonetheless aspects of wealth or risks to any of the important return. This framework describes above. how to include both in public valueassessments. Political impacts on personal or corporate influence on governmentThe framework also identifies the basic actions or policy, role in political affairs,ways government IT investments link to or influence in political parties or public value. The simplest link results from prospects for current of future public an IT investment embedded directly in a office.service delivery process (Figure 2:A) in away that enhances its value to citizens or Social impacts on family or communityofficials. The total value may be a composite relationships, social mobility, status, andof several separate measures: cost savings, identity.quality, enhanced trust. An indirect link(Figure 2:B) can result from infrastructure Strategic impacts on economic orimprovements to business processes. political advantage or opportunities, goals, resources for innovation or This link may be more difficult to trace and planning.can involve more risk. Enterprise Resource Figure 2. Direct and Indirect Links to Public Valuegovernmentold governmentold IT V1 with old IT V1business processinfrastructure Citizen/Citizen/ OfficialOfficial V2 governmentV2new government new ITwith new IT business processinfrastructure Return = V2 V1 Return = V2 V1 A - DirectB - IndirectCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 2 9. Figure 3. Mix of Direct and Indirect Links to Public Valueenvironment Citizen/Citizen/ government Vi Vd OfficialOfficial with new ITReturn = Vd + Vi Vi = indirect value Vd = direct valuePlanning (ERP) software implementations inState Digital Archives case study, for our case studies are excellent examples ofexample, putting birth and marriage this kind of investment. Value can also resultrecords online enabled research by from a mix of direct and indirect links (Figure local historians and genealogists. 3) when new IT changes the environment. Intrinsic enhancements changing the Identifying these links is just part of the story.environment or circumstances of a Each link can involve several value-stakeholder in ways that are valued for generating mechanisms. The frameworktheir own sake. For example, our Israel describes how to trace these mechanisms case showed how enhanced financial working through specific business accounting and reporting in the processes to produce different kinds of Merkava ERP system opened value. The value-generating mechanismsgovernment financial decision making are:to greater transparency. Increases in efficiency obtainingAn IT investment project can deliver publicincreased outputs or goal attainment value through any or all of thesewith the same resources, or obtainingmechanisms.the same outputs or goals with lowerresource consumption. In our AustrianThe framework is deliberately presented at aand Pennsylvania case studies, for moderate level of generality to make it mostexample, new ERP systems helpedwidely useful. Every government IT projectachieve substantial efficiencies inwill have its own unique goals, valuefinancial management.propositions, and stakeholders. So this framework can be used to plan and guide aIncreases in effectiveness public value assessment, in combinationincreasing the quality and/or quantity ofwith measurements, analysis tools, andthe desirable thing. Our case study of repor ting techniques chosen for the specificService New Brunswick, for example,situation. To aid in this regard, the paperrepor ts how an online registry for land presents an overview of more detailed anddata can contribute to improvements in highly specified assessment methods thatproper ty tax administration.can be used in conjunction with this framework. This includes a more detailedEnablement providing means orflow char t for the assessment process, aallowing otherwise infeasible or summary of several ROI models andprohibited desirable activity, ormethods, and suggestions for analysis andpreventing or reducing undesirable repor ting of or outcomes. In our WashingtonCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 3 10. Introduction Every investment decision requires aleap of faithsometimes a large one into an uncer tain future. When much of thatThe approach we take to answering thatquestion takes a point of view based on ourexperience, analysis, and the background uncertainty involves the value of theresearch for this paper. From this point of expected returns, risk increases and the view we see two sources of public returns: investment decision is all the more difficult. (1) value to the public that results from This is an even bigger problem for improving the government itself, and (2) government IT projects, which arevalue that results from delivering specific notoriously risky and aimed at public valuebenefits directly to persons or groups. We returns that are often very difficult to definealso see potential value creation that goes and even harder to measure. This white far beyond the traditional financial and paper takes on one important part of thatservice evaluation data. Value creation can problem: the question of describing andcome as much from increasing the integrity measuring public value.and transparency of government as fromreducing costs through online tax payments. We chose to focus on public value becauseThis expanded scope of value includes an it is both the most poorly understood and theoften wide range of stakeholders, each with most complex par t of the overall problem. their special interests and expectations from We can divide the overall analysis of return government. This point of view dictates an on investment (ROI) in government IT expansive way of seeking public value. We expenditure into four parts: costs, internal call this a public value framework, meaning returns (agency efficiencies, cost avoidance,that it is less a specific measurement tool etc.), public returns, and risks. Of these, theand more a way of identifying and as public returns element receives the leastsessing as wide a range of public value as amount of attention, though interest in this possible. This is not a small task, and topic is growing. The key question about cannot be completed in one such effor t. But public returns, of course, is what do we this approach can advance the search for a mean by public value and how can weway to effectively measure public value. observe, measure, and document its creation. The goal of this white paper is to help answer that question. CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 4 11. Section I: Why Assess Public ROI for Government IT? T he scope of government investment in IT and the associated problems certainly deserve serious attention. Both the levels on the public sector: If governments do not accurately measure the full value of their IT investments, they risk a serious political and growth rates in government IT spendingbacklash. They will be accused of wasting are substantial. The most recent data billions of pounds of taxpayers money on available shows that the level of thisunnecessary technology. This sentiment investment in both the industrialized has reached the US Congress. In July of this countries and the developing world hasyear the Senate Appropriations Committee grown to a ver y large scale. recommended no funding for the administrations 2007 e-governmentEuropean government IT spending is initiatives, reporting that the committeeexpected to increase from US$110 has no confidence that the amounts beingbillion in 2005 to US$119 billion by assessed have any relationship to the2007, with US$26 billion in the UK benefits anticipated to be returned. And inalone, which is about 40% abovethe House of Representatives, the currentGermany or France. appropriations bill calls for a cost-benefit analysis of all e-government initiatives. According to Mike Hettinger, staff director atIT spending by Asia-Pacific region the House Government Reform Committeesgovernments, excluding Japan, is Government Management, Finance andexpected to reach US$31.7 billion by Accountability Subcommittee, the language2010, from US$22.7 billion in 2006.speaks for itself and reiterates what the subcommittee has been saying for the pastThe Chinese government expendituresyear, that in order for this initiative to beof USD$5 billion in 2004 are expectedsuccessful, we need to have a betterto grow 16% in the following five years. understanding of the costs and benefits and clearer guidance for the agencies to follow. 2India spent US$943 million onCertainly this enhanced attention to ROI fore-government in 2002, and this figure is IT must include a comprehensive andpredicted to increase by 15% a year to effective way to deal with the public valueUS$3.3 billion by 2009.side of the problem.The U.S. federal budget for fiscal year2007 provides $64 billion in funding forIT investments, approximately a 3%increase from the 2006 enacted level of$62 billion. Total state and localgovernment IT spending was about $55billion in FY2004 and is expected togrow to $62.4 billion by FY2009.Though what is included in the IT category may vary across these examples, the overall scale of expenditure remains substantial.1This level of expenditure is receiving increasing scrutiny. The reason for this scrutiny is clear, according to Andrea DiMaio, a Gartner Vice President focusing CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 5 12. Section II: Research Summary of Public Value Case Studies T his framework is based in part on the results of five case studies that exam- ined how government IT investment projects These general understandings of public value guided the data collection and presentation of the case results. came to deliver value to the public.3 The projects were the Integrated Enterprise The case studies examined public returns to System in the Commonwealth of the overall IT investment, including returns Pennsylvania and the Washington State generated by a government IT investment Digital Archives in the US, the Merkava and the possible mechanisms to produce Project in Israel, the Austrian Federal them. We did this by considering the links Budgeting and Bookkeeping System in between goals, implemented systems, Austria, and Service New Brunswick in government performance, and public returns Canada. All five case study reports and the in terms of where they represent value in white paper are available on CTGs Webthe chain shown in Figure 4 (next page). site.2 In the case studies we examined how the IT investments were conceived and The public returns from the investment can developed, with particular attention to the flow from the internal improvements in the role of public value in the process. We tookagency resulting in returns to individual the view that government IT investmentcitizens and the public at large (the main generates public value in two ways: flow through the center). Other returns can flow to the political system and the economicBy improving the value of theenvironment (below the center), or throughgovernment itself from the perspective effects on other agencies (secondaryof the citizens, and performance gains). This general view of public returns informed the case studies andBy delivering specific benefits directly tohelped summarize the results.persons, groups, or the public at large.The first is based on the idea that, assuming a benign government, the better it functions overall, the better off its citizens will be. The government is an asset to the community or nation that delivers a wide range of values. Internal improvements make it a more valuable asset to the public. The second way of generating value has three forms: financial, political, and social. Financial value results from lowering the cost or increasing the efficiency of government or delivering direct financial benefits to the citizens. Political value consists of increasing participation, fairness, transparency, legitimacy, or conferring political capital to elected officials or citizens. Social returns include increased social status, stronger relationships, or opportunities; increased safety, trust in government, and economic advantage.CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 6 13. Figure 4. Public ROI Value Propositions SecondaryPerformancePublicGain Cost * ** Business GoalInternal Direct Public-at-ITor ProblemInvestment* Agency*Citizen*LargeReturnsBenefit BenefitPrivate* Cost (?) Policy/ * Vendor/*Electoral IndustryBenefitBenefit ** *Development Risk *General Economic * Benefit Risk Development Direct Benefit FlowIndirect Benefit FlowDirect BenefitAccumulationIndirect BenefitAccumulation CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 7 14. Section III: A Public Value Framework for Government IT AssessmentT his framework is designed to assist government IT executives and analysts in understanding and measuring the value toIn this way, the framework offers both asystematic way of thinking about publicvalue and a way to apply that thinking to the public of government IT investments. The par ticular IT investment decisions. The main goal is to produce an assessment of drilling down process is necessary to tailor a public returns that is credible, persuasive, specific public value assessment to the and highly relevant to the investmentnature of a par ticular investment decision. decisions faced by the government. We useThe framework shows how to take into value here, rather than return to emphasizeaccount how public value can change the broad scope of the framework. Most across the many interests of citizens and methods for assessing return on investment groups in interacting with governments. In focus on financial or economic metrics; this the morning, for example, an executive approach includes a much broader view of doing business with the government may how IT investments can produce results ofthink about how IT speeds payments on her value to citizens or to the society as a government contracts, in the evening while whole.4 This concept of value includes morehelping with homework she may observe than the usual financial or economic metrics how computers improve the quality of common in ROI analysis. It is a new andschools, and while watching the news on TV expanded way of understanding the resultsat night she might learn how a new crime of government IT expenditures. mapping system makes the neighborhoodsafer. These ways of thinking about public We call this set of ideas a framework to returns include both easily measured value, indicate that it is more than a particular like improved financial flows, as well as method for public value assessment. It ishighly subjective ones like public safety, broad in scope so that it can be applied toservice quality, or government integrity. A virtually any government IT investment, from framework for public value assessment must simple Web sites to governmentwide provide a way to deal with these many information systems and architectures. It is perspectives and possible measures of broad in scope because this range of public value. investments requires a comparable range of assessment methods. Our framework, Not every aspect of public value is relevant therefore is a way of thinking about and for a par ticular IT investment. The organizing the analysis of a family of Washington State Digital Archives project, problems that can encompass many meth- for example, had no particular public safety ods. A spreadsheet, for example, is arelated goals, but is of considerable value to framework for working on a broad class ofgenealogists and local historians. Our problems or analytical tasks. Any par ticularframework begins the process of narrowing spreadsheet may include specific methods,and focusing by starting with the three basic such as scenario analysis or a net present elements of analyzing public value: the value calculation. Instead of guiding theinvestment, the government operations assessment process in terms of calculationsaffected, and the stakeholders. At the in a matrix, however, this framework providesbeginning of a public value assessment, an an analysis process that starts with a highanalyst often knows that there are many level view of the IT investment and then connections among theses three elements, drills down through successive steps tobut those connections are poorly specified identify the specific measures and methods and understood. The situation might be that will reveal and document public value.described by the overlapping parts ofCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 8 15. Figure 5 below. The public value is to be Analyzing those connections can, in found by unpacking, so to speak, the area ofprinciple, start in any part of the problem. overlap. The task of the assessment,Ultimately, of course, these three elements therefore, is to identify the connections and must be considered together in order to gather data about how the IT investment ensure that the value assessment for any produces value for the relevant par ticular investment project or system is stakeholders. tailored to the specific value context, investment situation, and the beneficiaries Close attention to all three elements isinvolved. In practice, however, the essential. The connections among theseassessment must begin somewhere. The elements are the keys to a fully informed framework presented below describes the public value assessment. The frameworklogic and methods that guide the provides a way of describing theseassessment process. connections to show how public value is generated and the risks involved therein. The sequence of activities shown for an Focusing on one or two elements alone assessment is not designed or intended, cannot reveal the necessary scope ofhowever, to suggest how IT investment public values involved in an IT investmentdecisions should be made. or how they can be assessed. The risks of slighting one or another of these elements in an assessment can be substantial. In addition to missing significant public returns, such limited thinking can lead to stakeholder resistance, flawed technology decisions, or poor integration with or disruption of business processes.Figure 5. The Basic Elements in the Public Value Framework The Stakeholders Public ValueThe Technology Government InvestmentPrograms & Operations CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 9 16. A. The Framework general, however, the sequence follows the path shown in Figure 6 below.Strategy The steps illustrate how the process ofThe framework describes public value value identification and measurementassessment as a series of steps to gathercarries through from the goals of an ITspecific data and use it to answer questions investment, through the value generatingthat lead to public ROI results. The results mechanisms of the business processes,can include measures and documentation connection with stakeholders, to specificof public returns, risk analysis of the threatsdata and reporting. These are the mainto achieving those results, and suggestionscomponents of the framework.for presenting results to decision makers.Each component of the framework dealsBy starting with a deliberately broad scope,with a different set of questions andthen narrowing to specific questions forpossible data to use in subsequent steps,each project, this approach addresses oneand so for th. By working with the ideas and of the shortcomings of previous work onmethods presented in the framework, theROI for government IT, namely the narrowanalysis can produce a public ROIscope of possible value questionsassessment that is broad in scope, yet considered. By expanding the initial scopetailored to the par ticular investment ofof analysis to include a broad view of publicinterest.returns, the framework can lead to a more comprehensive result. The framework thus does not replace or supersede the existingThe steps are not intended to be used in a methods developed by others for assessinglock-step manner. The questions and returns to government IT investments.inquiry involved in one part of the frame- Rather the framework shows how thesework may be obvious and easily answered methods can be used in assessment, whatin some cases and quite complex and additional value issues should bedifficult in others. It may be necessary to addressed, and how additional assessmentexplore later steps before preceding ones methods may be used.are finished, and then cycle back. InFigure 6. Steps in Applying the Public Value FrameworkCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 10 17. B. How Does GovernmentFigure 7. Type 1: Direct Service IT Investment Link to the Impact Public?government V1 with old IT The question of how an IT investment can Citizen/ impact public value can be answeredOfficial V2 government initially in terms of three impactwith new IT mechanisms. These three, alone or in combination, describe the general ways ITReturn = V2 V1 investments can connect with the public, as revealed in our research. Identifying these linking mechanisms is an important step in the analysis process because it leads to athese investments go beyond the important more detailed examination of impact but obvious cost and time savings to include pathways than might otherwise occur. IT attention to service quality, access, equity, investment can enhance public value inand the full range of values described many ways, some of which are only below. indirectly related to the performance of a particular IT system. Attention to these2. Indirect Service Impacts indirect and more complex mechanisms can help ensure a full picture of public value results is developed. Indirect service impacts (Figure 8) occur when back office or infrastructure invest- ments produces changes in a government 1. Direct Service Impacts business process. As in Type 1 mechanisms, the value to the public is reflected in the Direct service impacts (Figure 7) occur whenchanged interaction or transaction with a IT is embedded directly in a service delivery government business process. However, the process, generating service changes thatprocess change results from an IT enhance value to the citizen or officialinvestment at least one step removed from involved in the service process. The netinteraction with the public. Because of this public value is just the difference between indirect route, it may be more difficult in the value of the new and the old. Value inType 2 cases to trace the links from the IT this sense, and in the other mechanisms asinvestment clearly and unambiguously to well, is assessed from the point of view of the public. Risks to the effectiveness of the the public participant in the transaction, notinvestment are higher as well, due to the government. The total value may be adependence on business process changes composite of several separate measures, that are independent of the IT itself. The IT such as lower cost to the citizen, plus investment may have potential results increased satisfaction with quality ofspread over many business process and service, plus enhanced trust in government. may interact with other technologies, further The specific kinds of value involved areobscuring the impact linkages. described in more detail in a later section. The ERP system implementations in Austria, The e-government investments in our caseIsrael, and Pennsylvania that were studies and much of the literature on e-examined in our case studies are excellent government provide many examples of thisexamples of this kind of investment. The kind of investment. These include systems primary impact of these systems is on the for fee and tax payments, license application internal, back office operations of the and renewals, obtaining information, filing government. How to identify and describe forms, etc. The public value propositions for these linked back office operations isCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 11 18. Figure 8. Type 2: Indirect Service or Impact old governmentold ITV1business processinfrastructure Citizen/ OfficialV2new governmentnew ITbusiness processinfrastructure Return = V2 V1discussed in a later section of theare more complex. The direct valueframework in terms of business reference mechanism shown here as Vd, is the samemodels (see Figure 10 on page 15). The as a Type 1 or Type 2 mechanism,impacts on these internal operations may beproducing a value increment for a citizen,very positive and extensive, and still remainofficial, or group (shown in a simplified way).hard to trace to many kinds of value gains In this type of system, the new IT is alsofor individual citizens or organizations. This linked to changes in the environment andis an impor tant part of the public valuerelationships between the direct beneficiaryproblem dealt with in detail in the valueand other entities (persons, groups,types discussion to follow. ERP systems also organizations) in the public arena. Thiscontribute to public value generationindirect mechanism can result in additionalthrough the Type 3 mechanism discussed public value flowing from interactionsbelow. outside the government, between other persons or organizations. The changes in the environment may also produce direct3. Mixed Service & Environmental value beyond the direct transactions.Value Impact There are many examples of this type of value mechanism in our cases and in otherIn this third type of impact mechanism settings. In the Washington State Digital(Figure 9 below), the links to public valueArchives case at least two such impactsFigure 9. Type 3: Mixed Direct, Indirect Service & Environmental ImpactenvironmentCitizen/ Citizen/ government ViVdOfficial Official with new IT Return = Vd + ViVi = indirect value Vd = direct value CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 12 19. occurred. The ability of the county auditorsthe same outputs or goals with lower to shift responsibility for preserving digitalresource consumption. In our Austria records to the state archive produced and Pennsylvania case studies, for savings in cost and workload at the countyexample, new ERP systems helped level (Vd). This allowed county auditors to achieve substantial efficiencies in improve services to their citizens in other financial management and other core areas (Vi). In addition, the accessibility of administrative functions of government. the government records online increases the overall environment of transparency for Increases in effectiveness the government, benefiting all citizens. In increasing the quality and/or quantity of the Service New Brunswick case, Web the desirable thing. Our case study of access to company registrations providedService New Brunswick, for example, direct value to citizens such as accountants, repor ts how an online registry for land lawyers, financial institutions, and thedata can contribute to improvements in general public searching for such businessproper ty tax administration. information (Vd). In addition, the decision by CGI to locate its Global E-Government Enablement providing means or Headquarters in Fredericton, New Brunswick, due in large part to its strongallowing otherwise infeasible or partnership with Service New Brunswick, prohibited desirable activity, or benefits the economic development of thepreventing or reducing undesirable city of Fredericton and surrounding areas inevents or outcomes. In our Washington the province (Vi). In a different way, puttingState Digital Archives case study, for some forms of information online, such as example, putting birth and marriage the Toxic Release Inventory in the US,records online enabled research by enable citizens to obtain benefits from other local historians and genealogists. transactions, such as lawsuits against polluters (Vi).5Intrinsic enhancements changing the environment or circumstances of a stakeholder in ways that are valued for C. What Kinds of Impactstheir own sake. For example, our Israel Matter for Public Value?case showed how enhanced financial accounting and reporting in the Merkava ERP opened government Just identifying these operationalfinancial decision making to greater mechanisms in general terms, however, transparency. does not tell the full value story. Each mechanism can involve more than one kindThe examples above illustrate public returns of value generator. These mechanisms show that accrue in addition to, and largely how IT investments can link to increasedindependently of, internal efficiency gains public value, but not how that increase isthat may accompany the IT investment. produced. Each mechanism has theThese value generators can also operate potential to generate more than one kind of together, increasing the overall return. public value increase, depending on the Research by the local historians mentioned details of the situation. The framework above was enabled by access to previously recognizes four basic kinds of public value inaccessible records; online access to these generators, listed below, each with a records also made research much more different range of measurements and efficient compared to paper records. Access implications for assessment:to crime mapping information by citizens could also improve the efficiency ofIncreases in efficiency obtainingchoosing a place to live or locate aincreased outputs or goal attainment business.with the same resources, or obtainingCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 13 20. D. What is the Investment? The first part of the linking takes advantageLinking IT to Goals andof the comprehensive integrated descriptions of government goals andBusiness Processes.processes that are found in the enterprise architecture work currently underway for US and other governments. For our purposes1. Links to the Business of here, the most useful comprehensiveGovernment description of government processes is in the US Federal Enterprise ArchitectureThis analysis of public value generation (FEA) Business Reference Model (seen inrequires identifying how the IT investment Figure 10).6 With some minor modifications,project links to government goals andthe components in this FEA model can beactivities. IT investments generate value in used to identify the business operations andrelation to the policy and administrativegeneral goals of an IT investment in mostsetting in which they operate. The goals and government agencies. Local governments,business processes provide links between for example, do not typically have defensehow the technology operates and theand national security operations. Theinteractions with stakeholders that generate supporting documents that form part of thevalue. The analysis, therefore, includes Business Reference Model containlinking the investment to the relevant descriptions of each of the components.government goals, operations, and business These descriptions can be used to clarifyprocesses. meaning and help identify which compo- nents are linked to a particular IT invest-This linking process is more complex than it ment.may appear, requiring a comprehensive andreasonably detailed picture of governmentWhile comprehensive, this FEA model is notgoals and operations. Such a picture willthe only enterprise architecture model thatensure that all the relevant links between may be used. There are a number ofthe technology and business processes will comparable models or business processbe identified. Many government ITframeworks that could work as well, such asinvestments have potential links acrossthose developed for the EU, the UK, Hongmany agencies and processes. The Kong, and Malaysia.7 The NationalWashington State Digital Archives, for Association of State Chief Informationexample, collects records from hundreds of Officers (NASCIO) has an enterprisestate and local agencies. The Merkavaarchitecture model, adapted for US stateproject in Israel will eventually involve allgovernments, that may also be used. Thegovernment agencies. The Pennsylvania IESEuropean Commission also suppor tssupports human resources management, additional framework activity through thebudgeting, and other administrativeAthena Project, which has published similarfunctions for all executive agencies in thedocuments.8Commonwealth. The relevant businessprocess setting of an IT investment, Using this model, the analysis proceedsbytherefore, can be quite extensive. To deal identifying where the IT investment links towith this contingency, this part of thegovernment business. This linking can beframework is based on just such abased on the new technologys statedcomprehensive picture. This is a usefulgoals, organizational location, and intendedanalysis strategy because it helps drawoperational profile. The detailedattention to value generating aspects of the descriptions of each Business Referenceinvestment that might be missed becauseModel component are useful in identifyingthey result from indirect effects or complex these links. If needed, those responsible forinteractions across many agencies or the analysis can enlist a team of IT andprocesses. CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 14 21. Figure 10. Elements in the FEA Business Reference ModelServices for Citizens Community and social servicesCorrectional activities Defense and national securityDisaster management Economic development Education Energy Environmental management Purpose ofScience and innovation Health GovernmentIncome securityInternational affairs & commerce Law enforcementJudicial activities Natural resourcesTransportation Workforce management Financial management * Information& technology management *Mode of DeliveryGovernment Service Delivery Financial VehiclesMechanisms usedto achieveDirect Services to Clients* Financial assistanceKnowledge creation & Management Credit & insurancepurpose Public Goods Creation Transfers to local governmentsRegulatory compliance & enforcement Support Service Delivery Legislative relationsPublic affairsGovernment Regulatory development Controls & oversight Operations Support Functions Planning & resource allocation * Revenue collection Internal risk management & mitigationGeneral government Management of Government Resources Resource ManagementSupply chain management *Human resource managementAdministrative managementFinancial management* FunctionsInformation& technology management* operational experts to help ensure all bidding continues until the low bid remains relevant links are identified. unchanged for a predetermined time (e.g.,five minutes), closing the auction. The low To illustrate how to use the model, we price bidder wins the contract. The entire marked several of the components in Figure auction takes place in public view and is 10 with asterisks. The asterisks mark an recorded in detail. example of the business links we identified for an IT investment project in the MerkavaFor an illustrative example, consider the case study used for this framework: an procurement of new police radios for a law online reverse auction system forenforcement agency. Based on the nature of procurement. The agency conducting the the e-procurement system and theonline auction sets the terms of a procure- component descriptions, we see it as linked ment, publishes them, and qualifies vendorsto the five marked components in Figure 10: as potential bidders. At a predetermined law enforcement, financial management, time, a Web site is opened for bidding. Allsupply chain management, direct service to qualified vendors can then submitclients (vendors), and planning and re- authenticated price bids to a public space source allocation. on the site, visible to all other vendors. TheCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 15 22. 2. Links to Business Processes The list could be extended for fur ther analysis by adding deployment of the radios, training for users (human resourceMaking further links to business processes management), and evaluation of themeans connecting the larger goal orimpacts of the new radios use in thefunction from the Business Reference agencies and business processes whereModel to specific activities identifiable as installed. To keep this examplebusiness processes. For this step, picking manageable, however, we are limiting thethe appropriate level of detail is important.analysis to the processes listed below. TheChoosing too general or too fine-grained a components identified from Figure 10 areprocess analysis will tend to obscurelinked to the business processes in the rowsstakeholder interactions that are necessaryof Table 1 identify public value. For this step in theanalysis, we recommend another enterpriseIdentifying the business processes leads toarchitecture tool: the Zachman Framework. 9two key questions: (1) How does a businessThe descriptions in the first two rows of theprocess generate increased public value?Zachman Framework (Scope and Businessand (2) For whom? A way of answering theModel) are a good guide for the most usefulfirst question is recorded in the third columnlevel of detail. of Table 1. Our analysis of public value generating mechanisms is shown in theThis method identifies several businessdiscussion of value impact mechanismsprocesses for the e-procurement example. below.They include: determining communication needs forE. What Kinds of Publicthe law enforcement agencies Value are Produced?budgeting for the procurement Identifying the linkages between businessrecruiting and qualifying vendorsprocesses and public value generators described above carries the analysis aconducting the auction critical step farther: from how the IT changes a business process to the impact of thosepaying the vendors changes for a par ticular public orTable 1. Business Component, Process, and Value Mechanism Business ComponentBusiness ProcessPublic Value MechanismLaw enforcementconduct auctioncitizen - type 3 Financial managementbudget for procurement citizen - type 3citizen - type 3 Supply chain managementqualify vendors for auction vendor - type 1citizen - type 3 Planning & resource allocation deter mine communication needs gov't staf f - type 1 Direct ser vice to clientsconduct auctionvendor - type 1 Financial managementpay vendorsvendor - type 1CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 16 23. stakeholders. That step is accomplished bya corporations integrity and assets on looking to the value of the impact in terms ofbehalf of stockholders, public managers the interests of one or another stakeholder.and elected officials have a parallel So the framework includes an analysis responsibility for government on behalf of scheme for taking into account interests that the public. Recent financial scandals in the can apply to the full range of stakeholders.US have demonstrated how lack of The framework employs these six basic stewardship in major corporations (e.g., kinds of interests: Enron, WorldCom) can destroy their financial value. The government officials inFinancial impacts on current orour cases expressed a similar stewardshipanticipated income, asset values,responsibility in terms of, for example, usingliabilities, entitlements, and other IT to increase transparency andaspects of wealth or risks to any of the accountability. Public surveys of trust inabovegovernment show that citizens have similar stewardship expectations of government.10Political impacts on personal or Clearly this list goes well beyond the internalcorporate influence on governmentefficiency or service quality impacts of an ITactions or policy, role in political affairs,investment. Of course, attending to thisor influence in political parties or expanded concept of value raises manyprospects for current or future public problems of definition and However, our case research and results from many other kinds of inquiry show theSocial impacts on family or communityimportance of this more comprehensiverelationships, social mobility, status, andtreatment of value and interests. Many of theidentity.measurements and data implied by these interest types are found in existing ROIStrategic impacts on economic or methods as well. However, none of thepolitical advantage or oppor tunities, methods we reviewed for this white papergoals, and resources for innovation or cover the full range represented here.planning Working with this expanded range of value types represents one impor tant contribution of this framework.Ideological impacts on beliefs, moralor ethical commitments, alignment of It is also clear from this way of describinggovernment actions or policies or social public value that it supplements rather thanoutcomes with beliefs, or moral or replaces methods that assess the efficiencyethical positionsgains or savings returns captured by government agencies internally. For ex-Stewardship impacts on the publicsample, it seems safe to say that aview of government officials as faithful Department of Motor Vehicles internal coststewards or guardians of the value ofsavings from putting license renewals onlinethe government itself in terms of public are invisible and largely irrelevant totrust, integrity, and legitimacy drivers. These savings would have no detectable impact on overall tax burdens or This last intereststewardshipis benefits resulting from a shift of government deliberately not included in the ideology resources to some other service. The categor y, though it could logically fit there. information about the savings, however, is Setting it off separately emphasizes itsanother matter. If government officials make importance in the overall public valuesome political use of the cost savings framework. Just as corporate managers and information, that would represent strategic directors are responsible for stewardship ofvalue to those officials or to their political CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 17 24. allies. The public learning about the savingsthinking and include multiple opinionsmight also improve their opinion of theand information sourcesgovernments stewardship, fairness, or otherideological value. Identifying multiple stakeholder roles, internal and external to the organization setting (e.g., internal user, customer,F. Who Receives Value? vendor, developer, manager, oversight, politician, taxpayer, analyst, advocate,The Stakeholder Analysis.etc.)Identifying stakeholder expectations,Completing the analysis of value impactsrequires identifying those with an interest in influence potential, past and futurethe value generating process: i.e., thepar ticipation possibilities, and level ofstakeholders. The kind of stakeholderinterestanalysis required for this task has threemain parts: (1) identifying the persons or In this kind of analysis, continuing referencegroups (including organizations) whose to the nature of the investment and impactinterests are potentially affected,mechanisms will help produce the needed(2) identifying what their specific interestsfocus and detail.11may be, and (3) assessing their role andpotential influence in the deliver y of public The results of a stakeholder analysis havevalue. The first two par ts are necessary forimplications for value assessment and forthe analysis of value impacts and possible understanding their potential to influenceassessment strategies and so are discussed the investment project. These results can behere. The third part is more relevant to the presented in ways that show the multiplerisk analysis discussed in more detail indimensions used in the analysis. Ansection III.H below. example of results from a hypothetical stakeholder analysis is shown in Figure 11.Identifying stakeholders for the frameworkwill depend on in-depth knowledge of the This figure combines data for 14context of the investment and the agencies stakeholders roles (SA = Stakeholder A,involved in its development and use. etc.) and times of possible influence in aTypically, stakeholder analysis engages aprojects lifecycle. This kind of displaygroup of participants with extensive illustrates some of the complexities ofknowledge of the political and stakeholder analysis and the possibilities fororganizational setting of the investment interactions among different stakeholderand, hopefully, with experience in this kind groups. A more complete analysis wouldof analysis. Because the analysis is soinclude estimates of the stakeholderscontext-dependent, there are no standard influence capabilities and specific interestsprocesses to follow. There are in the project development and outcomes.consistencies, however, among the many These issues are discussed in more detail inversions of stakeholder analysis methods the section on risk analysis (section III.H).found in the management, planning, andassessment literature. These include:To focus on stakeholder interests, the illustration in Table 2 below shows how theInvolving multiple participants with wideresults of a simple stakeholder analysis canknowledge of the stakeholder be linked to the business process, valueenvironmenttypes, and mechanisms. Looking widely to identify all relevantThe business processes identified in Table 2stakeholders through brainstorming and on page 20 link to a partial list ofrelated methods to stimulate divergent stakeholder types in the columns: citizens atCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 18 25. Figure 11. Stakeholder Analysis Matrix12Type of Participation InformConsultPartnershipControlInitiationIdentification SA SCSI SESHSD PlanningSK SB SJStage in LifecycleSFSN Execution ImplementationSM SGControllingSL Monitoring and Evolution SO Closing SD SA Sx = Stakeholder X large, vendors, and elected officials. Anassumptions about what advances the actual assessment would include a more interests of those staff members. A success- detailed list of stakeholders. The value ful procurement system would provide types most likely to map onto each stake-political value and some strategic advan- holder type are entered in the cells. Thetage to elected officials, as well as enhance basis for this mapping is linking the nature their reputation as good stewards of public of the value as understood in that context resources. with detailed knowledge of the stakeholder groups. This part of the assessment can include data collection about the interestsG. How to Demonstrate the and expectations of stakeholders through Value? Identifying Variables interviews, surveys, or focus groups if needed. The entries in the column for theand Methods. citizens at large, for example, are based on the assumption that their interests in Using the framework up to this point has government stewardship would be served identified many kinds of data relevant to by the e-procurement system. While a assessing public value. The next step is to reasonable assumption, this could be choose the specific variables or points of confirmed by collecting opinion data from aobservation for collecting the assessment sample of citizens, or relying on previous data. This is as much a practical problem as opinion research. Similarly, the entries in thean analytical one. Many variables may be column for government IT staff are based onCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 19 26. Table 2. Stakeholder Value MatrixStakeholders BusinessValue ImpactCitizens atElectedVendors Process GeneratorTypeLarge Officials Budget forintrinsic 3 stewardship procurement Deter mine intrinsic,ideology, political,3 needsef fectivenessstewardship strategic Qualifyef ficiency,ideology, financial,political, 1,3 vendorsintrinsic stewardship strategic strategicef ficiency,political, Conductideology, financial,intrinsic, 1,3strategic, auctionstewardship strategicef fectivenessstewardshipef ficiency,financial,political, Pay vendors 1,3stewardshipintrinsic strategic stewardship relevant for each combination of ROI methods described below include manystakeholder and value type. A few variablesvariables and additional methods that canthat would be relevant to this example are be helpful in that task. However, the moreshown in Table 3.the choice of variables can be tailored to the specific public value context, the more validThis table illustrates two important aspects and persuasive the assessment is likely toof its role in the framework and assessmentbe.process. First, it is highly unlikely thatapplying the framework to any IT investmentAn actual public value assessment shouldwould lead to variables in all, or even most be based, of course, on the best availableof the cells of this matrix. This matrix can beinformation. But actual assessments takethought of as an heuristic device, prompting place in practical settings of limitedquestions about what might be useful and resources and access to data, plus beingavailable variables for each row and columnpar t of the additional work needed forwithout expecting to fill them. The second isinternal returns and costs. The prioritythat identifying a specific variable relies on setting described in the risk analysis sectioncombining information about stakeholdercan narrow the field to only the mostinterests, the value type, the impactimportant public value outcomes. Themechanisms, and the context. This is a section below describes additionalcomplex and demanding process. Thisstrategies for choosing the appropriatesection offers additional guidance onvariables and analysis methods.choosing the best public return variables fora given assessment. In addition, the otherCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 20 27. Table 3. Stakeholder Variables by TypeStakeholder Impact Variables Value Type Citizens at LargeVendorsElected OfficialsFinancial local economicincreased revenuedevelopment Political suppor t from vendorspositive mediacoverage Socialenhanced reputation inagencies receivingradios Ideological increase level of marketenhanced reputationcompetition for promoting marketcompetition increase in citizenaccess to infor mationabout procurement:greater transparency Strategic increased market shareenhanced reputationfor being moder n &progressive increased knowledgeabout competition Stewardship increased trust inreputation for frugalitygover nment & good management1. Variables and Analysis The scope of this framework, as applied to a particular IT project, could encompass a Methods very wide range of data types. Many kinds of quantitative data from financial sources, The choices of variables and analysis operations research, and surveys are methods for the empirical par ts of a publicappropriate for statistical analysis, value assessment should be considered modeling, and simulations. Many of the together. In terms of basic measurement and social, political, and intrinsic value returns analytical methodology, what constitutes an can be expressed in normalized scales, or appropriate analysis depends, in part, on may best be revealed in qualitative terms or the types of data and variables involved. in simple dimensions that are not suitable CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 21 28. for much quantitative manipulation. To helprequires an inference, since value does notguide the assessment, given the very large stalk about wearing a label. Valid inferencesnumber of possible public return data typesabout value can be formed from qualitativeand variables, the framework provides twoas well as quantitative data, contentkinds of resources. The first is a general analysis as well as statistics. Taking thesescheme suggesting variable types and four principles into account, choices madesources for different value types (see Table about how a specific analysis proceeds4). The second consists of summaries of theshould be based on three criteria: 1) Whatapproaches and variables available in aconstitutes the best data? 2) What kind ofrange of existing ROI methods that can beanalysis is appropriate to that type? 3) Whoof value in completing an assessment (seewill be the audience for the conclusionsTable 5).reached? The best kind of data available will be specific to the operational andA wide range of possible public value data stakeholder context. The kinds of analysiscan be identified by the methods presented appropriate to various data types are shownhere. The framework approach is based on in summary form in Table 4.13the assumption that virtually any kind ofdata can be useful in describing publicBeyond these general considerations, thevalue creation, from the most precisechoice of variables and analyses for anquantitative figures available from financialassessment can draw on a volume ofor physical measurements, to material as existing work on ROI methods for guidance.diverse as the content of blogs or These methods, summarized in Table 5, varyobservation of user or customer behavior. Awidely in the number and type of variablesconclusion about public value creation used, the scope of public value co nsidered, Table 4. Variable Type for Public Value Data Variable TypeExamples Analysis Methods averages, totals, dif ferences,percents, parametric &financial data, time, age,nonparametric statisticsratio scale distance, frequency, transaction(cor relation, regression, etc.),volumesimulation, time series,optimization averages (with assumptions),opinion scales, rating scales,frequencies, scale analysis,inter val scalegrades, Liker t scalesnonparametric statistics (x2,etc.), cluster analysis cor relation, frequencies, scalingordinal scalesrankings, rating scalesanalysis gender, ethnicity, location,frequencies, clustering, networknominal variablesrelationships, user behaviors analysis content analysis, wordtext/discourse data inter views, blogs, news repor tsfrequency, semantic analysis user studies, field obser vationprocess maps, networkobser vational datanotes, photos, film/video analysis, behavior patter nsCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 22 29. Table 5. Summary of Selected ROI Methods and Models Methods/Models Value MeasuringBest Value Gar tnerSocial Retur n on Method Methodology, US Perfor mance Gover nmentInvestment, DescriptorsFederal CIOIndicatorsPerfor manceREDF (US)Council (ODPM, UK)Framework Time Orientation prospective retrospective retrospectiveretrospective Time Frame cross sectional longitudinalmixedcross sectional* Institutional publicprivate public publicFocus Gover nment national/federaln /alocal UK generalLevel generalProjectlocal gover nment ITsocial ser vices perfor manceOrientationprogramsorientationmixed,predominantlypredominantly predominantlyVariable Types predominantly quantitative quantitative quantitative quantitative+qualitative +qualitative highy general categoriesVariablequantitative, + examples &highly specific highly specificSpecification employing suggestionsindexesmixed mixed; manymixed inter nal & primarily inter nal,inter nal/exter nal inter nal andPROI Orientationexter nal some public with exter nalexter nal emphasisvariables PROI Variable limited nar row, limitedbroad large & diverseSetlimited toAnalytical some method method fully method specified variableMethodssuppor tspecified definition Analyticalunknown modest suppor t par tial suppor t not includedProcess Suppor t Risk Analysisincludednot includednot included limited inclusiondecision tool suppor t suppor t materialsDecision Suppor tnone includedmaterials Presentation examples results matrices minor suggestions noneToolsprovided*Includes some longitudinal variables CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 23 30. Table 5. Summary of Selected ROI Methods and Models (continued)Methods/Models Perfor manceMAREVA Demand andPublic Ser viceReference Model Method(Gover nment of Value AssessmentValue Method,(US Federal Descriptors France)Model (Australia) AccentureEnter prise Architecture) retrospectiveTime Orientation prospective combinedretrospective cross sectional*Time Frame cross sectional longitudinalmixed Institutional public publicpublicpublicFocus Gover nment national-France national/generalgeneral national-USLevel ProjectprogramIT general generalOrientationperfor mance mixed, largely mixed, largelypredominantlyVariable Types mixedquantitative quantitativequantitativemixed categoriesmostly specific predominantlyVariablelargely specified and someperfor mance &variableSpecification specified cost metricscategoriesmixed inter nal and primarilymixed, largelyPROI Orientation exter nal; fewinter nal, some mixedinter nal political public PROI Variablever y large &limited large, diversenar row, limitedSetdiverse Analytical highly specified &analysis methodincluded not includedMethodsdetailedspecified Analytical highly specified & included includednot includedProcess Suppor t detailed includedRisk Analysisincludednot includedn/asuppor t materialssuppor t material includedDecision Suppor tmaterial included and guidelinesmaterials Presentation standard par tial includedpar tialToolsdisplays*Includes some longitudinal variablesCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 24 31. and the level of analytical detail and technique included. Some are intended for use prospectively, in planning for and justifying an investment. Others are aimed primarily at showing impacts of investments after the fact. They also vary in terms of the degree to which they deal with both internal and public value results of the investment, and whether they are designed specifically for IT or government investments generally. The summary of these methods or models in Table 5 can help in the selection of variables and analysis to fit the IT project.The SROI (Social Return on Investment) model, the only private sector oriented one in the summary, is included for its special features. Its private orientation refers not to the commercial sector but to a private philanthropic orientation. The method was developed by a San Francisco foundation to assess its social and economic development programs. It illustrates not so much how to assess an IT investment but rather how to deal with personal and community impacts in a systematic and comprehensive way. CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 25 32. H. What Can Interfere Withmitigation strategy. To identify and evaluatethreats, we return to the overall valueor Prevent Public Returns?proposition schematic presented earlier(Figure 12).The Risk Analysis.In this figure, we identified two kinds of risk:Government IT innovation is risky business. 1) development risk and 2) benefit risk.Evidence from many surveys and casesDevelopment risk, simply put, is risk that thesuggests that the threats and other development and implementation of the ITchallenges of IT projects often overwhelm will fail outright or will not perform asthe capabilities of the developers anddesigned and intended. Benefit risk appliesimplementers. Data from the private sectorto whether the IT investment will fail totells a similar story. For this paper, we focus produce the envisioned benefits in spite ofour attention not on the full range of risks to being successfully developed andgovernment IT project success, butimplemented. For the public valueconcentrate on the public return aspects of framework we focus on benefit risk. Ofthat problem. course any threat to the development of theIT is an indirect threat to public returns.The distinction between the public return However, several of the ROI methodsand more generic aspects of risk analysis isdescribed below include adequate analysisnot, however, a simple one. The approach is methods for development risk. Thesethe same: identify and evaluate threats,methods do not deal adequately, however,develop and evaluate response methods,with the additional threats and issues inand produce a summary analysis andbenefit risk.Figure 12. Public Return on Investment Value Chain Secondary Performance PublicGainCost ***Business Goal InternalDirect Public-at- IT or ProblemInvestment* Agency *Citizen*LargeReturns Benefit Benefit Private*Cost (?) Policy/* Vendor/ *Electoral IndustryBenefitBenefit *** Development Risk *General Economic* Benefit Risk DevelopmentDirect Benefit Flow Indirect Benefit Flow Direct Benefit Accumulation Indirect Benefit AccumulationCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 26 33. Table 6. Examples of Threats to Public ValueSources of Threats Threat Type Theory Failure Exogenous Factors overestimate of public demand confounding or cancellingleads to non-use of ser vice; ef fects: down-tur n in economyflawed understanding of impactcancels financial advantages;To creation of retur ns mechanisms; misunderstandingchanges in gover nment counterof stakeholder interestsstewardship outcomes; politicalscandal cancels positive mediarepor ts lack of or er roneous baselineindirect and second-orderdata; mis-specification ofef fects undetectable; dataoutcome variables; inappropriateinaccessible due to privacy orTo detection of retur nsanalysis methods or models; sensitivityaccounting systems based oninappropriate data definitionsand models The benefit risks associated with the managers, and policy makers. This will creation and assessment of public returns provide a basis for accurate estimates and come from threats to the creation of theconcentrating attention on the threats with returns and to their detection. And there the highest combined loss-likelihood appear to be two main sources of thoseestimates. In some cases, the value, cost, threats: one is what we will call theory and loss-likelihood estimates can be failure and the other is exogenous factors.quantified to yield decision tools for moving In theory failure, the underlying assumptions ahead with an investment. The displays in or theory on which the project is based are Figure 13 and Figure 14 on the next page, flawed or simply wrong. One such theory show such a result from the US Federal CIO failure caused the US Department of Councils Value Measurement Method, Education to abandon a multi-million dollar which employs voting and other methods to pilot project for online college studentprioritize and estimate quantitative values financial aid administration. Developed for cost and their related risks. The decision without significant par ticipation from college boundaries shown in the figures come from financial aid officers, the system did notmanagement decisions or some other attract supporters and generated much deliberative process.15 stakeholder resistance. 14 It is important to use both sources and types of benefitsThe threats due to difficulty in detecting threats in a full risk analysis, as suggested indirect or second-order effects deserve by the two-by-two array in Table 6. some added discussion. Public value outcomes can extend beyond those involved The risk analysis process can then use thein the direct interaction with government. For threats identified in this way to estimate theexample, drivers are the direct beneficiaries potential loss or cost that each threat of an online license renewal system. represents and the likelihood of the threat However, the scope of possible indirect materializing. These estimates should bevalue outcomes and beneficiaries can be based to the extent possible on input fromvery broad. Residents in the neighborhoods stakeholders, analysts, users, developers,of the license renewal agencies will benefitCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 27 34. Figure 13. Example of a Value-Risk Boundary Analysis Figure 14. Example of a Cost-Risk Boundary AnalysisCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 28 35. to some degree from reduced traffic one step can easily lead to revisions in congestion and pollution. Shopkeepers inconclusions or data collected at preceding that neighborhood may lose business for the steps. This is par ticularly so for the risk same reasons. Second order effects may be analysis steps in the process. It is not even more diffused and difficult to detect. feasible to complete the risk analysis for Learning to trust the online process forachieving any particular public value return license renewal, for example, can result in until the details about that return and how to more use and greater benefits from usingassess it are known. Loops can occur in the other online services.process when a step produces unexpected or challenging results. For example, Since risks are tied both to stakeholders and identifying all related business processes possible value outcome variables, pursuingfor a large, complex project may produce too this line of reasoning can lead to a very many results to carry forward in the large, and likely infeasible, list of riskanalysis. That would require looping back to analysis factors and tasks. To work withinreduce the scope. So the potential user resource constraints will require limiting theshould view this process summary as a risk analysis to the most important value rough guide rather than a lock-step proce- variables and stakeholders. Setting prioritiesdure. for this kind of analysis will therefore be essential, and must be based on the goals ofThe process also represents a the project and stakeholder value estimates.comprehensive approach to public value assessment. We argue that all the steps are needed to ensure that the full range of I. Overview of Using thepublic value returns will be included in any assessment and report. Some small or very Framework.narrow projects, of course, may have limited public value potential and not require such The framework is a combination of individualan extensive review. But the research and analysis steps combined into an overall analysis leading to this framework have process or plan for public ROI assessment. Aclearly demonstrated that the scope of summary of that overall process is shown in public value possible from government IT Figure 15 on pages 30-31. As shown there, investments can far exceed what an the process follows the arrows, starting with internally focused, financial assessment will a clear understanding of the larger context reveal. Unless assessments use a much of public value generation. The investmentbroader view, as represented in this goals that emerge from the investment plans framework and other models cited above, and understanding of the context link to thethe full value of government IT investments business processes and public value impactwill not come to light. It also seems likely mechanisms. Analysis of the impactthat this same kind of analytical framework mechanisms and public return types leadswould be useful for other forms of to specific impacts listed in the stakeholder government investments. We did not, interest analysis. Identifying the interestshowever, explore that possibility for this and impacts will then guide choosing public report. value variables, analysis methods, risk analysis, and ultimately, reporting tools and techniques.For simplicity, this process diagram is somewhat idealized. It shows only the basic linear path through the various steps. In practice, the actual path is rather more likely to have backtracking and loops. Learning atCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 29 36. Figure 15. The Public Value Framework OverviewSecondary PerformancePublic GainCost ***Business GoalInternal Direct Public-at-IT or Problem Investment*Agency*Citizen*Large ReturnsBenefit Benefit Private *Cost (?)Policy/* Vendor/ * Electoral Industry BenefitBenefit * * * Development Risk *General * Benefit Risk EconomicDevelopmentDirect Benefit Flow Indirect Benefit Flow Direct Benefit Accumulation Indirect Benefit Accumulation Business Reference ModelValue Impact TypesBusiness Component Business ProcessPublic Value Mechanism Law enforcement conduct auction citizen - type 3 Financial managementbudget for procurementcitizen - type 3 citizen - type 3 Supply chain management qualify vendors for auction vendor- type 1 Planning & resource deter mine communicationcitizen - type 3 a l l o c a ti o nneeds gov't staf f - type 1 D i r e c t s e r v i c e t o c l i e n tsconduct auction vendor - type 1 Financial managementpay vendors vendor - type 1 Public Return TypesF = FinancialP = PoliticalSo = SocialI = IdeologicalSt = Strategic (Continued on next page)Sw = StewardshipCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 30 37. Figure 15. The Public Value Framework Overview (continued)Identify Risk Risk Mitigation StrategiesAnalysis Results &Reporting Value Variable and Analysis Analysis Selection Stakeholder Impact Variables Value TypeCitizens at LargeVendors Elected OfficialsFinancial local economic increased revenuedevelopmentPoliticalsuppor t from vendors positive media coverageSocial enhanced reputation in agencies receiving fur nitureIdeological increase level ofenhanced reputationmarket competition for promoting market competitionincrease in citizenaccess to infor mationabout procurement:greater transparencyStrategicincreased marketenhanced reputation share being moder n & progressive increased knowledge about competitionStewardship increased trust in reputation for frugalitygover nment& good management StakeholdersBusinessValue Impact Citizens atVendors ElectedProcess GeneratorType Large OfficialsBudget forprocurement intrinsic3 stewardship(Continuedfrom previouspage)Deter mineintrinsic, 3ideology, political,needs ef fectivenessstewardship strategicQualify ef ficiency,ideology,financial, political,vendors intrinsic 1,3 stewardshipstrategicstrategicConduct ef ficiency,ideology,financial, political,auction intrinsic,1,3 stewardshipstrategicstrategic,ef fectivenessstewardshipef ficiency, financial, political,Pay vendors 1,3stewardshipintrinsicstrategicstewardship CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 31 38. J. How to Summarize andresults, with the estimated relative magnitudes as well. The down arrow in thePresent Results? stewardship row for vendors indicates a potential loss to vendors due toThis framework for public value assessment transparency increases that diminishpresents both problems and opportunities opportunities for some forms of influencingfor summarizing and repor ting. Theprocurement. Whether this is a positive orproblems arise from the large number and negative result overall may not be clear.types of results that the assessment canproduce. For presentation to policy makers For the more quantitative results, where theand non-technical audiences, the results public value variables lend themselves toshould be as simple and accessible ascalculation and statistical analysis, manypossible. Simple char ts and summary tablespresentation and summary methods areare best for this purpose. For multipleavailable. Clear and simple visuals arestakeholders and value variables, thegenerally preferable to tables of quantitativenumber and complexity of charts maydata for non-technical audiences. For crossbecome a problem. This section discusses sectional data, a column char t, such as insome of the specific issues and alternativeFigure 16, can be used to present the samemethods available. comparative public value data converted to a ten-point index or scale.For qualitative variables, the presentation ofresults can show the presence of a value Such a multidimensional chart can beresult, and information about magnitude anddifficult to interpret, however, and some ROIdirection, if relevant and available. Usingmethods use the so-called radar char t forthe information in Table 3 (page 21), athe same data, shown in Figure 17. Thissummary display similar to Table 7 below kind of char t provides a clearer image of thecan present the types and direction of pattern of results for each stakeholder type and value dimension. However, this kind ofTable 7. Qualitative Display of Stakeholder Impacts Stakeholder Impact VariablesValue Type Citizens at LargeVendorsElected OfficialsFinancialPoliticalSocialIdeologicalStrategicStewardshipCENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 32 39. Figure 16. Stakeholder Impact Column Chart Public Value Index by Stakeholder Groups1098Public Value Index Financial76Political5Social4Ideological3Strategic2 Stewardship10citizens at largevendors elected officials Figure 17. Stakeholder Impact Radar ChartFinancial 10 Stew ardship5Political 0StrategicSocial IdeologicalCitizens at Large Vendors Elected Officials CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 33 40. Figure 18. Portfolio Risk and Value Comparisons - DVAM Financial and Social Worthdisplay becomes much more difficult to example of this form of results presentationinterpret if the number of axes or is shown in Figure 19 for the Arizonastakeholders is large. Depar tment of Revenue. Overall, the performance shown is improving from 1999The use of indexes can also be used to to 2001 and 2002 to 2003 when thecompare the relative risk and value of organization is creating increased outcomesalternative investments on a value scale ifand doing so more cost effectively.the variables can be combined into a singleindex. The Demand and Value Assessment Documentation of results through this kindModel (Australia) provides guidance on how of chart or visual device should includeto produce such a public value index and background material about methods anduse it for comparison purposes. An example measurement issues. This frameworkof that kind of result is shown in Figure 18 advocates the use of a wide range of dataabove. and analytical styles, many of which are considered controversial or suspect in someFor cross sectional assessments, moreenvironments. Decision makers and analystselaborate presentations are available, often have deep-seated biases about thepar ticularly if the value variables are validity of qualitative data or social statistics,indexed or based on quantitative data. The or other non-traditional material for ROIAccenture Public Service Value Model (PSV) analysis. Those performing a public valuecan use historical data about government assessment must, therefore, be thorough inprogram effectiveness and costs to showproviding rationale and supporting materialchanges in performance over time. 16 The for all results. They must be attentive to themodel is based on the principle that publicissues of interpretation and validity that mayvalue is created when both outcome results affect how key members of their audienceand cost effectiveness increases. An respond to the assessment results. CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 34 41. Figure 19. Historical Performance Change Model (PSV - Accenture) PSV Agency Performance Matrix (FY 1996 FY 2003)10%Higher 8%6% 1997 4%1996Outcomes 2%2001AVERAGE 0% 2003SCORE-20%-15%-10% 1998 -5%0%5%2002 10%15%20%-2% 2000-4%1999-6%Lower-8% -10% Lower Cost EffectivenessHigher The principles for SROI analysis used by the4. Replicable - The framework should REDF organization provide valuable result in similar conclusions when guidance for the conduct of the kind ofapplied by different practitioners who assessment described here. These use similar parameters (such as the principles can be applied to traditional ROI scope and options). Thus, results analysis as well, but seem particularly well should also be comparable over time suited to the public value issues involved and among organizations, at least with this framework and related methods. among analyses that use similar options The principles make an appropriate bridgeand where the options are clearly noted. from the general ideas and methods presented here to the difficult work of 5. Transparent - The process by which the carrying out public value and ROIanalysis was prepared, and the contextassessment in practice. in which results should be seen, shouldbe transparent. SROI Design Principles: 17 6. Credible - The results should be 1. Feasible - A basic SROI Analysis should credible to investors, purchasers,be something any organization can mangers, and other users.afford to prepare itself. 7. Integrative - The framework should 2. Accessible - The process should berelate to, and where possible integrateunderstandable and relevant towith, other approaches to understandorganizations at various stages ofsocial value.development. 8. Avoids misuse - Proper application of 3. Rigorous - The method should be the framework should reduce the risk ofsubstantive and well-executed, andmisuse of, or misleading, SROI numbersbased upon premises that are validatedor informed practitioners.CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 35 42. 9. Open source - The framework should continuously be informed and improved by the collective wisdom of practitioners in an inclusive, iterative process. 10. Useful - Applying the framework shouldresult in information that enables usersto make decisions or take actions thatfurther their goals. CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 36 43. IV. Reflections on the Framework and the Value of Public ValueT his framework grows from the rather simple principle: that the value of a governments investment in IT should be of assessing public value that matches their greater scope and complexity, a way that can build the needed public support and guide assessed from the point of view of thedevelopment. In this light, the cost of using a public it serves. That principle leads us toframework such as this should be easily identify two distinct but equally important justified. Moreover, the new knowledge types of public value: delivering benefitsabout public value possible from such an directly to citizens and enhancing the valueassessment can help guide other forms of of government itself as a public asset. investment and contribute to long-term From this seemingly simple beginninggovernment improvement. grows the rather high level of complexity involved in working through this framework and its potential application. That complexity derives in part from the way these ideas radically expand the possible scope of inquiry needed to identify and document public value creation. That expanded scope brings with it a host of measurement problems that emerge when the many social and political outcomes come into play. The cost of this expanded and more complex assessment can be quite high.So government executives and IT planners can rightly question whether assessing public value is wor th the effort. Part of the answer may be that they will have no choice. As we noted earlier, elected officials have begun insisting on more comprehensive cost and return analyses for IT investment proposals. Conventional approaches to ROI analysis may simply be inadequate for these increased demands.More importantly, however, the desire for a more comprehensive and robust justification for new IT investments reflects their greater complexity and ambition. The low-hanging fruit available from earlier IT investments, such as establishing Web presence and automating simple service transactions, has typically been harvested. More substantial improvements in government are now possible by exploiting the integrative and transformative potential of IT, but they require much larger investments. These projects require a way CENTER FOR TECHNOLOGY IN GOVERNMENTADVANCING RETURN ON INVESTMENT ANALYSIS FOR GOVERNMENT IT: A PUBLIC VALUE FRAMEWORK 37 44. V. Appendix: The Research Basisfor the Public Value Framework A. Consultative Workshop what should be done, but not on what has already been done. On July 13, 2005, the Center for TechnologyGovernments have trouble harvestingin Government, in collaboration with SAP,savings, which often get moved aroundhosted a consultative workshop onthe budget.assessing public return on governmentinvestments in IT. A select internationalThere is no straightforward quantitativegroup of 22 experts from government, bottom line value measure for ROI inacademia, and the private sector werepublic sector.invited to meet with researchers from CTG todiscuss the core issues and the