Implementation Completion and Results Report · Web viewIMPLEMENTATION COMPLETION AND RESULTS...

Document ofThe World Bank

Report No: ICR00003850

IMPLEMENTATION COMPLETION AND RESULTS REPORT(GRANT NO: TF010455)

ON A

GRANT AMOUNT OF USD $ 2.3 MILLION

TO

THE OFFICE OF THE AUDITOR GENERALGOVERNMENT OF NEPAL

FOR

STRENGTHENING OFFICE OF THE AUDITOR GENERAL PROJECT

May 8, 2017

Governance Global PracticeNepalSouth Asia Region

CURRENCY EQUIVALENTS(Exchange Rate Effective February 23, 2012)

Currency Unit = Nepalese Rupee = NPR 1.00 NPR = US$ 0.012US$1.00 = 78.61 NPR

FISCAL YEAR July 1 to June 30

ABBREVIATIONS AND ACRONYMS

ASOSAI Asian Organization of Supreme Audit InstitutionsCAAT Computer Assisted Audit TechniqueCSO Civil Society OrganizationsDFID Department for International DevelopmentEWP Electronic Working PaperFCGO Financial Controller General’s OfficeFMIS Financial Management Information SystemGDP Gross Domestic ProductICR Implementation Completion ReportIDF Institutional Development FundINTOSAI International Organisation of Supreme Audit InstitutionsIPSAS International Public Sector Accounting StandardISN Interim Strategy NoteISR Implementation Status and Results ReportISSAI International Standards of Supreme Audit Institutions IT Information TechnologyM&E Monitoring and EvaluationMDTF Multi Donor Trust FundNAMS Nepal Audit Management SystemNPSAS Nepal Public Sector Accounting StandardsOAGN Office of the Auditor General of NepalPAC Public Accounts CommitteePDO Project Development ObjectivePEFA Public Expenditure and Financial AccountabilityPFM Public Financial ManagementPRAN Program for Accountability in NepalQA Quality AssuranceSAI Supreme Audit Institution SAI PMF Supreme Audit Institute Performance Measurement FrameworkSIL Specific Investment LoanSOAGN Strengthening Office of Auditor General Nepal (Project)SPFM Strengthening Public Financial Management (Project)TSA Treasury Single Account

Vice President: Annette DixonCountry Director:Country Manager:

Qimiao FanTakuya Kamata

Senior Global Practice Director: Deborah L. WetzelPractice Director: Edward Olowo-OkerePractice Manager: Fily SissokoProject Team Leader: Akmal MinallahICR Team Leader: ICR Main Author:

Krishnamurthy Sankaranarayanan /Arvind Prasad Mantha

i

NepalStrengthening Office of the Auditor General of Nepal

CONTENTS

Data SheetA. Basic Information.......................................................................................................1B. Key Dates....................................................................................................................1C. Ratings Summary........................................................................................................1D. Sector and Theme Codes............................................................................................2E. Bank Staff....................................................................................................................2F. Results Framework Analysis.......................................................................................2G. Ratings of Project Performance in ISRs.....................................................................7H. Restructuring (if any)..................................................................................................7I. Disbursement Profile...................................................................................................7

1. Project Context, Development Objectives and Design................................................82. Key Factors Affecting Implementation and Outcomes.............................................103. Assessment of Outcomes...........................................................................................154. Assessment of Risk to Development Outcome - Rating: “Moderate”..........................205. Assessment of Bank and Borrower Performance..........................................................206. Lessons Learned............................................................................................................217. Comments on Issues Raised by Grantee/Implementing Agencies/Donors...................22Annex 1. Project Costs and Financing (to be filled based on final figures)......................23Annex 2. Outputs by Component......................................................................................24Annex 3. Economic and Financial Analysis......................................................................25Annex 4. Grant Preparation and Implementation Support/Supervision Processes............26Annex 5. Beneficiary Survey Results................................................................................27Annex 6. Stakeholder Workshop Report and Results.......................................................28Annex 7. Summary of Grantee's ICR and/or Comments on Draft ICR.............................29Annex 8. Comments of Co-financiers and Other Partners/Stakeholders..........................30Annex 9. List of Supporting Documents...........................................................................31Annex 10. Comparison between original Results Framework and revised Results Framework.........................................................................................................................32Annex 11. Results Chain: Linkage between PDO, intermediate indicators and components........................................................................................................................35

ii

A. Basic Information

Country: Nepal Project Name:NP: Strengthening the Office of the Auditor General

Project ID: P127040 L/C/TF Number(s): TF-10455ICR Date: 12/30/2016 ICR Type: Core ICR

Lending Instrument: SIL Grantee:GOVERNMENT OF NEPAL

Original Total Commitment:

USD 2.30M Disbursed Amount: USD 2.22M

Revised Amount: USD 2.22MEnvironmental Category: CImplementing Agencies: Office of the Auditor General of Nepal Co-financiers and Other External Partners:

B. Key Dates

Process Date Process Original Date Revised / Actual Date(s)

Concept Review: 03/24/2011 Effectiveness: 02/23/2012 Appraisal: 10/25/2011 Restructuring(s): 03/10/2015 Approval: 08/16/2011 Mid-term Review: 01/26/2014 Closing: 06/30/2015 06/30/2016

C. Ratings Summary C.1 Performance Rating by ICR Outcomes: Moderately Satisfactory Risk to Development Outcome: Moderate Bank Performance: Moderately Satisfactory Grantee Performance: Moderately Satisfactory

C.2 Detailed Ratings of Bank and Borrower Performance (by ICR)Bank Ratings Borrower Ratings

Quality at Entry: Moderately Satisfactory Government: Moderately Satisfactory

Quality of Supervision: Satisfactory Implementing Agency/Agencies: Satisfactory

Overall Bank Performance: Moderately Satisfactory Overall Borrower

Performance: Moderately Satisfactory

1

C.3 Quality at Entry and Implementation Performance IndicatorsImplementation

Performance Indicators QAG Assessments (if any) Rating

Potential Problem Project at any time (Yes/No):

Yes Quality at Entry (QEA): None

Problem Project at any time (Yes/No):

YesQuality of Supervision (QSA):

None

DO rating before Closing/Inactive status:

Moderately Satisfactory

D. Sector and Theme Codes Original Actual

Sector Code (as % of total Bank financing) Public administration - Financial Sector 20 20 Central Government (Central Agencies) 80 80

Theme Code (as % of total Bank financing) Other accountability/anti-corruption 80 80 Public expenditure, financial management and procurement 20 20

E. Bank Staff Positions At ICR At Approval

Vice President: Annette Dixon Isabel M. Guerrero Country Director: Qimiao Fan Ellen A. Goldstein Practice Manager/Manager: Fily Sissoko Jennifer K. Thomson Project Team Leader: Akmal Minallah Leslie Isao Kojima ICR Team Leader: Krishnamurthy Sankaranarayanan ICR Primary Author: Arvind Prasad Mantha

F. Results Framework Analysis

Project Development Objectives (from Project Appraisal Document)The objective of the project is to enhance the quality and impact of public sector audit in Nepal.

Revised Project Development Objectives (as approved by original approving authority)There was no change in the Project Development Objective. However, the Results Framework has undergone a change through project restructuring on March 10, 2015 to bring clarity to indicators taking into account the ground realities. PDO indicators have been modified during mid-term review keeping in

2

view the additional information available at that point of time. Similarly, Intermediate Outcome Indicators have also been modified during the Mid-Term Review.

(a) PDO Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)

Formally Revised Target

Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Audit of Financial Statements Completed in accordance with ISSAI Standards

Value quantitative or Qualitative)

35% 60% 7 9

Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016

Comments (incl. % achievement)

Target Surpassed. Quality assurance review of pilot financial audits indicated compliance of 70% to ISSAI framework. Financial Statement Audit of entities in accordance with ISSAI standards are included in every years audit work plan.

Indicator 2 : Financial Audit of Spending Units completed in accordance with ISSAI Standards


NA NA 21 28

Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016


Target Surpassed. Quality assurance review of pilot financial audits indicated compliance of 70% to ISSAI framework. From this financial year audits of 4,300 spending units are being planned and carried out in accordance with ISSAI standards.

Indicator 3 : Performance Audit conducted in accordance with ISSAI framework & submitted to parliament.


1 3 6 6

Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016


Target achieved. Quality assurance review of pilot performance audits indicated satisfactory compliance to ISSAI framework. The performance audits which were conducted have been incorporated in the annual report and submitted to the Parliament.

Indicator 4 : Audit observations contained in ISSAI compliant audit reports (pilot audits) acted upon by the executive within 90 days

Value quantitative or

NA NA 30% 32%

3

Qualitative) Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016


Target Achieved. As against 842 audit observations reported in pilot audits, the executive has acted upon 271 audit observations within 90 days. The audit observations are recorded in NAMS, but its compliance is manual.

(b) Intermediate Outcome Indicator(s)

Indicator Baseline Value

Original Target Values (from

approval documents)


Values

Actual Value Achieved at

Completion or Target Years

Indicator 1 : Office wide audit plan implements risk-based concepts

Value (quantitative or Qualitative)

Nil Fully implemented Fully implemented Fully implemented

Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016Comments (incl. % achievement)

Target achieved. Audit planning and execution is being carried out on risk basis for all entities. Nepal Audit Management System is used for risk based audit and creating audit work plan

Indicator 2 : Peer Review Assessment using the SAI PMF framework


NA Implemented

Next long term Strategic Development Plan of OAGN prepared and agreed on the basis of peer-review report

Peer review completed and actions proposed as part of strategic plan 2016-20.

Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016


Target fully met. Peer review conducted using SAI-PMF which covered all strategic areas of OAGN. The suggestions of peer review has been accepted and adopted as a part of the current strategic plan 2016-2020.

Indicator 3 : Number of Person days of classroom training in new financial audit procedures


NA NA 600 3737


Target surpassed. Training has been conducted both in the country and abroad and has covered all the staff across various levels. Training has been identified as one of the

4

key activities in Strategic Plan 2016-20 for sustenance.

Indicator 4 : Lessons learned from current year pilots documented and reflected in revised audit manuals, guides and training material


Not done Yes

Documented and reflected in audit manuals, guides

Performance audit guide updated based on pilots

Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016


Target partly achieved. Performance Audit Guide has been revised after pilot audits were conducted, and training materials has been revised, but both documents are yet to be approved, by the OAGN. Once approved it will be operationalized.

Indicator 5 : Enhanced Public Participation of CSOs / other stake-holders in the performance audit process (number of participants/CSOs at the events)


NA NA 30 30


Target achieved. CSOs now involved in performance audit and strategy developed for future engagement.

Indicator 6 : Number of times press meetings were arranged to disseminate the results of performance audits with extensive coverage by media


NA NA 6 6


Target achieved. Press briefings done to disseminate audit findings

Indicator 7 : Person days of class-room training in performance audit


NA NA 300 732

Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016


Target surpassed. Training has been conducted both in the country and abroad and has covered staff across various levels. Training has been identified as one of the key activities in Strategic Plan 2016-20.

Indicator 8 : Audit observations contained in ISSAI compliant audit reports (pilot audits) accepted by the executive within 35 days

5


60% of current year audits 75% 50% of pilot audits

89% of pilot audits

Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016


Target Surpassed. As against 842 audit observations reported in pilot audits, the executive has accepted 753 audit observations within 35 days. The audit observations are recorded in NAMS, however, its compliance is manual.

Indicator 9 : Strategy document prepared with actions for improving relations with Executive, PAC, Media, &stakeholders


No Yes

Strategy document prepared and implemented

Communication strategy document has been prepared but yet to be approved.


Target partly achieved. The communication strategy document has been prepared and submitted to OAGN for approval. Once approved it would be operationalized.

Indicator 10 : Computerized databank of audit observations to produce monthly status reports


No Yes

Software implemented and databank populated for last 2 years audit observations

Software developed but data not populated.

Date achieved 12/17/2011 06/30/2015 06/30/2016 06/30/2016


Target partly achieved. NAMS has the requisite fields for capturing audit observations as well as compliance. However for the past data, which is mostly manual in nature the observations needs to be reviewed and entered in the software.

6

G. Ratings of Project Performance in ISRs

No. Date ISR Archived DO IP Actual Disbursements

(USD millions) 1 04/21/2014 Moderately Satisfactory Moderately Unsatisfactory 0.99 2 08/06/2014 Moderately Satisfactory Moderately Satisfactory 0.99 3 02/20/2015 Moderately Satisfactory Moderately Satisfactory 1.73 4 10/08/2016 Satisfactory Satisfactory 2.24

H. Restructuring (if any)

Restructuring Date(s)

Board Approved PDO Change

ISR Ratings at Restructuring

Amount Disbursed at

Restructuring in USD millions

Reason for Restructuring & Key Changes MadeDO IP

03/10/2015 N MS MS 1.73

The project has gone through Level 2 project restructuring on March 10, 2015 to (i) extend the closing date of the project by one year i.e. June 30, 2016, and (ii) revise the results framework to effectively monitor and evaluate the progress of the project.

I. Disbursement Profile

7

1. Project Context, Development Objectives and Design

1.1 Context at Appraisal

1. Country context: Nepal has recorded impressive economic and social achievements and outperformed most low-income countries on a range of social indicators. Real GDP growth between 2002 and 2011 averaged about 3.8 percent per annum, and this growth was achieved in combination with relatively low inflation and stable domestic debt. Nepal had made progress toward establishing peace and democracy. Its Constituent Assembly was elected in 2008 and functioned as a regular Parliament.

2. Sectoral context: Improving governance and accountability underpinned the agenda of the Government of Nepal for building a New Nepal, one that is prosperous, peaceful, and just. Strengthening public financial management (PFM) was a key element of the Government’s strategy for strengthening governance, optimizing outputs from public resources, and ensuring inclusive and broad-based development. The Government conducted a Public Expenditure and Financial Accountability (PEFA) assessment in 2008 to assess the performance of the country’s PFM system in comparison with international benchmarks. The assessment revealed many opportunities for strengthening PFM and suggested that external scrutiny and audit is the weakest link in the budget cycle in Nepal, which needed immediate attention.1

3. Institutional context: External audit in Nepal is carried out by the Office of the Auditor General of Nepal

(OAGN), which is the country’s Supreme Audit Institute (SAI). Its aim is to enhance good governance by promoting public accountability and transparency through provision of independent professional audit services to the nation. Current mandate empowers OAGN to audit the accounts of all government offices, including fully owned government undertakings. The OAGN was determined to improve both the quality and timeliness of audit. The need for better audits and improved transparency from citizens and government necessitated the audit reforms. One of the goals of OAGN Strategic Plan 2010-2012 was to provide timely and quality audit services, including six key objectives: (a) develop and implement a Financial Audit Manual; (b) enhance the quality of audit through IT application; (c) strengthen the Quality Assurance (QA) function; (d) develop and use specific audit guidelines for the audits of boards and state-owned enterprises; (e) enhance quality in performance auditing; and (6) adopt time-bound reporting mechanisms. Strengthening the OAGN Project was designed to support the achievement of the above objectives.

4. Rationale for Bank assistance: The World Bank through a series of IDF grants (1993-2006) has supported OAGN in achieving (a) development of financial and performance audit guides; (b) establishment of training division and conducting training programs; (c) entering into a twinning arrangement with the SAI of Malaysia; (d) establishment of a website; and (e) improvement of its overall audit capacity. The Bank brought both technical knowledge from other countries and financial support from other donors. Among key areas of interventions, the Bank also supported PFM reforms in Treasury Single Account under the Strengthening Public Financial Management Project I, which was a perfect complement to the overall PFM reforms looking at the audit, accountability, and transparency side of governance.

5. The Bank’s Interim Strategy Note (ISN) 2011 for Nepal identified governance as a cross-cutting theme to be mainstreamed in all World Bank-supported projects and this project supported that theme. The project also contributed towards the Government PFM Reform strategy (2009) and the OAGN Strategic Plan 2010-2012, two key elements in the Government’s effort to improve governance and enhance transparency and accountability.

1 PEFA rating for PI-26 was D+ in 2008 and was C+ during the repeat assessment in 2013.

9

1.2 Original Project Development Objectives and Key Indicators (as approved)

6. The project development objective (PDO) is “to enhance the quality and impact of public sector audit in Nepal”. The original Results Framework2 identified three PDO performance indicators to measure the expected outcome:

Audits substantially meeting OAGN financial statement auditing standards; Number of in-depth performance audits reported to Parliament per annum; and Current year audit observations implemented by the Executive within 1 year.

1.3 Revised PDO and Key Indicators, and reasons/justification

7. The PDO was not revised during the project period. Changes were made to the Results Framework 3 and/or the corresponding end targets consequent to project restructuring in 2015. It identified four PDO indicators to measure the expected outcome:

Audits of financial entities completed in accordance with ISSAI framework; Financial audits of spending units completed in accordance with ISSAI framework; Performance audits conducted in accordance with ISSAI framework and submitted for transmission to

Parliament; and Audit observations contained in ISSAI-compliant audit reports [pilot audits] acted upon by the

Executive within 90 days.

8. The Results Framework was revised to enhance some of the indicators and match them with internationally accepted standards as well as sharpen the focus on results being achieved. This restructuring was necessary as the original PDO and intermediate indicators were not easily measurable, and some of the intermediate indicators for were not well designed.

1.4 Main Beneficiaries

9. The direct beneficiaries of this project were expected to be OAGN and the Public Accounts Committee (PAC). The staff of OAGN was to be provided with hands-on support by project-financed consultants in planning and conducting actual financial and performance audits that would be reported to Parliament. Direct support was expected to be provided to the PAC Secretariat to conduct more efficient and effective hearings and to draft the committee reports to Parliament.

1.5 Original Components

10. The SOAGN Project had four components that complemented each other:

Component 1, Enhancing the quality of financial statement audits (US$1.0 million). This component aimed at improving the quality and timeliness of the financial statement audits carried out by OAGN by providing technical support for capacity building of staff, implementing risk-based audit, and implementing a computerized audit management system. This component included the following key activities: (a) provide technical advice in preparation of the annual audit plan; (b) upgrade financial audit manuals and conduct pilot financial statement audits; (c) support QA arrangements in OAGN; (d) provide

2 Refer to Annex 10. See page 4 of the Project Paper dated December 17, 2011. 3 Refer to Annex 10. See page 10 of the Project Restructuring Paper dated March 6, 2015.

10

for a peer review by a sister SAI; and (e) introduce the use of computer audit technologies like electronic working papers (EWP) and computer-assisted audit techniques (CAATs).

Component 2: Upgrading the capacity to conduct performance audits (US$0.73 million). This component extended technical support to OAGN in upgrading its capacity to conduct in-depth performance audits through following activities: (a) revise performance audit guidelines, including manuals for implementation; (b) conduct in-depth performance audits; and (c) support strategic planning and development of work plan by the Performance Audit Division.

Component 3: Enhancing the impact of audit (US$0.5 million). This component aimed at improving transparency and accountability by involving the Public Accounts Committee, Executive, civil society, and media. The key activities under this component were (a) provide technical support to the PAC Secretariat; (b) summarize key recommendations from audit reports; (c) design and implement a communications strategy for dissemination of results of the audit; and (d) redesign and computerize the databank of audit and PAC recommendations.

Component 4: Project management (US$0.15 million). This component provides support to OAGN for managing project activities.

1.6 Revised Components

11. There were no major changes to project components and costs during project implementation. However, Sub-component 3(a), Provide technical support to PAC, was scaled down in the absence of PAC.

1.7 Other Significant Changes

12. Extension of closing date. The project closing date was extended by one year (i.e., until June 30, 2016) to support the completion of remaining project activities. It had resulted in a slight revision in the agreed implementation schedule of the project.

13. Changes in the Results Framework. The Results Framework was revised to bring clarity in the context and measurement of indicators. The baseline and targets/values were updated based on ground realities; and certain intermediate indicators, which were not interlinked with PDO, were dropped.

2. Key Factors Affecting Implementation and Outcomes

2.1 Project Preparation, Design and Quality at Entry

14. Relevance and analytical underpinning. The project was prepared based on the request received from the Government of Nepal. The project design and components were formulated in close consultation with OAGN and OAG Norway4. PDO was aligned with the strategic Government priorities, OAGN Strategic Plan 2010-2012, and development partners.5 Following an innovative approach, the project design was selective by focusing on strengthening OAGN and PAC to improve governance and enhance accountability and transparency. The Bank had partnered with OAGN;6 on a series of IDF grants and based on lessons learned, the project was designed to ensure maximum knowledge transfer and capacity building in OAGN. The project design also recognized the PEFA 2008 score PI-26,7 focusing on scope, nature, and follow-up of external audit.

4 In 2008, OAGN signed a Memorandum of Understanding with OAG Norway, the country’s SAI. With technical support from OAG Norway, OAGN developed a Strategic Plan and a capacity-building initiative for the period 2010-2012. The role of Norway SAI in this project was to mainly support OAGN in the implementation of performance audit.5 PFM reform strategy and strategic plan of OAGN 2010-2012.6 The traditional approach did not provide for extensive technology transfer.7 The PEFA rating for indicator PI-26 was D+ in 2008.

11

15. Institutional arrangements. The OAGN was uniquely placed to implement project components given its strategic role, constitutional mandate, constitutional position, and solid foundation. The original project design did not recognize the PAC Secretariat as an implementing agency for activities relating to technical support to PAC, which led to a gap in institutional arrangement.

16. Quality at entry. The Bank team delivered the project within one year of concept review and rightly identified the core weaknesses and implementation risks. The institutional and governance risks affecting the PDO — as a result of the long, vacant position of Auditor General and non-availability of PAC — were well identified during project preparation and appropriately articulated in the project paper. Even though it was a small trust fund used for looking into the impact of the interventions, the project risk was appropriately rated as “Substantial”. To expedite project start-up, the Bank team supported OAGN in the finalization of terms of reference for the appointment of consultants. The project had procured an experienced international consulting firm within few months of project effectiveness to implement the project components. However, the results framework and monitoring indicators for the PDO and intermediate level were not objectively defined in the beginning. As a result, during implementation phase, the project team was unable to gather reliable data to measure outcome indicators

17. Key challenges at entry. The political instability and overall delay in taking critical decisions was a key challenge at the time of project preparation. The absence of PAC and delay in appointment of Auditor General led to a leadership vacuum. Administrative and financial dependence of OAGN on Government was a risk identified at the time of entry.

2.2 Implementation

18. The project experienced some implementation delays, especially in the first year; but the pace largely picked up after Mid-Term Review and achieved reasonable success. Although the consultants were appointed within the first six months of the project, their mobilization was delayed, which contributed to the slow start. Also in the first year, budget was not approved for the project which hampered the implementation. Up to Mid-Term Review, the project was lagging behind mainly in Components 1 and 3. Thereafter, consistent supervision and follow up by the Bank, along with proactive action from the client, turned the project around. Restructuring was also carried out during Mid-Term Review to modify certain indicators and set realistic targets for project interventions. The project managed to achieve the targets under Component 1 and 2 by the close of the project. Effective and timely implementation support by the Bank task team ensured identifying issues on timely basis, which led to restructuring of the Results Framework and extension of time. The Strengthening PFM (SPFM) project on overall PFM reforms in government also complemented the SOAGN project to a great extent in achieving the desired results. However, the project did face several external constraints, namely political instability and fragility, the April 2015 earthquake during which OAGN buildings were severely damaged and declared unsafe, the absences of the Auditor General in initial years and PAC through most of the project period, and the September 2015 fuel crisis that affected implementation for considerable time. In spite of these challenges, the client ensured reforms remained on track.

19. In the case of Component 1, there was delay in the first year, which led to spillover of activities. During the second year, key activities like the risk-based audit plan, financial audit manuals, and QA handbook were updated and accepted by OAGN and disseminated to stakeholders. First 13 pilot financial statement audits were delayed due to non-availability of transactional-level data in TSA from Financial Controller General’s Office (FCGO), and CAATs could not be piloted in the field. Delay in deciding suitable electronic working paper software and subsequent procurement also resulted in non-usability of electronic working paper while conducting

12

these pilot audits. Due to the above reasons, the pilot could not be completed until mid-term review as designed. After mid-term review, the electronic working paper and Nepal Audit Management System (NAMS) were implemented in OAGN and used for carrying out the next set of 15 pilot financial statement audits. Training and capacity building on these systems was provided to the staff both in-house and abroad on financial and performance audits. During this period, peer review assessment based on SAI-performance management framework was also carried out by Comptroller and Auditor General of India, which paved the way for developing the current strategic plan. For the second set of 15 pilot financial statement audits, the CAAT software IDEA was used, and data from TSA was made available by FCGO based on which vertical, risk-based audit of ministries was carried out by OAGN. The QA review of pilot financial audits indicated compliance of 70 percent with the ISSAI framework. While electronic working paper and NAMS have been implemented in OAGN, CAAT was used only for pilots and has not been procured and implemented till date, which limits OAGN to do data-mining and auditing of transactions carried out by FCGO.

20. In the case of Component 2, the progress from the beginning was on track. This component was supported by SAI Norway as well as by the project consultants who supported OAGN to develop the strategic plan for performance audit. The Performance Audit Manual was updated and aligned with ISSAI in the first year, and training on performance audit methodology was conducted in collaboration with SAI Norway. The OAGN conducted three in-depth performance audits and incorporated these reports in their annual report submitted to the Parliament. In year 2, a second round of training was conducted and three more in-depth performance audits were carried out by OAGN. Consultants carried out the QA reviews of these audits, and a workshop was organized to capture the lessons learned from this round of audits. The QA review of pilot performance audits indicated satisfactory compliance to ISSAI framework. Performance Audit Guide has been revised after pilot audits were conducted, and training materials has been revised; but both documents are yet to be approved by OAGN. Once approved, it will be operationalized.

21. The OAGN officials collaborated with Commission of Audit Philippines to understand citizen participation audits. Based on this interaction, OAGN Strategic Plan 2016-20 aims to enhance the participation of civil society organizations (CSO) in the auditing process. A demand also arose for exploring the possibilities of involving CSOs in audit as the extended mandate required additional resources that are difficult to come by in the short term. The PRAN (Program for Accountability in Nepal), a unit in World Bank office, worked on the demand side of PFM by enlisting CSO support to build their capacity for gathering audit evidence. A number of workshops, capacity-building training, and interactions with OAGN staff were organized under this component. A set of Guidelines for Citizens’ Participation in Audit have been developed.

22. In the case of Component 3, performance has largely remained below par. Two major sub-components did not progress as designed. The PAC was not available for most of project implementation (from project effectiveness till mid-term review), and there were several activities in the project design that were planned for PAC Secretariat. The Constituent Assembly was dissolved in May 2012; and in the absence of PAC, there was no mechanism for legislative scrutiny of audit reports. As an interim measure, a number of initiatives were taken by Government and OAGN to enhance the impact of audit: (a) Government in 2013 established a cabinet-level committee chaired by Minister of Finance to review OAGN reports and to follow up on implementation of audit recommendations; and (b) in 2014, OAGN team together with Chief Secretary from Prime Minister’s office visited regions and discussed the challenges faced by heads of administration and accounts to clear audit observations. There were workshops organized around these themes with follow-up meetings in Kathmandu with invitations to secretaries and account chiefs from all the ministries, public enterprises, and private audit firms. These set of activities undertaken by OAGN as possible ways to enhance the impact of audit given the absence of PAC has added value in the accountability process but have unfortunately not shown measurable results in the project outcome indicators.

13

23. The new PAC was finally constituted in April 2014 with many members; however, a chairperson was not appointed. Under the project there were efforts made to harmonize the functioning between Government, PAC, and OAGN. A high-level stakeholder workshop with support of SAI Norway was conducted in September 2014 to discuss new audit legislation for Nepal and its adherence to INTOSAI standards, highlighting the importance of SAI independence. The focus of the new PAC was to handle immediate issues and not to review backlog of audit reports from the previous PAC, which also added to delays. Notably, there was no indicator to measure this important sub-component.

24. A communications strategy for effective dissemination of audit findings has been developed but is awaiting the approval of OAGN for implementation. The OAGN has been engaging regularly with media and CSOs to disseminate audit findings and recommendations.

25. The OAGN has obtained the technical support of project consultants to summarize audit observations and to improve the presentation of annual audit reports. Additionally, a 15-member Advisory Committee comprising a cross-section of experts was formed to look into the quality of audit reports. The audit report for FY2014 to Parliament was commended for its quality by this Committee. However, improvements could be made in the presentation of audit reports like preparing separate ministerial audit reports or separate performance audit reports and tabling it to the legislature. The audit reports for the last 10 years were published on the OAGN website and are accessible to general public.

26. The NAMS software was procured in mid-2014 and configured to support recording of past audit observations and PAC recommendations. While NAMS captures the audit observations for new audits, OAGN has yet to populate the database with past audit observations to maintain repository of audit objections for past years. The OAGN should develop a strategy to ensure that old data is scrutinized, validated, and entered in the system for future compliance.

2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization

27. The original indicators were not designed appropriately because some indicators were not clear. The flaws in both PDO indicators and intermediate indicators surfaced during project implementation and were addressed either by redrafting and aligning with international standards or redefining them, taking into account the revised baseline during the mid-term review. The World Bank, in consultation with client, revised the results framework during mid-term review and carried out a formal restructuring. The restructuring helped both the team and client in measuring the results better after mid-term review as well as measuring during the Implementation Completion Report (ICR). The M&E implementation and utilization was working well in the project. The OAGN submitted six monthly status reports on a regular basis, highlighting the achievements in relation to the components and also outlining the ongoing issues and future action plan.

2.4 Safeguard and Fiduciary Compliance

28. Environment and social safeguards. The environment and social safeguards for the project were not required as the grant funds were earmarked toward development of PFM systems.

29. Procurement. The procurement rating remained “Satisfactory” throughout project implementation. There was one large-value consultancy contract, which was awarded in a timely manner; and the clients were proactive in handling the contract management issues faced during project implementation. From a procurement perspective, it is one of the most successfully implemented projects in the Nepal portfolio.

14

30. Financial management: The overall FM performance was “Satisfactory”. The FM arrangements were aligned with the country systems. Separate budget head was created for the project, including counterpart activities. To facilitate timely payments, the fund flow arrangements were streamlined through the provision of a Designated Account. The project used a report-based disbursement method, and advances into the Designated Account were provided based on two trimester forecasts of expenditures. A full-time accounts officer was dedicated for managing and reporting the use of project funds. There were no delays in the submission of implementation progress reports, which included financial monitoring reports. Initially, some delays were noted in the submission of annual audit reports due to which the FM performance rating was downgraded. As implementation progressed, the project was timely in the submission of annual audit reports, and FM performance was upgraded to “Satisfactory”. 2.5 Post-completion operation/next phase

31. The initiatives supported in the current project have been included in the OAGN Strategic Plan 2016-20. Since financial and performance audit constitute the core operations of OAGN, sustainability of these reforms will continue after project closure. Capacity building and training are being carried out on a regular basis, and strong capabilities have been developed in the second level of officers to carry forward the reforms. The OAGN has appointed domain experts on a contract basis to support it with IT initiatives and future rollouts.

32. This project has been a catalyst for reforms in OAGN and, through pilot interventions, demonstrated the effectiveness of reforms. It is imperative that these reforms be scaled up, rolled out, and sustained across the organization. With OAGN’s expanding mandate to carry out audit of local bodies and requirements for sustaining the reforms and internalizing them, it is imperative that the next phase of technical and financial support be provided as early as possible to ensure continuity and keep up pace of reforms. This will not only support them financially but also would help them with technical assistance and proper supervision leading to better outcomes.

33. The interventions that need augmentation in the next phase together with other weak areas identified in the SAI PMF report may be considered under the follow on project. Some areas which could be supported in the next phase are as follows:

Support capacity building and training of OAGN staff, including (i) refresher training on conducting audits as per newly developed Financial Statement and Performance Audit Framework; (ii) issuance of stand-alone ministry/entity-wise financial statements; and (iii) special audits focusing on IT systems, disaster management and externally aided projects;

Develop audit guidelines and strengthen technical capacity of OAGN staff to conduct audit of village development committees and municipalities;

Provide technical support and capacity building to stakeholders involved in the implementation of social audits;

Induct qualified accounting and audit professionals in OAGN, along with suitable capacity building to existing staff to handle NPSAS-based financial statement certification;

Create sustainable IT infrastructure, separate IT directorate, and develop disaster recovery plan for OAGN;

Support OAGN in implementation of CAAT and improve functionality and use of NAMS with FMIS; Support OAGN in the design and implementation of new Human Resources framework; Develop strategy for validating and creating databank for audit observations to ensure repository of audit

reports and managing audit paras and compliance; Support audit impact through proper dissemination of audit findings, and improve compliance by the

Executive through policy decisions and effective monitoring mechanisms.

15

3. Assessment of Outcomes

3.1 Relevance of Objectives, Design, and Implementation

Rating: Substantial

Relevance of objectives: High34. The SOAGN project is relevant even in today’s context as it contributes towards the overall Government reform agenda to improve governance and enhance transparency and accountability in the country. An important player in the PFM arena, OAGN is responsible for external audit of government institutions, including fully owned government undertakings as mandated by the Constitution of the country. Going forward with the new Constitution expanding the OAGN mandate to carry out audit of local bodies, these reforms are more relevant and should be implemented as early as possible. The project continues to be relevant under the Bank’s Interim Strategy Note (ISN) 2011 for Nepal as it supports “governance and accountability” and identifies governance as a cross-cutting theme to be mainstreamed in all World Bank-supported projects. Improving the quality, timelines, and impact of audit also augurs well for the overall efficiency and transparency of government functioning.

Relevance of project design and implementation: Substantial35. The project design was largely consistent with PDOs and targeted important areas for achieving the objectives. The PDO, revised intermediate indicators, and components were interlinked in a harmonious manner. The selection of sub-activities was notably relevant since they drew on financial audits and performance audits, which are core areas of OAGN operation. The planned provision of capacity support and functional reforms was also consistent with identified needs. The actual implementation of the project was largely relevant to the PDO achievement. Project implementation was largely consistent with the PDOs and responded to changing circumstances. For example, the Bank responded to a request to extend Component 1 and, in mid-term review, sought to clarify and restructure the Results Framework based on the actual implementation experience in the field, which was vital for proper measurement of project success.

3.2 Achievement of Project Development Objectives

36. The PDO, “to enhance the quality and impact of public sector audit in Nepal”, can be divided into two distinct development objectives: (a) enhance the quality of audits and (b) enhance the impact of audit. The first development objective, “enhance the quality of audits”, is rated “Substantial” and is important since it would be achieved in the project period and would be the stepping stone for achieving the second development objective, “enhance the impact of audit”, which is rated “Moderate” and would be achievable in the long run. It is judged that the Results Framework indicators are, in many ways, appropriate measures of progress toward PDO. As such, the following assessment is based on the given core target indicators: 4 PDO indicators and 10 intermediate indicators. Given that the Results Framework was restructured, results are judged against the revised outcome indicators and target values that are more amenable for measurement.8

37. Enhance the quality of public sector audit in Nepal – “Substantial”. The PDO indicators and corresponding intermediate indicators have been successfully achieved. In case of PDO Indicator-1, the project conducted 9 pilot financial statement audits of entities in accordance with ISSAI-issued standards against a target of 7. Similarly for PDO Indicator-2 against a target of 21 spending units, the pilot financial audits were completed

8 Refer to Annex 11 for results chain.

16

for 28 spending units. The consultants conducted the QA review on a structured scale for 25 financial audit reports completed by OAGN. The review indicated 70-percent compliance to ISSAI framework, which is encouraging but with a reminder that the gaps need to be filled as implementation progresses. With respect to PDO Indicator-3, OAGN has completed 6 performance audits, which were also the target for this indicator. The QA review of pilot performance audits indicated “satisfactory” compliance to ISSAI framework.

A. Enhance the Quality of Financial statement Audits

38. Indicator-1, Office-wide audit plan implements risk-based concepts (Target: Full implementation. Achievement: Fully achieved). Risk-based audit planning and implementation is an integral part of ISSAI framework for financial statement audits. The OAGN has developed and implemented a Financial Audit Manual as per ISSAI standards. The annual audit planning methodology based on risk scoring of entities has been developed under the project and implemented across OAGN. The methodology classifies audit entities into three main categories (A, B, and C) based on risk parameters that are defined and used for developing the audit plan. For Categories A and B, detailed audits are conducted; while for Category C, desk audits are conducted. The annual audit plan provides directions on undertaking various types of audits, including financial audits, concurrent audits, IT audits, and performance audits, which are approved by Auditor General. Currently, the financial audits being carried out are as per ISSAI standards. For FY16-17, the annual audit plan has been prepared following a risk-based concept. The NAMS has been developed and implemented across OAGN following ISSAI standards and framework, with main focus on risk-based audit methodology which is used by OAGN for the entire audit cycle, including planning, execution, documentation, and generation of reports (except compliance). The OAGN follows strict protocols to ensure quality of audit. The QA Handbook has been updated and implemented in OAGN. The QA Committee headed by a senior Deputy Auditor General goes through a sample of audit files and prepares a consolidated QA Review Report. In addition, a Code of Ethics Monitoring Committee headed by a Deputy Auditor General ensures an oversight on compliance with ethics code.

39. Indicator-2, Peer review assessment using the SAI Performance Measurement Framework (Target: Next long-term, OAGN Strategic Development Plan prepared and agreed on the basis of peer-review report. Achievement: Report accepted and incorporated in Strategic Plan 2016-20). Office of the Comptroller and Auditor General of India carried out peer review in 2014, using INTOSAI Performance Measurement Framework to assess the various processes, practices, methodologies, and SAI environment and issued a report. The report highlighted the achievements made by OAGN as well as gaps in the various domains of SAI operational framework. It identified the areas that need strengthening and indicated strategic direction for future capacity building. The report has been accepted by OAGN, and all recommendations have been suitably incorporated in the Strategic Plan 2016-20, which is under implementation.

40. Indicator-3, Number of person days of classroom training in new financial audit procedures (Target 600 person-days. Achievement: 3,737 person-days). A structured training on the financial audit procedures was delivered to more than 300 OAGN staff in 2014. In the first phase, OAGN staff were trained on the application of risk-based audit methodology aligned with ISSAI and on the use of audit working papers. In the second phase, training was provided on the use of NAMS and electronic working papers. Training-of-trainers course was developed, and training on NAMS software was delivered to a core group of 30 OAGN staff. Exposure visits to Malaysia for 60 OAGN staff and to the Philippines for 45 OAGN staff provided opportunities to interact with experts from respective SAIs and gain firsthand experience on the financial audit methodologies and application of new audit technologies. The training was also provided to 20 internal auditors from FCGO. The benefits of CAAT to analyze data obtained the FCGO TSA system in support of financial audits was also demonstrated. As a result, majority of OAGN auditors are fully conversant in conducting financial audits applying risk-based audit methodology of ISSAI standards and possess practical knowledge of using audit working papers. The quality of

17

training is also evident since the staff have been able to carry off financial statement audit as per ISSAI framework. However, training on ISSAI framework, NAMS, and electronic working paper should be provided on a continuous basis to ensure long-term sustainability.

B. Upgrading the Capacity to Conduct Performance Audits

41. Indicator-1, Lessons learned from current-year pilots documented and reflected in revised audit manuals, guides and training material (Target: Documented and reflected in audit manuals, guides. Achievement: Fully achieved, Performance Audit Guidelines updated based on pilots). The OAGN updated the previously developed performance audit manual and guidelines and adopted it for pilot performance audits. The OAGN staff conducted six performance audits applying revised performance audit guide procedures; and final reports were incorporated in the Annual Report of Auditor General, which was submitted to the President. The consultant conducted a QA review of these audits, and a debriefing workshop was organized in order to capture the lessons learned. The quality of performance audit has improved as per the quality assessment of the consultants and ICR team. However, these reports should be issued as standalone reports in future for better discussions and actions at the Executive level. Following the risk-based methodology outlined in the OAGN Performance Audit Guidelines 2012, INTOSAI, and ASOSAI performance audit guidelines, OAGN developed a Performance Audit Strategic Plan and a risk-based Annual Audit Plan for Performance Audits.

42. Indicator-2, Enhanced public participation of CSOs and other stakeholders in the performance audit process (Target: 30 participants. Achievement: Fully achieved). The draft Audit Act 2016 provides for involvement of CSOs in the audit process as deemed necessary by OAGN. There are two aspects to engagement with citizens/CSOs, and there is a need to distinguish between them. On the one hand, there is engagement with citizens/CSOs to raise their awareness about PFM issues and build their capacity to explore TSA/budget to be able raise relevant questions. This is the demand side of PFM. There is, on the other hand, engagement with CSOs to seek their participation in public audit to supplement the efforts of OAGN. This has become imperative in view of OAGN’s lack of resources to comply with expanded mandate.

43. The OAGN reached out to CSOs through round-table discussions, series of workshops, and focus group discussions for enhancing their roles and participation in the performance audit. In addition to holding a series of workshops with CSOs in Nepal, OAGN hosted a series of regional village committees with SAIs and CSOs in other countries in South Asia, with resource speakers from SAIs of Korea, Philippines, India, and Argentina. The performance audits conducted by OAGN accordingly included CSO input to planning performance audits (identification of risks and audit criteria); performance audit fieldwork (in gathering evidence through focus group discussions and questionnaire administration); and, in future, report dissemination. The OAGN officials also participated in a South-South knowledge exchange with Commission on Audits of the Philippines in collaboration with other Bank programs. After this visit, OAGN reviewed the audit mandate to ensure that citizen participatory audit is added as a part of the Strategic Plan 2016-20. Experiences in other countries with direct citizen engagement in the audit process have shown greater demand for accountability and heightened the impact from audit.

44. Indicator-3, Number of times press meetings were arranged to disseminate the results of performance audits with extensive coverage by media (Target: 6 meetings. Achievement: Fully achieved). The OAGN held regular press briefings to disseminate audit findings from the reports submitted to the Parliament. The minutes of press briefings are available both in English and Nepali in the OAGN website.

45. Indicator-4, Person-days of classroom training in performance audit (Target: 300 person-days; Achievement: 732 person-days): Training programs for staff across various levels were conducted within country

18

and abroad (India, Pakistan, and UK). Training has been identified as one of the key activities in the Strategic Plan 2016-20. Performance audit training in the field of environment audit and reconstruction and rehabilitation audit was also carried out during the project. Such training should be provided on a continuous basis to ensure long-term sustainability.

46. Enhance the impact of public sector audit in Nepal (“Moderate”). This PDO indicator has been partly achieved. For pilot audits, the audit process was carried out using NAMS, and audit paras were recorded in NAMS. The compliance rate for action by Executive is 32 percent, which meets the end target of this PDO. However, compliance is not recorded in NAMS, and it has not yet been made systematic and internalized by all stakeholders. The audit reports are published and public accessible on OAGN website. A communication strategy for effective dissemination of audit findings has been developed, and its operationalization would improve impact of audit. The OAGN is actively involving CSOs in the audit process and has developed a set of Guidelines for Citizen Participatory Audit, which is incorporated in the Strategic Plan 2016-20. The proposed Audit Act provides for involvement of CSOs in the audit process. The OAGN is conducting outreach and dissemination activities with CSOs and the public on findings of the audit report, which enhances the impact of audit.

47. The ultimate stakeholder in the accountability cycle for audit is PAC, which was targeted in the project but could not be carried out due to non-availability of PAC for a large part of the project. The new PAC was finally constituted in April 2014, with many members; however, a chairperson was not appointed. The focus of new PAC was to handle immediate issues and not on review of audit reports and the backlog from previous PAC, which also added to delays. Under the project, there were efforts made to harmonize the functioning between Government, PAC, and OAGN. A key point is that no indicators related to PAC.

C. Enhance the Impact of Audit – Rating of “Moderate”

48. Indicator-1, Audit observations contained in ISSAI compliant audit reports (pilot audits) accepted by the executive within 35 days (Target: 50 percent; Achievement: Fully achieved). Of the 842 audit observations reported in pilot audits, the Executive has accepted 753 audit observations within 35 days. It is to be noted that compliance is not recorded in NAMS and are presently available in manual form.

49. Indicator-2, Strategy document prepared with actions for improving relations with Executive, PAC, media, and other stake-holders (Target: Strategy document prepared and implemented. Achievement: Partly achieved, the document prepared but yet to be approved by OAGN). Prepared and submitted to OAGN, the draft communication strategy document advocates extending the OAGN outreach program to create awareness about its function; deepen citizen engagement in the audit process; and enhance the quality of information broadcast to citizens, media, and CSOs about the results of its audit. The strategy document addresses the challenges of communication with both external and internal stakeholders and recommends establishing a communication cell. This document is yet to be approved by OAGN, after which it would be operationalized.

50. Indicator-3, Computerized databank of audit observations to produce monthly status reports (Target: Software implemented and databank populated for last 2 years audit observations. Achievement: Partly achieved, Software developed but data not populated). The NAMS has the requisite fields for capturing the audit observations as well as tracking compliance. For new audits, NAMS would be able to capture the audit issues and compliance in the system; however, for the past data, which is mostly manual in nature, the observations should be reviewed and entered in the software as one time exercise. This has not been done. This indicator has not been achieved, and OAGN needs to fully capitalize on this newly developed feature of repository of audit observations in NAMS by populating data that could be used for improving the effectiveness of compliance and therefore improve the impact of audit. A strategy for updating the old paras and compliance should be worked out by

19

OAGN with a practical timeline and for all future audits, the audit paras and compliance should be recorded in NAMS mandatorily.

3.3 Efficiency Rating - “Substantial”

51. The project interventions were apt since it focused on improving the accountability processes, information systems, and functioning of OAGN and PAC in order to bring good governance and transparency in the PFM system. It followed a holistic approach and sequenced the reforms in a phased manner. The transformation in technical capacity, performance, and behavioral change inculcated in the rank and file of OAGN due to project interventions increasing Executive performance and improving quality of service delivery. With the overall outlay of US$2.22 million, the outcomes achieved under the project are substantial and provide value proposition for a reform-oriented PFM program. The overall efficiency gains for the society in the long run will be significant.

3.4 Justification of Overall Outcome Rating - “Moderately Satisfactory”

52. Based on the PDO, Relevance (“Substantial”), and Efficiency (“Substantial”), the overall outcome rating is “”Moderately Satisfactory”. It is pertinent to note that the impact of public sector audit should be measured in the long run and not in the short run of project period.

Relevance of Efficacy: PDOEfficiency Overall OutcomeObjectives Design Audit Quality Audit Impact

High Substantial Substantial Moderate Substantial Moderately Satisfactory Substantial Moderate Substantial

3.5 Overarching Themes, Other Outcomes, and Impacts (a) Poverty impacts, gender aspects, and social developmentNA

(b) Institutional Change/Strengthening53. The project had a positive impact on strengthening of OAGN. Through its interventions, the project has supported improving the audit systems and procedures and usage of information technology. Training on modern audit techniques and methodologies has improved the overall capacity of OAGN staff. The NAMS and electronic working paper have been implemented in OAGN, which will go a long way in improving the quality of the audit. Another positive aspect is that the Strategic Plan 2016-20 addresses all areas of improvement, including human resources as identified in the peer review report, which will further strengthen the capacity of OAGN.

20

(c) Other unintended outcomes and impacts (positive or negative)

54. The project has brought behavioral change in OAGN staff to shift from old audit procedures to the risk-based audits, which is commendable.

3.6 Summary of Findings of Beneficiary Survey and/or Stakeholder WorkshopsNA

4. Assessment of Risk to Development Outcome - Rating: “Moderate”

55. This project has been a catalyst for reforms and has laid the path for improving OAGN both internally and externally. The OAGN has internalized the reforms and changes in audit techniques. The reforms have gained sustainable momentum. Although quality of audit would be improving, OAGN should make effort to improve the impact of audit by collaborating with PAC and other external stakeholders. The key risk as of now is the impending retirement of the current Auditor General, which may affect the pace of reforms to some extent. Also the risk of staff movement from OAGN due to retirement policies of Government would affect the capacity of the OAGN in the short run.

5. Assessment of Bank and Borrower Performance

5.1 Bank Performance

(a) Bank performance in ensuring quality at entry - Rating “Moderately Satisfactory”

56. The project design addressed the strategic need of Government while project components were designed based on consultations with OAGN and wide range of analytical work. The risks relating to implementation were adequately identified and have proven to be sound. The project was also prepared within a quick timeframe, which ensured the relevance of the project for the client. The quality at entry could have been rated as ‘Satisfactory’ except for (a) issues in results framework that did not provide reliable data for measurement of outcome indicators and led to subsequent restructuring; and (b) PAC Secretariat should have been made a separate implementing agency for their sub-component and held accountable for results.

(b) Quality of supervision - Rating “Satisfactory”

57. The ICR task team continuously monitored project progress and provided support to the client to help stay engaged and focused on the reform agenda. The team also restructured the project with revised Results Framework at an appropriate time. The implementation support missions were held regularly to monitor project progress, findings were reflected in the aide memoirs, and compliance was regularly monitored. The Implementation Status and Results Report (ISR) is an important management tool to monitor project performance and record implementation issues. The ICR team has noted that although periodic implementation support missions have been undertaken (as evidenced from the team aide-memoires and management letters), the ISR portal has not been updated regularly, which is a lacunae in documentation.

(c) Justification of rating for overall Bank performance - Rating “Moderately Satisfactory”

58. With rating of “Moderately Satisfactory” for preparation and “Satisfactory” for supervision, overall World Bank performance is rated as “Moderately Satisfactory”.

5.2 Borrower Performance

21

(a) Government performance - Rating “Moderately Satisfactory”

59. Government’s ownership of the project was good, but a timely commitment could have achieved the desired outcomes. The funding for the project, including government contribution, was provided in the yearly budget (except for the first year) and was made available to OAGN. Competent government officers were also assigned to the project. Vacant since 2007, the position of Auditor General was filled in 2013, albeit with substantial delay. The PAC has remained absent for most of project implementation, which resulted in mild achievement of certain outputs under Component 3. A Cabinet-level committee was formed in May 2013 (as an interim measure in the absence of PAC). The interim committee, chaired by Minister of Finance to review past reports of the Auditor General and to follow up on implementation of audit recommendations, was disbanded with limited if any evident results. The new PAC was subsequently formed by the Government in 2014; but due to lack of focus, it could not deliver desired results. The Government is expanding the mandate of OAGN and envisages providing greater independence and financial powers. This is a welcome step taken by the Government since it would further improve the governance environment that the project aims to support. The Government, in spite of external issues like earthquake, fuel crisis, and political uncertainty, has supported the SOAGN project.

(b) Implementing agency or agencies performance - Rating “Satisfactory”

60. The OAGN, the main implementing agency, handled major activities of the project. Working in spite of political issues and natural disasters, OAGN was committed to reforms as part of its strategic plan. The OAGN Strategic Plan was the basis for developing and implementing this project, which ensured total ownership. Within one month of project effectiveness, the project team constituted a High-level Steering Committee comprising senior members from government, which met regularly to review the project progress and provided policy-level guidance and strategic direction to OAGN. To manage day-to-day operations and implementation of the audit reform agenda, a Management Committee was formed once the World Bank approved the project. The program coordinator, designated as the primary focal point, was a senior OAGN official and served the project from beginning to end. The project team was experienced and successfully managed implementation of many project activities in different technical areas. It maintained a strong focus on capacity building, skill enhancement, and developed professional training capacity in the area of financial and performance audit. The project team was able to successfully award complex, large-value consultancy contracts within few months of project effectiveness. Contract management issues, faced during project implementation, were handled proactively. The project activities continued its momentum and pace even during crisis situations. There was good coordination and exchange of information among project team, World Bank officials, and donor partners to implement the project activities. In spite of all external difficulties, the project has achieved most of the desired outcomes.

(c) Justification of Rating for Overall Borrower Performance - Rating “Moderately Satisfactory”

61. Based on the above ratings, the overall borrower performance is rated as “Moderately Satisfactory”.

6. Lessons Learned

62. The SOAGN project has provided the following valuable lessons for sustaining reforms and scaling up support for future:

Sustenance and capacity building. While most of the project outputs and outcomes have been achieved, there is a need for continuous capacity building and sustenance of efforts in OAGN. The client has recognized this need and has incorporated it in its Strategic Plan 2016-2020.

22

Continuous client ownership for reforms. A project should be designed as per field requirements to get full ownership. In spite of unfavorable external circumstances, the client achieved most of the outcomes.

Holistic PFM reform approach. This project complemented the overall PFM reforms in Government and was aligned with the Bank’s SPFM and PRAN projects to a great extent. The PFM projects should be designed taking into account the overall holistic reform agenda of the country and should be sequenced properly to attain the maximum benefit. Such technical assistance projects should be complemented by policy reform operations.

No PAC affected project outcomes. As one of the key stakeholders in the accountability cycle, PAC was not available during most of the project period, which affected the compliance of audit issues at the government level in a systematic way and also affected the impact of audit.

Skewed Results Framework not capturing impact. Both original and revised Results Framework were lop-sided and focused more on the audit quality aspects of PDO rather than its impact.

Constraints in Capacity. The existing human resource profile and many vacancies are a serious constraining factor for OAGN. To sustain reforms, OAGN should recruit fresh talent with commerce background, invest in IT systems, and develop a talent pool of IT and audit professionals.

Enhancing audit impact. Even though audits are being carried out using NAMS, its compliance is not being tracked. To enhance the impact, an audit compliance module needs to be operationalized. Timely acceptance and dissemination of communication strategy by OAGN would have provided further impetus to the impact.

Proactive contract management. The OAGN identified key staffing issues and swiftly mobilized external consultants, which worked in favor of the project. Clients, with support of the Bank, addressed the issues in interpretation of scope, especially in the area of software implementation. The contract terms and conditions, including scope, should be clearly defined and regularly revisited between the parties to avoid ambiguity and ensure consistency.

Timely mid-term review and restructuring. The client and Bank took necessary steps to modify the Results Framework to meet project reporting and monitoring requirements on a timely basis and restructured the project during mid-term review, which facilitated proper measurement of results at the end of project.

Customizing commercial-off-the-shelf solution. It was a good strategy to adopt the existing commercial-off-the-shelf solution in Bangladesh and customize it for OAGN. This action resulted in faster development and deployment of software and also met the training needs of OAGN.

Next phase of support: With an expanding mandate of OAGN and immediate future requirements for internalization and sustainability of reforms, it will be imperative to obtain Bank’s technical knowledge and MDTF financial support for implementation of next phase of project.

7. Comments on Issues Raised by Grantee/Implementing Agencies/Donors

(a) Grantee/Implementing agenciesThe comments provided by OAGN have been positive and suitably reflected in the report. The OAGN has raised the issue of “Satisfactory” rating for the project. Since impact side of PDO was partly achieved, the ICR team has provided the overall rating of “Moderately Satisfactory”.

(b) Co-financiers/DonorsThe donor community has been extremely supportive and expressed satisfaction on the project performance.

(c) Other partners and stakeholders

23

The technical support received from OAG Norway and Office of the Comptroller and Auditor General of India were important factor for the success of this project.

24

Annex 1. Project Costs and Financing (to be filled based on final figures)

(a) Project Cost by Component

ComponentsUS$ million equivalent Percentage (%) of

appraisalAppraisal estimate Actual estimate

Component-1: Enhancing the quality of financial statement audits 1.00 1.00 100

Component-2: Upgrading the capacity to conduct performance audits 0.73 0.73 100

Component-3: Enhancing the impact of audit 0.50 0.46 92Component-4: Project Management 0.15 0.03 20

Total Baseline Cost 2.38 2.22 93.2Physical contingenciesPrice contingencies

Total Project Cost 2.38 2.22 93.2Project preparation costs

Total Financing Required 2.38 2.22 93.2

(b) FinancingUS$ millions equivalent

Source of Funds Type of Co-financing

Appraisal estimate

Actual/Latest estimate

Percentage (%) of appraisal

Borrower 0.08 0.05 62.5 Grant financing MDTF 2.30 2.22 96.5

25

Annex 2. Outputs by Component

Component 1: Enhance the quality of financial statement AuditsFinancial Audit Manual 2015 (updated)

Well drafted and serves the purpose of a reference document on finance audit.

Financial Audit Training Courseware Outlines the training modules on financial audit Guidelines on Annual Audit Planning Outlines the risk based audit planning methodology Technical report on pilot financial audits – 1 and 2

Reviews the pilots carried out by OAGN and rates them based on a pre-determined factor scale and provides suggestion for improvement.

NAMS finalization report, NAMS and EWP software

Outlines the design and development of the NAMS software. Also NAMS and EWP software has been developed and operationalized.

Cross-cutting themes supported under Component 1Peer review report by SAI India and consultants’ report on peer review report

Peer review done as per SAI PMF and outlines the strength, weakness, opportunities and areas of improvement for OAGN.

Human resources report and report on human resource retreat

Outlines the (a) organizational structure, (b) human resource management, (c) training, and (d) staff motivation. It also includes job descriptions, performance appraisal process, and competency-based model.

OAGN Strategic Plan and Operational Plan 2016-2020

Set outs the strategic plan and key areas to be addressed by OAGN. This has been developed based on the peer review report. The operational plan has been developed and is under implementation.

Component 2: Upgrading capacity of OAGN to conduct Performance auditsSector Audit Guide: Performance Audit of Local Government Sector

This audit guide has been prepared to assist performance auditors in the performance audit process at the local government sector.

Technical report on pilot Performance Audits – 1 and 2

Review the pilot performance audits carried out by OAGN as per ISSAI framework and provide guidance for improvement.

Technical report on selection of themes for performance audit

Outlines criteria as per ISSAI framework for selecting the themes where performance audits would be conducted.

Performance Audit Manual Outlines the methodology for conducting performance audits as per ISSAI framework.

Component 3: Enhancing the impact of auditStrategic development plan for PAC This report conducts a review of the existing challenges faced by PAC and

provides recommendations for improving its relationship with OAGN. Technical report on audit report format

Outlines the best practices in audit reporting

Communication strategy The report conducts SWOT analysis of OAGN’s current communication practices and recommends a new approach to enhance the OAGN credibility, trust and reliability with stakeholders

1

Annex 3. Economic and Financial Analysis

NA

2

Annex 4. Grant Preparation and Implementation Support/Supervision Processes

(a) Task Team members

Names Title Unit Responsibility/Specialty

Lending/Grant PreparationLes Kojima Sr. Financial Management Specialist GGO24 TTL World BankBigyan Pradhan Sr. Financial Management Specialist SACNP Co TTL, World BankBjorn Langerud Assistant Director General, International Division OAG NorwayMs. Lise Styrk Hansen Assistant Director General, International Division OAG NorwaySupervision Les Kojima Sr. Financial Management Specialist GGO24 TTL World BankBigyan Pradhan Sr. Financial Management Specialist SACNP World BankManoj Jain Lead Financial Management Specialist GGO24 TTL World BankRobert Bou Jaoude Program Manager of MDTF for PFM MNCIQ World BankMs. Lise Styrk Hansen Assistant Director General, International Division OAG NorwayAkmal Minallah Sr. Financial Management Specialist GGO24 TTL World Bank

Shambhu Prasad Uprety Sr. Procurement Specialist GGO24 World BankYogesh Bom Malla Financial Management Specialist GGO24 World BankPragya Shrestha Operations Analyst GGO24 World BankNagendra Nakarmi Sr. Program Assistant SACNP World BankICRS. Krishnamurthy Sr. Financial Management Specialist GGO24 TTL World BankArvind Prasad Mantha Financial Management Specialist GGO24 ICR Author, World Bank

Srinivas Alamuru Consultant GGO24 Technical Specialist, World Bank

(b) Staff Time and Cost

Stage of project cycleStaff Time and Cost (Bank Budget Only)

No. of staff weeks US$ thousands(including travel and consultant costs)*

Lending NA 38Supervision/ICR ICR – 3 Weeks 182

Total 220* Supervision and lending cost has been charged to TF010482.

3

Annex 5. Beneficiary Survey Results

(if any)NA

4

Annex 6. Stakeholder Workshop Report and Results

(if any)NA

5

Annex 7. Summary of Grantee's ICR and/or Comments on Draft ICR

The comments on Draft ICR provided by OAGN are given below:

1. The content and facts seems reasonable however we have some suggestion and comments on ICR which are as follows:

i. The project was operated during very critical political instability period. In the initial two years there was no Public Account Committee therefore this component of the project was slightly delayed however we have tried our best to hold discussion with PAC secretariat.

ii. The devastating earthquake in Nepal also hampered the activities at the final stage of the project. The consultant had concern about the force majeure but we had settled it amicably without paying additional cost.

iii. We have achieved the objectives as mentioned in the result framework.iv. Different manuals, guidelines and strategies has been prepared and approved which are in the process of

implementation.v. The procurement of consultant was done and handled very efficiently which was very large contract in

the project.vi. The system of computerized methodology Nepal Audit Management System NAMS was developed

which was very tough.vii. Participation of civil society organization CSOs in public audit is also a commendable initiative.

2. In this background the project should be rated as 'Satisfactory'.

7

Annex 8. Comments of Co-financiers and Other Partners/Stakeholders

1. DFID has provided an output score of A in its project completion review report. A summary of the DFID project completion report is provided below:

a. MDTF has supported the gradual roll-out of pilots using a risk-based audit methodology consistent with the International Standards of Supreme Audit Institutions (ISSAI) framework. All audits were quality assured and a workshop was conducted to exchange lessons learnt and refine methodologies. Overall, there have been a range of achievements to improve the quality of public sector audits since the project to strengthen the capacity of the OAGN commenced in February 2012. The financial audit manual has been updated, assisted by field testing on pilots, and translated into Nepalese; a bespoke computerized audit management system – the Nepal Audit Management System (NAMS) – has been developed; and a range of capacity building activities have been undertaken.

b. Further to the catalytic work of the MDTF’s demand side component, Citizen Participatory Audit could prove to be a highly valuable approach to improve the robustness and impact of audit reports. Citizen participatory audit has now been included in the OAGN’s long-term strategy and guidelines have been drafted to support its implementation. The OAGN plans to introduce this approach gradually over the next 3-4 years.

c. The original vision for this sub-project to enhance the impact of public sector audit has been difficult to realize due to factors beyond the MDTF’s control. There was no Public Accounts Committee for the first two years of the project. Once established in April 2014, due to various factors it proved difficult to provide capacity building support to them.

Recommendations for future program

2. The OAGN faces a huge organizational challenge in coming years as it continues to strengthen the quality of it audits and in parallel, substantially extends its coverage to include municipalities and Village Development Committees as mandated in the 2015 Constitution. The OAGN also plans to establish regional offices to fulfil its extended mandate. Consideration should be given to substantially scaling up MDTF support to strengthen to OAGN, in order to consolidate gains achieved and assist with the implementation of parts of its 2016-2020 Strategic Plan.9 This should include activities to strengthen audit follow-ups – the current overall compliance rate on implementing recommendations is low at around 50% (OAGN data) – and strategies to ensure that human resource capacity is sustainably built. Support should also be provided to strengthen citizen participatory audit.

3. While the Auditor General of Nepal is specified in the Constitution as an independent body, its budget and staffing levels continue to be controlled by the Ministry of Finance and Public Service Commission respectively. The OAGN has submitted a request to the Ministry of Finance to amend the Audit Act to enable the PAC to approve its budget and staffing, thereby enforcing its independence

4. Given weak accountability structures at the local level, the remoteness of some districts, and the lack of capacity at the OAGN, the OAGN should pilot test and if successful, scale up citizen participatory audit approaches.

5. With the political context unlikely to substantially reduce the number of PAC members, support should nonetheless be provided to assist the PAC in devising effective operational approaches, building the capacity of its Secretariat and strengthening its relationship with the OAGN.

6. Consideration should also be given to supporting activities to strengthen internal auditing to improve the internal control environment.

9 http://www.oagnep.gov.np/uploads/05092072-%20SP%202016-20-Final_1451555461.pdf

8

Annex 9. List of Supporting Documents

1. Project Completion Report by Co-Water International INC. – January 20162. Project Completion Review Report by DFID – March 20163. OAGN Strategic Plans for 2010-2012, 2013-15 and 2016-2020.4. OAGN Operational Plan for 2016-20205. OAGN Financial Audit Manual 20156. Technical Report on Financial Audits by Co-Water International INC.7. Technical Report on Performance Audit by Co-Water International INC.8. OAGN Implementation Progress Report 2015-169. OAGN Draft Communication Strategy Document10. SAI PMF Report – 201411. Human Resources Report by Co-Water International INC.12. Project Paper of the Bank – December 201113. Grant Agreement – February 201214. Restructuring Paper – March 201515. Amendment to Grant Agreement – March 2015

9

Annex 10. Comparison between original Results Framework and revised Results Framework

PDO Level Result Indicators

Original Indicator

Baseline Value

Original Target Values Revised Indicator


Values

Actual Value Achieved at Completion

Remarks

Audits substantially meeting OAGN financial statement auditing standards

35% 60% Audit of financial statements completed in accordance with ISSAI Standards

7 9 Indicator and target revised

NA NA NA Financial audit of spending units completed in accordance with ISSAI

21 28 Indicator added

In-depth performance audit reports submitted to Parliament in year

1 3 Performance audit conducted in accordance with ISSAI framework and submitted for transmission to Parliament.

6 6 Indicator and target revised

Current year audit observations implemented by the Executive within 1 year

<35% 50% Audit observations contained in ISSAI compliant audit reports (pilot audits) acted upon by Executive within 90 days

30% 32% Indicator and target revised

10

Intermediate Outcome Indicators

Original Indicator

Baseline Value



Values


Remarks

Office-wide audit plan implements risk-based concepts

N Full implementation

Office wide audit plan implements risk-based concepts

Full implementation

Fully implemented

No change

Number of financial audit teams included in pilot audits

0 15 This indicator has been dropped

Peer review recommendations by an SAI

N Implemented Peer Review Assessment using the SAI PMF framework

Next long-term Strategic Development Plan of OAGN prepared and agreed on the basis of peer-review report

Peer review completed and actions proposed as part of Strategic Plan 2016-20.

Indicator revised

NA NA NA Number of person-days of classroom training (including training-the-trainer in new financial audit procedures)

600 3,737 Indicator added

Strategic plan for performance audit division prepared and updated

N Y Indicator dropped

Lessons learned from current year pilots documented and reflected in revised audit guides and training materials

N Y Lessons learned from current year pilots documented and reflected in revised audit manuals, guides and training materials

Documented and reflected in audit manuals, guides

Performance Audit Guide updated based on pilots

No Change in indicator, target value revised

NA NA NA Enhanced public participation of CSOs and other stakeholders in performance audit process (number of participants/CSOs at events)


NA NA NA Number of times press meetings were arranged to

6 6 Indicator added

11

Original Indicator

Baseline Value



Values


Remarks

disseminate the results of performance audits with extensive coverage by media

NA NA NA Person-days of class-room training in performance audit


Current year audit observations accepted by the Executive within 35 days

60% 75% Audit observations contained in ISSAI compliant audit reports (pilot audits) accepted by the Executive within 35 days

50% 89% Indicator and target revised

Strategy document prepared with actions for improving relations with Executive, PAC, media, and other stakeholders

N Y Strategy document prepared with actions for improving relations with Executive, PAC, media and other stakeholders

Strategy document prepared and implemented.

Communication strategy document has been prepared, but yet to be approved.

No change in indicator, target value revised

Computerized databank of audit observations produces monthly status reports

N Y Computerized databank of audit observations to produce monthly status reports

Software implemented and databank populated for last 2 years audit observations

Software developed, but data not populated.

No change, target values revised

Annual work plan approved by Steering Committee

N Y NA NA NA Indicators dropped

Inception report approved by Steering Committee

N NA NA NA NA Indicators dropped

Mid-Term Report approved by Steering Committee

N NA NA NA NA Indicators dropped

Final Report approved by Steering Committee

N Y NA NA NA Indicators dropped

12

Annex 11. Results Chain: Linkage between PDO, intermediate indicators and components

13

Implementation Completion and Results Report · Web viewIMPLEMENTATION COMPLETION AND RESULTS...

Documents

Transcript of Implementation Completion and Results Report · Web viewIMPLEMENTATION COMPLETION AND RESULTS...