Special Study: Local Capital Markets Development - Interim Evaluation

EvD Approach Paper to Special Study: Local Capital Markets Development – Interim Evaluation 1

APPROACH PAPER – SPECIAL STUDY

Local Capital Markets Development – Interim Evaluation

Regional

July 2016

EBRD EVALUATION DEPARTMENT


The Evaluation department (EvD) at the EBRD reports directly to the Board of Directors, and is

independent from the Bank’s Management. This independence ensures that EvD can perform two critical

functions, reinforcing institutional accountability for the achievement of results; and, providing objective

analysis and relevant findings to inform operational choices and to improve performance over time. EvD

evaluates the performance of the Bank’s completed projects and programmes relative to objectives.

EvD’s Special Studies review and evaluate Bank activities at a thematic or sectorial level. They seek to

provide an objective assessment of performance, often over time and across multiple operations, and to

extract insights from experience that can contribute to improved operational outcomes and institutional

performance.

This report has been prepared by EvD and is circulated under the authority of the Chief Evaluator. The

views expressed herein do not necessarily reflect those of EBRD Management or its Board of Directors.

Responsible members of the relevant Operations teams were invited to comment on this report prior to

internal publication. Any comments received will have been considered and incorporated at the discretion

of EvD. Whilst EvD considers Management’s views in preparing its evaluations, it makes the final

decisions about the content of its reports.

Nothing in this document shall be construed as a waiver, renunciation or modification by the EBRD of any

immunities, privileges and exemptions of the EBRD accorded under the Agreement Establishing the

European Bank for Reconstruction for Development, international convention or any applicable law.

This report was prepared by Tomasz Bartos, Senior Evaluation Manager and reviewed by Joe

Eichenberger, Chief Evaluator of the EBRD Evaluation department.

© European Bank for Reconstruction and Development, 2016

One Exchange Square

London EC2A 2JN

United Kingdom

Web site: www.ebrd.com

All rights reserved. No part of this publication may be reproduced or transmitted in any form or by any

means, including photocopying and recording, without the written permission of the copyright holder. Such

written permission must also be obtained before any part of this publication is stored in a retrieval system

of any nature.


Contents

1. Introduction ...................................................................................................................... 4

1.1 Scope and nature of the evaluation

1.2 Rationale for inclusion in the work programme

1.3 Background to the evaluation

1.4 Other relevant evaluation work

2. Monitoring and self-assessment .................................................................................... 7

2.1 Monitoring reports

2.2 Transition monitoring

2.3 Self-evaluation

3. Evaluation methodology ................................................................................................. 7

3.1 Evaluation questions

3.2 Methodology

3.2.1 Strategic Relevance

3.2.2 Strategic Responsiveness

3.2.3 Strategic Results

3.3 Potential problems and limitations

4. Administrative arrangements ....................................................................................... 11

4.1 EvD evaluation team

4.2 Timetable

Annex 1: Results Framework for LC2 initiative (copied from LC2 Currency and Capital market Development – Strategic Initiative, November 2013)


1. Introduction

1.1 Scope and nature of the evaluation

This paper outlines the proposed approach for the Evaluation Department (EvD) to prepare an interim evaluation of

the European Bank for Reconstruction and Development’s (the Bank or the EBRD) outputs and outcomes designed

to support the development of local capital markets (LCM) in its countries of operation (COO).

The development of LCMs is a high priority for the EBRD and it was specifically identified as a core function in its

founding documents1. The importance of the LCM (and local currency financing) was highlighted in the EBRD’s

2006 and 2012 Transition Reports. Initially the EBRD mainly performed this function through the Treasury

Department’s borrowing and lending operations in various currencies, and the Banking Department’s issuance of

corporate bonds and support of Initial Public Offerings (IPOs).

The financial crisis of 2008 highlighted LCM risks arising from excessive reliance on foreign currency funding. The

crisis prompted the EBRD to deepen its engagement in the development of LCMs and it launched a Local Currency

and Capital Markets Development Initiative (LC2) in 2010. This initiative was designed to provide capital market

participants with access to local currency financing through the development of efficient and self-sustaining LCMs.

To achieve this objective, the EBRD established a dedicated team (LC2 team) in December 2012. The LC2 team

coordinated the activities of many operations in this area, including Banking, Treasury, Office of General Counsel

(OGC)/ Legal Transition Program (LTP) and Office of the Chief Economist (OCE). In November 2013 the Board

approved the “Local Currency and Capital Markets (LC2) Development Strategic Initiative” paper (the Strategy),

which outlined the EBRD’s main LCM goals and set specific objectives for the period 2013 - 2016.

Reflecting these developments, the evaluation will review the four years of LC2 operations from 2012 to 20152. The

evaluation will describe the evolution of the LCM development program, and how it led to the formulation of the LC2

Strategy3. It will identify the program’s links to country and sector strategies, and how LCM development efforts

were implemented. The evaluation will then examine a sample of LCM development projects, identify the

achievement of the planned outputs and outcomes, and to the extent possible, determine whether any broader

impacts can be attributed to LCM projects implemented to date. The evaluation will identify critical issues, and draw

lessons that can be used to help guide future LCM strategy development and the design of associated projects.

1.2 Rationale for inclusion in the work programme

LC2 is one of the EBRD’s three key strategic initiatives. As LC2 operations have been increasing in recent years,

there is a growing interest among Board members, management and staff to learn more about the performance of

its efforts to promote capital markets development. Given the importance of the initiative, the Board and

management decided to include this interim evaluation in the EvD’s 2016 work programme. At the same time, it is

understood the LCM program is on-going and the achievement of tangible results takes time. Therefore, this

evaluation is designed as an interim report, rather than a final analysis of the overall performance of the LCM

portfolio.

1 The Agreement Establishing the European Bank for Reconstruction and Development, signed on 29 May 1990, describes in its article 2 – Functions, measures through which the Bank is to fulfil its function of fostering the transition of its countries of operation towards open market-oriented economies and the promotion of private and entrepreneurial initiative. Point (v)of this article says that the Bank will “stimulate and encourage the development of capital markets” 2 The EBRD only began to classify projects as “supportive of LCM” in the Bank’s systems in 2014. Data for 2012 to 2013 was compiled by LC2 team manually. Data for earlier LCM-supportive projects (2010-11) was difficult to compile, as the initiative was at a nascent stage and only a few LCM projects were signed at that time. Therefore the earlier (pre-2012) period will be omitted from this review. 3 The EBRD’s local currency financing operations were evaluated by EvD during 2011 to 2013 (“Local Currency Lending and Borrowing 2000 to 2010”) and these findings will help set the context for the LC2 evaluation.


1.3 Background to the evaluation

The LC2 strategy set out five priority interrelated themes where the EBRD would concentrate its work:

1. Building stable and sustainable macroeconomic policy frameworks – this theme focused on promoting

prerequisites for LCM development, primarily in areas such as policies, skills and tools required to stabilise

inflation (e.g. inflation forecasting and development of yield curve models);

2. Improving the legal and regulatory environment to support capital market activity – this theme

targeted the improvement or development of legal frameworks and regulatory regimes for selected, more

advanced financing products, primarily through Technical Cooperation (TC) projects;

3. Developing financial market infrastructure including clearing and settlement – projects under this

theme supported LCM infrastructure for processing financial transactions such as: stock exchanges (SEs),

payment systems, central counterparties, central securities depositories, securities settlement systems and

trade repositories;

4. Developing the institutional investor base – this theme targeted the weak institutional investor base in

the Bank’s COOs (one of the key impediments to LCM development). This constraint was addressed

through a combination of TCs supporting pension reforms and several banking projects, which invested in

life insurance funds; and

5. Promoting a more efficient transaction environment and expanding the product range – most of the

LCM banking projects fell into this category

The EBRD implemented this strategic program by: (i) providing assistance to develop a favourable environment for

market-related activity (usually though policy dialogue and TCs), and (ii) investing (usually co-investing with private

investors) in local securities (bonds and stocks of listed companies, including IPOs).

During the first years of the LCM initiative the activities of the Bank (primarily LTT) and the LC2 team focused on

diagnosing the legal and regulatory environment (theme 2 of the LC2) related to local capital markets and to date

16 assessments have been produced. This activity was supported by capacity building/training provided to local

decision-makers to create a better understanding of the risks associated with overdependence on foreign funding

and emphasise the need for LCM development.

In 2012 the EBRD approved the LC2 Initiative Framework budget of €1.3 million, to support LCM development

through TCs. Between 2012 and 2015 the LC2 team initiated and implemented nearly 40 TC projects related to

LCM development4. The LC2’s TCs ranged from diagnostic and capacity building, to supporting integration of stock

exchanges or preparation of new laws and regulatory regimes (e.g. for derivatives or covered bonds). A total of 18

projects (mostly market assessments and other smaller TCs) were completed and the balance of the TC projects

are on-going. The EBRD’s TC efforts were concentrated on the Early Transition Countries (ETC) and Southern and

Eastern Mediterranean (SEMED) countries. The first TC supported the preparation of the first corporate bond issue

in the Kyrgyz Republic and TC projects in other ETCs followed. The EBRD signed Memorandums of Understanding

(MoUs) with six ETCs on cooperation in LCM development that provided frameworks to initiate several TCs.

Overall, about 20 TCs (half of the total) benefited at least one ETC or SEMED country.

4 This number includes market assessments, workshops/conferences and extensions.


Geographically, Poland accounted for over a quarter of all investment/financing projects (24), followed by Romania

(14) and Turkey (13). In 2015 Greece became one of the main recipients of LCM projects with five transactions

signed in that year. To date, the EBRD has signed only six LCM-supportive banking projects in the ETCs and two in

SEMED countries.

In terms of sectors, 70% of LCM projects (64) were Financial Institution (FI) transactions, and the balance was

spread over nine other sectors. These sectors were: Manufacturing and Services (M&S), Transport and Natural

Resources each having six projects; and Information and Communication Technology (ICT), Property and Tourism

(P&T), Agribusiness, Power and Energy (P&E), Equity Funds and Municipal and Environmental Infrastructure (MEI)

each having signed between one and three projects.

Almost 50% of the banking projects (45) classified as LCM development were investments in bonds (including

Eurobonds) issued in both local or foreign currencies. These projects were followed by: equity investments in listed

companies (32); Swap transactions (10); Deposit insurance facilities (3); and Asset Backed Securities (2).

A preliminary review of the portfolio indicates some of these transactions supported innovative financing products.

However, a large majority of LCM banking projects were in either corporate bonds or listed equities (in most cases

in more advanced countries). Nevertheless, these projects supported development of LCMs, as even in more

advanced COOs (e.g. Poland, Slovenia) corporate bond markets were still relatively underdeveloped, while

investments in listed equities in smaller countries boosted capitalisation of their stock exchanges and provided them

with the EBRD’s “seal of approval”. Given these results, in recent years the LC2 team concentrated TC on

supporting the development of the legal and regulatory environment (see theme 2) for new, more innovative

financing products, which could lead to potential investment transactions. Two main sub-themes/products

supported by the team were TC for: (i) derivatives regulatory reform, and (ii) covered bonds legal reform.

The LC2 team implemented the Strategy by developing and managing certain activities itself (mainly TCs), and

coordinated and advised the EBRD’s various departments on implementation of other activities (such as actual

investments) related to LCM development. In particular, the LC2 team worked closely with: the Banking department

(e.g Non-Banking Financial Institutions (NBFIs) on investments in stock exchanges and insurance funds; MEI

department on municipal bonds; Industry, commerce and agribusiness (ICA) department on corporate bonds;

Treasury on local currency borrowing and lending, and regulatory reform; and OGC/LTP on legal and regulatory

reforms.

Treasury department’s statutory activities (funding the Bank’s operations) were particularly conducive to supporting

LCM development. It issued bonds (some in local currency) in 14 markets of the Bank’s COO (including first local

currency bonds in ETCs such as Armenia and Georgia). To enable these issues, the EBRD worked with the

countries’ regulatory authorities and legislative bodies to prepare new or refine existing regulations in line with best

international practice. Moreover, there have been two LCY/LCM-related operations in the Treasury department’s

books - the TCX Currency Exchange Fund (signed in 2007) and the Frontier Clearing Fund (signed in 2015).

1.4 Other relevant evaluation work

There have not been any previous dedicated evaluations of the EBRD’s LC2 or LCM initiatives. During 2011 to

2013 EvD evaluated the early local currency financing and funding initiatives, known as “Local Currency Lending

and Borrowing 2000 to 2010”. The evaluation presented some findings and recommendations relevant to the LC2

team and Treasury. However the evaluation was focused on local currency lending and borrowing, and it did not

specifically address the LCM development issues.


An evaluation of the Legal Transition Programme until 2010 was carried out from 2011 to 2012 that reviewed the

legal and regulatory frameworks assessments (which related in part to the LCM assessments). This component of

the Legal Transition Programme was rated highly successful.

EvD prepared Operation Performance Assessment Validations (OPAVs) for two projects classified as LCM-

supportive. An additional seven projects are scheduled for validation in 2016.

During the course of this study, the EvD team will review relevant COO and the EBRD strategic documents, case

studies and sample projects, selected articles and publications related to any evaluation of the local capital markets

development projects financed by other international organisations or International Financial Institutions (IFIs), and

in particular, International Finance Corporation (IFC).

2. Monitoring and self-assessment

2.1 Monitoring reports

Most TCs on enabling environment initiatives are on-going and only a few completion reports have been prepared.

They are informative but very brief. The reports for banking projects proposed as case studies and sample projects

under this evaluation have been reviewed. These reports focused primarily on the clients’ financial performance and

provided few references to the extent the projects supported LCM development. No PMMs have been produced yet

for several of the more “landmark”/important LCM projects as they were signed recently and some of them have not

yet been fully implemented.

2.2 Transition monitoring

Transition Impact Monitoring System (TIMS) reports for most LCM projects reviewed by EvD were up to date, and it

will request an update for those projects which are behind schedule. Similar to portfolio management reports, TIMS

have not yet been produced for several recent projects.

2.3 Self-evaluation

This evaluation will not rate specific projects. It will concentrate on identifying early results of the EBRD’s LCM

projects and derive relevant lessons, rather than provide a detailed assessment of the performance of any

individual project. As a result, full Operation Performance Assessments (OPAs) on the case study/sample projects

will not be required.

The evaluation team will rely on interviews with Operation Leaders (OLs), bankers in relevant Resident Offices

(RO), and most importantly, the clients to obtain information on the current status and overall performance of the

case study and sample projects.

3. Evaluation methodology

3.1 Evaluation questions

The evaluation will seek to answer the following study questions:

(i) Were the LC2 strategy and its objectives relevant to the requirements of the COOs?


(ii) How responsive was the EBRD implementing the LC2 strategy and what is the status of

implementation?

(iii) What have been the early results of the LCM projects?

(iv) What have been the key issues and lessons that can be drawn from the implementation of the LCM

projects so far, which could improve future project design and execution effectiveness and efficiency?

Answers to these questions will contribute to an understanding of the larger objectives of the LCM program such as

the need to decrease dollarization, grow and diversify capital markets, and increase liquidity in the targeted

markets. The analysis will aim to understand which projects were most effective in meeting these objectives and the

critical constraints and risks impacting on program and project performance.

3.2 Methodology

The evaluation will review the EBRD’s approach to identify early results of promoting LCM development during the

four years between January 2012 and December 2015. The review will consist of three parts:

Strategic Relevance 3.2.1

(i) Review of the evolution of the EBRD’s approach to LCM development, particularly the differences

between 2010 Local Currency Initiative and 2013 LC2 Strategy;

(ii) Review of the alignment of key country strategies and sector strategies with the LC2 Strategy,

identifying objectives, expected outcomes and outputs, and expected scale and timing to achieve

results; and

(iii) An analysis of the relevance of the Strategy (the degree to which the Strategy reflected LCM

development gaps identified in the Capital Market Assessments for the Bank’s key COO).

Strategic Responsiveness 3.2.2

(i) Review of organisational arrangements within the Bank to execute the Strategy. In particular the review

will assess the arrangements for LC2 team’s coordination efforts, internally and externally (including the

extent of Bank’s collaboration with other IFIs);

(ii) Identify the level of resources allocated to LC2 over the evaluation period in terms of objectives,

instruments, countries, and sectors; and

(iii) Assess the evaluability of the program and projects, focussing on adequacy of definitions of outputs,

outcomes, and impacts; availability of baselines and targets; and adequacy of data collection and

reporting arrangements.

Strategic Results 3.2.3

The study will review the status of Strategy implementation and identify early results of selected LCM projects. The

two themes of building stable and sustainable macroeconomic policy framework; and developing the institutional

investor base were targeted relatively infrequently and it is premature to review the performance of these two

themes. The reasons why these two themes were not targeted to a greater extent will be explored to form a view on

their relevance.

The following three LCM themes were pursued by the EBRD and they will be the primary focus of the analysis of

results:


1. Improving the legal and regulatory environment for LCM;

2. Developing the capacity of LCM infrastructure; and

3. Promoting more efficient transactions environment and processes that can be used to expand

LCM product volumes and range.

These themes accounted for about 85% of the EBRD’s LCM development-related projects implemented to date. In

evaluating these themes, it is import to note that some LCM key projects are not yet operationally mature.

For each of the three focus themes, the evaluation team will select one case study and two sample projects.

A case study might be a TC project, an investment project, or both (if they are connected). The case studies will be

examined in detail, primarily based on a document review, and internal and external stakeholder interviews. The

case studies will be described and presented in the evaluation report’s annex.

Two sample projects will be selected for each focus theme to increase the project sample size5. The sample

projects will be subject to a detailed review, including internal and external stakeholder interviews. Key issues,

findings and recommendations derived from the sample projects reviews will be highlighted in the report. Unlike

case study projects, the sample projects will not be described in detail in the evaluation report.

Only those case study and sample projects which have been completed and where sufficient evidence exists to

assess their performance will be rated. The rating will reflect the delivery of the outputs and achievement of their

targeted versus actual outcomes and impacts on LCM development. No other aspects of the projects’ performance

will be assessed and no rating of overall project performance will be derived. The evaluation of all the LCM projects

during the evaluation period is expected to generate practical findings, which could be useful for the LC2 team in

the design and implementation of its programs and projects in the future.

Following preliminary discussions with the LC2 team, Banking, Treasury and other teams across the EBRD, it is

proposed to focus the evaluation (in terms of case studies and sample projects) on more innovative (“landmark”)

projects and TCs, which are expected to bring substantial change and have a strong demonstration effect on the

local capital markets, in some cases, even despite their recent vintage.

Table 2 presents case study and sample projects for each focus theme, which are proposed for the study. The

projects have been selected following a review of LCM-supportive TCs and investments, and extensive

consultations with LC2 and banking teams. Geographic and thematic diversity was also taken into account when

selecting these projects sample. A brief description of the current status of the proposed case study projects is

presented in Annex 2. However, following an in-depth review and interviews for each of these projects, the

evaluation team might decide to swap some of the projects’ categories, and present in the report a case study as a

project, which is currently designated as a sample project. These adjustments will depend on the nature of the

findings and learning potential the projects present.

5 An evaluation based on only one case study would not be sufficient to identify important issues and draw lessons


Table 2. The evaluation’s case study and sample projects

Theme Case study projects Sample projects

Improving legal and regulatory

environment

Derivatives Law and Regulations

Development TC - Ukraine

Covered Bonds Law and

Regulations Development TC -

Poland

Derivatives Law and Regulations

Development TC - Morocco

Developing financial markets

infrastructure

SEE Link TC – integration of smaller

stock exchanges in Western

Balkans (managed by Zagreb Stock

Exchange, Croatia)

2 investments in stock exchanges

Promoting more efficient transaction

environment and expanding product

range

Muni bond

2 bond projects

TC = Technical Cooperation

As part of the evaluation of the case studies and sample projects, the review team will interview representatives of

the following groups (other than the EBRD staff):

Bank clients (private and/or public – including ministries and/or central banks);

Bank’s or clients’ consultants;

Security issuers/arrangers (if different from the client);

Market regulators (security commissions);

Industry associations (where feasible);

Market participants (banks, selected institutional investors or mutual funds); and

Co-financiers and/or other IFIs active in this field in a given country.

The output of this exercise will be a Special Study report (of approximately 30 pages plus linked documents and

annexes providing detailed case study analyses). A presentation and Q&A session with the Audit Committee will be

held if requested.

3.3 Potential problems and limitations

The main challenge of this evaluation arises from the relatively recent nature of the EBRD’s most significant LCM

development projects. Most TCs are ongoing (although in most cases their first phases have been completed) and

some of the investment projects that will be reviewed were only signed in 2015. Therefore, a full evaluation of the

results of these activities will not be possible and the special study is designed as an interim evaluation. More

recent projects (TCs and investments) were selected as they best illustrate the EBRD’s “added value” in promoting

LCM development. These projects were more innovative from an LCM development perspective than the earlier

projects, which were primarily standard investments in corporate bonds and IPOs. As most of the field visits and

stakeholder interviews will take place in the autumn of 2016, these projects will be more mature and it will be

possible to observe early performance. Part of their impact relates to structure and innovativeness, which are

largely time-neutral. It is expected there will be sufficient data to perform the evaluation, and many of the areas

related to relevance, organizational arrangements, and implementation of existing operations are ready for

evaluation.


Finally, the study will require a high degree of sensitivity and tact (including observation of internal “Chinese walls”

and confidentiality arrangements) when reviewing projects, many of which have been classified as “highly

confidential” due to their potential to influence market movements in the case of information leaks (although this risk

is/was related primarily to the pre-approval stage). Any conclusions will have to take into account the wider country

context and acknowledge the limitation to generalisation of findings and recommendations.

4. Administrative arrangements

4.1 EvD evaluation team

This evaluation will be executed by Tom Bartos, Associate Director/Senior Evaluation Manager. It is envisaged that

one other staff member of the Evaluation Department will be involved in this study. They will be supported by a

sector consultant (Terms of Reference are attached as annex 3). Keith Leonard, Consultant or/and Barry Kolodkin,

Deputy Chief Evaluator will act as peer reviewer(s). An external panel member (to be identified) may be requested

to review the draft report.

4.2 Timetable

Table below presents the evaluation’s timetable.

Milestone (2016) Date

Approach paper prepared, circulated and approved May – mid June

Consultant contracted 1 July

Background documentation review April - July

Internal data collection and internal interviews May-August

Field missions and report drafting July – October

Draft report circulated to internal peer reviewer mid November

Draft sent to External Panel early December

Draft submitted to CE mid December

Draft circulated for Management Comments January 2017

Final editing and final distribution Late January 2017


Annex 1: Results Framework for LC2 initiative (copied from LC2 Currency and Capital market Development – Strategic Initiative, November 2013)

Key LC2 themes Output Outcome Impact 1. Building stable and sustainable macroeconomic policy frameworks

- Policy dialogue/ technical

assistance/ capacity building to authorities in CoO promoting stability oriented policies/improved risk management

- Loans/investments in instruments providing long term local (currency) funding (see Section 1.4)

- Lower and less volatile inflation

- Smaller asset/liability mismatches

Lower loan and deposit dollarization

2. Improving the legal and regulatory environment to support capital market activity

Policy dialogue/ technical assistance/ capacity building in CoOs promoting international standards/best practices in capital market regulation/ transactions

Improved regulatory framework/ transactions environment

- Larger market size - Improved market

liquidity

3. Developing financial market infrastructure including clearing and settlement

Policy dialogue/ technical assistance/ capacity building/ investments in CoOs establishing more effective financial market institutions

Improved operating environment in capital markets (faster/more secure execution, lower transaction costs)


liquidity

4. Developing the institutional investor base

Policy dialogue/ technical assistance/ capacity building/ investments in CoOs establishing/ enhancing institutional investors (e.g. pension funds, insurance companies)

More assets under management at institutional investors


liquidity

5. Promoting a more efficient transaction environment and expanding product range

Policy dialogue/ technical assistance/ capacity building/ investments in CoOs establishing/ enhancing financial market products/ benchmarks

- New instruments traded

- New benchmarks utilised


liquidity

Special Study: Local Capital Markets Development - Interim Evaluation

Documents

Transcript of Special Study: Local Capital Markets Development - Interim Evaluation