BEST PRACTICES FOR THE SUBMISSION OF ECONOMIC … · 2019-10-21 · European Commission, 1 DG...

1European Commission, DG Competition, Chief Economist

BEST PRACTICES FOR THE SUBMISSION OF ECONOMIC EVIDENCE AND DATA COLLECTION

Damien Neven*Chief competition economist, European Commission

Conference in honour of B. Hawk

Brussels, March 17, 2010

*The views expressed are those of the speakers and do not necessarily reflect those of DG Competition or the European Commission


Introduction

• More analysis of effects (more focused objectives, better technology, increased scrutiny by the courts)

• Identify the consequences of mergers, agreements or single firm conduct (by formulating a theory of harm and/or understanding of efficiencies)

• Assess the direction and magnitude of these effects empirically

• Trade-off between precision and administrability


Introduction • Popular scepticism arises from the observation that apparently

sound but contradictory pieces of evidence can be submitted

• Understandable but incorrect belief that professional application of rigorous methods should produce unambiguous and consistent results

• However, apparent divergences can be due to

– Intrinsic differences in quality. Some studies may be methodologically superior

– Differences in data, modelling approaches, empirical techniques (and unintentional mistakes)

– Understanding the sources of the differences (data, models) may provide important insights


Goals of the Best practices• Develop some quality standard

– To help those involved in the production and presentation of economic evidence

– Most importantly, to help those with responsibilities to make decisions

• Ensure that economic analysis meets certain minimum standards at the outset,

• Facilitate the efficient gathering and exchange of facts and evidence, in particular any underlying quantitative data, and

• Use in an efficient way reliable and relevant evidence obtained during the administrative procedure, whether quantitative or qualitative


Content of the Best Practices

• Provides recommendations regarding the content and presentation of economic or econometric analysis. To facilitate its assessment and replication

• Provides guidance to respond to Commission requests for quantitative data to ensure that timely and relevant input for the investigation can be provided.

• The desire to ensure transparency and accountability that guide these best practices apply to all parties involved in proceedings


Some basic principles

• An economic analysis may support an assessment of the anticompetitive or pro-competitive effects of a merger.

• Counterfactual• Assumptions need to be consistent with the facts of the industry under

consideration.– These assumptions should be carefully laid out and– the sensitivity of its predictions to changes to the assumptions should be made explicit.

• It is not necessary for economic submissions to actually formalize verbal arguments in a model, but this may help to:– clearly spell out the assumptions underlying an argument,– to check its logic consistency,– to assess effects of a high degree of complexity, or– to use the model as the theoretical basis for an empirical estimation


Criteria for assessing the quality of economic analysis

• The relevant question• Data• Methodology• Results• Robustness


A. The relevant question

• Needs to be formulated unambiguously• Properly motivated, taking into account:

– the nature of the competition issue– the institutional features of the markets and industries

under consideration– the relevant economic theory.

• The hypothesis to be tested (or null hypothesis) must be clearly spelled out as well as the alternative hypothesis or hypotheses under consideration.


B. The data• The intrinsic quality of an economic theory depends on the

extent to which the underlying assumptions match the corresponding economic facts.

• Likewise, empirical analysis depends on the relevance and the reliability of the underlying data.

• Not all facts can be observed or measured with high accuracy and most datasets are incomplete or otherwise imperfect. Hence, parties and/or DG Competition should become familiar with the facts and data and acknowledge its limitations explicitly.


• Describe data sources. Provide the data!• Report how the data were gathered, the sample selection

process, the measurement of the variables

• Explain/Describe data handling and data management

– Provide software code employed to generate the final dataset.

• Describe final data set (including variables!!)

• Explain anomalies in the data and efforts made to correct them.


C. Methodology• Motivate the choice of empirical methodology• Discuss choices in light of (a) their data limitations, (b) the

features of the market under investigation, and (c) the economic issues under consideration (the relevant question).

• Identification!! can be understood as:– clarifying the basis upon which one theory can be preferred to

another.– More than one set of parameter of the econometric model generates

the same distribution of observations

• Alternative methodologies should also be discussed


D. Results and implications

• Explain the details of the models• Share any documentation needed to allow timely

replication (e.g. the programming code used to run the analysis).

• Clear reporting understandable to non-economists• Discuss not only the statistical significance of their

results but also their practical relevance.• Discuss results with reference to relevant economic

theory


E. Robustness tests

• Check whether the empirical results are sensitive to changes in the data, the choice of empirical method, and the precise modelling assumptions;

• Test whether the results of the analysis can be generalised; and

• Compare the results of the empirical work in question with previous results in the relevant literature.

• In the case of econometric or simulation models: always provide a sensitivity analysis with respect to the key variables– (All results from the sensitivity analysis conducted should also be

reported and not only those that support the argument)


Practical recommendations for all parties

• Credibility is enhanced when limitations are addressed and acknowledged

• Facts that could not be audited/observed should not be endorsed

• Opinions should not disguised as statements of facts

• Dialogue between economists is useful, but needs to happen early and should not interfere with the process of developing the arguments


Responding to data requests• Parties should provide the data and codes timely, in an

appropriate format to allow immediate replication– Note the Commission may not be able to take the analysis further into

consideration if replication is not possible• The Commission shall also provide the data and codes (upon

request).– A data room may be necessary to protect the confidentiality of other

parties– Third parties are expected to allow access to the data they submit

• Third parties or complainants are equally expected to submit all the underlying data used in the analysis and allow access to a data room


• Dual objective:– Reduce the burden on the involved parties– Ensure data gathered is (i) Accurate and

Complete and (ii) that there is sufficient time to analyze it

• Ask for the adequate amount of data• Completeness, correctness, timely submission• Cooperation in good faith, early consultation


Final advice…Quantitative analysis needs to…

• Be based on clear economic theory that generates • …testable propositions (prices in the input market will go up

in vertical mergers?)– Which distinguish between alternative theories (estimating the

elasticity of demand of a firm in a dominance?)

• Be intuitive (be suspicious of price effects of 400%)• and replicable (track down aggregation in variables)• Use simple, well tested methods whenever possible

– Complexity of the analysis can lead to courts simply choosing to ignore the evidence


Examples of Poor Economic Submissions• Irrelevant submissions

– Example: report comparing financial multiples in different industries, which was unrelated to the issue under investigation (input foreclosure).

• Pseudo data analysis– Example: report which argues that a chain of substitution exists

because the price difference across end points of segments was smaller than the price difference between end points within segments

• Results hidden in legal submissions– Example: results of win/loss analysis inserted in form CO without any

data description or rigorous interpretation


• Theoretical claims that do not fit the facts of the case– Example: report arguing that the merger under review would take

place in an idealised bidding market and hence could not have any anticompetitive effect (without any data analysis supporting this claim)

• Analysis of unreliable data– Example: report analysing an unrepresentative sample of bidding data

• No data or methodology description– Example: results of econometric estimation initially submitted without

proper methodological explanation and data description


• Results poorly reported or interpreted– Example: econometric estimation presented without reporting standard errors

• Lack of sensitivity analysis– Example: report presenting one single result without any indication of its robustness to

underlying assumptions

• Replication issues– Example: data and computers codes not submitted, or submitted in such a way that

they do not allow timely replication and assessment

⇒ Dealing with some of these submissions may capture precious resources from DG COMP

⇒ Importance for submission to meet minimum standards given short deadlines


Conclusion

• Complementarity with best practice on procedures

• Economic evidence in courts

• Comparison with US

BEST PRACTICES FOR THE SUBMISSION OF ECONOMIC … · 2019-10-21 · European Commission, 1 DG...

Documents

Transcript of BEST PRACTICES FOR THE SUBMISSION OF ECONOMIC … · 2019-10-21 · European Commission, 1 DG...