Epistemology and Methods

Small-N and Large-N Studies

May 18 2010

Conflict vs. co-existence

• Methods are used to test theories or assist in theory-building

• Quantitative or qualitative methods have different strengths and weaknesses

• Different “group think” attitudes have led to sharp divisions

• Common quest, different routes…

Qualitative methods: what is this?

Other “label”: case study methods (single cases and comparison of cases)

• Mostly used qualitative method is:

• Process-tracing – Whether intervening variables between a hypothesized cause and

observed effect move as predicted by theories

• Also used, albeit less frequently, is:

• Counterfactual analysis– Whether x in a specified case was necessary for y

Case study designs

Forms of single case study design

1) Descriptive case study Written by participants or historians

2) Preliminary illustration of a theoryKeohane (1984) on the role of regimes

Case study designs

3) Disciplined interpretative case study

• Interpretation/explanation of an event by applying a known theory

• Could lead to improvement of theory

• Risk: underplaying evidence inconsistent with the argument, eclectic approach (which factors are more important)

• Remedy: Engage sincerely in alternative explanations, add counterfactual arguments

Case study design

4) Hypothesis-generating case study

• Schattschneider (1935) Politics, Pressures, and the TariffPressure group politics

• Kindleberger (1973)“that for the world economy to be stabilized, there has to be a stabilizer, one stabilizer“

Case study design

5) Least-likely (theory-confirming) case study

• Extreme case that is highly unlikely to confirm

• Lends strong support if confirmed

• Example: The WTO treaties constrain actor’s national policies – case-study on the US

Case study design

6) Most-likely (theory-infirming) case study

• An important single case study that disconfirmed the expected outcome even though conditions make the case favorable for theory

• Example: The WTO dispute settlement system is biased against emerging developing countries – case-study on Brazil’s application and success rate…

Case study design

7) Deviant case study (outlier cases)

• Shedding light on the limits of a theory

• Suggesting new hypotheses

• Japan’s attack on Pearl Harbor and Deterrence Theory (Russett 1967)

Case study design

Comparative methods (Mill’s Methods and Least-Similar and Most-Similar Case Comparisons)

• The method of agreement (least similar case design)

• Search for similar antecedent conditions / ideally necessary conditions

• E.g. negotiations in GATT vs. WTO (A: G2 power)

IV DV

Case 1 ABCDE Y

Case 2 AFGHI Y

Case study design

Comparative methods

• The method of difference (most similar case design)

• Method of controlled comparison

• BCDE (constant)

• E.g. disputes on similar cases: GATT vs. WTO

IV DV

Case 1 ABCDE Y

Case 2 ~ABCDE ~Y

Case study design

Advantages of case studies

• Generate valid theory

• Refining theory, generate new hypotheses

• Strong for documenting processes /making inference regarding causal mechanisms

• Finding omitted variables

• Key events better explained than in large-n statistical tests…

Case study design

Limits of case studies

• Less useful for systematic testing a theory

• Case selection bias

• Confirmation bias

• Potential indeterminacy

• Representativeness (generalizability vs. specificity)

• Lesser precision of magnitude of causal effects

Quantitative methods

What is statistical method capable of doing?

• Short-cut: “it permits the researcher to draw inferences about reality based on the data at hand and the laws of probability”

Quantitative methods

Advantages:

• Powerful tool to “aggregate information” from a large amount of data

• Clear transparent coding process (high reliability, possibility for replication)

• Visual display

• Test whether association between variables is a product of chance

Quantitative methods

Advantages:

• Measure the effect of a change on the IV on the DV

• Assess the “contribution” (explanatory power) of an IV (average explanatory effects)

• Mapping of “deviant cases”

• Generalizability

Quantitative methods

Limits:

• Identifying new variables

• Dealing with multiple conjunctural causality or equifinality

• Validity of operationalization of variables

• Role of important cases

Quantitative methods

Errors of Specification:

• Too much effort calculating correlations with little attention to theory (i.e. Democratic peace)

• Theory itself often imprecise/shallow – does not lend itself to be tested (i.e. Waltzian balancing vs. bandwagoning)

• Imposing a statistical model on the theory (inattention to causal processes...)

Quantitative methods

Errors of Inference:

• Focus on statistical significance (probability that relationship between A and B occured by chance) vs. substantive significance (magnitude of the relationship)

• Mining datasets /few non-results make it to publication

Summing Up (Mahoney and Goertz 2006)

Summing Up (Mahoney and Goertz 2006)