Testing theories vs explaining outcomes

5th seminar, reading group qualitative methodology

Carl Henrik Knutsen 14/7-08

Two modes of doing science?

• An analogy from econometrics: Calibration vs hypothesis testing

• How strongly do we believe in our priors, our theory?

• Theories, verification and falsification, Popper• Do we separate in practice between verification

and explanation?• W.V.Quine and the critique of Kant and other

empiricists: Testing and falsification

Explanation• Elster: must not confuse with provision of narratives…• Mechanisms again• Explaining by theory alone? One or many theories? The purpose

of the study: Paint the “whole picture” or focus on parts• How do we know if our explanation is the correct one? Multiple

coherent theories• The role of deducing several different observational implications,

and checking…but then we are involving testing• We can provide narratives that are coherent with the outcome, but

how can we assure the plausibility of our narrative being correct?• The two cornerstones of science: Logical coherence and empirical

evidence!

Testing theories• “Testing” whether our theory sketched up plausible explanations in a

particular case (observable implications, strategic tests with alternative theories, other evidence) VS testing the theory in general

• 1) Falsificationism gone mad in the social sciences: Rejecting theories on the basis of a case study– Deterministic theories– Most likely and unlikely cases, role of prior knowledge

• 2) Theoretical “believers” gone mad:– Stretching concepts– Monocausal explanations (awareness)– Ignoring alternative explanations

• Example of 1) Rejecting modernization theory on the basis of single case. Ex of 2) Rational addictions

Platt

• Why do some areas have more scientific progress than others?• Strong inference: Formally, regularly and explicitly:

– 1)Devise alternative hypothesis– 2) Devise crucial experiment– 3) Recycle the procedure, subhypothesis, sequential

hypothesis..exclude more possibilities• Being aware of the structure..problem oriented rather than

method oriented..the important role of induction• Aim: Interconnecting theory and empirical evidence in most

fruitful way to achieve scientific progress• The danger of tautological theories

Chamberlin• Creative and novel thinking in science, rather than mere following• Problematic: “the habit of some to hastily conjure up an

expIanation for every new phenomenon that presents itself”• The importance of solid description and descriptive inference

before explanation• Multiple working hypothesis: The danger of falling in love

reduced!• “Just as the investigator armed with many working hypotheses is more likely to

see the true nature and significance of phenomena when they present themselves, so the instructor equipped with a full panoply of hypotheses ready for application more readily recognizes the actuality of the situation, more accurately measures its significance, and more appropriately applies the methods which the case calls for.”

Chamberlin• “The explanation offered for a given phenomenon is

naturally ,under the impulse of self-consistency, offered for like phenomena as they present themselves, and there is soon developed a general theory explanatory of a large class of phenomena similar to the original one. This general theory may not be supported by any further considerations than those which were involved in the first hasty inspection... With this tentative spirit and measurable candor, the mind satisfies its moral sense, and deceives itself with the thought that it is proceeding cautiously and impartially toward the goal of ultimate truth. It fails to recognize that no amount of provisional holding of a theory, so long as the view is limited and the investigation partial, justifies an ultimate conviction.”

Geddes

• The consequence of lack of focus on basic design issues: Relative failure of parts of comparative politics in knowledge accumulation (development studies, democratization etc..)

• “Steer the course between lovely, untested theories and information unstructured by theories”..either grand schemes or unfocussed case-studies..

• Symptom: Sand castles

Geddes cont’d

• The main problem: Theories not checked against the facts• Unfruitful response: “inductive fact-gathering missions

resulting in a disorganized mass of information”..Why does this case study matter for my research and how can it be used?

• Induction and speculative theories not tested on new cases!!• Focus on finer mechanisms rather than inductive search

large correlates. Big questions, little answers. Theory-based disaggregation and process focus.

• Additional problems: Case-selection and casual attitude towards non-quantitative measurement

Geddes cont’d• How do we deal with big macro-structures..• The quest for explaining everything and developing grand schemes• Selective tests and disaggregation of grand theories..Are there

plausible elements from for example modernization and dependency theory?

• Paradigms and Kuhn: Successful research frontiers in need of smaller puzzles and paradoxes. Many unexplained paradoxes, from normal science to scientific revolution..

• The grand schemes and their dependence on empirical events…• Reason: problematic aspects of theory not taken out through testing

and checks against evidence..all or nothing approach makes it easier to throw out the whole scheme when difficult empirical events occurs. Theory disaggregation and theory refinement.

Geddes cont’d

• The importance of drawing out clear and testable implications

• The importance of precise operational definitions and classification/coding (measurement also non-quantitative)..sensitivity analysis..

• Trying to explain compound outcomes can be hurtful, testable hyp on processes and smaller sub-question

• More cases are better..Cases which generated theoretical framework vs novel evidence.