Relations Publisher: Routledge Revolution and Qualitative ...amoravcs/library/Final Copy.pdf ·...

This article was downloaded by [Princeton University]On 06 December 2014 At 1321Publisher RoutledgeInforma Ltd Registered in England and Wales Registered Number 1072954 Registeredoffice Mortimer House 37-41 Mortimer Street London W1T 3JH UK

Security StudiesPublication details including instructions for authors andsubscription informationhttpwwwtandfonlinecomloifsst20

Trust but Verify The TransparencyRevolution and Qualitative InternationalRelationsAndrew MoravcsikPublished online 03 Dec 2014

To cite this article Andrew Moravcsik (2014) Trust but Verify The TransparencyRevolution and Qualitative International Relations Security Studies 234 663-688 DOI101080096364122014970846

To link to this article httpdxdoiorg101080096364122014970846

PLEASE SCROLL DOWN FOR ARTICLE

Taylor amp Francis makes every effort to ensure the accuracy of all the information (theldquoContentrdquo) contained in the publications on our platform However Taylor amp Francisour agents and our licensors make no representations or warranties whatsoever as tothe accuracy completeness or suitability for any purpose of the Content Any opinionsand views expressed in this publication are the opinions and views of the authorsand are not the views of or endorsed by Taylor amp Francis The accuracy of the Contentshould not be relied upon and should be independently verified with primary sourcesof information Taylor and Francis shall not be liable for any losses actions claimsproceedings demands costs expenses damages and other liabilities whatsoever orhowsoever caused arising directly or indirectly in connection with in relation to or arisingout of the use of the Content

This article may be used for research teaching and private study purposes Anysubstantial or systematic reproduction redistribution reselling loan sub-licensingsystematic supply or distribution in any form to anyone is expressly forbidden Terms ampConditions of access and use can be found at httpwwwtandfonlinecompageterms-and-conditions

Security Studies 23663ndash688 2014Copyright copy Taylor amp Francis Group LLCISSN 0963-6412 print 1556-1852 onlineDOI 101080096364122014970846

Trust but Verify The Transparency Revolutionand Qualitative International Relations

ANDREW MORAVCSIK

ldquoДоверяй но проверяйrdquo (ldquoTrust but verifyrdquo)ndashOld Russian proverb often used by President Ronald Reagan

Qualitative analysis is the most important empirical method in the field ofinternational relations (IR) More than 70 percent of all IR scholars conductprimarily qualitative research (including narrative case studies traditionalhistory small-n comparison counterfactual analysis process-tracing analyticnarrative ethnography and thick description discourse analysis) comparedto only 20 percent whose work is primarily quantitative Total use is evenmore pervasive with more than 85 percent of IR scholars conducting somequalitative analysis1 Qualitative analysis is also unmatched in its flexibilityand applicability a textual record exists for almost every major internationalevent in modern world history Qualitative research also delivers impressiveexplanatory insight rigor and reliability Of the twenty scholars judged bytheir colleagues to have ldquoproduced the best work in the field of IR in thepast 20 yearsrdquo seventeen conduct almost exclusively qualitative research2

Moreover controlled studies reveal that experts on world affairs whose anal-yses are informed by more eclectic theory and the myriad ldquosituational factsof each historical episoderdquo (a mode in which qualitative analysis excels)

Andrew Moravcsik is professor of politics and public affairs at Princeton UniversityColor versions of one or more figures in the article can be found online at www

tandfonlinecomFSST1 Daniel Maliniak Susan Peterson and Daniel J Tierney TRIP around the World Teaching Research

and Policy Views of International Relations Faculty in 20 Countries (Williamsburg Virginia Institute forthe Theory and Practice of International Relations College of William and Mary 2012) Charts 28ndash30available at httpwwwwmeduofficesitpir_documentstriptrip_around_the_world_2011pdf SeeActive Citation 1 in online supplemental material Instructions for accessing this material can be found atthe end of this article

2 Maliniak et al Chart 42 These seventeen scholars are Michael Barnett Barry Buzan MarthaFinnemore Samuel Huntington John Ikenberry Robert Jervis Peter Katzenstein Robert KeohaneStephen Krasner David Lake John Mearsheimer Joseph Nye John Ruggie Jack Snyder Stephen WaltKenneth Waltz and Alexander Wendt

663

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

664 A Moravcsik

tend to predict future events significantly better than those who seek to pre-dict future events using average tendencies and abstract theory (hallmarksof formal and quantitative analysis) To borrow Tolstoyrsquos famous metaphorldquofoxesrdquo consistently outperform ldquohedgehogsrdquo3 No wonder scholars hold awidespread conviction that qualitative analysis is more policy relevant thanany other type of IR scholarship4

Yet IR researchers like all political scientists face a shifting environmentAcross the social sciences a transparency revolution is underway Concernabout the inability to replicate results or locate data in natural medical andsocial sciences clear evidence that scientific studies generate a suspiciousnumber of confirming results questions about whether scholarly citations areaccurate or informed and resulting questions about the legitimacy and cred-ibility of all types of academic scholarship have led journals professionalassociations foundations governments colleagues journalists and the pub-lic to press researchers to open their data analysis and methods to greaterscrutiny5 Such demands for transparency cross disciplinary and methodolog-ical barriers Statisticians are being asked to publicize datasets robustnesschecks and the procedures by which data are selected and manipulated Ex-perimentalists increasingly preregister experimental questions and protocolsand track subsequent protocol revisions in the course of conducting exper-iments Formal analysts post appendices with extensive derivations Quali-tative political scientists have also been at the forefront of the transparencyrevolution often working closely with their quantitative experimentaland normative colleagues through the American Political Science Association(APSA) National Science Foundation (NSF) Social Science Research Coun-cil (SSRC) various interdisciplinary research communities qualitative traininginstitutes university departments and major journals All this is part of a gen-eral trend in IR over the past two decades led by top research scholars in thesubdiscipline toward paying more explicit attention to qualitative methods6

3 Philip Tetlock Expert Political Judgment How Good is It How Can We Know (Princeton NJPrinceton University Press 2005) Chapter 3

4 Maliniak et al Chart 57 See Active Citation 2 in online supplemental material5 Jon Wiener Historians in Trouble Plagiarism Fraud and Politics in the Ivory Tower (New York

New Press 2005) Malcolm Wright and J Scott Armstrong ldquoThe Ombudsman Verification of CitationsFawlty Towers of Knowledgerdquo Interfaces 38 no 2 (MarchndashApril 2008) 125ndash32 David Goodstein OnFact and Fraud Cautionary Tales from the Front Lines of Science (Princeton NJ Princeton UniversityPress 2010) Ole Bjoslashrn Rekdal ldquoAcademic Citation Practice A Sinking Sheeprdquo Libraries and the Academy14 no 4 (2014) 567ndash85

6 For example Gary King Robert O Keohane and Sidney Verba Designing Social Inquiry ScientificInference in Qualitative Research (Princeton NJ Princeton University Press 1994) Steven Van EveraGuide to Methods for Students of Political Science (Ithaca NY Cornell University Press 1997) ColinElman and Miriam Fendius Elman eds Bridges and Boundaries Historians Political Scientists andthe Study of International Relations (Cambridge MA MIT Press 2001) Andrew Bennett Case Studiesand Theory Development in the Social Sciences (Cambridge MA MIT Press 2005) Audie Klotz andDeepa Prakash eds Qualitative Methods in International Relations A Pluralist Guide (Hampshire UKPalgrave Macmillan 2008) Henry Brady and David Collier Rethinking Social Inquiry Diverse ToolsShared Standards (Lanham MD Rowman amp Littlefield 2010)

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

Trust but Verify 665

This article explores what the transparency revolution may mean forqualitative political scientists particularly those in IR The central argumentis that although openness does pose some challenges of adaptation it pro-vides important positive opportunities The major challenge is to assure thatresearch transparency is implemented in a way appropriate to qualitativeresearch The first section of this article defines three types of researchtransparencymdashdata analytic and production transparencymdashand explainshow each is best understood in the qualitative IR tradition It then evalu-ates the practical tools currently available to enhance each An emergingconsensus among scholars journals associations and funders in politicalscience views ldquoactive citationrdquo as the most generally applicable and logisti-cally manageable standard of qualitative research transparency for generaluse It melds traditional citation web technology and new thinking on qual-itative methods to permit scholars to hyperlink citations directly to annotatedsources in an appendix The goal is to place a researcherrsquos qualitative data in-terpretive data analysis and methodological choices just one click away fromreaders Active citation can be supplemented by other transparency tools in-cluding traditional citations data archiving hyperlinks and databasesmdasheachof which is essential for particular purposes

The second section of this article weighs the costs and benefits of en-hancing qualitative transparency with active citation and other means Thebasic message is that greater transparency offers large potential opportunitiesand benefits without imposing an undue logistical burden Research trans-parency promises to enhance the richness rigor and policy relevance ofqualitative IR This is likely not just to improve the quality of IR scholarshipand attract more researchers to it but will encourage all political scientists toinvest in language skills area expertise policy and functional expertise his-torical knowledge interviewing and archival techniques innovative methodsof qualitative inference and other such skills All this could help revitalizequalitative IR research as well as increase the recognition (both inside andoutside academia) accorded those who conduct it

WHAT IS RESEARCH TRANSPARENCY AND WHY DOES IT MATTER

Defining Research Transparency

Research transparency mandates that ldquoresearchers have an ethical obligationto facilitate the evaluation of their evidence-based knowledge claimsrdquo Thisobligation can usefully be divided into three dimensions7 The first datatransparency rests on the premise that social scientists should publicize the

7 This tripartite distinction revises that found in American Political Science Association (APSA) AGuide to Professional Ethics in Political Science 2nd ed (Washington DC APSA 2012) 9ndash10 alsoavailable at Arthur Lupia and Colin Elman ldquoOpenness in Political Science Data Access and ResearchTransparencyrdquo PS Political Science amp Politics 47 no 1 (January 2014) 19ndash42 appendices A and B

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

666 A Moravcsik

data and evidence on which their research rests This helps readers appre-hend the richness and diversity of the real-world political activity scholarsstudy and to assess for themselves to what extent (and how reliably) thatactivity confirms particular descriptive interpretive or causal interpretationsand theories linked to it

The second dimension analytic transparency rests on the premise thatsocial scientists should publicize how they measure interpret and analyzedata Social scientific evidence does not speak for itself but is used to infer un-observable characteristics like identity preferences power beliefs strategicintent and causality For readers to understand and engage in scholarshipthey must be able to assess what the data measure how descriptive andcausal inferences are drawn from them and how precise and unbiased theyare

The third dimension production transparency rests on the premise thatsocial scientists should publicize the broader set of research design choicesthey make which gave rise to the particular combination of data theoriesand methods they use for empirical analysis Decisions on how to select datameasure variables test propositions and weight overall findingsmdashbeforeduring and after data analysismdashare often decisive in driving research re-sults Any such choices inevitably exclude other possible data measure-ments theories specifications tests and summaries These design decisionscan induce significant methodological biases invisible to most readers byexcluding consideration of ldquohiddenrdquo options When scholars run many testsin different configurations random chance dictates that some are likely togenerate positive results In qualitative political science specific concerns in-clude oversampling of confirming evidence (ldquocherry-pickingrdquo) unfair framingof alternative theories (ldquostraw-manningrdquo) conducting idiosyncratic and non-robust tests and aggregating findings unfairly For research to be productiontransparent authors must explain the processes and decisions through whichthey made these choices

Research Transparency as a Fundamental Norm of Social Science

Transparency is a foundational principle of scientific scholarship embracedby scholars across the full range of epistemological commitments theoreticalviews and substantive interests The celebrated physicist Richard Feynmanlocates the essence of scientific investigation in an ldquointegrity that corre-sponds to a kind of utter honestyrdquo which he defines in terms of transparencyldquoThe idea is to try to give all of the information to help others to judge thevalue of your contribution not just the information that leads to judgment inone particular direction or anotherrdquo8 Yet transparency is central not only to

8 Richard P Feynman ldquoCargo Cult Science Caltech 1974 Commencement Addressrdquo Engineeringand Science 37 no 7 (June 1974) 10ndash13

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


natural scientists and unambiguously positivist modes of inquiry it is just asessential perhaps even more so to the human and social scientists and moreinterpretivist modes of inquiry Philosopher of history R J Collingwood whofamously maintains that historical analysis involved carefully describing con-textually interpreting and ldquoreenactingrdquo past subjective experiences arguesldquoHistory has this in common with every other science that the historian isnot allowed to claim any single piece of knowledge except where he canjustify his claim by exhibiting to anyone else who is both able and willingto follow his demonstration the grounds upon which it is based [and] whatthe evidence at his disposal proves about certain eventsrdquo9

Transparency enjoys this unique status as a fundamental principleacross academia because nearly all scholars view scholarship as a collectiveenterprisemdasha conversation among scholars sometimes extending to thoseoutside academia as well This conversation cannot take place and thus so-cial science as we know it can have little intersubjective meaning withoutopenness and honesty among scholars about data theory and methods Re-search transparency fuels collective social science in two ways which aresummarized in the Russian proverb that President Reagan made famousldquoTrust but Verifyrdquo10

Transparency invites scholars to verify what their colleagues have writ-ten thereby fueling an essential collective conversation In its ideal formsound and relevant scholarship describes a cycle When new work appearsother scholars in the same research community are inspired to debate itand to conduct new research that challenges extends or borrows from itto move in innovative directions renewing the flow of research Citizenspractitioners and political leaders may apply elements of it feeding backtheir experiences in the form of new data and questions for researchers toanalyze Scholars are trained to contribute to the advancement of this collec-tive enterprise and are recognized and rewarded for doing so The smoothfunctioning of this cycle is what gives social science its credibility and le-gitimacy both inside and outside academia and what ultimately justifiessocietyrsquos investment in it

Transparency makes these things possible It does not achieve themdirectly but rather empowers scholars to achieve them collectively In thecase of any given piece of scholarship each reader is better able to assessthe richness and rigor of it to appreciate replicate criticize and debate it toimprove or extend it and even to borrow from it in order to produce workin unrelated directions Transparency is also a precondition for scholars todemonstrate excellence publicly as well as for their research communitiesto recognize and reward their contributions fairly It encourages researchers

9 RG Collingwood The Idea of History (Oxford Oxford University Press 1946) p 25210 James Mann The Rebellion of Ronald Reagan A History of the End of the Cold War (New York

Penguin 2009) 65 100 267 273

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

668 A Moravcsik

to direct their training and skills in ways that are productive empiricallyTransparency permits citizens private organizations sponsoring bodies andpublic decision makers to evaluate and apply the results with confidence andprecision In the end all this not only closes the cycle of research but displaysit publicly thereby enhancing the credibility and legitimacy of research

A research community in which scholars accept research because of theprominence of the author or the abstract sophistication of the methods isquestionable A community in which scholars can read and understand oneanotherrsquos work and verify and debate it when they choose fosters legiti-mate confidence Massive datasets copious citations clever arguments orsophisticated methods should not inspire trust without transparency Whyfor example should a reader be convinced by a ldquostructured focused com-parisonrdquo of foreign policy decision-making cases unless he or she can perusethe evidence track why the evidence supports one theory rather than an-other and understand why particular data and theories were selected11 Thisis not to say of course that all social science is exact and replicable in thestrict sense of interpreting any given body of evidence in only one way Itis to say that whatever their epistemology scholars owe their readers open-ness about data theory and methods even when (indeed especially when)they are making close calls about complex interpretive issues because thisis what constitutes scholarly debate and community

For these reasons transparency in contrast to almost all other method-ological ideals in social science tends to unify rather than divide scholarsThough specific research communities in political science are often sepa-rated sometimes irremediably by diverse methodological theoretical sub-stantive and normative commitments most scholars accept that transparencyis necessary for the members of each group to converse with one anotherfor each group to engage in productive disagreement with others and foroutsiders to view the overall enterprise as credible and legitimate Evidenceof the consensual tendency comes from the recent establishment of APSArsquosData Access and Research Transparency (DA-RT) initiative the incorpora-tion of new norms of research transparency into the APSA Guide to Pro-fessional Ethics in Political Science the positive consideration of new trans-parency standards by the American Political Science Review the launchingof a new interdisciplinary SSRC social scientific transparency project andmany other recent transparency initiatives in political science12 All of theseefforts were led by qualitative and quantitative scholars working togetherand secured the approval not just of mainstream empirical researchers but

11 Bennett Case Studies and Theory Development David Collier ldquoUnderstanding Process TracingrdquoPS Political Science amp Politics 44 no 4 (2011) 823ndash30 Peter A Hall ldquoSystematic Process Analysis Whenand How to Use Itrdquo European Management Review 3 (2006) 24ndash31

12 APSA A Guide to Professional Ethics

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


also interpretivists post-modernists ethnographers political theorists formalanalysts and others

Research Transparency in the Qualitative Tradition

Although the research transparency is a general ideal its proper applicationvaries greatly across specific research communities Qualitative analysis aspracticed in contemporary IR differs in relevant ways from other researchmethods These differences stem from three characteristics such researchtends to examine few cases intensively to employ diverse forms of tex-tual and factual evidence and to be structured around temporal or causalnarratives backed by ldquocausal processrdquo observations

These characteristics have implications for transparency The first im-plies that transparency standards must be consistent with intensive casestudy analysis The second implies they must be designed to handle abroad range of textual (and sometimes non-textual) sources The impli-cations of the third characteristicmdashthe use of narrative backed by processobservationsmdashrequires more elaboration Qualitative analysis generally con-ceives data as a set of heterogeneous ldquocausal process observationsrdquo withina case rather than as homogeneous ldquodataset observationsrdquo across cases asis common in statistical work A causal-process observation ldquois an insightor piece of data that provides information about context or mechanism andcontributes a different kind of leverage in causal inference It does not nec-essarily do so as part of a larger systematized array of observations Acausal-process observation may be like a lsquosmoking gunrsquo It gives insightinto causal mechanisms insight that is essential to causal assessment and isan indispensable alternative andor supplement to correlation-based causalinferencerdquo13 Often a single case can generate dozens or hundreds of theoret-ically relevant causal process observations This has important implicationsfor what data analytic and production transparency means and how it ispresented

Qualitative work normally appears in narrative form with myriad in-tervening observations In a formal sense the analyst enjoys considerableflexibility in the role weight and meaning assigned to each piece of evi-dence depending not just on its position in the underlying temporal narrativeor causal model as well as its intrinsic contextual reliability This type of nu-anced but rigorous and informed contextual interpretation of each sourcehighly prized in fields such as history and law is more appropriate to qualita-tive case study analysis than imposing on this process any set of simple rulessuch as weighting all data equally or randomly or simply placing all data

13 Brady and Collier Rethinking Social Inquiry 252ndash53 See also James Mahoney and Gary GoertzA Tale of Two Cultures Qualitative and Quantitative Research in the Social Sciences (Princeton NJPrinceton University Press 2012) 230ndash31 Van Evera Guide to Methods for Students of Political Science

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

670 A Moravcsik

in a general database as is appropriate to statistical work In order for suchan analysis to be transparent readers should be able to move efficiently inreal time from a point in the main narrative directly to the source and thescholarly interpretation that explains its significance and back again

In recent years political scientists have moved to establish norms oftransparency appropriate to the specific research community of qualitativepolitical scientists Building on the APSADA-RT initiative described abovea team of scholars has developed specific applied transparency guidelinesfor qualitative research14 The NSF is funding a Qualitative Data Repository(QDR) based at Syracuse University as well as various projects demonstrat-ing new transparency standards and instruments that use new software andInternet technologies15 A series of conferences workshops journal articlesand foundation projects are further elaborating how best to implement qual-itative transparency in practice16

A Core Instrument of Qualitative Transparency Active Citation

What practical tools are best able to enhance transparency in qualitativepolitical science given the specific characteristics outlined above Severalinstruments are available including traditional citation active citation ex-ternal hyperlinks data archives and qualitative databases Although perfectresearch transparency is an attractive ideal an appropriate and workablestandard of transparency for empirical researchers must take account of fivetypes of real-world constraints logistics intellectual property law confi-dentiality of data and other human subject constraints existing publicationpractices and the right of scholars to exploit new data Scholars are converg-ing to the view that in light of all these practical considerations the mostpromising and widely applicable basic standard for enhancing qualitativetransparency is active citation Other instruments and standards can also beuseful under narrower circumstances and are often best used in combinationwith active citation

Active Citation (AC) is a digitally-enhanced mode of citing empiricalmaterial In actively cited research any empirical citation to a contestableempirical claim is hyperlinked to an annotated excerpt from the originalsource which appears in an appendix attached to the scholarly work Themain text and citations of a scholarly article book or chapter remain justas they are traditionally no matter their format the difference lies entirelyin the new ldquoTransparency Appendixrdquo (TRAX) appended to the work whichcontains an entry for each source cited in support of a contestable claim (see

14 APSA Guidelines for Qualitative Transparency (2013) see Footnote 715 Qualitative Data Repository (QDR) Center for Qualitative and Multi-Method Inquiry Syracuse

University httpwwwqdrorg16 For guidance on qualitative methods see ibid

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


FIGURE 1 Active citation with transparency appendix

Figure 1) Each TRAX entry contains four elements (1) an excerpt from thesource presumptively fifty to one hundred words long (2) an annotation ofa length at the authorrsquos discretion explaining how the source supports theunderlying claim in the main text (3) a copy of the full-source citation fromthe footnote and optionally (4) a link to or scan of the full source TheTRAX also reserves an exceptional first entry to address general issues ofproduction transparency The excerpt and linkscan can easily be adaptedto presenting sources in visual audio cinematic graphic and other me-dia17 Active citations can be employed in almost any form of scholar workdirectly attached to unpublished papers manuscripts submitted for publica-tion online journal articles and e-books or as separate online appendicesto printed journals and books Like all appendices the TRAX would lie out-side word limits For budgetary reasons most journals would probably notinclude them in printed versions

Each element of AC promotes transparency in a distinct way The sourceexcerpt offers basic data transparency Its relatively short length provides in-terpretive context while avoiding most logistical legal confidentiality and

17 This is increasingly commonplace outside of academia For a creative use of multi-media activefootnotes in a museum website see Metropolitan Museum of Art wwwmetmuseumorg

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik

data-ownership constraints The optional scan or link offers the possibilityof referencing a complete source when it is feasible and informative Theannotation delivers basic analytic transparency by offering an opportunityfor researchers to explain how the source supports the main text The full ci-tation assures that each TRAX entry is entirely self-contained for convenientdownloading into word-processing bibliographical or database softwareThe exceptional first entry (as well as any specific methodological citations)enhances production transparency by addressing research design selectionbias and other general methodological concerns that remain insufficientlyexplained in the main text footnotes or empirical entries As it focuseson methodology the current article contains relatively few contestable em-pirical claims that properly illustrate AC still for examples readers mayexamine notes 2 4 19 and 37 for which active footnotes have been pro-vided18 In addition the pilot-data collections currently in development atQDR (described in more detail in the next paragraph) include active-citationcompilations with their accompanying TRAXs19

AC is rapidly taking root Published articles presentations at disci-plinary and interdisciplinary conferences and modules at training institutesand graduate seminars have elaborated AC in detail20 The new norms andstandards that APSA promulgated for qualitative transparency were devel-oped with prototypes of AC in full view and hence are compatible withthis approach to transparency21 With National Science Foundation fundingthe QDR has hosted various activities to refine the guidelines for AC withinput from methodologists and field researchers who employ archives in-terviews ethnographic observation secondary sources and other materialsThe repository has also commissioned ten ldquoactive citation compilationsrdquo aspilot projects with several by leading scholars of international relations andcomparative politicsmdashincluding Jack Snyder Elizabeth Saunders and myselffrom this symposiummdashto retrofit classic articles and chapters to the activecitation format22 Under the same grant developers have been creating toolsto assist in preparing TRAXs in particular via software add-ons to automatethe formatting of transparency appendices and individual entries in popu-lar word processing programs In September 2013 the editorial board ofthe American Political Science Review received a presentation on the new

18 Andrew Moravcsik ldquoExample of an Active Citation about Steven Spielbergrsquos lsquoLincolnrdquorsquo Metho-dological Memo Princeton University 2014 httpwwwprincetonedusimamoravcslibrary (see Stevens-ExampleActiveCitationpdf)

19 QDR httpwwwqdrorg20 Andrew Moravcsik ldquoActive Citation A Precondition for Replicable Qualitative Researchrdquo PS

Political Science amp Politics 43 no 1 (January 2010) 29ndash35 Moravcsik ldquoActive Citation and QualitativePolitical Sciencerdquo Qualitative amp Multi-Method Research 10 no1 (Spring 2012) Moravcsik ldquoTransparencyThe Revolution in Qualitative Political Sciencerdquo PS Political Science amp Politics 47 no 1 (January 2014)48ndash53

21 APSA Guidelines for Qualitative Transparency22 Moravcsik ldquoActive Citationrdquo Moravcsik ldquoActive Citation and Qualitative Political Sciencerdquo Moravc-

sik ldquoTransparencyrdquo

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


DA-RT principles and their instantiation in quantitative and qualitative re-search including active citation The board voted unanimously to furtherinvestigate bringing the journal in line with the new principles A work-shop co-hosted by APSA in September 2014 assembled thirteen editors oftop political science journals and resulted in a joint statement and commonadoption date for a package of quantitative and qualitative research trans-parency standards including AC23 In addition scholars representing themove toward qualitative transparency are participating in multi-method andinterdisciplinary initiatives by major foundations in this area including theSSRC There is good reason to expect that in coming years active citationwill become the norm for qualitative papers articles and books

Combining Active Citation with Other Transparency Instruments

AC appears to be the most efficient and effective general standard cur-rently available to enhance qualitative transparency in political science yetother instruments remain useful even essential for more specific purposesThese include traditional citation hyperlinks to webpages data archivingand databases None match ACrsquos overall combination of broad applicabilitylow demands of time and logistical effort consistency with the distinctiveepistemology of qualitative analysis and ability to deliver all three types ofresearch transparency Yet each has specific advantages and can be par-ticularly useful when employed in tandem with AC which is designed tofacilitate such mixed-method qualitative transparency strategies

TRADITIONAL CITATION

Legal academia and history rank highly among the disciplines in data andanalytic transparency This is achieved through ubiquitous discursive foot-notes containing long annotated quotations In many ways such traditionalfootnotes remain the most efficient transparency instrument Readers canscan the main argument citation source and interpretation at a glance Yetrecent trends in formatting publicationsmdashthe move to endnotes ever tighterword limits and so-called scientific citationsmdashhave all but banished discur-sive footnotes from political science24 These trends are not methodologicallyneutral they privilege quantitative research that employs external datasets

23 In addition to APSA serving a lead host the workshop is co-sponsored by Syracuse UniversityrsquosCenter for Qualitative and Multi-Method Inquiry and University of Michiganrsquos Center for Political Studiesand the Inter-university Consortium for Political and Social Research University of Michigan Ann Arbor

24 International Security provides admirably detailed traditional footnotes though it does not con-sistently provide either annotations or extended quotations For similar book citations see ThomasJ Christensen Worse Than a Monolith Alliance Politics and Problems of Coercive Diplomacy in Asia(Princeton NJ Princeton University Press 2011) Jack L Snyder The Ideology of the Offensive MilitaryDecision Making and the Disasters of 1914 (Ithaca NY Cornell University Press 1984)

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik

and cites secondary journals rather than data while blocking qualitative re-search from achieving data and analytic transparency Given the economicsof social science journals this trend is unlikely to reverse Fortunately how-ever AC can match the epistemic advantages of traditional citationmdashof whichit is essentially a technologically enabled extensionmdashwhile leaving the exist-ing (pre-appendix) format of articles and footnotes (and therefore their hardcopy format also) essentially unchanged

HYPERLINKING WEBSITES

Scholarly journals and papers that cite secondary research as well as jour-nalistic articles policy papers social media and government documentsregularly employ characteristic bright blue hyperlinks to external web pagesIn the modern world such links are often useful which is why AC makesexplicit a provision for them as an optional supplement Yet three barrierspreclude hyperlinks alone from serving as the primary transparency instru-ment25 First most citations in political sciencemdashincluding many primary doc-uments informal printed material interview transcripts ethnographic fieldmaterials and books published in the past half centurymdash remain unavail-able online (or hidden behind paywalls) in foreign languages or withinlong documents Human subject and intellectual property constraints limitthe scholarrsquos ability to save materials on the web Second web links are un-stable changing surprisingly often when periodicals and book series switchowners formats or archiving systems or when government agencies privatefirms or civil society groups reorganize26 One example must suffice in 2009the US State Department unexpectedly altered the web address of the canon-ical series of public documents on US foreign policy Foreign Relations of theUnited States (FRUS)27 Who knows what new formats and changes the futurewill bring Third even when efficient and stable web links are available theyoffer at best only data transparency while doing little to enhance analytic orproduction transparency At most we learn what a scholar cited but not why

DATA ARCHIVING

Quantitative social scientists promote data transparency primarily by archiv-ing datasets in repositories By analogy political scientists have created datarepositories for textual material notably the Qualitative Data Repository forpolitical science materials recently established with NSF funding at SyracuseUniversity28 Repositories are essential instruments especially for the purpose

25 Linked documentation is the principle underlying the software Zotero supported by the Mellonand Sloan Foundations httpswwwzoteroorgblogbuilding-a-sustainable-zotero-project

26 For an adventurous example see the Yale Law Journal Online httpwwwyalelawjournalorg27 US Department of State Foreign Relations of the United States 200928 QDR httpwwwqdrorg Henry Murray Data Archive at Harvard University

httpwwwmurrayharvardedu

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


of preserving complete collections of new and unique field data such asinterviews ethnographic notes primary document collections field researchmaterial and web-searches29 Archiving full datasets also builds strongerprocedural bulwarks against selection bias (cherry-picking) of specific quo-tations and documents out of a larger body of material

For these reasons AC is designed to be entirely compatible withdata archiving in that TRAX permits optional links to material in archivesYet practical considerations preclude using data archives as the primaryinstrument to achieve research transparency First most qualitative scholarssift through many times (even orders of magnitude) more source materialmost of it uninteresting than they peruse intensively or cite and thelogistical burden of depositing all such material is prohibitive Secondintellectual property law imposes narrow de jure and de facto limits oncopying textual sources including most secondary material publishedin the past half century a surprising amount of archival material someweb material much commercial material and almost all artistic productThird protection of human subjects creates an additional administrativebarrier enforced by university Institutional Review Boards (IRBs) that wouldrequire materials to be sanitized or placed under tight control before beingreleased Fourth if scholars want to exploit new data further then thepublication of complete datasets may have to be delayed as sometimesoccurs with quantitative data Fifth archiving at best enhances data andperhaps production transparency but it does little to improve analytictransparency Links from article to archives are often cumbersome fail todesignate particular passages or do not explain why a passage is important

DATABASES

Scholars increasingly employ databases such as Atlas Access Filemaker andEndnote to array and manipulate textual data This approach is extremelypromising even indispensable in encouraging scholars to analyze texts withcritical theoretical issues firmly in mind thereby enhancing data analyticand production transparency30 It is particularly useful in research designsthat emphasize macro-comparative inquiry systematic coding content anal-ysis mixed-method analysis or weighing of a large number of sources toestimate relatively few carefully predefined variables31 AC aims to be fully

29 Colin Elman Diana Kapiszewski and Lorena Vinuela ldquoQualitative Data Archiving Rewards andChallengesrdquo PS Political Science amp Politics 43 no 1 (January 2010) 23ndash27 QDR ldquoA Guide to Shar-ing Qualitative Datardquo Qualitative Data Repository Project funded by the National Science FoundationDecember 2012

30 Evan S Lieberman ldquoBridging the Qualitative-Quantitative Divide Best Practices in the Devel-opment of Historically Oriented Replication Databasesrdquo Annual Review of Political Science 13 (2010)37ndash59

31 For a fine example see the work of the Research Network on Gender Politics and the Statehttplibartswsueduppparngs

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik

compatible with this trend which is why TRAX entries are formatted (andsoftware designed) so they can be downloaded Yet there is little reasonto believe that databases will either become ubiquitous or displace the tra-ditional narrative structure (on which AC builds) as the primary mode ofqualitative presentation First the narrative structure is well suited to theepistemology of case study analysis because it easily permits the analyst topresent (and readers to follow) a multistage causal process backed by aflexible and nuanced contextual analysis of individual steps in the sequenceeach of which is linked to one or more documented and annotated pro-cess observations Second compared to databasing narrative analysis is lesscostly and more flexible analysts need not commit early to particular vari-ables divide sources into small snippets code each one separately entereverything formally or structure the study accordingly Because change islow cost narrative encourages open-ended exploratory analysis aimed attheory generation a comparative advantage of qualitative work Third aslong as other disciplines such as law and history retain narrative structuremaintaining a similar format in political science promotes interdisciplinarycompatibility and interaction

For all these reasons the emerging trend in political science appears tobe to employ AC as the general transparency standard for evidence presentedin a narrative form backed by traditional citations hyperlinks data archivingand databases in appropriate circumstances

ENHANCING QUALITATIVE TRANSPARENCY IN IRA COST-BENEFIT ANALYSIS

Using AC to enhance qualitative transparency comports with scholarly idealsIt is now enshrined as a formal APSA guideline Technologies and standardsexist to achieve it Yet qualitative IR scholars will naturally wonder about itspractical costs and benefits This section argues that research transparencyoffers unprecedented opportunities to improve the richness rigor and rele-vance of IR research and to bolster the reputation of qualitative scholarsmdashandit does so at surprisingly low cost

The Benefits of Qualitative Research Transparency

Enhanced qualitative transparency offers at least five major benefits

MORE QUALITATIVE DATA

Greater data transparency promises to make available more abundant high-quality and accurate textual evidence thereby helping to improve the qualityof current scholarship and to reduce the cost of future research

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


Anyone who seeks to examine the evidence underlying contemporaryqualitative IR scholarship is likely to be frustrated Genuine source materiallies beyond the reach of nearly all readers Articles rarely quote sources ver-batim and even when they do almost never at sufficient length for a readerto assess the evidence in context Incomplete or vague citations (withoutpage numbers for example) are common Often references invoke genericmaterial such as chapters of secondary sources leaving the exact evidentiarybasis of the claim obscure Even when sources are precisely cited in a worldof burgeoning research publication they are often too costly to replicate32

Relatively few are available online Under the best circumstances findingsuch sources requires days in a university library usually with the help ofan interlibrary loan Specialized and informal publications foreign materialsgovernment documents interviews or ethnographic observation notes areoften available only from authors and it is not uncommon for authors to beunable or unwilling to provide themmdashparticularly years later Language andlegibility barriers can be prohibitive In two recent graduate seminars mystudents found that even in highly praised IR research a substantial portionof cited materials (generally around 20 percent) cannot be identified or lo-cated at all even by an entire class that is working together with the helpof authors The result generally few if any IR political scientists have anyinkling what cited sources actually say

This lack of transparency is itself worrisome as it leaves us with no pre-cise way to assess the accuracy of evidence in contemporary political sciencethus violating a foundational practice of social science IR scholars almostnever challenge the quality or reliability of one anotherrsquos textual evidenceeven though qualitative scholarship in IR is supported by a thinner and morederivative level of textual evidence than in law history policy analysis or asubdiscipline such as comparative politics Some make a virtue of this basedon the questionable belief that political scientists should abjure primary evi-dence and stick only to secondary sources33 This is problematic howeversince it would mean that qualitative researchers must renounce archival andfield research and can never revise the received empirical wisdom of otherdisciplines

Even more troubling are recent examples suggesting that the primaryand secondary sources cited in qualitative political science contain signif-icant inaccuracies Students in my graduate seminar on qualitative meth-ods have used AC in recent years to replicate a number of highly praisedworks source-by-source Consistently around 10 percent (beyond the 20

32 Mark Bauderlein ldquoWe Must Stop the Avalanche of Low-Quality Researchrdquo The Chronicle of HigherEducation (13 June 2010)

33 Ian S Lustick ldquoHistory Historiography and Political Science Multiple Historical Records and theProblem of Selection Biasrdquo American Political Science Review 90 no 3 (September 1996) 605ndash18

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik

percent that could not be located) do not support underlying claims34 Thediscrepancy moreover tends to increase as the sources become more cen-tral to core causal arguments which suggests the evidence is not just sloppybut biased One example must suffice35 a recent major press book backedby articles in International Security and policy recommendations in For-eign Policy selectively edits quotations from numerous primary and sec-ondary sources so they state the opposite of their unambiguous meaningin contextmdashconsistently in ways favoring the authorrsquos thesis36 Several disci-plinary referee processes failed to catch these errors

Of course pockets of scholarship employing exceptionally rich nu-anced well-documented work exist in which scholars do engage in knowl-edgeable debate Yet these are exceptions that prove the rule in the sensethat they tend to cluster where scholars can recapitulate sources (as well asarguments and methods) from other disciplines (eg the causes of WorldWar I international law) rich area studies literatures (eg Chinese foreignpolicy Vietnam War end of the Cold War) or extensive policy analysis tra-dition (eg European integration nuclear policy and other military strategyissues)37 Such work is therefore not as common as it should be Moreoverbecause in the eyes of most scholars richly sourced work is indistinguishablefrom work with thinly or incorrectly sourced work these scholars do notalways receive the recognition they deserve as methodological models

AC backed by other tools addresses this issue by enhancing data trans-parency This in turn encourages scholars to attend to the quality accuracy

34 For examples from natural science see K S Larsson ldquoThe Dissemination of False Data throughInadequate Citationrdquo Journal of Internal Medicine 238 (1995) 445ndash50 James K Wetterer ldquoQuotationError Citation Copying and Ant Extinctions in Madeirardquo Scientometrics 6 no 7(2006) 351ndash72 Wrightand Armstrong ldquoThe Ombudsmanrdquo 125ndash32

35 It would be unprofessional on my part to criticize specific works without extensive documentationand an opportunity for rebuttal by the author both of which go beyond the scope and space limits of thisarticle However one recent work Sebastian Rosatorsquos book Europe United Power Politics and the Makingof the European Community (Ithaca NY Cornell University Press 2011) does make claims that morethan one scholar has chargedmdashand in some cases Rosato has admittedmdashare based on erroneous datadata analysis and biased methodology See Andrew Moravcsik ldquoDid Power Politics Cause EuropeanIntegration Realist Theory Meets Qualitative Methodsrdquo Security Studies 22 no 4 (December 2013)773ndash90 Robert H Lieshout ldquoReview of Europe Unitedrdquo Journal of Cold War Studies 14 no 4 (Fall 2012)234ndash37 Sebastian Rosato ldquoTheory and Evidence in Europe United A Response to My Criticsrdquo SecurityStudies 22 no 4 (December 2013) 802ndash20 Accordingly I have cited it as an example

36 For an example of a debate over whether verbatim quotations were edited so as to change theirmeaning radically which in two cases the author concedes see Moravcsik ldquoDid Power Politics CauseEuropean Integrationrdquo 786ndash88 For Rosatorsquos admissions see Rosato ldquoTheory and Evidence in EuropeUnitedrdquo 808n13 807n12 808n15 respectively

37 Jack Levy Thomas J Christensen and Marc Trachtenberg ldquoCorrespondence Mobilization andInadvertence in the July Crisisrdquo International Security 16 no 1 (Summer 1991) 189ndash204 ChristensenWorse Than a Monolith Snyder The Ideology of the Offensive Elizabeth Saunders Leaders at WarHow Presidents Shape Military Interventions (Ithaca NY Cornell University Press 2011) Keir A LieberldquoThe New History of World War I and What It Means for International Relations Theoryrdquo InternationalSecurity 32 no 2 (Fall 2007) 155ndash91 Keren Yarhi-Milo Knowing Thy Adversary Leaders Intelligenceand Assessments of the Adversaryrsquos Intentions (Princeton NJ Princeton University Press 2014)

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


and originality of evidence The need to present an explicit and annotatedquotation in context is likely to make researchers more aware of what theycite at a minimum outright misquotations of the type mentioned abovewould become nearly impossible unless the author committed outrightfraud Referees critics and readers would recognize superior evidence morequickly shifting disciplinary norms and incentives to favor it The reputa-tion of journals would increasingly come to rest not just on enforcing trans-parency but on presenting rich and accurate textual evidence while avoidingembarrassing errors and omissions This in turn requires that referees andcritics be knowledgeable qualitative researchers able to recognize the dif-ference In training and advising graduate students hiring and promotionof scholars research funding and professional recognition greater valuewould be placed on increasingly rare skills and expertise required to col-lect and manipulate high-quality textual evidence foreign language abilityarea knowledge historical expertise policy experience ethnographic obser-vation archival inquiry interviewing design and other field research skillsThis is an opportune moment given innovative new methodological workin some of these areas38

Transparent evidence does not only help encourage greater accuracyIt also helps scholars convey a deeper more vivid and nuanced image ofpolitics Through AC political actorsmdashpoliticians social organizers interestgroup leaders contemporary observers technical specialists and commoncitizensmdashcan address readers directly expressing their perceptions beliefsinterests cultural frames deliberative processes and strategic choices in theirown words Scholars of all epistemic persuasions should welcome this Tradi-tional historians interpretive analysts and cultural theorists should applaudthe implication that to grasp politics fully one must comprehend the sub-jective meanings and beliefs cultural frames and discourses through whichsocial actors interpret and communicate their situations More positivistic ra-tionalist or materialist scholars should welcome the opportunity to assemblemore observable implications concerning the motivations strategies knowl-edge and beliefs of actors in order to test conjectures about their behavior

A data-rich environment does not simply enrich current scholarship Itencourages future research by allowing scholars to reuse published evidenceat low cost Today this dynamic rarely works well One study concludesldquoMost data generated by American qualitative and multi-method social sci-ence are used only oncerdquo39 This is understandable since the evidence itselfis not truly visible AC radically reduces the cost of viewing and therefore

38 Marc Trachtenberg The Craft of International History A Guide to Method (Princeton NJ PrincetonUniversity Press 2009) Layna Mosley ed Interview Research in Political Science (Ithaca NY CornellUniversity Press 2013) Lisa Wedeen ldquoReflections on Ethnographic Work in Political Sciencerdquo AnnualReview of Political Science 13 (2010) 255ndash72 Diana Kapiszewski et al Field Research in Political SciencePractices and Principles (New York Cambridge University Press forthcoming)

39 Elman Kapiszewski and Vinuela ldquoQualitative Data Archivingrdquo 23

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik

reusing data An afternoon with a dozen articles can yield hundreds ofprecisely cited copiously quoted carefully annotated sourcesmdashsomethingthat today can take weeks or months to assemble Scholars can collect evalu-ate download and reanalyze them or more likely use them as a springboardto deeper research This particularly benefits younger scholars catching upto the research frontier This virtuous circle of data creation also helps facili-tate comparative analysis in which similar situations are examined in varioussettings (eg countries issues time periods) using evidence from differentsources meta-analysis in which the empirical results and content of differentstudies is assessed and secondary use of data commonplace in quantitativetradition in which evidence collected in one context is employed for newand entirely unrelated purposes All this promises to expand research op-portunities for individual qualitative scholars to render qualitative analysismore prestigious and to improve research across the sub-discipline

MORE RIGOROUS ANALYSIS

Greater analytic transparency encourages more nuance and rigor in drawingcausal theoretical inferences from individual sources

Qualitative analysis is based in part on contextual analysis of individ-ual pieces of evidence To this end scholars invoke textual exegesis sec-ondary history the relationship among various pieces of data and arealinguistic and policy expertise Yet readers often find it difficult to replicateprecisely how scholars draw descriptive and causal inferences Accordinglyeven when data is available such inferences are rarely debated in IR Thisis troubling not simply because we do not know how reliable qualitativedata analysis is in IR as a whole but because specific cases clearly exist inwhich IR scholars lack the contextual knowledge to draw valid causal or de-scriptive inferences40 As with data exceptions prove the rule rare pocketsof exceptionally high-quality analysis exist but they tend to cluster wherescholars can draw on prior work in area studies policy analysis history orlaw and they are underappreciated 41

AC backed by other transparency tools enhances analytic transparencyby permitting scholars to add annotations explaining how they interpretspecific qualitative evidence Now is an opportune moment to encouragegreater care in drawing causal inferences from qualitative data and to facil-itate greater recognition for superior interpretive skills In recent years IR

40 For an example of how a major theoretical point can turn on contextual interpretation of adocument see Moravcsik ldquoTransparencyrdquo 782 Rosato ldquoTheory and Evidence in Europe Unitedrdquo 815ndash16Documents Diplomatiques Francais no 297 tome 1 1955 I have activated the final citation to the originalsource and my annotation addresses the debate See Active Citation 3 in online supplemental material

41 Yuen Foong Khong Analogies at War Korea Munich Dien Bien Phu and the Vietnam Decisionsof 1965 (Princeton NJ Princeton University Press 1992) examples in note 38

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


scholars qualitative methodologists and scholars in other disciplines haverefined techniques of qualitative data analysis and these innovations are nowtaught to graduate students These include improved counterfactual analy-sis coding schemes process-tracing various schemes of necessary andorsufficient conditions discourse analysis and content analysis42 AC wouldencourage such techniques to be employed in an explicit or sophisticatedmanner rarely the case today and more robust debate and replication ofsuch analysesmdashas explained in greater detail in the next section All thispromises to enhance the analytical rigor of qualitative analysis and thusbolster the academic respect and recognition such analysis receives

IMPROVED RESEARCH DESIGN

Greater production transparency encourages improvements in methodolog-ical rigor and general research design particularly regarding issues of selec-tion bias

Qualitative IR scholars are not immune from bias in the selection ofdata theories and methods They often enjoy almost unchecked discretion tochoose among thousands of primary and secondary sources numerous casesand observations myriad formulations of IR theories and many standardsof theory testing With the exception of theoretical literature reviews andthe issue of case selection scholars almost never acknowledge (let alonemake transparent) procedural elements of qualitative research design FewIR works employ explicit strategies to select reliable and high-quality datato sample that data in an unbiased fashion to test theories with an equal exante probability of (dis)confirmation to aggregate results systematically or todefend the appropriateness of particular qualitative testing proceduresmdashletalone to triangulate or conduct robustness checks

The current lack of transparency makes it impossible to know preciselyhow common cherry-picking and similar biases are in qualitative politicalscience yet they are probably widespread One reason is that prevailingmethodological standards do not even pretend to be unbiased Often schol-ars present dozens of pages of qualitative evidence confirming a favoredtheory followed by a few pages of evidence disconfirming alternative theo-ries regardless of the state of the historical or primary literature being sam-pled43 Summaries of findings are presented almost without constraint even

42 See Derek Beach and R B Pedersen Process-Tracing Methods Foundations and Guidelines (AnnArbor University of Michigan Press 2013) James D Fearon ldquoCounterfactuals and Hypothesis Testing inPolitical Sciencerdquo World Politics 43 no 2 (January 1991) 169ndash95 Levy et al ldquoCorrespondencerdquo BennettCase Studies and Theory Development Mahoney and Goertz A Tale of Two Cultures James Mahoneyand Rachel Sweet Vanderpoell ldquoSet Diagrams and Qualitative Researchrdquo Comparative Political Studies(forthcoming)

43 For a critique see Lustick ldquoHistory Historiography and Political Sciencerdquo Some scholars maintainthat sampling is unnecessary Challenged on the lack of an explicit strategy and potential biases in

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik

in the natural sciences evidence shows considerable error44 Thesis commit-tees journal referees conference commentators or job committees almostnever challenge or discount such results Examples exist of IR scholars pub-lishing cherry-picked evidence for a favored theory against straw-mannedalternatives and framing the results favorably45

The exceptional first TRAX entry especially dedicated to this purposeprovides space in AC (outside the word limit) to elaborate selection biasresearch design robustness checks and to describe how the research wasconducted This can be supplemented by actively cited individual refer-ences to methodological points in the main text This encourages scholarsto reduce these biasesmdashor at least to make readers aware of themmdashandempowers readers to assess the reliability of research for themselves to bemore cautious in accepting results to challenge publicly the methods orto conduct methodologically improved research All this parallels the com-mon practice in quantitative and experimental political science of submittingarticles for publication with long appendices justifying the choice of datacoding scheme choice of variables statistical approach and experimentalprotocol backed by robustness checks

It is an opportune moment for such reforms because scholars can ex-ploit a large overhang of recent innovations in qualitative methods Asidefrom the work on field research and data analysis noted above importantwork has appeared on process tracing structured focused comparison caseselection formal analytic narrative QCA and fuzzy set analysis medium-n comparative case-study design interview design and databases46 De-tailed methodological analyses now exist on issues such as dealing withextreme cases how to take account of randomness and historical contin-gency in small-n designs coping with multiple necessary conditions and soon Although today one rarely sees today such cutting-edge methods actuallyapplied in empirical research because there is no way to make either themethodological approach or the empirical payoff transparent Active citationwould fundamentally alter this dynamic

sampling secondary and primary material one scholar has argued ldquoRather than relying on some kind ofsample my claims rest on an informed reading of the secondary literaturerdquo Yet the same scholar alsoconcludes ldquoThe existing historiography leans toward my viewrdquo Rosato ldquoTheory and Evidence in EuropeUnitedrdquo 811 This begs the basic methodological issue How can we know what direction the historicalliterature leans unless we have sampled it fairly

44 Robert Siebers ldquoData Inconsistencies in Abstracts of Articles in Clinical Chemistryrdquo Clinical Chem-istry 47 no1 (January 2001)149

45 For a debate over cherry-picking and evidence on both sides see Moravcsik ldquoTransparencyrdquo779ndash80 Rosato ldquoTheory and Evidence in Europe Unitedrdquo 812n31

46 See notes 6 13 42 Robert H Bates et al Analytic Narratives (Princeton NJ Princeton UniversityPress 1998)

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


MORE ROBUST DEBATE PUBLICATION AND THEORETICAL DEVELOPMENT

Greater transparency introduces more vigorous replication and criticism ofresearch findings higher quality academic debate more publication oppor-tunities and new theoretical insights into world politics

Today scholarly debate in qualitative IR over data analytic and pro-cedural issues remains rare More common are abstract debates over con-tending grand theoretical or conceptual positions thatmdashdue to the lack oftransparent data analysis and proceduremdashare often difficult to adjudicateempirically Whereas in the quantitative tradition it is now commonplacein early graduate statistics training to replicate and extend recently pub-lished quantitative scholarship using easily available datasets often with aneye to publication such replication remains rare in the qualitative traditioneven though myriad opportunities exist To be sure islands of high-qualityqualitative empirical debate exist They are however instructive exceptionsbecause they arise in areas where political scientists can draw on disciplinesor specific area and policy studies communities more committed to dataanalytic and production transparency Discussions in H-DIPLO which bringtogether historians and political scientists or in international law which linkpolitical scientists with legal academics are exemplary47 AC and other in-struments of research transparency promise to help focus attention on suchintense high-quality empirical debates as methodological models and tomultiply them across IR

The ultimate payoff for this effort is not better footnotes active orotherwise It lies in the main text of articles we care about transparencybecause it permits us to make new arguments about politics and betterreassess the relative real-world importance of old ones As transparency en-courages scholars to replicate critique and debate published work moreintensively this is more likely to occur Now is an optimal moment toencourage such empirical debates As we have seen above new inno-vations in qualitative methodology are at hand Moreover opportunitiesto publish replicationsmdasha career opportunity qualitative scholars particu-larly younger ones do not want to missmdash are increasing rapidly48 Mostof all it is an optimal moment because qualitative replication and cri-tique are proving to be devastatingly effective in advancing our theoreticalknowledge of world politics Qualitative replications and extensions haverecently called into question widely accepted theories of audience costs

47 H-DIPLO H-Net Network on Diplomatic History and International Affairs httpsnetworksh-netorgh-diplo (accessed 1 July 2014)

48 APSA and various publishers are considering proposals for replication journals or sections ofjournals on replication Qualitative replication is increasingly taught at the graduate level Collier ldquoUn-derstanding Process Tracingrdquo

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik

by James Fearon nuclear crisis diplomacy by Matthew Kroenig and ToddSechser and credible commitment by Robert North and Barry Weingast49

In a parallel manner transparency may also enhance the theoreticaldiversity and substantive scope of IR Critical here is not simply that qualita-tive replication offers a new empirical tool Relative to statistical or experi-mental analysis qualitative data and analysis are also extremely flexible andrelatively inexpensive Textual information is readily available on almost ev-ery aspect of world politics in recorded history Qualitative work also playsa unique role in generating new causal conjectures and theories and un-covering precise causal mechanisms Moreover it may also be particularlywell suited to cope with rare complex and contingent events50 Methodol-ogist Gary King argues that the limitations imposed by causal endogeneitysocial complexity and extreme cases inherently relegate issues such as theimpact on international conflict and cooperation of democracy ethnic di-versity interdependence international institutions and nuclear weapons tothe category of ldquobig social science problems we canrsquot answerrdquo using statis-tical experimental and formal deductive theories51 If political science is toremain diversemdashand in particular is to avoid increasingly looking under sta-tistical and experimental lamppostsmdashyet at the same time reliable rigorousand respected throughout the social sciences high standards of qualitativetransparency are imperative Given how little has been done rigorous qual-itative replication has the potential to be one of the largest growth areasand revolutionary forces in political science and in IR specifically over thecoming decade The result will likely be to invigorate the field theoreticallyand strengthen the role of qualitative scholars within it

49 Marc Trachtenberg ldquoAudience Costs in 1954rdquo 2014httpwwwh-netorgsimdiploISSFPDFISSF-Forum-1pdf Jack Snyder and Erica D Borghard ldquoThe Costof Empty Threats A Penny Not a Poundrdquo American Political Science Review 105 no 3 (August 2011)437ndash56 Francis Gavin et al ldquoForum on lsquoWhat We Talk About When We Talk About Nuclear WeaponsrsquordquoH-DiploISSF Forum (15 June 2014) available at httpissforumorgISSFPDFISSF-Forum-2pdfTommaso Pavone ldquoConstitutional Constraints Organizational Preferences and Qualitative Methods AReplication and Reassessment of North and Weingastrdquo (unpublished manuscript Department of PoliticsPrinceton University May 2014)

50 Harry Eckstein ldquoCase Study and Theory in Political Sciencerdquo in Handbook of Political Science edFred Greenstein and Nelson Polsby (Reading MA Addison-Wesley 1975) 79ndash117 Jon Elster ldquoA Plea forMechanismsrdquo in Social Mechanisms ed P Hedstrom and R Swedborg (Cambridge Cambridge UniversityPress 1998) 1ndash31

51 Gary King ldquoA Hard Unsolved Problem Post-Treatment Bias in Big Social Science Questionsrdquopresentation at Hard Problems in Social Science Symposium Institute for Quantitative Social ScienceHarvard University 10 April 2010) httpGKingHarvardedu Bent Flyvbjerg ldquoCase Studyrdquo in The SAGEHandbook of Qualitative Research 4th ed ed Norman Denzin and Yvonna Lincoln (Los Angeles Sage2011) 301ndash16 Peter J Katzenstein and Rudra Sil ldquoAnalytic Eclecticism in the Study of World PoliticsReconfiguring Problems and Mechanisms across Research Traditionsrdquo Perspectives on Politics 8 no 2(2010) 411ndash31

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


MORE INTERDISCIPLINARY INTERACTION AND POLICY RELEVANCE

Broader effects of greater research transparency include more intense inter-disciplinary interaction and greater policy relevance

Greater research transparency encourages greater cooperation withneighboring disciplines particularly those that employ best practices in qual-itative methodology such as history legal academia education and sociol-ogy Law reviews for example normally require that an author provideextensive quotations from cited sources and an annotation interpreting itwhen citing jurisprudence they must even cite all relevant cases for andagainst any interpretation Archival historians are traditionally responsiblenot simply for quoting specific documents accurately but also for report-ing a balanced sample of relevant contextual material in the broader docu-ment box archive or other location from which it came Scholars in otherfields often hold political science research in low regard because it fails tomeet these professional standards of transparency AC and other instrumentswould permit political scientists to cooperate across disciplines on an equalbasis52

Transparency would also encourage more policy-relevant IR researchScholars should be encouraged to speak to public affairs and we haveseen that qualitative work in IR tends to be considered more policy relevantthan quantitative work Yet two barriers stand in the way much qualitativeempirical work attains a level of regional or functional expertise requiredto be credible in the policy world and transparency and replicability ofpolicy and journalistic work are now often out ahead of political scienceReaders would be surprised to encounter electronic journalism or reportsfrom a government international organization NGO or think-tank withouthyperlinks to underlying data sources and analysis US government intel-ligence officials and decision makers burned several times in recent yearswith vague or manipulated evidence increasingly structure even confidentialassessments with footnotes that drill down to (properly sanitized) primarysources It is impracticalmdasheven unprincipledmdashfor scholars to seek to influ-ence public policy debates with scholarly conclusions that are empiricallyless transparent and methodologically less reliable than what policymakersand journalists already produce If qualitative political scientists wish to re-tain their respect they must follow suit At the same time public authoritiesand research funders are seeking guidance on how to generate richer andmore rigorous qualitative results There are real opportunities for collabora-tion between scholars and public affairs practitioners not just on substantiveresearch but on qualitative methodology but a precondition is enhancedresearch transparency

52 Francis Gavin ldquoPanel Commentsrdquo (54th Annual Convention of the International Studies Associa-tion San Francisco CA 5 April 2013)

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik

Are the Costs of Transparency Too High

Despite these advantages some may worry that AC (or any other trans-parency instrument) consumes more time energy and resources than it isworth At first glance this fear seems plausible Yet the experiences of otherresearch communities and disciplines and of those (including myself andothers in this symposium) who have begun employing AC in their researchsuggest that the fear is exaggerated At least six considerations suggest thatthe costs are lower than skeptics fear

First transparency publicizes what many qualitative scholars alreadydo Many scholars already conduct careful field research assemble archivaldocuments in multiple languages master voluminous historical literaturesreconstruct complex decisions and increasingly have been trained in qual-itative causal analysis AC simply provides a formal structure in which suchscholars can demonstratemdashand others can properly recognizemdashprevailinghigh standards

Second only contestable empirical claims and exceptionally issuesof production transparency need to be actively cited Contestable empiri-cal claims as defined by the author are often a small subset of the totalNo obligation exists to actively cite introductory and background materialliterature reviews conventional methodological issues extensions or im-portantly uncontroversial empirical claims

Third the length of quotations and annotations is flexible and discre-tionary AC obliges scholars to provide a modest quotation and some anno-tation clarifying analytical ambiguity Yet implementation remains largely atthe authorsrsquo discretion the suggested length of sources at fifty to one hun-dred words is only a guideline If the evidentiary context is clear the authormay simply enter a single line Similarly if the link between to the underlyingclaim is obvious or trivial a one-sentence annotation may suffice Moreoverall requirements defer to intellectual property human subject and logisticalconstraints An article based on confidential or non-transcribed interviews(eg with contemporary Chinese military officers) probably cannot be asopenly sourced as one based on public documents (eg about nineteenth-century British imperialism) if necessary verbatim text may be omitted orreplaced by a summary Diverse research communities will surely evolvevaried expectations about reasonable and feasible transparency appropriateto specific research conditions

Fourth advance preparation and modern technology lighten the loadJack Snyder Elizabeth Saunders and I recently contributed pilot projectsfor the QDR by retrofitting previous book chapters and articles The burdenwas misleadingly heavy because we retrofitted research conducted years(even decades) ago with obsolete technology Most scholars today work(or should work) far more efficiently researching from beginning to endin digital media employing readable e-copies of secondary sources like

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014


journals and books online or scanned copies of primary sources managinginformation with programs such as Endnote and Scrivener photographingarchival documents with small digital cameras taping interviews taking andscanning notes electronically and managing sources using spreadsheet anddatabase programs Such data can be searched accessed edited and insertedwithin seconds Even the financial cost is modest but a silver lining is thatqualitative scholars can now apply for outside funding to defray the researchcosts of achieving transparency just as their quantitative and experimentalcolleagues do

Fifth AC imposes little format change on writers editors reviewersand publishers AC leaves scholarly formats of main text and citations essen-tially unchanged Only the appendix is new and it lies outside word limitsDedicated software is available Since actively cited articles would appearin electronically published versions increases in publication costs would bemarginal

Sixth AC imposes a lighter logistical burden than that borne by pastpolitical scientists contemporary scholars in other academic disciplines andnatural scientists AC is not a radical and untested scheme Rather it closelyresembles the system of discursive footnotes common a generation agoacross the social sciences and humanities including political sciencemdashalbeitin a technologically turbo-charged form It is a less demanding version ofcurrent and long-established practices in legal academia history and otherneighboring fields It is a functional parallel to the derivations data androbustness checks formal and quantitative political scientists must submit tojournals as appendices and to mandatory materials that customarily appearwith articles in natural science Is there some reason why political scientistscannot meet the same challenge

CONCLUSION

Research transparency is a basic social scientific principle shared by researchcommunities of all types It is a precondition for rich and rigorous qualitativescholarship a spur to productive debate and extension of existing researchan instrument to promote theoretical diversity a way to encourage policyrelevance an instrument to forge interdisciplinary links a source of newpublication outlets and less expensive research opportunities for qualitativescholars and a means to enhance the credibility and legitimacy of qualitativeresearch and those who conduct it both inside and outside academia Otherdisciplines have long since taken the lead in assuring data analytic andproduction transparency for qualitative research Given the benefits the costsare remarkably low

Transparency is however more than an academic ideal It is a straight-forward adaptation to modern life Today transparency envelops us Not

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik

just academics and scientists but journalists government officials civil soci-ety groups corporations and individual citizens are embracing technologiesmuch like active citation Whether we peruse the morning paper online as-sess our legal rights book airplane seats visit a museum read a governmentintelligence briefing or chat with friends we would be shocked to findourselves without the blue links to vital underlying information and analysisPolitical science has fallen behind the curve but it is responding The ques-tion is no longer whether qualitative political scientists will move in the samedirection but when and how With active citation and other instruments athand transparency is now just one click away

ACKNOWLEDGMENTS

This research was supported by the Woodrow Wilson School of Public andInternational Affairs Princeton University and the National Science Founda-tion Grant SES-0838716 The author thanks Nathaniel Bell Andrew BennettColin Elman Van Harvey Diana Kapiszewski Arthur ldquoSkiprdquo Lupia JohnOwen Elizabeth Saunders Jack Snyder Allison Stanger Michael Tierneyand several anonymous reviewers for their help and comments

SUPPLEMENTAL MATERIAL

An appendix containing the Active Citations mentioned in this article can beaccessed at httpdxdoiorg101080096364122014970846

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

Security Studies 23663ndash688 2014Copyright copy Taylor amp Francis Group LLCISSN 0963-6412 print 1556-1852 onlineDOI 101080096364122014970846

Trust but Verify The Transparency Revolutionand Qualitative International Relations

ANDREW MORAVCSIK

ldquoДоверяй но проверяйrdquo (ldquoTrust but verifyrdquo)ndashOld Russian proverb often used by President Ronald Reagan

Qualitative analysis is the most important empirical method in the field ofinternational relations (IR) More than 70 percent of all IR scholars conductprimarily qualitative research (including narrative case studies traditionalhistory small-n comparison counterfactual analysis process-tracing analyticnarrative ethnography and thick description discourse analysis) comparedto only 20 percent whose work is primarily quantitative Total use is evenmore pervasive with more than 85 percent of IR scholars conducting somequalitative analysis1 Qualitative analysis is also unmatched in its flexibilityand applicability a textual record exists for almost every major internationalevent in modern world history Qualitative research also delivers impressiveexplanatory insight rigor and reliability Of the twenty scholars judged bytheir colleagues to have ldquoproduced the best work in the field of IR in thepast 20 yearsrdquo seventeen conduct almost exclusively qualitative research2

Moreover controlled studies reveal that experts on world affairs whose anal-yses are informed by more eclectic theory and the myriad ldquosituational factsof each historical episoderdquo (a mode in which qualitative analysis excels)

Andrew Moravcsik is professor of politics and public affairs at Princeton UniversityColor versions of one or more figures in the article can be found online at www

tandfonlinecomFSST1 Daniel Maliniak Susan Peterson and Daniel J Tierney TRIP around the World Teaching Research

and Policy Views of International Relations Faculty in 20 Countries (Williamsburg Virginia Institute forthe Theory and Practice of International Relations College of William and Mary 2012) Charts 28ndash30available at httpwwwwmeduofficesitpir_documentstriptrip_around_the_world_2011pdf SeeActive Citation 1 in online supplemental material Instructions for accessing this material can be found atthe end of this article

2 Maliniak et al Chart 42 These seventeen scholars are Michael Barnett Barry Buzan MarthaFinnemore Samuel Huntington John Ikenberry Robert Jervis Peter Katzenstein Robert KeohaneStephen Krasner David Lake John Mearsheimer Joseph Nye John Ruggie Jack Snyder Stephen WaltKenneth Waltz and Alexander Wendt

663

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

664 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

666 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Penguin 2009) 65 100 267 273

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

668 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

670 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

664 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

666 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Penguin 2009) 65 100 267 273

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

668 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

670 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

666 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Penguin 2009) 65 100 267 273

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

668 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

670 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

666 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Penguin 2009) 65 100 267 273

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

668 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

670 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Penguin 2009) 65 100 267 273

Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

668 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

670 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

668 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

670 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

670 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

670 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

672 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

674 A Moravcsik




DATA ARCHIVING





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014




DATABASES





Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

676 A Moravcsik









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

678 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

680 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014









Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

682 A Moravcsik








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014








Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

684 A Moravcsik






Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

686 A Moravcsik







Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014





CONCLUSION



Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

688 A Moravcsik


ACKNOWLEDGMENTS




Dow

nloa

ded

by [

Prin

ceto

n U

nive

rsity

] at

13

21 0

6 D

ecem

ber

2014

Relations Publisher: Routledge Revolution and Qualitative ...amoravcs/library/Final Copy.pdf ·...

Documents

Transcript of Relations Publisher: Routledge Revolution and Qualitative ...amoravcs/library/Final Copy.pdf ·...