I N S T I T U T E

Methodology

Copyright © University of Gothenburg, V-Dem Institute,

University of Notre Dame, Kellogg Institute. All rights reserved.

Version 5 - Dec 2015

1

Suggestedcitation:Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,FridaAndersson,KyleL.Marquardt,ValeriyaMechkova,FarhadMiri,DanielPemstein,JosefinePernes,NataliaStepanova,EitanTzelgov,andYi-tingWang.2015.“V-DemMethodologyv5.”VarietiesofDemocracy(V-Dem)Project.

Authors Collaborators• MichaelCoppedge–U.ofNotreDame • DavidAltman–PontificiaU.CatólicadeChile• JohnGerring–BostonUniversity • MichaelBernhard–UniversityofFlorida• StaffanI.Lindberg–U.ofGothenburg • M.StevenFish–UCBerkeley• Svend-ErikSkaaning–AarhusUniversity • AdamGlynn–EmoryUniversity• JanTeorell–LundUniversity • AllenHicken–UniversityofMichigan• FridaAndersson–U.ofGothenburg• KyleL.Marquardt–U.ofGothenburg• ValeriyaMechkova–U.ofGothenburg• FarhadMiri–Uof.Gothenburg• DanielPemstein–NorthDakotaStateU.• JosefinePernes–U.ofGothenburg• NataliaStepanova–U.ofGothenburg

• CarlHenrikKnutsen–UniversityofOslo• KellyMcMann–CaseWesternReserve

PamelaPaxton–U.ofTexas• JeffreyStaton–EmoryUniversity• BrigitteZimmerman–U.ofNorthCarolina

• EitanTzelgov–U.ofEastAnglia&U.ofGothenburg

• Yi-tingWang–NationalChengKungU& U.ofGothenburg

2

TableofContents

1. CONCEPTUALSCHEME 4

PRINCIPLES–MEASUREDBYV-DEM’SDEMOCRACYINDICES 4COMPONENTS 10INDICATORS 11SUMMARY 12

2. DATACOLLECTION 14

HISTORYOFPOLITIES 14CODINGTYPES 17EXPERTRECRUITMENT 18EXPERTCODINGPROCESS 21BRIDGE-ANDLATERALCODING 25PHASES 26

3. MEASUREMENT 27

THEQUESTIONNAIRE 27IDENTIFYING,CORRECTING,ANDQUANTIFYINGMEASUREMENTERROR 28

MeasurementModels 29CorrectingErrors 35

VERSIONSOFC-VARIABLES 37ADDITIONALPOSSIBILITIESFORIDENTIFYINGSOURCESOFMEASUREMENTERRORINTHEFUTURE 39

REFERENCES 42

APPENDIXA:V-DEMINDICES,COMPONENTS,ANDINDICATORS 46

3

This document outlines the methodological considerations, choices, and procedures

guidingthedevelopmentoftheVarietiesofDemocracy (V-Dem)project.Part Isetsforth

theconceptual scheme.Part IIdiscusses theprocessofdatacollection.Part IIIdescribes

themeasurementmodelalongwitheffortstoidentifyandcorrecterrors.

Wecontinuallyreviewourmethodology—andoccasionallyadjustit—withthegoal

ofimprovingthequalityofV-Demindicatorsandindices.Wethereforeissueanewversion

ofthisdocumentwitheachnewversionofthedataset.

Additional project documents complement this one.V-Dem Codebook includes a

comprehensive list of indicators, response-categories, sources, and brief information

regarding theconstructionof indices.V-DemCountryCodingUnits explainshowcountry

unitsaredefinedand listseachcountry included inthedataset,withnotespertainingto

the years covered and special circumstances that may apply. V-Dem: Comparisons and

ContrastswithOtherMeasurementProjectssurveysthefieldofdemocracyindicatorsand

situates the V-Dem project in relation to these efforts. V-Dem Organization and

Management introduces the project team, the web site, outreach to the international

community,funding,progresstodate,andsustainability.

Versioning of the current document, V-Dem Codebook, V-Dem Country Coding

Units and V-Dem Organization & Management documents are synchronized with the

release of each new dataset. Versioning of other documents is not synchronized.

(Currently,weareatv5.)

Several configurationsof theV-Demdatasetareavailable, including country-date

andcoder-leveldatasets.Foradditionaldocumentationandguidance,users should refer

totheOtherProjectDocumentationfilethatisappendedtoeachdatadownload.

IntheV-DemWorkingPaperSeries,userscanfindamoretechnicaldiscussionof

the measurement model we use to aggregate coder-level data to point estimates for

country-years (Pemstein et al. 2015, WP #21). Working Paper #6 introduces the

democracyindices.WorkingPaper#24(forthcoming2016)detailstheElectoralDemocracy

index.WorkingPaper#25 (also forthcoming)describesother indices.Additionalworking

papers provide in-depth treatments of more specialized indices such as the Female

Empowerment Index (#18), the Core Civil Society Index (#13), the Party System

Institutionalization Index (#26, forthcoming), the Corruption Index (#23), and ordinal

4

versions of the V-Dem indices (#20). The V-Dem Working Paper Series is available for

downloadontheV-Demwebsite(v-dem.net).

V-Dem isamassive,global collaborativeeffort.Anup-to-date listingofourmany

collaborators,withoutwhom this projectwouldnotbepossible, is also availableon the

web site. Collaborators include Program Managers, Regional Managers, International

AdvisoryBoardmembers,theV-DemInstitutestaff(Director,Program-,Operations-,Data

Processing and Data Managers, Assistant Researchers, and Post-Doctoral Fellows and

Associate Researchers), Research Assistants, and Country Coordinators. We are also

especially indebted to some 2,500 Country Experts, whose identities must remain

anonymousforethicalreasons.

The website serves as the repository for other information about the project,

including Country and Thematic Reports, Briefing Papers, publications, grant and

fellowshipopportunities,andthedataitself.Dataforall173countriesincludedinthefirst

publicrelease(V-DemDatasetv5)isalsoavailableforexplorationwithonlineanalysistools

(countryandvariablegraphs,motioncharts,and–soon–globalmaps).

1. ConceptualScheme

Anymeasurementschemerestsonconcepts. Inthissection,wesetforththeconceptual

schemethatinformstheV-Demproject–beginningwith“democracy”andproceedingto

thepropertiesandsub-propertiesofthatfar-flungconcept.Bywayofconclusion,weissue

severalclarificationsandcaveatsconcerningtheconceptualscheme.V-Dem:Comparisons

andContrastsprovidesamoredetaileddiscussion,butwerecaptheessentialpointshere.

Principles–MeasuredbyV-Dem’sDemocracyIndices

Thereisnoconsensusonwhatdemocracywrit-largemeansbeyondavaguenotionofrule

bythepeople.Politicaltheoristshaveemphasizedthispointforsometime,andempiricists

woulddowelltotakethelessontoheart(Gallie1956;Held2006;Shapiro2003:10–34).At

thesametime,interpretationsofdemocracydonothaveanunlimitedscope.

A thorough search of the literature on this protean concept reveals seven key

principles that inform much of our thinking about democracy: electoral, liberal,

5

majoritarian, consensual, participatory, deliberative, and egalitarian. Each of these

principlesrepresentsadifferentwayofunderstanding“rulebythepeople.”Theheartof

thedifferences between theseprinciples is in the fact that alternate schools of thought

prioritize different democratic values. Thus, while no single principle embodies all the

meanings of democracy, these seven principles, taken together, offer a fairly

comprehensiveaccountingoftheconceptasemployedtoday.1

The V-Dem project has set out to measure these principles, and the core values

whichunderliethem.Wesummarizetheprinciplesbelow.

• Theelectoralprincipleofdemocracyembodiesthecorevalueofmaking

rulers responsive to citizens through periodic elections, as captured by

Dahl’s (1971, 1989) conceptualization of “polyarchy.” Our measure for

electoral democracy is called the “V-Dem Electoral Democracy Index.”

We consider this measure fundamental to all other measures of

democracy:wewouldnotcallaregimewithoutelections“democratic”in

anysense.

• The liberal principle of democracy embodies the intrinsic value of

protecting individualandminority rightsagainstapotential “tyrannyof

the majority” and state repression. This principle is achieved through

constitutionally-protectedcivil liberties,strongruleof law,andeffective

checksandbalancesthatlimittheuseofexecutivepower.

• Theparticipatoryprincipleembodiesthevaluesofdirectruleandactive

participation by citizens in all political processes.While participation in

elections counts toward this principle, it also emphasizes nonelectoral

forms of political participation, such as civil society organizations and

other forms of both nonelectoral and electoral mechanisms of direct

1Thisconsensusonlyholdsinsofarasmostscholarswouldagreethatsomepermutationoraggregationoftheseprinciplesunderlieconceptionsofdemocracy.Forexample,scholarscanreasonablyarguethatthelistcouldconsistofseven,six,orfiveprinciples;our“principles”maybe“properties”or“dimensions;”and“majoritarian”and“consensual”areactuallyoppositepolesofasingledimension.Asaresult,weintendforthisdiscussiontoassureconsumersofthedataofthecomprehensivenatureofourinventoryofcorevaluesofdemocracy:namely,thatitincludesalmostalltheattributesthatanyuserwouldwanttohavemeasured.

6

democracy.

• Thedeliberativeprincipleenshrinesthecorevaluethatpoliticaldecisions

in pursuit of the public good should be informed by a process

characterizedbyrespectfulandreason-baseddialogueatalllevels,rather

thanbyemotionalappeals,solidaryattachments,parochial interests,or

coercion.

• Theegalitarianprincipleholdsthatmaterialand immaterial inequalities

inhibit the actual use of formal political (electoral) rights and liberties.

Ideally,allgroupsshouldenjoyequaldejureanddefactocapabilitiesto

participate;toserve inpositionsofpoliticalpower;toput issuesonthe

agenda; and to influence policymaking. Following the literature in this

tradition, gross inequalities of health, education, or income are

understood to inhibit the exercise of political power and the de facto

enjoymentofpoliticalrights.

Theconceptualschemepresentedabovedoesnotcaptureallthetheoreticaldistinctions

atplayinthecomplexconceptofdemocracy.Wehavechosentofocusonthecorevalues

and institutions that the other principles emphasize in their critique of the electoral

conceptionas a stand-alone system.Eachof theseprinciples is logicallydistinct and—at

least for some theorists—independently valuable.Moreover,we suspect that there is a

considerable divergence in the realizationof theproperties associatedwith these seven

principles among the world’s polities. Some countries will be particularly strong on

electoraldemocracy;otherswillbestrongontheegalitarianproperty,andsoforth.

AggregationProcedures

Atthispoint,V-Demoffersseparateindicesoffivevarietiesofdemocracy:electoral,

liberal,participatory,deliberative,andegalitarian.Weanticipateprovidingindicesforthe

remainingtwoprinciples–majoritarianandconsensual–inthenearfuture.2V-Dem

2Themajoritarianprincipleofdemocracy(reflectingthebeliefthatamajorityofthepeoplemustbecapacitatedtoruleandimplementtheirwillintermsofpolicy);andtheconsensualprincipleofdemocracy(emphasizingthatamajoritymustnotdisregardpoliticalminoritiesandthatthereisaninherentvaluein

7

CodebookcontainstheaggregationrulesforeachindexandseveralV-DemWorking

Papers(presentandforthcoming)layoutjustificationsforthechoicesmadeineach

aggregationscheme.Thehigh-levelindices,measuringcoreprinciplesofdemocracy,are

referredtoasdemocracyindices.

Sartoriheldthateverydefiningattributeisnecessaryfortheconcept.Thislogic

requiresmultiplyingtheattributessothateachofthemaffectstheindexonlytothe

degreethattheothersarepresent.Familyresemblancedefinitionsallowsubstitutability:a

highvalueononeattributecancompensateforalowvalueonanother.Thislogic

correspondstoanadditiveaggregationformula.Therearesoundjustificationsfortreating

alloftheseattributesasnecessary,ormutuallyreinforcing.Forexample,ifopposition

candidatesarenotallowedtorunforelectionortheelectionsarefraudulent,thefactthat

alladultshavevotingrightsdoesnotmattermuchforthelevelofelectoraldemocracy.But

therearealsogoodreasonstoregardtheseattributesassubstitutableaswell.Wherethe

suffrageisrestricted,thesituationislessundemocraticifthedisenfranchisedarestillfree

toparticipateinassociations,tostrikeandprotest,andtoaccessindependentmedia

(Switzerlandbefore1971)thaniftheylacktheseopportunities(ItalyunderMussolini).

Evenwheretheexecutiveisnotelected,citizenscanfeelthattheyliveinafairly

democraticenvironmentaslongastheyarefreetoorganizeandexpressthemselves,asin

Liechtensteinbefore2003.

Becausewebelieveboththenecessaryconditionsandfamilyresemblancelogicsare

validforconceptsofdemocracy,ouraggregationformulasincludeboth;becausewehave

nostrongreasontoprefertheadditivetermstothemultiplicativeterm,wegivethem

equalweight.TheElectoralDemocracyindexistherefore:

ElectoralDemocracy(polyarchy)=.5*(Familyresemblance)+.5*(Necessaryconditions)=.5*(.2*Sumofelectedexecutive,etc.)+.5*(Productofelectedexecutive,etc.)=.1*electedexecutive+.1*cleanelections+.1*freedomofexpression+.1*freedomofassociation+.1*suffrage+.5*electedexecutive*cleanelections*freedomofexpression*freedomofassociation*suffrage.

therepresentationofgroupswithdivergentinterestsandview).

8

Thesumoftheweightsoftheadditivetermsequalstheweightoftheinteraction

term.Eachadditivetermhasthesameweightbecausethereisnoobvious,uncontested

reasontopreferoneovertheothers.3Inanyevent,becausemostofthevariablesare

stronglycorrelated,differentaggregationformulasyieldverysimilarindexvalues.The

officialformulapresentedherecorrelatesat.94to.99withapurelymultiplicativeformula,

apurelyadditiveformula,onethatweightstheadditivetermstwiceasmuchasthe

multiplicativeterm,onethatweightsthemultiplicativetermtwiceasmuchastheadditive

terms,andonethatweightssuffragesixtimesasmuchastheotheradditiveterms.The

maindifferenceacrosstheseformulasisintheirmeanvalues,withsomebeingcloserto

oneandothers(i.e.themoremultiplicativeformulas)beingclosertozero.

TheElectoralDemocracyIndexalsoservesasthefoundationfortheotherfour

indices.Therecanbenodemocracywithoutelectionsbut,followingthecanonineachof

thetraditionsthatarguesthatelectoraldemocracyisinsufficientforatruerealizationof

“rulebythepeople,”thereismoretodemocracythanjustelections.Wetherefore

combinethescoresforourElectoralDemocracyIndex(v2x_polyarchy)withthescoresfor

thecomponentsmeasuringdeliberation,equalitarianism,participation,andliberal

constitutionalism,respectively.Thisisnotaneasytask.Imaginetwocomponents,

P=PolyarchyandHPC=HighPrincipleComponent(liberal,egalitarian,participatory,or

deliberative),4thatwewanttoaggregateintomoregeneraldemocracyindices,whichwe

willcallDI(DeliberativeDemocracyIndex,EgalitarianDemocracyIndex,andsoon).For

convenience,bothPandHPCarescaledtoacontinuous0-1interval.Basedonextensive

deliberationsamongtheauthorsandothermembersoftheV-Demresearchgroup,we

tentativelyarrivedatthefollowingaggregationformula:

DI=.25*P1.6+.25*HPC+.5*P1.6*HPC

TheunderlyingrationaleforthisformulaforallfourDIsisthesameasthatforthe

ElectoralDemocracyIndex:equalweightingoftheadditivetermsandthemultiplicative

3OnecouldarguethatthesuffragedeservesgreaterweightbecauseitliesonadifferentdimensionthantheothersandisthekeycomponentofoneofDahl’stwodimensionsofpolyarchy(Dahl1971;Coppedgeetal.2008).However,ourformulaallowsarestrictedsuffragetolowertheElectoralDemocracyIndexconsiderablybecauseitdiscountsalltheothervariablesinthemultiplicativeterm.

4TheHPCsareindicesbasedontheaggregationofalargenumberofindicators(liberal=23,egalitarian=8,participatory=21,deliberative=5).

9

terminordertorespectboththeSartoriannecessaryconditionslogicandafamily

resemblancelogic.Forexample,thedegreeofdeliberationstillmattersfordeliberative

democracyevenwhenthereisnoelectoraldemocracy,andelectoraldemocracystill

mattersevenwhenthereisnodeliberation;butthehighestlevelofdeliberative

democracycanbeattainedonlywhenthereisahighlevelofbothelectoraldemocracyand

deliberation.

Themoreacountryapproximatespolyarchy,themoreitscombinedDIscore

shouldreflecttheuniquecomponent.Thisperspectiveisacontinuousversionof

theoreticalargumentspresentedintheliteraturesayingthatpolyarchyorelectoral

democracyconditionsshouldbesatisfiedtoareasonableextentbeforetheother

democracycomponentgreatlycontributestothehighlevelindexvalues.Atthesametime,

itreflectstheviewintheliteraturethat,whenacertainlevelofpolyarchyisreached,what

mattersintermsof,say,participatorydemocracyishowmuchoftheparticipatory

propertyisrealized.Thisargumentalsoresemblesthewidespreadperspectiveinthe

qualityofdemocracyliteratureemphasizingthatthefulfillmentofsomebaseline

democracycriteriaisnecessarybeforeitmakessensetoassessthequalityofdemocracy.5

Giventhisbodyofliterature,itbecomesnecessarytospecifytherateatwhicha

componentshouldinfluenceaDIscore.Wedosobyraisingthevalueofacomponentby

1.6.Weidentifythisnumericvaluebydefiningananchorpoint:whenacountryhasa

polyarchyscoreof.5(inpractice,thisisathresholdontheElectoralDemocracyIndex

beyondwhichcountriestendtobeconsideredelectoraldemocraciesinaminimalsense)

anditsHPCisatitsmaximum(1),thehighlevelindexscoreshouldbe.5.6

Takentogether,theseindicesofferafairlycomprehensiveaccountingof“varieties

ofdemocracy.”Thefive(soontobeseven)democracyindicesconstituteafirststepin

disaggregatingtheconceptofdemocracy.Thenextstepisthecomponents.

5Foranoverview,seeMunck(2016).6Definetheexponentasp.SettingPolyarchy=.5,HPC=1,andHLI=.5,andsolvingforDI=.25*Polyarchy^p+.25*HPC+.5*Polyarchy^p*HPC,p=log(base0.5)of.25/.75≈1.6.

10

Components

Themain democracy components, already included in the discussion above, specify the

distinct properties associatedwith the principles. TheV-DemElectoralDemocracy Index

consists of five components (each of these components being indices themselves built

fromanumberofindicators)thattogethercaptureDahl’sseveninstitutionsofpolyarchy:

freedom of association, suffrage, clean elections, elected executive, and freedom of

expression.Thecomponent indicesmeasuringthe liberal,deliberative,participatory,and

egalitarianpropertiesof democracy (majoritarian and consensualwill be released in the

near future) follow the principles of democracy described in the previous section – but

without the core, unifying element of electoral democracy. They capture only what is

unique for eachof the principles. As such, these components aremutually exclusive, or

orthogonaltoeachother.

These main democracy components typically have several sub-components. For

example, the liberal democracy component consists of three sub-components, each

capturedwithitsownindex:theEqualitybeforethelawandindividuallibertyindex;the

Judicialconstraintsontheexecutiveindex;andtheLegislativeconstraintsontheexecutive

index.

InadditiontothecomponentandsubcomponentindicesthatarepartoftheV-Dem

democracy indicesconceptual scheme,membersof theV-Demteamhaveconstructeda

series of indices of lower-level concepts such as civil society, party system

institutionalization, corruption, andwomen’s political empowerment.We also list these

indicesintheappendix.Intotal,V-Demoffers39indicesofcomponents,subcomponents,

and related concepts. TheV-Demdataset includes all of these indices. PublishedV-Dem

working papers already detail many of these indices (e.g. papers #6, #13, #17-20).

Additionalworkingpaperswillprovidefurtherdetailsonotherindices.

Weusetwotechniqueswhenaggregatingintodemocracyindices,components,and

subcomponents, as well as related concepts’ indices. For the first step, going from

indicatorsto(sub-)components,weaggregatethelatentfactorscoresfrommeasurement

model (MM) output. More specifically, we use relevant theoretical distinctions in the

literature togroup interval-levelMMoutput into setsof variables that sharea common

underlying concept.We then randomly select 100 draws from each variable’s posterior

11

distribution(seedetailsunder“MeasurementModels”below),anduseaunidimensional

Bayesian factor analysis (BFA) to measure this latent concept sequentially for each

randomly-selected draw in each grouping of variables. We then combine the posterior

distributions of the latent factor scores in each variable group to yield the latent factor

scores.Inallanalysesthevariablesgenerallyloadhighlyontheunderlyingfactor.

For the next level in the hierarchy –another subcomponent, a component, or a

democracyindexdependingonthecomplexityoftheconceptualstructure(seeAppendix

A)–we take the latent factor scores from the separateBFAsanduse in combination in

constructingthe“HigherLevelIndices”(HLIs).HLIsarethuscompositemeasuresthatallow

thestructureoftheunderlyingdatatopromulgatethroughthehierarchyinthesameway

astheBFAsdo–andcriticallycarryoverthefullinformationaboutuncertaintytothenext

level in order to avoid allowing the aggregation technique artificially increase the

estimated confidence – while being faithful to the theoretically informed aggregation

formula.FollowingtheformulaofeachHLI(seetheV-DemCodebook),wetakeaveragesor

products of each of the relevant BFA factor score posterior distributions, and then

calculatethepointestimates(means)andconfidenceintervalsacrosstheresultingmatrix

to generate the HLI estimates. For example, the liberal component of democracy index

comprises three elements: equality before the law and individual liberties, judicial

constraintsontheexecutive,andlegislativeconstraintsontheexecutive.Webelievethese

threeelementsaresubstitutiveandthereforetaketheaverageofthesethreeelementsto

constructtheliberalcomponentindex.FortheDIs,weusetheequationsdiscussedabove

toassignweightstothecombinations.

Indicators

Thefinalstepindisaggregationistheidentificationof indicators. In identifyingindicators

welookforfeaturesthat(a)arerelatedtoat leastonepropertyofdemocracy;(b)bring

thepoliticalprocessintocloseralignmentwiththecoremeaningofdemocracy(rulebythe

people);and(c)aremeasurableacrosspolitiesandtime.

Indicatorstaketheformofnominal(classifications,text,dates),ordinal(e.g.,Likert-

stylescales),orintervalscales.Somerefertodejureaspectsofapolity–rulesthatstatute

or constitutional law (including the unwritten constitution of states like the United

12

Kingdom) stipulate. Others refer tode facto aspects of a polity – theway things are in

practice.

Therearesome350uniquedemocracyindicatorsintheV-Demdataset.Welisteach

indicator, along with its response-type, in the V-Dem Codebook. We discuss coding

procedures in greater detail in the next section. The V-Dem dataset contains many

indicators that we do not include in the component and democracy indices discussed

above,thoughtheyarerelatedtodemocracy.Theirabsencereflectsthefactthatwehave

soughttomakethecomponent-anddemocracyindicesasorthogonalaspossibletoeach

other,andalsoasparsimoniousaspossible.Furthermore,wheneverwehavemeasuresof

boththedejureandthedefactosituationinastate,ourindicesbuildprimarilyonthede

facto indicators because we want the measures to portray the “real situation on the

ground”asfaraspossible.

Summary

Tosummarize,theV-Demconceptualschemerecognizesseverallevelsofaggregation:

●Coreconcept(1)●DemocracyIndices(5,soontobe7)

●DemocracyComponents(10)●Subcomponents,andrelatedconcepts(29)

●Indicators(≈350)

As anappendix to this document,weattacha tablewith a completehierarchyof

democracyindices,democracycomponentindices,democracysubcomponentindices,and

indicators,aswellasthehierarchyofrelatedconceptindices.

Several important clarifications apply to this taxonomy. First, our attempt to

operationalize democracy does not attempt to incorporate the causes of democracy

(exceptinsofarassomeattributesofourfar-flungconceptmightaffectotherattributes).

Regime-typesmay be affected by economic development (Epstein et al. 2006), colonial

experiences (Bernhard et al. 2004), or attitudes and political cultures (Almond& Verba

1963/1989; Hadenius & Teorell 2005; Welzel 2007). However, we do not regard these

attributesasconstitutiveofdemocracy.

Second,ourquesttoconceptualizeandmeasuredemocracyshouldnotbeconfused

13

with the quest to conceptualize and measure governance.7 Of course, there is overlap

betweenthesetwoconcepts,sincescholarsmayconsidermanyattributesofdemocracyto

beattributesofgoodgovernance.

Third,werecognizethatsome indicatorsandcomponents (listed intheCodebook)

are more important in guaranteeing a polity’s overall level of democracy than others,

thoughthepreciseweightingparametersdependuponone’smodelofdemocracy.

Fourth, aspects of different ideas of of democracy sometimes conflict with one

another.At the levelofprinciples, there isanobviousconflictbetweenmajoritarianand

consensualnorms,whichadoptcontraryperspectivesonmost institutional components.

Forexample,protecting individual libertiescan impose limitson thewillof themajority.

Likewise,strongcivilsocietyorganizationscanhavetheeffectofpressuringgovernmentto

restrict thecivil libertiesenjoyedbymarginalgroups (Isaacn.d.).Furthermore, thesame

institutionmaybedifferently viewedaccording todifferentprinciplesofdemocracy. For

example,thecommonpracticeofmandatoryvotingisclearlycontrarytotheliberalmodel

(where individual rights are sacrosanct and include the right not to vote), but the

participatorymodelsupportsthispractice,since ithasademonstratedeffect inboosting

turnoutwhereversanctionsaremorethannominal.

Such contradictions are implicit in democracy’s multidimensional character. No

wide-ranging empirical investigation can avoid conflicts among democracy’s diverse

attributes. However, with separate indicators representing these different facets of

democracyitshouldbepossibletoexaminepotentialtradeoffsempirically.

Fifth, our proposed set of democracy indices, components, and indicators, while

fairlycomprehensive,isbynomeansexhaustive.Theproteannatureofdemocracyresists

closure; there are always potentially new properties/components/indicators that, from

one perspective or another, may be associated with this essentially contested term.

Moreover, some conceptions of democracy are difficult to capture empirically; this

7SeeRose-Ackerman(1999)andThomas(2010).Inglehart&Welzel(2005)arguethateffectivedemocracy–asopposedtopurelyformalorinstitutionaldemocracy–islinkedtoruleoflaw:aformallydemocraticcountrythatisnotcharacterizedbytheruleoflawisnotdemocraticinthefullsenseoftheterm.InordertorepresentthisthickconceptofdemocracytheymultiplytheFreedomHouseindicesbyindicesofcorruption(drawnfromTransparencyInternationalortheWorldBank),producinganindexofeffectivedemocracy.SeeHadenius&Teorell(2005)andKnutsen(2010)forcriticaldiscussions.

14

difficultyincreaseswhenanalyzingtheseconceptionsovertimeandacrosscountriesona

globalscale.Thisfactlimitsthescopeofanyempiricalendeavor.

Sixth,principlesandcomponents,whilemucheasiertodefinethandemocracy (at-

large), are still resistant to authoritative conceptualization. Our objective has been to

identifythemostessentialanddistinctiveattributesassociatedwiththeseconcepts.Even

so,wearekeenlyawarethatothersmightmakedifferentchoices,andthatdifferenttasks

require different choices. The goal of the proposed conceptual framework is to provide

guidance,notto legislate inanauthoritativefashion.Theschemademonstrateshowthe

various elements of V-Dem hang together, according to a particular set of inter-

relationships. We expect other writers will assemble and dis-assemble these parts in

whateverfashionsuitstheirneedsandobjectives.Inthisrespect,V-Demhasthemodular

qualitiesofaLegoset.

Finally,asshouldbeobvious,thissectionapproachesthesubjectfromaconceptual

angle.Elsewhere(e.g.,intheV-DemCodebookandinV-DemComparisonsandContrasts,

aswellasinworkingpapersfoundontheV-Demwebsite),wedescribetechnicalaspects

ofindexconstructioninmoredetail.

2. DataCollection

Theviabilityofanydatasethingescriticallyonitsmethodofdatacollection.V-Demaimsto

achievetransparency,precision,andrealisticestimatesofuncertaintywithrespecttoeach

(evaluativeandindex)datapoint.

HistoryofPolities

Ourprincipalconcern iswiththeoperationofpolitical institutionsthatexistwithin large

andfairlywell-definedpoliticalunitsandwhichenjoyamodicumofsovereigntyorserveas

operational units of governance (e.g., colonies of overseas empires).We refer to these

unitsaspolitiesorcountries.8

8Wearenotmeasuringdemocracywithinverysmallcommunities(e.g.,neighborhoods,schoolboards,municipalities,corporations),incontextswherethepoliticalcommunityisvaguelydefined(e.g.,

15

Wearenotconcernedmerelywiththepresentandrecentpastofthesepolities.In

our view, understanding the present – not tomention the future – requires a rigorous

analysisofhistory.Theregimesthatexisttoday,andthosethatwillemergetomorrow,are

theproductofcomplexprocessesthatunfoldoverdecades,perhapscenturies.Although

regime changes are sometimes sudden, like earthquakes, these dramatic events are

perhaps sometimes to be understood as a combination of pent-up forces that build up

over longspansof time,notsimplytheprecipitating factorsthatreleasethem.Likewise,

recent work has raised the possibility that democracy’s impact on policies and policy

outcomestakeeffectoveraverylongperiodoftime(Gerringetal.,2005)andthatthere

are indeed sequences in terms of necessary conditions in democratization (Wang et al.

2015).Arguably,short-termandlong-termeffectsarequitedifferent,whetherdemocracy

isviewedasthecauseoroutcomeoftheoreticalinterest.Forallthesereasons,webelieve

thatafullunderstandingofdemocratizationdependsuponhistoricaldata.9

The advantage of our topic – in contrastwith other historicalmeasurement tasks

suchasnationalincomeaccounts–isthatmuchoftheevidenceneededtocodefeatures

of democracy is preserved in books, articles, newspapers archives, and living memory.

Democracyis,afterall,ahigh-profilephenomenon.Althoughasecretiveregimemayhide

thetruevalueofgoodsandservicesinthecountry,itcannotdisguisetheexistenceofan

election; those features of an election that might prejudice the outcome toward the

incumbent are difficult to obscure completely. Virtually everyone living in that country,

studyingthatcountry,orcoveringthatcountryforsomeforeignnewsorganizationoraid

organizationhasaninterestintrackingthisresult.

Thus,weregardthegoalofhistoricaldatagatheringasessentialandalsorealistic,

even if it cannot be implemented for every possible indicator of democracy. V-Dem

thereforeaimstogatherdata,wheneverpossible,backto1900forallterritoriesthatcan

transnationalmovements),oronagloballevel(e.g.,theUnitedNations).Thisisnottosaythattheconceptofdemocracyshouldberestrictedtoformalandwell-definedpolities.Itissimplytoclarifyourapproach,andtoacknowledgethatdifferentstrategiesofconceptualizationandmeasurementmayberequiredfordifferentsubjectareas.

9ThisechoesapersistentthemepresentedinCapocciaandZiblatt(2010),Knutsen,Møller&Skaaning(forthcoming),Teorell(2011),andinotherhistoricallygroundedwork(Nunn2009;Mahoney&Rueschemeyer2003;Pierson2004;Steinmo,Thelen,&Longstreth1992).

16

claimasovereignorsemi-sovereignexistence(i.e.theyenjoyedadegreeofautonomyat

leastwithrespecttodomesticaffairs)andserveastheoperationalunitofgovernance.The

latter criterionmeans that they are governed differently from other territories and we

might reasonably expect many of our indicators to vary across these units. Thus, in

identifying political units we look for those that have the highest levels of autonomy

and/or are operational units of governance. These sorts of units are referred to as

“countries,” even if they are not fully sovereign. This means, for example, that V-Dem

provides a continuous time-series for Eritrea coded as an Italian colony (1900-41), a

provinceofItalianEastAfrica(1936-41),aBritishholdingadministeredunderthetermsof

aUNmandate(1941-51),afederationwithEthiopia(1952-62),aterritorywithinEthiopia

(1962-93), and an independent state (1993-). For further details, see V-Dem Country

CodingUnits.Inthefuture,weplantoaddinformationinthedatasetanddocumentation

to link predecessor and successor states, facilitating panel analysis with continuous

country-levelunits.

V-Dem provides time-series ratings that reflect historical changes as precisely as

possible. Election-specific indicators are coded as events occurring on the date of the

election.Wecodeotherindicatorscontinuously,withanoption(thatsomecodersutilize)

tospecifyexactdates(day/month/year)correspondingtochangesinaninstitution.

Date-specificdatacanbeaggregatedat12-monthintervals,whichmaybeessential

for time-series where country-years form the relevant units of analysis. The V-Dem

“standard”dataset is inthecountry-year format,wheredate-specificchangeshavebeen

aggregatedtogetherattheyearlevel.However,wealsoprovideacountry-datedatasetfor

userswhowant greater precision. In the data archive accessible via the data download

pageonourwebsite,wealsoprovidetherawcoder-leveldata.Doingsoallowsusers to

inspect thedatadirectlyoruse it for alternateanalyses. Finally, in the samearchivewe

also provide the posterior distributions from the Bayesian ordinal IRT model for each

variabletofacilitatetheirdirectuseinanalyses.

Currently,weareworking toextendV-Demcodingback further inhistorical time,

i.e., to1789, for85sovereigncountriesandforaselectionof indicators.Thiscodingwill

enhance our knowledge of democratic development for countries whose process of

democratizationbeganpriortothetwentiethcentury.Itwillalsoenhanceourknowledge

17

of the pre-democratic history of all countries, a history that may exert an enduring

influenceoversubsequentdevelopmentsinthe20thand21stcenturies.

CodingTypes

The350+V-DemspecificindicatorslistedinV-DemCodebookfallintofourmaintypes:(A)

factual indicatorscodedbymembersof theV-Demteam, (B) factual indicatorscodedby

Country Coordinators, (C) evaluative indicators based on multiple ratings provided by

experts, and (D) composite indices.Part IofV-DemCodebook describes these indicators

Parts II and III provide a fifth type of indicators: (E) extant data (both factual and

subjective).

We gather Type (A) data from extant sources, e.g., other datasets or secondary

sources,as listed intheCodebook.Thesedataare largelyfactual innature,thoughsome

coderjudgmentmayberequiredininterpretinghistoricaldata.PrincipalInvestigatorsand

Project Managers supervise the collection of these data, which Assistant Researchers

connected to the project carry out using multiple sources, with input from V-Dem’s

CountryCoordinators.

CountryCoordinators,underthesupervisionofRegionalManagers,gatherType(B)

datafromcountry-specificsourcesby.AswithType(A)data,thissortofcodingislargely

factualinnature.

Type(C)datarequiresagreaterdegreeof judgmentaboutthestateofaffairs ina

particular country at a particular point in time. Country Experts code these data. These

experts are generally academics (about 80%) or professionals working in government,

media, or public affairs (e.g., senior analysts, editors, judges); they are also generally

nationalsofand/orresidents inacountryandhavedocumentedknowledgeofboththat

country and a specific substantive area. Generally, each Country Experts code only a

selection of indicators following their particular background and expertise (e.g. the

legislature).

Type (D) data consists of indices composed from (A), (B), or (C) variables. They

includecumulativeindicatorssuchas“numberofpresidentialelectionssince1900”aswell

as more highly aggregated variables such as the components and democracy indices

18

describedintheprevioussectionanddetailedinAppendixA.

WedrawType(E)datadirectlyfromothersources.TheyarethereforenotaV-Dem

product.TherearetwogenresofE-data.Thefirstgenreconsistsofalternativeindicesand

indicators of democracy found in Part II of V-Dem Codebook, which may be useful to

compare and contrast with V-Dem indices and indicators. This genre also includes

alternative versions of the V-Dem indices that are ordinal instead of interval (Lindberg

2015).ThesecondtypeofE-indicatorsconsistoffrequentlyusedcorrelatesofdemocracy

suchasGDP.TheyarefoundinPartIII.

ExpertRecruitment

Type (C) coding–byCountry Experts –involves evaluative judgmentson thepart of the

coder.Asaresult,wetakeanumberofprecautionstominimizeerrorinthedataandto

gaugethedegreeofimprecisionthatremains.10

An important aspect of these precautions is the fact thatwe endeavor to find a

minimumoffiveCountryExpertstocodeeachcountry-yearforeveryindicator.Thequality

and impartiality of C-data naturally depends on the quality of the Country Experts that

provide the coding. Consequently, we pay a great deal of care and attention to the

recruitmentofthesescholars,whichfollowsanexactingprotocol.

First,weidentifyalistofpotentialcodersforacountry(typically100-200namesper

country).RegionalManagers,inconsultationwithCountryCoordinators,usetheirintimate

knowledge of a country to compile the bulk of the experts on this list. Assistant

Researchers locatedattheV-DemInstitute(UniversityofGothenburg)alsocontributeto

thislist,usingreadilyavailableinformationdrawnfromtheInternet.11Othermembersof

theproject team(PIs,PMs,andassociates)mayalsosuggestcandidates.Atpresent,our

databaseofpotentialCountryExpertscontainssome18,000names.

RegionalManagers and Country Coordinators thus play a critical role in the data

collectionprocess.V-Dem’sapproachistorecruitRegionalManagerswhoarenationalsor

10ForaperceptivediscussionoftheroleofjudgmentincodingseeSchedler(2012).11ResearchAssistantsattheUniversityofNotreDamealsosuppliedmorethan3,000namesforallregionsin2011-2013,usinginformationfromtheInternet.

19

residentsofoneofthecountriesineachregionwheneverpossible.TheRegionalManagers

aretypicallyprominentscholars inthefieldwhoareactiveasprofessors intheregion in

question.Insomecases,RegionalManagersarelocatedoutsideoftheregion,iftheyare

currentlyactiveinwell-respectedinternationalthinktanksorsimilarinstitutions.Country

Coordinatorsarealmostalwaysnationalsandresidentsofthecountrytobecoded.They

arealsoscholars,althoughtheyaretypicallymorejuniorthanRegionalManagers.

Using short biographical sketches, publications, website information, or similar

material we compile basic information for each Country Expert: their country of origin,

current location, highest educational degree, current position, and area of documented

expertise(relevantfortheselectionofsurveystheexpertmightbecompetenttocode)to

makesureweadheretothefiverecruitmentcriteria.

RegionalManagers,CountryCoordinators,andotherprojectteammembersreferto

five criteriawhen drawing up the list of potential Country Experts. Themost important

selectioncriterionisanindividual’sexpertiseinthecountry(ies)andsurveystheymaybe

assigned to code. This expertise is usually signified by an advanced degree in the social

sciences, law,orhistory;arecordofpublications;orpositions inoutsidepoliticalsociety

that establish their expertise in the chosen area (e.g. a well-known and respected

journalist; a respected former high court judge). Regional Managers and Country

Coordinatorsmayalsoindicatewhichsurveysapotentialcoderhasexpertisein.Naturally,

potentialcodersaredrawntoareasofthesurveythattheyaremostfamiliarwith,andare

unlikelytoagreetocodetopicstheyknowlittleabout.Asaresult,self-selectionalsoworks

toachieveourprimarygoalofmatchingquestionsinthesurveywithcoderexpertise.

Thesecondcriterionisconnectiontothecountrytobecoded.Bydesign,threeout

of five(60%)oftheCountryExpertsrecruitedtocodeaparticularcountry-surveyshould

be nationals or permanent residents of that country. Exceptions are made for a small

numberofcountrieswhere it isdifficulttofind in-countrycoderswhoarebothqualified

andindependentofthegoverningregime,orwherein-countrycodersmightbeplacedat

risk.ThiscriterionhelpsusavoidpotentialWesternorNorthernbiasesincoding.

The third criterion is the prospective coder’s seriousness of purpose, i.e. her

willingness to devote time to the project and to deliberate carefully over the questions

asked in the survey. Sometimes, personal acquaintanceship is enough to convince a

20

RegionalManager and a Country Coordinator that a person is fit, or unfit, for the job.

Sometimes, this feature becomes apparent in communications with ProgramManagers

thatprecedetheoffertoworkonV-Dem.Thiscommunicationisquiteintensive,withan

averageof13interactionsbeforecodingisconcluded,andinvolvesrequiringthepotential

coder to read and work with several lengthy, detailed documents. This process readily

identifiespotentialcoderswhoarenotseriousenough.

The fourth criterion is impartiality. V-Demaims to recruit coderswhowill answer

surveyquestionsinanimpartialmanner.Wethereforeavoidthoseindividualswhomight

bebeholdentopowerfulactors–byreasonofcoercivethreatsormaterialincentives–or

whoserveasspokespersonsforapoliticalpartyorideologicaltendency.Closeassociation

(current or past) with political parties, senior government officials, politically affiliated

think-tanks or institutes is grounds for disqualification. In cases where finding impartial

coders is difficult, we aim to include a variety of coderswho, collectively, represent an

arrayofviewsandpoliticalperspectivesonthecountryinquestion.

The final criterion is obtaining diversity in professional background among the

coders chosen for a particular country. For certain areas (e.g., themedia, judiciary, and

civil societysurveys)suchdiversityentailsamixtureofacademicsandprofessionalswho

study these topics. It also means finding experts who are located at a variety of

institutions,universitiesandresearchinstitutes.

Afterweighingthesefivecriteria,wegivethe100-200potentialexpertsonourlist

ofcandidatesarankfrom“1”to“3,”indicatingtheorderofprioritywegivetorecruiting

anExpert.TheRegionalManagersandCountryCoordinatorsareprimarilyresponsiblefor

the ranking, but Program Managers and one of the Principal Investigators may review

thesechoices.

Using this process,wehave recruitedover 2,500 scholars and experts fromevery

corneroftheworld.About30percentoftheCountryExpertsarewomen,12andover80

percent have PhDs or MAs and are affiliated with research institutions, think tanks, or

similarorganizations.

12ThenumberofwomenamongtheranksofourCountryExpertsislowerthanwewouldhaveliked,anditoccurreddespiteourstrenuousefforts.However,itreflectsgenderinequalitieswithregardtoeducationanduniversitycareersintheworld.

21

In order to preserve confidentiality, V-Dem has adopted a policy of neither

confirmingnordenyingtheidentitiesofCountryExperts.OnlythetwoProgramManagers

are actively involved in this final stage of recruitment (and two of the Principal

Investigators,whohavesupervisoryauthorityovertheprocess)areawareoftheidentities

ofthefinalchosenCountryExperts.Theseindividualsalsohandleallcorrespondencewith

Country Experts, so this confidentiality is not inadvertently revealed through

communication..

Thus,whiletheidentityofothermembersoftheV-Dementerpriseispublicizedon

ourwebsite,wepreservetheconfidentialityofCountryExperts.Severalreasonsliebehind

thisdecision.First,thereareanumberofcountriesintheworldwhereauthoritiesmight

sanctionCountryExperts,ortheirfamiliesorfriends,fortheirinvolvementintheproject.

Second,thereisnowaytopredictwhichcountrymayinthefuturebecomerepressiveand

thereforesanctiontheCountryExperts.Third,weanticipatethatV-Demdatamaybecome

used inevaluationsandassessments internationally inwaysthatcouldaffectacountry’s

status. Thus, one may foresee incentives for certain countries’ governments and other

actors to try to affect their ratings. For all these reasons, we consider it essential to

preserveCountryExpertanonymity.

ExpertCodingProcess

ThetwoProgramManagersattheV-DemInstitute(UniversityofGothenburg)thenissue

invitationsuntilthequotaoffivecodersisobtained.13Wereplacethosewhofailtobegin

orcompletethesurveyinareasonabletimeinasimilarmanner.Codersreceiveamodest

honorarium for their work that is proportional to the number of surveys they have

completed.

C-indicatorsareorganizedintofourclustersandelevensurveys:

1. Elections Politicalparties/electoralsystems

2. Executive Legislature

13BeforeJuly2014,therewasathirdProgramManagerattheKelloggInstituteoftheUniversityofNotreDamewhomanagedmostcountryexpertsinLatinAmericaandafewintheMiddleEastandNorthAfrica.

22

Deliberation

3. Judiciary Civilliberty Sovereignty

4. CivilsocietyorganizationsMedia

Politicalequality

Wesuggest(butdonotrequire)thateachCountryExpertcodeatleastonecluster.

In consultation with the Country Coordinators and Principal Investigators, Regional

ManagerssuggestwhichCountryExpertmightbemostcompetenttocodewhichsurveys.

We then consult with the Country Expert about which cluster(s) they feel most

comfortablecoding.Mostcodeonlya fewofthesurveys.Thismeansthat, inpractice,a

dozenormoreCountryExpertsprovideratingsforeachcountry(withatargetoffivefor

eachcountry/indicator/year,asstated).14

AllCountryExpertscarryouttheircodingusingaspeciallydesignedonlinesurvey.

Theweb-basedcodinginterfacesaredirectlyconnectedwithapostgresdatasetwherewe

storetheoriginalcoder-leveldata.Figure4providesanexampleofthecodinginterface.

ThecodinginterfaceisanessentialelementofV-Dem’sinfrastructure.Itconsistsof

aseriesofweb-basedfunctionsthatallowCountryExpertsandCountryCoordinatorsto(1)

logintothesystemusingtheirindividual,randomizedusernameandself-assigned,secret

password;(2)accesstheseriesofsurveysassignedtothemforaparticularcountry(orset

ofcountries);and(3)submitratingsforeachquestionoveraselectedseriesofyears.

Thecoding interfaceallows formanytypesofquestions (binary,ordinal,multiple

selection,etc.),country-specificandquestion-specificyearmasks(e.g.,allowingthecoding

of elections only in years they occurred), and question-specific instructions and

clarifications.

The interface also requires that, for each rating, experts assign a level of

confidence,indicatinghowconfidenttheyarethattheirratingiscorrect(onascaleof0-

14Insomerarecases---mainlysmallandunder-studiedcountries---weaskindividualexpertstocodethewholesetofsurveys,simplybecauseexpertsonthevariousspecificpartsofthesurveyarenotavailable.Similarly,itisalsonotalwayspossibletoreachthegoalofhavingfivecountryexpertscodeeachindicatorforthesecountries.

23

100,where each 10-percent interval has a substantive anchor point), providing another

instrumentformeasuringuncertaintyassociatedwiththeV-Demdata.Weincorporatethis

confidence into the measurement model. Country Experts also have an opportunity to

register uncertainty in the “Remarks” field that lies at the end of each section of the

survey.Here, experts can comment (inprose)onany aspectof the indicatorsor ratings

thatshefoundproblematicordifficulttointerpret.

Fig.4ExampleofCodingInterface

Finally, in order to ensure wide recruitment of potential experts, and minimize

confusiondue tounfamiliaritywith English,we translate all type-Cquestions, aswell as

coder-instructionsanddocumentationforthem,intofiveotherlanguages:Arabic,French,

Portuguese,Russian,andSpanish.Approximately15percentofthecoderscodeinanon-

24

Englishversionofthequestionnaire.CountryExpertsgetasmallremunerationasatoken

ofappreciationfortheirtime.15

Wetakeanumberofstepstoassureinformedconsentandconfidentialityamong

participants.Theon-linesurveyprovidesfullinformationabouttheproject(includingthis

document)andtheuseofthedata,sothatcodersarefullyinformed.Italsorequiresthat

prospectivecoderscertify that theyaccept the termsof theagreement.Theyaccess the

surveysonlywitharandomizedusernamethatweassignandasecretpasswordthatthey

createthemselves.Westorethedatatheysupplyonafirewall-protectedserver.Anydata

we release to thepublic excludes information thatmightbeused to identify coders.All

personal identifying information is kept in a separate database in order to ensure

confidentiality.

Aspeciallydesignedprogramminginterfaceisemployedtomanagethedatabaseof

potential country experts. It includes many tools that enable us to handle over 2,500

Country Experts while guaranteeing their safety and confidentiality. These tools also

ensureconsistencyininstructionsandinformationsenttoCountryExperts,qualitycontrol

andcleaningofdata,followupandevaluationofthecodingprocess.Itisdirectlylinkedto

thepostgresdatabasewhereratingsarestored.

For example, the roster of hundreds of potential experts for Country X and all

associatedinformationisuploadedintothemanagementdatabaseanddirectlyaccessible

from the interface. Then perhaps 20 or so Country Experts are invited to participate as

coders,using speciallydesigned invitation letters in six languages that is associatedwith

standardizedinformationmaterials.

When a potential Country Expert accepts an invitation, the Program Managers

mark their acceptance, the areas of coding, and assign themas coders for oneormore

countries.Themanagementsoftwarethenautomaticallycommunicateswiththepostgres

database and the coding interfaces, creating a coder ID. The software then creates the

15Fromwhatwecantell,thisisnotasignificantthreattocodingvalidity.Fewindividualsseemtohavebeenmotivatedtoconductthisarduouscodingassignmentforpurelymonetaryreasons:V-Dempaysverylittlerelativetowhathighlyqualifiedexpertscouldearnforthesameamountofworkfromotherpursuits.Furtherstrengtheningthispoint,thereseemstobenorelationshipbetweenthewealthofthecountryandourabilitytorecruitcoders:wehavefacedchallengesgettingexpertstoagreetoconductcodingforthepoorestaswellastherichestcountriesintheworld.

25

sameIDinrelevanttablesandinterfacecommunications,generatesauseridtobeusedin

the coding interfaces, and sends an email to the new coder with their username and

instructionsforhowto log inandcreateauniqueandsecretpassword.Fromthatpoint,

themanagement software communicates automaticallywith the postgres database and

determines each coder ID’s progress on coding for each of the indicators to which the

coder is assigned. The software also reports to the Program Managers on the coder

management toolpages.Otherpartsof theprocess including thehandlingof signed tax

forms and applications; as well as payments are similarly connected in the coding

managementtool.

Thecodermanagementtoolisjustoneofover20sophisticatedtoolsamongtheV-

Dem management interfaces in the software. There are tools for management of

countries, rounds of surveys, surveys and questions, country coordinators, regional

managers, for logging activities, analyses of progress on recruitment as well as coding,

planning, and general management. A web-interface portal is connected to the

management software, allowing Regional Managers to securely upload Country Expert

rosterstothedatabasewithouthavingtoshareconfidentialinformationviaemail.

Bridge-andlateralcoding

In addition to regular ratings by multiple Country Experts for C-type indicators, we

encourage Country Experts to conduct bridge coding (coding ofmore than one country

throughtime)and lateralcoding(coding limitedtoasingleyear–2012).Thepurposeof

this additional coding is to assure cross-country equivalence by forcing coders tomake

explicit comparisons across countries. This helps themeasurementmodel estimate, and

correctfor,systematicbiasesacrosscodersandacrosscountriesthatmayresultifCountry

Expertsemployvaryingstandardsintheirunderstandingofaquestion,e.g.,aboutwhata

“high”levelofrepressionmightconsistof.

Throughoutimplementationoftheproject,wehaveencouragedCountryExpertsto

codemultiplecountries.Thistypeofcodingiscalledbridgecoding.Anexpertwhoagrees

tocodeoneormoreadditional countries receives the samesetof surveys for the same

time period as the original country they coded; bridge coding therefore typically covers

1900 to the present. Bridge coding helps us better model how Country Experts make

26

judgments between different response categories, and allows us to incorporate this

informationintotheestimatedscoreforeachcountry-indicator-year/date.

Bridge coding is most useful when the chosen countries have different regime

histories.ThisgeneratesvarianceacrossaCountryExpert’sratings,whichinturnprovides

information about the coder’s judgments that can be used to inform themeasurement

model.Inordertomaximizevariance,andthereforegainasmuchinformationaspossible

abouteachexpert’s thresholdsand reliability,weencourageCountryExperts to select–

from among countries they are familiar with – those that have the most distinctive

historicaltrajectories.

AsofDecember2015,wehaveover360bridge coders–about14percentof all

CountryExperts.Onaverage,theseexpertshavecoded6.1surveysfor2.1countries.

Constraints of time or expertise sometimes prevent Country Experts from

conductingbridge coding. In these situations,weencourageCountryExperts toperform

thesimplertypeofcross-countrycomparisoncalled lateralcoding.That is, inadditionto

theiroriginalcodingofonecountryovertime(e.g.,from1900tothepresent),theycodea

numberofcountriesforasinglepointintime–January1,2012–focusingonthesameset

ofquestions.

SomeCountryExpertshavecodedupto14countries.Moretypically,lateralcoding

extends to a few countries. To date, 350 Country Experts (about 15%) have performed

lateral coding, covering on average of 5.5 countries and 6.3 surveys. As a result, lateral

coding by regular Country Experts has provided linkages equivalent to over 1,100 “fully

covered” countries – in other words, countries that have been “cross-coded” by

lateral/bridgecodingacrossallindicatorsinthedataset.

Phases

In the firstphaseofdatacollection (2012 to2014),weaskedCountryExperts tocodea

cluster(s) of surveys for a single country from 1900 (or the relevant first year for a

particularcountry)totheendof2012.

FromNovember2014toMarch2015weconductedthefirstupdate.Itcovered54

countries–bringingtheirdatacurrentuptoend-2014–andalsoaddedsixnewcountries

27

(with data from 1900 to 2014). Due to coder attrition, coding for the update was

conducted by amix of returning Country Experts and new Country Experts.When they

codedfor2013and2014,returningCountryExpertssawtheirpreviously-submittedratings

for the years from 2010 to 2012, so as to encourage consistency in ratings over time,

thoughwedidnot allow them to alter those ratings.WeaskednewCountry Experts to

codetenyears(2005-2014)soastoensurethattheirscoresoverlapbyanumberofyears

withreturningCountryExperts’ratings.

Wearecurrentlyconductingasecondroundofannualupdates,covering2015.We

will release data from this round byMarch 31, 2016. To enhance consistency in coding

acrossrounds, returningcoderswillagainseetheirpriorratings,butwillnowbeableto

revise them, if theywish todoso.Finally,weare implementingaseriesofvignettes for

eachsurveytogiveusadditionalleverageonmeasurementerror.

This round of updates will occur between November 2015 and March 2016. It

covers76countries,22ofwhichwerealsocovered inthefirstupdate.Hence,byMarch

31st2016,theV-Demdatasetwillhavedatafor173countries:upto2012for59countries,

andupto2014or2015for114countries.

3. Measurement

Havingdiscussedtheprocessofdatacollection,weproceedtothetaskofmeasurement.

Under this rubric, we include (a) the questionnaire, (b) our measurement model, (c)

methodsof identifyingerror inmeasurement, (d) studiesofmeasurementerror,and (e)

methodsofcorrectingerror.Inprinciple,thediscussionsarerelevantfordifferenttypesof

data(A,B,andCintheV-Demscheme)butmostifnotallofthemaremuchmoreacute

when it comes to expert-based coding of evaluative, non-factual yet critical indicators.

Hence,mostofthefollowingfocusesontheC-typeindicators.

TheQuestionnaire

Themost important featureofasurvey is theconstructionofthequestionnaire itself. In

craftingindicatorstomeasuretheC-typedata,wehavesoughttoconstructquestionswith

bothspecificandclearmeanings,andwhichdonotsuffer fromtemporalorspatialnon-

28

equivalence.Todesignthesequestions,weenlisted leadingscholarsondifferentaspects

ofdemocracyanddemocratizationasProjectManagers.

We enrolled each Project Manager because of her record of scholarly

accomplishment in a particular area related to issues of democracy (e.g. legislatures,

executives, elections, and civil society), with the goal of creating a team that also had

substantiveexperiencesandexpertiseonallregionsoftheworld.ProjectManagersbegan

designingsurvey-questionsintheirareaofexpertisein2009,andwecollectivelyreviewed

andrefinedtheirquestionsoverthecourseoftwoyears.

WeimplementedapilotoftheV-Demsurveyin2011,whichservedasaninitialtest

of our questionnaire. It was implemented for 12 countries, two (one “easy” and one

“hard”) fromeachof the sixmajor regionsof theworldenlistingover120pilot-Country

Experts and resulted in some 450,000 ratings on preliminary indicators. The results

promptedrevisionsinthenextroundofsurveys.Anotherroundofcollectivedeliberation

followed, involving consultationswith scholars outside of the project team. The revised

questionsforC-codingthusenduredseveralroundsofreviewwithProjectManagersand

outside experts over the course of two years before emerging in their final form, as

describedintheCodebook.

Identifying,Correcting,andQuantifyingMeasurementError

Evenwithcarefulquestiondesign,aprojectofthisnaturewillencountererror.Sucherror

may be the product of linguistic misunderstandings (most of our coders do not speak

English as their first language, and some take the survey in a translated form),

misunderstandingsaboutthewayaquestionappliestoaparticularcontext,factualerrors,

errors due to the scarcity or ambiguity of the historical record, differing interpretations

abouttherealityofasituation,variationinstandards,coderinattention,errorsintroduced

bythecoderinterfaceorthehandlingofdataonceithasbeenenteredintothedatabase,

orrandommistakes.

Someof theseerrorsare stochastic in the senseofaffecting theprecisionofour

estimatesbutnot their validity.Othererrors are systematic, potentially introducingbias

intotheestimates thatweproduce. In thissection,we firstdescribethemethodological

29

toolsweusetomodelandcorrectforsystematicbiasincoders’answerstoourquestions,

aswellas toprovideestimatesof thereliabilityof thesecodings. Wethendescribe the

procedures we use to assess the validity of our estimates. Finally, we explain how we

identifythemostserioussourcesofmeasurementerror,inordertocontinuouslyimprove

howwegatherandsynthesizedata.

MeasurementModels

The most difficult measurement problems concern the C-type questions, all of which

require substantial case knowledge and generally somedegree of subjective evaluation.

Having five coders for each of these questions is immensely useful, as it allows us to

conductinter-coderreliabilitytests.Thesesortsoftests–standardinmostsocialscience

studies–areonlyrarelyifeveremployedinextantdemocracyindices.

Whileweselectexpertscarefully,theyexhibitvaryinglevelsofreliabilityandbias,

and may not interpret questions consistently. In such circumstances, the literature

recommends that researchers usemeasurementmodels to aggregate diversemeasures

wherepossible,incorporatinginformationcharacterizedbyawidevarietyofperspectives,

biases,and levelsof reliability (Bollen&Paxton2000,Clinton&Lapinski2006,Clinton&

Lewis2008, Jackman2004,Treier& Jackman2008,Pemstein,Meserve&Melton2010).

Therefore,tocombineexpertratingsforaparticularcountry-indicator-yeartogeneratea

single“bestestimate”foreachquestion,weemploymethodsinspiredbythepsychometric

and educational testing literature (see, e.g., Lord&Novick 1968, Jonson&Albert 1999,

Junker1999,Patz& Junker1999).Theunderpinningsof thesemeasurementmodelsare

straightforward:theyusepatternsofcross-rater(dis)agreementtoestimatevariations in

reliabilityandsystematicbias.Inturn,thesetechniquesmakeuseofthebiasandreliability

estimates to adjust estimates of the latent—that is, only indirectly observed—concept

(e.g.,executive respect for theconstitution, judicial independence,orproperty rights) in

question. These statistical tools allow us to leverage our multi-coder approach to both

identifyandcorrectformeasurementerror,andtoquantifyconfidenceinthereliabilityof

our estimates. Variation in these confidence estimates reflect situationswhere experts

disagree, orwhere little information is available because few raters have coded a case.

These confidence estimates are tremendously useful. Indeed, to treat the quality of

30

measures of complex, unobservable concepts as equal across space and time, ignoring

dramatic differences in ease of access andmeasurement across cases, is fundamentally

misguided,andconstitutesakeythreattoinference.

ThemajorityoftheC-typequestionsareordinal: theyrequireCountryExpertsto

rankcasesonadiscretescale. Take,forexample,thefollowingquestionaboutelectoral

violence:

Question:Inthisnationalelection,wasthecampaignperiod,electionday,andpostelectionprocessfreefromothertypes(notbythegovernment,therulingparty,ortheiragents)ofviolencerelatedto the conductof theelection and the campaigns (butnot conductedby the government and itsagents)?Responses:

0. No. There was widespread violence between civilians occurring throughout the electionperiod,orinanintenseperiodofmorethanaweekandinlargeswathsofthecountry.Itresultedinalargenumberofdeathsordisplacedrefugees.

1. Notreally.Thereweresignificantlevelsofviolencebutnotthroughouttheelectionperiodorbeyondlimitedpartsofthecountry.Afewpeoplemayhavediedasaresult,andsomepeoplemayhavebeenforcedtomovetemporarily.

2. Somewhat.Thereweresomeoutburstsoflimitedviolenceforadayortwo,andonlyinasmall part of the country. The number of injured and otherwise affected was relativelysmall.

3. Almost. There were only a few instances of isolated violent acts, involving only a fewpeople;noonediedandveryfewwereinjured.

4. Peaceful.Noelection-relatedviolencebetweenciviliansoccurred.

Note, inparticular, thattheserankingsdonot followan interval-levelscale. Onecannot

subtractalmost frompeaceful andgetnot really. Furthermore, it neednotbe the case

thatthedifferencebetweennotreallyandsomewhatisthesameasthatbetweenalmost

and peaceful. Perhaps most importantly, although we strive to write questions and

responses thatarenotoverlyopen to interpretation,wecannotensure that twocoders

look at descriptions like somewhat in a uniform way—even when somewhat is

accompaniedbyacarefullyformulateddescription—especiallybecausecodershavewidely

varying backgrounds and references. In other words, one coder’s somewhat may be

another coder’s not really; a problem known as scale inconsistency. Therefore, we use

Bayesian item response theory (IRT)modeling techniques (Fox 2010) to estimate latent

politycharacteristicsfromourcollectionofexpertratingsforeachordinal(C)question.

Specifically, we fit ordinal IRTmodels to each of our ordinal (C) questions. (See

Johnson&Albert1999foratechnicaldescriptionofthesemodels.)Thesemodelsachieve

threegoals.First,theyworkbytreatingcoders’ordinalratingsasimperfectreflectionsof

31

interval-levellatentconcepts.Withrespecttotheexamplequestionabove,ourIRTmodels

assumethatelectionviolencerangesfromnon-existenttoendemicalongasmoothscale,

and coders observe this latent characteristic with error. Therefore, while an IRTmodel

takesordinalvaluesasinput,itsoutputisaninterval-levelestimateofthegivenlatenttrait

(e.g.electionviolence). Interval-valuedestimatesarevaluableforavarietyofreasons; in

particular,theyareespeciallyamenabletostatisticalanalysis.Second,IRTmodelsallowfor

the possibility that coders have different thresholds for their ratings (e.g. one coder’s

somewhatmight fall above another coder’salmost on the latent scale), estimate those

thresholds from patterns in the data, and adjust latent trait estimates accordingly.

Therefore,theyallowustocorrectforthispotentiallyserioussourceofbias.16Thisisvery

important in a multi-rater project like V-Dem, where coders from different geographic,

cultural,andotherbackgroundsmayapplydifferingstandardstotheirratings.Finally,IRT

modelsassumethatcoderreliabilityvaries,produceestimatesofraterprecision,anduse

these estimates—in combination with the amount of available data and the extent to

whichcodersagree—toquantifyconfidenceinreportedscores.

Sinceourcodersgenerallyrateonecountrybasedontheirexpertise,itisnecessary

toutilizelateralcoders.Aspreviouslydescribed,thesecodersratemultiplecountriesfora

limitedtimeperiod(mostlyoneyear,butinsomecasesten).Wehaveatpresentsome350

lateralcoders.Inaddition,wehaveover360bridgecoders,asdiscussedabove.Theseare

coderswho code the full time series (generally 1900-2012) formore than one country,

coveringoneormoreareas (“surveys”).17 Essentially, this codingprocedureallowsus to

mitigate the incomparability of coders’ thresholds and the problem of cross-national

estimates’calibration(Pemsteinetal.2015).Whilehelpfulinthisregard,ourtestsindicate

that,giventhesparsityofourdata,eventhisextensivebridge-coding isnotsufficient to

fully solve cross-national comparability issues. We therefore employ a data-collapsing

16Givencurrentlyavailabledata,wemustbuildinassumptions—formally,theseareknownashierarchicalpriors—thatrestricttheextenttowhichcoders’thresholdestimatesmayvary.Informally,whileweallowcoderstolookatordinalrankingslikesomewhatandalmostdifferently,weassumethattheirconceptionsarenottoodifferent.Weareworkingtorelaxtheseassumptionsbycollectingmoredata.TechnicaldetailsareavailableinV-DemWorkingpaperno.19whichwillbeavailableinDecember2015,andfullcodeisreleasedwiththedataset.

17Thuswehavelateral/bridgecodingcoveringtheequivalentofover1,100“fullcoverage”ofallcountry-questions.

32

procedure.Atitscore,thisprocedurereliesontheassumptionthataslongasnoneofthe

experts change their ratings (or their confidence about their ratings) for a given time

period, we can treat the country-years in this period as one year. The results of our

statisticalmodelsindicatethatthistechniqueisextremelyhelpfulinincreasingtheweight

given to bridge coders, and thus further ameliorates cross-national comparability

problems.

As a final note, our model diverges from more standard IRT models in that it

employsempiricalpriors.Specifically,wemodelacountry-year’s latentscore foragiven

variableasbeingdistributedaccordingtoanormaldistributionwithanappropriatelywide

standarddeviationparameterandameanequaltotherawmeanofthecountry’sscores,

weighted by coder confidence and normalized across all country-years. More formally,

!! ~ !(!! , 1), where ! is the latent score for country-year !, and ! is the normalized

confidence-weighted average from the raw data. In contrast, most standard models

employavaguemeanestimate,i.e.!! ~ !(0,1).Ourapproachofusingempiricalpriorsis

similar to the standardapproach:ourwide standarddeviationparameter still allows for

the model output to diverge from prior as the data warrant. However, our approach

incorporatesouractualpriorbeliefsaboutacountry’sscoreandthusyieldsmoreaccurate

measures.Especially inthecaseofcountrieswithextremevalues,atraditionalapproach

risksbiasingoutputtowardthemean.

FutureversionsofourordinalIRTmodelswillimproveoncurrentestimatesintwo

primary ways. First, hierarchical IRTmodeling techniques (Patz et al. 2002,Mariano &

Junker2007)wouldallowustoborrowstrengthfromdifferentvariableestimates,yielding

more precise measures of each variable. Second, all raters complete a post-survey

questionnaire that asks demographic and attitudinal questions. Coders also report

personal assessments of confidence in their responses to each question. At present, of

these datawe only incorporate confidence into themodel, using it toweight our prior

meanestimates;furtheruseoftheseformsofdatainourmodelswillallowustoteaseout

patternsconcerningbiasesandreliabilityacrossdifferent typesofexperts,andgenerally

improvethequalityofourestimates.

For nominal and some dichotomous variables we use IRT techniques when

sufficient variation exists to identify rater thresholds. For the remaining variables we

33

providetheunweightedmean.

IdentifyingRemainingErrors

Toevaluatepossibleerrorsweemployanumberoftests,someofwhichareincorporated

into themeasurementmodels and others of which are applied ex post to examine the

validityofmodeloutput.

First, we have used data from the post-survey questionnaire that every V-Dem

coder completes to identify potential sources of bias. This survey delves into factors of

possible relevance to coder judgments, such as personal characteristics like sex, age,

country-of-origin,educationandemployment. Italso inquires intoopinions thatCountry

Expertsholdaboutthecountrytheyarecoding,askingthemtoassignapointscoreona0-

100scalesummarizingtheoverall levelofdemocracy in thecountryonJanuary1,2012,

using whatever understanding of democracy they choose to apply. We ask the same

questionaboutseveralprominentcountries fromaroundtheworldthatembodyvarying

characteristics of democracy/autocracy. Finally, the questionnaire contains several

questions intendedtoelicit thecoder’sviewsabouttheconceptofdemocracy.Wehave

runextensive testsonhowwell such individual-level factorspredictscountry-ratingsbut

havefoundthattheonlyfactorconsistentlyassociatedwithcountry-ratings iscountryof

origin (with “domestic” coders being harsher in their judgments). This is also the

individual-levelcharacteristicincludedinthemeasurementmodelestimates.

Inthefuture,weneverthelessplantouseeachpieceofinformationfromthispost-

survey questionnaire to help inform themeasurementmodel, i.e., to enhance precision

andlimitpossibleundetectedbiases.Themeasurementmodelwillalsotakeintoaccount

information we can glean from the performance of the coders that might serve as an

indicationoftheirlevelofattentiveness,effort,andknowledge.Thisinformationincludes

inter-coder reliability (assessed at the coder level across all codings), self-reported

confidence(ineachcoding),numberofcountry-yearscoded(alltogether),codingchanges

(the number of times that a coder changes their coding fromT-1 toT relative to other

codersforthatcountry/indicator,aggregatedacrossallcodings),timeontask(thenumber

of hours a coder is logged into the on-line system, discounted by the number of

country/indicator/yearss/hehascoded),accesses(thenumberoftimestheon-linesurvey

34

isaccessed),contacts(writingcommentsoraskingquestionsoftheV-Demteamthatare

non-logistical in nature), and response rate (assessed at the country level). (With the

exception of inter-coder reliability, these elements have not yet been included in the

model.)

Eachof theaforementioned featureswill alsobe tested independently. Thus,we

will be able to report on whether, and to what extent, each of the observed and self-

reportedfeaturesofthecodersaffectstheirratings.Inparticular,byincludinghierarchical

priors that dependon observed rater characteristics and behavior in our latent variable

model specifications—an approach often referred to as “empirical Bayes”—we can

evaluatetheextenttowhichsuchfeatureshelptoexplainraterbiasandreliability,while

simultaneouslyincorporatingthatinformationintoindicatorestimates.

Inaddition,wewillapplyseveralexpost tests toevaluatethequalityof thedata

emanatingfromthemeasurementmodel.Onesortoftestreliesonthedistributionofthe

data.Ifthedistributionofresponsesforaparticularcountry/indicator/yearisbi-modalwe

haveanobviousproblem:codersdisagreewildly.Thisalsomeansthatthepointestimate

fromthemeasurementmodelisunstable:achangeofcodingforanysinglecoder,orthe

additionofanewcoder,islikelytohaveabigimpactonthepointestimate.Disagreement

as registeredbyabi-modaldistributioncould representa situation inwhich the truth is

recalcitrant – presumably because available information about a topic is scarce and/or

contradictory.Oritcouldrepresenterrorsthatarecorrigible.

A secondapproach tovalidationcomparesV-Dem indiceswithother indices that

purport to measure similar concepts, i.e., convergent validity. For example, a set of

regressionsusingallavailabledataoftheV-DemElectoralDemocracyIndex–andsomeof

itsconstituentindicators–againstPolity2indicatesrelativelyhighcorrelations(Pearson’s

r= .85) and (separately) against FH Political rights (Pearson’s r= .90). Unfortunately,

techniquesof convergent validity are limited in their utility. First,wehave somedoubts

aboutthevalidityofstandardindices(seeComparisonsandContrasts).Second,standard

indices tend to hover at a higher level of aggregation, thus impairing comparability

betweenV-Demindicesandalternativeindices.Indeed,onlyafewextantindicesareclose

enoughinconceptionandconstructiontoprovideanopportunityfordirectcorroboration

withV-Demindices.

35

A third approach to validation focuses on face validity. Once data collection is

completeforagroupofcountries,RegionalManagersandothermembersoftheV-Dem

teamlookcloselyatpointestimatesinanattempttodeterminewhethersystematicbias

may exist. One major such review was conducted in October 2013 when almost all

RegionalManagers,allProjectManagers,ResearchFellows,PIsandstaff,spentfourdays

collectively reviewing all data collated at that point to validate the approach and

aggregationmethods.Theprocessoffacevaliditycheckshassincethenbeenrecurrent.

Finally,inthepresentroundofupdate(2015/2016),weareimplementingaseries

ofvignettesforeachsurveythatCountryExpertscode.Thevignettesarecalibratedatthe

thresholdsbetweenanswercategoriesandwillgiveusadditional leverageonsystematic

differencesinCountryExperts’ratingsdependingontheir“harshness”asraters.Thiswill

furtherreducemeasurementerrorinfuturereleasesofthedata.

CorrectingErrors

We correct problems with factual questions (B-type indicators) whenever the Principal

Investigators,inconsultationwiththerelevantProjectManagers,becomeconvincedthata

better (i.e., more correct) answer is available. Based on analysis of submitted data by

CountryCoordinators,certainvariablesweredesignatedasB+A.UsingtheoriginalB-data

as a point of departure and cross-checking with external resources, we designed and

implemented a coding scheme to re-code these indicators, as the Codebook describes.

Indicatorsaffectedincludeallindicatorsfromthedirectdemocracysurvey,fourindicators

ontheexecutive,fouronelectionsandnineonlegislature.Thedecisiontore-assignthese

indicators was also due to the interaction between question formulation and coder

interpretation, e.g. in some instances the meaning of “plebiscite” was interpreted in a

different way than what the Project Manager envisaged, leading to discrepancies in

coding.

We handle problemswithevaluative questions (C-type indicators)with restraint.

Wefullyexpectthatanyquestionrequiring judgmentwillelicitarangeofanswers,even

whenall codersarehighly knowledgeableabouta subject.A keyelementof theV-Dem

project – setting it apart frommost other indices that rely on expert coding – is coder

independence:eachcoderdoesherworkin isolationfromothercodersandmembersof

36

theV-Dem team (apart from clarifyingquestions about theprocess). Thedistributionof

responses across questions, countries, and years thus provides vital insight into the

relative certainty/uncertainty of each data point. Since a principal goal of the V-Dem

projectistoproduceinformativeestimatesofuncertaintywedonotwishtotamperwith

evidence that contributes to those estimates. Arguably, the noise in the data is as

informativeasthesignal.Moreover,waywardcoders(i.e.,coderswhodivergefromother

coders)areunlikelytohaveastronginfluenceonthepointestimatesthatresultfromthe

measurementmodel’saggregationacrossfiveormorecoders.Thisisespeciallythecaseif

thewaywardcodersareconsistentlyoff-center(acrossalltheircodings);inthiscase,their

weightindeterminingmeasurementmodelscoresisreduced.

That said, there have been instances in which we have altered C-data. A few

questions were largely of factual nature (e.g. number of legislative chambers; if a local

governmentexists,whichofficeswereelectedinaparticularelection,etc.).Sincewelater

acquired enough funding to have assistants conduct the factual coding based on

systematicconsultationofcrediblesources,wedischargedthedatasubmittedbyCountry

Experts for these particular questions and any “downstream” data. For example, if a

CountryExpert indicatedthatthereweretwochambersinthelegislatureforaparticular

year,shethencoded“downstream”inthequestionnaireaseriesofquestionsregarding

boththelowerandupperchamber.Ifourresearchestablishedthatanupperchamberdid

not in fact exist in that particular year,we cleaned the records of data providedby the

expertfortheupperchamber.Thiscleaningaffected19%ofallexecutivedatasubmitted

for thosedownstreamvariables, 7.7%of thedata in theelection surveyand11% in the

legislative survey. These numbers reflect places where coders unnecessarily coded due

eithertoa)problemwiththeskippingfunctioninthesurveys,b)coders’abilitytochange

the pre-coded, factual data, or c) an initial decision, subsequently reversed, to have

CountryExpertstoanswersomeoftheA-coded(morefactual)questions.

Inafinalcase,weremovedoriginalcodingbysomeCountryExpertsbecauseofa

factual misunderstanding (or misunderstanding about response-categories) about the

existenceoftheinternetineraspriortoitsinvention.

In all these situations,wemaintain the original coder-level data in archived files

thatmayberetrievedbyspecialrequestofthePIs.

37

VersionsofC-Variables

TheV-DemdatasetthencontainsA,B,C,andDindicatorsthatareallunique.Inaddition,

tofacilitateeaseofuseforvariouspurposes,theC-variablesaresuppliedinthreedifferent

versions(alsonotedintheV-DemCodebook):

1.“RelativeScale”-MeasurementModelOutput–hasnospecialsuffix(e.g.

v2elmulpar).Thisversionofthevariablesprovidescountry-year(country-dateinthe

alternativedataset)pointestimatesfromtheV-Demmeasurementmodeldescribed

above.Thepointestimatesarethemedianvaluesofthesedistributionsforeach

country-year.Thescaleofameasurementmodelvariableissimilartoanormal(“Z”)

score(i.e.typicallybetween-5and5,with0approximatelyrepresentingthemean

forallcountry-yearsinthesample)thoughitdoesnotnecessarilyfollowanormal

distribution.Formostpurposes,thesearethepreferredversionsofthevariablesfor

time-seriesregressionandotherestimationstrategies.

“MeasureofUncertainty”–MeasurementModelHighestPosteriorDensity(HPD)

Intervals–havethesuffixes–"codelow"and"codehigh"(e.g.,v2elmulpar_codelow

andv2elmulpar_codehigh).Thesetwovariablesdemarcateonestandarddeviation

upperandlowerboundsoftheintervalinwhichthemeasurementmodelplaces70

percentoftheprobabilitymassforeachcountry-yearscore.Thespreadbetween

"codelow"and"codehigh"isequivalenttoatraditionalonestandarddeviation

confidenceinterval;alargerrangeindicatesgreateruncertaintyaroundthepoint

estimate.

2.“OriginalScale”–LinearizedOriginalScalePosteriorPrediction–hasthesuffix

“_osp,”(e.g.v2elmulpar_osp).Inthisversionofthevariables,wehavelinearly

translatedthemeasurementmodelpointestimatesbacktotheoriginalordinalscale

ofeachvariable(e.g.0-4forv2elmulpar_osp)asanintervalmeasure.18Thedecimals

18Morespecifically,weusethemeasurementmodeltoestimatetheposteriordistributionaroundthepredictedprobabilitythatatypicalcoderwouldplaceacountry-yearestimateateachleveloftheoriginalcodebookscale.Wethenlinearlymapthesepredictedprobabilitydistributionsontotheoriginalscale,producingadistributionofinterval-valuedscoresontheoriginalcodebookscaleforeachcountry-year.

38

inthe_ospversionindicatethedistancebetweenthepointestimatefromthe

linearizedmeasurementmodelposteriorpredictionandthethresholdforreaching

thenextlevelontheoriginalordinalscale.Thus,a_ospvalueof1.25indicatesthat

themedianmeasurementmodelposteriorpredictedvaluewasclosertotheordinal

valueof1than2ontheoriginalscale.Sincethereisnoconventionaltheoretical

justificationforlinearlymappingordinalposteriorpredictionsontoaninterval

scale,19thesescoresshouldprimarilybeusedforheuristicpurposes.However,since

the_ospversionmapsontothecodingcriteriafoundintheV-DemCodebook,andis

stronglycorrelatedwiththeMeasurementModeloutput(typicallyat.98orhigher),

someusersmayfindthe_ospversionusefulinestimatingquantitiessuchas

marginaleffectswithaclearsubstantiveinterpretation.Usingthe“OrdinalScale”

estimates---orincorporatingthepropertiesofordinalprobitmodelsintothe

estimationprocedure---isgenerallypreferabletousingthe_ospestimatesin

statisticalanalyses.Thatsaid,ifauseruses_ospdatainstatisticalanalysesitis

imperativethatshefirstconfirmthattheresultsarecompatiblewithestimations

usingMeasurementModeloutput.

“MeasureofUncertainty”–LinearizedOriginalScaleHPDIntervals–havethe

suffixes–"codelow"and"codehigh"(e.g.,v2elmulpar_osp_codelowand

v2elmulpar_osp_codehigh).Weestimatethesequantitiesinasimilarmannerasthe

MeasurementModelHighestPosteriorDensityIntervals.Theydemarcateone

standarddeviationupperandlowerboundsoftheintervalinwhichthe

measurementmodelplaces70percentoftheprobabilitymassforeachcountry-year

score.Thespreadbetween"codelow"and"codehigh"isequivalenttoatraditional

onestandarddeviationconfidenceinterval;alargerrangeindicatesgreater

uncertaintyaroundthepointestimate.

19ThemaintheoreticalandpragmaticconcernwiththesedataisthatthetransformationdistortsthedistancebetweenpointestimatesintheMeasurementModeloutput.Forexample,thedistancebetween1.0and1.5inthe_ospdataisnotnecessarilythesameasthedistancebetweena1.5and2.0.

39

3.“OrdinalScale”-MeasurementModelEstimatesofOriginalScaleValue–hasthe

suffix"_ord"(e.g.v2elmulpar_ord).Thismethodtranslatesthemeasurementmodel

estimatesbacktotheoriginalordinalscaleofavariable(asrepresentedinthe

Codebook)aftertakingcoderdisagreementandmeasurementerrorintoaccount.

Moreprecisely,itrepresentsthemostlikelyordinalvalueontheoriginalcodebook

scaleintowhichacountry-yearwouldfall,giventheaveragecoder’susageofthat

scale.Specifically,weassigneachcountry-yearavaluethatcorrespondstoits

integerizedmedianordinalhighestposteriorprobabilitycategoryoverMeasurement

Modeloutput.

“MeasureofUncertainty”–OriginalScaleValueHPDIntervals–havethesuffixes–

"codelow"and"codehigh"(e.g.,v2elmulpar_ord_codelowand

v2elmulpar_ord_codehigh).Weestimatethesevaluesinasimilarmannerasthe

MeasurementModelHighestPosteriorDensityIntervals.Theydemarcateone

standarddeviationupperandlowerboundsoftheintervalinwhichthe

measurementmodelplaces70percentoftheprobabilitymassforeachcountry-year

score.Thespreadbetween"codelow"and"codehigh"isequivalenttoatraditional

onestandarddeviationconfidenceinterval;alargerrangeindicatesgreater

uncertaintyaroundthepointestimate.

AdditionalPossibilitiesforIdentifyingSourcesofMeasurementErrorintheFuture

A final approach to validationanalyzes various featuresof thedatagatheringprocess in

ordertogaugepossiblesourcesoferror.Thisanalysistakestheformofvariousstudiesin

which a particular issue is probed in an intensive fashion. The following studies are

underwayoronthedrawingboard–thoughwecannotsayforsurehowlongitwilltakeus

tocompletethem.

Onesuchstudywillfocusoncodertypes.Akeychallengetothevalidityisthatdata

maybesubjecttothesubjectiveperceptionsandopinionsofthechosencoders. Is itthe

casethatadifferentsetofcodersmightarriveataverydifferentsetofanswers?Features

of the coders captured in our post-survey questionnaire can be tested systematically

across the entire dataset, as noted. However,we cannot test the potential impact of a

40

differentkindofcodernot includedinourusualsample.Thisstudythereforefocuseson

comparisons across different coder types, e.g., partisans, academics, civil society

professionals,businesspeople,cosmopolitans(thosespeakingforeign languagesandwith

travel or educational experience abroad), educated lay citizens, and less educated lay

citizens.Resultsofthisstudyshouldindicate(a)howfartheconsensusoncodingextends

(i.e., to what types of coders), (b) how much difference the background of the coder

makes,(c)forwhattypesofquestions itmatters,and(d)whichsortsofcodershavethe

most positive view of a country. More generally, we hope to learn more about the

sensitivityofV-DemdatatooursamplingofCountryExperts.

Asecondstudywouldbetofocusoncountrysequencing.Doesitmatterifcoders

have considered other countries prior to coding CountryA? Such a studywould involve

randomizing respondents into two groups. Group 1 is asked to code Country A. Several

weekslater,theyareaskedtocodeahandfulofcountriesincludingCountryA,whichthey

mustre-code.Thecomparisoncasesshould includethosethatare inthesameregionas

wellasacountry(preferablyinthesameregion,orwithahistoryofcolonialinvolvement

intheregion)generallyregardedashighlydemocratic.Respondentsarenotremindedof

theiroriginal codings forCountryAandareencouraged toadjust theiroriginal coding if

they feel that amore accurate assessment is possible, in light of their consideration of

other countries. Group 2 repeats this procedure in reverse. That is, they first code a

handfulofrelatedcountriesandthenareaskedtocodeCountryA.

Athirdstudywouldbetofocusonquestionordering.TheV-Demquestionnaireis

not randomized for several reasons. First, somequestionsmust be asked in a particular

order(laterquestionsareactivatedorskippeddependingupontheanswers).Second,we

wish tomaintain a logical flowacrossquestions and tomake the flowaspredictable as

possible, so that inadvertent errors are minimized. Finally, we wish to maintain

equivalenceacrosssurveys.However,onemayalsowishtoknowwhethertheorderingof

questionson thequestionnaireaffects responses, and if sohow.Toprobe thisquestion

onewouldhavetorandomizequestionswithinasurvey(butnotacrosssurveys),without

upsettingquestionsthataredependentuponothers,andwhilemaintainingsomedegree

of logical flow. For example,wewill reverse the order of questions that are asked first

aboutmenandnextaboutwomen.

41

A fourth study could explore the quality of model-based bias adjustment. In

particular, because coders from different countries may understand both question

wordingsandconceptsindifferentways,twocodersoperatingindifferentcontextsmight

ratetwoidenticalcasesdifferentlyfromoneanother.Acommonapproachtoaddressing

thisproblemistoconstructanchoringvignettes—shorthypotheticaldepictionsofcases—

andthenaskcoderstoevaluatevignettesinadditiontorealcases,andtousedifferences

in vignette evaluations to correct for inter-personal differences in coder perceptions or

understandingsofconcepts (Kinget.al.2004;King&Wand2007;Hopkins&King2010).

Because the vignettes are fixed, these techniques assume that differences in rater

evaluationsmustrepresentdifferencesinpersonalinterpretation,andthensubtractthese

differences from responses for real cases, ostensibly correcting for respondent

incomparability. Similarly, given sufficient overlap in observed coding across raters, our

latentvariablemodelingtechniquescanusepatternsofinter-coderagreementtoidentify

and correct for systematic differences in raters' perceptions and conceptual

understandings. In otherwords, differences in howexperts rate identical cases help to

identify inter-expert variation in interpretation in much the same way that variation in

ratingsoffixedvignettesdoes.Wecanvalidatethisfeatureofthemodelbycomparingits

performance to a vignette-based approach for controlling incomparability in survey

responses.Focusingonasubsetofindicators,wewouldrecruitcountry-expertstoratean

anchoringvignette, theirowncountry,andsomecomparisoncountries. Thenwewould

apply both vignette-based and measurement-model based corrections to responses to

determineiftheyproducecomparableresults.Anexperimentalcomponentcanalsoseek

to determine if vignettes themselves alter coder behavior. In particular, we could use

patternsofagreementbetweenraterstodetermineiftreatedexperts(vignettecondition)

produce codings that are systematicallydifferent froma controlpopulation (novignette

condition).

42

References

Almond,GabrielA.,SidneyVerba.1963/1989.TheCivicCulture:PoliticalAttitudesandDemocracyinFiveNations.NewburyPark,CA:Sage.

Bernhard,Michael,EitanTzelgov,Dong-JoonJung,MichaelCoppedge,&StaffanI.Lindberg.2015.TheVarietiesofDemocracyCoreCivilSocietyIndex.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPaperSeries,No.12.

Bernhard,Michael,ChristopherReenock,andTimothyNordstrom.2004.“TheLegacyofWesternOverseasColonialismonDemocraticSurvival.”InternationalStudiesQuarterly48(3),225-250.

Bollen,KennethA.,PamelaPaxton.2000.“SubjectiveMeasuresofLiberalDemocracy.”ComparativePoliticalStudies33(1):58–86.

Capoccia,Giovanni,DanielZiblatt.2010.“TheHistoricalTurninDemocratizationStudies:ANewResearchAgendaforEuropeandBeyond.”ComparativePoliticalStudies43(8-9):931-968.

Clinton,JoshuaD.,DavidLewis.2008.“ExpertOpinion,AgencyCharacteristics,andAgencyPreferences.”PoliticalAnalysis16(1):3–20.

Clinton,JoshuaD.,JohnS.Lapinski.2006.“MeasuringLegislativeAccomplishment,1877-1994.”AmericanJournalofPoliticalScience50(1):232–249.

Collier,DavidandJamesMahon(1993).“Conceptual‘Stretching’Revisited:AdaptingCategoriesinComparativeAnalysis.”AmericanPoliticalScienceReview87(4):845-855.

Coppedge,Michael,StaffanLindberg,Svend-ErikSkaaning,andJanTeorell.2015.MeasuringHighLevelDemocraticPrinciplesusingtheV-DemData.UniversityofGothenburg,TheVarietiesofDemocracyInstitute:V-DemWorkingPaperseriesNo.6

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,DavidAltman,MichaelBernhard,M.StevenFish,AdamGlynn,AllenHicken,CarlHenrikKnutsen,KyleMarquardt,KellyMcMann,FarhadMiri,PamelaPaxton,DanielPemstein,JeffreyStaton,EitanTzelgov,Yi-tingWang,andBrigitteZimmerman.2015.V-Dem[Country-Year/Country-Date]Datasetv5.VarietiesofDemocracy(V-Dem)Project.

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,withDavidAltman,MichaelBernhard,M.StevenFish,AdamGlynn,AllenHicken,CarlHenrikKnutsen,KellyMcMann,PamelaPaxton,DanielPemstein,JeffreyStaton,BrigitteZimmerman,FridaAndersson,ValeriyaMechkova,andFarhadMiri.2015.V-DemCodebookv5.VarietiesofDemocracy(V-Dem)Project.

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,FridaAndersson,KyleMarquardt,ValeriyaMechkova,FarhadMiri,DanielPemstein,JosefinePernes,NataliaStepanova,EitanTzelgov,andYi-tingWang.2015.V-DemMethodologyv5.VarietiesofDemocracy(V-Dem)Project.

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,andVladCiobanu.2015.V-DemCountryCodingUnitsv5.VarietiesofDemocracy(V-Dem)Project.

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,JanTeorell,FridaAndersson,ValeriyaMechkova,JosefinePernes,andNataliaStepanova.2015.V-DemOrganizationandManagementv5.VarietiesofDemocracy(V-Dem)Project.

43

Coppedge,Michael,JohnGerring,StaffanI.Lindberg,Svend-ErikSkaaning,andJanTeorell.2015.V-DemComparisonsandContrastswithOtherMeasurementProjects.VarietiesofDemocracy(V-Dem)Project.

Dahl,RobertA.1971.Polyarchy:ParticipationandOpposition.NewHaven:YaleUniversityPress.

Dahl,RobertA.1989.DemocracyanditsCritics.NewHaven:YaleUniversityPress.Epstein,DavidL.;RobertBates;JackGoldstone;IdaKristensen;SharynO’Halloran.2006.

“DemocraticTransitions.”AmericanJournalofPoliticalScience50(3):551-569.Fox,Jean-Paul.2010.BayesianItemResponseModeling:TheoryandApplications.New

York:Springer.Gallie,W.B.1956.“EssentiallyContestedConcepts.”ProceedingsoftheAristotelian

Society56:167–220.Gerring,John,PhilipBond,WilliamBarndt,andCarolaMoreno.2005.“Democracyand

Growth:AHistoricalPerspective.”WorldPolitics57(3):323–364.Goertz,Gary.2006.SocialScienceConcepts:AUser’sGuide.Princeton:Princeton

UniversityPress.Hadenius,AxelandJanTeorell.2005.“CulturalandEconomicPrerequisitesofDemocracy:

ReassessingRecentEvidence.”StudiesinComparativeInternationalDevelopment39(4):87–106.

Held,David.2006.ModelsofDemocracy,3ded.Cambridge:PolityPress.Hopkins,Daniel,andGaryKing.2010.“ImprovingAnchoringVignettes:DesigningSurveys

toCorrectInterpersonalIncomparability.”PublicOpinionQuarterly:1-22.Inglehart,RonaldandWelzel,Christian.2005.Modernization,CulturalChangeand

Democracy:TheHumanDevelopmentSequence.Cambridge:CambridgeUniversityPress.

Isaac,JeffreyC.n.d.“ThinkingAbouttheQualityofDemocracyanditsPromotion.”Unpublishedms.

Jackman,Simon.2004.“WhatDoWeLearnfromGraduateAdmissionsCommittees?AMultipleRater,LatentVariableModel,withIncompleteDiscreteandContinuousIndicators.”PoliticalAnalysis12(4):400–424.

Johnson,ValenE.,JamesH.Albert.1999.OrdinalDataModeling.NewYork:Springer.Junker,Brian1999.SomeStatisticalModelsandComputationalMethodsthatmaybe

UsefulforCognitively-RelevantAssessment.http://www.stat.cmu.edu/~brian/nrc/cfa/documents/final.pdf

King,Gary,ChristopherMurray,JoshuaA.Salomon,andAjayTandon.2004.“EnhancingtheValidityandCross-culturalComparabilityofMeasurementinSurveyResearch.”AmericanPoliticalScienceReview98(1):191–207.

King,Gary,andJonathanWand.2007.ComparingIncomparableSurveyResponses:NewToolsforAnchoringVignettes.PoliticalAnalysis15:46-66.

Knutsen,CarlHenrik.2010.“MeasuringEffectiveDemocracy.”InternationalPoliticalScienceReview31(2):109-128.

44

Knutsen,CarlHenrik,JørgenMøller,andSvend-ErikSkaaning.Forthcoming.GoingHistorical:MeasuringDemocraticnessbeforetheAgeofMassSuffrage.InternationalPoliticalScienceReview.

Lindberg,StaffanI.2015.OrdinalVersionsofV-Dem’sIndices:ForClassification,Description,SequencingAnalysisandOtherPurposes.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.19.

Lord,FredericM.,andMelvinNovick.1968.StatisticalTheoriesofMentalTestScores.Reading,MA:Addison-Wesley.

Mahoney,James,andDietrichRueschemeyer,eds.2003.ComparativeHistoricalAnalysisintheSocialSciences.Cambridge:CambridgeUniversityPress.

Mariano,LouisT.andBrianW.Junker.2007.“CovariatesoftheRatingProcessinHierarchicalModelsforMultipleRatingsofTestItems.”JournaloftheEducationalandBehavioralStatistics32(2):287-314.

Munck,GerardoL.2009.MeasuringDemocracy:ABridgebetweenScholarshipandPolitics.Baltimore:JohnHopkinsUniversityPress.

Munck,GerardoL.2016.“WhatisDemocracy?AReconceptualizationoftheQualityofDemocracy.”Democratization23(1):1-26.

Nunn,Nathan.2009.“TheImportanceofHistoryforEconomicDevelopment.”AnnualReviewofEconomics1(1):1–28.

Patz,RichardJ.,andBrianW.Junker.1999.“AStraightforwardApproachtoMarkovChainMonteCarloMethodsforItemResponseModels.”JournalofEducationalandBehavioralStatistics24:146-178.

Patz,RichardJ.,BrianW.Junker,MatthewS.Johnson,andLouisT.Mariano.2002.“TheHierarchicalRaterModelforRatedTestItemsanditsApplicationtoLarge-ScaleEducationalAssessmentData.”JournalofEducationalandBehavioralStatistics27(4):341-384.

Pemstein,Dan,KyleL.Marquardt,EitanTzelgov,Yi-tingWang,andFarhadMiri.2015.“TheV-DemMeasurementModel:LatentVariableAnalysisforCross-NationalandCross-TemporalExpert-CodedData.”UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.20

Pemstein,Daniel,StephenMeserve,andJamesMelton.2010.“DemocraticCompromise:ALatentVariableAnalysisofTenMeasuresofRegimeType.”PoliticalAnalysis18(4):426–449.

Pierson,Paul.2004.PoliticsinTime:History,Institutions,andSocialAnalysis.Princeton:PrincetonUniversityPress.

Rose-Ackerman,Susan.1999.CorruptionandGovernment:Causes,Consequences,andReform.Cambridge:CambridgeUniversityPress.

Sartori,Giovanni.1970.“ConceptMisformationinComparativePolitics.”AmericanPoliticalScienceReview64(4):1033-1053.

Schedler,Andreas.2012.“JudgmentandMeasurementinPoliticalScience.”PerspectivesonPolitics10:1,21-36.

Shapiro,Ian.2003.TheStateofDemocraticTheory.Princeton:PrincetonUniversityPress.Sigman,RachelandStaffanI.Lindberg.2015.TheIndexofEgalitarianDemocracyandIts

Components:V-Dem'sConceptualizationandMeasurement.UniversityofGothenburg,

45

VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.21Steinmo,Sven,KathleenThelen,andFrankLongstreth,eds.1992.StructuringPolitics:

HistoricalInstitutionalisminComparativeAnalysis.Cambridge:CambridgeUniversityPress.

Teorell,Jan.2011.“OverTime,AcrossSpace:ReflectionsontheProductionandUsageofDemocracyandGovernanceData.”ComparativeDemocratization9:1(February)1,7.

Teorell,Jan,MichaelCoppedge,JohnGerring&StaffanLindberg.n.d.2016"MeasuringElectoralDemocracywithV-DemData:IntroducingaNewPolyarchyIndex.”UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.23

Teorell,Jan,RachelSigman,andStaffanI.Lindbergn.d.2016.V-DemIndices:RationaleandAggregations.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.22

Teorell,JanandStaffanI.Lindberg.2015.TheStructureoftheExecutiveinAuthoritarianandDemocraticRegimes:RegimeDimensionsacrosstheGlobe,1900-2014.UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.5

Thomas,MelissaA.2010.“WhatDotheWorldwideGovernanceIndicatorsMeasure?”EuropeanJournalofDevelopmentResearch22(1):31–54.

Treier,Shawn,andSimonJackman.2008.“DemocracyasaLatentVariable.”AmericanJournalofPoliticalScience52(1):201–217.

Wang,Yi-ting,PatrikLindenfors,AkselSundström,FredrikJansson,andStaffanI.Lindberg.2015.NoDemocraticTransitionWithoutWomen’sRights:AGlobalSequenceAnalysis1900-2012.VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.12.

Wang,Yi-ting,AkselSundström,PamelaPaxton,andStaffanI.Lindberg.2015.”Women’sPoliticalEmpowermentIndex:ANewMeasure.”UniversityofGothenburg,VarietiesofDemocracyInstitute:V-DemWorkingPapersSeriesNo.18.

Welzel,Christian.2007.“AreLevelsofDemocracyAffectedbyMassAttitudes?TestingAttainmentandSustainmentEffectsonDemocracy.”InternationalPoliticalScienceReview28(4):397–424.

46

APPENDIXA:V-DemIndices,Components,andIndicators

DemocracyIndices

Names

Mid-LevelDemocracy

andGovernance

IndicesNames

Lower-Level

Democracyand

GovernanceIndices

Names NamesIndicators

v2_tagIndicesand

Indicators

ElectoralDemocracyIndex

v2x_polyarchy

Expandedfreedomofexpressionindex

v2x_freexp_thick

Governmentcensorshipeffort-Media

v2mecenefm

Governmentcensorshipeffort-Internet

v2mecenefi

Harassmentofjournalists v2meharjrn Mediaself-censorship v2meslfcen Mediabias v2mebias Print/broadcastmediacritical v2mecrit Print/broadcastmedia

perspectives v2merange Freedomofdiscussionformen v2cldiscm Freedomofdiscussionfor

womenv2cldiscw

Freedomofacademicandculturalexpression

v2clacfree

Alternativesourceinformationindex

v2xme_altinf

Mediabias v2mebias Print/broadcastmediacritical v2mecrit Print/broadcastmedia

perspectivesv2merange

Freedomofassociationindex(thick)

v2x_frassoc_thick

PartyBan v2psparban Barrierstoparties v2psbars Oppositionpartiesautonomy v2psoppaut Electionsmultiparty v2elmulpar CSOentryandexit v2cseeorgs CSOrepression v2csreprss Shareofpopulation

withsuffrage v2x_suffr

Percentofpopulationwithsuffrage

v2elsuffrage

Cleanelectionsindex v2xel_frefair EMBautonomy v2elembaut EMBcapacity v2elembcap Electionvoterregistry v2elrgstry Electionvotebuying v2elvotbuy

47

Electionothervotingirregularities

v2elirreg

Electiongovernmentintimidation

v2elintim

Electionotherelectoralviolence v2elpeace Electionfreeandfair v2elfrfair Electedexecutive

index(dejure) v2x_accex

Lowerchamberelected v2lgello Upperchamberelected v2lgelecup Legislaturedominantchamber v2lgdomchm HOSselectionbylegislaturein

practicev2exaphos

HOSappointmentinpractice v2expathhs HOGselectionbylegislaturein

practicev2exaphogp

HOGappointmentinpractice v2expathhg HOSappointscabinetinpractice v2exdfcbhs HOGappointscabinetin

practicev2exdjcbhg

HOSdismissesministersinpractice

v2exdfdmhs

HOGdismissesministersinpractice

v2exdfdshg

HOSappointscabinetinpractice v2exdfcbhsLiberalDemocracyIndex

v2x_libdem

ElectoralDemocracyIndex

v2x_polyarchy

LiberalComponentIndex

v2x_liberal

Equalitybeforethelawandindividuallibertyindex

v2xcl_rol

Rigorousandimpartialpublicadministration

v2clrspct

Transparentlawswithpredictableenforcement

v2cltrnslw

Accesstojusticeformen v2clacjstm Accesstojusticeforwomen v2clacjstw Propertyrightsformen v2clprptym Propertyrightsforwomen v2clprptyw Freedomfromtorture v2cltort Freedomfrompoliticalkillings v2clkill Freedomfromforcedlaborfor

menv2clslavem

Freedomfromforcedlaborforwomen

v2clslavef

Freedomofreligion v2clrelig Freedomofforeignmovement v2clfmove

48

Freedomofdomesticmovementformen

v2cldmovem

Freedomofdomesticmovementforwomen

v2cldmovew

Judicialconstraintsontheexecutiveindex

v2x_jucon

Executiverespectsconstitution v2exrescon Compliancewithjudiciary v2jucomp Compliancewithhighcourt v2juhccomp Highcourtindependence v2juhcind Lowercourtindependence v2juncind Legislative

constraintsontheexecutiveindex

v2xlg_legcon

Legislaturequestionsofficialsinpractice

v2lgqstexp

Executiveoversight v2lgotovst Legislatureinvestigatesin

practicev2lginvstp

Legislatureoppositionparties v2lgoppartDeliberativeDemocracyIndex

v2x_delibdem

ElectoralDemocracyIndex

v2x_polyarchy

DeliberativeComponentIndex

v2xdl_delib

Reasonedjustification v2dlreason Commongood v2dlcommon Respectcounterarguments v2dlcountr Rangeofconsultation v2dlconslt Engagedsociety v2dlengageEgalitariandemocracyIndex

v2x_egaldem

ElectoralDemocracyIndex

v2x_polyarchy

EgalitarianComponentIndex

v2x_egal

Equalprotectionindex

v2xeg_eqprotec

Accesstojusticeformen v2clacjstm Accesstojusticeforwomen v2clacjstw Socialclassequalityinrespect

forcivillibertiesv2clacjust

Socialgroupequalityinrespectforcivilliberties v2clsocgrp

Weakercivillibertiespopulationv2clsnlpct

Equaldistributionofresourcesindex

v2xeg_eqdr

49

Powerdistributedbysocioeconomicposition

v2pepwrses

Powerdistributedbysocialgroup

v2pepwrsoc

Educationalequality v2peedueq Healthequality v2pehealth Powerdistributedbygender v2pepwrgen Encompassing-ness v2dlencmps Means-testedvs.universalistic v2dlunivlParticipatoryDemocracyIndex

v2x_partipdem

ElectoralDemocracyIndex

v2x_polyarchy

ParticipatoryComponentIndex

v2x_partip

Civilsocietyparticipationindex

v2x_cspart

Candidateselection--National/local

v2pscnslnl

CSOconsultation v2cscnsult CSOparticipatoryenvironment v2csprtcpt CSOwomensparticipation v2csgender DirectPopularVote

Index v2xdd_dd

Initiativespermitted v2ddlegci

Initiativessignatures% v2ddsigcip

Initiativessignature-gatheringtimelimit

v2ddgrtlci

Initiativessignature-gatheringperiod

v2ddgrgpci

Initiativeslevel v2ddlevci

Initiativesparticipationthreshold

v2ddbindci

Initiativesapprovalthreshold v2ddthreci

Initiativesadministrativethreshold

v2dddistci

Initiativessupermajority v2ddspmjci

Occurrenceofcitizen-initiativethisyear

v2ddciniyr

Localgovernmentindex

v2xel_locelec

Localgovernmentelected v2ellocelc Localofficesrelativepower v2ellocpwr Localgovernmentexists v2ellocgov Regional

governmentindex v2xel_regelec

Regionalgovernmentelected v2elsrgel Regionalofficesrelativepower v2elrgpwr Regionalgovernmentexists v2elreggov

50

CoreCivilSocietyIndex

v2xcs_ccsi

CSOentryandexit v2cseeorgs CSOrepression v2csreprss CSOparticipatoryenvironment v2csprtcpt Party

Institutionalizationindex

v2xps_party

Partyorganizations v2psorgs PartyBranches v2psprbrch Partylinkages v2psprlnks Distinctpartyplatforms v2psplats Legislativepartycohesion v2pscohesv Womenpolitical

empowermentindex v2x_gender

Womencivillibertiesindex

v2x_gencl

Freedomofdomesticmovementforwomen

v2cldmovew

Freedomfromforcedlaborforwomen

v2clslavef

Propertyrightsforwomen v2clprptyw Accesstojusticeforwomen v2clacjstw Womencivilsociety

participationindex

v2x_gencs Freedomofdiscussionfor

womenv2cldiscw

CSOwomensparticipation v2csgender Percent(%)FemaleJournalists v2mefemjrn Womenpolitical

participationindex

v2x_genpp Powerdistributedbygender v2pepwrgen Lowerchamberfemale

legislatorsv2lgfemleg

ElectoralRegimeIndex

v2x_elecreg

Legislativeorconstituentassemblyelection

v2xel_elecparl

v2eltype v2eltype_0 v2eltype v2eltype_1 v2eltype v2eltype_4 v2eltype v2eltype_5 Legislatureclosed

downoraborted v2xlg_leginter

Legislaturebicameral v2lgbicam Presidential

election v2xel_elecpres

v2eltype v2eltype_6 v2eltype v2eltype_7

51

Chiefexecutivenolongerelected

v2x_hosinter

HOS=HOG? v2exhoshog HOGappointmentinpractice v2expathhg HOSappointmentinpractice v2expathhs Corruptionindex v2x_corr Legislaturecorruptactivities v2lgcrrpt Judicialcorruptiondecision v2jucorrdc Publicsector

corruptionindex v2x_pubcorr

Publicsectorcorruptexchanges v2excrptps Publicsectortheft v2exthftps Executive

corruptionindex v2x_execorr

Executivebriberyandcorruptexchanges

v2exbribe

Executiveembezzlementandtheft

v2exembez

ElectoralComponentIndex

v2x_EDcomp_thick

Freedomofassociationindex(thick)

v2x_frassoc_thick

PartyBan v2psparban Barrierstoparties v2psbars Oppositionpartiesautonomy v2psoppaut Electionsmultiparty v2elmulpar CSOentryandexit v2cseeorgs CSOrepression v2csreprss Shareofpopulation

withsuffrage v2x_suffr

Percentofpopulationwithsuffrage

v2elsuffrage

Cleanelectionsindex

v2xel_frefair

EMBautonomy v2elembaut EMBcapacity v2elembcap Electionvoterregistry v2elrgstry Electionvotebuying v2elvotbuy Electionothervoting

irregularitiesv2elirreg

Electiongovernmentintimidation

v2elintim

Electionotherelectoralviolence v2elpeace Electionfreeandfair v2elfrfair Electedexecutive

index(dejure) v2x_accex

Lowerchamberelected v2lgello Upperchamberelected v2lgelecup

52

Legislaturedominantchamber v2lgdomchm HOSselectionbylegislaturein

practicev2exaphos

HOSappointmentinpractice v2expathhs HOGselectionbylegislaturein

practicev2exaphogp

HOGappointmentinpractice v2expathhg HOSappointscabinetinpractice v2exdfcbhs HOGappointscabinetin

practicev2exdjcbhg

HOSdismissesministersinpractice

v2exdfdmhs

HOGdismissesministersinpractice

v2exdfdshg

HOSappointscabinetinpractice v2exdfcbhs Freedomof

expressionindex v2x_freexp

Governmentcensorshipeffort-Media

v2mecenefm

Harassmentofjournalists v2meharjrn Mediaself-censorship v2meslfcen Freedomofdiscussionformen v2cldiscm Freedomofdiscussionfor

womenv2cldiscw

Freedomofacademicandculturalexpression

v2clacfree