Political Ideology and Polarization of Policy Positions:A Multi-dimensional Approach

Barea Sinno1∗ Bernardo Oviedo2∗ Katherine Atwell3∗Malihe Alikhani3 Junyi Jessy Li41 Political Science, Rutgers University

2 Computer Science, 4 Linguistics, The University of Texas at Austin3 Computer Science, University of Pittsburgh

barea.sinno@gmail.com, bernyoviedo@utexas.edu, kaa139@pitt.edu

malihe@pitt.edu, jessy@austin.utexas.edu

Abstract

Analyzing political ideology and polariza-tion is of critical importance in advancingour understanding of the political contextin society. Recent research has made greatstrides towards understanding the ideologi-cal bias (i.e., stance) of news media alonga left-right spectrum. In this work, we takea novel approach and study the ideology ofthe policy under discussion teasing apart thenuanced co-existence of stance and ideol-ogy. Aligned with the theoretical accountsin political science, we treat ideology as amulti-dimensional construct, and introducethe first diachronic dataset of news articleswhose political ideology under discussion isannotated by trained political scientists andlinguists at the paragraph-level. We show-case that this framework enables quantitativeanalysis of polarization, a temporal, multi-faceted measure of ideological distance. Wefurther present baseline models for ideologyprediction.

1 Introduction

Polarization can present a problem. It can congestessential democratic functions with an increase inthe divergence of political ideologies. Defined as agrowing ideological distance between groups, po-larization has waxed and waned since the adventof the American Republic (Pierson and Schickler,2020).1 Two eras—post-1896 and -1990s—havewitnessed deleterious degrees of polarization (Jenk-ins et al., 2004; Jensen et al., 2012). More recently,COVID-19, the murder of George Floyd, and theCapitol riots have exposed ideological divergences

* Equal contribution ordered by first name.1We distinguish ourselves from work that considers other

types of polarization, e.g., as a measure of emotional dis-tance (Iyengar et al., 2019) or distance between political par-ties (Lauka et al., 2018).

Twodimensions:trade andeconomicliberalism

The U.S. aim is to create a monetary sys-tem with enough flexibility to prevent bar-gain-hungry money from rolling aroundthe world like loose ballast on a ship dis-rupting normal trade and currency flows.Nixon goals: dollar, trade stability. Thismust be accompanied, Washington says,by reduction of the barriers ...

Onedimension:tradeprotectionism

The controls program, which Mr. Nixoninaugurated Aug. 15, 1971, has helpedto reduce inflation to about 3 percentyearly, and to boost annual U.S. eco-nomic growth to more than 7 percent...

Table 1: Excerpts from article #730567. The firstparagraph advocates for liberalism and the reduc-tion of trade barriers. It also has a domestic eco-nomic dimension. The second paragraph, on thecontrary, advocates for protectionism and a domes-tic controls program.

in opinion in the US through language on newsmedia and social media.

Political ideology rests on a set of beliefs aboutthe proper order of a society and ways to achievethis order (Jost et al., 2009; Adorno et al., 2019;Campbell et al., 1980). In Western politics, theseworldviews translate into a multi-dimensional con-struct that includes: equal opportunity as opposedto economic individualism; general respect for tra-dition, hierarchy and stability as opposed to ad-vocating for social change; and a belief in theun/fairness and in/efficiency of markets (Jost et al.,2009). Propaganda and misinformation effective-ness rely on ideology and thus polarization (Vi-cario et al., 2019; Bessi et al., 2016). With thehope of taking steps towards combating unwantedinfluence and misinformation, we study this socialphenomenon through the lens of computational lin-guistics by presenting a carefully annotated corpusand examining the efficacy of a set of computa-tional and statistical analyses.

arX

iv:2

106.

1438

7v1

[cs

.CL

] 2

8 Ju

n 20

21

We take a novel approach and instead of study-ing the bias or the stance of the author of the text(Kulkarni et al., 2018; Kiesel et al., 2019; Balyet al., 2019, 2020; Chen et al., 2020; Stefanov et al.,2020), we opt to study the ideology of the discussedpolicy or entity under discussion. That is, in lieu ofexamining the author’s stance, we focus on address-ing the at-issue content of the text and the ideologythat it represents in the implicit social context. Thenuanced co-existence of stance and ideology canbe illustrated in the following excerpt from an ar-ticle in the Federalist authored by Hadley HeathManning on September 27, 2019:

“Republicans and Joe Biden are making a hugemistake by focusing on cost. The implicationis that government-run health care would bea good thing–a wonderful thing!– if only wecould afford it.”

The author is attacking a liberal social and eco-nomic policy; therefore, the ideology being dis-cussed is liberal on two dimensions - social andeconomic, while the author’s slant is conservative.

Moreover, our novel approach acknowledgesthat ideology can also vary within one article. InTable 1, we show an example in which one partof an article advocates for trade liberalism, whileanother advocates for protectionism.

Together, author stance and ideology inform usnot only that there is bias in the media, but alsowhich beliefs are being supported and/or attacked.A full analysis of polarization (that reflects a grow-ing distance of political ideology over time) canthen be derived if diachronic data for both slant andideology were available. However, while there hasbeen data on media bias (with articles from recentyears only) (Kiesel et al., 2019), to date, there hasbeen no temporal data on the ideology of policypositions.

In this paper, we present a multi-dimensionalframework, and an annotated, diachronic corpus,for the analysis of ideology in text, and study po-larization as a state of ideological groups with di-vergent positions on a political issue as well aspolarization as a process whose magnitude growsover time (DiMaggio et al., 1996). We use center-left and center-right media outlets who claim to beobjective in order to focus exclusively and more ob-jectively on the ideology being discussed, withoutthe subjectivity of annotating author stance. Westudy ideology within every paragraph of an article

and aim to answer the following question2: whichideological dimension is present and to which def-inition does this ideology correspond to on theliberal-conservative spectrum.

Our extensive annotation manual is developed bya political scientist, and the data then annotated bythree linguists after an elaborate training phase. Af-ter 150 hours of annotation, we present a dataset of721 fully adjudicated annotated paragraphs, from175 news articles and covering an average of 7.86articles per year (excerpts shown in Tables 1, 3,and 4). These articles originate from 5 news outletsrelated the US Federal Budget from 1947-1975 cov-ering the center-left, center, center-right spectrum:Chicago Tribune (CT), Christian Science Monitor(CSM), the New York Times (NYT), Time Maga-zine (TM), and the Wall Street Journal (WSJ).

With this data, we reveal lexical insights on thelanguage of ideology across the left-right spectrumand across dimensions. We observe that linguis-tic use even at word level can reveal the ideologybehind liberal and conservative policies. Our frame-work also enables fine-grained, quantitative analy-sis of polarization, which we demonstrate in Sec-tion 5. This type of analysis, if scaled up usingaccurate models for ideology prediction, has thepotential to reveal impactful insights not only forpolitical science research but also for a deeper un-derstanding of the political context of our societyas a whole.

Finally, we present baseline models for the au-tomatic identification of multi-dimensional ideol-ogy at the paragraph level. We show that this isa challenging task with our best baseline yieldingan F measure of 0.55; exploring pre-training withexisting data in news ideology/bias identification,we found that this task is different in nature from,although correlated with, labels automatically de-rived from news outlets. We will contribute ourdata and code to the community upon publication.

2 Task

Many political scientists and political psycholo-gists argue for the use of at least a bidimensionalideology for domestic politics that distinguishesbetween economic and social preferences (Carseyand Layman, 2006; Carmines et al., 2012; Feldmanand Johnston, 2014).3 In line with this literature,

2We use automatically segmented paragraphs since theraw texts were not paragraph-segmented.

3It is important to distinguish between ideology and parti-sanship (party identity) (Campbell et al., 1980). Partisanship

we start with these two dimensions; we add a thirddimension “Foreign” when the article tackles for-eign issues.

Specifically, our annotation task entails exam-ining a news article and annotating each dimen-sion (detailed below) along three different levels—liberal, conservative, neutral—at the paragraphlevel. The neutral level for every dimension isreserved for paragraphs related to a specific dimen-sion but either (a) contain both conservative andliberal elements that annotators were unable to as-certain an ideological dimension with confidence,or (b) do not portray any ideology. We additionallyprovide an irrelevant option if a dimension doesnot apply to the paragraph. The three dimensionsare:

Social Dimension: While the (1) socially con-servative aspect of this dimension is defined asrespect for tradition, fear of threat and uncertainty,need for order and structure, concerns for personaland national security, and preference for confor-mity, its (2) socially liberal counterpart has beenassociated with a belief in the separation of churchand state, tolerance for uncertainty and change (Jostet al., 2009).

Economic Dimension: Similarly, while the (3)economically conservative aspect of this dimen-sion refers to motivations to achieve social rewards,power, and prestige such as deregulation of theeconomy, lower taxes and privatization (i.e., be-ing against deficit) spending and advocating for abalanced budget, its (4) economically liberal coun-terpart refers to motivation for social justice andequality such as issues related to higher taxes onrich individuals and businesses and more redistri-bution.

Foreign Dimension: After piloting the bi-dimensional approach on 300 articles, we find thata bi-dimensional conception conflates two impor-tant aspects of ideology related to domestic econ-omy and foreign trade. Tariffs, import quotas,and other nontariff-based barriers to trade that areaimed at improving employment and the compet-itiveness of the United States on the internationalmarket did not map well onto the bidimensional

refers to an affiliation with a party and its political positions.A partisan person changes their ideology when their partychanges its ideology, whereas an ideological person changestheir party when their party changes its ideology. Partisanshipis easily conflated with party ID when using a unidimensionalconceptualization of ideology, but not when using a multi-dimensional one.

framework. After consulting several senior polit-ical scientists, we adopted a third dimension forideology that dealt with the markets as well as therelations of the United States with the rest of theworld.

While the (5) globalist counterpart of this di-mension accounts for free-trade, diplomacy, immi-gration and treaties such as the non-proliferationof arms, its (6) interventionist aspect is nationalistin its support for excise tax on imports to protectAmerican jobs and economic subsidies and anti-immigration.

With the annotated data, we demonstrate quan-titative measures of polarization (Section 5) andintroduce the modeling task (Section 6) of auto-matically identifying the ideology of the policypositions being discussed: given a paragraph, foreach dimension, we classify its ideology as liberal,conservative, or neutral.

3 Data collection and annotation

In this section, we describe the details of our datacreation, preparation, and annotation.

3.1 Curation of raw texts

Since polarization is a process (DiMaggio et al.,1996), a diachronic corpus is required to measureand analyze polarization over time. We collect andannotate data across a long period to address theissue of distributional shifts across years (Desaiet al., 2019; Rijhwani and Preotiuc-Pietro, 2020;Bender et al., 2021) and help build robust modelsthat can generalize beyond certain periods.

Additionally, the raw data on top of which weannotate needs to satisfy the following constraints:(1) for human annotation to be tractable, the articlesshould cover content that demonstrate some levelof coherence in terms of topic; (2) for the datato be useful for the larger community, the contentshould also cover a range of common discussions inpolitics across the aisle; and (3) the articles shouldcome from a consistent set of news outlets, forminga continuous and ideologically balanced corpus.

We start with the diachronic corpus of politicalnews articles of Desai et al. (2019) which coversyears 1922-1986, the longest such period to ourknowledge. This corpus is a subset of news articlesfrom the Corpus of Historical American English(COHA, Davies (2012)). To extract topically co-herent articles, we run LDA (Blei et al., 2003) overthis corpus. We then investigate the topics and arti-

Topic1:Trade

bank, market, farm, loan, export, agricul-tur, farmer, dollar, food, debt

Topic2:Business

incom, tax, revenu, profit, corpor, financ,treasuri, pay, sale, bond

Topic3:Education

school, univers, educ, student, colleg, pro-fessor, institut, teacher, research, graduat

Topic4:Defense

nuclear, missil, weapon, atom, test, energi,strateg, bomb, space, pentagon

Topic5:Economy

budget, billion, economi, inflat, economic,deficit, unemploy, cut, dollar, rate

Topic6:Health/Race

negro, hospit, medic, health, racial, south-ern, discrimin, doctor, contra, black

Topic7:Industry

compani, contract, plant, steel, coal, wage,railroad, corpor, manufactur, miner

Table 2: Top words from topics selected in ourcluster, from the 50-topic LDA model that yieldedthe most well-deliminated topics.

cles across multiple LDA runs varying the numberof topics (15, 20, 30, 50), aiming to arrive at acluster of topics that share common points of dis-cussion and collectively will yield a sizable numberof articles each year from the same news outlets.

The LDA models consistently showed oneprominent topic—the federal budget—across 5news outlets with balanced ideology: ChicagoTribune (CT), Wall Street Journal (WSJ), Chris-tian Science Monitor (CSM), the New York Times(NYT), and Time Magazine (TM). Because federalbudget stories touch on all aspects of the federalactivity, this topic appeals to both liberal and con-servative media and thus can provide a good testingground to showcase our proposed ideological an-notation method. In addition to the core federalbudget topic (topic 5 of Table 2), we also includeother topics such as health and education that areintegral parts of ideological beliefs in the UnitedStates, and when discussed at the federal govern-ment level, are typically related to the federal bud-get. The top vocabulary of the cluster is shownin Table 2. In an effort to purge articles unrelatedto the federal budget, we selected only those thatcontain words such as “federal” and “congress”,and excluded those that mention state budget, andletters to editors. (Note that when we annotate ide-ology, we also discard articles that are unrelated tothe federal budget.) After this curation, the totalnumber of articles is 5,706 from the 5 outlets.

To account for the sparsity of articles in the firstdecades and their density in later decades, we nar-rowed down the articles to the period from 1947to 1974. We believe this period is fitting because

it includes various ideological combinations of thetripartite composition of the American government,Congress and presidency.4 The total number ofarticles in the final corpus of political articles onthe federal budget from 1947 to 1974 is 1,749.

Paragraph segmentation. The raw texts fromDesai et al. (2019) originate from COHA and mostwere not segmented into paragraphs. We usedTopic Tiling (Riedl and Biemann, 2012) for au-tomatic segmentation. Topic Tiling finds segmentboundaries again using LDA and, thus, identifiesmajor subtopic changes within the same article.The segmentation resulted in articles with 1 to 6paragraphs. The average number of paragraphs perarticle was 4.

3.2 AnnotationProtocol A political science graduate studentdeveloped an elaborate annotation protocol afterstudying over 350 randomly sampled paragraphs.The protocol provides a step-by-step progressionof the annotation task that explains the dimensionsto lay readers, includes an extensive list of fre-quently asked questions and lists specially confus-ing annotation cases she encountered during thepre-annotation process. The annotation process us-ing this protocol was independently reviewed byfour political science professors from two univer-sities in the US and are not authors of this paper;the research area for two of them is ideology andpolarization in the US.

Process We sampled on average 7.86 articles peryear for annotation, for a total of 721 paragraphsacross 175 articles. We divided the annotation taskinto two batches of 45 and 130 articles, the smallerbatch was for training purposes.

We recruited three additional annotators, all ofwhom have recently graduated from a linguists de-partment from a US university. The training ses-sions consisted of one general meeting (all anno-tators met) and six different one-on-one meetings(each annotator met with another annotator once).During initial training, the annotators were asked tohighlight sentences based on which the annotationwas performed. We analyze these rationales, whichis present in 25 articles, in Section 4.3.

After the annotations of this batch were final-ized, the annotators met with the political science

4For example, between 1947-49, Congress was Republi-can and the President was a Democrat while the story flippedbetween 1955-57.

Twodimensions:socially andeconomicallyliberal

... Secretary of Defense Robert S. threwhis full support today behind the Admin-istration’s drive against poverty. Citingfigures showing that, about a third of thenation’s youths fail either mental or phys-ical examinations given by, the SelectiveService, Mr. Mc-Namara said : “It is theyouth that we can expect to be the mostimmediate beneficiaries of the war onpoverty.” He said he was endorsing the“entire program” both as a citizen and asa member of the Cabinet. His endorse-ment came as his fellow Republicans inCongress continued to hammer away atparts of the Administration’s antipovertyprogram. . . .

Twodimensions:socially andeconomicallyconservative

The antipoverty program, the Republi-cans insisted, would undercut the author-ity of the Cabinet members by makingSargent Shriver a ”poverty czar.” “I don’tsee how you can lie down and be a door-mat for this kind of operation, ”Represen-tative Peter Frelinghuysen Jr., Republicanof New Jersey, commented. ...

Table 3: Excerpts from article #723847. Becausethe first paragraph calls for minimizing incomeinequality, it is socially liberal and because advo-cating for such a program call for an budgetaryexpenditure, it is also has a economic liberal di-mension. The second paragraph advocates for theexact opposites of the positions in the first para-graph. Therefore, it is socially and economicallyconservative. Sentences most relevant to these la-bels are highlighted.

student to create ground truth labels in cases ofdisagreement. Then, the three annotators receivedthe second batch and each article was annotatedby 2 annotators. This annotation was composedof two stages to account for possible subjectivity.In stage 1, each annotator worked on a batch thatoverlapped with only one other annotator. In stage2, the two annotators examine paragraphs that theydo not agree with, and met with the third annotatoracting as consolidator to adjudicate. Tables 1, 3and 4 are examples of adjudicated annotation in thedata.

Agreement To assess the inter-annotator agree-ment of stage 1, we report Krippendorf’s α (Hayesand Krippendorff, 2007) for each dimension for the135 articles after training and before any discus-sion/adjudication: economic (0.44), social (0.39),foreign (0.68). The agreements among annotatorsthe for economic and foreign dimensions are mod-

Zerodimension

. . . “The committee is holding public hear-ings on President Eisenhower’s EconomicReport, which he sent to Congress last week.The Secretary’s [Humphrey] appearance be-fore the group provided an opportunity forpolitical exchanges.

Onedimension:econom-icallyliberal

Senators Paul H. Douglas of Illinois, J. W.Fulbright of Representative Wright Patmanof Texas, all Democrats, were active in ques-tioning Mr. Humphrey. The Democrats as-serted that the Administration’s tax reduc-tion program was loaded in favor of businessenterprises and shareholders in industry andagainst the taxpayer in the lowest incomebrackets. . . .

Onedimension:economi-callyneutral

Senator Fulbright .. declare[d] that the prob-lem was to expand consumption rather thanproduction. ... “Production is the goose thatlays the golden egg,“ Mr. Humphrey replied.“Payrolls make consumers.”

Table 4: Excerpts from article #716033. Because inthe second paragraph the topic is anti tax reductionon businesses, it is economically liberal. The thirdparagraph is at the same time economically conser-vative and liberal because one speaker is advocatingfor decreasing tax on businesses and asserting thatproduction gives an advantage to businesses, theother is advocating for decreasing tax on the poorbecause they need the income and asserting thathealthy businesses are the ones who pay salariesfor the low income bracket worker.

erate & substantial (Artstein and Poesio, 2008),respectively; for social, the ‘fair’ agreement wasnoticed during annotation, and additional discus-sion for each paragraph was then held. Afterwards,25 more articles were independently annotated andassessed with an α of 0.53. Although the agree-ments were not perfect and reflected a degree ofsubjectivity in this task, all dimensional labels wereadjudicated after discussions between annotators.In total, creating this dataset cost ∼150 hours ofmanual dimensional labeling.

4 Ideology analyses

In this section we present the overall statistics of thedataset annotations and insights into the languagebeing used to describe each policy position acrossdimensions.

4.1 Dimensional label distribution

Table 5 shows the number of articles and para-graphs (in total and by dimension) for each of the

#Docs Econ. Soc. Fgn. Total

CSM 37 115 63 82 260CT 14 48 33 16 97NYT 60 219 114 130 463TM 52 134 60 89 283WSJ 12 42 21 21 84Total 175 558 291 338 1187

Table 5: Dimensional label counts across all 721paragraphs in the adjudicated data (there can bemultiple dimensions per paragraph).

Figure 1: Dimensional label distribution per dimen-sion.

news outlets; Figure 1 shows the dimensional labeldistributions per outlet for each dimension. Ex-pectedly, the dimensional labels often diverge fromproclaimed ideology of the news outlet.

We also analyze the percentage of articles thatcontain at least one pair of paragraph labels thatlean in different directions; for instance, a para-graph with a label of globalist in the foreign dimen-sion and a paragraph with a label of conservative orneutral in the fiscal dimension. The percentage ofsuch articles is 78.3%. Out of this set of articles, weexamine the average proportion of neutral, liberal,and conservative paragraph labels, and find neutrallabels have the highest share (43.27%), followedby liberal (33.20%) and conservative (23.53%).

In Figure 2 (right), we visualize the percentageof articles where two dimensional labels co-occurwithin the same article. The figure indicates thatideology varies frequently within an article, furthermotivating our paragraph-level approach.

In Figure 2 (left) we also show paragraph-levellabel co-occurrence. Unlike the article-level, theco-occurrences are less frequent and we are morelikely to observe co-occurrences along the sameside of ideology. Still, we see interesting nuances;for example, on both the paragraph and the arti-cle level, the economic dimension is often neutral,and this tends to co-occur with both liberal and

Figure 2: Co-occurrence matrices on the paragraphlevel (left) and article level (right)

Eco

nom

ic C: mr (5.2), tax (5.0), truman (3.7), business(3.3), billion (3.2)L: school (-4.3), education (-3.3), commission(-3.2), senator (-3.0), plan (-2.5)

Soci

al C: defense (7.5), tax (4.6), air (4.4), billion(3.9), missile (3.8)L: federal (-3.6), wage (-3.6), would (-3.5),policy (-3.2), labor (-2.9)

Fore

ign C: defense (6.9), force (5.3), north (5.2), air

(5.0), vietnam (4.6)L: aid (-9.3), economic (-5.5), foreign (-5.3),germany (-4.3), make (-4.1)

Table 6: Words with the most positive and negativeweights from a logistic regression model trainedto predict liberal/conservative ideology for eachdimension.

conservative positions in other dimensions.

4.2 Lexical Analysis

To understand ways policy positions are reflectedin text, we also look into the top vocabulary thatassociates with conservative or liberal ideology. Todo so, we train a logistic regression model for eachdimension to predict whether a paragraph is labeledconservative or liberal on that dimension, using un-igram frequency as features. In Table 6 we showthe top most left-leaning (L) or right-leaning (R)vocabulary with their weights. The table intimatelyreproduces our annotation of ideology. For exam-ple, words like federal and Senator allude to thefact that the topic is at the federal level. The im-portance of education and labor to liberals is alsoevident in the economic and social dimensions inwords like school, education, wage and labor. Theimportance of the topic of taxation and defense isevident in conservative ideology in words such astax, business, missile, force, and defense.

4.3 Qualitative analysis of text highlights

For 25 articles during initial training, all annotatorshighlighted the sentences that are relevant to eachdimension they annotated. This helps annotatorsfocus on the sentences that drive their decision,and provides insights to the language of ideologywhich we discuss here. On average, 21%–54% ofthe sentences in a paragraph were highlighted.

While entities such as “President” and”Congress” were the most prevalent, they tackledsocial and economic issues combined. Thisis not surprising as it suggests that when themedia quotes or discusses the “President” and“Congress”, they do so with reference to morecomplex policy issues. In contrast, individualcongresspeople tackled mostly economic or socialissues. This also is not surprising as it suggeststhat individual congresspeople are more concernedwith specific issues. Interestingly, “House” and“Senate” almost always figured more in socialissues. This suggest that when news media speaksabout a specific chamber, they do so associatingthis chamber with social issues. Finally, partyaffiliation was infrequent and was mostly associatewith social issues.

5 Polarization

In this section, we demonstrate how our frameworkcan be used to analyze ideological polarization,quantitatively. To say that two groups are polar-ized is to say that they are moving to oppositeends of an issue on the multi-dimensional ideo-logical spectrum while, at the same time, their po-litical views on ideological issues converge, i.e.socially liberals become also economically lib-eral (Fiorina and Abrams, 2008). In political sci-ence when ideology is multidimensional, polar-ization is quantified by considering three mea-sures that capture complementary aspects (Lelkes,2016): (1) sorting (Abramowitz and Saunders,1998) (to what extent the annotated ideology de-viates from an outlet’s ideological bias); (2) issueconstraint (Baldassarri and Gelman, 2008) (corre-lation between different dimensions); (3) ideolog-ical divergence (Fiorina and Abrams, 2008) (thedistance between two groups along a single dimen-sion). Together they describe changes in the ideo-logical environment over time: only a concurrentincrease in all three measures indicates polarizationin media.

Limitations: We use only the fully adjudicated

Adfontes Allsides MBFC Average

CSM -.06 0.00 -.16 -.07CT -.04 NA .34 .15NYT -.20 -.5 -.4 -.36TM -.10 -.5 -.6 -.4WSJ .15 .25 .58 .32

Table 7: Ideological bias of news outlets from com-mon references of media bias. We use the averagein our analyses.

Figure 3: The evolution of the sorting measure, ag-gregating conservative/neutral/liberal outlets. Mov-ing further away from the zero means articles de-viate more from the proclaimed ideology of theiroutlets.

data and refrain from using model predictions,since our baseline experiments in Section 6 showthat predicting ideology is challenging. Hence, theanalysis are demonstrations of what our frameworkenables, which we discuss at the end of this section,and the conclusions are drawn for our annotatedarticles only.

Measure 1: Sorting We adapt the sorting prin-ciple of Abramowitz and Saunders (1998) to ourdata and investigate the difference between the ide-ological bias of a news outlet and the ideology ofannotated articles from the outlet. To obtain thebiasBj of a news outlet j, we average the ratings ofeach news outlet across commonly used referencesfor media bias, shown in Table 7. To obtain an ar-ticle i’s overall ideology Ii we take the average ofliberal (-1), neutral (0), and conservative (1) acrossits paragraphs in all three dimensions. Thus, foreach 4-year time period with m articles for outlet j,the sorting measure would be the absolute distanceof article vs. outlet ideology |avgmi=1(Ii)−Bj |/Bj .

In Figure 3, we plot the sorting measure, aver-aged across news outlets of the same ideologicalbias. The figure shows that in our sample of arti-cles, the liberal news outlets were closest to their

Figure 4: The evolution of the issue constraint mea-sure, stratified by pairs of dimension. Higher val-ues mean some pairs of dimensions correlate morestrongly than other. Due to the lack of annotateddata that simultaneously contains social & eco-nomic dimensions (1st graph), and economic &foreign dimensions (3rd graph) from conservativeoutlets their respective blue lines start in 1958.

ideological bias measure over time whereas theneutral outlets were more liberal before 1957 andafter 1964. The conservative outlets were moreconservative at that time than their ideological bias.

Measure 2: Issue constraint This measurerefers to the tightness between ideological dimen-sions over time (Baldassarri and Gelman, 2008) soas to assess, for example, if socially liberal dimen-sions are more and more associated with economicliberal dimensions at the outlet level. Concretely,for each article we derive its ideology along a singledimension as the average of paragraph annotationsalong that dimension. We, then, calculate the Pear-son correlation between the article ideology of eachpair of dimensions, over all articles from one outletin the same period.

Results in Figure 4, again averaged across newsoutlets of the same ideological bias, show that forconservative media, the correlation between anytwo dimensions in the annotated data are largelypositive (e.g., economically conservative were alsosocially conservative) until 1967 or 1970. How-ever, for liberal and neutral outlets, the correlations

Figure 5: The evolution of the ideological diver-gence measure stratified by dimension. The dot-ted line refers to the bimodality threshold (Lelkes,2016). Higher values mean the ideology of an arti-cle along that one dimension is bimodal.

fluctuates especially when considering the foreigndimension.

Measure 3: Ideological divergence This mea-sures the distance between two ideological groupson a single dimension (Fiorina and Abrams, 2008).We follow Lelkes (2016) to calculate the bimodal-ity coefficient (Freeman and Dale, 2013; Pfisteret al., 2013) per dimension over articles from allnews outlets over the same time period. The bi-modality coefficient ranges from 0 (unimodal, thusnot at all polarized) to 1 (bimodal, thus completelypolarized).

Figure 5 shows the evolution of the ideologi-cal divergence measure of every dimension. A bi-modality measure assessed whether this divergenceattained the threshold for the cumulative distribu-tion to be considered bimodal. Ideological distance,as a result, refers to the three bimodal distances be-tween pairs of dimensions. We note for examplethat the foreign dimension crossed this thresholdbetween 1956 and 1968. This means that liberalsand conservatives grew further apart on foreignissues during this time period.

Discussion Taken together, the graphs indicatethat the years between 1957 and 1967 are mostnoteworthy. During this period, from our sample ofarticles, we see that polarization was only presentin conservative news media because it (1) sorted,as it was significantly more conservative than itscomposite bias measure, (2) constrained its issues,as evidence by high correlation values, and (3) be-came bimodal, as the ideological distance betweentheir positions and those of their liberal counterparton foreign issues increased.

Econ Social Foreign Average

Majority 0.30 0.23 0.25 0.26

BiLSTM 0.44 0.37 0.33 0.38BERT no-ft 0.46 0.31 0.53 0.44

+pre-training 0.42 0.32 0.46 0.40BERT ft 0.64 0.50 0.52 0.55

+pre-training 0.56 0.47 0.46 0.49-focal loss 0.61 0.50 0.50 0.54

Table 8: Macro F1 scores of the models averaged across10 runs.

While this conclusion applies to only the setof articles in our dataset, the above analysis illus-trates that our framework enables nuanced, quan-titative research into polarization. This analysis,paired with the political context of the time (e.g.,all three branches of government were Republicanuntil 1957 and Congress flipped Democrat that yearduring the Midterm Elections. In 1961, all threebranches of government became Democrat until1969 when only the presidency flipped Republicanagain), can lead to high-impact insights into the po-litical dynamics of our society. We leave for futurework, potentially equipped with strong models forideology prediction, to analyze the data at scale.

6 Experiments

We present political ideology detection experi-ments as classification tasks per-dimension on theparagraph level. We performed an 80/10/10 split tocreate the train, development, and test sets. The de-velopment and test sets contain articles uniformlydistributed from our time period (1947 to 1975)such that no particular decade is predominant. Toensure the integrity of the modeling task, all para-graphs belonging to the same article are present ina single split. The number of examples in the splitsfor each dimension for the adjudicated data are asfollows: for the economic dimension, we had 450training, 50 development, and 58 test examples.For the social dimension, we had 253 for training,13 for development, and 25 for testing. For theforeign dimension, we had 266 for training, 33 fordevelopment, and 39 for testing.

6.1 ModelsRecurrent neural network. We trained a 2-layer bidirectional LSTM (Hochreiter and Schmid-huber, 1997), with sequence length and hidden sizeof 256, and 100D GloVe embeddings (Penningtonet al., 2014).

Pre-trained language models We used BERT-base (Devlin et al., 2019) from HuggingFace (Wolfet al., 2020) and trained two versions, with andwithout fine-tuning. In both cases we used a cus-tom classification head consisting of 2 linear layerswith a hidden size of 768 and a ReLU betweenthem. To extract the word embeddings we followedDevlin et al. (2019) and used the hidden states fromthe second to last layer. To obtain the embeddingof the whole paragraph5 we averaged the word em-beddings and passed this vector to the classificationhead.

To find the best hyperparameters we performeda grid search in each dimension. For the economicdimension, the best hyperparameters consisted of alearning rate of 2e-6, 6 epochs of training, a gammavalue of 2, no freezing of the layers, a 768 hiddensize, and 10% dropout. For the social dimension,the best hyperparameters were a learning rate of2e-5, 12 epochs, a gamma of 4, no freezing of thelayers, a 768 hidden size, and 10% dropout. Finally,for the foreign dimension the best hyperparametersconsisted of a learning rate of 2e-5, 6 epochs, agamma of 2, no freezing of the layers, a 768 hiddensize, and a 10% dropout.

Focal loss. To better address the imbalanced la-bel distribution of this task, we incorporated focalloss (Lin et al., 2017), originally proposed for denseobject detection. Focal loss can be interpreted as adynamically scaled cross-entropy loss, where thescaling factor is inversely proportional to the con-fidence on the correct prediction. This dynamicscaling, controlled by hyperparameter γ, leads to ahigher focus on the examples that have lower con-fidences on the correct predictions, which in turnleads to better predictions on the minority classes.Since a γ of 0 essentially turns a focal loss intoa cross entropy loss, it has less potential to hurtperformance than to improve it. We found the bestγ values to be 2 or 4 depending on the dimension.

Task-guided pre-training. We also explored su-pervised pre-training on two adjacent tasks that cangive insights to the relationship between tasks. Weused distant supervision that labeled the ideologicalbias of each article according to that of its news out-let from www.allsides.com (Kulkarni et al.,2018). This procedure allowed us to use the unan-notated US Federal Budget articles (Section 3). 6

599% of the paragraphs in the dataset have ≤512 tokens.6We also experimented with pre-training on the dataset

from Chen et al. (2020).

6.2 Results

Table 8 shows the macro F1 for each configura-tion, averaged across 10 runs with different ran-dom initializations. The fine-tuned BERT model,with no task-guided pre-training shows the bestperformance across all 3 ideology dimensions. Itis important to note that all the models do betterthan randomly guessing, and better than predictingthe majority class. This shows that the models arecapturing some of the complex underlying phenom-ena in the data. However, the classification tasksstill remain challenging for neural models, leavingplenty of room for improvement in future work.

The BERT ft -focal loss setting ablates the effectof focal loss against a weighted cross entropy loss.with weights inversely proportional to the distri-bution of the classes in the dimension. This losshelped get a bump in the macro F1 score of around0.1 for each dimension compared to an unweightedcross entropy loss. However, the focal loss gavefurther improvements for 2 of the 3 dimensions.

Although task-guided pre-training improved theBERT (no fine-tuning) model for 1 of the 3 dimen-sions, it led to worse performance than BERT (fine-tuned). The improvement on the no fine-tuningsetting indicates that there is a potential correlationto be exploited by the ideology of the news outlet,but such labels are not that informative for multi-dimensional prediction. We hope that this datasetprovides a testbed for future work to evaluate moredistant supervision data/methods.

7 Related work

In contrast to our multi-dimensional approach thatexamines the ideology of the policy and not theauthor stance, much of the recent work on the topicof ideology in computational linguistics has beendedicated to the detection of ideology or ideologi-cal bias in news media while collapsing ideology toone dimension (Budak et al., 2016; Kulkarni et al.,2018; Kiesel et al., 2019; Baly et al., 2019, 2020;Chen et al., 2020; Ganguly et al., 2020; Stefanovet al., 2020). The proposed computational mod-els classify the partiality of media sources withoutquantifying their ideology (Elejalde et al., 2018).

Other researchers interested in the computationalanalysis of the ideology have employed text datato analyze congressional text data at the legisla-tive level (Sim et al., 2013; Gentzkow et al., 2016)

However, because their dataset starts from 2006 (outside ofour time domain), this setting performed poorly.

and social media text at the electorate level (Saez-Trumper et al., 2013; Barbera, 2015).

In political science, the relationship between(news) media and polarization is also an active areaof research. Prior work has studied media ideolog-ical slant in coverage (Valentino et al., 2009) aswell as media slant in choice of coverage (Georgeand Waldfogel, 2006). Prior (2013) argues there isno firm evidence of a direct causal relationship be-tween media and polarization and that this relation-ship depends on preexisting attitudes and politicalsophistication. On the other hand, Gentzkow et al.(2016) have established that polarization languagesnippets move from the legislature in the directionof the media whereas (Baumgartner et al., 1997)have shown that the media has an impact on agendasettings of legislatures.

8 Conclusion

We take the first step in studying multi-dimensionalideology and polarization over time and in news ar-ticles relying on the major political science theoriesand tools of computational linguistics. This opensup new opportunities and invites researchers to usethis corpus to study the spread of propaganda andmisinformation in tandem with ideological shiftsand polarization. The presented corpus also pro-vides the opportunity for studying ways that socialcontext determines interpretations in text while dis-tinguishing author stance from content.

Our work has several limitations. We only focuson news whereas these dynamics might be differ-ent in other forms of communication such as socialmedia posts or online conversations, and the leg-islature. Further, our corpus is relatively smallalthough carefully annotated by experts. Futurework may explore semi-supervised models or ac-tive learning techniques for annotating and prepar-ing a larger corpus that may be used in diverseapplications.

Acknowledgements

We thank Zachary Elkins, Richard Lau, Beth Leech,and Katherine McCabe for their feedback of thiswork and its annotation protocol, and Katrin Erkfor her comments. Thanks to Pitt Cyber https://www.cyber.pitt.edu/ for supporting thisproject. We also acknowledge the Texas AdvancedComputing Center (TACC) at UT Austin and TheCenter for Research Computing at the University

of Pittsburgh for providing the computational re-sources for many of the results within this paper.

References

Alan I Abramowitz and Kyle L Saunders. 1998.Ideological realignment in the us electorate. TheJournal of Politics, 60(3):634–652.

Theodor Adorno, Else Frenkel-Brenswik, Daniel JLevinson, and R Nevitt Sanford. 2019. The au-thoritarian personality. Verso Books.

Ron Artstein and Massimo Poesio. 2008. Inter-coder agreement for computational linguistics.Computational Linguistics, 34(4):555–596.

Delia Baldassarri and Andrew Gelman. 2008. Par-tisans without constraint: Political polarizationand trends in american public opinion. AmericanJournal of Sociology, 114(2):408–446.

Ramy Baly, Giovanni Da San Martino, JamesGlass, and Preslav Nakov. 2020. We can detectyour bias: Predicting the political ideology ofnews articles. In Proceedings of the 2020 Con-ference on Empirical Methods in Natural Lan-guage Processing (EMNLP), pages 4982–4991.

Ramy Baly, Georgi Karadzhov, Abdelrhman Saleh,James Glass, and Preslav Nakov. 2019. Multi-task ordinal regression for jointly predicting thetrustworthiness and the leading political ideol-ogy of news media. In Proceedings of the 2019Conference of the North American Chapter ofthe Association for Computational Linguistics:Human Language Technologies, Volume 1 (Longand Short Papers), pages 2109–2116.

Pablo Barbera. 2015. Birds of the same feathertweet together: Bayesian ideal point estimationusing twitter data. Political analysis, 23(1):76–91.

Frank R Baumgartner, Bryan D Jones, and Beth LLeech. 1997. Media attention and congressionalagendas. Do the media govern, pages 349–363.

Emily M Bender, Timnit Gebru, AngelinaMcMillan-Major, and Shmargaret Shmitchell.2021. On the dangers of stochastic parrots: Canlanguage models be too big? In Proceedings ofthe 2021 ACM Conference on Fairness, Account-ability, and Transparency, pages 610–623.

Alessandro Bessi, Fabio Petroni, Michela Del Vi-cario, Fabiana Zollo, Aris Anagnostopoulos, An-tonio Scala, Guido Caldarelli, and Walter Quat-trociocchi. 2016. Homophily and polarization inthe age of misinformation. The European Physi-cal Journal Special Topics, 225(10):2047–2059.

David M Blei, Andrew Y Ng, and Michael I Jordan.2003. Latent dirichlet allocation. Journal ofmachine Learning research, 3(Jan):993–1022.

Ceren Budak, Sharad Goel, and Justin M Rao.2016. Fair and balanced? Quantifying me-dia bias through crowdsourced content analysis.Public Opinion Quarterly, 80(S1):250–271.

Angus Campbell, Philip E Converse, Warren EMiller, and Donald E Stokes. 1980. The ameri-can voter. University of Chicago Press.

Edward G Carmines, Michael J Ensley, andMichael W Wagner. 2012. Who fits the left-right divide? partisan polarization in the ameri-can electorate. American Behavioral Scientist,56(12):1631–1653.

Thomas M Carsey and Geoffrey C Layman. 2006.Changing sides or changing minds? party identi-fication and policy preferences in the americanelectorate. American Journal of Political Sci-ence, 50(2):464–477.

Wei-Fan Chen, Khalid Al Khatib, HenningWachsmuth, and Benno Stein. 2020. Analyz-ing political bias and unfairness in news articlesat different levels of granularity. In Proceedingsof the Fourth Workshop on Natural LanguageProcessing and Computational Social Science,pages 149–154.

Mark Davies. 2012. Expanding horizons in histor-ical linguistics with the 400-million word cor-pus of historical american english. Corpora,7(2):121–157.

Shrey Desai, Barea Sinno, Alex Rosenfeld, andJunyi Jessy Li. 2019. Adaptive ensembling: Un-supervised domain adaptation for political doc-ument analysis. In Proceedings of the 2019Conference on Empirical Methods in NaturalLanguage Processing and the 9th InternationalJoint Conference on Natural Language Process-ing (EMNLP-IJCNLP), pages 4712–4724.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, andKristina Toutanova. 2019. Bert: Pre-trainingof deep bidirectional transformers for languageunderstanding. In Proceedings of the 2019 Con-ference of the North American Chapter of theAssociation for Computational Linguistics: Hu-man Language Technologies, Volume 1 (Longand Short Papers), pages 4171–4186.

Paul DiMaggio, John Evans, and Bethany Bryson.1996. Have american’s social attitudes becomemore polarized? American journal of Sociology,102(3):690–755.

Erick Elejalde, Leo Ferres, and Eelco Herder. 2018.On the nature of real and perceived bias in themainstream media. PloS one, 13(3):e0193765.

Stanley Feldman and Christopher Johnston. 2014.Understanding the determinants of political ide-ology: Implications of structural complexity. Po-litical Psychology, 35(3):337–358.

Morris P Fiorina and Samuel J Abrams. 2008. Po-litical polarization in the american public. Annu.Rev. Polit. Sci., 11:563–588.

Jonathan B Freeman and Rick Dale. 2013. Assess-ing bimodality to detect the presence of a dualcognitive process. Behavior research methods,45(1):83–97.

Soumen Ganguly, Juhi Kulshrestha, Jisun An, andHaewoon Kwak. 2020. Empirical evaluation ofthree common assumptions in building politicalmedia bias datasets. In Proceedings of the In-ternational AAAI Conference on Web and SocialMedia, volume 14, pages 939–943.

Matthew Gentzkow, Jesse M Shapiro, and MattTaddy. 2016. Measuring polarization in high-dimensional data: Method and application tocongressional speech. National Bureau of Eco-nomic Research.

Lisa M George and Joel Waldfogel. 2006. The newyork times and the market for local newspapers.American Economic Review, 96(1):435–447.

Andrew F Hayes and Klaus Krippendorff. 2007.Answering the call for a standard reliability mea-sure for coding data. Communication methodsand measures, 1(1):77–89.

Sepp Hochreiter and Jurgen Schmidhuber. 1997.Long short-term memory. Neural Comput.,9(8):1735–1780.

Shanto Iyengar, Yphtach Lelkes, Matthew Leven-dusky, Neil Malhotra, and Sean J Westwood.2019. The origins and consequences of affectivepolarization in the united states. Annual Reviewof Political Science, 22:129–146.

Jeffery A Jenkins, Eric Schickler, and Jamie L Car-son. 2004. Constituency cleavages and congres-sional parties: Measuring homogeneity and po-larization, 1857–1913. Social Science History,28(4):537–573.

Jacob Jensen, Suresh Naidu, Ethan Kaplan, Lau-rence Wilse-Samson, David Gergen, MichaelZuckerman, and Arthur Spirling. 2012. Polit-ical polarization and the dynamics of politicallanguage: Evidence from 130 years of partisanspeech [with comments and discussion]. Brook-ings Papers on Economic Activity, pages 1–81.

John T Jost, Christopher M Federico, and Jaime LNapier. 2009. Political ideology: Its structure,functions, and elective affinities. Annual reviewof psychology, 60:307–337.

Johannes Kiesel, Maria Mestre, Rishabh Shukla,Emmanuel Vincent, Payam Adineh, David Cor-ney, Benno Stein, and Martin Potthast. 2019.Semeval-2019 task 4: Hyperpartisan news detec-tion. In Proceedings of the 13th InternationalWorkshop on Semantic Evaluation, pages 829–839.

Vivek Kulkarni, Junting Ye, Steven Skiena, andWilliam Yang Wang. 2018. Multi-view modelsfor political ideology detection of news articles.In Proceedings of the 2018 Conference on Empir-ical Methods in Natural Language Processing,pages 3518–3527.

Alban Lauka, Jennifer McCoy, and Rengin B Firat.2018. Mass partisan polarization: Measuring arelational concept. American behavioral scien-tist, 62(1):107–126.

Yphtach Lelkes. 2016. Mass polarization: Man-ifestations and measurements. Public OpinionQuarterly, 80(S1):392–410.

Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaim-ing He, and Piotr Dollar. 2017. Focal loss for

dense object detection. In Proceedings of theIEEE international conference on computer vi-sion, pages 2980–2988.

Jeffrey Pennington, Richard Socher, and Christo-pher Manning. 2014. GloVe: Global vectorsfor word representation. In Proceedings of the2014 Conference on Empirical Methods in Natu-ral Language Processing (EMNLP), pages 1532–1543, Doha, Qatar. Association for Computa-tional Linguistics.

Roland Pfister, Katharina A Schwarz, MarkusJanczyk, Rick Dale, and Jon Freeman. 2013.Good things peak in pairs: a note on the bi-modality coefficient. Frontiers in psychology,4:700.

Paul Pierson and Eric Schickler. 2020. Madison’sconstitution under stress: A developmental anal-ysis of political polarization. Annual Review ofPolitical Science, 23:37–58.

Markus Prior. 2013. Media and political polariza-tion. Annual Review of Political Science, 16:101–127.

Martin Riedl and Chris Biemann. 2012. Topictil-ing: a text segmentation algorithm based on lda.In Proceedings of ACL 2012 Student ResearchWorkshop, pages 37–42.

Shruti Rijhwani and Daniel Preotiuc-Pietro. 2020.Temporally-informed analysis of named entityrecognition. In Proceedings of the 58th AnnualMeeting of the Association for ComputationalLinguistics, pages 7605–7617.

Diego Saez-Trumper, Carlos Castillo, and MouniaLalmas. 2013. Social media news communi-ties: gatekeeping, coverage, and statement bias.In Proceedings of the 22nd ACM internationalconference on Information & Knowledge Man-agement, pages 1679–1684.

Yanchuan Sim, Brice DL Acree, Justin H Gross,and Noah A Smith. 2013. Measuring ideolog-ical proportions in political speeches. In Pro-ceedings of the 2013 conference on empiricalmethods in natural language processing, pages91–101.

Peter Stefanov, Kareem Darwish, Atanas Atanasov,and Preslav Nakov. 2020. Predicting the topi-cal stance and political leaning of media using

tweets. In Proceedings of the 58th Annual Meet-ing of the Association for Computational Lin-guistics, pages 527–537.

Nicholas A Valentino, Antoine J Banks, Vincent LHutchings, and Anne K Davis. 2009. Selectiveexposure in the internet age: The interaction be-tween anxiety and information utility. PoliticalPsychology, 30(4):591–613.

Michela Del Vicario, Walter Quattrociocchi, Anto-nio Scala, and Fabiana Zollo. 2019. Polarizationand fake news: Early warning of potential mis-information targets. ACM Transactions on theWeb (TWEB), 13(2):1–22.

Thomas Wolf, Lysandre Debut, Victor Sanh, JulienChaumond, Clement Delangue, Anthony Moi,Pierric Cistac, Tim Rault, Remi Louf, MorganFuntowicz, Joe Davison, Sam Shleifer, Patrickvon Platen, Clara Ma, Yacine Jernite, Julien Plu,Canwen Xu, Teven Le Scao, Sylvain Gugger,Mariama Drame, Quentin Lhoest, and Alexan-der M. Rush. 2020. Huggingface’s transformers:State-of-the-art natural language processing.