u n i ve r s i t y o f co pe n h ag e n

Københavns Universitet

Book reviews in humanities research evaluations

Zuccala, Alesia Ann; Van Leeuwen, Thed

Published in:Journal of the American Society for Information Science and Technology

DOI:10.1002/asi.21588

Publication date:2011

Document VersionPublisher's PDF, also known as Version of record

Citation for published version (APA):Zuccala, A., & Van Leeuwen, T. (2011). Book reviews in humanities research evaluations. DOI:10.1002/asi.21588

Download date: 12. jul.. 2018

Book Reviews in Humanities Research Evaluations

Alesia Zuccala and Thed van LeeuwenCentre for Science and Technology Studies, Leiden University, Willem Einthoven Building, Wassenaarseweg 62A,2333 AL Leiden, The Netherlands. E-mail: {a.a.zuccala, leeuwen}@cwts.leidenuniv.nl

Bibliometric evaluations of research outputs in the socialsciences and humanities are challenging due to limi-tations associated with Web of Science data; however,background literature has shown that scholars are inter-ested in stimulating improvements.We give special atten-tion to book reviews processed byWeb of Science historyand literature journals, focusing on two types: Type I(i.e., reference to book only) and Type II (i.e., referenceto book and other scholarly sources). Bibliometric dataare collected and analyzed for a large set of reviews(1981–2009) to observe general publication patterns andpatterns of citedness and co-citedness with books underreview. Results show that reviews giving reference onlyto the book (Type I) are published more frequently whilereviews referencing the book and other works (Type II)are more likely to be cited. The referencing culture ofthe humanities makes it difficult to understand patternsof co-citedness between books and review articles with-out further in-depth content analyses. Overall, citationcounts to book reviews are typically low, but our datashowed that they are scholarly and do play a role in thescholarly communication system. In the disciplines ofhistory and literature, where book reviews are prominent,counting the number and type of reviews that a scholarproduces throughout his/her career is a positive step for-ward in research evaluations. We propose a new set ofjournal quality indicators for the purpose of monitoringtheir scholarly influence.

Introduction

The purpose of this study is to examine the scholarly roleof book reviews and to determine whether or not there areeffective methods of including them in bibliometric researchevaluations for the humanities.

Figure 1 illustrates two types of book reviews. ReviewType I differs from review Type II given that the first includesonly a reference to the book that has been reviewed whilethe second includes both the book and references to otherscholarly sources.

Received April 19, 2011; revised May 17, 2011; accepted May 17, 2011

© 2011 ASIS&T • Published online 20 July 2011 in Wiley Online Library(wileyonlinelibrary.com). DOI: 10.1002/asi.21588

FIG. 1. Types of book reviews categorized by references.

Now consider the citing behavior of a humanities scholarwho writes a journal article. There are at least four waysin which a book review may be cited. Table 1 below liststhese options. An author may cite the book review alone;thereby omitting a separate citation to the book. An authormay cite separately both the review and the published book.Additionally, if the book review was written in the style ofa literary essay (i.e., Type II) an author may cite the review,the published book (or not), and some of the other scholarlyreferences acknowledged in the review.

We are interested in how book reviewing has evolvedwithin a Web of Science (WoS) context for the database periodof 1981 to 2009. In the first section of this article, we presentsome general statistics pertaining to the humanities fields thatproduce a significant number of book reviews per year. Next,we focus on two leading book-reviewing fields, history andliterature, retrieve citations to the reviews published in thesefields, and identify the reviews that were co-cited with thebooks. Using both the citation and co-citation data, we willdetermine if it is feasible to utilize book reviews in bibliomet-ric research evaluations, and consider the development of anew indicator for measuring the influence that book reviewshave on scholarly communication.

Background

Bibliometric evaluations of research outputs in the socialsciences and humanities (SSH) are riddled with drawbacks,

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY, 62(10):1979–1991, 2011

TABLE 1. Options for citing a book review in a journal article.

Citing Options Example

Book review∗ ∗Cooper, F. 1996. “Black liberation: A comparative history of black ideologies in the United States andSouth Africa – Frederickson, GM.” American Historical Review, 101(4): 1122.

Book review∗ + BookB ∗Cooper, F. 1996. “Black liberation: A comparative history of black ideologies in the United States andSouth Africa – Frederickson, GM.” American Historical Review, 101(4): 1122.BFrederickson, G.M. 1995. Black liberation: A comparative history of black ideologies in the United Statesand South Africa. New York: Oxford University Press, Inc.

Book review∗ and other references ∗Leffler, M.P. (1999). “We now know: Rethinking Cold War History” American Historical Review, 104: 501.

appearing in the review. Roberts G. 1994. “Moscow and the marshall-plan – politics, ideology and the onset of the cold-war, 1947.”Europe-Asia Studies, 46:1371.

Rotter AJ. 1994. “Gender relations, foreign-relations – the united-states and south-asia, 1947–1964.”Journal of American History, 81: 518.

Book review∗ + BookB and otherreferences appearing in the review

∗Asad, T. 1987. “Europe and the people without history – Wolf, E.” Comparative Studies in Society andHistory, 29(3): 594–607.BWolf, E.R. 1982. Europe and the people without history. Berkeley, CA: University of California Press.Wickham, C. 1985. “The Uniqueness of the East”. Journal of Peasant Studies, 12(2–3): 166–96.

yet many scholars have been interested in stimulatingimprovements. Archambault and Vignola Gagné (2004)reminded us that bibliometric analyses require large quanti-ties of data and that the pace of theoretical development in theSSH can be slower than that in the natural sciences: The “timerequired to accumulate citations makes analyses more diffi-cult, particularly when the goal is to assist in decision makingand policy setting” (p. 24). Humanities scholars often dis-seminate information using media other than journals, mainlybooks (Huang & Chang, 2008), and many contribute to local-ized outlets, including those directed to the nonscholarlypublic (Nederhof, 2006).

Hicks and Wang (2009) as well as Moed et al. (2009)focused on the requirements for creating appropriate datainfrastructures for the SSH. Hicks and Wang suggested thatit is perhaps best to rely on national research-documentationsystems, where “universities submit bibliometric records oftheir publications;” thus, taking responsibility for data qual-ity while “agencies then validate and standardize the data”(p. 18). Another recommendation is to persuade publishers tosubmit records to a “database of published scholarly bookswith records that include book author affiliation” (Hicks &Wang, 2009, p. 20). Moed et al. (2009) advocated the WorldWide Web (e.g., Google Scholar) as a source of data forSSH metrics, emphasizing the role of open access and thedevelopment of institutional repositories. The authors alsocommented on the potential for combining a number ofspecial bibliographies across Europe to create one compre-hensive SSH database. Finally, due to the commercial natureof Thomson Reuters’ WoS and Elsevier’s Scopus, it is plau-sible to assume that both providers may be willing to expandtheir SSH coverage.

At present, collecting data from the WoS for humanitiesevaluations is a challenge. Books are a predominant aspect

of this literature and can be identified only with special filter-ing procedures applied to compiled reference lists (Lewison,2001, 2004). For evaluation purposes, researchers are eitherbypassing the WoS to explore the potential of library catalogsas tools for bibliometric analyses (Torres-Salinas & Moed,2009) or turning to Google Books as a resource (Kousha &Thelwall, 2009). Book reviews, on the other hand, are pro-cessed by the WoS, and considerable space is devoted to themin scholarly journals.

Book reviews fit within the realm of scholarly communica-tion because they involve scholarly producers and users, andthey are disseminated through formal channels (Borgman,1990). As a result, the practice of writing book reviews hasoften been scrutinized (Cortada, 1998; Miranda, 1996). Forinstance, Glenn (1978) suggested that reviews “are not as ade-quate for evaluating books and authors as many people seemto think” and that this is due to “a lack of consensus on stan-dards” as well as “motives ranging from altruistic to selfish”(pp. 254–255). Reviewers “may often refrain from publish-ing their private negative evaluations” or “publish evaluationsthat are honest but different from the evaluations they wouldhave made” (Glenn, 1978, p. 255) if they had not been askedto write the review in the first place.

Glenn’s (1978) argument may be made about any typeof peer review in academia; thus, if a scholar is asked towrite a book review, it is his or her responsibility to elimi-nate personal biases, to ensure that it is a trustworthy pieceof information, and to make it publicly accessible (Kling &McKim, 1999). Guidelines, such as the Alberta Book ReviewWriting Guide (University of Alberta, 2010), can help thescholar to accomplish what is expected: “Rather than a simplesummary of a book’s contents, a review is a critical essay. Itspurpose is not to prove that you read the book … but to showthat you can think critically about what you read.” The role

1980 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011DOI: 10.1002/asi

of an effective reviewer is to (a) critique the author’s writ-ing style, (b) evaluate the author’s intent behind the book,(c) determine if the author has presented his or her ideaslogically and consistently, (d) confirm the author’s ability tocontextualize the work or connect it to wider developments inthe field, and (e) examine critical “silences” or source omis-sions that might weaken the book’s content (University ofAlberta, 2010).

Research pertaining to book reviews has focused less ontheir use in academic evaluations and more on their con-tent and applicability for library selection processes (Blake,1989; Furnham, 1986; Natowitz & Wheeler Carlo, 1997;Parker, 1989; Serebnick, 1992). Librarians use reviews for thedevelopment of book collections, but studies have shown thatscholars consider them to be useful as well. Humanities/artsand social science scholars read book reviews, normallybetween 1 and 10 per month (Spink, Robbins, & Schamber,1998), and value them more for teaching and research thando scholars in science and technology (Hartley, 2006). Thefeatures that most scholars value in a good review includethe presentation of a straightforward overview of the book,a strong critique of the book’s main argument, and a strongevaluation of the book’s academic credibility (Hartley, 2006).In addition, many scholars—60% in the arts, 50% in the socialsciences, and 41% in the sciences—seem to agree that theacademic standing of book reviews would be enhanced “ifinstitutions gave academic credit for writing [them]” (Hartley,2006, p. 1201).

Early work by Diodato (1984) has indicated that bookreviews are rarely cited; hence, citation studies have not beena priority in past years. Nicolaisen (2002a) revived this sub-ject and found that books receiving positive or favorablereviews tend to be cited more often than those receiv-ing neutral or negative comments from a reviewer. In theinternational literature of the social sciences (1997–2001),Nicolaisen (2002b) also examined the share of book reviewscontaining additional references to works other than the bookunder review and found that reviews of this type have beengrowing rapidly (Note that this work inspired the Type Iand Type II classifications for this article.) A review withmany references was characterized as trustworthy or more“scholarly” because the book had been related to previousworks in the field (Nicolasen, 2002b).

According to Hartley (2006), “few studies have been car-ried out to assess the impact of book reviews on scholarlyfields” (p. 1194). The question of impact depends, however,on what is being measured. Lindholm-Romantschuk (1998)focused on the flow of information into or out of a disciplinebased on the proportion of book reviews that are published inthe discipline’s own journals that are reviews of books orig-inating in other disciplines. The following example is given:“if a review of an economics book appears in a history jour-nal, it is counted as one unique of inflow from economics intothe discipline of history” (p. 93). Lindholm-Romantschuk’sinput–output model has shown that books written in sociol-ogy have the most impact on other disciplines since “morethan two thirds of the book reviews are found externally”

(p. 135). In this article, we also are concerned with the impactof reviews. Our objective is to understand more clearly thepossibilities and limitations associated with the influence thatbook reviews have within the scholarly communication sys-tem from a citation perspective, using data from the WoS Artsand Humanities Citation Index.

Book Reviews in the Humanities: DescriptiveStatistics

Bibliometric research techniques using the WoS arenormally applied to three different types of scientificcommunication in biomedicine and the natural sciences:(a) journal articles, (b) reviews, and (c) letters. Other doc-ument types such as meeting abstracts, editorials, and bookreviews are often excluded because they do not play a signif-icant role in scientific communication across these domains.By comparison, the process of communicating knowledge inthe arts and humanities is not typically formed by journalpublications but rather by books or monographs: “Mono-graphs are like the main course of a meal, journal articlesand other scholarly communication are like tapas” (Williams,Stevenson, Nicholas, Watkinson, & Rowlands, 2009, p. 76).This is reflected in part by the referencing pattern, shown inFigure 2, where a large percentage of the references givenby authors publishing in arts and humanities journals are todocuments that have not been processed by the WoS. Booksare the most predominant within this set.

The writing of a book review can have critical implica-tions for the humanities scholar: A well-written review canstrongly reflect or even strongly disapprove of the qualityor significance of a book. Those who publish a book willwant it reviewed in the best possible light since it is likely tobe a prerequisite for both promotion and awarding of tenure(Cronin & La Barre, 2004; Williams et al., 2009). Figure 3shows that book reviews constitute a large portion of the doc-uments housed in Thomson Reuter’s Arts and Humanitiessection of the WoS. In fact, there are 15% more book reviewspublished than journal articles, and of interest are the journalsthat may actually “specialize” in reviewing books.

Humanities fields that publish the most book reviewsin WoS journals include, in ranked order, history, litera-ture, humanities–multidisciplinary, philosophy, and religion(Figure 4). History’s lead role is echoed in Thinking aboutReviews, where Stowe (1991) stated that due to “the increas-ing number of books published each year, the Journal ofAmerican History’s commitment to covering all significantnew books . . . demands that we publish as many reviewsper issue as we can” (p. 593). A trend analysis for the top-reviewing fields (1981–2000) has shown that there has indeedbeen a growth in the number of book reviews published inthe field of history. In the field of literature, there has beena slight decline in reviews published after 1995, and in thethree other disciplines (humanities–multidisciplinary, philos-ophy, and religion), there has been a steady, but overall lower,production of reviews (see Figure 5).

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011 1981DOI: 10.1002/asi

FIG. 2. Share of references toward other Web of Science/Arts & Humanities Citation Index publications and publications outside the Web of Science/Arts &Humanities Citation Index. Journal Subject Categories (1990–2009).

FIG. 3. Percentage of document outputs: Arts & Humanities Citation Index (1981–2009).

1982 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011DOI: 10.1002/asi

FIG. 4. Top-ranking humanities disciplines with the most reviews (1981–2009). [Color figure can be viewed in the online issue, which is available atwileyonlinelibrary.com.]

FIG. 5. Trend analysis of top humanities disciplines publishing the most book reviews (1981–2009). [Color figure can be viewed in the online issue, whichis available at wileyonlinelibrary.com.]

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011 1983DOI: 10.1002/asi

TABLE 2. Perfect match for book and book review referenced in same citing journal article.

Book Review Citing Journal Article

Reference Units Reference Units

Book title /A /D /T /A /D /T

Listening in Paris: JOHNSON, JH 1995 LISTENING PARIS CULT JOHNSON, JH 1995 LISTENING PARIS CULTA Cultural Historyby Johnson, James H.

TABLE 3. Total number of Type I & Type II reviews and citations in history and literature.2

History 1981–2009 Literature 1981–2009465,769 370,458

Total book reviews Total count Total cited %Cited Total count Total cited %Cited

Type I reviews 443,422 10,446 2.4 341,845 6,282 1.8Type II reviews 22,259 1,711 7.7 28,482 1,129 4.0

Retrieving Citation Data From the WoS

The citation data used in this study focuses on the fieldsof history and literature. We selected two fields because theyhave published the most reviews in journals processed bythe WoS from 1981 to 2009 and because a workable datasetwas required due to a lack of coverage related to books.Book reviews are indexed in the WoS as unique documents,but books are not, and this currently makes it difficult todetermine co-citation rates between books and book reviewsreferenced by journal articles.1 Another reason for select-ing two fields relates to the probability of focusing on bookreviews in a real evaluation context, and we suggest for nowthat it may be of most interest for historians and literaryscholars.

The boundaries for the field of history are delineated byWoS journal subject categories, and include history, the his-tory of social sciences, and the history & philosophy ofscience. Likewise, we use the journal subject categories forthe field of literature (i.e., literary theory & criticism; literaryreviews; literature–African, Australian, Canadian, American,British Isles, German, Dutch, and Scandinavian; literature–romance, Slavic). Next, we list the steps taken to retrievecitation and co-citation data:

1. Collect book reviews written in history and literature(1981–2009) and remove duplicates where the samereview is assigned to more than one journal subjectcategory.

2. For each book review (Types I and II), obtain a list of citedreferences.

1Currently, Thomson ISI is improving its WoS coverage in the SSH bydeveloping an index of books, but for the Year 2011, it will include onlythose with a copyright date of 2003 to the present. At the time that thisresearch was carried out, this index was not yet available.

3. Identify the book as it appears in the book review’s refer-ence list. With a Type I review, the isolation procedure isstraightforward: If there is only one reference, it is usuallyto the book under review. Type II reviews require the useof a complex selection algorithm to isolate the book fromadditional references.

4. Determine which book reviews have received citations,and for each year from 1981 to 2009, calculate the aver-age number of citations per publication (CPP) as the ratiobetween the total number of reviews published and thesum of the citations received in that year.

5. Among the reviews cited in journal articles, determine howmany were co-cited with the reviewed book. Retrievingco-citation counts requires matching the book’s author,publication date, and title appearing in the reference listof a citing journal article, with the same iteration of author,publication date, and title appearing in the reference listof the review article (see Table 2).

Results: Review Types and Their Impact

The most visible document processed for the WoS Arts &Humanities Index is the book review (see Figure 2); however,Table 32 indicates that the total number of citations to bookreviews in journal articles is quite low. From the period of1981 to 2009, 2% of book reviews published in both historyand literature, and referencing the book only (Type I), werecited. Type II reviews, or those that give reference to the bookand other scholarly sources, received slightly more citations:8% in history and only 4% in literature.

Figures 6 and 7 illustrate long-term book-reviewingimpacts in history and literature, where impact is measuredas CPP and calculated by dividing the total number of bookreviews published in a given year by the sum of citation

2Eighty-eight book reviews in history and 131 book reviews in literaturecould not be assigned a type due to a lack of references.

1984 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011DOI: 10.1002/asi

FIG. 6. Publication years of history book reviews and citations per publication (1981–2009). [Color figure can be viewed in the online issue, which isavailable at wileyonlinelibrary.com.]

FIG. 7. Publication years of literature book reviews and citations per publication (1981–2009). [Color figure can be viewed in the online issue, which isavailable at wileyonlinelibrary.com.]

counts that they received for that year. There is a slight timeeffect where book reviews processed for the WoS in 2008and 2009 have not been published long enough to receivecitations. Figure 6 indicates that history book reviews givingreference to both the book under review and other schol-arly sources (Type II) received more CPP than did Type I(i.e., references only the book) reviews, and dramatically so;but for the field of literature, the CPP values for the two types

barely differ, with Type II reviews showing only a slightlygreater impact from 1991 and 2001 (see Figure 7). During this10-year period, the number of Type II book reviews publishedin literature declined, thus leading to a similar drop in averagecitation rates.

In Table 4, we present co-citation counts resulting fromthe book as it appeared in the citing document’s reference listand the book as it appeared in the book review’s reference

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011 1985DOI: 10.1002/asi

TABLE 4. Matching books in citing document and book review. Co-citation counts.

History Literature

Type I Type II Type I Type II

Book title matched in citing document and review 5,092 1,901 2,472 704Book title and publication date matched in citing document and review 4,513 1,433 1,668 440Book title and book author matched in citing document and review 4,340 1,302 2,121 556All three variables—title, author, date—matched in the citing document and review 3,896 1,101 1,442 359

TABLE 5. Percentage of book reviews co-cited with book (history and literature).

%Reviews co-citedNo. co-cited with book (minimum–maximum values)

No. reviews cited Perfect match Title only Perfect match Title only

HistoryType I 10,446 3,593 4,576 0.34 0.44Type II 1,711 631 820 0.37 0.48

LiteratureType I 6,282 1,301 2,174 0.21 0.35Type II 1,129 248 450 0.22 0.40

TABLE 6. Book review impacts (CPP) relative to four different citing options.

No. of reviews cited/ Sum of citation CPP (minimum– maximumCiting options co-cited counts values)

HistoryType I 10,446 14,074 1.35Type II 1,711 3,516 2.05Type I + Book 3,593–4,576 3,896–5,092 1.08–1.11Type II + Book 631–820 1,101–1,901 1.74–2.31

LiteratureType I 6,282 8,501 1.35Type II 1,129 1,612 1.43Type I + Book 1,301–2,174 1,442–2,472 1.11–1.14Type II + Book 248–450 359–704 1.45–1.57

list. Matching errors occurred most often at the level of theauthor, and second, with the year. In cases where the authorname did not match, we found either a misspelling of theauthor’s surname or an omission of one initial. If we founda mismatch due to the year, this was because it was eithermissing from one document or written incorrectly. Addi-tional errors occurred if one document referenced the bookeditor’s surname and the other referenced the author whowrote a chapter in the same edited book.Yet another problemoccurred when the book title was part of a series and appearedmore than once in a reference list with a different author pervolume.

Data retrieved from the perfect match procedure (i.e., allthree units) is the most accurate, albeit not perfectly reliablein the sense that other ‘intended’ co-citations are missed. Asa result, we calculated the minimum to maximum percent-ages of book reviews that were co-cited with the book, fromthe total number of book reviews cited in both history and

literature. Table 5 now shows that for history and literature,slightly more reviews classified as Type II (i.e., referencingthe book and other scholarly sources) were co-cited with thereviewed book than were those classified as Type I. In history,approximately 37 to 48% of the book reviews co-cited withthe book may have had some influence on how that bookwas received by the author of a journal article. We do notknow whether the influence was positive or negative; thus,content analyses of these reviews would be the next step forunderstanding the meaning of the co-citations.

A final summary of findings based on the citing optionsoutlined in the Introduction (see Table 1) is presented inTable 6. Of interest is the citation impact calculated in termsof CPP values for each of the citing options. For all bookreviews that were co-cited with the book (i.e., Type I + Book;Type II + Book), we include minimum to maximum values.Again, the first value relates to the perfect matching proce-dure (i.e., title, author, date), and the second value to the

1986 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011DOI: 10.1002/asi

matching procedure based on title only. Type II reviews havethe most impact (in bold), thus providing clear evidence oftheir scholarly role. Since citations are by authors of journalarticles only, more insight is to be gained if we also couldobtain citation counts given by authors of books.

Citing Reviews Alone or With Book

Here, we focus on the following question: In which cir-cumstance would an author cite a book review alone in ajournal article or co-cite the review with the reviewed book?Our data-retrieval method indicates that this is possible, butdoes it mean that there is a strong contextual differencebetween the two cases, or is this mostly a WoS artifact? Toaddress this issue, we took a random sample of recently pub-lished reviews from the history dataset and carried out a smallcontent analysis (see Appendix).

The purpose of this content analysis was to obtain hintsas to where a more comprehensive study might take us. Itis not part of the scope of this bibliometric study to engagein a full-scale qualitative analysis, but initial insights aboutwhy a review was cited alone or why it was co-cited with thebook help to determine whether it makes sense to develop abibliometric weighting system for evaluation purposes. Forinstance, if a book review has not been cited, we could applya weight of “0” to a total citation count of “0.” If the bookreview was cited alone, we could apply a weight of “.5” tothe total citation count, and if a book review was co-citedwith the book, we could apply a weight of “1” to the totalcitation count. Our rationale for granting the highest weightto a review co-cited with a book is based on the notion thatit has had some influence on how the content of a book wasreceived. Research already has confirmed that books receiv-ing favorable reviews tend to be cited more often than dothose receiving negative reviews (e.g., Nicolaisen, 2002a);hence, we suggest that a co-citation may be a reflection ofthe reviewer’s positive or negative influence.

A brief content analysis indicates that book reviews arecited because of their scholarly value. Authors of journal arti-cles in history do not necessarily elaborate on their reasons forciting a book review, but do indicate how critical the reviewerwas of the book’s thesis and may comment on the status ofthe reviewer (i.e., whether he or she is a well-known scholarin the same field as the author of the published book). Anauthor also might state why he or she was “inspired” by abook review to adopt a particular term or theoretical stand-point. Moreover, we know that authors of book reviews alsomay cite other reviewers or persons who have in fact reviewedtheir own books (see Appendix).

A book review can be influential to an author writing ascholarly journal article, but this influence seems to makelittle difference whether it was cited alone or co-cited withthe book. In the absence of a formal “rule” for how creditmust be given to book reviews and books (i.e., separatelyor together), it is common for historians to use a footnotereference style. With a footnote, the author may separatelyreference a review and a book, but an author also may give

credit to both within the same note. Later, the WoS processesthe footnote as one reference to the review and not to thebook. In the sciences and the social sciences, where we seedifferent norms for citing, it is easier to detect co-citations ifan author gives distinct credit to the review and the book ina full reference list at the end of an article.

Book Reviews in Humanities ResearchEvaluations

To carry out valid bibliometric evaluations, robust cita-tion counts are essential; hence, some bibliometricians mightargue that there is little incentive to focus on book reviewsknowing that they are so infrequently cited. Others will rec-ognize that the standards set for meaningful citation-basedindicators in the natural sciences are not directly applicableto the humanities and may need to be accommodated to thisfield. In a research-evaluation context, we give considerationto the value of a scholarly book review, including surround-ing circumstances such as the quality of the journal in whichit was published and whether the journal’s editor invited (andperhaps paid) the scholar to write the review. A first step intaking book reviews seriously may be to count or includethem in assessment studies, thus making them more visible.As long as they are counted, a new type of indicator couldbe valuable if it is used in conjunction with other forms ofevaluation.

A “healthy” impact metric is not feasible at theindividual-author level; however, it may be useful to evaluatejournals that have published book reviews over a significantperiod of time, based on the influence that some have hadwithin the scholarly communication system. This means thatwe focus on the journal as a benchmark for the individ-ual’s reviewing performance and reward individuals whocontribute to quality review journals. A ranking may beformulated in one of two ways:

• An indicator termed the Book Review Influence Share (BRIS),which recognizes the inherent value of a review, but distin-guishes between reviews that are not cited and those thathave been cited in the journal literature. With this, we givea weighted value of 1 to the book reviews not cited and aweighted value of 1.5 to the book reviews that have beencited.

BRIS =(

(W1 × BRNotCited) + (W1.5 × BRCited)

Number Of Re views

)− 1

• Another indicator termed the Book Review Influence Fac-tor (BRIF), which places emphasis on the total number ofcitations that a set of book reviews has received. Here, weconsidered applying weights to reviews cited alone versusreviews co-cited with the book, but we do not yet have strongqualitative evidence to apply such a weighting system. For themoment, this indicator is basic and constitutes the summingof all citations given to each cited book review, divided bytotal number of reviews.

BRIF =∑

i ci

n

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011 1987DOI: 10.1002/asi

TABLE 7. Top ranking History journals (n = 35) based on the BRIS and BRIF (1981–2009).

Journal title (abbreviated) No. reviews No. cited %Type II cited Total citations BRIS BRIF

Historical Reflections-Reflexions Historiques 15 9 73% 11 30% 0.73Past & Present 14 6 57% 31 21% 2.21Biology & Philosophy 230 77 87% 192 17% 0.83Social Studies Of Science 263 79 58% 297 15% 1.13Studies In History And Philosophy Of Science 165 49 72% 118 15% 0.72Science And Engineering Ethics 7 2 50% 2 14% 0.29Geschichte Und Gesellschaft 50 14 50% 25 14% 0.50British Journal For The Philosophy Of Science 561 147 57% 310 13% 0.55History And Theory 601 155 57% 343 13% 0.57Synthese 67 14 57% 23 10% 0.34Historical Methods 125 26 54% 55 10% 0.44Philosophy Of Science 519 96 36% 164 9% 0.32Social Science History 38 5 40% 7 9% 0.18Archiv Fur Reformationsgeschichte-Archive For 29 5 60% 6 9% 0.21

Reformation HistoryJournal Of The History Of Medicine And Allied 885 146 6% 162 8% 0.18

SciencesAcadiensis 25 4 50% 9 8% 0.36Vierteljahrshefte Fur Zeitgeschichte 40 6 20% 7 8% 0.18Minerva 342 50 66% 70 7% 0.20History Of The Human Sciences 238 33 67% 67 7% 0.28British Journal For The History Of Science 2337 297 6% 343 6% 0.15Diplomatic History 646 71 42% 145 5% 0.22Annals Of Science 3101 335 23% 375 5% 0.12Journal Of The History Of Ideas 94 10 20% 20 5% 0.21Reviews In American History 3265 346 34% 497 5% 0.15Isis 9016 915 3% 1053 5% 0.12French Historical Studies 40 4 0% 20 5% 0.50Huntington Library Quarterly 121 12 0% 17 5% 0.14Bulletin Of The History Of Medicine 2219 209 5% 256 5% 0.12Quaderni Storici 93 8 25% 8 4% 0.09American Journal Of Bioethics 107 9 56% 12 4% 0.11Public Understanding Of Science 141 11 27% 11 4% 0.08Technology And Culture 4206 328 1% 404 4% 0.10Southern Cultures 261 19 5% 32 4% 0.12Contemporary European History 14 1 0% 1 4% 0.07

An example of how the BRIS and the BRIF may becalculated is shown for the Review of American History(1981–2009).

BRIS(Rev.Am.Hist.) =

⎛⎜⎜⎝

(W1 × BRNotCited)

+ (W1.5 × BRCited)

TotalNumberOf Re views

⎞⎟⎟⎠ − 1

=(

(1 × 2,919) + (1.5 × 346)

3,265

)− 1

=(

2,919 + 519

3,265

)− 1

=(

3438

3,265

)− 1

= 1.050 − 1

= 0.050

= 5%

BRIF(Rev.Am.Hist.) =∑

i ci

n= 497

3265= 0.15

Table 7 presents the top-ranking WoS history journals(n = 35) based on the new BRIS and the BRIF3 indicators(sorted respectively) for the citation period of 1981 to 2009.If we compare certain journals, we see that one may presenta higher output of book reviews than may another (e.g., Isisn = 9,016 vs. History and Theory n = 601), but this does notnecessarily mean that it has produced the higher share ofinfluential reviews (e.g., Isis BRIS = 5% vs. History and The-ory BRIS = 13%). Only 3% of the Type II reviews were citedfrom Isis compared to 57% from History and Theory, thuscontributing to a much higher BRIF value for the later jour-nal. Bibliometric evidence now shows that Type II reviewsoverall tend to be cited more frequently than do Type I, butwithout such evidence, it makes sense to assume that a reviewthat functions more like a literary critique will be cited more.Again, this type of review is considered to be more scholarlybecause it engages the reader in a wider discussion of theacademic background surrounding the book’s contribution(Nicolaisen, 2002b).

3For the full title of the journal, refer to theWoS Journal TitleAbbreviationsat http://images.isiknowledge.com/WOK45/help/WOS/E_abrvjt.html

1988 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011DOI: 10.1002/asi

Since we are using the WoS journal subject categories toselect all journals related to the humanities discipline of his-tory, we also recognize that some journals are related to otherdisciplines. For example, the journal Social Sciences His-tory is categorized as both as a history-related and a socialscience-related journal. The same holds true for Social Stud-ies of Science, and with this journal, we observe strong BRISand BRIF values given that it had published S. Shapin’swell-cited review of Bruno Latour’s notable book Sciencein Action. Naturally, outliers (i.e., exceptional reviews ofnotable books) will play a role in the inflation of these newindicator values, and there will be differences related to howeditors of multidisciplinary journals select reviews to publishand how authors in different disciplines later use them.

Conclusions

In the present study, we examined book reviews as schol-arly pieces of information and their influence on the scholarlycommunication system within the humanities. We focusedon two prominent reviewing disciplines, history and litera-ture, and we used the WoS as a data source for observinggeneral output and citation patterns to book reviews. Thereare clear advantages and disadvantages associated with thisdata source. One advantage is that it is an index to qualityinternational journals in the sciences, social sciences, andhumanities, and has been a tried and true source for cita-tion analyses in the past, particularly for research-evaluationpurposes. With this database, it is possible to identify andisolate book reviews from other types of documents, clas-sify reviews according to their reference lists (i.e., Type Ivs. Type II), and detect the degree to which the two typeshave been cited in journal articles. Nevertheless, book reviewsare intrinsically linked to books, and the challenge of usingthe WoS rests upon identifying books as unique documentsin reference lists, and obtaining clear information about therelationship between book reviews and books, specifically inco-citation.

In our analyses of book reviews referencing the book(Type I) and book reviews referencing the book as well asother scholarly sources (Type II), we found that Type IIreviews tend to be cited more often than do Type I reviews.The citation counts are not robust unless we focus on a 20- to30-year period; therefore, it is difficult to use them for sophis-ticated individual-level measures, akin to those used in thenatural sciences. Nevertheless, a reward system may be cre-ated for scholars who contribute to quality review journals. Ifthe number of Type II scholarly reviews grows significantlyin coming years, the process of monitoring citations is morepromising. We also may find that if data providers such asThomson Reuters (WoS) and Elsevier (Scopus) expand theirscope to include more humanities-related journals in additionto an index of books, the researcher’s ability to evaluate theimpact that reviews have on books will be much improved.Access to both journal and book citation data through theWoS is just one external condition that will make it easier to

generate more reliable and valid statistics. Some bibliometri-cians might choose not to wait for external developments, andfocus instead on devising more sophisticated analytic toolsfor mining citation data from Google Scholar or other digitalrepositories with book-related data.

In conclusion, we have proposed the development of aset of indicators for monitoring WoS journals, which havebecome prominent outlets for book reviews. Our indicatorsmay be used with other similar data sources (e.g., Elsevier’sScopus), and although they are preliminary and experimental,we consider them to be a starting point for further in-depthanalyses. There are other patterns to consider, such as thecorrelation between BRIFs and the general Impact Factors ofbook-reviewing journals.We have yet to examine the relation-ship between scholars writing book reviews and their overallproductivity within a WoS context. More work needs to bedone to understand the book-reviewing culture in general,and in particular, why certain books receive more scholarlyreviews than do others, the degree to which a book’s authorand the reviewer are peers from the same discipline, and themotivation that scholars have for writing different types ofreviews (i.e., Type I vs. Type II).

Acknowledgments

We thank Ludo Waltman and Rodrigo Costas for engagingin many discussions regarding the development of the BRISand BRIF indicators for book reviews. We also acknowledgethe detailed and helpful comments provided to us by ouranonymous JASIST reviewers.

References

Archambault, E., & Vignola Gagné, E. (2004). The use of bibliometricsin the social sciences and humanities. Science-Metrix, Final Report. Pre-pared for the Social Sciences and Humanities Research Council of Canada(SSHRCC).

Blake, V. (1989). The role of reviews and reviewing media in the selectionprocess: An examination of the research record. Collection Management,11(1/2), 1–40.

Borgman, C.L. (1990). Editor’s introduction. In C.L. Borgman (Ed.), Schol-arly communication and bibliometrics (pp. 10–27). Newbury Park, CA:Sage.

Cortada, J.W. (1998). Five ways to be a terrible book reviewer. Journal ofScholarly Publishing, 30(1), 34–37.

Cronin, B., & La Barre, K. (2004). Mickey Mouse and Milton: Bookpublishing in the humanities. Learned Publishing, 17, 85–98.

Diodato, V. (1984). Impact and scholarliness in arts and humanities bookreviews: A citation analysis. Proceedings of the 47th annual meeting ofthe American Society for Information Science, 21, 217–221.

Furnham, A. (1986). Book reviews as a selection tool for librarians:Comments from a psychologist. Collection Management, 8(1), 33–43.

Glenn, N. (1978). On the misuse of book reviews. Contemporary Sociology,7(3), 254–255.

Hartley, J. (2006). Reading and writing book reviews across disciplines.Journal of the American Society for Information Science and Technology,57(9), 1194–1207.

Hicks, D., & Wang, J. (2009). Towards a bibliometric database for the socialsciences and humanities. Retrieved from http://works.bepress.com/cgi/viewcontent.cgi?article=1017&context=diana_hicks

Huang, M.-H., & Chang, Y.-W. (2008). Characteristics of research output insocial sciences and humanities: From a research evaluation perspective.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011 1989DOI: 10.1002/asi

Journal of the American Society for Information Science and Technology,59(11), 1819–1828.

Kling R.K., & McKim, G. (1999). Scholarly communication and the con-tinuum of electronic publishing. Journal of the American Society forInformation Science, 50(10), 890–906.

Kousha, K., & Thelwall, M. (2009). Google book citation for assessinginvisible impact? Journal of theAmerican Society for Information Scienceand Technology, 60(8), 1537–1549.

Lewison, G. (2001). Evaluation of books as research output in the history ofmedicine. Research Evaluation, 10(2), 89–95.

Lewison, G. (2004). James Bond and citations to his books. Scientometrics,59(3), 311–320.

Lindholm-Romantschuk, Y. (1998). Scholarly book reviewing in the socialsciences and humanities. The flow of ideas within and among disciplines.Westport, CT: Greenwood Press.

Miranda, E.O. (1996). On book reviewing. Journal of Educational Thought,39(2), 191–202.

Moed, H.F., Linmans, J., Nederhof, A., Zuccala, A., López Illescas,C., & de Moya Anegón, F. (2009). Options for a comprehen-sive database of research outputs in social sciences and humanities(Version 6). Retrieved from http://www.dfg.de/download/pdf/foerderung/grundlagen_dfg_foerderung/informationen_fachwissenschaften/geisteswissenschaften/annex_2_en.pdf

Natowitz,A., &Wheeler Carlo, P. (1997). Evaluating review content for bookselection: An analysis of American history reviews in Choice, AmericanHistorical Review, and Journal of American History. College & ResearchLibraries, 58(4), 322–335.

Nicolaisen, J. (2002a). The J-shaped distribution of citedness. Journal ofDocumentation, 58(4), 383–395.

Nicolaisen, J. (2002b). The scholarliness of published peer reviews: A bib-liometric study of book reviews in selected social science fields. ResearchEvaluation, 11(3), 129–140.

Nederhof, A.J. (2006). Bibliometric monitoring of research performance inthe social sciences and the humanities: A review. Scientometrics, 66(1),81–100.

Parker, J.M. (1989). Scholarly book reviews in literature journals as col-lection development sources for librarians. Collection Management,11(1&2), 41–57.

Serebnick, J. (1992). Selection and holdings of small publishers’ booksin OCLC libraries: A study of the influence of reviews, publishers andvendors. Library Quarterly, 62(3), 259–294.

Spink, A., Robbins, D., & Schamber, L. (1998). Use of scholarly bookreviews: Implications for electronic publishing and scholarly communi-cation. Journal of the American Society for Information Science, 49(4),364–374.

Stowe, S. (1991). Thinking about reviews. Journal of American History,78(2), 591–595.

Torres-Salinas, D., & Moed, H.F. (2009). Library catalog analysis asa tool in studies of social sciences and humanities: An exploratorystudy of published book titles in economics. Journal of Informetrics, 3,9–26.

University of Alberta. (2010). Department of History and Classics. BookReview Writing Guide. Retrieved from http://www.uofaweb.ualberta.ca/historyandclassics/BookReviewWritingGuide.cfm

Williams, P., Stevenson, I., Nicholas, D., Watkinson, A., & Rowlands, I.(2009). The role and future of the monograph in arts and humani-ties research. ASLIB Proceedings: New Information Perspectives, 61(1),67–82.

1990 JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011DOI: 10.1002/asi

Appendix

Article citing review Review title Notes

Book review and book co-cited (i.e., in separate footnotes)

Preston, A. (2006). Bridging the Gapbetween the Sacred and the Secular in theHistory of American Foreign Relations.

Leffler, M.P. (1999). We now know:Rethinking Cold War history. (review ofGaddis, J.L., 1997)

Few scholars know as much about the origins of the Cold Waras John Gaddis and Melvyn Leffler, yet on the most importantquestions of causation they do not agree.97

Goodman, J. (1998). For the Love ofStories.

Rabinowitz, H.N. (1993). The Promiseof the New South—Life AfterReconstruction. (review of Ayers, E.L.,1992)

41For an example of a critic pointing his finger at post-modernismfor a historian’s experiment, see Howard N. Rabinowitz,“The Origins of a Poststructural New South: A Review ofEdward L. Ayers’ The Promise of the New South: LifeAfter Reconstruction” Journal of Southern History 59(August 1993): 505–15.

Kamrath, M.L. (2001). Charles BrockdenBrown and the “Art of the Historian:”An Essay Concerning (Post)ModernHistorical Understanding.

Haskell, T.L. (1998). Beyond the GreatStory: History as Text and Discourse.(review of Berkhofer, R.F., 1997)

Berkhofer. . .challenges us to not only “surmount the dilemma ofrepresentationalism or the semiotic absolute” but to also dealwith issues of multicultural representation and anachronism aswell as new ways of representing the past.86

86One of the most recent assessments of Berkhofer’s studyis Thomas Haskell’s review essay in History and Theory, 37(October 1998), 347–369.

Book review cited alone (i.e., in one footnote)

Best, J. (2004). Deviance May Be Alive,But Is It Intellectually Lively? A Reactionto Goode.

Valverde, M. (2000). Controlling Vice:Regulating Brothel Prostitution inSt. Paul, 1865–1883. (review of Best,J., 1998)

I [the Author] was pleased when my monograph on nineteenth-century brothels was reviewed in the Journal of AmericanHistory, but bemused by the review’s first sentence: “This studyis meant primarily as a contribution to a field whose centralconcept has been thoroughly discredited but that nonethelessrefuses to make a graceful exit from curricula: the sociology of‘deviance.”’ (Valverde, 2000, p. 1802).

Cutcliffe, S.H. (2010). Travels In andOut of Town. William Cronon’s Nature’sMetropolis: Chicago and the Great West.

Hays, S.P. (1992). Natures Metropolis-Chicago and the Great West. (review ofCronin, W., 1991)

Thus it was that environmental historians Richard White andSamuel Hays could refer to the book as “extraordinary” andas “innovating and exciting” while at the same timedeclaring Cronon’s “linkage of capital and nature . . .

sometimes . . . problematic” (White) and his analysis “highlyselective” (Hays).4

Reich, S.A. (2009). The Great Migrationand the Literary Imagination.

Coclanis, P.A. (2004). Generations ofCaptivity: A History of African-AmericanSlaves. (review of Berlin, I., 2003)

3See also Peter Coclanis, “The Captivity of a Generation,” reviewof Generations of Captivity: A History of African-AmericanSlaves, by Ira Berlin, William and Mary Quarterly 61:3(July 2004): 544–556. 90.53Two sources inspire my use of the term morality tale, both ofwhich come from critiques of the excessive use of agency andthe tendency toward romanticization in recent historicalscholarship, one in the writing of American labor history andthe other in the writing of the history of American slavery.On labor history, see Eric Arnesen, “Passion and Politics: Raceand the Writing of Working-Class History,” Journal of theHistorical Society 6:3 (September 2006): 323–356; on slavery,see Coclanis, “Captivity of a Generation,” 544–556.

JOURNAL OF THE AMERICAN SOCIETY FOR INFORMATION SCIENCE AND TECHNOLOGY—October 2011 1991DOI: 10.1002/asi