STM social bookmarking (by S. Haustein)

September 9th 2010

Mit

glie

d d

er

Helm

holt

z-G

em

ein

sch

aft

Journal Evaluation and Science 2.0Using Social Bookmarks to Analyze Reader Perception

| Stefanie Haustein, Evgeni Golov, Kathleen Luckanus, Sabrina Reher & Jens Terliesner

Outline

1. Introduction

Multidimensional Journal Evaluation

Journal Usage

Social Bookmarking in STM

2. Retrieval of Bookmarking Data

3. Journal EvaluationUsage Indicators

Tags

4. Conclusions

5. Literature

Introduction: Multidimensional Journal Evaluation

one indicator cannot measure true influence of periodicals

five dimensions of journal evaluation are identified

test set: 45 solid state physics

journals all publications from

2004 to 2008 bibliographic data for

168,109 documents from Web of Science

MULTIDIMENSIONALJOURNAL

EVALUATION

Introduction: Journal Perception

measuring problematic despite standards only available for usage at own institutions publishers shut global statistics away alternative needed to measure worldwide journal usage

bookmarking articles on social bookmarking platforms indicates journal usage

Introduction: Social Bookmarking in STM

modeled on delicious

allow academic users to store references online

indexing resources with tags sharing literature with

other users finding new content

through other users


CiteULike Connotea BibSonomy 2collab

Launch 2004 2004 2006 2007

Responsibility Oversity Ltd. sponsored by Springer

Nature Publishing Group

Universityof Kassel

Elsevier B.V.

Unique visitors1 360,000 380,000 350,000 40,000

Total visits2 480,000 690,000 620,000 44,000

Page views3 2,100,000 3,100,000 2,800,000 180,000

1 Approximate number of estimated cookies on a site for April 2010 (Google, Inc., 2010)2 Estimated number of times a site is accessed by unique visitors in April 2010 (Google, Inc., 2010).3 Total estimated number of times pages on a site have been accessed by users (Google, Inc., 2010)

Retrieval of Bookmarking Data

bookmarks from BibSonomy, CiteULike and Connotea to articles published in 45 Physics journals from 2004 to 2008

search strategies applied to compensate incomplete and erroneous bookmarking data journal title abbreviations ISSNs DOIs of articles

download of bookmarks via API or by parsing


completeness / correctness of bookmarking data

0%

10%

20%

30%

40%

50%

60%

70%

80%

90%

100%

journal title DOI ISSN journal abbreviation

Perc

enta

ge o

f boo

kmar

ks f

ound

by

sear

ch st

rate

gy

Search stragey

all services BibSonomy CiteULike Connotea


completeness / correctness of bookmarking data

26.82%

65.11%

21.19%

38.61%

58.67%

32.34%

60.70%

60.26%

14.37%1.81%

18.00%

1.13%0.14% 0.74% 0.11%

all services BibSonomy CiteULike Connotea

all fields (journal title, abbreviation, ISSN, DOI)

three fields

two fields

one field


download was checked and limited to bookmarks to documents published in 45 journals from 2004 to 2008

13,760 correct bookmarks retrieved

940

10778

2042

BibSonomy

CiteULike

Connotea

Number of bookmarks retrieved


missing DOIs were replaced by CrossRef and manually bookmarks were matched to the bibliographic entries from

Web of Science via DOI 98.9% of bookmarks matched

2.8% doubles in user names between services

results were combined

145 users

2054 users

313 users

802 articles

8082 articles

1995 articles

BibSonomy

CiteULike

Connotea

Number of unique users and articles


13,608 bookmarks were matched to 10,280 articles 2,441 unique users

0

50

100

150

200

250

300

350satoshi (322 posts)

bronckobuster (238 posts)rice (234 posts)

Num

ber o

f bo

okm

arks

per

use

r

Users from BibSonomy, CiteULike and Connotea

1,179 users posted one article 75% of content is created by 21%

of users 8,511 articles were only

bookmarked once

Journal Evaluation: Usage Indicators

bookmarking an article indicates journal usage

Usage ratio journal articles with bookmarks compared to all publications

0

500

1000

1500

2000

2500

0%

10%

20%

30%

40%

50%

60%

70%

RE

V M

OD

PH

YS

RE

P P

RO

G P

HY

SP

HY

S R

EP

NE

W J

PH

YS

PH

YS

RE

V E

SO

FT

MA

TT

ER

J S

TAT

ME

CH

PH

YS

RE

V A

EU

R P

HY

S J

EE

PL

AC

T C

RY

ST

A

AP

PL

PH

YS

LE

TP

HY

S F

LU

IDS

EU

R P

HY

S J

B

J S

TAT

PH

YS

AN

N P

HY

SJ

PH

YS

CO

ND

EN

S M

AT

TE

RN

AN

OT

EC

HN

OL

J AP

PL

PH

YS

J P

HY

S A

J P

HY

S D

IEE

E N

AN

OT

EC

HN

OL

CO

MP

UT

MA

TE

R S

CI

AP

PL

PH

YS

AJ

LO

W T

EM

P P

HY

S

PH

YS

STA

T S

OL

BN

UC

L I

NS

TR

UM

ME

TH

AP

HY

S L

ET

T A

PH

YS

STA

T S

OL

AS

UP

ER

CO

ND

SC

I T

EC

HN

OL

PH

YS

SC

RN

UC

L I

NS

TR

UM

ME

TH

B

PH

YS

TO

DA

YJE

TP

LE

TT

INT

J T

HE

RM

OP

HY

S

AC

T C

RY

ST

BJ

VA

C S

CI

TE

CH

NO

L A

J M

AG

N M

AG

N M

AT

ER

PH

YS

ICA

C

J R

HE

OL

PH

YS

ICA

BP

RA

MA

NA

HY

PE

RF

INE

IN

TE

RA

CT

SO

LID

STA

TE

IO

NP

HY

S S

OL

ID S

TAT

E

Num

ber o

f bo

okm

arke

d ar

tice

ls

Usa

ge ra

tio

articles bookmarked

share of articles bookmarked


Usage diffusion number of unique users per journal depicts the diffusion of journal content into the community

0

100

200

300

400

500

600

700

800

900

PH

YS

RE

V E

AP

PL

PH

YS

LE

TJ A

PP

L P

HY

SP

HY

S R

EV

AJ

PH

YS

CO

ND

EN

S M

AT

TE

RR

EV

MO

D P

HY

SN

EW

J P

HY

SE

PL

J P

HY

S A

NA

NO

TE

CH

NO

LP

HY

S R

EP

EU

R P

HY

S J

BJ

STA

T M

EC

HR

EP

PR

OG

PH

YS

J P

HY

S D

PH

YS

FL

UID

SP

HY

S L

ET

T A

NU

CL

IN

ST

RU

M M

ET

H A

AP

PL

PH

YS

AJ

MA

GN

MA

GN

MA

TE

RE

UR

PH

YS

J E

PH

YS

STA

T S

OL

BJ

STA

T P

HY

SN

UC

L I

NS

TR

UM

ME

TH

BS

OF

T M

AT

TE

RP

HY

S S

TAT

SO

L A

PH

YS

ICA

BP

HY

S S

CR

PH

YS

TO

DA

YC

OM

PU

T M

AT

ER

SC

IS

UP

ER

CO

ND

SC

I T

EC

HN

OL

PH

YS

ICA

CA

CT

CR

YS

T A

JET

P L

ET

TJ

LO

W T

EM

P P

HY

SA

NN

PH

YS

J V

AC

SC

I T

EC

HN

OL

AIE

EE

NA

NO

TE

CH

NO

LP

RA

MA

NA

SO

LID

STA

TE

IO

NIN

T J

TH

ER

MO

PH

YS

AC

T C

RY

ST

BH

YP

ER

FIN

E I

NT

ER

AC

TP

HY

S S

OL

ID S

TAT

EJ

RH

EO

L

Num

ber o

f un

ique

use

rs


Article Usage Intensity average number of bookmarks per article shows how often articles are read

0

1000

2000

3000

0.0

1.5

3.0

4.5

RE

V M

OD

PH

YS

PH

YS

RE

P

RE

P P

RO

G P

HY

S

J S

TAT

ME

CH

EU

R P

HY

S J

B

PH

YS

RE

V E

IEE

E N

AN

OT

EC

HN

OL

NE

W J

PH

YS

AC

T C

RY

ST

A

J P

HY

S A

EU

R P

HY

S J

E

EP

L

J P

HY

S C

ON

DE

NS

MA

TT

ER

PH

YS

RE

V A

J R

HE

OL

SU

PE

RC

ON

D S

CI

TE

CH

NO

L

PH

YS

TO

DA

Y

PH

YS

ICA

C

J S

TAT

PH

YS

PH

YS

FL

UID

S

SO

FT

MA

TT

ER

AP

PL

PH

YS

LE

T

AN

N P

HY

S

PH

YS

LE

TT

A

J M

AG

N M

AG

N M

AT

ER

NA

NO

TE

CH

NO

L

J P

HY

S D

AC

T C

RY

ST

B

JET

P L

ET

T

PH

YS

STA

T S

OL

A

J AP

PL

PH

YS

NU

CL

IN

ST

RU

M M

ET

H A

CO

MP

UT

MA

TE

R S

CI

INT

J T

HE

RM

OP

HY

S

AP

PL

PH

YS

A

PH

YS

STA

T S

OL

B

NU

CL

IN

ST

RU

M M

ET

H B

J L

OW

TE

MP

PH

YS

PH

YS

SC

R

J V

AC

SC

I T

EC

HN

OL

A

PH

YS

ICA

B

HY

PE

RF

INE

IN

TE

RA

CT

PH

YS

SO

LID

STA

TE

PR

AM

AN

A

SO

LID

STA

TE

IO

N

Num

ber o

f bo

okm

arks

Art

icle

usa

ge in

tens

ity

number of bookmarks

article usage intensity


Journal Usage Intensity mean number of articles per user measures how intensive the average user reads the journal

0

100

200

300

400

500

600

700

800

0.00

1.00

2.00

3.00

4.00

5.00

PH

YS

RE

V A

AP

PL

PH

YS

LE

T

J AP

PL

PH

YS

NU

CL

IN

ST

RU

M M

ET

H A

NU

CL

IN

ST

RU

M M

ET

H B

PH

YS

RE

V E

PH

YS

FL

UID

S

PH

YS

ICA

C

SO

FT

MA

TT

ER

J L

OW

TE

MP

PH

YS

J P

HY

S C

ON

DE

NS

MA

TT

ER

INT

J T

HE

RM

OP

HY

S

PH

YS

ICA

B

J V

AC

SC

I T

EC

HN

OL

A

NA

NO

TE

CH

NO

L

J M

AG

N M

AG

N M

AT

ER

HY

PE

RF

INE

IN

TE

RA

CT

CO

MP

UT

MA

TE

R S

CI

PH

YS

SC

R

EP

L

J P

HY

S D

EU

R P

HY

S J

E

PH

YS

LE

TT

A

AC

T C

RY

ST

A

PH

YS

STA

T S

OL

B

JET

P L

ET

T

J R

HE

OL

SO

LID

STA

TE

IO

N

PH

YS

STA

T S

OL

A

J S

TAT

PH

YS

NE

W J

PH

YS

IEE

E N

AN

OT

EC

HN

OL

J P

HY

S A

AP

PL

PH

YS

A

SU

PE

RC

ON

D S

CI

TE

CH

NO

L

AC

T C

RY

ST

B

PR

AM

AN

A

AN

N P

HY

S

PH

YS

SO

LID

STA

TE

J S

TAT

ME

CH

PH

YS

TO

DA

Y

EU

R P

HY

S J

B

RE

P P

RO

G P

HY

S

PH

YS

RE

P

RE

V M

OD

PH

YS

Usa

ge d

iffu

sion

Jour

nal u

sage

inte

nsit

y

usage diffusion

journal usage intensity


Pearson correlation

Usage ratio

Usage diffusion

Article usage

intensity

Journal usage

intensityP 5-year IF h-index

Cited Half-Life

Usage ratio

0.33 0.90 -0.22 -0.09 0.86 0.50 0.02

Usage diffusion

0.33 0.27 0.45 0.72 0.14 0.78 -0.24

Article usage intensity

0.90 0.27 -0.36 -0.16 0.88 0.46 0.09

Journal usage intensity

-0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10

P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17

5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21

h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22

Cited Half-Life

0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22


Pearson correlation

Usage ratio

Usage diffusion

Article usage

intensity

Journal usage


Cited Half-Life

Usage ratio

0.33 0.90 -0.22 -0.09 0.86 0.50 0.02

Usage diffusion

0.33 0.27 0.45 0.72 0.14 0.78 -0.24

Article usage intensity 0.90 0.27 -0.36 -0.16 0.88 0.46 0.09


-0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10

P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17

5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21

h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22

Cited Half-Life

0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22


Pearson correlation

Usage ratio

Usage diffusion

Article usage

intensity

Journal usage


Cited Half-Life

Usage ratio 0.33 0.90 -0.22 -0.09 0.86 0.50 0.02

Usage diffusion 0.33 0.27 0.45 0.72 0.14 0.78 -0.24


0.90 0.27 -0.36 -0.16 0.88 0.46 0.09

Journal usage intensity -0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10

P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17

5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21

h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22

Cited Half-Life

0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22


Pearson correlation

Usage ratio

Usage diffusion

Article usage

intensity

Journal usage


Cited Half-Life

Usage ratio

0.33 0.90 -0.22 -0.09 0.86 0.50 0.02

Usage diffusion

0.33 0.27 0.45 0.72 0.14 0.78 -0.24


0.90 0.27 -0.36 -0.16 0.88 0.46 0.09


-0.22 0.45 -0.36 0.77 -0.29 0.40 -0.10

P -0.09 0.72 -0.16 0.77 -0.16 0.69 -0.17

5-year IF 0.86 0.14 0.88 -0.29 -0.16 0.39 0.21

h-index 0.50 0.78 0.46 0.40 0.69 0.39 -0.22

Cited Half-Life

0.02 -0.24 0.09 -0.10 -0.17 0.21 -0.22

Journal Evaluation: Tags

88.4% of all bookmarks contained tags 8,208 unique tags assigned 38,241 times

1

10

100

1000

10000

1 10 100 1000 10000

freq

uenc

y

tags

tagcloud: all tags assigned at least 50 times


2,535

2,139

1,453

1,227

657

711

546

637

564

494

0 2000 4000 6000 8000 10000

PHYS REV E

APPL PHYS LET

PHYS REV A

J APPL PHYS

J PHYS CONDENS MATTER

NEW J PHYS

REV MOD PHYS

EPL

NANOTECHNOL

J PHYS A

tag frequency

2004

2005

2006

2007

2008number of unique tags

2004

2005

2006

2007

2008

Number of tags per publication year for all bookmarked articles


tags assigned to journal articles depict the users’ perspective on content

analysis over time can reveal shifts in thematic focus areas

tags assigned to articles published in J Phys Condens Matter in 2004

tags assigned to articles published in J Phys Condens Matter in 2008

Conclusions

data from social bookmarking services can be used in journal evaluation

number of bookmarks can be used alternatively to measure worldwide usage of journals

tags reveal the users’ perspective on journal content

poor quality of bookmarking data makes retrieval and analysis complex

social bookmarking in STM is still in its infancy and services are competing for a critical mass

Literature

BERNIUS, S., HANAUSKE, M. AND DUGALL, B. (2009). Von traditioneller wissenschaftlicher Kommunikation zu "Science 2.0". ABI-Technik 29, 214-226.

HAMMOND, T., HANNAY, T., LUND, B. AND SCOTT, J. (2005). Social Bookmarking Tools (I). D-Lib Magazine 11.

HAUSTEIN, S. (2010). Multidimensional Journal Evaluation. Proceedings of STI 2010, Leiden.

JUCHEM, K., SCHLOEGL, C. AND STOCK, W.G. (2006). Dimensionen der Zeitschriftenszientometrie am Beispiel von "Buch und Bibliothek". Information Wissenschaft & Praxis 57, 31-37.

PROJECT COUNTER (2008). The COUNTER Code of Practice. Journals and Databases Release 3, in: http://www.projectcounter.org/r3/Release3D9.pdf.

REHER, S. AND HAUSTEIN, S. (to be published). Social Bookmarking in STM: Putting Services to the Acid Test. ONLINE.

ROUSSEAU, R. (2002). Journal Evaluation: Technical and Practical Issues. Library Trends 50, 418-439.

SHNEIDERMAN, B. (2008). Computer Science - Science 2.0. Science 319, 1349-1350.

Thank you for your kind attention!

STM social bookmarking (by S. Haustein)

Health & Medicine

Transcript of STM social bookmarking (by S. Haustein)