Computer Networks 51 (2007) 4574–4585

www.elsevier.com/locate/comnet

Supporting collaborative hierarchical classification:Bookmarks as an example

Dominik Benz a,*, Karen H.L. Tso b, Lars Schmidt-Thieme b

a Computer-based New Media Group (CGNM), Department of Computer Science, University of Freiburg,

Georges-Kohler-Allee 51, 79110 Freiburg, Germanyb Information Systems and Machine Learning Lab (ISMLL), University of Hildesheim, Samelsonplatz 1, 31141 Hildesheim, Germany

Available online 27 June 2007

Abstract

Bookmarks (or favorites, hotlists) are popular strategies to relocate interesting websites on the WWW by creating apersonalized URL repository. Most current browsers offer a facility to locally store and manage bookmarks in a hierarchyof folders; though, with growing size, users reportedly have trouble to create and maintain a stable organization structure.This paper presents a novel collaborative approach to ease bookmark management, especially the ‘‘classification’’ of newbookmarks into a folder. We propose a methodology to realize the collaborative classification idea of considering howsimilar users have classified a bookmark. A combination of nearest-neighbor-classifiers is used to derive a recommendationfrom similar users on where to store a new bookmark. A prototype system called CariBo has been implemented as a pluginfor the central bookmark server software SiteBar. All findings have been evaluated on a reasonably large scale, real userdataset with promising results, and possible implications for shared and social bookmarking systems are discussed.� 2007 Elsevier B.V. All rights reserved.

Keywords: WWW; Social bookmarking; Bookmark classification; Collaborative filtering; Recommender systems

1. Introduction

The continuing, explosive growth of the WWWstrengthens its role as a prevalent source of informa-tion for scientific research as well as everyday workand leisure. Studies on Web usage like [6] reportedthat revisits are a major part (58%) of website visits.Bookmarks (or favorites, hotlists) are widely used

1389-1286/$ - see front matter � 2007 Elsevier B.V. All rights reserved

doi:10.1016/j.comnet.2007.06.014

* Corresponding author.E-mail addresses: dbenz@informatik.uni-freiburg.de (D.

Benz), tso@ismll.uni-hildesheim.de (K.H.L. Tso), schmidt-thie-me@ismll.uni-hildesheim.de (L. Schmidt-Thieme).

strategies to relocate sites of interest that allow theuser to create a personalized URL repository, whichfacilitates easy and fast access to relevant informa-tion [1]. Traditionally, these repositories are storedon the client-side and can be organized in a hierar-chical folder structure via the browser interface.Recently, server-side mechanisms like the so-called‘‘Social Bookmarking’’ have gained popularity [8].The latter is often characterized by a non-hierarchi-cal classification paradigm known as Collaborative

Tagging. For the hierarchical case, however, diffi-culties remain in organizing and maintaining thehierarchical structure with growing size of therepository. An example is the classification of new

.

D. Benz et al. / Computer Networks 51 (2007) 4574–4585 4575

bookmarks, i.e., finding or creating an appropriatefolder to store them.

This paper presents a novel approach to auto-mate the bookmark classification process, aimingat recommending appropriate folders to a user whenfiling new bookmarks. There are two basic strategiesto solve the problem of how to generate such recom-mendations: The first one, commonly referred to asinformation filtering or content-based filtering [14],draws inferences from the user’s past behavior. Inthis context, ‘‘behavior’’ means which bookmarksthe user stored in which folders.

The second strategy, usually referred to as collab-

orative filtering [14], takes the behavior of othersinto account, especially of those who displayed sim-ilar interests in the past. In other words, the basicidea is to find similar users who have already classi-fied a bookmark, and then to derive recommenda-tions on where the target user could store thisbookmark.

The central contribution of this paper is to present acollaborative classification algorithm for bookmarks.The novelty hereby consists of recommending struc-tural information from similar users. This has scarcelybeen researched in the context of bookmark classifica-tion, where content-based approaches prevail. A pro-totype of a collaborative bookmark classificationsystem, CariBo, has been built, and experimentationresults with this prototype and real user data confirmthat the presented approach can outperform con-tent-based approaches.

This paper is structured as follows: Section 2details on general aspects of bookmarking andproblems that exist hereby. Section 3 gives an over-view of existing work. Our collaborative approach ispresented in detail in Section 4, while Section 5 pre-sents a content-based approach. Implementationdetails can be found in Section 6. The evaluationprocedure and results are given in Section 7, andin Section 8 we conclude.

2. Bookmarks in general

Studies on Web and bookmark usage as well asthe current rapid growth of social bookmarkingtools like del.icio.us1 (see Section 3) suggest thatbookmarks are a popular method among users tofacilitate the access to WWW information. Abrams[1], as an example, cited a survey conducted in 1996

1 http://www.del.icio.us.

with 6619 Web users, where 80% of the subjectsreported bookmarks as a strategy for locating infor-mation. Ninety-two percent of them had a book-mark archive, 37% had more than 50 bookmarks.Three years after its foundation in September2003, del.icio.us claimed to serve 1 million users.2

The studies of Abrams and others [6] confirmedthat users tended to have problems with bookmarkmanagement, especially when the size of the collec-tion increased. Kanawati and Malek categorized theproblems into three classes [10]:

• Resource discovery

The problem of ‘‘finding good bookmarks’’ thatmatch the user’s information needs. This is nota purely bookmark-specific problem, but corre-sponds heavily to the problem of ‘‘finding inter-esting websites’’. This is addressed by a largeresearch community in the area of recommendersystems [14].

• Recall

The problem of locating an appropriate book-mark at a given time.

• Maintenance

The problem of keeping the set of bookmarks up-to-date and well-organized; difficulties herebyarise from discovering broken links, modifyingthe organization scheme due to changes in per-sonal interest, and creating and maintaining thetaxonomy implied by the bookmark folder hier-archy. The classification of new bookmarks alsobelongs to this category.

Abrams et al. pointed out that the crucial trade-off in bookmarking is between organization costsand future benefit: ‘‘Users must weigh the cost oforganizing bookmarks against the expected gains’’

[1]. Roughly half of their subjects turned out to be‘‘sporadic filers’’, i.e., users who occasionally sche-dule reorganization sessions when their bookmarkrepository became too complex. This task was gen-erally reported to be time-consuming and tedious.Among others, their implications for the design ofa (possibly shared) bookmark management systemincluded means to ‘‘provide users with an immediate

filing mechanism when creating a bookmark’’. Weargue that using a collaborative classification algo-rithm for this purpose is a sensible choice.

2 http://blog.del.icio.us/blog/2006/09/million.html.

4576 D. Benz et al. / Computer Networks 51 (2007) 4574–4585

3. Related work

As bookmarking is one of the most commonlyused features on the Web, there is a vast numberof programs and tools with the purpose to alleviatedifferent aspects of bookmark management. A largenumber of these tools can be assigned to the category‘‘centrally store and browse’’, whereby the core ben-efit is to make bookmarks available when the usermoves to another physical machine. This concept isextended in some cases by making bookmarks share-able with other users. Already mentioned above,del.icio.us3 is a popular online service which transfersthe usual client-side bookmarking mechanism onto acentral server to enable roaming and bookmarksharing. This has become known as social bookmark-

ing. Instead of a hierarchical classification scheme,each user can tag his bookmarks with a set of arbi-trary keywords, facilitating a ‘‘by-keyword’’-accessto own or other users’ bookmarks. Spurl4 is anexample of a social bookmark service that retainsthe hierarchical folder structure known from client-side mechanisms. It is important to notice that inboth cases, no personalized recommendation takesplace on how a particular user could classify a newbookmark. The individual repositories are simplymade ‘‘browsable’’.

An example for a more personalized solution tothe resource discovery problem is GroupMark [13],a WWW recommender system. It takes the users’bookmarks as the primary source of informationto assign them to peer recommender groups. Fromthose, they will receive suggestions for potentiallyinteresting websites.

In addition to website recommendation, InLinx byBighini et al. [3] facilitates the automatic classificationof bookmarked websites into globally predefined cat-egories. The basis for the classification is the user’sprofile and the content of the Web page. Two furtherapproaches that use this basis are [12] (employing asemi-automatic clustering algorithm for reorganiza-tion of the bookmark hierarchy) and [11] (comparingdifferent document classification methods).

All of the described approaches address differentaspects, but leave out an important source of infor-mation, namely to consider the bookmark organiza-tion habits and strategies of similar users. The focusof this paper is to propose an algorithm to automat-

3 http://www.del.icio.us.4 http://www.spurl.net.

ically classify bookmarks based on the classificationsof similar users. This collaborative methodology ofrecommending structural information has scarcelybeen researched in the context of hierarchical classi-fication schemes. Haase et al. [7] presented a moregeneral approach of how the evolution and manage-ment of personal ontologies can be supported by acollaborative recommendation algorithm.

4. Collaborative approach

Pemberton et al. pointed out that the basic idea ofcollaborative filtering is ‘‘to recruit others to act as

our filtering agents on the assumption that they are

our peers, i.e., like us in tastes and judgements of qual-

ity’’ [13]. For the case of bookmark management,one could replace ‘‘filtering agents’’ with ‘‘classifica-tion agents’’ or ‘‘annotation agents’’. Differentgroups of people obviously have different needsand strategies to organize and annotate bookmarksbelonging to a certain category. A computer scientistfor example might store a bookmark about Webdevelopment with PHP in a relatively sophisticatedhierarchy like development > Web development >languages > PHP. A sales consultant, however,would probably file the same bookmark in a less dif-ferentiated organization scheme, possibly somethinglike marketing > websites. Analogously, the annota-tions that both these persons would use for this web-site will in all probability differ. The computerscientist might annotate the PHP page with some-thing like ‘‘dynamic, script language, LAMPP’’,whereas one could imagine annotations like‘‘advanced webdesign, programming, webserver’’ forthe sales consultant.

Consequently, having a look in our peer group,i.e., people who are interested and engaged in simi-lar topics as we are, is highly likely to give us valu-able information how to classify and annotate ourown bookmarks. The system described in this paperaims at generating two substantially separate rec-ommendations: keyword recommendations on theone hand, i.e., which keywords to use for annotatinga new bookmark, and a recommendation of a clas-sification on the other hand. We will focus on thelatter aspect in the following discussion; refer to[2] for details on the keyword recommendation.

4.1. Data model

In order to facilitate the measurement of differenttypes of similarity, we have chosen a uniform repre-

5 http://www.lextek.com/manuals/onix/stopwords1.html.

D. Benz et al. / Computer Networks 51 (2007) 4574–4585 4577

sentation of the following three basic entities in thesystem:

• Links, i.e., the actual ‘‘bookmarks’’, consisting ofa URL, and optionally a title and a description.

• Folders that contain the bookmarks, labelledwith a folder title and optionally annotated witha folder description.

• Users that own a hierarchy of folders, optionallyannotated with a user description.

WWW recommender systems like [3] often exam-ine the complete content of websites as data founda-tion and analyze it with information retrievaltechniques. Instead, the presented approach relieson information extracted from the bookmarkedURL itself and manually assigned annotations(title, description). One of the main reasons for thisdecision was that analyzing possibly large HTMLdocuments might slow down the classification pro-cess significantly. This would have detrimentaleffects especially for an everyday task like book-marking. However, if an analysis technique is ableto extract highly descriptive keywords from theactual page content, it can be expected to furtherimprove the recommendation quality of the pre-sented algorithm. For the scope of this work, thishas to be left for future work.

4.1.1. Term vector space

For data representation, the vector space model,a popular information filtering model for textualmaterial, is used [15]. It has widely been tested andis expressive enough to describe the informationcontent available. Furthermore, it allows, in combi-nation with an appropriate database design, for afast computation of recommendations or profileupdates, which is crucial to an everyday task likebookmarking.

In the vector space model, links, folders and usersare described by a profile vector. Each term thatoccurs in any title or description in the system addsone dimension to the vector space. In addition, eachhostname occurring in any URL adds one moredimension. The normalized term frequency wasused as weight for each term. The dimensionalityof the vector space was reduced by stemming, a pro-cedure that tries to reduce the keywords to theirword stems. We used Porter Stemming, a popularstemming algorithm for the English language [3],which removes suffixes based on a set of condi-tion/action rules that specify, for example, how to

remove the plural-‘‘s’’ from plural terms. Addition-ally, very common words or terms with little infor-mation content (‘‘and, or’’, etc) were removed byusing a stopword list5 containing 429 common Eng-lish words. We modified this list by adding stop-words belonging to the area of the WWW, e.g.,index, home, homepage, website. After this, the listcontained 460 entries.

4.1.2. Taxonomy representation

To represent the hierarchical organization of abookmark collection, the terms were aggregated ina bottom-up manner through the taxonomy tree.Starting from the links as ‘‘leaves’’, all foldersinherit all terms and the corresponding frequenciesof their contained links. Then, all parent foldersinherit all terms and frequencies describing theirdescendant folders, up to the user’s root folder.The user profile itself inherits all terms and frequen-cies of the user’s root folder. Hence, the profile of afolder becomes more general the closer it is to thehierarchy root. We argue that this simple mecha-nism reflects the intuitive organization principle ofincreasing folder specificity with increasing depthin the hierarchy. This is why we consider this aggre-gation as a sufficient representation of the hierarchi-cal structure. Furthermore, the additional storageand computational consideration of the graph struc-ture itself might lead to a complexity overheadhardly justified in relation to the possible benefits.

4.1.3. Similarity measure

To measure similarity between two profile vec-tors, this approach uses the cosine vector similarity,a common measure in the context of the vectorspace model. It defines the similarity of two profilevectors, profx and profy, as the cosine of the anglebetween them and can be computed as:

simðprofx; prof yÞ ¼profx � prof y

jprofxkprof y j:

Obviously, for the computation of the dot productin the numerator of this fraction, only those entriesthat have a value greater than zero in both vectorsare relevant. In combination with computing andstoring the norm at vector creation time, this allowsfor an efficient computation of this similarity mea-sure, considering only the intersection of the key-word sets of two entities (links, folders or users).

4578 D. Benz et al. / Computer Networks 51 (2007) 4574–4585

The uniform vector representation of links, fold-ers and users in combination with the mentionedsimilarity measure provides us with the ability tomeasure various relations inside our domain. Firstof all, we can measure the similarity between twousers, two folders or two links. But similarities canalso be computed between different entities, e.g.,between a link and a folder.

4.2. Classification process

Given that a similar user has already bookmarkeda certain URL in one folder of his bookmark hierar-chy, the basic problem consists in mapping the loca-tion of this folder to a folder location in the targetuser’s bookmark hierarchy. This can be seen as aproblem of taxonomy mapping. Another aspect thatneeds to be considered is what to do if we do not findsuch a corresponding folder. As there is no approachof collaborative classification found in the researcharea of bookmark management, it is difficult to drawcomparisons or to point out the predominance of theapproach presented here. This is why we have imple-mented a content-based classification algorithm aswell as a random algorithm to compare the results(see Section 5).

Fig. 1 gives an overview of the process of collab-orative classification. The figure is to be read fromleft to right. It depicts the process when a user u

adds a new bookmark l. The first step is to find sim-ilar users in the system that have already book-marked l. Usim,l is the set of those users, sorted in

Fig. 1. Overview of the collaborative classification process. Abbrj 2 {1, . . .,5}); cs ¼ cusþcfs

2.

descending order by user similarity. Two parame-ters control the size of the group: (i) the maximalnumber of similar users to consider; (ii) the similar-ity threshold to which extent a user is considered tobe similar. Table 1 contains the values used for theevaluation.

Fsim,l contains all folders in which the users fromUsim,l have stored the link l. Assuming that there areno URL duplicates for each user, it is obvious thatjUsim,lj = jFsim,lj.

For each of the folders in Fsim,l, we now try to findthe most similar folders of user u himself. This resultsagain in a set of folders Frec,l, containing only foldersowned by user u. Two parameters control the cardi-nality of Frec,l: (i) the number of similar folders to beconsidered for each folder fsim 2 Fsim,l; and (ii) thefolder similarity threshold to which extent a folderis considered to be similar (see Table 1).

For the purpose of finding the best folder recom-mendation among Frec,l, we can consider three vari-ables (as explained above): (i) the similarity of therecommending user (denoted as us in the diagram);(ii) the folder similarity of his folder with our corre-sponding folder (denoted as fs); (iii) the number oftimes a folder has been recommended, i.e., howoften it occurs in Frec,l. The following ideas of howto combine them are intuitive:

• Choose the folder which has been recommended

most often.

This completely neglects user and folder similari-ties, and is hence insufficient. If folder A is recom-

eviations used: us = sim(u,ui), fs ¼ simðfsimi ; frecj Þ (i 2 {1,2,3},

Table 1Parameters controlling the collaborative classification and valuesused for the evaluation

Parameters controlling the size of Usim,l

Number of similar users to consider 3Threshold of user similarity 0.1

Parameters controlling the size of Frec,l

Number of similar folders to consider 3Threshold of folder similarity 0.01

Parameter controlling new folder creation

Threshold when to create a new folder 0.2

D. Benz et al. / Computer Networks 51 (2007) 4574–4585 4579

mended by three marginally similar users andfolder B by two very similar users, folder A wouldbe the choice – which is not the desired behavior.

• Sum up the user and folder similarities for each

folder.

Once again, this would lead to a strong domina-tion of folders that were recommended often,with the same disadvantages just mentioned.

• Average the user and folder similarities for each

folder.

Hereby, the number of times a folder has beenrecommended would lose influence. If folder Ahas been recommended 10 times, but its averagesimilarity values are slightly smaller than the onesof folder B who has been recommended just once,this approach would wrongly choose to recom-mend B. If there is one very similar user who hap-pens to have a very similar folder, this user wouldstrongly dominate the recommendation process.

We argue that a combination of the above ideasis required that strikes the balance between thenumber of times a folder has been recommendedand the average similarity values. In this way, theeffect of dominating users or folders as in the givenexamples would be smoothed. Of course, thisbecomes necessary only when a folder has been rec-ommended by more than one user. In the area ofcollaborative filtering, usually ratings for certainitems are predicted, e.g., by computing a weightedsum of other users’ votes [4]. In our case, we wantto predict classifications instead of ratings; so wehave adapted this technique to compute for eachrecommended folder a combined user similarity ofall users who have recommended it (denoted cusin the diagram) and a combined folder similarity(cfs) of all folders the recommended folder has beenmapped from. Taking the combined user similarityfor the recommended folder frec as an example, thefollowing values are used for its computation:

• the average user similarity of all users who rec-ommended frec (denoted avgU sim;l;frec

below),• the number of users that recommended frec

(denoted jU sim;l;frec j below) and• the total number of recommending users

(denoted jUsim,lj below).

The core idea is to weigh the average by the pro-portion of all recommending users who have recom-mended the particular folder frec. Notated formally,the combined user similarity, cusfrec , the combinedfolder similarity, cfsfrec , and the final combined sim-ilarity value csfrec of a recommended folder, frec, touser u are computed according to:

cusfrec ¼ avgU sim;l;frec1þ ð1� avgU sim;l;frec

Þ jU sim;l;frec jjU sim;lj

� �;

cfsfrec ¼ avgF sim;l;frec1þ ð1� avgF sim;l;frec

Þ jF sim;l;frec jjF sim;lj

� �;

csfrec ¼cusfrec þ cfsfrec

2;

whereas

• Average user and folder similarity:

avgU sim;l;frec¼ 1

jU sim;l;frec jX

usim2U sim;l;frec

simðusim; uÞ;

avgF sim;l;frec¼ 1

jF sim;l;frec jX

fsim2F sim;l;frec

simðfsim; frecÞ:

• Set of all similar users that would recommend toput link l in folder frec:

U sim;l;frec :

• Set of all folders containing l mapped to the rec-ommended folder frec:

F sim;l;frec :

The resulting combined similarities are found inthe rightmost columns of Fig. 1. The final combinedsimilarity of a recommended folder (denoted cs inthe diagram) is computed as the mean of its com-bined user similarity and its combined folder simi-larity. In the example, folder frec2

would berecommended with a final similarity value of 0.82.

Fig. 2 visualizes the weighting effect on the aver-age user similarity (found on the x-axis) of a folder

frec. The higher the proportionU sim;l;frecj j

U sim;lj j of users

who have recommended frec for link l, the higher

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 10

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

average user similarity

com

bine

d us

er s

imila

rity

(cus

)

|Usim,l,frec| / |Usim,l| = 0

|Usim,l,frec| / |Usim,l| = 0.5

|Usim,l,frec| / |Usim,l| = 1

Fig. 2. Weighting effect on the combined user similarity.

4580 D. Benz et al. / Computer Networks 51 (2007) 4574–4585

the combined similarity value. Please note thatU sim;l;frecj j

U sim;lj j ¼ 0 can never be reached and is just

included as a lower boundary.

To establish a connection to standard classifica-tion methods, this approach can be considered asan application of a two-step k-nearest-neighbor-classifier (k-NN). Usually, this algorithm is usedfor document classification. A new document isclassified to the category the majority of its k mostsimilar documents in the training set belong to.The approach described above considers in the firststep a set of k similar users who have bookmarked alink l (forming the set Usim,l). In the second step itconsiders for each of those users a set of k similarfolders of the target user (forming the set Frec,l).So the first step can be regarded as classifying thecurrent user into an interest group, the second stepas classification of the current bookmark accordingto the needs and habits of that group. For the scopeof the current evaluation, a value for k of 3 has beenchosen. More systematic variation of this parameteris subject to further evaluation.

4.2.1. Creating new folders

The methodology described above will performbest if the current user already has an existing folderwith a sufficient total similarity to a new bookmark.If the latter is not the case, a recommendation isdesirable to create a new folder. We are mainly con-cerned with (i) how to label the new folder, (ii)where to place it in the target user’s hierarchy and(iii) to which degree ancestors of the new foldershould also be created.

Generating such a recommendation is a veryinteresting task, but is not the main focus of thispaper. Hence we have excluded this aspect in theevaluation. Nevertheless we have implemented amethodology, that might serve as a basis for futurework and evaluation. We recommend creating anew folder under the following conditions: (i) Inthe case when the recommended folder happens tobe the target user’s root folder, storing a link in thisroot folder would not contribute to increase thelevel of organization. (ii) In the case when the totalsimilarity of the recommended folder falls below acertain threshold, it can be assumed that the contentof this folder is somehow related to the new book-mark, but is probably not a very specific match.Creating a new subfolder inside this folder can beconsidered as an appropriate way to create a morespecific storage location.

As a folder recommendation might stem fromseveral users having folders with different names, adecision is necessary as to which name to recom-mend as a label for the folder to be created. For thisapproach we adapted the most intuitive idea to usethe name the most similar of the recommendingusers has used. The final question is to which extenta hierarchy of folders is to be created. Before recom-mending the creation of a new folder, its ancestorsare checked for similarity with the target folder. Ifa higher value is found, it creating a hierarchy withappropriate depth is recommended.

Besides the proposed collaborative approach, thefollowing Section presents a content-basedapproach for bookmark classification.

5. Content-based approach

As detailed at the beginning of this section,another source of information to reason aboutautomatic bookmark classification is the targetuser’s own classification history. This is a classicalcase of content-based recommendation. We haveimplemented this algorithm for comparison reasons.In a prior study, we also considered a public direc-tory as another source of information, yieldingrather poor recommendation results [2]. For furthercomparison, we also implemented a random classifi-cation algorithm.

5.1. User’s classification history

Having the vector space model described aboveat hand, finding the best existing folder for a new

Fig. 3. Screenshots of the user interface: (a) adding a new bookmark and (b) display a folder profile.

D. Benz et al. / Computer Networks 51 (2007) 4574–4585 4581

bookmark can be done in a straightforward man-ner. First, a profile vector for the new link is gener-ated, based on the URL as well as title anddescription the user has assigned to it (eventuallysupported by meta information found in the pagecontent). The resulting profile vector is comparedwith the profile vectors of all existing folders. Themost similar folder is recommended. This is a typi-cal application of a nearest-neighbor-classifier(NN). A requirement for a reliable recommendationis the existence of a sufficiently similar folder. Theabilities of this method to recommend the creationof new folders are very limited.

5.2. Random algorithm

For comparison reasons, we implemented analgorithm that randomly recommends a folder ofthe target user for a new bookmark.

7

6. Implementation

The prototype implementation is called CariBo

(Collaborative Bookmark Classifier) and is basedon the open source bookmark server SiteBar.6 Site-Bar as a sourceforge-project is an open-source soft-

6 http://www.sitebar.org.

ware written in PHP to centrally store and sharebookmarks on a webserver. All system data isstored in a MySQL database. The implementationwas done using PHP 5.0.4 along with MySQL4.0.21 and was tested on a machine equipped witha 2.8 GHz Intel Xeon Processor, 2 GB RAM andthe SuSE Linux Operating System (version 9.3).Fig. 3 shows the user interface where the outcomeof the collaborative classification is presented tothe user; Fig. 3 depicts the display of a folder profile.Installation instructions and downloads can befound at our group website.7

7. Experimental evaluation

In a prior case study with 15 subjects, our collab-orative approach clearly outperformed the content-based one [2]. Encouraged by these results, we useda larger scale real-user dataset crawled by Herold [9]from the social bookmarking platform spurl.net8 forfurther evaluation. As mentioned in Section 2, Spurlenables each user to mark his bookmarks as ‘‘pri-vate’’ or ‘‘public’’ and to store them in a hierarchicalfolder structure. Herold crawled the public book-marks of all users visible in the ‘‘discover users’’ –

http://www.informatik.uni-freiburg.de/cgnm/software/car-ibo/index_en.html.

8 http://www.spurl.net.

Table 2Experimentation data statistics

General

Number of test users 619Total number of bookmarks 206,365Total number of distinct URLs

among bookmarks155,736

Min/Max/average number of bookmarksper user

5/7364/332.18

Min/Max/average number of foldersper user

1/1024/41.6

Average number of bookmarks per folder 7.99

Extracted terms and profile vectors

Total number of terms in databaseafter initializing all profiles

135,208

Number of English termsa 15,709 (11.6%)Number of numeric terms 3020 (2.2%)Number of domain names among terms 67040 (49.6%)Number of other terms 49,439 (36.6%)Min/Max/average number of terms

assigned to a bookmark1/34/4.15

Min/Max/average number of termsassigned to a folder (after bottom-upkeyword inheritance, see Section 4.1.2)

1/10,735/60.82

a Counted by looking up the terms in a MySQL version of theWordNet 2.0 database (http://www.androidtech.com/html/wordnet-mysql-20.php).

9 To enhance clarity, the values for Fig. 4b and c weresmoothed with a Savitzky–Golay filter using polynomial order 1and frame size 21 and 361, respectively.

4582 D. Benz et al. / Computer Networks 51 (2007) 4574–4585

section along with the categories they were orga-nized in. From this initial dataset, we included eachuser who had bookmarked at least one of the 200most bookmarked URLs, leading to a set of 619 testusers. As Spurl allows several root folders, we addedan artificial single root folder for each user andappended his original root folders as subfolders.After the users along with their complete bookmarkcollections were imported into our system, the pro-file for each test user was initialized. Refer to Table2 for further statistics. The parameters controllingthe collaborative classification were set accordingto Table 1. The given values were taken over fromthe prior case study. It is subject to further investi-gation how to optimize the classification perfor-mance by modifying those parameters.

For each test user and each of his test links (i.e.,the intersection of his URL collection and the 200most bookmarked URLs), a ‘‘leave-one-out’’-test-ing was applied: the current test link was removedfrom his collection, and given to the three classifica-tion algorithms (collaborative, content-based, ran-dom). This led to a total of 5015 classificationdecisions for each algorithm. The outcome of eachalgorithm was a list with the top 5 recommendationsof folders where the user could classify the book-mark. A classification was judged as a ‘‘hit’’ when

the algorithm recommended the folder from whichthe bookmark was taken from. For each rankingposition, the success rate was measured as the pro-portion of hits among all classifications. Fig. 4 dis-plays the results.9

7.1. Discussion

With a success rate of 70.73% among the top fiverecommended folders (denoted as pos 6 5 in Fig. 4),the collaborative algorithm clearly outperforms thecontent-based approach (44.73%). This predomi-nance is retained on every ranking position (denotedas pos = 1, pos = 2, . . .). The random algorithmshows a nearly constant hit rate of around 4% acrossall positions. The average time needed to generate acollaborative recommendation (1.82 s) is signifi-cantly higher than for the content-based algorithm(0.38 s), but can be seen as justified regarding theobserved gain of recommendation quality.

For the collaborative algorithm, an interestingquestion is how many users need to be present inthe system that have already bookmarked a certainURL in order to generate a recommendation. Fig. 4displays the success rates depending on the book-marking frequency, i.e., the number of times a cer-tain URL has been bookmarked. Althoughslightly decreasing for the collaborative algorithm,the performance is rather stable throughout all fre-quencies. This implies that even a small number ofother people who have bookmarked a certainURL can serve as a sufficient basis for a successfulrecommendation. Fig. 4 compares the performancedepending on the different bookmark repositorysizes, i.e., the number of bookmarks inside a partic-ular collection. Hereby the content-based algorithmseems to perform equally throughout all bookmarkrepository sizes. The random algorithm naturallyperforms better on smaller repositories, as the prob-ability of a hit increases with decreasing repositorysize. Interestingly, the success rate of the collabora-tive algorithm increases in a similar fashion, whichmeans that it is able to produce sensible recommen-dations even for users with a relatively small set ofbookmarks. Of course small bookmark repositoriesare easier to manage manually than larger ones. Butnevertheless these results corroborate the generalidea of collaborative classification.

pos<=5 pos=1 pos=2 pos=3 pos=4 pos=50

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

ranking position

succ

ess

rate

random

contentbased

collaborative

0 200 400 600 800 1000 1200 1400 1600 1800 20000

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

number of bookmarks

succ

ess

rate

collaborative

contentbased

random

10 20 30 40 50 60 700.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

bookmarking frequency

succ

ess

rate

collaborative

contentbased

random

Fig. 4. Evaluation results: (a) success rate on ranking position pos, (b) success rate on top five ranking positions (pos 6 5) depending onthe number of users that have bookmarked a certain URL (bookmarking frequency) and (c) success rate on top five ranking positions(pos 6 5) depending on the size of bookmark repositories (number of bookmarks).

D. Benz et al. / Computer Networks 51 (2007) 4574–4585 4583

Kim [11] reported success rates between 53% and59% for his purely content-based k-nearest-neigh-bor-classifier approach, based on 400–800 trainingdocuments. In this method, the text content of thewebsites was also analyzed. Comparing this to therelatively sparse information available for eachbookmark in our case (roughly four terms; see Table2), the advantages of the collaborative classificationidea become clear: the sparsity of the informationinside a single personal bookmark repository canbe counterbalanced when it is shared with a largercommunity and information of similar users is takeninto account. Interestingly, one can assume that thecontributors have structured their repositories onlyfor personal use, e.g., not adhering to a standardvocabulary to maximize the benefit for the commu-nity. Nevertheless, the aggregation of genuinelyself-centered structuring activity can lead to a benefit

for the community, as the results of our evaluationsuggest. Dynamics like these have also beenobserved in the context of collaborative tagging [5].It is notable at this point that the Spurl test datasetwas naturally limited to bookmark folders whichwere not marked as private by their owners. Fromour results, we speculate that the inclusion of thesefolders would not make a big difference; but, dueto the inherent privacy restrictions, this has toremain a hypothesis.

8. Conclusions and outlook

This paper presented a novel approach to auto-matic bookmark classification, based on the classifi-cations of similar users. The main method presentedwas an application of a k-NN-classifier to generatecollaborative recommendations for classifying new

4584 D. Benz et al. / Computer Networks 51 (2007) 4574–4585

bookmarks. The latter uses a weighted average tech-nique to regulate the influence of several users.

The central contribution of this paper is to dem-onstrate that the classifications of other users, espe-cially similar ones, are a valuable source ofinformation for an automatic bookmark classifica-tion process that should not be neglected whendesigning shared bookmark systems. In the evalua-tion presented, the collaborative classificationclearly outperformed the content-based approachand compared well to other studies. Especially associal bookmarking systems like del.icio.us gain pop-ularity, the results of this study open a new perspec-tive for extending the functionality of such sharedbookmark repositories.

Nevertheless, another result is that collaborativeclassification alone cannot be seen as the goldenmechanism that relieves a user from all tasks of book-mark organization. The user cold start probleminherent to all recommender systems is alleviatedwhen users submit their bookmarks when joiningthe system. But the system cold start problem is morecritical, as recommendations from other usersrequire other users to be present in the system. Forfurther research it could be promising to examinehow synergies can arise from combining the resultsof invoking additional sources for automatic classifi-cation, i.e., the user’s classification history and publicdirectories – the first used only for comparison here.

Another promising direction for further improve-ment of the presented approach is to provide userswith mechanisms to control the taxonomy structure.If a user defines, e.g., a maximum depth level of thetaxonomy, a maximum number of folders or a max-imum number of bookmarks per folder, clusteringtechniques might help to support the process ofsplitting or merging folders.

Long-term experimentation with direct user feed-back on the classifications can be expected to furtherprove the utility of collaborative classification.Another critical aspect of the real-world applicationis how to ensure privacy, e.g., by a mechanism con-trolling which folder or link information should beavailable for others. SiteBar, the base system of thepresented implementation, already offers the basicaccess control features; these could be extended.

References

[1] D. Abrams, R. Baecker, M.H. Chignell, Information archiv-ing with bookmarks: personal web space construction andorganization, in: Proceedings of the SIGCHI Conference on

Human Factors in Computing Systems, ACM Press/Addi-son-Wesley Publishing Co., New York, NY, USA, 1998, pp.41–48.

[2] D. Benz, K.H.L. Tso, L. Schmidt-Thieme, Automatic book-mark classification – a collaborative approach, in: Proceedingsof the 2nd Workshop in Innovations in Web Infrastructure(IWI2) at WWW2006, Edinburgh, Scotland, May 2006.

[3] C. Bighini, A. Carbonaro, G. Casadei, Inlinx for documentclassification, sharing and recommendation, in: V. Devedzic,J.M. Spector, D.G. Sampson, M. Kinshuk (Eds.), Proceed-ings of the 3rd International Conference on AdvancedLearning Technologies, IEEE CS, Los Alamitos, CA, USA,2003, pp. 91–95.

[4] J.S. Breese, D. Heckerman, C.M. Kadie. Empirical analysisof predictive algorithms for collaborative filtering, in: G.F.Cooper, S. Moral (Eds.), Proceedings of the 14th Conferenceon Uncertainty in Artificial Intelligence, 1998, pp. 43–52.

[5] C. Cattuto, V. Loreto, L. Pietronero, Semiotic dynamics andcollaborative tagging, Proceedings of the National Academyof Sciences United States of America 104 (2007) 1461.

[6] A. Cockburn, B. McKenzie, What do web users do? Anempirical analysis of web use, International Journal ofHuman–Computer Studies 54 (2002) 903–922.

[7] P. Haase, A. Hotho, L. Schmidt-Thieme, Y. Sure, Usage-driven evolution of personal ontologies, in: Proceedings ofthe 3rd International Conference on Universal Access inHuman–Computer Interaction (UAHCI), Las Vegas,Nevada USA, July 2005, pp. 22–27.

[8] T. Hammond, T. Hannay, B. Lund, J. Scott, Socialbookmarking tools (i) – a general review, D-Lib Magazine11 (4) (2005).

[9] M. Herold, Collaborative personal ontology evolution,Diploma thesis, Albert-Ludwigs-University, Freiburg, Ger-many, December 2005.

[10] R. Kanawati, M. Malek, informing the design of sharedbookmark systems, in: Proceedings of the InternationalConference on Computer Assisted Information Retrieval(RIAO), Paris, France, 2000.

[11] I.-C. Kim, A personal agent for bookmark classification, in:S.-T. Yuan, M. Yokko (Eds.), Intelligent Agents: Specifica-tion, Modeling, and Applications. 4th Pacific Rim Interna-tional Workshop on Multi-Agents, PRIMA 2001.Proceedings, Lecture Notes in Artificial Intelligence, vol.2132, Springer-Verlag, 2001, pp. 210–221, Dept. of Comput.Sci., Kyonggi Univ., Suwon, South Korea.

[12] Y.S. Mareek, I.Z.B. Shaul, Automatically organizing book-marks per contents, in: Proceedings of the Fifth Interna-tional World Wide Web Conference on Computer Networksand ISDN Systems, 6–10 May, 1996, Elsevier SciencePublishers B.V., Amsterdam, The Netherlands, 1996, pp.1321–1333.

[13] D. Pemberton, T. Rodden, R. Procter, Groupmark: a WWWrecommender system combining collaborative and informa-tion filtering, in: Proceedings of the 6th ERCIM Workshopon ‘User Interfaces for All’, number 12 in Long Papers,ERCIM, 2000, p. 13.

[14] P. Resnick, H.R. Varian, Recommender systems, Commu-nications of the ACM 40 (3) (1997) 56–58.

[15] G. Salton, Automatic Text Processing: The TransformationAnalysis and Retrieval of Information by Computer, Addi-son-Wesley Longman Publishing Co., Inc, Boston, MA,USA, 1989.

Networks 51 (2007) 4574–4585 4585

Dominik Benz obtained his Diploma inComputer Science with minor Psychol-ogy at the University of Freiburg, Ger-many in 2007. He is employed currentlyas a Ph.D. candidate in Computer Sci-ence at the University of Kassel, Ger-many, in the research unit Knowledgeand Data Engineering. His mainresearch interests lie in the area of col-laborative knowledge management,ontology learning and collaborative

tagging systems.

D. Benz et al. / Computer

Karen H.L. Tso received her Bachelor inElectrical Engineering at the Universityof British Columbia, Canada, in 2002, aMaster in Communication and MediaEngineering at University of AppliedScience Offenburg, Germany, in 2004.She is currently a Ph.D. candidate inComputer Science at University of Hil-desheim. Her main research interests arerecommender systems that incorporatecontent information such as attributes

and tags.

Lars Schmidt-Thieme obtained hisDiploma in Mathematics at the Univer-sity of Heidelberg in 1999 and his Ph.D.in the Department of Economics andBusiness Engineering of University ofKarlsruhe, Germany, in 2003. Currently,he is a full professor at institute forEconomics and Information Systems aswell as at Institute of Computer Scienceat University of Hildesheim, Germany.His main research interests are Data

Mining and Machine Learning, especially in connection withInternet technologies and e-commerce applications.