User Interfaces for Semantic Content Authoring:A Systematic Literature Review

Ali Khalili ∗, Soren Auer,

Universitat Leipzig, Institut fur Informatik, AKSW,Johannisgasse 26, 04103 Leipzig, Germany

Abstract

Practical approaches for managing and supporting the life-cycle of semantic content on the Web of Data haverecently made quite some progress. In particular in the area of the user-friendly manual and semi-automatic creationof rich semantic content we have observed recently a large number of approaches and systems being described inthe literature. With this survey we aim to provide an overview on the rapidly emerging field of Semantic ContentAuthoring (SCA). We conducted a systematic literature review comprising a thorough analysis of 31 primary studiesout of 175 initially retrieved papers. We obtained a comprehensive set of quality attributes for SCA systems togetherwith corresponding user interface features suggested for their realization. The quality attributes include aspects suchas usability, automation, generalizability, collaboration, customizability and evolvability. The primary studies weresurveyed in the light of these quality attributes and we performed a thorough analysis of four SCA systems. Theproposed quality attributes and UI features facilitate the evaluation of existing approaches and the development ofnovel more effective and intuitive semantic authoring interfaces.

Key words: Semantic Web, User Interface, Content Authoring, Applications

1. Introduction

Practical approaches for managing and support-ing the life-cycle of semantic content on the Webof Data have recently made quite some progress.On the backend side, a variety of triple stores weredeveloped and their performance and maturity im-proves steadily. Similarly tools and algorithms forlinking data and schemata are progressing and ap-proaches are deployed for the use on the emergingWeb of Data. The quantity of semantic content be-ing made available on the Data Web is rapidly in-

∗ Corresponding author. Tel: +49 341 97-32320Email addresses: khalili@informatik.uni-leipzig.de

(Ali Khalili), auer@informatik.uni-leipzig.de (Soren

Auer).

creasing, mainly due to the use of automated knowl-edge extraction techniques or due to the seman-tic enrichment and transformation of existing struc-tured data. Despite many interesting showcases (e.g.Sindice, Parallax or PowerAqua), we still lack moreuser friendly and scalable approaches for the ex-ploration, browsing and search of semantic content.However, the currently least developed aspect of thesemantic content life-cycle is from our point of viewthe user-friendly manual and semi-automatic cre-ation of rich semantic content. Results of surveysreported by Heitmann et al. 2009 [26], Paulheim etal. 2010 [49], and Hachey 2011 [22] support this factas well.

We define semantic content authoring as the tool-supported manual composition process aiming atthe creation of documents which are:

Preprint submitted to Elsevier 18 May 2012

– fully semantic in the sense that their original datamodel uses a semantic knowledge representationformalism (such as RDF, RDF-Schema or OWL)or

– based on a non-semantic representation form (e.g.text or hypertext), which is enriched with seman-tic representations during the authoring process.

A semantic authoring UI is a human accessible in-terface with capabilities for writing and modifyingsemantic documents. Semantic documents facilitatea number of important aspects of information man-agement:– For search and retrieval enriching documents

with semantic representations helps to createmore efficient and effective search interfaces, suchas faceted search [61] or question answering [39].

– In information presentation semantically en-riched documents can be used to create moresophisticated ways of flexibly visualizing infor-mation, such as by means of semantic overlays asdescribed in [8].

– For information integration semantically enricheddocuments can be used to provide unified viewson heterogeneous data stored in different applica-tions by creating composite applications such assemantic mashups [2].

– To realize personalization, semantic documentsprovide customized and context-specific informa-tion which better fits user needs and will resultin delivering customized applications such as per-sonalized semantic portals [54].

– For reusability and interoperability enrichingdocuments with semantic representations (e.g.using the SKOS and Dublin Core vocabularies)facilitates exchanging content between disparatesystems and enables building applications suchas executable papers [42].

There are already many approaches and toolsavailable for semantic content authoring which ad-dress different aspects of this task by proposingappropriate user interfaces. Due to the wealth ofdifferent approaches emerging it is crucial to obtainan overview on the advancement in this emergingfield. Furthermore, having a holistic view on ap-proaches and tools provides us with an exhaustiveset of quality attributes, which are important forconceiving guidelines for developing more effectiveand intuitive semantic authoring interfaces.

In this article, we summarize the findings of a sys-tematic literature review on semantic content au-

thoring. We extract different types and propertiesof user interfaces proposed for semantic content au-thoring. The results reveal a set of quality attributeswhich can be used for classification of semantic au-thoring tools. Furthermore, we report on the sug-gested user interface types and features proposed inthe literature to realize these quality attributes.

The rest of the paper is organized as follows. InSection 2 we describe the research method and thereview protocol used for conducting the systematicreview. In Section 3 we first define the terminologyof the paper then we elaborate on the results of thereview by surveying the extracted quality attributes.In Section 4 we discuss four existing semantic au-thoring tools and describe them in the light of thequality attributes. In Section 5 we report on the gapsand open research issues revealed from the results ofour systematic literature review. Finally in Section 6we conclude and present some ideas for future work.

2. Research Method

We followed a formal systematic literature reviewprocess for this study based on the guidelines pro-posed in [14,31]. A systematic literature review is anevidence-based approach to thoroughly search stud-ies relevant to some pre-defined research questionsand critically select, appraise, and synthesize find-ings for answering the research questions at hand.Systematic reviews maximize the chance to retrievecomplete data sets and minimize the chance of bias.As part of the review process, we developed a pro-tocol (described in the sequel) that provides a planfor the review in terms of the method to be followed,including the research questions and the data to beextracted.

2.1. Research Questions

The goal of our survey is analyzing existinguser interfaces for semantic content authoring andthereby providing a set of quality attributes, whichcan serve as guidelines for designing suitable andeffective user interfaces for semantic content au-thoring. To achieve this goal we aim to answer thefollowing general research question:

What are existing approaches for user-friendly se-mantic content authoring?

We can divide this general research question intothe following more concrete sub-questions:

2

– RQ1. How to classify existing approaches for se-mantic content authoring?

– RQ2. What type of user interfaces are used by eachapproach?

– RQ3. What are the features supported by the pro-posed user interfaces?

– RQ4. What type of users are targeted in each ap-proach?

– RQ5. How is the user interface evaluated?

After doing some pilot searches and consultingexperts in the field, we obtained a list of pilot studieswhich served as a basis for the systematic review.

2.2. Search Strategy

To cover all the relevant publications, we used thefollowing electronic libraries:– ACM Digital Library– IEEE Xplore Digital Library– ScienceDirect– SpringerLink– ISI Web of Sciences

Based on the research questions and pilot stud-ies, we found the following basic terms to be mostappropriate for the systematic review:

(i) semantic OR linked data OR web of data ORdata web

(ii) content OR web page OR document(iii) authoring OR annotating OR annotation OR

annotate OR enrich OR editTo construct the search string, all these search

terms were combined using Boolean “AND” as fol-lows:

i AND ii AND iiiThe next decision was to find the suitable field

(i.e. title, abstract and full-text) to apply the searchstring on. In our experience, searching in the ‘ti-tle’ alone does not always provide us with all rele-vant publications. Thus, ‘abstract’ or ‘full-text’ ofpublications should potentially be included. On theother hand, since the search on the full-text of stud-ies results in many irrelevant publications, we choseto apply the search query additionally on the ‘ab-stract’ of the studies. This means a study is selectedas a candidate study if its title or abstract containsthe keywords defined in the search string. In addi-tion, we limited our search to the publications thatare written in English and are published after 2002(when the first ISWC conference was held).

Fig. 1. Steps followed to scope the search results.

2.3. Study Selection

Some of the studies might contain the keywordsused in the search string but might still be irrele-vant for our research questions. Therefore, a studyselection has to be performed to include only stud-ies that contain useful information for answering theresearch question.

Peer-reviewed articles that satisfy all the follow-ing inclusion criteria are selected as primary studies:– I1. A study that focuses on semantic content au-

thoring.– I2. A study that either proposes a user interface

or a set of user interface features for the purposeof semantic content authoring.

Studies that met any of the following criteria wereexcluded from the review:– E1. A study that does not focus on semantic con-

tent authoring but only mentions the term e.g. asan example or use case.

– E2. A study that does not propose any user inter-face or user interface feature for semantic contentauthoring but only a generic, non-user interfacesupported method, approach or algorithm for se-mantic annotation.

– E3. A study that is not about Web-based contentauthoring (e.g. studies about semantic authoringin word processors like LATEX).

– E4. A study that is only about the ontology cre-ation or ontology annotation (e.g. using the nat-ural language).

– E5. A study that does not discuss textual Webcontent authoring but other modalities such asimage, audio or video annotation.

We conducted our review in early July 2011. Asa consequence, our review included studies thatwere published and/or indexed before that date.As shown in Figure 1, we first applied the searchquery on each data source separately. Subsequently,to remove duplicate studies, we merged the resultsobtained from the different data sources. To remove

3

Fig. 2. The screenshot of the coding software showing the generated list of codes from the primary studies.

irrelevant studies, we scanned the articles by titleand thereby reduced the number of studies to 175.Then, we read the abstract of each publication care-fully and further decreased the number of studiesto 70. Finally, we added a list of additional papersrecommended by experts and then scanned the full-text of the publications. We checked the full-text ofstudies to see if they fit with our predefined selec-tion criteria. The result comprised 31 publicationsthat represented our final set of primary studies.

2.4. Data Extraction and Analysis

The bibliographic metadata about each primarystudy were recorded using the bibliography manage-ment platform JabRef 1 . In addition, we extractedthe following information from each paper:– The used approach for semantic content author-

ing.– The type of user interface.– The features supported by the user interface.– The domain and type of user.– The evaluation method used in the paper.

1 http://jabref.sourceforge.net/

To analyze the information appropriately, we re-quired a suitable qualitative data analysis methodapplicable to our dataset. A common method that isused for this purpose is the grounded theory methodbecause the theories (the SCA approaches and UIfeatures) are “grounded” in the data [19].

Constant comparison method, one of thegrounded theory techniques, has been often used inanalyzing data and generating categories of data.Although constant comparison method can be usedon any set of data, it is particularly suitable for thedata that are context sensitive [55] (i.e. data can beinterpreted differently in different contexts). To in-terpret SCA approaches and UI features correctly,one often needs to understand in which contextthe approach and feature is proposed and howit is addressed. For instance, consider one studythat mentions “evolvability” as a feature for UI.Without understanding the context of this feature,we cannot conclude whether this feature is aboutdesigning evolvable UIs or about supporting anno-tation/ontology evolvability in the UI (which is ouraim here).

Miles and Huberman [41] described coding asa procedure for the constant comparison method.Codes are tags or labels for assigning units of mean-ing to the descriptive or inferential information

4

compiled during a study. Codes are efficient data-labeling and data-retrieval devices. One method ofcreating codes which is employed in our review iscreating a provisional “start-list” of codes prior tofieldwork. We created this list from our researchquestions and the pilot studies. To carry out theanalysis systematically, we used the following cod-ing procedures proposed by Lincoln Guba[41]:– Filling-in: we read each study carefully and added

the codes for related fragments and items. Asnew insights or new ways of looking at the dataemerged, we reconstructed our coherent codingschema.

– Extension: if needed, we returned to materialscoded earlier and interrogated them in a new way,with a new theme, construct, or relationship.

– Bridging : if new or previously not understood re-lationships within units of a given category werefound, we recorded that relationship.

– Surfacing : we identified new categories whichcontained the previously created codes.

As shown in Figure 2, we used the Weft QDAsoftware 2 to record the codes. The final list of codesare available online 3 .

2.5. Overview of Included Studies

For quantitative analysis purposes, we performedsome queries on the collected database of primarystudies. The distribution of studies per year asshown in Figure 3 indicates an increasing intensityof research in the area of semantic content author-ing. The remarkable rise after 2008 can be explainedwith the emergence and adoption of weak seman-tic techniques (the so-called ‘lowercase’ SemanticWeb), such as the use of Microformats 4 , RDFa 5

and Microdata 6 . These techniques facilitate se-mantic content authoring by embedding semanticannotations into the HTML Web pages.

The primary studies included 14 conference pa-pers, 11 journal articles, 4 workshop papers, onethesis and one technical report. Among them, thefollowing four studies are survey papers. Uren etal. [62] have reported a comprehensive review ofthe studies and applications for semantic annota-

2 http://www.pressure.to/qda/3 http://rdface.aksw.org/SLR/codes.qdp4 http://microformats.org/5 http://www.w3.org/TR/rdfa-syntax/6 http://www.w3.org/TR/microdata/

Fig. 3. Publications per year.

tions which were published before 2006. In [26], Heit-mann et al. conducted an empirical survey of Seman-tic Web applications and have reviewed the chal-lenges of them. Paulheim and Probst [49] surveyedthe ontology-enhanced user interfaces and have in-troduced a schema for characterizing the require-ments of ontology-enhanced user interfaces. In [22],Hachey and Gasevic provide an overview of the cur-rent progress and gaps in the area of Semantic Webuser interfaces in general.

3. Results

In this section we first define the basic conceptsused in the paper and then elaborate on the resultsof our qualitative data analysis.

3.1. Terminology

The terminology basis of this article is depictedin Figure 4. In the sequel we describe the individualconcepts in more detail:

i. Semantic Gap is a term coined to describe thediscrepancy between low-level technical features ofmultimedia, which can be automatically processedto a great extent, and the high-level, meaning-bearing features a user is typically interested in [58].As discussed in [11], semantic gaps in the process ofconstructing and managing digital content can bedivided into three types namely human-to-machine,machine-to-machine, and machine-to-human. Inthis article we mainly focus on the machine-to-machine semantic gaps which are important whensearching or reusing content by machines. In thiscontext, semantics consists of concepts and theirlogical relationships in an explicit form. When thesemantics is processed by a machine, the lack of acommon vocabulary may lead to alterations in theoriginal semantics thus resulting in semantic gaps.

5

Semantic Computing

Human Computer Interaction

Social Semantic Web

Document

Semantics

Se

ma

nti

c D

oc

um

en

t

Semantic Gap

Authoring

User Interface

Semantic

Conte

nt

Authorin

g

Se

ma

nti

c A

na

lys

is - Processing of natural

language, multimodal

content & speech

- Web, Data & Process

mining

- Semantic link discovery

- Semantic enrichment &

repair

Users

Interactions

- Designing interfaces

- Implementing interfaces

- Evaluating & comparing

interfaces

- Descriptive & predictive

models & theories of

interaction - Personalization & contextual

browsing

- Integrated social networks

- Semantic information

mashups

- Crowdsourcing

- Human-machine synergy

Fig. 4. Semantic content authoring eco-system.

ii. Semantic Computing is a research field thataddresses the extraction and processing of the se-mantics of digital content and naturally expresseduser intentions to help retrieve, manage, manipu-late, or even create the content. Semantic comput-ing aims to bridge the semantic gap by employingappropriate semantic analysis techniques such asnatural language processing, processing of multi-modal content, speech recognition, Web, data andprocess mining, semantic link discovery as well assemantic enrichment and repair. Semantic Webknowledge representation techniques (e.g. OWL,RDF, RDFa, SPARQL, SKOS) help to bridge thesemantic gap through a common ground of sharedvocabularies and ontologies [57,23].

iii. Semantic Document is an intelligent document(with explicit semantic structure) which “knowsabout” its own content so that it can be automati-cally processed in unforeseen ways. These benefits,however, come at the cost of increased authoringeffort [23,62].

iv. Semantic Content Authoring (SCA) is a tool-supported manual composition process aiming atthe creation of semantic documents. With an on-tology and a user interface appropriate for the typeof content, semantic authoring can be easier than

traditional composition of content and the resultingcontent can be of higher quality [23].

v. Semantic Authoring User Interface (SAUI) isa human accessible interface with capabilities formodifying and writing semantic documents.

vi. Human Computer Interaction (HCI) is a re-search field that aims to improve the interactionsbetween users and computers by making computersmore usable and receptive to the user’s needs.

vii. Social Semantic Web is a very general researchfield triggered by the advent of Web 2.0. It aimsat bringing a social novelty, rather than a techni-cal one by providing user-friendly tools to facilitatebroad user participation in the process of creatingsemantic content [58]. The Social Semantic Web vi-sion comprises many of the aforementioned domainsand techniques.

3.2. Semantic Authoring Approaches

There are already different approaches proposedfor semi-structured but non-semantic content au-thoring (e.g. [5]). These approaches aim at immedi-ate user gratification in the form of useful visualiza-tions and interesting data aggregation but do not

6

deliverable

Nature

deliveryDate

leading

partner

work

package

label

seeAlso

Deliverable LO

D2

On

tolo

gy

Article

Person

...

...

...

Do

ma

in

On

tolo

gy

Location

Organization

Person

label

label

label

...

...

...

...

Top-Down

Bottom-Up

OntologiesContent

Fig. 5. Top-Down and Bottom-Up approaches for semantic

content authoring.

focus on using shared vocabularies and formal on-tologies which ultimately facilitate portability andreuse. With regard to explicit semantic content au-thoring recent approaches can be roughly classifiedinto the categories Top-Down and Bottom-Up. Asdemonstrated in Figure 5, the classification is basedon the starting point of the authoring process whichcan be ontologies (with upper level of expressive-ness) or unstructured content (with lower level ofexpressiveness).

3.2.1. Bottom-Up ApproachesThese approaches which are usually called se-

mantic annotation techniques (a.k.a. semanticmarkup [3]) aim to annotate existing documentsusing a set of predefined ontologies. The basic in-gredients of a semantic annotation system are on-tologies, the documents and the annotations thatlink ontologies to documents [62]. Here, we needtwo kinds of ontologies [63]: Annotation ontolo-gies (i.e. metadata schemata) which define whatkind of properties and value types should be usedfor describing a resource. For example, the DublinCore schema uses elements such as dc:title,dc:creator, dc:subject, etc. Domain ontologieswhich are used to define vocabularies providingpossible values for metadata properties. Examplesare eClassOWL 7 defining products and services,MeSH 8 defining medical subjects or the DBpediaknowledge base, which is a cross-domain ontologyextracted from Wikipedia.

7 http://www.heppnetz.de/projects/eclassowl/8 http://www.ncbi.nlm.nih.gov/mesh

The result of the annotation process is a documentthat is marked-up semantically. For that concern,some markup strategies are already proposed:– Microformats 9 is an approach to integrate se-

mantic markup into XHTML and HTML docu-ments. Microformats re-purpose existing markupdefinitions (particularly the HTML class at-tribute) in a non-standard way to convey (meta-)data. This approach is limited to a set of fewpublished Microformat templates and thus noteasily extensible for domain-specific applications.Moreover, it is not possible to validate Microfor-mat annotations since no proper grammar is usedfor their definition.

– eRDF (embedded RDF) is similar to Microfor-mats but annotates HTML using RDF. However,it faces the same criticism as Microformats, sinceit uses the same non-standard compatible anno-tating strategy[3].

– RDFa 10 (Resource Description Framework inattributes) is a W3C Recommendation that addsa set of attribute level extensions to XHTMLfor embedding RDF metadata within web docu-ments. RDFa fulfills the principles of interoper-able metadata such as publisher independence,data reuse, self containment, schema modularityand evolvability to a good extent.

– Microdata 11 is an HTML5 specification used tonest semantics within existing content on Webpages. It is already in use by popular search en-gines for interpreting the information containedin a Web page. Microdata is complementedby schema.org – a repository of annotationschemata.There are normally two types of metadata applied

to a document in the process of semantic annotation:– Content metadata describe specific things the au-

thor of the document wishes to write about (e.g.people, cities, etc.). These content-related meta-data cover a broad domain of information [43].Natural Language Processing (NLP) annotationAPIs (e.g. DBpedia Spotlight 12 ) are one ap-proach to automatically add content metadatainto a document.

– Context metadata refers to the general topic,structure or temporal aspects of a document (e.g.title, theme or creation date of a document).

9 http://microformats.org10http://www.w3.org/TR/rdfa-syntax/11http://www.w3.org/TR/microdata/12http://spotlight.dbpedia.org

7

These context-related metadata cover a very spe-cific domain of information. Semantic Tagging(e.g. Faviki 13 ) and structured templates [51] aretwo approaches to automatically embed context-related metadata in a document.

3.2.2. Top-Down ApproachesThese approaches which are also called Ontology

Population [58] techniques aim to create semanticcontent based on a set of initial ontologies whichare extended during the population process. Whencompared with the bottom-up approaches, these ap-proaches deal with semantic representations fromthe beginning instead of lifting unstructured contentto a semantic level.

These approaches combine ontological rigourwith flexible user interface constructs to create se-mantic content. Semantic templates as discussedin [4,13,59] are one technique to realize this goal.In this approach each class of the ontology has anassociated template. Each instance of a class isrepresented by a page using that template. Dataproperties are displayed as simple text while objectproperties are displayed as links to other pages (rep-resenting other instances of the ontology). Userscan also edit the underlying ontology which willresult in changes of the corresponding templates.

3.3. Quality Attributes

In order to evaluate the strengths and weaknessesof different SCA systems, we assess the systems ac-cording to predefined criteria which we call QualityAttributes in this paper. Quality attributes are non-functional requirements used to evaluate the perfor-mance of a system. They are widely used in architec-ture development and assessment as high level char-acteristics which systems enclose. In the context ofthis paper, quality attributes represent the areas ofconcern regarding the development of an SCA sys-tem from the viewpoint of its consumers.

Based on the qualitative analysis of our primarystudies, we obtained 11 quality attributes. For eachquality attribute we extracted one or more UI fea-ture(s). Features describe a specific type or prop-erty of UI that can be used to realize an intendedquality attribute. The features are directly (e.g.faceted browsing) or indirectly (e.g. UIs for mobile

13http://www.faviki.com

Fig. 6. Relation between usability factors and criteria.

devices) addressing the required UI functionalitiesfor an SCA system. Table 1 surveys the qualityattributes and various UI approaches for their im-plementation. In the sequel we describe each of the11 quality attributes in more detail.

3.3.1. UsabilityUsability is a measure of the quality of a user’s

experience in interacting with a system. In ISO9241 usability is defined as the effectiveness, effi-ciency and satisfaction with which specified usersachieve specified goals in particular environments.Lauesen [37] and Nielsen [45] add more factors suchas learnability and utility to usability definition. Inthe context of this paper we consider the followingfactors for defining the usability:(a) Efficiency. How efficient is the system for the

frequent user to expend appropriate amountsof resources in relation to the effectivenessachieved in a specified context of use?

(b) Effectiveness. How effective is the system toachieve specified tasks with accuracy and com-pleteness?

(c) Satisfaction. How satisfied is the user with thesystem?

(d) Learnability. How easy is the system to learn forvarious groups of users?

(e) Utility (or Usefulness). Assesses whether thesystem enables users to solve real problems in

8

Quality Attribute Realization

UsabilitySingle Point of Entry Interface [29,4,62], Faceted Browsing [4,17], Faceted Viewing [40,24,47],

Inline Editing and View Editing [4,60]

Customizability Living UIs [21], Providing Different Semantic Views [4,13,42,6]

GeneralizabilitySupporting Multiple Ontologies [62,7,24,44,1,53], Supporting Ontology Modification [62,63,13],

Supporting Heterogeneous Document/Content Formats [62,24]

CollaborationAccess Control [62,53,60], Support of Standard Formats [62,63,7,24,40,53,60,33], UIs for Social

Collaboration [4,6]

Portability Cross-browser Compatibility [63,44], UIs for Mobile Devices [15]

Accessibility Accessible UIs [22]

ProactivityResource Suggestion [40,7], Real-time Semantic Tagging [24,47], Concept Reuse [24,47,4,7],

Real-time Validation [63,13]

Automation Automatic Annotation [58,3,33,32,63,7,40]

EvolvabilityResource Consistency [24], Document and Annotation Consistency [62], Versioning and Change

Tracking [4]

Interoperability Support of Standard Formats [62,63,7,24,40,53,60,33], Semantic Syndication [4]

Scalability Support of Caching [4,27], Suitable Storage Strategies [4,62,44,40]

Table 1List of quality attributes together with their corresponding UI features suggested for SCA systems.

an acceptable way.Simplicity is the main prerequisite of usability.

An SCA system should, as a rule, hide technicalconcepts related to markup languages and ontolo-gies from the non-expert end-users [63,60]. It is cru-cial to provide end-users with easy to use interfacesthat simplify the annotation process and place it inthe context of their everyday work. More attentionneeds to be paid to decrease or blur the gap betweenthe normal authoring process and the semantic au-thoring process. SCA systems should focus on usersmain task [24]. Usually, a user wants to perform thetask of writing some text and not to annotate con-tent. Integrating semantic authoring process intothe commonly used packages is one approach to en-courage users to view semantic authoring as part ofthe authoring process not as an afterthought pro-cess [62].

The following features of UIs are proposed for im-proving the usability of SCA systems:– Single Point of Entry Interface.

It means the environment in which users anno-tate documents should be integrated with the onein which they create, read, share and edit them.So, there is no added user effort involved in cre-ating a semantic content versus a conventionalapproach, because the real work is done by thesoftware through capturing semantics that is al-ready being provided by the user [29,4,62]. This

will minimize user actions as well as memory loadthereby increasing the efficiency, user satisfaction,learnability and utility of the system.

– Faceted Browsing.Faceted browsing is a technique for accessinga collection of information represented using afaceted classification, allowing users to exploreby filtering the available information. In the UIwhich implements this technique, all propertyvalues (i.e. facets) of a set of selected instances areanalyzed. If for a certain property the instanceshave only a limited set of values, those values areoffered to further restrict the instance selection.Hence, this way of navigation through data willnever lead to empty results [4,17]. This feature isuseful when searching for available resources orvocabularies.

– Faceted Viewing.Faceted viewing [40,24] also known as augmentedbrowsing [47] is very similar to faceted browsingbut is used to distinguish the semantically anno-tated content from the normal content based onthe different facets selected by user. For example,highlighting the names of members of a specificworking group with a yellow background in thetext. Faceted browsing as well as faceted browsingfeatures increase the efficiency and effectivenessof the system by improving the navigability.

– Inline Editing and View Editing.

9

An SCA system should provide different editingmodes for editing single and batch items. Inlineediting allows editing items by clicking on them.View editing supports the editing of a combi-nation of items in a specific view in one singlestep [4,60]. This feature helps users to edit itemsin a minimum number of steps (minimal action)thereby increasing the efficiency and user satis-faction.Figure 6 shows how the aforementioned UI type

and properties affect our previously defined usabil-ity factors. It is based on the QUIM model definedby Seffah et al. [56]. Quality in Use Integrated Mea-surement (QUIM) model brings together usabilityfactors, criteria, metrics, and data mentioned in var-ious standards or models for software quality anddefines them and their relations with one another ina consistent way.

3.3.2. CustomizabilityCustomizability is the ability of a system to be

configured according to users’ needs and preferences.Instead of being a static form strictly dependent on agiven schema, an SCA system should provide mech-anism to tailor its functionalities based on the userneeds [13]. In [47] the concept of “semantics in theeyes of the end-user” is introduced which means anSCA system should provide different views for dif-ferent personas using the system.

The following features of UIs are proposed for im-proving the customizability of SCA systems:– Living UIs.

A Living UI is a user interface that configures itselfto automatically display the information most rel-evant to the user, dynamically adjusts to changingdata, and still allows single users to customize ac-cording to their preferences [21]. End-user devel-opment techniques like Programming by Example(PbE) allow inferring user intents in real interac-tions and according to that providing customizedoutputs [50].

– Providing Different Semantic Views.Semantic views allow the generation of differentviews on the same metadata schema and aggre-gations of the knowledge base based on the roles,personal preferences, and local policies of the in-tended users [4,13,42]. Such views can be eithergeneric or domain specific. Generic views providevisual representations of instance data accordingto certain property values (e.g. map view or cal-endar view). Domain specific views address the

requirements of a particular domain user (e.g.chemists need specific views for visualizing theatomic structure of chemical compounds).

3.3.3. GeneralizabilityGeneralizability is the ability of a system to adapt

to different situations or use cases. An SCA systemshould support wide a range of metadata schematain a flexible way. In fact, the more flexible and adapt-able a system is, the more valuable it is for differentcontexts and users. A generic SCA system reducesthe costs of supporting new schemata considerably,by following the evolution of existing standards andintegrating heterogeneous resources [13]. Adaptivityis an important capability of a generic system. AnSCA system should be adaptable to different an-notation and authoring uses with different kinds ofcontents to be processed [63,1]. In most of the casesGeneralizability is in opposition to Usability of asystem. For instance, adding more and more syn-tactic possibilities counteracts ease of use for SCAsystems [4].

The following features of UIs are proposed for im-proving the generalizability of SCA systems:– Supporting Multiple Ontologies.

A domain is usually described by several ontolo-gies. For example, in a medical context theremay be one ontology for general metadata abouta patient and other technical ontologies thatdeal with diagnosis and treatment. SCA sys-tems need to be able to support multiple ontolo-gies [62,7,24,44,1,53]. In a generic SCA system,the user interface must be completely decoupledfrom the ontological models. Models should beable to be added at runtime and become immedi-ately accessible to the users [53,7].

– Supporting Ontology Modification.A generic SCA system should provide users withuser-friendly interfaces to modify the structure(classes and properties) of ontologies [62,63,13].In this case, the system also needs to deal withconsistency issues which might arise between on-tologies and annotations with respect to ontologychanges (a.k.a. Ontology Maintenance [62]).

– Supporting Heterogeneous Document and ContentFormats.Supporting heterogeneous document and contentformats is a prerequisite for integrating semanticauthoring and annotation into the existing workpractices [62,24]. A generic SCA system shouldbe able to import documents in different formats

10

such as word processor files, spreadsheets, graph-ics files and complex mixtures of them. It alsoneeds to provide appropriate semantic annota-tions for different content types. For example, dur-ing the content annotation, a data table shouldbe treated differently then raw text, because a ta-ble implicitly expresses relationships between theentries of a row (or column).

3.3.4. CollaborationCollaboration refers to the ability of a system

to support cooperation between different users ofthe system. An SCA system should support collab-orative semantic authoring, where the authoringprocess can be shared among different authors atdifferent locations. This is a key requirement ofknowledge sharing between users from differentfields who are contributing to and reusing intelli-gent documents [62,63,12]. Web 2.0 applicationsand related technologies provide incentives to theirusers for collaboration and lead to rapidly grow-ing amounts of content. Triggered by the successof the Web 2.0 phenomenon the Social SemanticWeb idea has gained momentum yielding tools thatallow collaboration and participation incorporatingsemantics by lay users. As a result, many collabora-tive and community-driven approaches to semanticcontent creation have been proposed. Examplesare Semantic Wikis and Semantic Tagging Systems(e.g. Faviki 14 ) which exploit Web 2.0 principlesand technologies to facilitate broad user participa-tion and collaboration in the process of creatingsemantically enriched or annotated content [58,27].[13] divides semantic wikis into two main categoriesaccording to their connections with the ontologies:wikis for ontologies and ontologies for wikis. Theclassification is very similar to our proposed top-down (cf. Section 3.2.2) and bottom-up approaches(cf. Section 3.2.1).

Access control and supporting standard formatsare two additional independent prerequisites of col-laboration in an SCA system [62,53,60]. The SCAsystem should allow to distinguish between write-able and non-writeable content based on the userspermission level. It also needs to support standardformats which promote the collaboration and makeit possible to share and re-use the generated content.

To realize collaboration, an SCA system shouldprovide appropriate UI elements for meta-level in-

14http://www.faviki.com/

teractions around different types of semanticallycreated content such as rating, tagging and dis-cussing. Supporting social networking features suchas following other authors, subscribing to changesfor watching the evolution of content [6] as well asreusing and repurposing of content are also impor-tant to increase the collaboration in an SCA system.

3.3.5. PortabilityPortability is the ability of a system to run un-

der different environments. The user of an SCAsystem should be able to use the system at any loca-tion without installing any special software [63,44].When focusing on Web-based UIs, compatibilitybetween different existing web browsers and ac-cess technologies becomes an important issue. Asa requirement for UI, cross-browser compatibilityshould ideally be ensured in an SCA system. De-signing suitable UIs for mobile and ubiquitous de-vices is another aspect which needs to be taken intothe account as powerful mobile computing devicesare becoming common among the users [15].

3.3.6. AccessibilityAccessibility describes the degree to which a soft-

ware system is available to as many people as pos-sible. It can be viewed as the ability to access andbenefit from some system. Accessibility is often usedto focus on people with disabilities or special needsand their right of access to system. As mentioned in[22], papers discussing accessibility are clearly lack-ing in the context of Semantic Web UIs.

3.3.7. ProactivityProactivity is the ability of a system to act in

advance of a future situation, rather than just re-acting. It means taking control and making thingshappen rather than just adjusting to a situation orwaiting for something to happen. An SCA systemshould provide users with pre-filled form fields, sug-gestions, default values etc. These facilities simplifythe authoring process, as they reduce the numberof actions users have to perform. Moreover, they re-duce the possibility that users provide incompleteor empty metadata [13].

The following features of UIs are proposed for im-proving the proactivity of SCA systems:– Real-time Semantic Tagging.

Real-time tagging means creating annotationswhile the user is typing [24]. This will signifi-cantly increase the annotation speed [47]. Users

11

are not distracted since they do not have to in-terrupt their current authoring task. This type ofUI needs a client-side component which interactswith the server asynchronously.

– Resource Suggestion.An SCA system should provide users with a setof entity (i.e. URI) suggestions to facilitate theannotation process for non-expert users [40,7].

– Concept Reuse.An SCA system becomes increasingly advanta-geous, if once defined concepts (e.g. classes, prop-erties, or instances) are as much reused and inter-linked as possible [4]. Suggesting already definedconcepts to users (particularly new and inexpe-rienced users) will facilitate their contribution tothe system.

– Real-time Validation.When the annotation is completed by user, theSCA system should apply validation mechanismsto check the correctness of the values. Validatingmetadata while they are being created improvesthe overall quality of the documents and does notrequire further consistency checks, which mightbe difficult or even impossible once the providerof metadata has completed the job [63,13].

3.3.8. AutomationAutomation is the ability of a system to auto-

matically perform its intended tasks thereby reduc-ing the need for human work. In the context of se-mantic authoring it means the provision of facili-ties for automatic mark-up of documents to facili-tate the economical annotation of large documentcollections [62]. The automatic process of annotat-ing is composed basically of finding terms in doc-uments, mapping them against an ontology, anddisambiguating common terms. There are a widerange of approaches that carry out automatic anno-tation of texts. Most of them employ natural lan-guage processing and information extraction tech-niques. These approaches differ in architecture, in-formation extraction tools and methods, initial on-tology, amount of manual work required to performannotation, as well as performance [58,3].

Existing automated SCA systems can be di-vided into two categories: semi-automatic andfully-automatic systems. In semi-automatic sys-tems [63,40], the user is provided with a set ofsuggestions to select from. So, disambiguation isperformed with the help of user. In fully-automaticsystems [33,32], annotations are generated without

any intervention by users. Fully-automatic systemscan generally be regarded as falling into three cate-gories [62]. The most basic kind use rules or wrap-pers written by hand that try to capture known pat-terns for the annotations. Then there are two kindsof systems that learn how to annotate. Supervisedsystems learn from sample annotations marked upby the user. A problem with these methods is thatpicking enough good examples is a non-trivial anderror-prone task. In order to tackle this problemunsupervised systems employ a variety of strategiesto learn how to annotate without user supervision,but their accuracy is still limited.

Automated SCA systems should take into accountuser interface design issues related to minimizing in-trusiveness while maximizing accuracy. Completelyautomated systems which do not involve any userinteraction in the process of semantic content cre-ation are out of scope of this paper. User interactionis required to supervise, assess or evaluate the auto-mated annotation thereby creating accurate seman-tic content.

3.3.9. EvolvabilityEvolvability is defined as the capacity of a

system for adaptive evolution. An SCA systemshould support evolution of the annotated doc-ument [62,24,44,53,13]. To achieve this goal, itshould take into account the following consistencyconstraints:– Resource Consistency.

If users annotate the same resource in differenttexts, it is important to reference the same re-source in the generated RDF statements. Other-wise, we obtain many resources that are not in-terlinked and the statements in the repository arenot very useful and meaningful [24].

– Document and Annotation Consistency.One of the important issues for the design of asemantic authoring environment is to determinehow changes should be reflected in the knowl-edge base of annotated documents and whetherchanges to ontologies create conflicts with exist-ing annotations [62]. Ontologies change some-times but some documents change many times.So, it is crucial for an SCA system to track theannotation evolution.

An SCA system should provide appropriate UIs forversioning and change tracking to deal with docu-ment and annotation evolution.

12

3.3.10. InteroperabilityInteroperability is the ability of a system to work

and interact with other systems. An SCA systemshould provide mechanisms to interoperate togetherwith other systems which generate or consume thesemantic content created. The following features ofUIs are proposed for improving the interoperabilityof SCA systems:– Support of Standard Formats.

To minimize the problems of interoperabilitythe SCA system should be built on standards.There are already many standards for seman-tic content serialization (e.g. typical RDF se-rializations and particular RDFa), representa-tion (e.g. RDF/RDF-S/OWL/RIF and estab-lished vocabularies such as SIOC, SKOS, FOAF,rNews, etc.) and exchange (e.g. Linked Data,Web Services, REST). Supporting standard for-mats and avoiding proprietary formats are es-sential for compatibility of data with other sys-tems [62,63,7,24,40,53,60,33].

– Semantic Syndication.Semantic syndication supports the distribution ofinformation and their integration into other ap-plications by providing mechanisms such as Se-mantic Atom [48] and Semantic Pingback 15 [4].

3.3.11. ScalabilityScalability refers to the capability of a system

to maintain performance under an increased workload. An SCA system should support scalability asfor example, the number of users, data or annota-tions increase. Support of caching and implement-ing a suitable storage strategy play an important rolein achieving an scalable SCA system [4,62,44,27,40].Annotations can be directly stored in the documentor stored separately in a triple store. Most of the cur-rent SCA systems adopt a dual storage strategy ofsemantic annotations. In this case, annotations arestored in a server-side triple store and also embed-ded in the same document where annotations areundertaken in a way that is completely transparentto the user. A dual storage approach poses a redun-dancy but allows information from heterogeneousresources to be queried centrally and in real-time asa knowledge base [4,62].

15http://aksw.org/Projects/SemanticPingBack

Usability Customizability Generalizability Evolvability

Proactivity

Automation

CollaborationInteroperability Scalability

PortabilityAccessibility

+

+- ++

+

+

+

Fig. 7. Quality attributes dependencies (‘+’: positive effect,

‘+-’: reciprocal effect).

3.4. Quality Attributes Dependencies

The aforementioned quality attributes are notcompletely isolated and independent from eachother but have overlaps and relations with eachother. Figure 7 shows an overview of these qualityattributes with their interrelations. Proactivity, au-tomation and customizability will improve the us-ability of an SCA system. Proactive and automaticsystems provide users with helpers which facilitatethe usage of the system. Customized systems areconfigured based on the user needs thereby increasethe overall usability of the system.

Scalability will enhance the level of system collab-oration. A scalable system will support more usersand annotations thereby more collaboration in thesystem. Interoperability will also enhance the col-laboration support of a system, since an interoper-able system supports users of different systems. Itcan also support importing user data from other sys-tems which will play a positive role in enhancing thecustomizability.

Evolvability and generalizability are directly re-lated. The more evolvable to change a system is, themore generic it will be and vice versa. Customizabil-ity and generalizability share a reciprocal relation. Ageneric system will decrease its customization and acustomizable system needs to focus on specific userneeds and thus lacks generalizability.

3.5. User Types

Table 2 shows the list of tools discussed inour primary studies. The following tools weredescribed in the primary studies: OntoWiki [4],SAHA [17], OWiki [13], SemCards [59], Data-Press [5], Loomp [40], Semantic MediaWiki [27],SweetWiki [7], Information Workbench [42], RD-FAuthor [60], FLERSA [44], LinkedBlog [53],WordPress SIOC plugin [43], HayStack seman-tic blogging [29], Reflect [47], Ontos-feeder [33],

13

Tool User Type Domain Authoring Approach

OntoWiki [4] domain expert General

Top-Down

OWiki [13] domain expert General

SAHA [17] developer & domain expert Governmental data

SemCards [59] end-user (non-expert) General

RDFAuthor [60] end-user & domain-expert General

Tabulator [6] end-user & domain-expert General

Reflect [47] end-user (researchers) Chemistry

Bottom-Up

Epiphany [1] end-user CMS

Ontos-feeder [33] end-user (journalist) Journalism

DataPress [5] end-user (blogger) Blogs & Wikis

Loomp [40] end-user (journalist) Journalism

Semantic MediaWiki [27] end-user (Wiki users) General

LinkedBlog [53] end-user (blogger) Blogs

SweetWiki [7] end-user (Wiki users) General

Linkator [3] end-user CMS

FLERSA [44] end-user CMS

Information Workbench [42] end-user (researchers) Paper review & publishing

HayStack semantic blogging [29] end-user (blogger) Blogs

WordPress SIOC plugin [43] end-user (blogger) Blogs

Table 2

User types, domain and authoring approach of the surveyed SCA systems.

Epiphany [1], Linkator [3], Tabulator [6].For each tool, we extracted the type of user, do-

main of the tool and the authoring approach em-ployed in the tool. There are two genral types ofusers mainly discussed in the studies:– End-user or normal users which have no or lim-

ited knowledge of the domain on which the anno-tations or semantic structures are applied. Theyare consisted of the majority of the populationusing the Internet to browse for information andcommunicate with others.

– Domain experts which have a broad knowledge ofthe domain on which the annotations or semanticstructures are applied. They are usually consistedof the researchers or engineers with a top-downview of problems.

As our results revealed, the majority of studies (i.e.all the tools which employed the bottom-approach)were addressing tools which are appropriate for end-users. Tools which were adopting the top-down ap-proach needed users to have knowledge of the cor-responding domain as well ontology concepts.

3.6. User Interface Evaluation

In this section we briefly outline various methodsfor user interface evaluation and report about theirusage in the surveyed papers. Table 3 lists existingmethods for user interface evaluation adopted from[16,10].

Among the primary studies, the majority of stud-ies (19) were using an Example Application as theirevaluation method. Very few studies (2) were alsousing the discussion method. Other papers were ei-ther survey papers or the papers which only men-tioned some user interface features and did not pro-vide any UI evaluation method. The results dis-tinctly exposes the lack of formal and systematic UIevaluation methods in the context of SCA systems.

4. Tools

In this section we look at four available SCA sys-tems and compare them according to the quality at-tributes defined in Section 3.3. We will investigatetheir strengths and weaknesses based on our pro-

14

Evaluation Method Definition

Empirical Methods

(Case Study)

An empirical inquiry that investigates a contemporary phenomenon within its real-life context; Ausability evaluation specialist tests a well defined hypothesis by measuring subject (user) behavior

while he manipulates variables.

DiscussionProvided some qualitative, textual, opinion-oriented evaluation. e.g., compare and contrast, oral

discussion of advantages and disadvantages.

Example ApplicationAuthors describe an application and provide an example to assist in the description, but the exampleis “used to validate” or “evaluate” as far as the authors suggest.

Observation

(Experience)

The result has been used on real examples, but not in the form of case studies or controlledexperiments, the evidence of its use is collected informally or formally. A usability evaluation specialist

acts as the observer of users as they interact with computers, noting user successes, difficulties, likes,

dislikes, preferences and attitudes.

QuestionnaireThe use of a set of items (questions or statements) to capture statistical data relating to user profiles,

skills, experience, requirements, opinions, preferences and attitudes.

InterviewA formal consultation or meeting between a usability evaluation specialist and user(s) to obtain

information about work practices, requirements, opinions, preferences and attitudes.

User GroupsAvailing of the wealth of knowledge and experience of organized (user forum) and selected (beta

site) end users.

Cognitive

Walkthroughs

A step by step evaluation of a design by a cognitive psychologist in order to identify potential user

psychological difficulties with the system.

Heuristic MethodsThe use of a team of usability evaluation specialists to review a product or prototype in order to

confirm its compliance with recognized usability principles and practice.

Review MethodsThe review and reuse of the wealth of experimental and empirical evidence in the research literature

and in the de-facto standards established by the software industry.

Simulation Execution of a system within artificial data, using a model of the real world.

Modeling MethodsUsing models like GOMS (Goals, Operations, Methods and Selection) and KLM (Keystroke Level

Modeling) to predict and provide feedback on user interactions and difficulties.

Table 3User interface evaluation methods.

posed taxonomy of quality attributes and UI fea-tures which are required for SCA systems. Amongthe tools two (i.e. Ontowiki and SAHA 3) follow thetop-down approach (cf. Section 3.2.2) and two (i.e.Loomp and RDFaCE) follow a bottom-up approach(cf. Section 3.2.1) for semantic content authoring.Figure 8 summarizes the assessment of the tools ac-cording to the defined quality attributes.

4.1. OntoWiki

OntoWiki 16 [4] is a tool that provides supportfor agile, distributed knowledge engineering scenar-ios. Ontowiki facilitates the visual presentation of aknowledge base as an information map, with differ-ent views on instance data.

Regarding the technical realization, the system isimplemented in PHP using the Zend framework 17 .

16http://ontowiki.net17http://framework.zend.com/

It supports the MySQL database and the Virtuosotriple store 18 as storage backends and the authoringinterface is built using jQuery UI 19 .

Figure 9 shows a screenshot of OntoWiki. On-toWiki was applied in a number of use cases.Examples include: semantic content manage-ment [25], collaborative requirements engineeringwith SoftWiki [38] and historical, prosopographicalknowledge engineering with the Professor’s Cata-logue [52].

Ontowiki as a single point of entry UI adopts thetop-down approach for semantic authoring. It pro-vides a semantic search feature with support forfaceted browsing. It also supports two complemen-tary knowledge base authoring strategies: a) Inlineediting, which enables users to edit small informa-tion chunks (i.e. statements). b) View editing, whichenables users to edit common combinations of infor-

18http://virtuoso.openlinksw.com/19http://jqueryui.com/

15

OntoWiki SAHA 3 Loomp RDFaCE

Usability • Single point of entry UI

• Faceted browsing

• Inline editing/view editing

• Single point of entry UI

• Faceted browsing

• Inline editing

• Single point of entry UI

• Faceted viewing

• Single point of entry UI

• Faceted viewing

• Inline editing

Customizability • Semantic views: domain

specific & generic (e.g. map,

calendar)

- -

• Semantic views: WYSIWYM,

WYSIWYG, triple, source

code view

Generalizability • Multiple ontology support

• Ontology modification support • Multiple ontology support • Multiple ontology support • Multiple ontology support

Collaboration

• Access control

• Standard formats: RDF, RDFa

• Social collaboration UIs:

rating and commenting UIs

• Access control

• Standard formats: RDF

• Social collaboration UIs:

online chat

• Standard formats: RDF,

RDFa • Standard formats: RDFa

Portability • Cross-browser compatibility

• UI for mobile devices • Cross-browser compatibility • Cross-browser compatibility • Cross-browser compatibility

Accessibility - - - -

Proactivity • Resource suggestion

• Concept reuse

• Resource suggestion

• Concept reuse

• Real-time validation

• Resource suggestion

• Concept reuse • Resource suggestion

Automation - - - • Automatic annotation: using

NLP APIs

Evolvability

• Resource consistency

• Document & annotation

consistency

• Versioning & change tracking

• Resource consistency

• Document and annotation

consistency

• Document and annotation

consistency

• Document and annotation

consistency

Interoperability • Standard formats: RDF, RDFa

• Semantic syndication:

semantic pingback

• Standard formats: RDF • Standard formats: RDF,

RDFa • Standard formats: RDFa

Scalability • Caching support

• Storage strategy: backend

independent (Mysql, Virtuoso)

• Storage strategy: using a

server-side triple store

• Storage strategy: using a

server-side triple store

• Storage strategy: using on-

the-fly client -side triple

storage

Fig. 8. Comparison of OntoWiki, SAHA 3, Loomp and RDFaCE according to the quality attributes.

mation (such as an instance of a distinct class) in onesingle step. In order to do so, Ontowiki uses RDFAu-thor [60] to make generated RDFa views editable.Regarding the customizability, OntoWiki supportsdifferent semantic views of the knowledge base whichcan be generic or domain-specific. It also supportsediting multiple ontologies including both the in-stances and structures of the ontologies. As a Web-based system, it provides cross-browser compatibil-ity and has a specific UI for mobile devices. To pro-vide proactivity, OntoWiki uses the AJAX technol-ogy to interactively propose already defined con-cepts while the user types in new information to beadded to the knowledge base (i.e. Concept Reuse).

OntoWiki also provides versioning and evolu-tion features to track, review and selectively roll-back changes and supports semantic syndication(employing Semantic Pingback and Linked Datainterfaces) to interoperate with other systems. On-toWiki is backend independent to some extend andsupports two different types of storage engines. Italso provides a caching component to optimize the

performance of the system.As a drawback, OntoWiki does not provide any UI

elements to facilitate accessibility and automation.It supports only the editing of structured contentthus lacking UIs for the annotation of unstructuredor semi-structured content. Furthermore, it does notprovide real-time tagging and validation for increas-ing the overall proactivity.

4.2. SAHA 3 Metadata Editor

SAHA 3 20 [17] is an RDF metadata editor forcollaborative content creation and instant semanticcontent publishing on the Semantic Web. Regardingthe technical realization, the system is implementedin Java on top of the Spring framework 21 . The datamodel is based on the Jena TDB 22 RDF database

20http://demo.seco.tkk.fi/saha/saha3/21http://www.springsource.com/22http://openjena.org/TDB/

16

Fig. 9. Screenshot of the OntoWiki instance view with inlineediting.

and the editor interface is built using DWR 23 andthe Dojo 24 AJAX components.

Figure 10 shows a screenshot of SAHA 3. DataFin-land 25 as a semantic portal for open and linkeddatasets is one use case of SAHA 3.

Like OntoWiki, SAHA 3 uses the top-down ap-proach for semantic authoring and a single pointof entry UI with inline editing features. It supportsfaceted browsing when integrated into the facetedportal engine HAKO 26 for content publishing.SAHA 3 supports multiple ontologies as well as col-laborative simultaneous editing. Resources that arebeing edited by one user are locked for editing byother users. A chat facility has been integrated intothe editor to facilitate instant discussions betweenpeer editors.

Regarding proactivity, SAHA 3 also providesreal-time semantic validation and concept reuse. Asshown in [35], SAHA 3 has proven a good level ofscalability to support large projects.

As a drawback, SAHA 3 does not provide anyUI elements to provide customizability, accessibilityand automation. Like OntoWiki, it only supportsstructured content authoring thus lacking the appro-priate UIs for unstructured or semi-structured con-tent annotation. Although it provides some simpleUIs for supporting collaboration but lacks sophisti-cated features regarding social interactions. Since itdoes not provide any UI elements for versioning andchange tracking, evolvability is not well addressed.Scalability could be further improved by adding sup-

23http://directwebremoting.org/24http://www.dojotoolkit.org/25http://www.seco.tkk.fi/linkeddata/datasuomi/26http://www.seco.tkk.fi/tools/hako/

port for caching and alternative storage backends(i.e. client-side RDF processing).

Fig. 10. Screenshot of the SAHA 3 inline editing.

4.3. Loomp

Loomp 27 [40] is a tool representing a prove-of-concept for the One Click Annotation (OCA) strat-egy. The Web-based OCA editor allows for annotat-ing words and phrases with references to ontologyconcepts and for creating relationships between an-notated phrases [24].

Regarding the technical realization, Loomp isa typical Web application built on the LAMPstack 28 . It serves content either in RDF (e.g. forlinked data clients) or in XHTML/RDFa (e.g. forWeb browsers).

Figure 11 shows a screenshot of Loomp. Data-driven journalism is mentioned as one of the primaryuses cases of Loomp.

Loomp provides a WYSIWYG editor as a singlepoint of entry UI which adopts the bottom-up ap-proach for semantic content authoring. It supportsa faceted viewing feature which highlights user-selected annotations in the Web browser. Loompfacilitates concept reuse and suggestion in orderto reduce non-expert user efforts during the an-notation process. It also employs RDF and RDFa

27http://loomp.org28LAMP: Linux operating system, Apache web server,

MySQL database, PHP scripting language.

17

standard formats which make it interoperable withother systems.

As a drawback, Loomp lacks appropriate UI ele-ments to support customizability, accessibility andautomation. It does not provide any UI features forfaceted browsing and inline editing of annotations.It also does not allow to directly edit the underly-ing ontologies thereby extending the annotation do-main. Furthermore, Loomp lacks appropriate UI el-ements for real-time tagging and validation as wellas versioning and change tracking. Regarding scala-bility, no information could be found on how Loompsupports large amounts of users and annotations.

Fig. 11. Screenshot of the Loomp faceted viewing UI.

4.4. RDFaCE

RDFaCE 29 [30] (RDFa Content Editor) is anWeb-based RDFa content editor based on theTinyMCE 30 rich text editor. It supports differentviews for semantic content authoring and uses ex-isting Semantic Web APIs to facilitate annotatingand editing of RDFa content. RDFaCE introducesthe WYSIWYM (What-You-See-Is-What-You-Mean) concept for semantic text authoring whichis an extension (incorporating semantic views) tothe WYSIWYG (What-You-See-Is-What-You-Get)editing model.

Regarding the technical realization, RDFaCE iswritten completely in JavaScript and utilizes thejQuery UI library for the authoring user interface.Annotations are created on-the-fly using the client-side triple store RDFQuery 31 , which makes a sep-arate backend for storing annotations obsolete.

Figure 12 shows an screenshot of RDFaCE. Usecases of the RDFaCE include the annotation of

29http://rdface.aksw.org30http://tinymce.moxiecode.com31http://code.google.com/p/rdfquery/

news articles with metadata using the rNews vocab-ulary 32 and semantic blogging with Wordpress.

Like Loomp, RDFaCE consists of a single pointof entry UI which supports faceted viewing and in-line editing of annotations. It provides different se-mantic views for different personas involved in theprocess of semantic content authoring. Furthermore,RDFaCE supports resource suggestion and auto-matic content annotation using external NLP APIs.Since RDFaCE processes the annotations client-sidewithin the user’s browser and does not require anycentral storage backend, RDFaCE is highly scalable.

As a drawback, RDFaCE lacks the appropriateUI elements to support accessibility. It also does notprovide a faceted browsing UI and does not allowto directly edit the underlying annotation ontolo-gies. Beside supporting the RDFa standard format,it does not provide any UI features for social collab-oration as well as versioning and change tracking.

Fig. 12. Screenshot of the RDFaCE triple viewer and editor.

5. Research and Technology Challenges

The results of our systematic review revealedsome research and technology gaps and correspond-ing challenges with regard to the development ofSCA UIs.

i. Accessibility. There is a clear research gap inaddressing accessibility issues during the design ofSCA UIs. Many semantic authoring tools remain in-accessible to people with disabilities. Providing peo-ple with disabilities and special needs with appropri-ate SCA UIs can facilitate their tasks of informationseeking. Semantically annotated content allows al-ternatives or conditional content in different modal-ities to be selected based on the type of the user

32http://dev.iptc.org/rNews

18

disability and information need. For example, visu-ally impaired people, need significantly more timeto grasp the structure and gist of a Web site, sincevisual navigation and structuring elements are notaccessible to them. Once content is semantically an-notated, visually impaired people can use this se-mantic annotation as a means to access and explorethe content faster.

The Web Content Accessibility Guidelines(WCAG) 33 explain how to make Web contentmore accessible to people with disabilities. As partof WCAG, Authoring Tool Accessibility Guide-lines (ATAG) 34 define how authoring tools shouldsupport accessibility requirements. Consequently,a challenge is to apply and extend the series ofaccessibility guidelines proposed in ATAG for thepurpose of designing accessible SCA UIs.

ii. Handling complexity in UIs. One importantconcern when designing SCA UIs is how to makecomplex functionality available to the user in asimple way. There are two issues in this contextwhich need to be addressed. The first one is howto properly map complex functions and algorithms(e.g. entity disambiguation, recommendation andother machine learning algorithms) to correspond-ing user interface elements. The second issue is howto flatten the user’s learning curve by providingadaptive and intelligent UIs which take user knowl-edge into account. Many current SCA systems beara bewildering amount of functions and algorithmswhich confuses both the novice and expert users.This problem causes a significant impediment for abroader use of SCA systems.

Addressing the complexity problem requires thecreation of abstract models for complex tasks as wellas modeling the user characteristics and behavior.Ideally, the SCA UI should present the users withconcepts that are consistent with both designer andusers’ mental models of that phenomenon in thereal world. The above mentioned issues are well ad-dressed in designing Geographic Information Sys-tem (GIS) UIs [64,9]. Now it is a challenge to rethinkthese issues for the purpose of designing adaptiveand flexible SCA UIs.

iii. Formal and systematic methods for user inter-face evaluation. The results of our survey clearly

33http://www.w3.org/WAI/intro/wcag.php34http://www.w3.org/WAI/intro/atag.php

reflects the lack of formal and systematic UI eval-uation methods in evaluating SCA systems. As de-scribed in Table 3, there are several UI evaluationmethods which can be used in this context. Nielsenand Molich [46] enumerate four general categoriesof systematic user interface evaluation methods: for-mally by employing an analysis technique; automat-ically by a computerized procedure; empirically bytesting users performing experiments; and heuristi-cally.

In heuristic evaluation, evaluators inspect a userinterface against a guideline to identify usabilityproblems that violate any items on the guide-line [34]. Our list of quality attributes and UI fea-tures (cf. Section 3.3) can be used as a guideline forheuristic evaluation of SCA system UIs. This will re-quire less resources than testing with users and canbe applied to the system during the design phase.

iv. Support of crowdsourcing. One of the missingaspects of developing collaborative SCA systemsis the support of crowdsourcing. There are a hugeamount of amateur and expert users which arecollaborating and contributing on the Social Web.Harnessing the power of such crowds can signifi-cantly enhance and widen the results of semanticcontent authoring and annotation. Crowdsourcingas a distributed problem-solving and productionmodel is defined to address this aspect of collectiveintelligence [28].

In order to support crowdsourcing, an SCA sys-tem needs to provide appropriate UIs. In [18], Geigeret al. analyze the respective characteristics and re-quirements related to the design of crowdsourcingsystems. Providing small contributions with instantgratification, altruism and a way to establish a rep-utation are some of these requirements. As a newchallenge, it is worth to consider these characteris-tics when designing SCA UIs.

v. UIs for ubiquitous devices. As discussed in Sec-tion 3.3.5, creating UIs for mobile and ubiquitousdevices is an issue which is not well addressed inthe literature yet. Mobile and ubiquitous devicesare rapidly becoming the central computing andcommunication devices in people’s lives. Ubiquitouscomputing (a.k.a everyware [20]) presents new chal-lenges in user interface design. Emerging ubiquitousdevices are programmable and come with a grow-ing set of facilities including multi-touch screens andcheap powerful embedded sensors, such as an ac-

19

celerometer, digital compass, gyroscope, GPS, mi-crophone, and camera [36]. Utilizing these rich set ofUI facilities when developing SCA systems can im-prove the user experience in the process of seman-tic content authoring and annotation. For example,users can easily share their real-time activities withSCA system using mobile sensors or can use somegestures for annotating the content.

Another challenge here is the ability to provideoffline functionality with local updates for later syn-chronization with a web server. SCA systems shouldtake into account the reconciliation problem – theproblem of potentially conflicting updates from dis-connected clients.

6. Conclusions and Feature Work

In this paper we reported on the results of a sys-tematic literature review on user interfaces for se-mantic content authoring comprising initially 175papers. The review aimed to answer the five researchquestions defined in Section 2.1 by thorough analy-sis of the 31 most relevant papers. Before addressingthe defined research questions, we drew a terminol-ogy which defines the basic concepts used in the lit-erature as well our survey. To answer the RQ1, weclassified existing approaches for SCA into two cate-gories namely Top-Down and Bottom-Up discussedin Section 3.2. Furthermore, Our data analysis re-vealed a set of quality attributes for SCA systemstogether with the corresponding user interface fea-tures which are suggested for their realization. Thesequality attributes plus the UI features are used toanswer the RQ2 and RQ3. In order to answer RQ4and RQ5 we extracted the types of users as well asuser evaluation methods discussed in the primarystudies and reflected the results in Section 3.6. Openresearch and technological challenges in the domainof SCA systems were discussed as well. Addition-ally, to show the applicability of the results, we per-formed an in-depth comparison of four existing SCAsystems according to the defined quality attributesand described their strengths as well as their weak-nesses. Figure 13 shows an overview of the resultssurveyed in this paper.

Essential, foundational quality attributes for anSCA system are, in particular, usability, general-izability, customizability and evolvability. A basicSCA system should fulfill a reasonable level ofuser-friendliness and adopt to different situationsor use cases while providing mechanisms to tailor

its functionality based on specific user needs. Italso should take into account issues such as con-sistency constraints and content evolution whichare required for change management. Support ofcollaboration, interoperability and scalability arequality attributes required when an SCA system isemployed in a community-driven environment withlarge amount of users, systems and interactions.An SCA system should support standard formatsand provide appropriate UI elements for social net-working including both human-to-human as wellas system-to-system interactions. Additionally, itshould maintain performance under an increasedwork load by supplying appropriate storage andcaching mechanisms. Automation and proactivityare quality attributes which facilitate usability ofSCA systems especially for non-skilled users. Porta-bility and accessibility are, as our survey indicates,not well addressed by the literature so far. The de-mands for suitable UIs for mobile and ubiquitousdevices are increasing as powerful mobile computingdevices are becoming more common. Furthermore,providing accessible UIs for people with disabili-ties or special needs is another requirement whichshould be taken into account when designing SCAsystems.

While there are many benefits of systematic re-views, they also bear some limitations and validitythreats originating from human errors. The mainthreats to validity of our systematic review aretwofold: correct and thorough selection of the stud-ies to be included as well as accurate and exhaustiveselection of quality attributes together with theircorresponding UI features. With the increasingnumber of works in the area of semantic content au-thoring we can not guarantee to have captured allthe material in this area. The scope of our review isrestricted to the scientific domain. Therefore, sometools or approaches employed in the industry mighthave not been included in our primary studies.Furthermore, since the review process was mainlyperformed by one researcher a bias is possible. Inorder to mitigate a potential subjective bias, thereview protocol and results were checked and val-idated by a senior researcher and other colleaguesexperienced in the context of Semantic Web.

We see this effort and in particular the identifi-cation of a comprehensive set of quality attributesas a crucial step towards developing more effectiveand user-friendly authoring tools for realizing theSocial Semantic Web. New approaches and tools canbe evaluated in the light of these quality attributes,

20

thus revealing additional aspects to be taken intoconsideration. As a result, more user-friendly toolswill enable more people to interact with the Seman-tic Web thereby facilitating the realization of theintelligent Web vision.

As future work, we envision strategies to semi-automatically improve the realization of the qualityattributes, for example, using active machine learn-ing for better integration with approaches deliveringautomatic suggestions. Also extending the supportfor integration of multi-media and multi-modal se-mantic annotation (e.g. of images and multimediacontent) is a promising research direction.

Integrating SCA systems into other applicationslike speech recognition and question-answering sys-tems for improving the accuracy and quality ofresults is another important area of future work.At the moment, intelligent mobile assistants (e.g.Siri 35 for the iPhone) only allow delegation ofcertain programmed tasks (e.g. making restaurantreservations, getting movie tickets, etc.) by invokingcertain predefined web services. Employing seman-tically enriched content in the UI of mobile personalagents will extend their capability to inquiry theopen Web of Data thereby achieving more efficientand effective results.

Addressing open research and technology chal-lenges such as accessibility, handling complexity inUIs, formal and systematic methods for user inter-face evaluation, support of crowdsourcing and UIsfor ubiquitous devices discussed in the paper areother interesting areas for future research.

7. Acknowledgments

We would like to thank our colleagues fromAKSW research group for their helpful commentsand inspiring discussions during the conducting ofthis systematic review. This work was supported bya grant from the European Union’s 7th FrameworkProgramme provided for the project LOD2 (GA no.257943).

References

[1] B. Adrian, J. Hees, I. Herman, M. Sintek, A. Dengel,Epiphany: Adaptable rdfa generation linking the web of

documents to the web of data, in: P. Cimiano, H. Pinto(eds.), Knowledge Engineering and Management by

35http://www.siri.com

the Masses, vol. 6317 of Lecture Notes in Computer

Science, Springer Berlin / Heidelberg, 2010, pp. 178–

192, 10.1007/978-3-642-16438-5-13.URL

http://dx.doi.org/10.1007/978-3-642-16438-5-13

[2] A. Ankolekar, M. Krotzsch, T. Tran, D. Vrandecic, The

two cultures: mashing up web 2.0 and the semantic web,in: WWW ’07: Proceedings of the 16th international

conference on World Wide Web, ACM Press, New York,

NY, USA, 2007.URL http://dx.doi.org/10.1145/1242572.1242684

[3] S. Araujo, G.-J. Houben, D. Schwabe, Linkator:

Enriching web pages by automatically adding

dereferenceable semantic annotations, in: B. Benatallah,F. Casati, G. Kappel, G. Rossi (eds.), Web Engineering,

vol. 6189 of Lecture Notes in Computer Science, SpringerBerlin / Heidelberg, 2010, pp. 355–369, 10.1007/978-3-

642-13911-6-24.

URLhttp://dx.doi.org/10.1007/978-3-642-13911-6-24

[4] S. Auer, S. Dietzold, T. Riechert, Ontowiki – a

tool for social, semantic collaboration, in: I. Cruz,

S. Decker, D. Allemang, C. Preist, D. Schwabe, P. Mika,M. Uschold, L. Aroyo (eds.), The Semantic Web -

ISWC 2006, vol. 4273 of Lecture Notes in Computer

Science, Springer Berlin / Heidelberg, 2006, pp. 736–749, 10.1007/11926078-53.

URL http://dx.doi.org/10.1007/11926078-53

[5] E. Benson, A. Marcus, F. Howahl, D. Karger, Talking

about data: Sharing richly structured informationthrough blogs and wikis, in: P. Patel-Schneider, Y. Pan,

P. Hitzler, P. Mika, L. Zhang, J. Pan, I. Horrocks,

B. Glimm (eds.), The Semantic Web – ISWC 2010, vol.6496 of Lecture Notes in Computer Science, Springer

Berlin / Heidelberg, 2010, pp. 48–63, 10.1007/978-3-642-17746-0-4.

URL

http://dx.doi.org/10.1007/978-3-642-17746-0-4

[6] T. Berners-Lee, J. Hollenbach, K. Lu, J. Presbrey, E. P.d’ommeaux, m.c. schraefel, Tabulator redux: Writing

into the semantic web (2007).

URL http://eprints.ecs.soton.ac.uk/14773/

[7] M. Buffa, F. Gandon, G. Ereteo, P. Sander, C. Faron,Sweetwiki: A semantic wiki, Web Semantics: Science,

Services and Agents on the World Wide Web 6 (1) (2008)

84 – 97, semantic Web and Web 2.0.URL http://www.sciencedirect.com/science/

article/pii/S1570826807000522

[8] G. Burel, A. E. Cano1, V. Lanfranchi, Ozone browser:

Augmenting the web with semantic overlays, vol. 449 ofCEUR Workshop Proceedings ISSN 1613-0073, 2009.

URL http://CEUR-WS.org/Vol-449/Challenge1.pdf

[9] G. Camara, R. C. M. Souza, A. M. Monteiro, J. Paiva,J. Garrido, G. Camara, R. Cartaxo, M. D. Souza,A. Miguel, V. Monteiro, J. Carlos, P. D. Garrido,

D. Processamento, I. Dpi, Handling complexity in gis

interface design, in: In: Proceedings of the I BrazilianWorkshop on GeoInformatics, Campinas, Sao, 1999.

[10] L. Chen, M. A. Babar, A systematic review of evaluation

of variability management approaches in softwareproduct lines., Information & Software Technology53 (4) (2011) 344–362.

21

URL http://dblp.uni-trier.de/db/journals/

infsof/infsof53.html#ChenB11

[11] H.-C. Chu, M.-Y. Chen, Y.-M. Chen, A semantic-basedapproach to content abstraction and annotation for

content management, Expert Systems with Applications

36 (2, Part 1) (2009) 2360 – 2376.URL http://www.sciencedirect.com/science/

article/pii/S0957417407006860

[12] M. d’Aquin, E. Motta, M. Dzbor, L. Gridinoc, T. Heath,M. Sabou, Collaborative semantic authoring, Intelligent

Systems, IEEE 23 (3) (2008) 80 –83.

[13] A. Di Iorio, A. Musetti, S. Peroni, F. Vitali, Ontology-driven generation of wiki content and interfaces, NEW

REVIEW OF HYPERMEDIA AND MULTIMEDIA

16 (1-2, SI) (2010) 9–31.

[14] T. Dyba, T. Dingsoyr, G. K. Hanssen, Applying

systematic reviews to diverse study types: An experience

report, in: Proceedings of the First InternationalSymposium on Empirical Software Engineering and

Measurement, ESEM ’07, IEEE Computer Society,Washington, DC, USA, 2007.

URL http://dx.doi.org/10.1109/ESEM.2007.21

[15] T. Ermilov, N. Heino, S. Tramp, S. Auer, Ontowikimobile – knowledge management in your pocket, in:

8th Extended Semantic Web Conference (ESWC2011),

2011.URL http://data.semanticweb.org/conference/

eswc/2011/paper/mobile-web/3

[16] R. Fitzpatrick, Strategies for evaluating softwareusability, Methods 353 (1).

URL http://citeseerx.ist.psu.edu/viewdoc/

download?doi=10.1.1.169.9992&rep=rep1&type=pdf

[17] M. Frosterus, E. Hyvonen, J. Laitio, Datafinland—a

semantic portal for open and linked datasets, in:

G. Antoniou, M. Grobelnik, E. Simperl, B. Parsia,D. Plexousakis, P. De Leenheer, J. Pan (eds.), The

Semanic Web: Research and Applications, vol. 6644 of

Lecture Notes in Computer Science, Springer Berlin/ Heidelberg, 2011, pp. 243–254, 10.1007/978-3-642-

21064-8-17.URLhttp://dx.doi.org/10.1007/978-3-642-21064-8-17

[18] D. Geiger, M. Rosemann, E. Fielt, Crowdsourcinginformation systems : a systems theory perspective,

in: Australasian Conference on Information Systems

(ACIS) 2011, Sydney, Australia, 2011.URL http://eprints.qut.edu.au/47466/

[19] B. G. Glaser, A. L. Strauss, The Discovery of Grounded

Theory: Strategies for Qualitative Research, Aldine deGruyter, New York, NY, 1967.

[20] A. Greenfield, Everyware: The Dawning Age ofUbiquitous Computing, New Riders Publishing,Berkeley, CA, 2006.

[21] P. Haase, A. Eberhart, S. Godelet, T. Mathaß, T. Tran,

G. Ladwig, A. Wagner, The information workbench.interacting with the web of data, in: 3rd Future InternetSymposium (FIS2010), 2010.

URL http://data.semanticweb.org/conference/fis/

2010/paper/posters/11

[22] G. Hachey, D. Gasevic, Semantic web user interface: A

systematic survey, Master’s thesis, Athabasca University(2011).

[23] K. Hasida, Semantic authoring and semantic computing,in: A. Sakurai, K. Hasida, K. Nitta (eds.), New Frontiers

in Artificial Intelligence, vol. 3609 of Lecture Notes in

Computer Science, Springer Berlin / Heidelberg, 2007,pp. 137–149, 10.1007/978-3-540-71009-7-12.

URL

http://dx.doi.org/10.1007/978-3-540-71009-7-12

[24] R. Heese, M. Luczak-Rosch, R. Oldakowski, O. Streibel,

A. Paschke, One click annotation, in: Scripting and

Development for the Semantic Web (SFSW), 2010.URL http://www.semanticscripting.org/SFSW2010/

[25] N. Heino, S. Tramp, S. Auer, Managing web content

using linked data principles – combining semanticstructure with dynamic content syndication, in:

Proceedings of the 35th Annual IEEE International

Computer Software and Applications Conference(COMPSAC 2011), IEEE Computer Society, 2011.

URL http://svn.aksw.org/papers/2011/COMPSAC_

lod2.eu/public.pdf

[26] B. Heitmann,S. Kinsella, C. Hayes, S. Decker, Implementing semantic

web applications: reference architecture and challenges,

in: 5th International Workshop on Semantic Web-Enabled Software Engineering, 2009.

[27] D. Herzig, B. Ell, Semantic mediawiki in operation:

Experiences with building a semantic portal, in: P. Patel-Schneider, Y. Pan, P. Hitzler, P. Mika, L. Zhang, J. Pan,

I. Horrocks, B. Glimm (eds.), The Semantic Web –

ISWC 2010, vol. 6497 of Lecture Notes in ComputerScience, Springer Berlin / Heidelberg, 2010, pp. 114–

128, 10.1007/978-3-642-17749-1-8.

URLhttp://dx.doi.org/10.1007/978-3-642-17749-1-8

[28] J. Howe, The rise of crowdsourcing, Wired Magazine

14 (6).URL http:

//www.wired.com/wired/archive/14.06/crowds.html

[29] D. R. Karger, D. Quan, What would it mean to blog

on the semantic web?, Web Semantics: Science, Servicesand Agents on the World Wide Web 3 (2-3) (2005) 147

– 157, selcted Papers from the International Semantic

Web Conference, 2004 - ISWC, 2004.URL http://www.sciencedirect.com/science/

article/pii/S1570826805000107

[30] A. Khalili, S. Auer, D. Hladky, The rdfa contenteditor - from wysiwyg to wysiwym, in: IEEE Signature

Conference on Computers, Software, and Applications,

COMPSAC 2012, 2012.

[31] B. Kitchenham, Procedures for performing systematicreviews, Tech. rep., Keele University and NICTA (2004).

[32] N. Kiyavitskaya, N. Zeni, J. R. Cordy, L. Mich,J. Mylopoulos, Cerno: Light-weight tool support forsemantic annotation of textual documents, Data &

Knowledge Engineering 68 (12) (2009) 1470 – 1492,

including Special Section: 21st IEEE InternationalSymposium on Computer-Based Medical Systems

(IEEE CBMS 2008) - Seven selected and extendedpapers on Biomedical Data Mining.URL http://www.sciencedirect.com/science/

article/pii/S0169023X09001116

[33] A. Klebeck, S. Hellmann, C. Ehrlich, S. Auer,Ontosfeeder – a versatile semantic context provider for

22

web content authoring, in: G. Antoniou, M. Grobelnik,

E. Simperl, B. Parsia, D. Plexousakis, P. De Leenheer,

J. Pan (eds.), The Semanic Web: Research andApplications, vol. 6644 of Lecture Notes in Computer

Science, Springer Berlin / Heidelberg, 2011, pp. 456–

460, 10.1007/978-3-642-21064-8-34.URL

http://dx.doi.org/10.1007/978-3-642-21064-8-34

[34] E. D. Kock, J. V. Biljon, M. Pretorius, Usabilityevaluation methods : Mind the gaps, Evaluation (2009)

122–131.

URL http:

//portal.acm.org/citation.cfm?id=1632149.1632166

[35] J. Kurki, E. Hyvonen, Collaborative metadata editor

integrated with ontology services and faceted portals,in: 1st Workshop on Ontology Repositories and Editors

for the Semantic Web, 2010.

[36] N. D. Lane, E. Miluzzo, H. Lu, D. Peebles,T. Choudhury, A. T. Campbell, A survey of mobile

phone sensing (2010).

[37] S. Lauesen, User Interface Design: A SoftwareEngineering Perspective, Addison Wesley, 2005.

URL http://www.amazon.com/exec/obidos/redirect?

tag=citeulike07-20&path=ASIN/0321181433

[38] S. Lohmann, P. Heim, S. Auer, S. Dietzold, T. Riechert,Semantifying requirements engineering – the softwiki

approach, in: Proceedings of the 4th InternationalConference on Semantic Technologies (I-SEMANTICS

’08), J.UCS, 2008.

[39] V. Lopez, V. Uren, M. Sabou, E. Motta, Is question

answering fit for the semantic web? a survey, SemanticWeb ? Interoperability, Usability, Applicability 2 (2)

(2011) 125–155.URL http://oro.open.ac.uk/29573/

[40] R. H. M. Luczak-Roesch, Linked data authoring for

non-experts, in: Proceedings of the WWW09, Workshop

Linked Data on the Web (LDOW2009), 2009.URL http://events.linkeddata.org/ldow2009/

papers/ldow2009-paper4.pdf

[41] M. B. Miles, M. Huberman, Qualitative Data Analysis:An Expanded Sourcebook(2nd Edition), 2nd ed., Sage

Publications, Inc, 1994.

URL http://www.amazon.com/exec/obidos/redirect?

tag=citeulike07-20&path=ASIN/0803955405

[42] W. Muller, I. Rojas, A. Eberhart,

P. Haase, M. Schmidt, A-r-e: The author-review-executeenvironment, Procedia Computer Science 4 (2011) 627

– 636, proceedings of the International Conference on

Computational Science, ICCS 2011.URL http://www.sciencedirect.com/science/

article/pii/S1877050911001244

[43] K. Moller, U. Bojars, J. C. Breslin, Using semanticsto enhance the blogging experience, in: Y. Sure,

J. Domingue (eds.), SEMANTIC WEB: RESEARCH

AND APPLICATIONS, PROCEEDINGS, vol. 4011of LECTURE NOTES IN COMPUTER SCIENCE,Sekt; Dip; Knowledgeweb; Asg; Ontotext; iSOCO;Luisa; AKT; Oasis, 2006, 3rd European Semantic WebConference, Budva, SERBIA MONTENEG, JUN 11-14,

2006.

[44] J. L. Navarro-Galindo, J. Samos, Manual and automaticsemantic annotation of web documents: the flersa tool,

in: Proceedings of the 12th International Conferenceon Information Integration and Web-based Applications

& Services, iiWAS ’10, ACM, New York, NY, USA,

2010.URL http://doi.acm.org/10.1145/1967486.1967570

[45] J. Nielsen, Introduction to usability.

URL http://www.useit.com/alertbox/20030825.html

[46] J. Nielsen, R. Molich, Heuristic evaluation of user

interfaces, in: Proceedings of the SIGCHI conferenceon Human factors in computing systems: Empowering

people, CHI ’90, ACM, New York, NY, USA, 1990.URL http://doi.acm.org/10.1145/97243.97281

[47] S. I. O’Donoghue, H. Horn, E. Pafilis, S. Haag, M. Kuhn,V. P. Satagopam, R. Schneider, L. J. Jensen, Reflect:

A practical approach to web semantics, Web Semantics:Science, Services and Agents on the World Wide Web

8 (2-3) (2010) 182 – 189, bridging the Gap–Data

Mining and Social Network Analysis for IntegratingSemantic Web and Web 2.0; The Future of Knowledge

Dissemination: The Elsevier Grand Challenge for the

Life Sciences.URL http://www.sciencedirect.com/science/

article/pii/S1570826810000193

[48] D. R. Patel, S. A. Khuba, Realization of semantic atom

blog, CoRR abs/0912.3957.URL http://dblp.uni-trier.de/db/journals/corr/

corr0912.html#abs-0912-3957

[49] H. Paulheim, F. Probst, Ontology-enhanced user

interfaces: A survey, International Journal on SemanticWeb and Information Systems (IJSWIS), 6 (2010) 2.

URL http://www.heikopaulheim.com/docs/ijswis\

discretionary-2010.pdf

[50] F. Perdrix, R. Garcıa, R. Gil, M. Oliva, J. A. Macıas,Semantic web interfaces for newspaper multimedia

content management, in: New Trends on Human-

Computer Interaction, Springer London, 2009, pp. 1–10,10.1007/978-1-84882-352-53.

URLhttp://dx.doi.org/10.1007/978-1-84882-352-53

[51] V. Quint, I. Vatton, Structured templates for authoring

semantically rich documents, in: Proceedings of the 2007

international workshop on Semantically aware documentprocessing and indexing, SADPI ’07, ACM, New York,

NY, USA, 2007.

URL http://doi.acm.org/10.1145/1283880.1283889

[52] T. Riechert, U. Morgenstern, S. Auer, S. Tramp,M. Martin, The catalogus professorum lipsiensis –

semantics-based collaboration and exploration for

historians, in: Proceedings of the 9th InternationalSemantic Web Conference (ISWC2010), Lecture Notes

in Computer Science, Springer, Shanghai / China, 2010.

URL http://svn.aksw.org/papers/2010/ISWC_CP_

demo/iswc_cp_demo.pdf

[53] I. Ruiz-Rube, C. M. Cornejo, J. M. Dodero, V. M.

Garcıa, Development issues on linked data weblog

enrichment, in: S. Sanchez-Alonso, I. N. Athanasiadis(eds.), Metadata and Semantic Research, vol. 108

of Communications in Computer and InformationScience, Springer Berlin Heidelberg, 2010, pp. 235–246,

10.1007/978-3-642-16552-8-22.

URLhttp://dx.doi.org/10.1007/978-3-642-16552-8-22

23

[54] M. Sah, W. Hall, N. M. Gibbins, D. C. D. Roure,

Semport ? a personalized semantic portal, in: 18th ACM

Conference on Hypertext and Hypermedia, Sheridanprinting, 2007.

URL http://eprints.ecs.soton.ac.uk/14441/

[55] C. B. Seaman, Qualitative methods in empirical studiesof software engineering., IEEE Trans. Software Eng.

25 (4) (1999) 557–572.

URL http://dblp.uni-trier.de/db/journals/tse/

tse25.html#Seaman99

[56] A. Seffah, M. Donyaee, R. B. Kline, H. K. Padda,

Usability measurement and metrics: A consolidatedmodel, Software Quality Control 14 (2) (2006) 159–178.

URL http://dx.doi.org/10.1007/s11219-006-7600-8

[57] P. Sheu, H. Yu, C. V. Ramamoorthy, A. K. Joshi, L. A.

Zadeh, Semantic Computing, Wiley-IEEE Press, 2010.

[58] K. Siorpaes, E. Simperl, Human intelligence inthe process of semantic content creation, WORLD

WIDE WEB-INTERNET AND WEB INFORMATION

SYSTEMS 13 (1-2, SI) (2010) 33–59.[59] K. Thorisson, N. Spivack, J. Wissner, The semantic

web: From representation to realization, in: N. Nguyen,

R. Kowalczyk (eds.), Transactions on ComputationalCollective Intelligence II, vol. 6450 of Lecture Notes in

Computer Science, Springer Berlin / Heidelberg, 2010,

pp. 90–107, 10.1007/978-3-642-17155-0-5.URL

http://dx.doi.org/10.1007/978-3-642-17155-0-5

[60] S. Tramp, N. Heino, S. Auer, P. Frischmuth, Rdfauthor:

Employing rdfa for collaborative knowledge engineering,

in: P. Cimiano, H. Pinto (eds.), Knowledge Engineeringand Management by the Masses, vol. 6317 of

Lecture Notes in Computer Science, Springer Berlin /

Heidelberg, 2010, pp. 90–104, 10.1007/978-3-642-16438-57.

URL

http://dx.doi.org/10.1007/978-3-642-16438-57

[61] D. Tunkelang, Faceted Search (Synthesis Lectures on

Information Concepts, Retrieval, and Services), Morgan

and Claypool Publishers, 2009.URL http://www.amazon.com/exec/obidos/redirect?

tag=citeulike07-20&path=ASIN/1598299999

[62] V. Uren, P. Cimiano, J. Iria, S. Handschuh, M. Vargas-

Vera, E. Motta, F. Ciravegna, Semantic annotation for

knowledge management: Requirements and a survey ofthe state of the art, Web Semantics: Science, Services

and Agents on the World Wide Web 4 (1) (2006) 14 –

28.URL http://www.sciencedirect.com/science/

article/pii/S1570826805000338

[63] O. Valkeapaeae, O. Alm, E. Hyvoenen, An adaptableframework for ontology-based content creation on

the semantic web, JOURNAL OF UNIVERSALCOMPUTER SCIENCE 13 (12) (2007) 1835–1853, 28thAnnual Meeting of the Society-of-Behavioral-Medicine,

Washington, DC, MAR 21-24, 2007.[64] B. Yu, Cognitive aspects of human-gis interaction :

A literature review cognitive aspects of human-gis

interaction : A literature review, Interface (2006) 1–17.URL http://test.scripts.psu.edu/users/b/x/

bxy126/courses/files/IST501_TermPaper_BoYu.pdf

24

Qu

ali

ty A

ttri

bu

te

UI

Fea

ture

U

ser

Typ

e

Au

thori

ng

Ap

pro

ach

A

va

ila

ble

tool(

s)

Ex

per

t N

on

-ex

per

t T

op

-Dow

n

Bott

om

-Up

Usa

bilit

y

Sin

gle

Poin

t of

En

try I

nte

rfa

ce

On

toW

iki,

Hay

Stac

k se

man

tic

blo

ggin

g

Fa

cete

d B

row

sin

g

On

toW

iki,

SA

HA

Fa

cete

d V

iew

ing

R

efle

ct, L

oo

mp

Inli

ne

Ed

itin

g a

nd

Vie

w E

dit

ing

O

nto

Wik

i, R

DFa

uth

or

Cu

sto

miz

ab

ilit

y

Liv

ing

UIs

-

Pro

vid

ing

Diff

eren

t S

ema

nti

c V

iew

s

On

toW

iki,

Ow

iki,

Tab

ula

tor,

In

form

atio

n

Wo

rkb

ench

Ge

ne

raliza

bilit

y

Su

pp

ort

ing

Mu

ltip

le O

nto

log

ies

O

nto

Wik

i, Sw

eetW

iki,

FLER

SA, E

pip

han

y, L

inke

dB

log

Su

pp

ort

ing

On

tolo

gy M

odification

O

nto

Wik

i

Su

pp

ort

ing

Het

erog

eneo

us

Docu

men

t/C

on

ten

t F

orm

ats

-

Co

lla

bo

rati

on

Acc

ess

Con

trol

On

toW

iki,

RD

Fau

tho

r, L

inke

dB

log

Su

pp

ort

of

Sta

nd

ard

Form

ats

Sw

eetW

iki,

Lin

ked

Blo

g, R

DFa

uth

or

UIs

for

Soci

al

Coll

ab

ora

tion

O

nto

Wik

i, Ta

bu

lato

r, S

AH

A

Po

rta

bilit

y

Cro

ss-b

row

ser

Com

pa

tib

ilit

y

FLER

SA

UIs

for

Mob

ile

Dev

ices

O

nto

Wik

i

Ac

ce

ssib

ilit

y

Acc

essi

ble

UIs

-

Pro

ac

tivit

y

Res

ou

rce

Su

gg

esti

on

Lo

om

p,

Swee

tWik

i

Rea

l-ti

me

Sem

an

tic

Ta

gg

ing

R

efle

ct

Con

cep

t R

euse

R

efle

ct, S

wee

tWik

i

Rea

l-ti

me

Va

lid

ati

on

O

wik

i, SA

HA

Au

tom

ati

on

A

uto

ma

tic

An

nota

tion

Li

nka

tor,

On

tos-

Feed

er,

Epip

han

y

Evo

lva

bilit

y

Res

ou

rce

Con

sist

ency

O

nto

Wik

i, SA

HA

Docu

men

t a

nd

An

nota

tion

Con

sist

ency

Lo

om

p

Ver

sion

ing

an

d C

ha

ng

e T

rack

ing

O

nto

Wik

i

Inte

rop

era

bilit

y

Su

pp

ort

of

Sta

nd

ard

Form

ats

Sw

eetW

iki,

Lin

ked

Blo

g, R

DFa

uth

or

Sem

an

tic

Syn

dic

ati

on

O

nto

Wik

i

Sc

ala

bilit

y

Su

pp

ort

of

Ca

chin

g

Sem

anti

c M

edia

Wik

i , O

nto

Wik

i

Su

ita

ble

Sto

rag

e S

tra

teg

ies

On

toW

iki,

Loo

mp

, FL

ERSA

Fig. 13. Overview of the results.

25