Journal of Network and Computer Applications 34 (2011) 1200–1209

Contents lists available at ScienceDirect

Journal of Network and Computer Applications

1084-80

doi:10.1

n Corr

E-m

journal homepage: www.elsevier.com/locate/jnca

A model-based framework to automate the analysis of users’ activity incollaborative systems

Rafael Duque n, Crescencio Bravo, Manuel Ortega

Department of Information Systems and Technologies, School of Computer Science Engineering, University of Castilla-La Mancha, Spain

a r t i c l e i n f o

Article history:

Received 9 June 2010

Received in revised form

19 November 2010

Accepted 21 January 2011Available online 4 February 2011

Keywords:

Collaboration analysis

Collaborative systems

45/$ - see front matter & 2011 Elsevier Ltd. A

016/j.jnca.2011.01.005

esponding author.

ail address: Rafael.Duque@uclm.es (R. Duque

a b s t r a c t

The automated analysis of users’ activity in collaborative systems aims at characterizing the collective

process established between the participants and contributing to improving the deficiencies that occur

during the users’ activity and that are identified by means of this characterisation. This type of analysis,

however, requires the completion of complex and costly software development tasks such as capturing

information about the actions carried out by the users, calculating low and high level variables to

characterize the users’ work and behaviour and, finally, defining different kinds of interventions to

improve the users’ experience solving this way all of those problems that have been detected. To enable

the automation of these tasks, reducing the required development effort, we propose a framework that

allows software developers to specify the processes of analysis of the activity supported by a

collaborative system by means of visual languages in a model-based approach and to automate the

analysis accordingly. This framework is used in two case studies. Firstly, the framework is applied for

the collaboration analysis in a social network system. Secondly, the collaborative programming activity

supported by the COLLECE system is analysed using the framework, and the impact of using the

framework is then evaluated.

& 2011 Elsevier Ltd. All rights reserved.

1. Introduction

At present, the knowledge society in which we live has a clearneed for technological systems which can provide support forworking, learning and playing in groups. Recent technologicaladvances are introducing innovative improvements into thedevelopment of collaborative systems to support such activities.These innovations are not solely confined to the design ofcommunication mechanisms, but also include the building ofshared workspaces where users can solve or carry out commontasks collectively (Gonzalez and Mark, 2005) and of coordinationmechanisms that enable organised collaboration (Sire et al.,1999). In recent years, such improvements have resulted in aproliferation of systems to facilitate the communication betweenusers (social networks, instant messaging, wikis, etc.) and thecollaborative work aimed at solving problems in projects that,because of their scale or nature, must be approached in groups(electronic calendars, shared whiteboards, etc.).

At this point, it is worth noting that technology not only makesthese collective activities possible but also opens the doorto analyse how the activities occur (Dimitracopoulou, 2004).The process of collaboration analysis allows one to study the users’

ll rights reserved.

).

actions (Martınez et al., 2006) and to evaluate their impact on thecollective process (Jermann and Dillenbourg, 2008). The attain-ment of this characterisation should be the starting point toproviding perceptible improvements during the course of users’activity by means of assessments of the activity being carried outas well as modifications to the collaborative system’s behaviour.Moreover, the results of such an analysis could be of interest topeople outside the group of participants in the collective activity,who are usually evaluators or analysts interested in obtaining anassessment of specific aspects of the activity carried out by theusers in groups.

There is a wide range of technological systems supportingcollaborative activities that can benefit from a process of analysis.An example are computer-supported collaborative learning(CSCL) environments, which support the collaboration betweenstudents and their interaction with teachers in a structuredteaching-learning process (Bravo et al., 2006). The students,organised in groups, collaborate by exchanging ideas and solvingproblems as a team. The teachers organise and supervise thelearning activities by posing problems, uploading material andanswering questions. Here, a process of analysis would allow thecalculation of variables for characterizing the users’ learningactivity (e.g., students’ participation in the collaborative process,average number of problems considered by a group, qualityof the users’ learning, etc.) can contribute to make an evaluationof the teaching and learning processes. Computer-supported

R. Duque et al. / Journal of Network and Computer Applications 34 (2011) 1200–1209 1201

collaborative work (CSCW) systems are often used in professionalenvironments in which a group of users collaborate to solve acommon project. The processes of analysis are useful in the CSCWsystems to study the productivity and the coordination of thework teams. In other cases, the collaborative systems do notsupport such a close or structured interaction between the users;for example, this is the case of some social media systems. In asocial network system based on the web, the process of analysiswould allow the study of its social impact by means of variablesthat, for example, quantify the number of users, logons to thesystem, friends of a user or more elaborate issues.

Analysing a collaborative process is a complex task becausemultiple aspects must be taken into account, including theinformation that is to be processed, how to define variables toanalyse the group activity and how to use the analysis results toimprove the users’ activity. To enable one to accomplish thesetasks, we must rise to the challenge of building computationalsupport that enables software developers to automate the analy-sis in accordance with the particularities of each social scenario(collaborative system features, nature and type of the activity tobe carried out by the participants, etc.) and the interests of thosepeople involved in the analysis (participants in the collectiveactivity, external analysts, etc.).

When carrying out the collaboration analysis, with the aimof improving the users’ activity, a life cycle consisting of threephases, observation, abstraction and intervention, is usuallyfollowed (Bravo et al., 2008). The observation phase automaticallycollects data about the users’ activity (commonly known as rawdata). The abstraction phase involves the processing of the datacaptured in the observation phase in order to define variables thatcharacterize the users’ activity. The intervention phase takes asinput the analysis variables obtained during the previous phase inorder to make improvements in the users’ activity throughdifferent means including the modification of the system beha-viour or user interface.

To tackle the aforementioned challenge, this article presents atechnological framework that enables developers to produce com-putational support for automating the analysis of a collaborativeactivity according to the typology of the required analysis. Thistechnological framework is based on the theoretical process-solution analysis framework proposed by Bravo et al. (2008) andon the analysis life cycle above. In this context, where the analysismust be configured according to different particularities, the use of atechnological framework is considered particularly useful, sincesuch a framework is a partially complete software system to beextended by means of plug-in technology (Booch et al., 2005) inorder to produce a collaboration analysis system. Specifically, ourtechnological framework is based on the visual definition of modelsthat specify a particular process of collaboration analysis and on theproduction of an analysis system via a series of transformations onthese models and configurations.

This article contains four further sections. Section 2 reviewssome approaches for the analysis of the users’ activity in acollaborative system. Section 3 describes the structure andcomponents of the framework we have designed. The processby means of which the framework is configured by instantiatingmodels is presented and illustrated in Section 4 through two casestudies: social networks and collaborative programming. Finally,Section 5 presents the conclusions drawn from the work we havedone and suggests some future lines of work.

2. Related work

The type of characterisation carried out by collaborationanalysis tools generally varies according to the type of

collaborative system being used, the type of activity carried outby the users and the goals of the analysts. As such, the TagHelpertool (Rose et al., 2008) automates the analysis by focusing oncharacterizing the semantic content of the messages exchangedby the users of a system that supports communication (e.g., chattools, forums, etc.). Daradoumis et al. (2004) propose a set ofmeasurements for the activity supported by a social network thatquantify the relationships that each participant has with theothers and characterize how the activity is concentrated arounda particular participant or a very small number ofparticipants. Komlodi and Lutters (2008) perform an analysis thatfocuses on capturing the activity of each user and use it toimprove knowledge management within the social organisation.The Tatiana tool (Dyke et al., 2009) enables the execution of non-automated analysis where the collective activity is monitored viavideo and the analysts use a utility to attach notes or categorisa-tions about the users’ activities. We can observe that each of thesecollaboration analysis tools and some others (Baghaei et al., 2007;Bratitsis and Dimitracopoulou, 2006) in the literature areembedded into a specific collaborative system because they havebeen created to analyse the collective processes supported by aspecific type of such system. Therefore, they do not provide aflexible computational support for adapting the analysis to thecharacteristics of different collaborative systems, which wouldhelp developers to produce an analysis subsystem.

In recent years, there has been a proliferation of general-purpose analysis tools. These tools usually process statisticallythe log files that record the users’ actions when using theoperating systems and web servers (Google, 2009; WebAlizer,2009; WebTrends, 2009) in order to calculate variables that cancharacterize the users’ activity (number of logons, time spentusing the system, number of operations by category, etc.). How-ever, these tools lack automatic and methodological support fordefining the measurements and using them to make decisions.A paradigmatic case is that of e-business systems that integraterecommender systems that use the variables of the analysis tosuggest products or services (Wang et al., 2007).

In order to deal with the lack of analysis systems that canbe adapted to each collaborative system and allow the analysisprocedures to be configured, this paper presents a flexible model-based framework that enable developers to specify the analysisto be automated according to each collaborative system and typeof measurements required. The most important contribution ofthis framework is its potential for creating a ready-to-use colla-boration analysis system, following some guidelines and well-defined methodological procedures and by instantiating a numberof computational models. The framework can this way be appliedto different systems, ranging from social networks and wikis tocollaborative learning environments and collaborative work sys-tems. The framework allows for the measurement of differencesaspects, such as the frequency of communication messages, thetypes of actions carried out, the degree of agreement betweenparticipants, etc., and facilitates intervention to improve theusers’ experience with the collaborative system.

3. A model-based framework to automate the collaborationanalysis

3.1. Functional architecture

In designing the aforementioned technological framework, webegan by specifying the actors that interact with the technologicalframework and the basic information flows. The features of theframework are configured by means of a set of visual models. Forthis, the framework includes a set of authoring tools that enable

R. Duque et al. / Journal of Network and Computer Applications 34 (2011) 1200–12091202

software developers to design the models. The actors that interactwith the technological framework are:

�

Users of the collaborative system: their interaction with thecollaborative system is observed in order to create repositoriescontaining data about the actions they carry out. The data inthese repositories will subsequently be analysed. Then, theframework will show the analysis variables or change thesystem’s behaviour. � Analysts: they are interested in evaluating the collaborative

activity. As such, they define the collaborative activity to beanalysed (participants, groups, work units, etc.) using a webuser interface provided by the framework.

� Developers: they implement the required process of analysis

from the specification of computational models. Usually, thedevelopers require the help of analysts and experts in the typeof collaborative process and nature of the tasks supportedunder study. They interact with the framework by means ofauthoring tools that allow them to specify the process ofanalysis and by means of transformation tools to create thecomputational support that automates the analysis modelled.

Fig. 1 shows how the functional architecture of the frameworkis made up of a set of levels. Each level is made up of subsystemsgiving well-defined functionality. The information generated atone level of the architecture is processed by another level in orderto carry out its functions. In this way, a flow of informationbetween levels is established. The meta-information level is usedto manage all of the information necessary to automate theanalysis. Other levels are used to actually perform the processof analysis (analysis level) and to intervene in the activity of thecollaborative system users (collaboration level). The levels andsubsystems included in the framework’s architecture are thefollowing:

�

Meta-information level: at this level, the processes of analysisare configured according to the characteristics of the colla-borative scenario and the goals of the analysis. The subsystemsincluded at this level manage the definition and storage ofanalysis sessions specifying the collective processes that are tobe analysed. Its subsystems are as follows:J Design subsystem: this subsystem is used by the developers

to model the analysis. The subsystem includes a set ofauthoring tools that are used by software developers tospecify the processes of analysis through the visual designof models. The implementation of these authoring toolswas carried out using the support provided by Eclipse, thepopular integrated development environment. Eclipseincludes a framework, called the Eclipse Modelling Frame-work (EMF), which includes features for the implementa-tion of authoring tools that allow models to be instantiatedvisually. The authoring tools include a toolbar with the

Fig. 1. Levels and subsystems

building blocks the corresponding visual language that thedevelopers use to build the models that specify the processof analysis. These tools represent the designed models inthe XMI standard (XMI, 2010) for exchanging meta-infor-mation using XML so that these models can be processedautomatically.

J Analysis session manager subsystem: this subsystem is usedby the analysts to define the specific collective process tobe analysed (participants, activities, collaborative system,etc.). The subsystem provides a web interface and usesMySQL as relational database management system to storethe information of the collaborative process.

of th

�

Analysis level: the subsystems included at this level enable thesoftware developers to provide computational support thatimplements the analysis specified at the meta-informationlevel. These subsystems are:J Analysis subsystem: the developers interact with this sub-

system to automatically put into action the processes ofanalysis configured at the meta-information level. Theanalysis subsystem uses the models built by means of thedesign subsystem. For this, the analysis subsystem usesthe ATL language (ATL, 2010) that enables the processingof these models and their transformation into Java code.Then, this new Java code is used to generate automaticallya set of applications that carry out the modelled analysis.The applications are located on a central server that com-municates with the collaborative system by means ofTCP/IP ports.

J Communication subsystem: this includes the services forcommunicating the results of the process of analysis tothe collaboration level (see Fig. 1). This subsystem uses thesocket services provided by the Java language to commu-nicate the interventions to the TCP/IP ports of the colla-borative system.

�

Collaboration level: the subsystems included at this levelinteract with the collaborative system to capture the actionscarried out by the users and to make changes in its behaviourto improve the collaborative activities according to the inter-ventions of the analysis. This level contains two subsystems:J Monitoring subsystem: this allows for the automation of

those actions of the intervention phase that mainly requiresome information (values of the analysis variables, advicemessages, etc.) to be displayed. The technology used tobuild these visualisations is JavaFX, which allows userinterfaces to be specified more easily. This subsystemdisplays information that enables an improvement of thecollaborative process whose target is the user of thecollaborative system.

J Communication subsystem: this provides a set of services tocommunicate the actions carried out by the users to theanalysis level. The Java socket services are also used by thissubsystem.

e framework.

Fig. 2. Authoring tool for modelling the quantitative abstraction.

Fig. 3. Authoring tool for modelling the qualitative abstraction.

R. Duque et al. / Journal of Network and Computer Applications 34 (2011) 1200–1209 1203

When using the framework, the developers firstly specify theprocess of analysis which is to be automated by means ofauthoring tools integrated into the design subsystem. Secondly,they use the functionalities of the framework to provide compu-tational support that automates the specified analysis to beintegrated within a collaborative system. The following subsec-tions present the authoring tools and some implementationissues. Examples of models built with the authoring tools andtheir user interface are shown in Section 4.1 (see Figs. 2–4).

3.2. Authoring tool for modelling the quantitative abstraction

This authoring tool supports the visual representation of a setof components for defining how to calculate quantitative vari-ables that characterize the users’ activity. Unlike traditionalanalysis systems, which include a predefined set of quantitativevariables, this tool aims at giving developers the ability to specifyquantitative variables, calculated according to the needs of eachcase. These quantitative variables can be classified as either low-level variables or high-level variables. Low-level variables arecalculated by counting a subset of data stored in the actionsrepositories. High-level variables are calculated using mathema-tical operations.

The tool allows one to specify low-level variables whose valuescan show (i) the number of occurrences of specific types ofactions carried out by the users, (ii) the time spent using aparticular tool integrated into the system, or (iii) the time spentperforming a particular type of action. Therefore, the developersmust specify the characteristics of the actions that are to be

counted. These actions can be specified using some of thefollowing attributes:

�

Name: the name of the action to be processed (e.g., sending amessage, uploading a document, creating a concept or entity,etc.). � User: the user who performs the action. � Role: the role of the user who carries out the action (e.g., an

administrator of the collaborative system, a specific kind ofuser who participates in the collective process, etc.).

� Tool: the tool that supports the action (e.g., chat, wiki,

electronic whiteboard, etc.).

The authoring tool also allows one to specify high-levelvariables which are calculated by applying a set of basic arith-metic and statistical functions (addition, subtraction, multiplica-tion, division, percentage, average, standard deviation and so on).These functions take variables previously calculated as inputparameters and their resulting output is assigned to a newhigh-level variable.

3.3. Authoring tool for modelling the qualitative abstraction

The authoring tool for modelling the qualitative abstraction

allows the developers to design models that specify a numberof inference rules to characterize the users’ activity by means ofqualitative variables. This authoring tool enables the inferenceof variables to take place in accordance with the purposes ofeach process of analysis. Each inference rule is made up of an

Fig. 4. Authoring tool for modelling the intervention.

R. Duque et al. / Journal of Network and Computer Applications 34 (2011) 1200–12091204

antecedent and a consequent. The antecedent is made up ofquantitative and/or qualitative variables. When the antecedentvariables acquire certain values, the rule is activated. To specifythe values that the antecedent variables must take in order for therule to be activated, the authoring tool allows partitions to beestablished within the numerical domains of the quantitativevariables and the qualitative values of the qualitative variables tobe specified. Lastly, the consequent of each rule is defined by thename of the new qualitative variable and the value that it takes asa result of the activation of the rule.

3.4. Authoring tool for modelling the intervention

This authoring tool makes it possible to build interventionmodels so that the system can improve the users’ activity indifferent ways, in accordance with the previously calculatedquantitative and qualitative variables. The intervention phase isthe third phase in the analysis life cycle. This tool allows one tocomputationally define conditions for the values of some vari-ables (i.e., detecting when they are out of range) and othercircumstances (e.g., the type of actions carried out by the users,the time spent carrying out some task, etc.). In this way,deficiencies in the users’ activity can be identified and interven-tion mechanisms that can mean a contribution towards solvingthe detected problems can be triggered.

First of all, the authoring tool for modelling the intervention

allows developers to specify when the values of the analysisvariables should be checked. These moments can be defined asan elapsed period of time from the start of the activity or they canalso refer to the occurrence of certain types of action. For thispurpose, the tool allows the previously calculated variables to bedelimited using both a minimum and a maximum value. If thecurrent value of the variable falls outside of that range, it would benecessary to activate an intervention mechanism. The tool allowsfor the grouping of a set of variables using AND connectors and OR

connectors. In the first case, all of the grouped variables must beoutside of their ranges in order for the intervention mechanism tobe triggered. In the second case, a single variable outside of itsrange is enough to trigger the intervention mechanism.

Finally, it is necessary to specify how the intervention mechan-isms will intervene in the users’ activity. Each intervention isdefined through some dimensions: place, specifying the particulartool or workspace of the collaborative system where the

intervention is to take place; text, specifying the information tobe shown to the users in the intervention if necessary; form,describing which actions or behaviours should be carried out andwhether or not a representation of the variables that havetriggered the intervention should be displayed; and users, whoare the target of the intervention action.

3.5. Implementation issues

The purpose of the communication subsystem (see Fig. 1) is toorchestrate a process that coordinates the activities supported bythe collaborative system and the analysis subsystem. The need forthis coordination arises from the fact that each action carried outin the collaborative system is a new input for the process ofanalysis. Moreover, the results of the analysis in the form ofintervention actions have a direct influence on the activitysupported by the collaborative system. Therefore, both processes,collaborative system in execution and analysis, are interrelatedand influenced each other.

The communication subsystem includes an interface with a setof methods that are invoked by the analysis subsystem and thecollaborative system. The methods invoked by the collaborativesystem are those aimed at executing analysis tasks and sendinginformation of the actions carried out by the users. The analysis

subsystem invokes methods for sending descriptive information ofthe interventions to be carried out.

The operation of the communication subsystem is made up ofthree steps. First, the user accesses the collaborative system andstarts a working session; this situation is automatically commu-nicated to the analysis subsystem by means of the correspondingmethod. In the second step, each action carried out in thecollaborative system is communicated to the analysis subsystem

with descriptive data (name of the action, time when it wascarried out, etc.). The receipt of this information by the analysis

subsystem could involve only a confirmation that the informationhas been analysed or a request of intervention in the collaborativeprocess. The third step starts when the user closes the collabora-tive system which is communicated to the analysis subsystem tostop the process of analysis.

Thus, the framework can be used by any collaborative system(web-based social network system, wiki, CSCL system, etc.). Inorder that the framework can be integrated into a collaborativesystem, the developers must include in the collaborative system

R. Duque et al. / Journal of Network and Computer Applications 34 (2011) 1200–1209 1205

functions that capture and communicate the actions carried outby users, and functions that carry out those interventions thatcannot be automated by the monitoring subsystem. These tasksconsist in:

�

Capturing actions: the developers must implement functionsthat invoke the methods of the communication subsystem forsending information about the actions carried out by the users.This is necessary because the source code of the system shouldbe modified to detect the actions. � Implementing interventions: as mentioned, the framework

includes a monitoring subsystem whose main functionality atintervention level consists in displaying information (values ofanalysis variables, advice messages, etc.). However, when theinterventions involve other kind of actions (e.g., adapting themenus to the preferences of a participant, applying newcollaboration policies, etc.) the developers must implementspecific functionalities within the collaborative system. In thiscase, the communication subsystem provides the informationrequired for providing the intervention.

4. Application of the framework

After developing the framework, different case studies werecarried out to evaluate its effectiveness in automating the analysisof collective activities (Duque, 2010). We present here two casestudies. The first one is focused on putting the framework’sauthoring tools to the test of specifying the analysis of the activitysupported by a social network system. The second one is aimed atstudying the application of the framework in the automation ofthe analysis of collaborative programming practices.

4.1. The case of social network systems

A social network is a social structure made up of people(or organisations or other entities) who are connected by a varietyof relationships such as friendship, kinship, common interests,economic exchanges, etc. A considerable number of social net-works are web-based (e.g., Facebook, Twitter, etc.) and offer acollection of services that enable interaction between users bymeans of tools such as chat, e-mail, group discussion, blogs, etc.Usually, each participant has a directory of participants withwhich he or she has friendship relationships. Each participant hasthe ability to exchange messages, documents and files with theirfriends. In addition, each user can send invitations to other socialnetwork members in order to expand their circle of friends. Thedevelopers must implement functionalities that communicate theactions performed by the users (invitations proposed, invitationsaccepted, questions asked, etc.) via the communication subsystem.

Fig. 2 illustrates the use of the tool for modelling the quantita-

tive abstraction to specify how to calculate some variables of theanalysis to quantify the activity between the users of a computer-supported social network. The low-level variables considered are:(i) NDeletes, which quantifies the number of users who have beendeleted from the social network by the users playing the role ofadministrator; (ii) NActions, which quantifies the number ofactions (add a new friend, make a proposal to a friend, etc.)performed by each participant in the social network; (iii) NTime,which quantifies the number of seconds spent by each participantin the use of the system; (iv) NInvitations, which quantifies thenumber of times users have invited another participant to jointheir circle of friends in a period of time; (v) NAcceptances, whichquantifies the number of requests to join a circle of friends thathave been accepted; (vi) NMessages, which quantifies the numberof messages sent through the social network that propose a new

topic to be discussed; and (vii) RMessages, which quantifies thenumber of messages sent through the social network to replyprevious messages. Finally, the figure illustrates how to calculatethree high-level variables: (i) PAcceptance, which calculates whatpercentage of requests to establish a friendship relationship areaccepted; (ii) NConversations, which quantifies the total number ofmessages exchanged between users; and (iii) Actions_Density,which quantifies the number of actions performed by the partici-pants per unit of time.

These quantitative variables (see Fig. 2) enable an analysis ofvarious aspects of the users’ activity. As such, the analysts canstudy whether the social network supports a high activity(examining the Actions_Density variable), the flow of communica-tion between users (examining the NConversations variable), orthe predisposition for establishing new friendship relationships(examining the PAcceptance variable).

Fig. 3 shows a fragment of a qualitative abstraction model thatdefines how to infer two qualitative variables which analyse theactivity that has taken place in a social network: Communication

and Activity. This abstraction uses the quantitative variablescalculated in the previous example (see Fig. 2). The qualitativemodel defines a rule R1 for inferring the Communication qualita-tive variable with a high value. In order for rule R1 to be activated,the NConversations quantitative variable must take a value withinthe numerical range 80–100 and the PAcceptance quantitativevariable must take a value within the range 50–100. Secondly,the model defines the rule R2 for inferring the Activity qualita-tive variable. The rule R2 assigns a high value to the Activity

variable. In order for rule R2 to be activated, the Communication

variable must take a high value and the Actions_Density quantita-tive variable must take a value contained within the numericalrange 15–25.

Fig. 4 shows a fragment of the intervention model for ourexample social network. The model includes a cycle time of1800 s to determine whether or not the Communication andActivity variables are within their ranges. These two variablesare linked by an AND connector, so that the intervention shouldbe performed only when both of them are outside their ranges.The intervention consists in displaying the value of the variablesthat have triggered the intervention together with the message‘‘You can add new friends. This webpage includes a set of possible

new friends’’. The monitoring subsystem displays both the messageand the variables to the users playing the role of participants inthe social network.

We could have considered the use of other interventions,however, such as adding new tools to facilitate the exchange ofmessages between participants when the Communication variabledoes not achieve a high level, or the modification of the userinterface to make it more attractive and to encourage its use incases of low level of Activity. The interventions that involvemodifying the social network functionalities (modifying the userinterface, changing the communication tools, etc.) must beimplemented by the developers, since the monitoring subsystem

cannot perform them.

4.2. The case of collaborative programming

Collaborative programming involves groups of programmers,wherein each group member has a defined role. While one ofthe programmers plays the role of driver and implements theprogram source code, the other group members play the role ofobservers and are responsible for reviewing the source code andsuggesting improvements. During the course of programming,these roles are exchanged by the mutual agreement of the groupmembers. The groupware system used in this case study was

R. Duque et al. / Journal of Network and Computer Applications 34 (2011) 1200–12091206

COLLECE (Bravo et al., 2007), which supports the implementationof programs through the distributed synchronous collaborativework of programmers. This system features the following set oftools:

�

Shared editor (Fig. 5a): this is the shared workspace where thedriver inserts or modifies the source code of the program.A tele-pointer highlights at each moment the line of the sourcecode where the driver is working so that the observers canfollow their progress in real time. � Coordination Panels: with the aim of providing organised

coordination between the driver and the observers, COLLECEoffers a set of utilities that are used to establish the workingprotocol. Each of these panels allows any participant in thecollaboration to make a request to the group. This request isautomatically sent to the partners, who can either accept it orrefuse it, thereby preventing it from being carried out. In orderthat a request can be carried out, all the partners must acceptit. The coordination panels are as follows:J Editing panel (Fig. 5b): this is used by an observer to

assume the driver role, allowing him/her to take controland modify the source code.

J Compilation panel (Fig. 5c): when one of the programmersconsiders that the source code must be compiled, he/sheuses this panel to send the request to their partners andthen uses the console to carry out the compilation task.

J Execution panel (Fig. 5d): when one of the programmerswants to start a testing process to verify the validity of theprogram, he/she uses this panel to propose the execution ofthe program.

�

Structured chat (Fig. 5e): the process of collaborative program-ming requires constant communication and a constant flow ofinformation between the driver and the observers. To facilitatethis process, COLLECE contains a structured chat tool. This chattool contains buttons for making use of the conversational

Fig. 5. COLLECE u

actions that are most commonly used during the constructionof programs (for example, ‘‘In the loopy’’ or ‘‘I see a mistakeiny’’). In this way, when the programmer selects one of theseconversational actions, he/she then simply completes the restof the sentence.

In the next subsections, a study aimed at evaluating theanalysis framework in the collaborative programming domain isdescribed.

4.2.1. Evaluation method

We study here the benefits that the framework’s supportprovides to developers by studying its application to the COLLECEsystem. Accordingly, our objectives are the following:

�

ser

To study the feasibility of the framework for automatingprocesses for analysing the activity of collaborative program-ming with COLLECE.

� To compare the impact of integrating processes of analysis via

the framework with that of developing a specific analysissystem for the COLLECE system.

To address these objectives we followed a specific evaluationmethod. In the studies we considered different populations asfollows: (i) a software developer with an MSc in ComputerScience and Engineering, who used the framework in order toautomate the required analysis; (ii) 32 students of the Methodol-

ogy and Technology of Programming subject at the School ofComputer Science Engineering of the University of Castilla-LaMancha, who played the role of COLLECE users; and (iii) twoanalysts, who were teachers in the Computer EngineeringMaster’s programme and so were interested in the results of theanalysis. The 32 students were randomly arranged in pairs. Theyhad to implement collaboratively a program in Java with a

interface.

R. Duque et al. / Journal of Network and Computer Applications 34 (2011) 1200–1209 1207

recursive method that calculated all the integer numbers betweentwo values introduced by the program user. This program is verybasic but the study was carried out in an academic environ-ment where the participants were programming students whohave limited experience in programming, so the interventionmechanisms of the analysis are particularly useful for helpingstudents.

The case study of collaborative programming supported byCOLLECE was chosen since a previous version of this systemfeatured a built-in analysis system without making use of theframework, i.e., the analysis system was coded from scratchwithout any support for automating the analysis required. Thismade it possible to make a comparative study of the impact ofusing the framework for automating processes of analysis againstthe traditional approach involving the creation of a specificanalysis system for each collaborative system.

4.2.2. The framework in action

The work of the developer started with the specification of aquantitative abstraction model. This model specifies the calcula-tion of variables that show the number of communicationmessages carried out using the chat tool, the number of requestsfor editor’s floor control, compilation and execution in thecoordination panels, and the amount of time spent solving theproblem. In addition, a second set of variables quantify, as apercentage, the degree of agreement between the users incoordinating the participation in each programming task (edition,compilation, execution). Further variables are also calculated inorder to evaluate the speed of the users in the different workareas (editor, console, chat, etc.).

The next step for the developer was to specify the qualitativeabstraction model, as previously described. The purpose of thevariables considered in this model was as follows:

�

Speed: this seeks to evaluate the number of actions per unitof time. � Edition work: this rates the average actions to edit the source

code with regard to all of the work actions performed bythe group.

� Compilation work: this rates the average actions to compile

the program with regard to all of the actions performed bythe group.

� Execution work: this rates the average actions to run the

program with regard to all of the actions performed bythe group.

� Effort: this rates the number of actions performed by

the group.

Fig. 6. Intervention performed b

�

y th

Communication: this measures the number of messagesexchanged between users per unit of time.

� Preparation: this rates the level of discussion of the problem

statement and to establish communication between the part-ners in order to plan the work prior to starting work.

� Coordination: this rates the overall level of agreement

between the members of the working group in arrangingactivities through the coordination panels.

� Collaboration: this rates both the level of communication and

the level of coordination between the members of theworking group.

Last of all, the developer defined an intervention model toimprove the activity of the COLLECE’s users. Summing up, we candistinguish three different types of intervention:

�

Regulating the activity of the users when the Preparation

variable has a low value and detects that the actions neededto plan how the problem is to be solved do not take place.These interventions include the displaying of a text messagethat suggests to users how they might correct these problems;the intervention action attaches a graphical representationof the value taken by the Preparation variable that hastriggered the intervention.

� Correcting those situations in which a task (edition, compila-

tion or execution) lacks a sufficient workload. The insufficientworkloads of the tasks are detected by means of the valuesthat have the Edition_Work, Compilation_Work, Execution_Work

and Effort variables. To correct this situation, the interventionmechanisms assign the role of driver to the observer pro-grammer to carry out the task that has a work deficit. Thisintervention is not automated by the monitoring subsystem sothat the developer implemented a functionality in COLLECEconsisting of selecting as a driver the observer programmer.Figure 6 shows a message displayed by the monitoring sub-

system in a situation of very low numbers of execution thatjustifies the selection of a new driver.

� Displaying tips or advice messages to correct those problems

arising between users due to a poor use of the communicationmechanisms offered by the structured chat tool and/or thecoordination mechanisms supported by the correspondingpanels. The user interface in Fig. 6 is also used for thesevisualisations.

After defining the analysis models, the developer proceeded toautomate the specified analysis. This task consisted in creatingfunctionalities to send information about the actions carried outby the users via the methods of the communication subsystem. In

e monitoring subsystem.

R. Duque et al. / Journal of Network and Computer Applications 34 (2011) 1200–12091208

addition, the developer implemented functions for carrying outthe interventions that involve assigning the control of a task tothe observer programmer corresponding to the second type ofintervention above.

After finishing the collaborative programming experiment con-sisting of solving the proposed program, the programmers answereda questionnaire giving this way their opinions on the process ofanalysis. The programmers answered each question with a five-point Likert scale ranging from 1 (worst score) to 5 (best score).Table 1 contains a summary of the questions and the mean (M) andstandard deviation (SD) values of the results of the questionnaire. Ineach case, the programmers (N¼32) gave a mean of at least 3.66 tothe different aspects of the process of analysis.

4.2.3. Comparison with a hand-made specific-purpose analysis

system

Unlike previous studies (Bravo et al., 2008; Daradoumis et al.,2006) that are aimed at assessing only the results of the auto-mated analysis (values of the calculated variables, interventionsperformed, etc.) from a social view point, this study has the goalof assessing the cost of using this framework in the developmentsoftware process in contrast to traditional approaches where thesystem of analysis is created from scratch without using aframework. Bravo et al. (2008) built a system for analysing theactivity supported by the COLLECE system. This system wasdeveloped specifically for the COLLECE system and neither theframework proposed here nor any other mechanisms that allowfor the automation of analysis tasks were used.

Table 2 compares the impact of both implementation approaches.First of all, regarding learning issues, we can see that the use of theframework requires an effort because the developer is not onlyrequired to carry out learning work to study the characteristicsof the COLLECE system and the type of analysis to be automated buthe/she also need to study the characteristics of the framework(how to instantiate the models using authoring tools, the function-alities of the subsystems, etc.). However, the framework carries outautomatically the process that calculates variables to characterizethe users’ work. Therefore, the developer does not have to learntechniques that calculate these variables (e.g., arithmetic operations,statistical methods, etc.).

However, this table also shows how using the frameworksignificantly reduces the developers’ effort in terms of imple-menting the analysis source code, since the interventions thatinvolve the display of an advice are automated by the monitoring

Table 1Programmers’ opinions.

Question M SD

Assessment of the advice messages 3.66 0.90

Quality of the analysis variables 3.87 0.78

Suitability of the interventions that select a new driver 4.13 0.95

Global evaluation of the analysis 3.96 0.72

Table 2Comparison between an analysis system built using the framework and an analysis sy

Analysis system built using the framework

Learning COLLECE, type of analysis, framework

Cost Design models using visual languages, 112 source code lin

Maintainability Modify the models

Modularity Physical and functional independence

Reusability Applicable to other collaborative systems

subsystem and the developers do not have to implement specificfunctionalities. The developers only have to implement featuresto capture the actions performed by the COLLECE’s users and toperform the intervention that involves selecting a new driverprogrammer.

Furthermore, the framework facilitates the maintenance taskssince the developers only have to update the models in order toadapt the analysis to new analysis situations (e.g., to specify newanalysis variables) and to implement the new interventions thatare not automated by the monitoring subsystem, whereas adaptingthe analysis system that was developed specifically for COLLECEwould involve making modifications to 60% of the source code ofthe analysis system.

It is also worth noting that the framework is a separate moduleof COLLECE from both a functional and a physical point of view,meaning that the framework communicates with COLLECE tocapture actions generated by the users and to intervene in theusers’ activity. In contrast, the system created specifically forCOLLECE is a component that is embedded in the system. Thismeans that this analysis system cannot be reused in othersystems for automating analysis.

The framework proposed in this paper, however, is an indepen-dent software system whose features are configured using a model-driven approach to fulfil the analysis requirements of each case andwhich can be applied in different collaborative systems. Thus,Duque, (2010) has used the framework to automate processes ofanalysis not only in the programming domain using COLLECE butalso in the Co-Lab (van Joolingen et al., 2005), SPACE-DESIGN(Gallardo et al., 2008) and DomoSim-TPC (Redondo et al., 2002)systems for, respectively, the domains of System Dynamics, DiagramBuilding and House Automation.

The works studied in Section 2 include generic proposals(Google, 2009; WebAlizer, 2009; WebTrends, 2009) that can alsobe integrated to analyse the basic aspects of the use of collaborativesystems. However, the features of these proposals do not include theability to customise the process of analysis by defining the variablesto be calculated to analyse the collaborative activity in a structuredapproach. Nor do these proposals automate an intervention phasewhere the results of the analysis are used to improve the collabora-tion between the users. Therefore, the proposed framework can beconsidered as a means for reducing the cost of developing analysissystems and for extending the features of these proposals toautomate the collaboration analysis.

5. Conclusions

The framework we have presented here allows softwaredevelopers to automate the analysis of users’ activity withincollaborative systems. The main aim of the framework is toreduce the effort required by software developers when imple-menting a software subsystem, to be integrated into a collabora-tive system, for the automation of the collaboration analysis task.The framework follows a model-driven approach in which visuallanguages are used to specify the analysis and several

stem built from scratch specifically for COLLECE.

Analysis system created from scratch

COLLECE, type of analysis, techniques to calculate analysis variables

es Many more source code lines

Re-implement 60% of the source code

Embedded into the COLLECE system

Not applicable to other collaborative systems

R. Duque et al. / Journal of Network and Computer Applications 34 (2011) 1200–1209 1209

transformation tools generate the analysis subsystem from themodels. Moreover, the framework includes functionalities thatenable not only the registration of actions and the calculation ofvariables to characterize the users’ activities but also the inter-vention in the users’ activity in order to improve it in some way.

In order to show that the proposed framework is a suitabletool for specifying and automating the processes of analysis, weworked with two case studies. In the first case study, the frame-work is applied to specify a process that analyses the collectiveactivity in a social network. This case study shows that theframework provides a novel approach in which the softwaredevelopers use visual languages for specifying the analysis. Thesevisual languages allow them not only to specify variables thatcharacterize the activity of the social network but also to definemechanisms that, for instance, enable the participants to find newfriends and to improve the communication.

In the second case study, the framework was used to automatethe analysis of collaborative programming practices supported bythe COLLECE system. The COLLECE’s users gave a positive assess-ment of the analysis automated. Moreover, the impact of usingthe framework to automate the process of analysis was comparedwith the building of a specific analysis system created fromscratch with no automation support. The analysis to be carriedout using the framework was specified by means of models andthe developer’s effort to create code was thereby reduced. Thedeveloper only had to implement functionalities to communicatethe actions performed by the users and to automate thoseinterventions that constituted a change to the access policies ofthe shared workspaces, since the framework only automatesinterventions that consist of visualisations (i.e., displaying advice,showing the value of a variable, etc.). Therefore, we can state thatthe proposed framework allows analysis in collaborative systemsto be automated and that the cost of integrating this analysis isless when using the framework than when developing a specificanalysis system.

The model-driven approach followed by the framework alsofacilitates software maintenance tasks. Should the analysis sys-tem be needed to be adapted to new requirements, developershave only to update the analysis models and generate the system.In contrast, the software developers have to recode an importantpart of the analysis system when this is created from scratch. Inaddition, an analysis system created from scratch and embeddedin a collaborative system cannot be easily reused in a differentsetting. To this concern, the analysis framework is a softwaremodule separated from the collaborative system that can bereused in other systems. For this purpose, the framework includesa communication subsystem that offers an interface for anycollaborative system interested in using the framework.

Throughout this project, we have seen that collaborativesystems are powerful tools that allow groups of users to collabo-rate in the completion of common tasks. However, the develop-ment of mechanisms that can evaluate the use of these systemsand characterize the type of activities carried out by the users canbe a costly task. The framework described here makes theautomation of the analysis easier and also introduces additionaldynamism into the collaborative activity using interventions thatrecommend actions to the users or change the functionality of thecollaborative system.

In the future, we will further reduce the effort required by thedevelopers in order to automate these processes of analysis.To this end, machine learning techniques will be introduced toautomatically infer analysis variables and generate inferencerules. These techniques will reduce the work of developersbecause they will not be required to specify a large numberof rules through the authoring tool for modelling qualitative

abstraction.

Acknowledgment

This research has been partially supported by the Junta deComunidades de (Regional Government of) Castilla-La Mancha(Spain), Consejerıa de Educacion y Ciencia, under Projects PAC07-0020-5702 and PCI08-0069-7887.

References

ATL, /http://www.eclipse.org/m2m/atl/S; 2010.Baghaei N, Mitrovic A, Irwin W. Supporting collaborative learning and problem

solving in a constraint-based CSCL environment for UML class diagrams.Computer-Supported Collaborative Learning 2007;2:159–90.

Booch G, Rumbaugh J, Jacobson I. Unified modeling language user guide. 2ndedition Addison-Wesley Professional; 2005.

Bratitsis T, Dimitracopoulou A. Monitoring and Analyzing Group Interactions inAsynchronous Discussions with the DIAS System. Groupware: Design, Imple-mentation, and Use; 2006. p. 54–61.

Bravo C, Duque R, Gallardo J, Garcıa J, Garcıa PA. Groupware system for distributedcollaborative programming: usability issues and lessons learned. In: VanHillegersberg FH J, Amrit C, Geisberger E, Keil P, Kuhrmann M, editors. Theinternational workshop on tools support and requirements management forglobally distributed software development; 2007. p. 50–6.

Bravo C, Redondo MA, Ortega M, Verdejo MF. Collaborative distributed environ-ments for learning design tasks by means of modelling and simulation. Journalof Network and Computer Applications 2006;29:321–42.

Bravo C, Redondo MA, Verdejo MF, Ortega M. A framework for process-solutionanalysis in collaborative learning environments. International Journal ofHuman–Computer Studies 2008;66:812–32.

Daradoumis T, Martinez A, Xhafa F. An Integrated approach for analysing andassessing the performance of virtual learning groups. In: Proceedings of the10th international workshop on groupware. Springer; 2004. p. 289–304.

Daradoumis T, Martinez Mones A, Xhafa F. A layered framework for evaluating on-line collaborative learning interactions. International Journal of Human–Computer Studies 2006;64:622–35.

Dimitracopoulou A. State of the art on interaction and collaboration analysis.Kaleidoscope; 2004.

Duque R. Un framework para analisis automatizado proceso-producto de lacolaboracion en sistemas groupware. University of Castilla-La Mancha; 2010.

Dyke G, Lund K, Girardot J-J. Using the analysis tool Tatiana to visualize andcalculate interaction indicators. In: Proceedings of the workshop interactionanalysis and visualization: analysis’ methods, tools and research questions.Rhodes, Greece; 2009.

Gallardo J, Bravo C, Redondo MA. Developing collaborative modeling systemsfollowing a model-driven engineering approach. In: Proceedings of the OTMconfederated international workshops and posters on the move to meaningfulinternet systems: 2008 workshops: ADI, AWeSoMe, COMBEK, EI2N, IWSSA,MONET, OnToContent+QSI, ORM, PerSys, RDDS, SEMELS, and SWWS. Mon-terrey, Mexico: Springer-Verlag; 2008.

Gonzalez V, Mark G. Managing currents of work: multi-tasking among multiplecollaborations. Paris, France: Springer-Verlag New York, Inc.; 2005.

Google, /http://www.google.com/intl/es_ALL/analytics/S; 2009.Jermann P, Dillenbourg P. Group mirrors to support interaction regulation in

collaborative problem solving. Computers & Education 2008;51:279–96.Komlodi A, Lutters W. Collaborative use of individual search histories. Interacting

with Computers 2008;20:184–98.Martınez A, Yannis D, Gomez-Sanchez E, Rubia-Avi B, Jorrın-Abellan I, Marcos JA.

Studying participation networks in collaboration using mixed methods.International Journal of Computer-Supported Collaborative Learning 2006;1:383–408.

Redondo MA, Bravo C, Ortega M, Verdejo F. PlanEdit: an adaptive problem solvingtool for design. In: Proceedings of the second international conference onadaptive hypermedia and adaptive web-based systems. Springer-Verlag; 2002.

Rose C, Wang Y-C, Cui Y, Arguello J, Stegmann K, Weinberger A, et al. Analyzingcollaborative learning processes automatically: exploiting the advances ofcomputational linguistics in computer-supported collaborative learning. Inter-national Journal of Computer-Supported Collaborative Learning 2008;3:237–71.

Sire S, Chatty S, Gaspard-Boulinc H, Colin F-R. How can groupware preserve ourcoordination skills? Designing for direct collaboration. In: Sasse A, Johnson C,editors. Conference on human–computer interaction. Edinburgh, Scotland;1999. p. 304–12.

van Joolingen WR, de Jong T, Lazonder AW, Savelsbergh ER, Manlove S. Co-Lab:research and development of an on-line learning environment for collabora-tive scientific discovery learning. Computers in Human Behaviour 2005;21:671–88.

Wang F-Y, Carley KM, Zeng D, Mao W. Social computing: from social informatics tosocial intelligence. IEEE Intelligent Systems 2007;22:79–83.

WebAlizer, /http://www.mrunix.net/webalizer/S; 2009.WebTrends, /http://www.webtrends.com/S; 2009.XMI, /http://www.omg.org/spec/XMI/S; 2010.