02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 1 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

TCDL BulletinCurrent 2006Volume 2 Issue 2

Actual Use of Learning Objects and Metadata

An Empirical Analysis

Jehad Najjar and Erik DuvalComputer Science Department, K.U.Leuven

B-3001 Leuven, Belgium{Jehad.Najjar, Erik.Duval}@cs.kuleuven.ac.be

Abstract

Learning Object technology is being considered as the natural form of content re-use that willconsiderably decrease the cost of developing electronic learning (educational) material. The focus of myresearch is evaluating and tracking the actual use made of learning objects, metadata and their associatedtoolsets in Learning Object Repositories. This will enable users and tools to learn from the way users uselearning objects and technology in general, and hereby, improve the use of learning objects and theirassociated toolsets.

1. Introduction

In this research, we are carrying out empirical studies to evaluate the actual usage of learning objectsand metadata in Learning Object Repositories (LORs). We have statistically analyzed the actual usemade of metadata elements when indexing and searching learning objects in the ARIADNE knowledgePool System (KPS). Moreover, we conducted usability studies to evaluate tools and functionalities usedto index or find learning objects.

In order to learn from the way people actually use new technologies for learning, we have beendeveloping a framework [13] that helps us to track the behavior of users and learners. This frameworktracks and publishes attention given to learning objects and notifies the user about objects in which he orshe might be interested.

In the coming sections, we give a very brief description of our pieces of work, in addition to someselected results (see the references part of this paper for the full versions of the work). Section 2discusses the use of metadata in the indexing and search processes. Section 3 studies the usability of

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 2 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

learning object indexing and search tools. Section 4 presents a new framework proposed to track, publishand share the behavior of learning object users. Section 5 summarizes the research issues discussed inthis paper.

2. Actual Use of Metadata

This section studies the actual use made of metadata elements in Learning Object Repositories (LORs).Those elements form the application profile of LORs and are provided to facilitate the finding oflearning objects. This section is based on two publications. Sub-section 2.1 is based on the Actual Use ofMetadata in ARIADNE: An Empirical Analysis [9], published at the ARIADNE 2003 conference. Sub-section 2.2 is based on User Behavior in Learning Object Repositories: An Empirical Analysis [10],published at the ED-MEDIA 2004 World Conference on Educational Multimedia, Hypermedia andTelecommunications.

Results and findings of such studies provide us with empirical guidelines to assess the development andevaluation of application profiles and metadata toolsets.

2.1 Use of Metadata in Learning Object Indexing Process

In this study, we present a statistical analysis of the actual use of ARIADNE metadata elements inindexing learning objects. The results are derived from analyzing the empirical data (usage logs) of 3,700ARIADNE metadata instances (the number available when we started the analysis).

Table 1 shows the percentage of times each ARIADNE data element was filled in by indexers during theindexing process.

Table 1. Percentage of usage made of data elements by ARIADNE indexers

From the data shown in Table 1, we notice that only one data element is almost always used: theGranularity element. Other elements are used in about 50 % of the descriptions and the rest are rarelyused in the indexing process.

For the values of data elements, we can see that indexers often use just one value. However, this shows

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 3 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

that indexers are different in the way they choose data elements to describe their learning objects as wellas vocabulary values assigned to each data element. Moreover, by looking to the metadata informationfilled in by each indexer, we noticed that indexers often use mental templates of elements and valuesevery time they index new learning objects.

Predicting relationships between data elements is not an easy job. The relationships between the studieddata elements will form the guidelines for successful automatic indexing or application profilesdevelopment in general. In Table 2, the high correlation between Interactivity Level and SemanticDensity proves that choosing an Interactivity Level means a high probability for Semantic Density to befill-in and vice versa. Moreover, if the value of Semantic Density is "high" then Interactivity Level willmost probably be "high" too.

Based on the correlations within the elements, we may automatically fill-in or suggest values of otherco-related elements. We may also hide some elements from the user based on a correlation with otherelements.

Table 2. Measures of association's strength between data element

These kinds of studies will allow us to enhance the use of ARIADNE metadata and provide guidelinesfor developing and evaluating application profiles and their associated tools such as automatic indexing.

2.2 Use of Metadata in Learning Object Search Process

In this study, we investigate the ways in which users interact with Learning Objects Repositories (LORs)when searching for relevant learning objects. We present a statistical analysis of ARIADNE query logfiles of readily available data on 4,723 queries launched by approximately 390 users in six ARIADNELKPs [Genoa, Galati, Grenoble-UJF, Lausanne-EPFL, Lausanne-UNIL and Leuven-CS] over differenttime periods.

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 4 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

Figure 1: Frequency of elements used in searchers' queries

Figure 1 shows the frequency that the different ARIADNE elements have been used insearchers' queries. Analysis for the frequency of elements used in searchers' queriesreveals that searchers mostly accept the provided default data elements. The most used 20data elements by searchers are default data elements. Remarkably, the query tool allowssearchers to change the default settings for the query tool and show the whole list ofelements. Few searchers change the default element settings provided.

These results can be interpreted in two ways. First, the default settings are the mostrelated elements to ARIADNE users. Second, searchers have a tendency to accept defaultsettings. Giving the precise reason for these results requires further investigation to testthe above two mentioned hypothesis.

A comparison between the actual usage of data elements in both indexing (see section2.1) and search processes, reveals that data elements that have been used by more than50% of the indexers are not used by the majority of searchers, such as granularity,didactical context and semantic density elements. In addition, for values of such dataelements, we noticed that both indexers and searchers mostly select same values to indexor search learning objects.

Table 3: Frequency of elements used in searchers queries

No. elements used in Queries 0 1 2 3 4 >=5 Total

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 5 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

Frequency 548 2488 701 498 258 230 4723

Percent 11.6 52.7 14.8 10.5 5.5 4.9 100.0

Table 3 shows that searchers are more interested in forming queries that contain relativelyfew metadata elements. The majority of queries (75%) contain one to three data elements.Less than 5% of the queries contain five or more data elements. The mean of the numberof elements in queries is 1.7 elements and the SD is 1.6. About 12% of queries containedno metadata elements at all. In fact, this is related to some usability problems with thequery tool; some searchers directly launch queries without selecting any data element.Also, they might select their appropriate data element, but without specifying theappropriate string or mathematical operator such as: starts with, contains, Ends with, =, >,<, etc., which is required by the system to successfully issue a query in the ARIADNEquery tool; this problem has been removed in the last version of the tool.

3. Usability Evaluation of Learning Object Tools

The work presented in this section discusses the usability of learning object indexing andsearch tools. The goal of such studies is to determine the influence of usabilityperspectives on the performance of users, and also to uncover the type of complicationsthat may face users of such tools. In addition, provide recommendations to improve theusability of those tools. The research presented in this section was published in twopapers. Sub-section 3.1 is based on Usability Evaluation of Learning Object Indexing: theARIADNE Experience [11], published at the European Conference on E-Learning, 2004.Sub-section 3.1 is based on Finding Appropriate Learning Objects: An EmpiricalEvaluation [14], published at the European Conference on Research and AdvancedTechnology for Digital Libraries, 2005. Findings and recommendations of these studiesare generalized for other related tools.

3.1 Usability of Learning Object Indexing Tools

This study investigates usability problems of indexing tools for Learning Objects. Thecomplexity of manually indexing learning objects results in a bottleneck for theintroduction of such objects.

A usability test was performed on the two ARIADNE indexing clients: SILO [6] andToledo [7]. We collected detailed data from extended sessions with seven users.Participants were selected to be representative of an intended user community, includinguniversity professors, research assistants, and university students.

The goal of the usability test was to cover a wide range of abilities and sophistications,not to constitute a statistically balanced sample, because the aim of this study is todetermine whether such clients help target users to achieve their goals and discoverproblems that may influence users' effectiveness, efficiency and satisfaction.

Results of this study show that indexing performance and semantics of metadata(provided through indexing tools) can be influenced by different usability perspectives(see Table 4):

Interface of indexing tools.Functionalities provided to facilitate the indexing process, such as automatic

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 6 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

indexing.Indexer domain knowledge about the introduced learning objects.

Table 4: Metadata usage and semantics

We believe that the main problems behind the complexity of the indexing of learningobjects are:

User interface of indexing tools in most of LORs are more adapted to the metadatastandards and not to the indexer. This is related to a misunderstanding aboutimplementing interoperability with metadata standards. For example, in someLORs, terminologies and organization of metadata elements are copied as is fromdocuments of metadata standards, while terminologies (labels of metadata elementsand vocabulary values) and information organization should be adapted to the localcommunity of the LOR.Functionalities provided to facilitate the extraction of metadata are not as intelligentas they should be. More advanced techniques should be used to improve theindexing process, for example, algorithms that generate automatic values based onempirical analysis of actual usage, text or image recognition techniques orontologies. In order to save the quality of metadata semantics, we may allow usersto change automatically provided information whenever they feel it is necessary.

A full version of the list of findings and recommendations related to this study isprovided in [11].

3.2 Usability of Learning Object Search Tools

In this study [14] we investigate usability problems of search tools for learning objects.We present findings and recommendations of an iterative usability study conducted toexamine the usability of a search tool used to find learning objects in ARIADNEKnowledge Pool System (KPS). Findings and recommendations of this study aregeneralized to other similar search tools.

We aim to evaluate and improve the usability of a search tool (SILO) used by ARIADNEusers to search learning objects in the KPS. We want empirically to answer the following

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 7 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

questions:

How do users use the search tool?How effectively and efficiently does the search tool help users perform their searchtasks?What are the factors that may increase the performance of finding learning objects?Are users satisfied with the overall use of the tool?

In order to collect primary data on the usability of SILO and double check and validateour findings we conducted two iterative usability phases:

First phase:We collected primary data from extended sessions with 16 participants to determinethe initial usability of the tool.Second Phase:Here, we evaluated the tool after solving the usability problems and integrating therecommendations that appeared in the first phase. In this second phase, wecollected data from extended sessions with 10 new participants who had no priorexperience with the tool.

Results of this study shows that finding appropriate learning objects is still not an easytask. The usability of the search interface may noticeably decrease the performance ofusers searching for relevant material. The use of terminology and structure of informationin the old SILO was more adapted to the metadata standard than to user needs. Thatpractice of metadata use can be noticed in the search interface of other existingrepositories such as Merlot and SMETE.

Figure 2: Comparison between responses of participants who evaluated SILO in thetwo phases

Figure 2 illustrates a comparison between the overall use of SILO before and afteridentifying the usability problems. Five participants (group 2) who participated in phasetwo were asked to use the old interface of SILO. Moreover, we asked another fiveparticipants (group 1) who participated in the first phase to evaluate SILO interface of thesecond phase (SILO 2). We did that to revalidate the recommendations and usabilityproblems obtained from the first evaluation phase and to draw some comparisonsbetween the two interfaces. Based on participants' feedback, we found that SILO 2 wasmuch easier to use and much less overwhelming than the old SILO interface for both ofthe groups.

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 8 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

As shown in Figure 2, in both groups 1 and 2 the overall usability of SILO 2 is higherthan SILO. In addition, the level of user satisfaction on all the studied factors (ease ofuse, use of terminology, information organization, etc.) with SILO 2 is higher in bothgroups.

4. Actual Use of Learning Objects

In order to create a feedback loop that enables learning from the way people actually usenew technologies and tools for learning, it is essential to track the behavior of users andlearners [3]. In this section, we propose and discuss a framework (see Figure 3) that weare developing for automatic collection and management of attention metadata. Theresearch presented in this section was published in Attention Metadata Management:Tracking the use of Learning Objects through Attention.XML [13], which appeared in theproceedings of the ED-MEDIA 2005 World Conference on Educational Multimedia,Hypermedia and Telecommunications.

The framework shown in Figure 3 enables keeping track of learning objects that peopleuse, how they use them, the time they spend with them, etc. This framework tracks userinteractions with the different tools they use and then publishes that data in astandardized form of attention metadata to enable the use and exchange of these data.

The release of the new attention.xml specification [1] inspired us to examine thepossibilities for using that specification to track and publish attention given to learningobjects, and moreover, to notify the user about objects in which he or she might beinterested.

Figure 3: The AMM framework that tracks and publishes attention of users

As shown in Figure 3, users may interact with a wide variety of tools that make use oflearning objects:

Learning Object Repositories (LORs): Users may interact with a LOR (such asMERLOT, EdNa, ARIADNE and SMETE) to introduce or search relevant learningobjects.Learning Management systems (LMSs): Teachers may interact with an LMS (such

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 9 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

as Blackboard, Moodle, and WebCT) to manage objects of their courses. Studentsalso can use an LMS to access those objects.Internet Browsers: Users may download their relevant learning objects from theappropriate LOR and then open and read it (learn it) in another application like webbrowsers.Messaging Systems: Users may also chat about a learning object using a messagingsystem.Email Clients: Users may send or receive learning objects or information aboutthem by email messages or RSS feeds.Audio and Video Players: Users may use an audio or video player to learn.Other tools, such as MSWord and Powerpoint.

Our framework logs users' activities while interacting with the different tools, thenpublishes the collected attention metadata (user behavior) related to each tool in aseparate attention.XML file or stream. Afterwards, the set of those attention.xml sourcesare merged into one attention.XML source.

Figure 4: A small part of the technical representation of our framework

To see a larger version of Figure 4, click here

In Figure 4, we represent our scope of the technical representation framework. Thedescription of the whole framework incorporating all the components is still underinvestigation. Therefore, in this paper we intend to demonstrate an example of loggingactivities of people using the ARIADNE SILO search tool [6], and their activities whenopening those objects (for example, in a Mozilla Firefox browser) to learn them. Then,we merge the collected information and publish it in an attention.XML source.

For the SILO search tool, we can log information, like the title, subject, file name, or Idof learning objects, downloaded by the different users. We can use the slogger logging

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 10 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

extension provided with the Firefox browser to collect objective information aboutlearning objects the user accessed, such as file name, URL, host, title and access date-time. In addition, slogger enables us to collect information like the description andkeywords provided by the learner himself after reading the learning object. Moreover, wecan use the StumbleUpon extension of Firefox to track information about the relevance ofa learning object to the user: the user can click on "I-Like it" when the object is relevantfor the user and "Not-for me" for objects that not found to be relevant.

After we track the information, our framework will generate (this is under development)an attention.XML file for user interaction with the different tools and then merge theproduced sources in one attention.XML source that includes user attention given to thedifferent objects received, read or listened to.

The merging of different attention sources presents various implementation challenges,because learning objects might be processed by different tools. That requires us to beable to do the following:

Manage the attention.XML source so that it contains the updated information(collected from different attention.XML sources of each tool), for example, of readtimes, last time read, time spent on and followed links attributes of each object.Identify the same learning object and its attention data. We are researchingapproaches to tackle this issue.

There are different uses for the resulting attention.XML data:

Based on the subject, publisher, learning duration, etc. of objects with which a userinteracts, we can build a recommender system that suggests new objects. Inaddition, we can track other information that can be used by the recommendersystem, like the relevance of the object to the user, using features as "I Like it"provided by Firefox StumbleUpon.Attention metadata can also provide feedback to tools about user behavior. Forexample, we can to update a user metadata profile in a search tool, based on theattention metadata of that user.The same attention metadata also enable empirical analysis of the actual use oflearning objects and user interaction models [10].

For future work, we want to start implementing the different components of the proposedframework and build tools that use the produced attention files to enhance learners'productivity and efficiency.

5 Summary

This paper introduced the research activities that we are carrying out to evaluate theactual use of learning objects and metadata in Learning Object Repositories (LORs). Thisresearch is based on analyzing real-life data of users. User logs were analyzed todetermine the real use of metadata in both indexing and search activities. Extendedusability sessions were conducted to determine to what extent indexing and search toolsenable users to reach their goals effectively and efficiently. Moreover, a framework thatis intended to track, publish and share behavior of learners was introduced.

The research presented in this paper has been carried out at the Hypermedia and

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 11 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

Databases research group at the Computer Science Department of Katholieke UniversiteitLeuven, Belgium. More information on the research conducted in this research group isavailable at <http://ariadne.cs.kuleuven.ac.be/hmdb/jsp/Wiki?Research>.

References

[1] Attention.XML, (2004), Attention.XML specifications.<http://developers.technorati.com/wiki/attentionxml>.

[2] Blackboard. Available at <http://www.blackboard.com/>.

[3] Duval, E., Hodgins, W., (2003). A LOM Research Agenda, Proceedings of the 12thint. conf. on World Wide Web (Hencsey, G. and White, B. and Chen, Y. and Kovacs, L.and Lawrence, S., eds.), pp. 1-9, 2003.

[4] MERLOT, Multimedia Educational Resource for Learning and Online Teaching,Available at <http://www.merlot.org/>.

[5] Moodle. Available at: <http://moodle.org/>.

[6] Neven, F., Duval, E., Ternier, S., Cardinaels, K., Vandepitte, P., (2003). An Open andFlexible Indexation and Query tool for ARIADNE. Proceedings of World Conference onEducational Multimedia, Hypermedia and Telecommunications, 2003, (Lassner, D. andMcNaught, C., eds.), pp. 107-114, 2003.

[7] Toledo, <http://toledo.kuleuven.be/>.

[8] Jehad Najjar, Erik Duval, Stefaan Ternier and Filip Neven, Towards InteroperableLearning Object Repositories: the ARIADNE Experience, Proceedings of the IADISInternational Conference WWW/Internet 2003, pp. 219-226, 2003.

[9] Jehad Najjar, Stefaan Ternier and Erik Duval, The Actual Use of Metadata inARIADNE: An Empirical Analysis, Proceedings of the 3rd Annual Ariadne Conference,pp. 1-6, 2003.

[10] Jehad Najjar, Stefaan Ternier and Erik Duval, User Behavior in Learning ObjectRepositories: An Empirical Analysis, Proceedings of ED-MEDIA 2004 World Conferenceon Educational Multimedia, Hypermedia and Telecommunications, pp. 4373-4378, 2004.

[11] Jehad Najjar, Joris Klerkx, Stefaan Ternier, Katrien Verbert, Michael Meire and ErikDuval, Usability Evaluation of Learning Object Indexation: The ARIADNE Experience,Proceedings of ECEL 2004 European Conference on e-Learning, pp. 281-290, 2004.

[12] Jehad Najjar, Stefaan Ternier and Erik Duval, Interoperability of Learning ObjectRepositories: Complications and Guidelines, IADIS International Journal ofWWW/Internet (ISSN: 1645-7641) 2004.

[13] Jehad Najjar, Michael Meire and Erik Duval, Attention Metadata Management:Tracking the Use of Learning Objects through Attention.XML, Proceedings of ED-MEDIA 2005 World Conference on Educational Multimedia, Hypermedia andTelecommunications (Kommers, P. and Richards, G. eds.), pp. 1157-1161, 2005.

02/01/08 19:21Actual Use of Learning Objects and Metadata: An Empirical Analysis

Page 12 of 12http://www.ieee-tcdl.org/Bulletin/v2n2/najjar/najjar.html

[14] Jehad Najjar, Joris Klerkx, Riina Vuorikari and Erik Duval, Finding AppropriateLearning Objects: An Empirical Evaluation, Proceedings of ECDL 2005 9th EuropeanConference on Research and Advanced Technology for Digital Libraries, (Rauber, A., etal. eds. ), pp. 323-335, 2005.

© Copyright 2006 Jehad Najjar and Erik Duval

Top | ContentsPrevious Article

Next Article Home | E-mail the Editor