Aigle Ovacs Tao Icl2012 v 1 0

From informal knowledge to High Quality e-Assessment process

The OAT Semantic Approach

15th International Conference on Interactive Collaborative Learning and 41st International Conference on Engineering Pedagogy Villach, Austria 26-28 September 2012

Younes Djaghloul, Muriel Foulonneau, Raynald Jadoul

The vision

11/26/12 OVACS-AIGLE-TAO 2

Origins of the experiment

Originally -  The generation of assessment items from semantic models -  The generation of ontologies from medical guidelines, courses,… Collaboration -  TAO implementation in dentistry -  Interest in presenting the research at a conference on medical IT

Opportunity = testing our tools and validating our approach in the educational field

-  Interest in innovative approach to learning -  Interest in creating variability in item banks -  Interest in innovative approach to assessment (e.g. for PISA)


Overview on approach


Knowledge: Informal models

Knowledge: Formal models

1. Model building: data mining, human methodology

Formal but not validated

2. Model validation •  Experts for model validation •  OVACS: to assist experts and to hide the complexity of the formalism (OWL, description logic )

Final Ontology

Experts, repositories, social media

3. Question generation •  AIGLE tool, Automatic QTI-based questions generation •  Semantic similarity techniques

List of assessment questions

Validate questions Experts for question validation

4. Delivery strategy •  TAO Delivery Module •  TAO QTI viewer

The Final test

The process


Capturing information from an expert (in our case, a teacher) à Creation of a domain ontology

Validating the ontology à Evaluation by the experts using OVACS

Generating assessment items à Evaluation of the assessment generation

approach (AIGLE) with a teacher

Delivering the assessment items à Delivery of the assessment test using

the TAO platform

The experiment

Stakes: -  Validation of the tools that we will create or reuse

-  Research questions

-  What makes a domain model an educational domain model? -  How to optimize the interaction between the model

and the user (teacher in this case) through -  formulation and -  number of questions?


Why we believe there is a future

Real problems in testing and learning -  Item variability and item banking -  Learning adaptivity

Real problems in the process from modelling to feedback and data collection -  We see, for instance, some partners create forms generated

from UML models but that process is not optimized -  There is a potential to apply our approach of ontology

validation to data collection


The vision on the development of semantic tools


Market Artefact:

Science

Solution Artefact

Design Theory

Problem based

Technological Knowledge

Tools to interact with semantic models

Semantic modelling And user interactions

Adaptivity of services

Quality issue of automated processes

OVACS

Ontology VAlidation for Common uSers How to validate formal knowledge

model by questions


OVACS architecture


Source Ontology OWL

OVACS Engine (Semantic web technologies)

Generated Question (Web based)

Ontology management

feedback

Evaluated ontology

Validated ontology

Expert feedbacks

• Manage history • Get past questions

OVACS interface


Defining questions in 3 sub-models


Actions on the ontology


Questions and ontologies instantiation recovery realization classification

RDFS Classes

rdfs:Resource Y Y N Y

rdfs:Class Y Y N Y

rdfs:Literal Y Y N Y

rdfs:Datatype Y Y N Y

rdf:XMLLiteral Y Y N Y

rdf:Property Y Y N Y


Relating patterns to ��question strategies

Instantiation Recovering Realization Classification

Closed questions (i.e., with a textual

context) Y Y Y Y

Graphic questions N N N N Questions with a pre-determined

choice Y Y Y Y

Questions for which a level must be

indicated N N N N

Open questions Y Y Y Y Ordering N N Y N

Setting relations Y Y Y Y Creating (in this case, Modeling) Y Y Y Y

Forms of questions and ��questioning strategies ��for the x propertyOf y ��pattern (i.e., range and ��domain properties)

Multiple questions to verify inter-related sub-graphs Instantiation : is x an instance of y? Recovering : which are all the instances of y? Realization test: among all the objects that x is a type of, which one is the most specific? Satisfaction test: if we say that x is an instance of y, does it create any inconsistency in the

model? Classification test: which types of objects does x belong to?

Implementation of validation strategies

11/26/12 Cc-eSanté 15

-

E.g., based on ontology patterns

Optimization of the validation strategy


Pattern P2: Class – Property – Instance

Pattern P1: Class – Subclass – Instance

Next steps

1.  Improve the linguistic capabilities of the tool 2.  Implement alternative question sequence optimization strategies 3.  Implement a knowledge base on question forms 4.  Implement a feedback mechanism 5.  Test the caries application with Karolinska 6.  Test the SANTEC application

11/26/12 Cc-eSanté 17

AIGLE Assessment Item Generator

in Learning Environment


AIGLE – Assessment item generator

q  Security issue Adding variability to an item no expected variation of the construct

q  Model-based learning Generating items from knowledge represented as a model the construct is modified for each item


Stem variables

options

key

Auxiliary information

IMS-QTI item generation process

Generating items from Web data sources


Test with DBpedia data Q Question model Ontology What was tested Outcome

Q1

What is the capital of {Azerbaijan} Infobox

The lowest level of semantic quality with the InfoBox dataset

3/30 Q/CA pairs not generated for a country 1 Q/CA generated for a country that does not exist any longer 1 capital as URI rather than a literal 1 capital with a labelling issue

Q2

Which country is represented by this flag?

Infobox FOAF YAGO

Improved set of properties Retrieval of pictures from Wikimedia

6 out of 30 items did not include a flag picture because the flag URI had a 404 error

Q3

Who succeeded to { Charles VII the Victorious } as ruler of France ?

Infobox Dbpedia ontology YAGO

The use of the Dbpedia ontology the historical domain

Inconsistent labels duplicates due to multiple labels assigned to each king 1 incorrect statement

Q4

What is the capital of { Argentina }? With feedback

Infobox YAGO

Feedback on items Retrieval of resources through data linkage

Full Flickr dataset unavailable at the time of the query

Q5

Which category does { Asthma } belong to?

SKOS Dublin Core MESH Infobox

Use of SKOS terminology the medical domain

SKOS concept not related to a SKOS scheme Usability of the SKOS structure of limited use as a domain model for education


From Q1, 53,33% were directly usable neither a defective prompt nor a defective correct answer nor a defective distractor .

Benchmark from unstructured content: between 3.5% and 21%. Issues

•  Ontology issue •  Labels •  Inaccurate statements •  Data linkage (resolvable URIs) •  Missing inferences

Measuring data quality


Chance that an item will have a defective distractor =

User testing with countries and their capital


Calculating the semantic similarity between distractors and a correct answer

Ulan Bator Libreville Manila


Gabon – Libreville

Maputo Port Louis Libreville

No SemSim With SemSim

Adapted 3 semantic similarity strategies �� to large scale semantic graphs

Results of user test Clear decrease of performance in the population

when using SemSim


Common vs. uncommon knowledge: annotating the semantic model for educational purpose

Item Construct Estimate T1 Estimate T2

IC1 0.072 -0.288

IC2 -0.298 0.337

IC3 1.961 0.560

IC4 1.755 0.186

IC5 0.084 0.305

IC6 -0.270 -0.349

IC7 -0.737 -1.076

IC8 -1.517 0.751

IC9 -0.284 0.077

IC10 -0.767 -0.502


Item difficulty estimate according to IRT

But role of ��the item layer?

Predicting item difficulty? Representation of the semantic distance of options in the items

(item dispersity)


Next steps: 1.  Refining the semantic similarity metrics 2.  Assessing different forms of questions 3.  Building an item bank with generated questions 4.  Test the value of the questions (with Karolinska)


TAO Testing Assisté par Ordinateur

(Computer-Aided Testing)


TAO – assessment and feedback loop TAO platform is based on semantic web paradigm, i.e. it manages

question items decorated with any needed ad-hoc properties

The TAO platform delivers questionnaires that can also be featured with any desiderated extra semantic properties

The TAO collects all answers and behaviors of the test-takers

à  If extra properties like the “provenance” (i.e. the source model built with OVACS and used by AIGLE) are attached to the question items or to the questionnaire, these properties are stored in tests results

à  Analysis of the tests results will be used in a feedback loop for a validation process impacting the AIGLE & OVACS phases.


OVACS AIGLE TAO

Initial test with Karolinska


Original hypothesis

The creation of the domain ontology can use semi-automatic strategies, or third party encoders, or a collaborative work: can we ask an expert to validate the assertions in the ontology?

-  What is lost in the expert’s speech when creating the ontology? -  Does the expert understand automatically generated questions? -  Does the expert flag the errors?


Creating the ontology

An ontology of the caries -  A one hour interview where the teacher

explained - the caries, their description, their causes, - how to handle them, - how to prevent them, - how to set a diagnostic

-  Definition of a list of concepts / keywords -  Creation of classes, instances, and properties -  Creation of the OWL ontology


Test set up Labels on stand alone Selected a subset of the ontology to keep the test short:

à instanceOf (13 items) and à subClassOf (11 items)

Only Boolean questions (yes/no) + “I do not know” option

24 questions 2 intentional mistakes:

on the content (causes of caries) and spelling (emanel instead of enamel)

Objectives: -  check if the teacher would find the validation mechanism usable -  verify whether errors would be detected and corrected

Video recording of the teacher


Test conclusions

Confusion between the role of domain expert validating knowledge

and the role of teacher who prepares questions for students

According the comments of our expert: “Difficulty level of the generated questions is generally low” “But with very different variations in the difficulty level”

The OVACS validation questionnaire led to: 6 removals (2 subClassOf, 4 instanceOf) 16 acceptances (9 for subclassOf, 7 for instanceOf) 2 answers “I do not know” for subclassOf à meant not relevant


OVACS-AIGLE-TAO

[email protected]

Aigle Ovacs Tao Icl2012 v 1 0

Documents

Transcript of Aigle Ovacs Tao Icl2012 v 1 0