Lecture 7: Further Student Modelling

AILE 2004/5 Mar-23-05

Lecture 7: Further Student Modelling 1

Mar-23-05 Advanced Interactive Learning Environments 2004/5 1

Lecture 7: Further StudentModelling:


Contents1. Analytic Methods

e.g. ArtCheck2. Other Machine Learning examples3. Current trends

AILE 2004/5 Mar-23-05



1. Analytic Models


Analytic ModelsAnalyse student’s input and try to account

for the structure

+ represents misconceptions+ not restricted to pre-defined errors+ often use of machine learning techniques

- doesn’t explain cause of error or bug- needs significant samples of data

AILE 2004/5 Mar-23-05



e.g. SpellingWork back from the misspelling to the correction

k w i c quick

r i d red, rid, ride, write, rite, readf a t fate

Letter as its name? i for i_e a for a_e


e.g. Phoncode -> Phonological InformationBased on phoneme-grapheme grammar: consider whatphonemes error was intended to represent, then see if any wordmatches

‘neiz’

|

‘n’ ‘ei’ ‘z’

| | |

/n/ (nO) /i/ (kEEp) /z/ (Zoo)

/ng/ (siNG) /IE/ (EAr) /zh/ (pleaSure)

/m/ (jaM) /aI/ (hIGH) /s/ (Sit)

| /I/ (bIt) /sh/ (buSH)

| /eE/ (AIr) |

| /eI/ (dAY) |

| | |

knees /n/ /i/ /z/

neice /n/ /i/ /s/

AILE 2004/5 Mar-23-05



WEST (Brown and Burton, 1979)Development of mathematical skills

through a game:

- student’s move compared to system’s- if differ, moves are analysed- identifies issues student is weak on- coaches on weakness, in context of move


Wenger, fig 7.1 “How the West was won”(Burton and Brown, 1979)

Insert figure

AILE 2004/5 Mar-23-05



Excerpt from a dialogue with WEST(Burton and Brown, 1976)

Bob is at 54 -- the COMPUTER is at 40.West: It’s MY turn … Spinners are 2 4 2

My expression is: 2 + (4/2) which gives ME 4I took a SHORTCUTI bumped YOUthe COMPUTER is at 54 -- Bob is at 40

West: It’s YOUR turn … Spinners are: 1 2 2What arithmetic expression did you form?

Student: 1 + 2 * 2West: What number does that give you?Student: 5West: You don’t seem to be bumping very much. Bumps are hard to get but

they are usually a good idea. One good example would be the expression:(1 * 2) + 2, which would give you a SHORTCUT and a BUMP!! So youcould have been at 54 while I would have ended up at 40. Bob is at 45 --the COMPUTER is at 54.


WESTIssues:Mathematical skills: use of addition, division,

parenthesesGame-specific skills:- use of shortcuts and landing on your opponent- development of strategy such as always aiming

for towns if possible- knowing that order of numbers in expression does

not have to be same as on the spinners;General game skills:- learning from your opponent- exploring the space of possible strategies

permitted by the game.

AILE 2004/5 Mar-23-05



Recognising and Evaluating IssuesFor each issue, there is a RECOGNISER- checks to see whether issue features in the

student's move,- whether it is necessary in that expression- whether it is necessary in the optimal move

For each issue there an EVALUATOR- determines whether student weak in issue, by

failing to use it to good effect.- blame shared equally among all issues that might

be implicated in failure to find best move.


Dealing with different strategiesWhat if the user employs different strategy?

WEST watches for what is called a ‘tear’ = the appearance of a fairly high number of issues

not used when they should have been, and othersused when they should not have been.

WEST considers whether another strategy by theexpert would give a better fit, (according toeach of pre-wired strategies)

AILE 2004/5 Mar-23-05



2. Analytic Methodse.g. ArtCheck


Detecting and Analysing ErrorsUsing Articles: - how do you know whether you are talking

about a specific object, or any old object?- how do we choose the correct article to

indicate the indefinite or definite propertyof the noun in an utterance?

Detecting Errors:- how do we detect errors when some uses

the incorrect article?- how could we generate an appropriate

explanation to help them learn to do thiscorrectly?

AILE 2004/5 Mar-23-05



Artcheck (Sentence, 1993)Indefinite– a/an, eg John is a teacher– zero, eg Do you take milk in coffee?

Definite– the, eg He is the only teacher in the school

Some native languages do not include an articlecategory, e.g. Finnish - definiteness andindefiniteness expressed in quite different ways

For such EFL learners, the correct use of Englisharticles can be a major problem.


e.g in Finnish….Otan kirjatI take the books

Otan kirjojaI take some books

Huoneessa on pöytäThere is a table in the room

Pöytä on huoneessaThe table is in the room

Similar problems in Basque and Russian

AILE 2004/5 Mar-23-05



Examples of errors

*I have visited Tower of LondonI have visited the Tower of London

* Aeroplane has revolutionised travelThe aeroplane has revolutionised travel

*We discussed our plans for the day overthe breakfast

We discussed our plans for the day overbreakfast


ArtCheck ITS:Aims to help such non-native speakers of

English use articles appropriately.

Domain knowledge: rules which determinecorrect article usage

Applying knowledge user input, in order todetect any article usage errors

Having detected errors, rules can be used asthe basis for generating explanations,customised to the individual learner.

AILE 2004/5 Mar-23-05



Determining correct article usageRules indicate whether article before noun should be:1. the definite article the2. the indefinite article a/an3. no article at all, (the zero article)

Some are fixed rules: the definite article should be usedwhen the noun is modified by a superlative adjective,eg the largest dog

Other depend on context of use: the indefinite articleshould be used to introduce new information.

The sources of information used by the system: the lexicon, the parser, the morphological analyser,

and a discourse history module.


Examples of rulesArticle Usage Rules Example Information

neededThe definite article can beused where the noun ismodified by a relative clause

The man who I saw . ..

Syntactic

The zero article can be usedbefore plural count nouns

Do you like eggs Morphological

The zero article can be usedwith proper nouns

My dog is calledMarcus

Lexical

The indefinite article is usedin some expressions offrequency

I go running twice aday

Idiomatic

AILE 2004/5 Mar-23-05



Easily implemented rules, e.g.Article When used Information Needed

For singular count nouns Number and count/mass nounAfter be,become,remain withsingular count complement

Number and count/mass nounand What precedes NP.

With the adjective certain If premodifying adjective.

a/an

After half and before singularcount nouns

Number and count/mass of nounand What precedes NP.

Before of prep.phrases Details of modifier.With relative clauses Details of modifier.After all,half and both What precedes NP.

the

Before ordinal numbers Morphological information andDetails of modifier.

With mass nouns Count/mass noun.With plural count nouns Number and count/mass of noun.

zero

With proper nouns Category of word.


Rules requiring more informatione.g. occurrence of the zero article in certain

prepositional phrases with nouns of somesemantic categories, as in “in spring” and “bycar”.

In this case, lexicon augmented with informationabout the semantic categories of nouns

So with certain singular count nouns to do withseasons, meals, time, transport, andinstitutions, and with certain prepositions,information required is:

List of types of nouns and Prepositional phrasemodifier

AILE 2004/5 Mar-23-05



Detecting article usage errors1. Student answer compared with expert model2. Incorrect article usage identified3. Uses rule induction to learn rules from

training instances4. Artcheck produce new rules, based on the

expert ones, describes what students doingi.e. identify the (incorrect) rules (= mal-rules)

5. If the system can determine a mal-rulewhich represents the student’s error, then thisis explained to the student


Form of the mal-rulesWhen conditions X apply, use article Y.

The conditions which can apply are:Whether a noun is a proper or common nounWhether a noun is a singular or plural nounWhether a noun is a count (e.g. three eggs) or

mass bread, milk nounWhether a noun is modified and with whatWhat immediately precedes the noun (for

example, the verb to be)How the noun is categorised in the lexicon

(semantic category)Whether a noun is new or given information

AILE 2004/5 Mar-23-05



Generating mal-rulesMal-rule cannot be generated from one error:- system does not have enough information

about the consistency of the error to be able todecide that a student is applying an incorrectrule.

A number of errors must be accumulated,before the system can generalise about theerrors and find a common rule to explainthem.

Errors in ArtCheck are the positive traininginstances from which the system can learn.


Positive and negative instancesPositive instances of the error (incorrect noun phrases) :

a. *John is teacher.b. *Sandy is pig.c. *I am doctor.

Negative instance of the error (correct noun phrases) :d. John is a good man.

(a) - (c) include incorrect noun phrases all using the zeroarticle where the following rule should have applied:Rule 11: Use the article a/an where a singular countnoun is used as the complement of the verb to be.

Mal-rule proposed to account for data:Where there is a singular, unmodified, common, countnoun preceded by a singular form of the verb be, usethe zero article.

AILE 2004/5 Mar-23-05



Learning from errorsOne goal is for user of the system to learn from any

errors made. To do this it must:- be able to understand the observed errors,- be able to communicate effectively with the user- provide a good explanation for that error.

The explanation is tailored to the learner in three ways:1. relating to the learner’s level of ability,2. learning style, and3. the type of error observed.

In addition, the learner is given some control over theinformation received.


Explanations:Relevant if it answers the user’s questions.Convincing justification is one which is sound, logical,

and based on facts which the user believes.Whether a user can understand the explanation depends

on:- being a well-organised explanation,- not unnecessarily long-winded, and- using terms with which the user is familiar.A tutoring explanation should vary its explanation

according to the individual student.The student should both understand and be helped by

the explanation.

AILE 2004/5 Mar-23-05



Example dialogueAC: Enter sentence:Student: I am doctorAC: identify error: doctor in I am doctor is incorrect.AC: correct error It should be: a doctor .AC: ask student for feedback

Select: m more q quit explanationStudent: mAC: state rule The rule is: RULE 11

Use a or an before singular count nouns which come after the verb to be .

AC: ask student for feedbackSelect: m more q quit explanation

Student: mAC: explain mal-rule:

I have noticed that you seem to use no article instead of a or an before a singular count and after the verb to be in the singular


Dialogue, continuedAC: ask student for feedback

Select: m more q quit explanationStudent: mAC: exemplify mal-rule

You also said: *** Sandy is pig *** John is teacher

which are similar errors.Try one of these again:

Sandy is **** pigChoose the correct article:

1 a2 an3 the4 no article

Student: 1AC: Well done. That is the correct answer.

Continue? (y/n) n

AILE 2004/5 Mar-23-05



Evaluation of ArtCheckCan understand many types of sentence structures.Cannot understand questions and imperatives.- grammar could easily be extendedSometimes wrongly predict appropriate article usage:- less common idiomatic usages.- distant or complex referring expressions, where semantic

information requiredLot of data required for mal-rule to be generated.Feedback during external evaluation generally positive:- students confirmed this was an area of difficulty- were enthusiastic about experimenting with system.- found the system easy and helpful to use.Most showed some improvement after using ArtCheck for a short

period of time.Verbal and written feedback generally very positive.


3. Other MachineLearning egs

AILE 2004/5 Mar-23-05



Machine LearningMachine Learning= developing computational theories of the

learning process and building machines whichlearn

Where used in Student Modelling:

- PIXIE (Sleeman)

- ACM Langley, Ohlsson and Sage

- Self - co-operative learning


PIXIE (Sleeman, 1983)Solving algebraic equationsHas set of mal-rules + heuristicsTakes student answer, works back to

question, infers missing steps, generatesnew mal-rule

Generates large search tree - not clear howdecides which possible mal-rule;

• only generates one mal-rule per problem;• no cause suggested for mal-rule; domain-

dependent heuristics.

AILE 2004/5 Mar-23-05



Automated Cognitive Modelling(Langley, Ohlsson and Sage 1984)

Production rule representation of skill• search through problem space• operators for chosen domain plus distinguishing properties of

domaine.g. subtraction: greater than, above, is zero

1. Obtains student response to problem2. Searches problem space - get 'solution path' (describes

operations performed by student)3. Applies inductive-learning techniques to distinguish those

operators applied successfully and those note.g learning by example +ve -ve instances

4. Constructs discrimination network, providing model ofconstraints student placed on operators

Depends a lot on properties supplied initiallyNo explanation of where errors come from, how to correct


Collaborative Learning System ForConcepts (Gilmore and Self, 1988)

Envisaged situation:- access to database of elements- has details of properties of elements- elements assigned to 3 classes- explore why assignments made- student and system work together- system has learning strategies and tactics

(= learning algorithm)

Focusing: generalises from +ve examples anddiscriminates from -ve examples(uses a relation tree)

System's knowledge in learning components

AILE 2004/5 Mar-23-05



4. Current Trends


Student Modelling - Current Trends• Continued Emphasis on Diagnosis and Error

Modelling

• Modelling Affective and Motivational Aspects ofLearner

• Open and Participative Models

• Use of Statistical and Machine Learning Techniques

• Adaptive to Wider Range of Users (IncludingDisabled)

AILE 2004/5 Mar-23-05



User Modelling: Motivation and Affect(de Vicente and Pain, 2000)

Dialogue Planning Rules:

What type of feedback to give?- depends on student's performance, effort and

confidence

Should we allow the student to give up?- depends on effort, challenge

Is the interest so low that we should interuptthe student?

- depends on sensory-interest, cognitive-interest


References:Burton,R.R. and Brown, J.S., (1982) An investigation of computer coaching

for informal learning activities, in Sleeman, D.H. and Brown, J.S. (eds),Intelligent Tutoring Systems, 79-98, London: Academic Press.

Burton,R.R. and Brown, J.S., (1979) An Investigation of Computer Coachingfor Informal Learning Activities, International Journal of Man-MachineStudies, Vol. 11, January 1979.

Burton, R.R. and Brown, J.S., (1976) A Tutoring and Student ModelingParadigm for Gaming Environments, Proceedings for the Symposium onComputer Science and Education, Anaheim, CA, February 1976.

de Vicente, A. and Pain, H. (2000) A Computational Model of AffectiveEducational Dialogues. Papers from the 2000 AAAI Fall Symposium:Building Dialogue Systems for Tutorial Applications, North Falmouth,Massachusetts, November 3-5, 2000. Technical Report FS-00-01, AAAIPress, Menlo Park, CA, USA. pp 113-121.

Gilmore, D. and Self, J. (1988), The application of machine learning to ITS, inSelf, J. (ed.), Artificial Intelligence and Human Learning: IntelligentComputer-Aided Instruction, Chapman and Hall, pp 179-196

AILE 2004/5 Mar-23-05



References:Kass, R. and Finin, T., (1988). The Need for User Models in

Generating Expert System Explanations, Technical Report MS-CIS-88-37, University of Pennsylvania.

Langley P., Olhsson, S.. & Sage. S. (1984) A Machine LearningApproach to Student Modelling Tech. Rep, CMU-RI-TR847,The Robotics institute, Carnegie University

Sentence, S.(1993). Recognising and responding to English articleusage errors: an ICALL based approach, Unpublished PhDThesis, 1993, University of Edinburgh.

Sleeman, D. H., (1983). Inferring Student Models for IntelligentCAI. Machine Learning, R Michalski, J Carbonell & T MMitchell (Eds). Tioga Press, pp 483-510.

Sleeman, D. and Brown, J.S. (1982), (eds) Intelligent TutoringSystems, Academic Press.

Lecture 7: Further Student Modelling

Documents

Transcript of Lecture 7: Further Student Modelling