Cross-System Personalization for College Students

Cross-System Personalization for College Students

Peter Brusilovsky with Sergey Sosnovsky

Michael Yudelson

Shaghayegh Sahebi

Chirayu Wongchokprasitti

Sharon Hsiao

User Model

Collects information about individual user

Provides adaptation effect

Adaptive System

User Modeling side

Adaptation side

User-Adaptive Systems Then…

Classic loop user modeling - adaptation in adaptive systems

University of Pittsburgh - PAWS Lab

… and now


Personalization Challenges in the New Context •  How an adaptive system can benefit from

information about users collected by other systems? – What is the framework for UM integration? – Will it improve cold-start situation? – Will it improve parallel use of multiple systems

•  Can we do it for different types of user models? – Knowledge model –  Interest model

University of Pittsburgh - PAWS Lab 4

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N 10 3

0

2 7

4

Concept-Level Knowledge Model


Cross-System Knowledge Modeling

http://adapt2.sis.pitt.edu/kt/

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes


Missing links

The Approach: Ontology-Based Cross-System Personalization


Connect DM (ontologies)

Main Stages of Our Work

•  Centralized user modeling (1990-1998) •  Multi-system personalization based on single

domain model: ADAPT2 (2003-2007) •  Cross-domain mapping for cold start (2007)

–  C to Java •  Single domain guided evidence mapping

(2008-2010) –  Topic to concept mapping for Java –  Constraints to concepts mapping for SQL

•  Single domain automatic mapping (2010-2012) University of Pittsburgh - PAWS Lab 8

UM of C knowledge

Java C

UM of Java

knowledge

How we started – from C to Java

•  Manual vs. Automatic ontology mapping

•  Knowledge mapping using ontology mapping

•  Compare predicted and demonstrated knowledge

•  Automatic mapping is comparable with manual

•  Overall gain for translated knowledge is not high

•  We got concerned about model to model mapping

•  Started exploring evidence mapping


SEDONA: UM exchange with ontology servers


Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes

Ontology A

Ontology B


SEDONA: UM Exchange

•  Ontology server is an exchange point for concept-level overlay student models that are based on the stored ontology

•  Each UM server or adaptive system that can deduce student knowledge in terms of this ontology reports it to the server

•  Each adaptive system that need to know the level of student knowledge for concepts of this ontology can query the ontology server


Lightweight event-based centralized user modeling


Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes

Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes

Central UM Concept 1

Concept 2

Concept 3

Concept 4

Concept 5

Concept N yes no

no

no yes

yes


•  Student side: – Use systems in parallel (any order, any

combination) – No extra overhead (single sign-on, single

place to access) •  System side:

– Integrated environment > (system1 + system2)

– Each system should try to increase the quality of user modeling and adaptation

Goal: True Integration


Java Problets: The Interface

Sample program

Student’s answer

Help

Question text

System’s feedback

Java Problets: Domain Model •  Problets implement traditional overlay user modeling to

adapt to student’s performance

l  The domain model of a problet is a concept map enhanced with learning objectives, that combine pedagogical and domain knowledge

QuizJET (1): System Description •  QuizJet (Java Evaluation Toolkit) is a system for authoring

and delivery of online self-assessment quizzes for Java programming language

•  A typical QuizJET problem is a sample program (consisting of one or several classes), that a student needs to evaluate and provide an answer a follow-up question

•  QuizJET generates problems by substituting a numerical value in the program template with a randomized parameter

•  Upon receiving a student’s answer QuizJET provides a feedback indicating the correctness of the answer and the right answer (if the student’s attempt was not successful)

QuizJET (2): Student Interface •  Students can access QuizJET problems through the

KnowledgeTree portal

Topics in the course

Activities available for the

current topic

Problem text

Problem's classes

QuizJET’s feedback

QuizJET (3): Domain Model •  Java Ontology

specifies about 500 classes connected with 3 types of relations: subClassOf, partOf/hasPart, and related

•  About 300 classes are available for indexing

•  A class can play one of two roles in the problem index: prerequisite or outcome


Domain Model Integration •  Main problem: different modeling paradigms

–  A learning objective models application of a concepts in the certain context

–  Extra classes from the Java ontology have been used for context modeling

–  Weights are assigned to prevent too aggressive propagation of classes responsible for context modeling

•  Example: –  This learning objective models a situation when the conditional part

of the if-else statement is a relational expression evaluated into true value

Evidence-based UM integration in CUMULATE


SQL-Exploratorium


SQL-Tutor

Goal: Integrated Environment

http://www.sis.pitt.edu/~paws/ont/SQL.owl

SQL Explorer: SQL Ontology


SQL-Tutor: Constraints


•  Constraints and Concepts are too difficult to map them automatically

•  A typical constraint models syntactic or semantic relation between several concepts

•  Manual connect constraint to concepts with some degree (small-1, medium-2, or large-3)

Domain Model Mapping


•  Solution to SQL-Tutor problem, triggers a number of constraints satisfied and or violated

•  Mapping model calculates knowledge update for every concepts related to every triggered constrained:

•  The updates are reported to SQL-Exploratorium’s user modeling server

Evidence-Based Modeling


Architecture

•  University of Pittsburgh, 2 courses: undergraduate and graduate

• ½ of semester

•  42 students tried SQL-KnoT, 18 – SQL-Tutor

•  Out of 103 sessions of using SQL-KnoT 66 co-located with SQL-Tutor usage

Evaluation


•  Questionnaire (21 students) – I1 / I2: Overall, I like the interface of SQL-

KnoT/SQL-Tutor. – U1 / U2: SQL-KnoT/SQL-Tutor is a useful

learning tool. – C1 / C2: SQL-KnoT/SQL-Tutor problems

challenged me intellectually.

Results

•  An example of semantic integration of two working adaptive systems relaying on very different domain models

•  Students used the systems together during single sessions and liked the opportunity

•  More evaluation is needed to verify the effect of integration of user modeling accuracy and adaptation

•  It is interesting to evaluate the combined adaption (adaptive navigation from SQL-Exploratorium followed by intelligent coaching from SQL-Tutor)

What was presented in the past UbiqUM


Discussion

+ Experts only need to produce relations b/w KIs – the rest is automatic

+ Relations can be removed (strength=0)

-  Cannot add relations


References: Past UbiqUM papers Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., Yudelson, M., Brusilovsky, V., and Sharma, D.

(2008) Towards integration of adaptive educational systems: mapping domain models to ontologies. Proceedings of 6th International Workshop on Ontologies and Semantic Web for E-Learning (SWEL'2008) in conjunction with ITS'2008, Montreal, Canada, June 23, 2008.

Sosnovsky, S., Mitrovic, A., Lee, D. H., Brusilovsky, P., and Yudelson, M. (2008) Ontology-based integration of adaptive educational systems. Proceedings of 16th International Conference on Computers in Education (ICCE’2008), Taipei, Taiwan, October, 27-31, 2008, pp. 11-18.

Brusilovsky, P., Sosnovsky, S., Yudelson, M., Kumar, A., and Hsiao, I.-H. (2008) User Model Integration in a Distributed Adaptive E-Learning Systems. Proceedings of Workshop on User Model Integration at the 5th International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems (AH'2008), Hannover, Germany, July 29, 2008.

Brusilovsky, P., Mitrovic, A., Sosnovsky, S., Mathews, M., Yudelson, M., Lee, D., and Zadorozhny, V. (2009) Database exploratorium: a semantically integrated adaptive educational system. In: Proceedings of Ubiquitous User Modeling Workshop at the 17th International Conference on User Modeling, Adaptation, and Personalization (UMAP 2009), Trento, Italy, June 22, 2009

Sosnovsky, S., Brusilovsky, P., Yudelson, M., Mitrovic, A., Mathews, M., and Kumar, A. (2009) Semantic Integration of Adaptive Educational Systems. In: T. Kuflik, S. Berkovsky, F. Carmagnola, D. Heckmann and A. Krüger (eds.): Advances in Ubiquitous User Modelling. Lecture Notes in Computer Science, Vol. 5830, pp. 134-158


Evaluating and improving mapping: SQL Exploratorium and SQL Tutor •  Authoring constraint mapping is time consuming

•  How we can evaluate weights?

•  How we can improve mapping?

34

147 "You have used some names

in the WHERE clause that are not

from this database."

207 "You need to specify the join

condition in FROM!"

Join

FROM Clause

Attribute

Table

Database

ConceptsConstraintsw=1

w=2/3

w=2/3

w=2/3

w=1/3


SQL KnoT and SQL-Tutor (2) •  6 experts (2 teachers, 2 GSA, 2 practitioners)

•  1012 constraint-concept relations: strong (1/1), medium (2/3), weak (1/3)

•  Usage log of 3544 SQL-Tutor problem-solving attempts of 38 users

•  Dataset specific subset – 282 constraints, 576 relations, 61 concepts


Fitting The Source (Constraint) Model

•  Experts only need to produce relations b/w KIs – the rest is automatic

36 University of Pittsburgh - PAWS Lab

What was not presented at UbiqUM

•  Sosnovsky, S., Dolog, P., Henze, N., Brusilovsky, P., and Nejdl, W. (2007) Translation of overlay models of student knowledge for relative domains based on domain ontology mapping. 13th International Conference on Artificial Intelligent in Education, AI-ED 2007, Marina Del Rey, CA, July 9-13, 2007, IOS, pp. 289-296

•  Yudelson, M., Brusilovsky, P., Mitrovic, A., and Mathews, M. (2010) Using Numeric Optimization To Refine Semantic User Model Integration Of Adaptive Educational Systems. Proceedings of the Third International Conference on Educational Data Mining (EDM 2010), Pittsburgh, PA, June 11-13, 2010, pp. 221-230.

•  Sosnovsky, S. (2011) Ontology-based Open-Corpus Personalization for e-Learning. PhD Thesis, University of Pittsburgh


What Happened with auto-mapping?

38 University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis

OOPS Interface: Reading Phase

39

content of the chosen topic

Navigation links to the next and the previous topics

Feedback/exit buttons

University of Pittsburgh - PAWS Lab Sergey Sosnovsky PhD Thesis

Cross System Interest Modeling

•  CoMeT: a social system for sharing information about research colloquia in Pittsburgh

•  Models user research interests by observing bookmarking and sharing behavior

•  Cold start problem – can’t recommend with no bookmarks

•  Can we seed user profiles using other systems that represent user research interests? – Paper bookmarking systems – CuteULike


CoMeT: Collaborative Management of Talks (try pittcomet.info)


Traditional Interest Modeling with Keywords

•  Document model –  a bag of words represented as a vector in keywords

vector space with TF.IDF weighting scheme


W1

W2

W3

W4

W5

W6

D1 0 1 0 0 0 0

D2 .5 0 0 .5 0 0

D3 .12 .13 0 .25 .5 0

D4 .25 0 .25 0 .25 .25

Talks/Papers

Keywords

Recommending Talks to Users •  User model

–  A combination of vectors of “interesting documents” –  Possibly weighted by the rare of interest

•  K-nearest neighbor method –  recommend top K closest documents to user profile


D1 D2 D3 D4

U1 1 0 0 0

U2 .25 0 .5 .25

U3 0 .5 .25 .25

W1

W2

w3

D1 0 1 0

D2 0 0 .5

D3 0 1 0

D4 0 0 .5

U: User Profiles in Talks/Papers Space

D: Documents in Keywords Space

users

Documents Keywords

w1 w2

w3

U1 1 0 1

U2 .25 0.5

.37

U3 0 .25

.37

UP: User Profiles in Keywords Space

users

Keywords

Recommendation with Additional Sources of Information •  Sources of information about user interests:

– Standard information: Keywords of bookmarked talks in CoMeT

– Tags of talks in CoMeT – Keywords of bookmarked papers from CiteULike – Tags of papers in CiteULike (CUL)

•  Explore the impact of additional sources

•  Also explore different models for fusion of tags and keywords


Document Representation Models

•  Control condition: Keywords Only (KO) –  Keywords extracted from documents’ titles and abstracts

•  Keywords+n*Tags (KnT) –  Keywords extracted from documents’ titles and abstracts +

tags assigned to documents

•  Keywords Concatenated by Tags (KCT) –  Keywords extracted from documents’ titles and abstracts +

tags assigned to documents


Keywords+n*Tags (KnT) Model •  Each document: a bag of words containing :

–  document’s abstract, title and tags

•  Tags: regular keywords –  Each tag appears n times

•  Merge CUL and CoMeT data in this model: same as KO


W1 W2 W3/T1

W4/T2 T3 T4

D1 0 1 1 0 0 0

D2 1 0 3 5 0 0

D3 1 2 3 0 1 0

D4 2 0 5 0 2 1

Keywords

Talks/Papers

Common Keywords & Tags Tag

s

Keywords: w1, w2, w3, w2

Tags: T1, T3

D3

n=2

W3=T1 W4=T2

Keywords Concatenated by Tags (KCT) Model •  Tags: a separated source of information

•  Each document: a bag of keywords and a bag of tags – Concatenating keywords and tags vectors – TF.IDF weightening scheme


W1 W2 W3 W4 T1 T2 T3 T4

D1 0 1 1 0 0 0 0 0

D2 1 0 3 1 0 2 0 0

D3 1 2 1 0 1 0 1 0

D4 2 3 3 0 1 0 2 1

Keywords

Talks/Papers

Tags

W3=T1 W4=T2

Keywords: w1, w2, w3, w2

Tags: T1, T3

D3

Merging CUL and CoMeT Data


w1 w2

T1 T2

P1 1 0 0 0

P2 .25 0 .5 .25

P3 0 .5 .25 .25

W2

W3

T1

C1 0 1 0

C2 0 0 .5

Dc: CUL Papers’ Matrix Dt: CoMeT Talks’ Matrix W1 w2 W3 T1 T2

C1 0 0 1 0 0

C2 0 0 0 .5 0

P1 1 0 0 0 0

P2 .25 0 0 .5 .25

P3 0 .5 0 .25 .25

k

m+i

e

l+j

K+e

l+m+i+j-o-p

D: Merged Documents’ Matrix

k- the number of CiteULike papers m- the number of keywords used in CiteULike papers i- the number of tags used in CiteULike papers e- total number of talks in CoMeT l- total number of keywords in CoMeT j- total number of tags in CoMeT o- the number of common keywords between two CoMeT and CiteULike systems P- the number of common tags between two CoMeT and CiteULike systems

Experimental Results

•  User study: –  8 real users of both CoMeT and CiteULike systems

•  Questionnaire for each recommended talk: –  Is this talk related to your interest? (yes/no question) –  How interesting this talk to you? (in 5-point scale) –  If the talk is related to your interests, how novel is this talk

to you? (in 5-step scale) •  Measures:

–  Relevance: precision by yes/no answers –  Interest: nDCG by 5-point scale –  Novelty: averaged the novelty ratings (Non-relevant = zero

novelty) University of Pittsburgh - PAWS Lab 49

Precision and Novelty for different number of recommendations


Conclusion

•  Including another reliable user profile → increase precision of recommendations;

•  Using CiteULike data for all models –  Increased Relevance of recommended documents –  Decreased novelty for KO model

•  CiteULike: adding, reviewing and rating related papers to their research field •  CoMeT: information about talks happening within a specific time given on a particular

date users bookmark a more novel, less relevant talk

•  Adding tags –  Increased novelty of recommendations (both using CoMeT and CUL data) –  Increased relatedness in larger number of recommendations

•  Injection of keywords from another source of data: more reliable than including tags for relevance

•  Including tags from various sources of information: more reliable for interestingness or novelty


Back to the start

One user, many models of the same user


User Model

Adaptive System

Let’s look from the other side

Many users, many models of different users

User Model

User Model

User Model


Treemap Group UM for Java Programming



The more the students compared to their peers, the higher post-quiz scores they received (r= 0.34 p=0.004)

Progressor for Java Programming


Progressor+ for rich content


And some more references

•  Sahebi, S., Wongchokprasitti, C., and Brusilovsky, P. (2010) Recommending research colloquia: a study of several sources for user profiling. In: Proceedings of the 1st International Workshop on Information Heterogeneity and Fusion in Recommender Systems (HetRec 2010) at the 2010 ACM conference on Recommender systems, RecSys '10, Barcelona, Spain, ACM, pp. 32-38

•  Brusilovsky, P., Hsiao, I.-H., and Folajimi, Y. (2011) QuizMap: Open Social Student Modeling and Adaptive Navigation Support with TreeMaps. Proceedings of 6th European Conference on Technology Enhanced Learning (ECTEL 2011), Palermo, Italy, Sptember 20-23, 2011, Springer-Verlag, pp. 71-82.

•  Hsiao, I.-H., Bakalov, F., Brusilovsky, P., and König-Ries, B. (2011) Open Social Student Modeling: Visualizing Student Models with Parallel Introspective Views. In: Proceedings of 19th International Conference on User Modeling, Adaptation, and Personalization, UMAP 2011, Girona, Spain, Springer-Verlag, pp. 171-182.

•  Hsiao, I.-H., Guerra, J., Parra, D., Bakalov, F., König-Ries, B., and Brusilovsky, P. (2012) Comparative Social Visualization for Personalized E-learning. Proceedings of the Working Conference on Advanced Visual Interfaces, AVI 2012, Capri, Italy, ACM Press, pp. 303-307.


Cross-System Personalization for College Students

Education

Transcript of Cross-System Personalization for College Students