Text Classification with Belief Augmented Frames

Text Classification withBelief Augmented Frames

Colin TanDepartment of Computer Science,

School of Computing,National University of Singapore.

Outline

• What are Belief Augmented Frames?• Motivation behind Belief Augmented Frames• Representing Beliefs in BAFs• Some Definitions• Belief Augmented Frame Logic (BAF-Logic)• Applying BAF-Logic to Text Classification• Experiment Protocol and Results• Conclusions

What are Belief Augmented Frames?

• Belief Augmented Frames (BAF) combine classical AI frames with belief measures.– Frame-based system to structure knowledge

and relations between entities.– Belief measures provide uncertain reasoning on

existence of entities and the relationships between them.

Motivation behind Belief Augmented Frames

• Why Belief Measures?– Statistical Measures

• Standard tool for modeling uncertainty.• Essentially, if the probability that a proposition E is true is p,

then the probability of that E is false is 1-p.– P(E) = p– P(not E) = 1-p

• This relationship essentially leaves no room for ignorance. Either the proposition is true with a probability of p, or it is false with a probability of 1-p.

• This can be counter-intuitive at times.


• Why Belief Measures?– [Shortliffe75] cites a study in which, given a set

of symptoms, doctors were willing to declare with certainty x that a patient was suffering from a disease D, yet were unwilling to declare with certainty 1-x that the patient was not suffering from D.


• Why Belief Measures?– To allow for ignorance our research focuses on

belief measures.– The ability to model ignorance is inherent in

belief systems.• E.g. in Dempster-Shafer Theory [Dempster67], if

our belief in E1 and E2 are 0.1 and 0.3 respectively, then the ignorance is (1 – (0.1 + 0.3)) = 0.6.


• Why Frames?– Frames are a powerful form of representation.

• Intuitively represents relationships between objects using slot-filler pairs.

– Simple to perform reasoning based on relationships.

• Hierarchical– Can perform generalizations to create general models

derived from a set of frames.

Example BAF

Alice,1.0, 0.0

owns0.7, 0.2

walks0.9, 0.1

Donkey0.6, 0.3

color1.0, 0.0

Grey,1.0, 0.0

Dog0.9, 0.0

color1.0, 0.0

location1.0, 0.0

Blue,1.0, 0.0

Bay,1.0, 0.0

Belief Representation in Belief Augmented Frames

• Beliefs are represented by two masses:– φT: Belief mass supporting a proposition.– φF: Belief mass refuting a proposition.– In general φT + φF 1

• Room to model ignorance of the facts.

• Separate belief masses allow us to:– Draw φT

and φF from different sources.– Have different chains of reasoning for φT and φF.

Belief Representation in Belief Augmented Frames

• This ability to derive the refuting masses from different sources and chains of reasoning is unique to BAF.– In Probabilistic Argumentation Systems (the

closest competitor to BAF) for example, p(not E) = 1 – p(E).

Some Definitions

• Degree of Inclination– The Degree of Inclination is defined as:

• DI = T - F – DI is in the range of [-1, 1].– One possible interpretation of DI:

-1 0 1

IgnorantMost

ProbablyFalse

MostProbably

True

ProbablyFalse

ProbablyTrue

-0.75 -0.5 0.5 0.75

False True

-0.25LikelyFalse

0.25LikelyTrue

Some Definitions

• Utility Value– The Degree of Inclination DI can be re-mapped

to the range [0, 1] through the Utility function:• U = (DI + 1) / 2• By normalizing U across all relevant propositions it

becomes possible to use U as a statistical measure.

Belief Augmented Frame Logic(BAF-Logic)

• Belief Augmented Frame Logic, or BAF-Logic, is used for reasoning with BAFs.

• Throughout the remainder of this presentation, we will consider two propositions A and B, with supporting and refuting masses T

A, FA, T

B, and FB.


• A B: T

A B = min(TA, T

B) F

A B = max(FA, F

B)• A B:

TA B = max(T

A, TB)

FA B = min(F

A, FB)

A: T

A = F

A F

A = T

A


• BAF-Logic properties that are identical to Propositional Logic:– Associativity, Commutativity, Distributivity,

Idempotency, Absorption, De-Morgan’s Theorem, - elimination.

• Other properties of Propositional Logic work slightly differently in BAF-Logic.– In particular, some of the properties hold true only if the

constituent propositions are at least “probably true” or “probably false”

• I.e. |DIP | 0.5


• An Example:– Given the following propositions in your

knowledge base:• KB = {(A, 0.7, 0.2), (B, 0.9, 0.1), (C, 0.2, 0.7), (A

B R, TONE , F

ONE,), (A B R, TONE , F

ONE)}

• We want to derive TR, F

R.


• Combining our clauses regarding R, we obtain:– R = (A B) (A B)

• = A B ( A B)

• With De-Morgan’s Theorem we can derive R: R= A B (A B)


TR = min(T

A , TB , max(F

A , TB ))

= min(0.7, 0.9, max(0.2, 0.9))

= min(0.7, 0.9, 0.9) = 0.7 F

R = max(FA , F

B , min(TA , F

B ))

= max(0.2, 0.1, min(0.7, 0.1))

= max(0.2, 0.1, 0.1) = 0.2


• DIR = TR - F

R

= 0.7 – 0.2

= 0.5

• UR = (1 + 0.5) / 2.0= 0.75

• Suppose now it is known that B C R


• Combining our clauses regarding R, we obtain:– R = (A B) (B C) (A B)

= A B C ( A B)

• With De-Morgan’s Theorem we can derive R: R= A B C (A B)


TR = min(T

A , TB , T

C , max(FA , T

B ))

= min(0.7, 0.9, 0.2, max(0.2, 0.9))

= min(0.7, 0.9, 0.2, 0.9) = 0.2 F

R = max(FA , F

B , FC , min(T

A , FB ))

= max(0.2, 0.1, 0.7, min(0.7, 0.1))

= max(0.2, 0.1, 0.7, 0.1) = 0.7


• DIR = TR - F

R

= 0.2 – 0.7= -0.5

• UR = (1 - 0.5) / 2.0= 0.25

• Here the new evidence that B C R fails to support R, because C is not true (DIC = -0.5)

Text ClassificationFirst Approach

• First Formulation:– Using Individual Word Scores– Assuming that a document di belongs to a class

ck, then for every term tij the following relation holds:

di ck (ti0 ck ti1 ck ti2 ck … ti,n-1 ck)


• The final score of a document di belong to class cj is given by:

20.1

ki

ki

cdcd

DIU

• Where:F

cdT

cdcd kikikiDI


• Individual term probabilities are derived using Bayesian probabilities:

)()()|(

)|(ij

kkijijk tp

cpctptcp

Text ClassificationSecond Approach

• We classify the entire document using Naïve Bayes assumption:

j

ijkik tcpdcp )|()|(

• Trivial to derive the supporting score that di ck.

– It is simply p(ck | di)


• Formulating the Refuting Score is straightforward too:di ck di cm di cn di cp…, m, n, p, etc k

• We can formulate both supporting and refuting scores in BAF-Logic:

)|( ikT

cd dcpki

),...)|(),|(),|(max( ipinimF

cd dcpdcpdcpki


• We retain the definitions of DI and U from the first approach.

Experiment Protocol

• Using Andrew McCallum’s “Bag of Words” or BOW library.– Extended “rainbow”, the text-classification

front-end, with two BAF classification methods.

• Methods are called BAF1 and BAF2– Also extended with two PAS methods (see

paper for more details)• Methods are called PAS1 and PAS2

Experiment Protocol

• Corpus:– 20 Newsgroups– 80% (16,000) documents used to generate

statistics.– 20% (4,000) documents used for testing– Choice of documents for training/testing

handled by BOW– Headers removed from all documents

Experiment Protocol

• Trials– 10 trials were performed using each

classification method.• Naïve Bayes, tf.idf, kNN, EMM, Max entropy,

Probabilistic Indexing, BAF1, BAF2, PAS1, PAS2– The average was taken from the 10 trials for

each method.

Experiment ResultsText Classification Accuray

82.09 79.88

35.17

82.01 81.15 77.7368.98

82.36

65.87 67.46

0

10

20

30

40

50

60

70

80

90

100

Method

Acc

urac

y (%

)

Analysis

• BAF1 performs poorly.– Using individual word scores appears to be a poor idea.

• BAF2 performs very well.– Better than the other methods attempted.

• BAF2 Performance slightly better than Naïve Bayers– Appears that considering a document to belong to

another class has a positive effect on classification scores.

Conclusion

• Experiment results show that the use of BAF-Logic to classify documents might be a good idea.

• In addition there are features of BAFs (e.g. daemons attached to slots) that might enhance classification performance further.

• More work should be done on this.– Understanding better why BAF-Logic works for text

classification.– Improving classification performance.

Text Classification with Belief Augmented Frames

Documents

Transcript of Text Classification with Belief Augmented Frames