1
Chapters 11 & 12, 13: Knowledge Acquisition,
Representation and Validation Opening Vignette: American Express
Improves Approval Selection with Machine Learning
The Problem: Loan Approval85 to 90 % Predicted Accurately10 to 15 % in Gray AreaAccuracy of Loan Officer’s Gray Area Decisions were at most 50 %
2
The Solution
ES with Knowledge Acquisition Method of Machine LearningRule Induction Method Gray Area: Induced Decision Tree Correctly Predicted 70 %Induced Rules Explain Why Rejected
3
Knowledge Engineering Deals with knowledge acquisition, representation, validation, inferencing, explanation and maintenance
It is the art of building complex computer programs that represent and reason with knowledge of the world
– (Feigenbaum and McCorduck [1983])
4
Knowledge Engineering Process
ActivitiesKnowledge AcquisitionKnowledge RepresentationKnowledge ValidationInferenceExplanation and Justification
5
Knowledge Acquisition Knowledge acquisition is the extraction of knowledge from sources of expertise and its transfer to the knowledge base and sometimes to the inference engine– Documented (books, manuals, etc.)– Undocumented (in people's minds)
• From people, from machines
– Knowledge Acquisition from Databases– Knowledge Acquisition Via the Internet
6
Difficulties in Knowledge Acquisition
Expressing the Knowledge– expert’s “rules” are compiled and often difficult to
articulate– not available to conscious introspection– knowledge engineer’s domain “knowledge”Transfer to a MachineNumber of ParticipantsStructuring the Knowledge
7
Experts may lack time or not cooperateTesting and refining knowledge is complicatedPoorly defined methods for knowledge elicitationSystem builders may collect knowledge from one source, but the relevant knowledge may be scattered across several sourcesCollect documented knowledge rather than use expertsThe knowledge collected may be incompleteDifficult to recognize specific knowledge when mixed with irrelevant dataExperts may change their behavior when observed and/or interviewedProblematic interpersonal communication between the knowledge engineer and the expert
Other Reasons
8
Overcoming the Difficulties
Knowledge acquisition tools with ways to decrease the representation mismatch between the human expert and the program (“learning by being told”)– Simplified rule syntax – Natural language processor to translate knowledge to a
specific representation
Critical– The ability and personality of the knowledge engineer – Must develop a positive relationship with the expert– The knowledge engineer must create the right impression
Computer-aided knowledge acquisition tools
9
Required Skills and Characteristics of Knowledge
Engineers Computer skillsTolerance and ambivalenceEffective communication abilitiesBroad educational backgroundAdvanced, socially sophisticated verbal skillsFast-learning capabilities (of different domains)Must understanding organizations and individualsWide experience in knowledge engineeringIntelligenceEmpathy and patiencePersistenceLogical thinkingVersatility and inventivenessSelf-confidence
10
Methods of Knowledge Acquisition: An Overview
Manual – Interviewing (Structured, Semistructured, Unstructured)– Tracking the Reasoning Process – Observing
Manual methods: slow, expensive and sometimes inaccurate
Semiautomatic– Support Experts Directly
Automatic (Computer Aided)– Expert’s and/or the knowledge engineer’s roles are
minimized (or eliminated) – Induction Method
11
Recommendation
Before a knowledge engineer interviews the expert(s)1. Interview a less knowledgeable (minor) expert
– Helps the knowledge engineer • Learn about the problem• Learn its significance• Learn about the expert(s)• Learn who the users will be• Understand the basic terminology• Identify readable sources
2. Next read about the problem3. Then, interview the expert(s) (much more
effectively)
12
Tracking Methods
Techniques that attempt to track the reasoning process of an expertFrom cognitive psychologyMost common formal method: Protocol Analysis
13
Protocol Analysis
Protocol: a record or documentation of the expert's step-by-step information processing and decision-making behaviorThe expert performs a real task and verbalizes his or her thought process (think aloud)
14
TABLE 13.3 Advantages and Limitations of Protocol Analysis
Advantages Limitations
Expert consciously considers decision-making heuristics
Requires that the expert be aware ofwhy he or she makes a decision
Expert consciously considers decisionalternatives attributes, values
Requires that the expert be able tocategorize major decision alternatives
Knowledge engineer can observe andanalyze decision-making behavior
Requires that the expert be able toverbalize the attributes and values of adecision alternative
Knowledge engineer can record, andlater analyze with the expert, keydecision points
Requires that the expert be able toreason about the selection of a givenalternative
Subjective view of decision makingExplanations may not track withreasoning
Source: K. L. McGraw and B. K. Harbison-Briggs, Knowledge Acquisition, Principles andGuidelines, Englewood Cliffs, NJ: Prentice-Hall, 1989, p. 217.
15
Observations and Other Manual Methods
Observe the Expert Workcard sortinginformation display boardsCase analysisCritical incident analysisDiscussions with the usersCommentariesConceptual graphs and modelsBrainstormingMultidimensional scaling
16
Expert-driven Methods Knowledge Engineers Typically – Lack Knowledge About the Domain– Are Expensive– May Have Problems Communicating With Experts
Knowledge Acquisition May be Slow, Expensive and UnreliableCan Experts Be Their Own Knowledge Engineers?Two approaches– Manual – Computer-Aided (Semiautomatic)
• To reduce or eliminate the potential problems – E.g. REFINER+ - case-based system – TIGON - to detect and diagnose faults in a gas turbine engine
17
Manual Method:Expert's Self-reports
Problems with Experts’ Reports and Questionnaires
1. Requires the expert to act as knowledge engineer
2. Reports are biased3. Experts often describe new and untested
ideas and strategies4. Experts lose interest rapidly5. Experts must be proficient in flowcharting6. Experts may forget certain knowledge7. Experts are likely to be vague
18
Machine Learning
Manual and semiautomatic elicitation methods: slow and expensiveOther Deficiencies– Frequently weak correlation between verbal reports
and mental behavior– Sometimes experts cannot describe their decision
making process– System quality depends too much on the quality of the
expert and the knowledge engineer– The expert does not understand ES technology– The knowledge engineer may not understand the
business problem – Can be difficult to validate acquired knowledge
19
Computer-aided Knowledge Acquisition, or Automated
Knowledge Acquisition Objectives
Increase the productivity of knowledge engineeringReduce the required knowledge engineer’s skill levelEliminate (mostly) the need for an expertEliminate (mostly) the need for a knowledge engineerIncrease the quality of the acquired knowledge
20
Automated Knowledge Acquisition (Machine
Learning)
Rule InductionCase-based ReasoningNeural Computing Intelligent Agents
21
Machine LearningKnowledge Discovery and Data Mining Include Methods for Reading Documents and Inducing Knowledge (Rules)Other Knowledge Sources (Databases)Tools– KATE-Induction – CN-2
22
Automated Rule Induction
Induction: Process of Reasoning from Specific to GeneralIn ES: Rules Generated by a Computer Program from CasesInteractive Induction
23
TABLE 13.6 Case for Induction - A Knowledge Map
(Induction Table)
Attributes
AnnualApplicant Income ($) Assets ($) Age Dependents Decision
Mr. White 50,000 100,000 30 3 Yes
Ms. Green 70,000 None 35 1 Yes
Mr. Smith 40,000 None 33 2 No
Ms. Rich 30,000 250,000 42 0 Yes
24
TABLE 13.12 Admission Cases
Case # GMAT GPA Decision
1 510 3.5 Yes
2 620 3.0 Yes
3 580 3.0 No
4 450 3.5 No
5 655 2.5 Yes
25
Case-based Reasoning (CBR) Adapt solutions used to solve old
problems for new problems CBR
– Finds cases that solved problems similar to the current one, and
– Adapts the previous solution or solutions to fit the current problem, while considering any difference between the two situations
Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ
26
Finding Relevant Cases Involves
– Characterizing the input problem, by assigning appropriate features to it
– Retrieving the cases with those features– Picking the case(s) that best match the
input best
– Extremely effective in complex cases– Justification - Human thinking does not
use logic (or reasoning from first principle)
– Process the right information retrieved at the right time
Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ
27
CBR Construction - Special Tools - Examples
ART*Enterprise and CBR Express (Inference Corporation)
KATE (Acknosoft)
ReMind (Cognitive Systems Inc.)
Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ
28
Example: College Major Advisor
• who are the experts?• what do they do?• how do they do it?• how do we know?• how can we represent their knowledge and
reasoning process?• how can the system mimic their reasoning
process?
29
APTITUDE INTERESTS FINANCIALNEED
RECOMMENDATIONOF A MAJOR
College Major Advisor: Initial Block Diagram
30
APTITUDE INTERESTS FINANCIALNEED
RECOMMENDATIONOF A MAJOR
College Major Advisor: Expanded Block Diagram
mathskills
prog.skills
manualdexterity
computers problemsolving
repairthings
desk vs.field
job atgraduation
31
math? (yes, no)
programming (yes, no)
manual dexterity (yes, no)
computers (yes, no)
problem solving (yes, no)
place (desk, field)
repair (yes, no)
finance (yes, no)
APTITUDE
INTERESTS
SUGGESTEDMAJOR
College Major Advisor: Dependency Diagram
32
Induction Table Example
Induction tables (knowledge maps) focus the knowledge acquisition process
Choosing a site for a hospital clinic facility
33
TABLE 13.9: Induction Table (Knowledge Map) Example
PopulationDensity
Densityover HowMany Sq.mi
Number of Near(within 2 miles)Competitors
AverageFamilyIncome
Near PublicTransportation?
Decision(Choices)
People /Square Mile
Numeric,Region Size
0, 1, 2, 3, ... Numeric,$ / Year
Yes, No Yes, No
>= 2000 >=4 0 Yes
>=3500 >=4 1 Yes
>=2 No
<30,000 No
34
Row 1: Factors Row 2: Valid Factor Values and Choices (last column)
Table leads to the prototype ESEach row becomes a potential ruleInduction tables can be used to encode chains of knowledge
35
Knowledge Representation
a variety of alternative representations have been developed: rules, frames, semantic nets
efficiency and effectiveness criteria incorporation into ES development
software
36
Major Advantages of Rules
Easy to understand (natural form of knowledge)
Easy to derive inference and explanations
Easy to modify and maintain Easy to combine with
uncertainty Rules are frequently
independent
37
Production Rules: Drawbacks variables
– unwieldy for large body of knowledge– only stand-alone, unrelated facts
production rules– have to embed all context into antecedent– not appropriate for all types of problem
solving or expertise
38
Frames an abstraction of a single object or a
concept– frame representing a car
slots are used to define properties and attributes of a frame
efficient for hierarchical knowledge structures: supports inheritance
inference can be done with production rules based on value expectations
KR With FramesANIMALS
MAMMALS REPTILES BIRDS FISH
DOG HUMAN WHALEPENGUIN
LONG JOHNSILVER
Reproduce: YesLife-Form: Yes
Scales: YesWarmblooded: No
Warmblooded: YesSpinal Cord: YesLegs: 4
Legs: 2
Legs: 1
Legs: 0Fins: Yes
Legs: 2Wings: YesFeathers: Yes
Feathers: No
Scales: YesFins: yes
40
Semantic Networks
network of nodes and links– nodes = concepts– links = relationships
supports certain types of inference– “is-a” = inheritance (generalization/specialization)– “has-a” = property aggregation– “part-of” = physical component aggregation
41
KR With Semantic Networks
LIGHT
WORKINGHEADLIGHT
FUNCTIONALLIGHT BULB
FUNCTIONALCIRCUIT
IS-A
HAS-A HAS-A
42
Semantic Networks advantages
– supports limited inferencing– good model of long-term memory
organization?– easy to understand
disadvantages– quickly becomes unwieldy– doesn’t distinguish between objects,
attributes, or values
KR with Multiple Schemes
IF Battery-Age < 5 and Battery_terminals are Uncorroded THEN Battery can work
If Battery can work and Fuse is Functional and Wiring is Functional THEN Electric Circuit can work
ELECTRIC SUBSYSTEM
IGNITION HEADLIGHT
LIGHT BULB ELECTRIC CIRCUITWorking
Head Light
FunctionalLight Bulb
FunctionalCircuit
WorkingBattery
FunctionalFuse
FunctionalWiring
5-YearsOld
UncorrodedTerminals
HAS-A HAS-A
HAS-A
HAS-A
NEEDS-A HAS-A
NEEDS-A
44
Choosing a Representation Scheme should support acquisition, retrieval,
and reasoning naturalness, uniformity, and
comprehensibility degree of explicitness Modularity and flexibility of the
knowledge base efficiency of knowledge retrieval and
heuristic power
45
Validation & Verification of the Knowledge Base
Evaluation– Assess an expert system's overall value– Analyze whether the system would be
usable, efficient and cost-effective
Validation – Deals with the performance of the system
(compared to the expert's)– Was the “right” system built (acceptable
level of accuracy?)
Verification– Was the system built "right"?– Was the system correctly implemented to
specifications?
46
TABLE 13.8 Measures of Validation
Measure (Criteria) Description
Accuracy How well the system reflects reality; how correct the knowledge is in theknowledge base
Adaptability Possibilities for future development, changes
Adequacy (or completeness) Portion of the necessary knowledge that is included in the knowledge base
Appeal How well the knowledge base matches intuition and stimulates thought andpracticability
Breadth How well the domain is covered
Depth Degree of the detailed knowledge
Face validity Credibility of knowledge
Generality Capability of a knowledge base to be used with a broad range of similarproblems
Precision Capability of the system to replicate particular system parameters;consistency of advice; coverage of variables in knowledge base
Realism Accounting for relevant variables and relations; similarity to reality
Reliability Fraction of the ES predictions that are empirically correct
Robustness Sensitivity of conclusions to model structure
Sensitivity Impact of changes in the knowledge base on quality of outputs
Technical andoperational validity
Quality of the assumed assumptions, context, constraints and conditions, andtheir impact on other measures
Turing Test Ability of a human evaluator to identify if a given conclusion is made by anES or by a human expert
Usefulness How adequate the knowledge is (in terms of parameters and relationships)for solving correctly
Validity Knowledge base's capability of producing empirically correct predictions
Source: Adapted from B. Marcot, "Testing Your Knowledge Base," AI Expert, August 1987.
47
Method for Validating ES
Test1. The extent to which the system
and the expert decisions agree2. The inputs and processes used by
an expert compared to the machine
3. The difference between expert and novice decisions
(Sturman and Milkovich [1995])
48
ES Elements to be Validated output, recommendations
– internal validity: completeness and consistency– external validity: test cases, Turing test
reasoning system/user interface efficiency
– hardware, software– KR and response time
cost-effectiveness
49
Inferencing Methods objective
– find logical reasoning path between known data and conclusions
backward chaining– goal-directed search– start with where you want to end up and see if
data will get you there
forward chaining– data-directed search– start with what you know and see where it takes
you
50
ExampleVariables: A = have $10,000
B = younger than 30C = education at college levelD = annual income > = $40,000E = invest in securitiesF = invest in growth stocksG = invest in IBM stocks
Rules: R1: if A and C then ER2: if D and C then FR3: if B and E then FR4: if B then CR5: if F then G
Initial Facts: A and B
51
Inferencing Methods
which way should you go?– forward chaining tends to collect more data– forward chaining is good for design problems– backward chaining tends to examine more rules – backward chaining is good for diagnosis problems
vp-expert is backward chaining other expert system shells are forward
chaining, backward chaining, or both
52
Inferencing with Uncertainty Uncertainty in AI - Three-step Process
1. An expert provides inexact knowledge in terms of rules with likelihood values. Also, users may be uncertain with respect to inputs.
2. The inexact knowledge of the basic set of events can be directly used to draw inferences in simple cases (Step 3)
3. Working with the inference engine, experts can adjust the Step 1 input after viewing the results in Steps 2 and 3.
– In Step 2: Often the various events are interrelated. – Necessary to combine the information provided in Step 1
into a global value for the system
Major integration methods: Bayesian probabilities, theory of evidence, certainty factors and fuzzy sets
Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ
53
Theory of Certainty (Certainty
Factors)
Uncertainty is represented as a Degree of Belief or Certainty FactorCertainty Factors (CF) express belief in an event (or fact or hypothesis) based on evidence (or the expert's assessment)CFs are NOT probabilitiesCFs need not sum to 100
Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ
54
Rule 9if wine_color = red and body = full and sweetness = drythen wine = cabernet_sauvignon cnf 80because "This is a good wine~";
Rule 1if main_component = meatthen wine_color = red; Rule 3if has-sauce = yes and sauce_type = creamythen body = full cnf 90because "Creamy sauce needs full bodied wine"; Rule 8if likes = drythen sweetness = dry;
INPUTS: Meat cnf=100; has_sauce=95; creamy=70; likes dry=.60INTERMEDIATE: wine-color=red 100; body=full 63 [min(95,70)*90]; sweetness=dry 60FINAL: wine=cabernet sauvignon 48 [(min 100, 63, 60)*80]
Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ
55
Calculating Confidence Factors
For conjunction (AND): – take min CNF of conditions
For disjunction (OR):– when one condition only is true take the CNF of true condition
– when both conditions are true:– CNF antecedent=max CNF of conditions
– CNF Rule: Antecedent CNF*Consequent CNF
56
AND
– IF inflation is high, CF = 50 percent, (A), AND– IF unemployment rate is above 7 percent, CF =
70 percent, (B), AND– IF bond prices decline, CF = 100 percent, (C)– THEN stock prices decline
CF(A, B, and C) = Minimum[CF(A), CF(B), CF(C)] The CF for “stock prices to decline” = 50 percentThe chain is as strong as its weakest link
57
IF inflation is low, CF = 70 percent; OR
IF bond prices are high, CF = 85 percent;
THEN stock prices will be high
Only one IF need be true Conclusion has a CF with the maximum of the two
– CF (A or B) = Maximum [CF (A), CF (B)]
CF = 85 percent for stock prices to be highDecision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ
OR
58
Combining Two or More Rules
Example:– R1: IF the inflation rate is less than 5 percent,
THEN stock market prices go up (CF = 0.7)
– R2: IF unemployment level is less than 7 percent,THEN stock market prices go up (CF =
0.6)
Inflation rate = 4 percent and the unemployment level = 6.5 percent Combined Effect– CF(R1,R2) = CF(R1) + CF(R2)[1 - CF(R1)]; or – CF(R1,R2) = CF(R1) + CF(R2) - CF(R1) CF(R2)
Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ
59
Assume an independent relationship between the
rules Example: Given CF(R1) = 0.7 AND CF(R2) = 0.6, then: CF(R1,R2) = 0.7 + 0.6(1 - 0.7) = 0.7 + 0.6(0.3) = 0.88
ES tells us that there is an 88 percent chance that stock prices will increase
Decision Support Systems and Intelligent Systems, Efraim Turban and Jay E. AronsonCopyright 1998, Prentice Hall, Upper Saddle River, NJ