Automatic Strategies for Decision Support in Telephone Triage
Framework and Testbed in Smalltalk
Carlos E. [email protected]: Dan Rozenfarb
Agenda
Introduction Software Application: ExpertCare Overview of the Framework:
Concept representation Session, automation and simulation Strategies
Some statistics Examples and results
Telephone Triage
Phone call from a patient
Initial data gathered Questions and
answers Presumptive
diagnosis Ambulance dispatch
or treatment indications
ExpertCare
Initial identification data on a standard form
ExpertCare
Selection of all initial symptoms reported by the caller
ExpertCare
New questions are suggested.
Answers are recorded.
Diagnoses are re-evaluated.
Question in plain Spanish
List of scored diagnosesList of symptoms
Session information
ExpertCare
As new information is gathered,
some diagnoses
are separated according
to their score
ExpertCare composition
Modules of ExpertCare system
Inference Engine
Interrogator
Ontology
Knowledge base:Symptoms, Syndromeswith attributes
Interrogatory Rules
Scorer
ExpertCare ontology
Symptoms (e.g. fever, headache, dyspnea)
Syndromes (e.g. appendicitis, osteomyelitis, asthma, schizofrenia)
Systems (e.g. circulatory, digestive) Severities (red, yellow, green) Frequencies (high, medium, low)
ExpertCare syndrome definition
Syndrome definitions are logical expressions in terms of symptoms.
Examples:Definition of Appendicitis :“right iliac fossa pain” AND “abdominal pain” AND NOT “appendix operation”Definition of Massive Obesity :“intense weight increase” OR “intense body fat increase”
ExpertCare size in numbers
Rules 3209Symptoms 2383Syndromes 673Other 157
Rules account for 50% of size, but 80% of complexity and 90% of costs.They also hinder software evolution.
Target
Our main metrics is the amount of questions:
Red (Emergency): 3 or 4 questions Yellow (Urgency): 4 or 5 questions Green: around 6, but may reach 12
Solution approach
Automated strategy Dynamic interrogatory Navigation and gathering of information
from the knowledge base Adaptation to session status
Framework for session and strategies Virtual lab as testbed
Concept representation
Logical expressions for definitions
Expression
value: aContextsatisfiersacceptVisitor: anExpressionVisitor
Constant
value: aContextsatisfiersacceptVisitor: anExpressionVisitorbe: aBoolean
falsetruevalue: aBoolean
Variable
value: aContextsatisfiersacceptVisitor: anExpressionVisitorname: aString
named: aString
Conjunction
value: aContextsatisfiersacceptVisitor: anExpressionVisitoroperator
Comparison
value: aContextsatisfiersacceptVisitor: anExpressionVisitor
Disjunction
value: aContextsatisfiersacceptVisitor: anExpressionVisitoroperator
BinaryExpression
value: aContextsatisfiersacceptVisitor: anExpressionVisitorleft: anExpressionright: anExpressionoperator
of: anExpression and: otherExpression
Negation
value: aContextsatisfiersacceptVisitor: anExpressionVisitorexpression: anExpressionoperator
of: anExpression
Session, Diagnoses and other
Automation AnswerProvider simulates a
patient/caller Strategy guides the interrogatory,
suggesting #nextQuestionFor: aCallSession
SUnit tests run through all syndromes using different strategies
StatisticalCollector gathers and caches information from the knowledge base
Grouping and statistics
Grouping and statisticsSeverity red - Syndromes by system
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15
Severity green - Syndromes by system
0 10 20 30 40 50 60
Severity yellow - Syndromes by system
0 5 10 15 20 25 30
Grouping and statistics
Step-by-step example This is a typical
red syndrome. According to the
definition, AnswerProvider can choose among 8 pairs of symptoms (2x4).
Each one is called a subsyndrome
Diabetic Ketoacidosis
System: MetabolicFrequency: LowSeverity: RedDefinition: (Diabetes OR
History of diabetes) AND (Unconsciousness OR Confusion OR Ketonic breath OR Dyspnea)
Step-by-step example
Choosing clues: Diabetes Systems pregnancy and metabolic Confusion Associated to 9 different systems Let’s choose Diabetes as a clue, and try to
establish the presence of Confusion, in order to make a Diabetic Ketoacidosis diagnosis.
Diabetic Ketoacidosis_2Definition: Diabetes AND Confusion
Example - Step 1
Choosing symptoms to ask:We should try to discern the system from among these two
Diabetes
772 syndromes51 systems
2 systems 18 syndromes
positive evidence
6 syndromes2 systems
Example - Step 1
Choosing symptoms to ask:Using information from the knowledge base and some abductive reasoning, we have 9 candidates left.We choose symptom pregnancy, in order to confirm or discard system pregnancy.
System pregnancy6 syndromes31 symptoms
System metabolic13 syndromes48 symptoms
System pregnancy1 syndrome1 symptom
Diabetes
System metabolic5 syndromes8 symptoms
Example - Step 2
Now we “know” that only one system has chances left.
DiabetesNot pregnancy
772 syndromes51 systems
1 system5 syndromes
positive evidence
Example - Step 2
Now we try to discern severity, first trying to decide whether it is red or yellow.Using information from the knowledge base and some abductive reasoning, we have 8 symptom candidates left.Here is where we need some tool for comparing or choosing among them. For instance, we could ask for symptom dyspnea.
System pregnancy6 syndromes31 symptoms
System metabolic13 syndromes48 symptoms
System pregnancy0 syndrome0 symptom
DiabetesNot pregnancy
System metabolic5 syndromes8 symptoms
1 syndrome
3 syndromes
9 syndromes
1 syndrome
2 syndromes
2 syndromes
Example - Step 3
The new information did not reduce syndromes
DiabetesNot pregnancyNot dyspnea
772 syndromes51 systems
1 system5 syndromes
positive evidence
Example - Step 3
We still try to discern severity, because “not dyspnea” only rejected some branches of some syndromes, but did not reduce the total number.Now we have 7 symptom candidates left. This way, we could use up to 7 more questions to “hit” the symptom that the simulated patient has and make a diagnosis.
System metabolic13 syndromes48 symptoms
DiabetesNot pregnancyNot dyspnea
System metabolic5 syndromes8 symptoms
1 syndrome
3 syndromes
9 syndromes
1 syndrome
2 syndromes
2 syndromes
Strategies One family of first attempts, using none or
little information:
SequentialStrategy, RandomStrategy, MoreSatisfiersStrategy, LessSatisfiersStrategy, MiddleSatisfiersStrategy, MoreCriticSeparationStrategy
Second family, attempting to guess the system by different indicators:
GuessSystemByFrequencyStrategy, MoreCorrelationStrategy, LessNegationStrategy, GuessSystemStrategy, GuessSystemUsingPairsStrategy, LessNegationPairStrategy
Results of preliminary strategies
StrategyRed
Average MedianDiag Error
%Sev Error
%
Sequential 739.04 745 19.00 5.58
Random 746.58 732.5 18.05 5.46
LessSatisfiers (*) 726.28 932 18.84 0.00
MiddleSatisfiers 579.28 794 18.05 2.44
MoreSatisfiers 85.29 52 14.23 2.51
MoreCriticSeparation 174.44 35 18.29 7.13
GuessSystemByFrequency 362.37 219 18.18 1.46
GuessSystemBySeverity 92.12 134 75.77 66.15
GuessSystemUsingPairs (**) 9.27 2 40.48 23.81
Strategies - Support
We coined the notion of support
Intuitively, it is a numeric representation of the degree of likelihood of a given set of syndromes in the current session.
Calculation is straigthforward. Syndromes with full diagnoses add a large positive
value. Syndromes with disproved diagnoses add a large
negative value. For the rest, symptoms confirmed add positive value
and symptoms negated add negative value. Finally we perform a normalization.
SupportSeparationStrategy
The third family of strategies is based on support.
Most promising results 15 different strategies Hierarchy 7 levels deep Every level evolving from the
previous one SUCCESS according to target
Results of support strategies
StrategyRed
Average Median Diag Error % Sev Error %
SupportSeparation 6.09 3 15.32 5.58
SupportSeparationWithImplication 4.62 2 18.29 7.13
SupportSeparationImplicationTracking 3.98 2 15.32 5.58
SupportSeparationImplicationTrackingClosing 2.95 1 16.86 7.13
SupportMainSyndrome 3.96 2 15.32 5.58
SupportMainSyndromeScoring 5.41 3 15.20 5.70
SupportOnlyPositive 3.34 2 15.20 5.11
SupportOnlyPositiveClosing 2.40 1 16.86 6.77
SupportOnlyPositiveCandidatesClosing 1.85 1 16.50 6.41
SupportOnlyPositiveClosingDifSev 2.12 1 20.55 6.29
SupportOnlyPositiveStrictClosing 2.13 1 16.86 6.77
SupportLessMissingScore 0.22 0 57.84 39.55
SupportMoreCoincidences 1.65 1 16.50 6.41
SupportMoreCoincidencesPassThru_1 1.65 1 16.50 6.41
SupportMoreCoincidencesPassThru_2 1.65 1 16.50 6.41
SupportMoreCoincidencesPassThru_3 1.96 1 15.32 5.23
SupportMoreCoincidencesPassThru_4 2.12 1 15.20 5.10
Results of support strategies
StrategyYellow
Average Median Diag Error % Sev Error %SupportSeparation 11.75 6 9.61 3.91SupportSeparationWithImplication 8.53 6 16.76 9.72SupportSeparationImplicationTracking 7.31 5 10.33 4.16SupportSeparationImplicationTrackingClosing 5.94 4 16.25 10.39SupportMainSyndrome 7.18 5 10.95 4.47SupportMainSyndromeScoring 7.29 6 10.80 4.88SupportOnlyPositive 5.11 4 10.69 4.27SupportOnlyPositiveClosing 4.24 3 12.08 5.91SupportOnlyPositiveCandidatesClosing 3.15 2 11.21 5.19SupportOnlyPositiveClosingDifSev 4.65 3 12.90 6.17SupportOnlyPositiveStrictClosing 3.61 2 11.52 5.40SupportLessMissingScore 0.48 0 55.89 35.63SupportMoreCoincidences 2.63 2 11.47 5.45SupportMoreCoincidencesPassThru_1 2.63 2 11.47 5.45SupportMoreCoincidencesPassThru_2 2.62 2 11.47 5.45SupportMoreCoincidencesPassThru_3 3.04 2 11.10 5.09SupportMoreCoincidencesPassThru_4 3.07 2 10.00 4.99
Results of support strategies
StrategyGreen
Average MedianDiag Error
% Sev Error %SupportSeparation 25.43 8 18.50 9.97SupportSeparationWithImplication 17.67 8 23.55 10.66SupportSeparationImplicationTracking 16.54 6 18.11 9.97SupportSeparationImplicationTrackingClosing 5.79 1 19.01 16.43SupportMainSyndrome 16.46 6 18.20 9.97SupportMainSyndromeScoring 11.92 6 19.65 10.00SupportOnlyPositive 13.44 5 18.08 10.00SupportOnlyPositiveClosing 5.18 1 18.34 10.48SupportOnlyPositiveCandidatesClosing 2.82 1 18.41 14.48SupportOnlyPositiveClosingDifSev 5.23 1 18.11 9.70SupportOnlyPositiveStrictClosing 3.39 1 18.14 13.82SupportLessMissingScore 0.43 0 38.97 13.91SupportMoreCoincidences 2.12 1 18.29 14.12SupportMoreCoincidencesPassThru_1 2.12 1 18.29 14.12SupportMoreCoincidencesPassThru_2 2.11 1 18.29 14.06SupportMoreCoincidencesPassThru_3 2.24 1 17.66 10.00SupportMoreCoincidencesPassThru_4 2.27 1 17.39 9.19
Conclusions and remarks
It was great doing this work because:
Enhancing the ExpertCare application could have a direct impact on the population’s health.
Automated strategies allow ExpertCare architecture to be used in other domains.
We applied a scientific research approach and techniques to this “real world” software problem.
We learned from Artificial Intelligence, Object-Oriented Programming and Medicine in an interdisciplinary work.
Conclusions and remarksSmalltalk proved to be an adequate tool because:
Representation of the knowledge base was almost trivial.
Building a virtual lab for essays and benchmarks was very easy.
Additional tools for exploring the knowledge base and studying it were easy to implement.
There were no barriers for implementing and testing several strategies with diverse heuristics.
It was easy to get feedback and to debug troublesome cases, in order to enhance and refine strategies
Future work (technical) A visual tool for representing the
session. It should be some navigational metaphor.
The tool could be enhanced for tracing during simulation runs.
More tools for developers to understand and interact with strategy/session.
More tools for better comparative benchmarking.
Future work (domain model)
Integrate with ExpertCare Incorporate exceptions and special
rules Test with real samples Try some adaptation to other
knowledge bases
The End
Questions?
Top Related