A Structured Aproach to Knowledge Elicitation Vennix

Jac A.M. Vennix, Jan W. Gubbels. Doeke Post, and Henk J. Poppen

Policy-oriented modeling demands client involvement in the model-building process. This poses a knowledge elicitation problem, i.e., how to obtain the necessary knowledge from a group of p e e ple. ’kaditionally. interviews and freely

have been employed most frequently for this purpose. In this article, we present an approach to knowledge elicitation that employs a combination of different techniques to arrive at a conceptual model of a regional health care system. The adopted approach is suf6- ciently general to be employed in other model development processes as well.

interacting groups

Jac A.M. Vennix is an associate professor in the department of gamma-informatics at the University of Utrecht. He has completed Ph.D. research on the design and evaluation of the impact of a computer-based learning environment on participants’ mental models. He has coauthored several articles in this field as well as a book on the uses of gaming and computer simulation as policy support tools. Address: De- partment of Gamma- Informatics, Uni- versity of Utrecht. Netherlands.

System dynamicists have long recognized the importance of involving the client in the process of model building (see Roberts 1978; Meadows, Richardson, and Bruckmann 1982; Meadows and Robinson 1985). Various effective procedures have been designed to work with client groups. The approach most commonly used is what Richardson et al. (1989) call the informal consulting approach, i.e., a modeler working with a small, freely interacting group of people from a client’s organization (Roberts 1978; Weil 1980; Wallace and Sancar 1988). Another well-known approach is the use of reference groups (Randers 1977; Stenberg 1980; Weil 1980). Recently, new software tools like STELLA have been employed to work with client groups (Richmond 1987; Morecroft, Lane, and Viita 1989; Senge 1990; Richardson and Senge 1989). In all these approaches the most commonly employed techniques for eliciting knowledge are interviews and group discussions. As Richardson et al. (1989) and Vennix (1990) show, there are a variety of knowledge elicitation techniques ranging from content analysis of written interview materials to structured workshops.

In this article we discuss a structured approach to knowledge elicitation, which involves a combination of different techniques to arrive at a conceptual model of the Dutch health care system. The techniques are chosen to fit the various tasks in knowledge elicitation and to account for differences between individual and group tasks. The procedure allows for a large number of participants, which is important because in large corporate and public policy organizations the needed information is often scattered among many different people (Stenberg 1980).

The approach consists of three stages. The first stage entails the development of a preliminary conceptual model by the project group, based on the relevant literature and on general insights. This first stage is necessary because, in our opinion, it is easier to confront people with a preliminary model in hand than to approach them for the first time unprepared. In the second and third stages, the actual consultation of experts takes place. A method frequently used when consulting a panel of experts is the Delphi method (Linstone and Turoff 1975; Dunn 1981; Nutt 1984; Sancar and Ozgul 1982). The Delphi method uses a series of mailed questionnaires. The first questionnaire starts the process. Subsequent questionnaires provide feedback from previous ones, often to promote consensus within the panel. A normal questionnaire format, however, does not allow the respondent to deal with complex interrelations between variables. Hence, in the second stage, we decided to use a questionnaire followed by a “workbook” (Underwood 1984; Duke, Gem, and Underwood 1990). In the workbook, we could more easily deal with complex submodels than in the questionnaire. Moreover, the traditional Delphi method is not intended for use in sit- uations that require direct interaction and confrontation between experts. To remove this disadvantage we employed a structured workshop in the third stage, which made possible direct confrontation between experts’ opinions. The two first cycles have a focusing function: they eliminate those elements from discussion in the workshop on which there is a great deal of consensus in the panel. The sequential stages in the modeling project are summarized in Figure 1.

System LlynMlin Review 6 (no. 2. Summer 1990): 194-208. ISSN 0883-7066. Q 1990 by the System Dynamics Society.

Vennix et al.: Knowledge Elicitation in Model Building 195

Jan W. Gubbels is a statistician and sci- entific programmer in the department of research methodology at the Catholic University of Nij- megen. He has coauthored several articles and books in the field of health care research and methodology. His research activities are primarily focused on policy-oriented computer simulation modeling.

Doeke Post was active as a general practitioner born 1967 until 1982. He holds a Ph.D. degree from the University of Amsterdam. Since 1982 he has worked as a medical advi- sor at the regional Health Care Insur- ance Organization in Zwolle. His research activities and publi- cations concentrate on policy making in Dutch health care.

He& J. Poppen is a medical sociologist. He is currently working as head of the statistical department at the regional Health Care Insurance Organiza- tion in Zwolle. His main professional interest is the use of computers and software in data collection and analysis in health care.

Fig. 1. Phases in the model-building project

The above-mentioned stages (2,3a, and 4a in Figure 1) produce a final conceptual model, which, in turn, has to be formalized, tested, and validated before policy analyses can be carried out by computer. The last step in the project involves the design of a computer-based learning environment that can be used by participant groups to conduct their own policy experiments and discussions.

Since the model is designed to provide insight into the development of health care costs, we also decided to make a list of potential policy options aimed at reducing these costs. Here we also used the Delphi method and a workshop to make an in- ventory of policy options and to develop criteria that could be considered important in selecting policy options. Discussions in the workshop added valuable information with regard to implementation of policy options in the simulation model.

In this article, we concentrate on the unshaded stages in Figure I, After discussing the main characteristics of the Dutch health care system, we first focus on the prob lem definition (stage 1). Next we discuss the preliminary model (stage 2). the Delphi questionnaire and workbook (stage 3a), and the workshop with regard to model development (stage 4a). In the two final sections, we present the key results with regard to the final conceptual model (stage 5) and prospects for future research.

198 System Dynamics Review Volume 6 Number 2 Summer 1990

The Dutch Care Sy8blIl

The Dutch health care system lies somewhere between a fully nationalized system like the British one and a market-based system like the one in the United States. Two important decisions have determined the character of Dutch health care. The first is the implementation of mandatory insurance. Individuals with an annual income below f 50,000 are insured mandatorily against the costs of medical health care; consequently 60 percent of the Dutch population is automatically insured against health care costs. There are two laws regulating this mandatory insurance: the Law on Health Care Insurance and the Law on Special Medical Care costs. The Law on Health Care Insurance covers the normal health care costs; the premium is paid by employers and employees on a fifty-fifty basis and amounts to about 10 percent of an employee’s gross income. The Law on Special Medical Care covers the costs arising from long-term illnesses; the premium is paid by employers. In addition to these two laws, there also exists private insurance for the remaining 40 percent of the Dutch population.

The second important decision, dating to 1974, is the division of the health care system into echelons. In the first echelon we find practical and community nurses, health care workers, and family physicians. Family physicians usually work indi- vidually from their own homes, sometimes in community centers, and only rarely in health care centers staffed by general practitioners, community nurses, or physio- therapists. None of the family physicians work in hospitals. In the second echelon one finds the medical specialists and the hospitals. Medical specialists work in both hospital settings and in facilities for long-term patient care. Patients have no direct access to the second echelon for consultation and treatment except in case of emer- gencies (e.g., car accidents, heart attacks). Patients must be referred to specialists by the family physician. The general practitioner and the specialist have strictly sepa- rated responsibilities.

Since 1974 several attempts have been made to reduce health care costs. The most conspicuous was the attempt to strengthen the first echelon at the cost of the second. Thus far, however, this has not proved to be a very successful initiative. In the United States the so-called Health Maintenance Organizations (HMOs), based on a new payment and delivery system, seem to be more successful. Cost reductions of 10 to 40 percent have been reported, due in part to the prevention of prolonged hospitalization. Within a mandatory insurance system, such as the Dutch one, however, numerous operational constraints make it di5cult for the health care system to function like an HMO. Some of these constraints are the following (van de Ven 1987):

Financially speaking, health insurance companies do not operate independently. The premiums paid by employers and employees are collected by one central public fund. This in turn pays the costs of the insurance companies. Insurance companies cannot refuse contracts with a hospital, specialist, physician, and so on. The contents of contracts are determined by the federal government. Insurance companies are not allowed to operate simultaneously as an insurance company and a caretaker.

Since 1985 there has been considerable discussion in the Netherlands with respect


to the possibilities and limitations of reshaping insurance companies into HMOs. At present, it does not appear that the situation will change in the immediate future. Insurance companies are attempting to prepare themselves for this change in the long- term future, however, by strengthening their policy development and implementation capacities.

RoMem definition and project organization

Our client is the regional Health Care Insurance Organization (HCIO) in Zwolle. The model is designed for the specific region covered by the Zwolle HCIO. 'Itvo HCIO planners (Post and Poppen) joined the two system dynamics modelers (Vennix and Gubbels) on the project team. In addition, five HCIO staff members assessed the preliminary model, the design of the questionnaire, the workbook, and the workshops.

The background of our study is formed by the problems relating to increasing costs of public health care in the Netherlands. Total health care costs increased fcom about 6 percent of the net national product in 1968 to about 10 percent in 1985 (approximately f 35 billion). There are several causes for this gradual but persistent increase. Most of the causes that have been identified, however, are exogenous influences, e.g., increasing wage rates and energy prices (see Griinwald 1987). Up to now, very little attention has been devoted to the internal dynamics of the health care system. De- velopment of a system dynamics model could help planners address the following questions:

What factors have been responsible for the increase in health care costs in the past? How will health care costs develop in the future? What would be the effects of several policy options in reducing costs?

Because the client was not readily convinced that the resulting conceptual model could be easily formalized and quantified, we defined two additional goals of the project. The first was to provide systematic insight into available knowledge (as well as lack of knowledge) about important processes in Dutch health care. In this case, the model would be used to direct future research efforts on processes in the health care system. Hence, the client was very interested in designing a conceptual model that would provide a comprehensive overview of the most important processes in health care. The second goal was to employ the conceptual model as a tool in discussions with general practitioners. We pointed out to the Zwolle HCIO that even if we were not able to develop a computer model, the project might prove to be very valuable because the development of a conceptual model provides the opportunity to conduct qualitative analyses, which are considered very insightful, according to several system dynamicists (Wolstenholme 1982; Wolstenholme and Coyle 1983; Eden 1985; Meadows 1989).

To facilitate the process of consulting a large number of experts, we concluded that as a first step it would be useful to develop a preliminary conceptual model and have

1- System Dynamics Review Volume 6 Number 2 Summer 1990

I 1

Fig. 2. Patients' flow in the health care system this criticized by a smaller group of experts. We first constructed the flow diagram

shown in Figure 2. People with health complaints initially consult their general practitioners (G.P.s),

who decide whether to refer patients to a medical specialist, to discharge them, or to perform the required medical treatment themselves. Medical specialists, in turn, decide whether patients should be admitted into the hospital and when they will be discharged. The portion of the system shown in Figure 2 accounts for approximately 75 percent of health care costs in the region.

In this patients' flow model we subsequently included the decisions of the patients, general practitioners, and medical specialists, producing a hybrid diagram, as suggested by Richardson and Pugh (1981) (see Figure 3). Patients decide whether to consult a doctor. General practitioners decide to discharge the patients, prescribe drugs, refer patients to a specialist, or request patients to return (order back). Medical specialists decide whether to re-examine patients, apply medical surgery (medical transactions), prescribe drugs, or have patients admitted into the hospital. The decisions have a volume and a cost component, for instance, the number of prescriptions by a general practitioner or a specialist (volume) and the price of the drugs prescribed (cost). The next step is to determine factors that influence these decisions and actions. Based on the literature plus insights from the project group, we decided to incorporate a number of factors into the model. Each of these influenced one or more decisions, as can be seen in Figure 3. For instance, the workload of the general practioner (workload G.P.) is said to affect the number of prescriptions. Further information on this initial conceptual model appears in a Dutch medical journal (Vennix, Gubbels, and Post 1986a; 1986b).

There are two major issues in the design of any Delphi study: selection of experts and questionnaire design. The selection issue is how to ensure that respondents with different viewpoints will be included. Our participants were selected from three fields: the actual care system (general practitioners, medical specialists, or patients), the policy-making field (e.g.. planning institutions), and the social behavioral research field [e.g., university health care departments). We identified a list of some 60 respondents, fairly well distributed over the three fields. To avoid the problem of low response in mailed questionnaires, we enclosed with the questionnaire an abstract


Fig. 3. Preliminary conceptual model

of the above-mentioned articles as well as an article on the preliminary model itself, and we pointed out to the respondents that we needed their expert opinion in order to be able to improve the preliminary model. The response rate exceeded 95 percent, which is very high for a mailed questionnaire.

The second Delphi issue is formulating the questions. For the sake of simplicity, the questionnaire contained questions on binary relations, i.e., relations between two variables. We subdivided the questionnaire into a number of sections, each dealing with one of the decisions mentioned in the previous section. One section, for instance, focused on the decision of general practitioners to prescribe drugs. This was then considered to be the dependent variable. In each section, a number of statements were presented describing a relation between this dependent variable and some independent variable. For instance, in the section “prescriptions by general practitioners,” one of the statements is “The heavier the workload of a general practitioner, the higher the number of this practitioner’s prescriptions.”

Participants were asked to state whether they “agreed fully,” “agreed partially,” or “disagreed fully” with the statement. However, we were also interested in uncovering causal arguments from the participants’ mental models, particularly those that were not included in our preliminary model. To extract these causal arguments, we asked the respondent after each statement to indicate why one did or did not agree with the statement. Content analysis of the answers to the “why” questions revealed new


concepts that might be included in a causal relation, thereby clarifying it. This is typical for those who agree with a statement. An example: Our statement “Older patients are referred more often to a medical specialist than younger patients” (a typical statistically established relation) produced “Agree, because older people often have a more complex pathology, which impedes a correct diagnosis; hence a specialist’s opinion is needed.”

Other interesting conclusions hom the “why” questions were related to the concepts themselves. For instance, with respect to the statement on the relation between workload and the number of prescriptions, approximately half the respondents agreed and the other half did not. This is, of course, quite confusing. Careful analysis of the arguments, however, revealed that the two groups did not hold the same kind of concept. One group obviously had the temporary rush during consulting hours in mind (caused by an epidemic of influenza, for instance), while the other group presented arguments related to what might be called structural workload. Each section in the questionnaire, focusing on one dependent variable, thus contained a number of these statements together with “why” questions. In addition, at the end of each section we asked respondents to add variables, not yet mentioned, that they perceived to affect the dependent variable. Here, too, some respondents came up with interesting new variables. For instance, the effect of advertising by the pharmaceutical industries, and the demands of patients regarding the manner in which general practitioners prescribe drugs. On the other hand, this generated a host of factors that might influence some dependent variable. Thus, we decided to end each section with a question in which the respondent was asked to name the three independent variables considered to have the greatest effect on the dependent variable. Answers to this question helped us decide which factors to incorporate into the final conceptual model.

In the second cycle of the Delphi we switched to a workbook based on the results of the questionnaire. The workbook enabled us to focus on sets of interrelated variables instead of only binary relations. The workboDk was also used to prepare participants for the workshop. We selected a subset of 18 respondents (from the original 60) who had presented us with the most detailed comments and arguments in the Del- phi questionnaires. These respondents were divided into two workshops, each with 9 participants. The workshops could only be held for the first echelon (general praction- ers), since a number of medical specialists (the second echelon) refused to cooperate because of a conflict between their interest group and the federal government.’

The workbook contained four submodels of the first echelon. Each of these incorpo- rated one central dependent variable: consultation by patients, prescription of drugs, referral, and order-backs by general practitioners. These submodels were designed by the project group, based on the preliminary model and the results of the questionnaire. The design of these submodels was not quite straightforward. Although the questionnaire did provide insights into important factors affecting certain processes, it did not generate variables that could in turn explain these factors. Hence, we had to


Fig. 4. Conceptual workbook submodel on prescriptions

design submodels ourselves based on the outcome of the questionnaires, knowledge in the project group, and available research literature.

These submodels were gradually introduced into the workbook. This was done by first describing the most important results of the questionnaire with respect to the submodel. More specifically, starting with the dependent variable (e.g., number of prescriptions by general practitioners), we first presented a number of statements with regard to variables directly affecting the dependent variable. These were derived from the respondents’ answers pertaining to the factors they considered most influ- ential. For the number of prescriptions, for example, these were the quality of the discussion between G.P. and patient during consultation, the G.P.’s view of his or her job, the demands of patients for medication, and the G.P.’s uncertainty about the diagnosis. Subsequently, new explanatory variables were added to one of these explaining variables (e.g., quality of discussion in consultation). The verbal description was summarized using a causal diagram. Next, we took a second explaining variable and discussed it in much the same way. The verbal descriptions were again summarized in a causal diagram and added to the previous one. Gradually, a complex network of interrelated variables was constructed, aimed at explaining the dependent variable in question. At regular intervals the respondent was asked to comment on our arguments and to indicate disagreements in the final diagram (see Figure 4). After having completed one submodel, the respondent continued with the next, following the same procedure. The completed workbooks were sent to us one week before the workshop.


ThemockldsvdopmantWorkrhop

The workshop encourages participants to discuss differences of opinion on the structure of the conceptual model. The workshop design was based on our experiences and the guidelines described by Hart et al. (1985), Mason and Mitroff (1981), and Duke (1980). There were 9 participants at each of the two four-hour workshops. The 18 subjects were selected from the 60 original participants in the first Delphi cycle based on the quality and elaborateness of their comments in the questionnaires. In both workshops we formed three groups of three persons each to allow in-depth discussions of different submodels during the workshop (see Eden 1985; Huxham et al. 1988). Each of the three subgroups discussed one of the submodels. From the four submodels in the workbook we selected the three that had received most criticism in the workbooks. The entire four-hour session was broken down as follows:

Introduction (half hour) Subgroup activities (one hour) Break (half hour) Plenary session (one and one-half hours) Evaluation (half hour)

Work in each subgroup was assisted by a person from the project team. We used a few aids to structure the discussions in the subgroups. First, each member of the group was assigned a role. For instance, one person took notes and presented the results in the plenary session; another person was responsible for time management. Second, we copied diagrams from the workbook and, by using three different colors, indicated which ofthe three persons had commented on which parts of the submodel in his or her workbook. The colored parts of the submodels had to be discussed first during the subgroup meeting. Participants were asked to make changes in the diagrams according to the results of their discussions. In order to accomplish this task, they were provided with the relevant diagram of the workbook, which was put on the table in the subgroup room. During the discussions they employed this diagram as a kind of scratch-paper to make changes according to their discussions, by removing or adding variables or relations between variables. For instance, the subgroup that had been working on the prescription submodel (shown in Figure 4) had been tossing around the idea that a G.P. with a high prescription rate somehow teaches a patient to consult the G.P. for almost every complaint. With the aid of the modeling team this idea was translated into feedback processes that were not originally included in the workbook. The two positive feedback loops are shown in Figure 5. The first runs from “number of prescriptions” via “legitimization of complaints” (a new variable) to “number of consultations,” which then feeds back to “number of prescriptions” via “workload G.P.” and “use of drugs to terminate consult.” The second loop feeds back from “workload G.P.,” which leads to “tiredness of G.P.,” decreasing “quality of consult discussion,” which causes again a higher number of prescriptions. The participants found these feedback processes interesting and in particular considered the first loop highly plausible and important in explaining a G.P.’s structural workload.


Fig. 5. Example of a new variable and additional feedback loops

colt3dh(bm

in G.P.

of infomution provided to GP.

susceptibility of C.P. \ for patient's mplatnts TRnporqyM

d ODlrarlhng hour

After the discussions were completed, one person was responsible for integrat- ing the final changes into a large diagram to be used in the plenary session. The spokesperson of each subgroup was given ten minutes to explain the results of the discussions and the changes made in the submodel by the group. During the next 20 minutes the other subgroups were permitted to ask questions and comment on the results. The total time for plenary discussion was thus one and one-half hours.

Participants in both the small groups and the plenary sessions were quite involved and reached consensus on many issues. However, it also became clear that several processes are poorly understood. Here the knowledge elicitation process was arrested at the point where there were only vague conjectures. This was the case, for instance, with regard to the number of order-backs by a general practitioner. Lack of knowledge on G.P. order-backs results from lack of interest by the insurance companies (as far as mandatory insurance is concerned, a change in the number of order-backs would not affect the number of payments to G.P.s).

The final half hour was devoted to a short evaluation of the workshop and the modeling project. The participants considered the session interesting and the modeling exercise valuable. There was, however, some skepticism about whether in fact the model could be formalized and quantified. We explained the various potential uses of the model, for instance, formalizing parts of the model and designing a computer-based learning environment in which participants can conduct their own policy experiments. All participants wanted to be involved in testing the design of this computer-based learning environment. All discussions during the workshops were recorded on tape,.and minutes were made.


The approach we have discussed aims at the development of a conceptual model. Usually the results of model runs are considered important with respect to policy making. However, building a conceptual model of a policy problem also generates certain, often unexpected, results. In our case, several tangible results materialized, which are based on this stage of the model-building process.

Changes in the conceptual model

There were three classes of changes from the Delphi cycles and workshops: changes in the number of variables, changes in concepts, and changes in relations. These changes were limited to the factors underlying processes in the health care system. (There was almost no discussion of the patients’ flow model itself, which contained the most important levels and rates.)

As might be expected, the number of variables included in the conceptual model was increased considerably during the Delphi and workshop stages. Almost no variables were removed during the process. Our preliminary model contains about 40 variables, and the final model contains more than 80.

Changes in concepts usually involved detailing a concept originally introduced by us (e.g., more detail on the G.P.’s workload). Another example concerns the uncertainty of general practitioners. Project participants distinguished between uncertainty about diagnosis, about the progression of a disease, and about the effect of the therapy. Each uncertainty leads to different effects on the health care system. Uncertainty with regard to diagnosis affects the number of referrals; uncertainty about disease progression affects the number of prescriptions; and uncertainty about the therapy affects the number of order-backs by a G.P.

Most changes in relations involved adding relations between variables. The result of the Delphi cycle, for example, was an elaboration of several binary relations in chains of relations containing more than two concepts. Removing relations was frequently accompanied by specifying other causal chains leading from one concept to another. Finally, feedback processes were added into the conceptual model by the participants.

Reframing the policy problem and its solutions

Another area in which we perceived results from our model study is with regard to the perception of the problem. As several authors have pointed out, impact of computer modeling is often indirect, affecting one’s mental model and the way of thinking about a policy problem (see Meadows and Robinson 1985; Vennix 1990). This is also one of the most clear and perhaps most profound results of our study. Before starting the model-building process, people in the Zwolle HCIO were quite convinced that in order to reduce health care costs one would have to decrease the number of referrals by G.P.s to medical specialists in the second echelon. Transactions by medical specialists ace more expensive than those by G.P.s, particularly if patients are admitted to a hospital. Hence, the most effective way to cut health care costs


seemed to be to reduce the number of referrals. However, the model-building process revealed two important feedback processes counteracting the effects of a reduced number of referrals. The first is the reaction of medical specialists to a decrease in inflow of patients. This might induce an intensification of the treatment of patients currently treated by specialists, which in turn increases costs. The second process involves an increase in the workload of G.P.s, probably resulting in more prescriptions and, in the long run, more referrals. We are currently discussing whether to continue attention on reducing the number of referrals or to explore other policy options. We are also discussing how to avoid the effects of the feedback processes. Formalizing and quantifying the conceptual model, Eurrently underway, will further clarify this issue, because more exact statements can be made with regard to the net effects of these processes.

New modeling projects

To conclude the conceptual model-building project, we organized a conference, at- tended by some 200 people from various health care organizations (e.g., general practitioners, researchers in the field of future scenario studies in health care, financial planners, hospital managers, modelers). We had a number of presentations and a panel discussion on the project. At the end of the conference we asked participants to fill out a questionnaire. 'Itvo-thirds of the respondents considered the subject interesting to very interesting on a five-point scale. Three-fourths felt that the conceptual model provides insight into the complexity of the Dutch health care system. About 60 percent were interested in talking to the project group further about the model study. And finally, about 60 percent were interested in assisting on model improve- ments. As a result, we are now formulating two new projects in the health care field. One project will focus on health care for the elderly. The second will have the future organization of home care as its subject. W e plan to follow the same procedure in the development of a conceptual model as was described in this article.

Further research

The discussions revealed that various parts of the health care system are poorly understood. For instance, very little is known about the phenomenon of order-backs by general practitioners. Consequently, a research project has been started specifically aimed at filling in the gaps in our knowledge of G.P. order-backs.

Summaryand discu.sion

In this article, we have concentrated on a structured approach to knowledge elicitation for conceptual model building. We suggested that a three-step approach, using different kinds of data collection methods, is an appropriate way to structure the knowledge elicitation process. A Delphi questionnaire elicited comments from a number of experts (60) on binary relations of a preliminary model designed by


the project group. Next, a workbook was employed to have a subset (18) of experts criticize a number of more complex submodels. Finally, in a structured workshop, participants were put in a position to discuss these submodels in more detail.

In sum, the project has taken about two years. However, the project group did not work full-time on the model. We estimate that, given our experience with this approach, it should be possible to finish a conceptual modeling phase using this approach in about three to six months. Although at first sight the procedure might look labor-intensive, this is only partly true. For instance, one place where time is saved is in documentation. The procedure is partly self-documenting, and the text of the workbooks. for instance, has been used to make a verbal description of the final conceptual model. The project was well received by the Zwolle HCIO, which is supporting a book on the project, our continuing effort to formalize the model, and the design of a computer-based learning environment.

At the moment, we have already successfully formalized and quantified part of the model, i.e.. the patients’ flow model and the costs involved. Our efforts are now aimed at including a number of influencing factors on this patients’ flow model. The final phase of the project, which has started recently, is the computer-based learning environment (see Figure 1) with which policymakers can conduct simulation experiments themselves (see Senge 1990; Kim 1989; Graham et al. 1989; Morecroft 1988; Vennix and Geurts 1987). This also includes challenging and changing the model’s assumptions.

The simulation model will be used to support groups of policymakers in regional health care organizations in designing policy scenarios that will aid in health care policy formation. Several organizations will be invited to participate in this learning laboratory (see Kim 1989; Richardson and Senge 1989; Senge 1990). Some of the sessions will be devoted to training participants in model-oriented thinking in health care.

1. The conflict was about the income level of medical specialists. The fact that they did not participate shows up in the final conceptual model, which is fairly elaborate when it comes to the first echelon but much less so for the second. We hope to be able to persuade this group to participate in this project in the future.

Duke, R. D. 1980. A Paradigm for Game Design. In Principles and Pmctices of Garning- Simulation, ed. C. S . Greenblatt and R. D. Duke, 63-72. London: Sage.

Duke, R. D., J.L.A. Geurts, and S. E. Underwood. 1990. Gaming/Simulation as a Tool for Strategic Policy Development: A Description of a Project on Pharmaceutical R&D Planning. Technological Forecasting and Social Change. Forthcoming.

Dunn, W. 1981. Public Policy Analysis: An Introduction. Englewood Cliffs, N. J.: Prentice- Hall.

Eden, C. 1985. Perish the Thought. Journal of the Opemtions Research Society 36 (9): 809-819.


Graham, A. K., P. M. Senge, J. D. Sterman, and J.D.W. Morecroft. 1989. Computer-Based Case Studies in Management Education and Research. In Proceedings of the 1989 Znter- national System Dynamics Conference, Stuttgart, 317-326.

Griinwald, C. A. 1987. Beheersing van de gezondheidszorg. The Hague: WGA. Hart, S. L., M. Boroush, G. Enk, and W. Hornick. 1985. Managing Complexity Through

Consensus Mapping: Technology for the Structuring of Group Decisions. Academy of Management Review 10: 587-600.

Huxham, C., C. Eden, S. Cropper, and J. Bryant. 1988. Facilitating Facilitators: A Story about Group Decision Making. Opemtions Research Insight: 13-1 7.

Kim, D. H. 1989. Learning Laboratories: Designing a Reflective Learning Environment. in Proceedings of the 1989 International System Dynamics Conference, Stuttgart, 327-334.

Linstone, H., and M. Turoff. 1975. The Delphi Method: Techniques and Applications. New York: Wiley.

Mason, 0. M., and I. I. Mitroff. 1981. Challenging Stmtegic Planning Assumptions: Theov, Cases, and Techniques. New York: Wiley.

Meadows, D. H. 1989. System Dynamics Meets the Press. System Dynamics Review 5 (1): 69-80.

Meadows, D. H., J. Richardson, and G. Bruckmann. 1982. Groping in the Dark: The First Decade of Global Modeling. Chichester: Wiley.

Meadows, D. H., and J. M. Robinson. 1985. The Electronic Omcle: Computer Models and Social Decisions. Chichester: Wiley.

Morecroft, J.D.W. 1988. System Dynamics and Microworlds for Policymakers. European Journal of Opemtional Research 35: 301-320.

Morecroft, J.D.W., D. C. Lane, and P. S. Viita. 1989. Modeling a Biotechnology Startup Firm. In Proceedings of the 1989 International System Dynamics Conference, Stuttgart, 157-1 76.

Nutt, P. C. 1984. Planning Methods for Health and Related Organizations. New York: Wiley.

Randers, J. 1977. The Potential in Simulation of Macro Social Processes, or How to Be a Useful Builder of Simulation Models. Oslo: Grupper for Ressursstudier.

Richardson, G. P., and A. L. Pugh In. 1981. Introduction to System Dynamics Modeling with DYNAMO. Cambridge, Mass.: MIT Press.

Richardson, G. P., and P. M. Senge. 1989. Corporate and Statewide Perspectives on the Liability Insurance Crisis. In Proceedings of the 1989 International System Dynamics Conference, Stuttgart, 442457.

Richardson, G. P., J.A.M. Vennix, D. F. Andersen, J. Rohrbaugh, and W. A. Wallace. 1989. Eliciting Group Knowledge for Model Building. In Proceedings of the 1989 International System Dynamics Conference, Stuttgart, 341-357.

Richmond, B. 1987. The Stmtegic Forum. Lyme, N.H.: High Performance Systems. Roberts, E. B. 1978. Strategies for Effective Implementation of Complex Corporate Models.

In Managerial Applications of System Dynamics, ed. E. B. Roberts, 77-85. Cambridge, Mass.: MIT Press.

Sancar, F. H., and B. Ozgul. 1982. Formal Aids to Model Formulation Using Worth As- sessment and Delphi. Dynamica 8: 9&95.

Senge, P. M. 1990. Catalyzing Systems Thinking in Organizations. In Advances in Orga- nization Development, ed. F. Masaryk. Norwood, N. J.: Ablex. Forthcoming.

Stenberg, L. 1980. A Modeling Procedure for Public Policy. In Elements of the System Dynamics Method, ed. J. Randers. Cambridge, Mass.: MIT Press.

Underwood, S. E. 1984. An Evaluation of a Participative Technique for Strategic Planning. Unpublished Paper, University of Michigan, Ann Arbor, MI 48109.

van de Ven, W. 1987. The Key Role of Insurance in a Cost-Effective Health Care System: Towards Regulated Competition in the Dutch Medical Market. Health Policy 7: 253-272.


Vennix, J.A.M. 1990. Mental Models and Computer Models: Design and Evaluation of a Computer-Based Learning Environment for Policy Making. Ph.D. Dissertation, Univer- sity of Nijmegen. Netherlands.

Vennix, J.A.M.. and J.L.A. Geurts. 1987. Communicating insights from Complex Simulation Models: A Gaming Approach. Simulation and Games 18 (3): 321-343.

Vennix, J.A.M., J. W. Gubbels, and D. Post. 1986a. Spreekuurbezoek, verwijzen en opnemen als proces. I: Patienten onder behandeling van de huisarts. Medisch Contact 27: 852- 854.

. 1986b. Spreekuurbezoek, verwijzen en opnemen als proces. 11: Patienten onder behandeling van de specialist en ziekenhuisopnames. Medisch Contact 28: 895-898.

Wallace, S. D., and F. H. Sancar. 1988. An Lntegrative Approach to Water Resource Manage- ment: An Application in Middleton, Wisconsin. In Proceedings of the 1988 International System Dynamics Conference, La Jolla, Calif.

Weil, H. B. 1980. The Evolution of an Approach for Achieving Implemented Results from System Dynamics Projects. In Elements of the System w a m i c s Method, ed. J. Randers. Cambridge, Mass.: MIT Press.

Wolstenholme, E. F. 1982. System Dynamics in Perspective. Journal of the Opemtions Research Society 33: 547-556.

Wolstenholme, E. F., and R. G. Coyle. 1983. The Development of System Dynamics as a Methodology for System Description and Qualitative Analysis. Journal of the Operations Research Society 33: 56!3-581.

A Structured Aproach to Knowledge Elicitation Vennix

Documents

Transcript of A Structured Aproach to Knowledge Elicitation Vennix