Eval Rev 1999 Schuerman 599 618

5/21/2018 Eval Rev 1999 Schuerman 599 618

1/21

http://erx.sagepub.com/Evaluation Review

http://erx.sagepub.com/content/23/6/599Theonline version of this article can be foundat:

DOI: 10.1177/0193841X9902300602

1999 23: 599Eval Rev

John Schuerman, Peter H. Rossi and Stephen BuddeDecisions on Placement and Family Preservation: Agreement and Targeting

Published by:

http://www.sagepublications.com

can be found at:Evaluation ReviewAdditional services and information for

http://erx.sagepub.com/cgi/alertsEmail Alerts:

http://erx.sagepub.com/subscriptionsSubscriptions:

http://www.sagepub.com/journalsReprints.navReprints:

http://www.sagepub.com/journalsPermissions.navPermissions:

http://erx.sagepub.com/content/23/6/599.refs.htmlCitations:

What is This? - Dec 1, 1999Version of Record>>

at Alexandru Ioan Cuza on June 29, 2014erx.sagepub.comDownloaded from at Alexandru Ioan Cuza on June 29, 2014erx.sagepub.comDownloaded from
http://erx.sagepub.com/http://erx.sagepub.com/http://erx.sagepub.com/content/23/6/599http://erx.sagepub.com/content/23/6/599http://erx.sagepub.com/content/23/6/599http://www.sagepublications.com/http://erx.sagepub.com/cgi/alertshttp://erx.sagepub.com/cgi/alertshttp://erx.sagepub.com/subscriptionshttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsReprints.navhttp://www.sagepub.com/journalsPermissions.navhttp://erx.sagepub.com/content/23/6/599.refs.htmlhttp://erx.sagepub.com/content/23/6/599.refs.htmlhttp://online.sagepub.com/site/sphelp/vorhelp.xhtmlhttp://online.sagepub.com/site/sphelp/vorhelp.xhtmlhttp://erx.sagepub.com/content/23/6/599.full.pdfhttp://erx.sagepub.com/http://erx.sagepub.com/http://erx.sagepub.com/http://erx.sagepub.com/http://erx.sagepub.com/http://erx.sagepub.com/http://online.sagepub.com/site/sphelp/vorhelp.xhtmlhttp://erx.sagepub.com/content/23/6/599.full.pdfhttp://erx.sagepub.com/content/23/6/599.refs.htmlhttp://www.sagepub.com/journalsPermissions.navhttp://www.sagepub.com/journalsReprints.navhttp://erx.sagepub.com/subscriptionshttp://erx.sagepub.com/cgi/alertshttp://www.sagepublications.com/http://erx.sagepub.com/content/23/6/599http://erx.sagepub.com/


2/21

EVALUATION REVIEW / DECEMBER 1999Schuerman et a l. / PLACEMENT AND FAMILY PRESERVATION

Vignettes presented to experts and workers in the child welfare field were used to explore the

degree of agreement on decisions to place children in substitute care or to refer them to family

preservation services. The design allowed for investigation of the problem of targeting in family

preservation programs. Findings indicate considerable inconsistency in decisions among

experts and workers, particularly in decisions to refer to family preservation andotherin-home

services. Contrary to the statedintentions of familypreservationservices,a majority of the refer-

rals to these programs do not involve children who would have been placed in the absence of

these programs.

DECISIONSON PLACEMENT

AND FAMILY PRESERVATION

Agreement and Targeting

JOHN SCHUERMAN

PETER H. ROSSI

STEPHEN BUDDEUniversity of Chicago

Perhaps the most important decision in child welfare services is that of

removing an abused or neglected child from his or her home and placing the

child in substitute care, such as a foster home, institution, or other group liv-ing situation. Most often, removal of a child from the home occurs in the

course of an investigation of a report of child maltreatment. Removal occurs

whenthe safetyof the child isin doubt because ofhighrisk offurther abuse or

neglect. In this article, we explore the degree of agreement among child wel-

fare experts and practitioners on decisions to remove children from their

homes and to refer cases to family preservation services (FPS).

In current child welfare practice, it is believed important for childrens

well-being that they be able to grow up in their birth families whenever

599

AUTHORS NOTE: This article is based on data collected by the Understanding Placement

Decisions in Child Welfare project, funded by Grant No. 90CW1081 from the ChildrensBureau,

Administration f or Children, Youth, and Families, Administration for Children and Families,

U.S. Department of Health and Human Services. Rossi and Schuerman served as coprincipalinvestigators and Budde as project director.

EVALUATION REVIEW, Vol. 23 No. 6, December 1999 599-618

1999 Sage Publications, Inc.

at Alexandru Ioan Cuza on June 29, 2014erx.sagepub.comDownloaded from
http://erx.sagepub.com/http://erx.sagepub.com/http://erx.sagepub.com/http://erx.sagepub.com/


3/21

possible, so extensive efforts are made to avoid placing children. Family

preservation programs area major wayin which this is done. These programs

provide intensive, short-term services intended to prevent the placement of

children in substitute care. Services include counseling, treatment for sub-

stance abuse, instruction on child-rearing practices, employment assistance,

and help with concrete needs, such as referrals to income support programs.

Child welfare caseworkers are supposed to refer families to these services

under two conditions: (a) when placement of a child in substitute care is

likely and (b) when it is believed that the child can be left in the birth family

without serious abuse or neglect if intensive services are provided.

The hallmark of an equitable decision-making process is consistency in

dealing with similar cases. Well-functioning child protective systems should

make consistent decisions about cases of abuse and neglect. In similar cases,the decisions made should not vary according to the assigned caseworker.

Disagreement on decisions would also lead to questions as to the extent to

whichdecisionsare basedon systematic analysis and established knowledge.

The initial goal of this study wasto determine the appropriateness of deci-

sions made by workers to place children in substitute care. This aim was

motivated by concern about the validity of assertions by family preservation

advocates that large numbers of children are being unnecessarily removed

from their homes, a claim for which there is no firm evidence. We thus

wanted to estimate the two kinds of errors that may occur in placement deci-

sions: unnecessary placements and failures to remove children from their

homes when that should be done to prevent further harm. To accomplish this

goal, we needed a way to determine the right decision in cases. We believed

that we might be able to identify decision-making standards by studying thedecision making of experts in the child welfare field. Once such standards

were developed, we could determine how frequently the decisions of actual

workers deviated from them.

Such an approach depends on substantial agreement among experts

otherwise, standards could not be developed. Unfortunately, it was apparent

early on that such agreement does not exist, so we were unable to pursue our

goal of estimating thedegree of error in placementdecisions. But thestudy of

the character of that disagreement proved fruitful. We went on to study front-

line workers, to compare their decision making with that of experts.

In addition to studying the placement and FPS referral decisions, we

sought to understand the problem of targeting of family preservation ser-

vices. Because family preservation programs are intended to prevent the

placement of children in substitute care, the target group ought to be families

that would experience the placement of children in the absence of these ser-

vices. However, experimental studies of family preservation programs have

600 EVALUATION REVIEW / DECEMBER 1999



4/21

suggested that substantial proportions of the families served by these pro-

grams are not at imminent risk of placement (Schuerman, Rzepnicki, and

Littell 1994; Rossi 1991). As a result, these programs cannot be expected to

achieve one of their major goals. This research was designed to further

explore this issue.

An additional objective of this research was to explore case and respon-

dent characteristics that were related to various decisions. The results of

those analyses are reported in a separate article in this issue.

METHODS

Our examination of these matters is based on a vignette study in which wepresented workers and experts a series of case descriptions, asking them to

make decisions about the cases under varying conditions. The overall

research strategy calledfor studying decisionsmade by experts andchildpro-

tective service workers (simply called workersfrom now on) concerning a

sample of serious maltreatment cases. The cases were chosen to represent

fairly the kinds of serious maltreatment encountered in the investigations of

reports to a large state child welfare agency. An ideal, but impractical, design

would call for panels of experts and workers to conduct independent investi-

gations of the same serious cases and to make decisions about each case. Our

approximation to theidealdesign wasto usewritten summariesof actualcase

investigations based on interviews with the caseworkers assigned to each

case, supplemented with information obtained from administrative records.

Thecases were handled by line investigators working in theAurora and Cook

County regional offices of the Illinois Department of Children and Family

Services (DCFS). All of the cases had been investigated following reports to

the state child abuse hot line. In telephone interviews, each investigator was

asked a series of questions about his or her most recent placement case and

hisor hernext substantiated nonplacementcase. The cases were written up in

2- to 5-page summaries. The interviews resulted in 64 case summaries sup-

plemented by 6 case summaries all from Cook County, which had been used

in thepretest phase of this study. Each of thecase summariescontained infor-

mation on the demographic composition of the family involved, previous

complaint history, the nature of the current complaint, plus a narrative

account of the investigation. The case summaries did not disclose the investi-

gating workers decisions in the case. It should be noted that family preserva-tion services were available in these areas at the time the cases were collected,

although a relatively small numberof cases were referred to these services.

Schuerman et al. / PLACEMENT AND FAMILY PRESERVATION 601



5/21

The data were collected from experts in 1994 and from workers in 1995.

Twenty-seven persons served on the expert panel. Each was chosen because

he or she had some high standing in the child welfare field. Twelve were

national leaders in academic social work research or persons who have writ-

ten extensively about practice in child welfare. We have classified them as

theoreticians. The remaining 15 were chosen because of their direct line

experience in child welfare and their reputations of excellence in their posi-

tions, classified for our purposes as practitioners. The experts had an average

of 20 years of experience in the child welfare field. All of our experts were

college graduates and 9 had earned doctorates; 16 were women; 8 were Afri-

can American, 15 were Caucasian, 3 were Hispanic American, and 1 was

Asian American. Seven of the 15 practitioners were from the Illinois DCFS,

the remainder were nominatedby childwelfare administratorsin otherstates.The expert panel cannot be regarded as composed of representative sam-

ples of either theoreticians or practitioners. The intended bias in the selection

of experts was toward picking persons of prominence in their respective

fields. Indeed, the names of many of our experts are well known in the child

welfare field.

For the worker sample, we wanted persons who investigated abuse and

neglect cases as their major activity, who had at least 1 year of actual experi-

ence, and who worked in major urban jurisdictions. Because our resources

were limited, we could not sample from all workers in the nation and had to

restrict our sample to three state child protective agencies who were willing

to allow access to their workers. A total of 103 child protection investigators

from urban sites in Michigan (n = 44),New York (n = 32), and Texas (n = 27)

participated in the study. The Michigan workers were from Wayne County(Detroit) and volunteered to participate. They completed the questionnaire in

a meeting held in Detroit. In New York, 36 workers were randomly selected

from a larger pool of workers in Erie County (Buffalo). Of these 36 workers,

32 (89%) returned questionnaires mailed to them. In Texas, 37 workers from

offices in Houston and Dallas volunteered to participate in the study and 27

(73%) returned mailed questionnaires.

Two thirds of the workers were White (67%) and almost all others were

Black (32%). Workers were predominantly female (83%). With a mean age

of 38, workers averaged 10 years of experience in social services positions,

almost 6 years of which involved working in child protection. Most workers

(71%) had bachelors degrees and the remainder had masters degrees. Fam-

ily preservation services were available in all three states and presumably all

respondents had some familiarity with them, because allhad at least 1 year of

investigatory experience.




6/21

Each expert was asked to read 70 case summaries and record two judg-

ments on each case:

Question 1:Regarding no FPS condition, under the circumstance that no family

preservation services are available, should the child(ren) be placed out of the

home, provided services as ordinarily available, or the case closed?

If the decisionwas to place outof thehome, theexpert was asked to specify which

of the following was intended:1. Place temporarily to provide time to assess family needs and the safety of

the child.2. Place for a longer period to give caretakers a chance to change enough to

resume caregiving responsibilities.3. Place with the expectation that long-term placement will be required

because it appears unlikelythat thecaretakers will be able to care for theirchildren.

Question 2:Regarding FPS condition, under the circumstance that family preser-

vation services are available, should the child(ren) be placed out of the home,

referred to family preservationservices, provided ordinarily available services,

or the case closed?

The questionnaires also asked about other issues in each case (e.g., place-

ment with kin) and the experts rated how much confidence they had in their

decisions. The data from these questions were analyzed in detail in Rossi,

Schuerman, and Budde (1996). In addition, each expert filled out a question-

naire concerning views on child welfare issues, professional experiences,

and background. The experts reported that the task of filling out their ques-

tionnairestookat least 4 hours, and sometook aslongas 20hours.Eachof theexperts received a modest honorarium for completing the task. The resulting

expertsdata setconsists of the decisions made in the 1,890 case readings (27

experts each reading 70 cases) plus data on each of the 27 experts.

Because we believed that we could not ask workers to spend as many

hours on the decision-making task, we gave each of them 18 cases. Four sets

of 18 cases were selected randomly from the 70 cases given to the experts.

Four cases were identical across the sets, and the other 14 appeared in only

oneset. Alltold,60 differentcaseswere used in theworker booklets.The four

booklets were randomly assigned to participating workers and they were

asked to recordthe same judgmentsas above. In addition, workers filled outa

questionnaire about their work experiences, backgrounds, and their views on

child welfare issues, identicalin most respects to that filled outby theexperts.

The resulting data set consisted of 1,854 decisions (18 cases for each of 103workers) plus background questionnaires from each worker.




7/21

VALIDITY

There can be no pretense that the decision tasks set before the experts and

workers reproduce precisely the decision tasks confronted by actual case-

workers in the field. First, not all the information acquired in an actual field

investigation can be incorporated fully into a 3- to 5-page single-spaced sum-

mary. We designed the case summaries to cover a uniform set of what we

believed to be essential information about each case. Second, to some

unknown degree, the information received from an investigator may be influ-

enced subtlyor otherwiseby thedecision made in thecase. That is, it is possi-

ble that the information we obtained from the workers tends to support the

decisions they made. Finally, there is the question of the validity of the

responses provided by our experts and workers, that is, to what extent dothese responses reflect the decisions they would have made had they actually

been in a child protective office and confronted with these cases?

The main argument that can be made for the procedures we employed is

that they were the best we could do to produce case summaries that were as

close as possible to actualcases. To a large degree, theexperts thought that we

had prepared summaries that were adequate for the purposes. Experts were

asked to rate the adequacy of each case along a 5-point scale anchored by the

phrasesadequateand not adequatewith the midpoint designated as some-

what adequate. Nearly one third (31%) of the ratings were adequate and

another 36% were given the rating just below that (above somewhat ade-

quate). Perhaps more telling was that only 9% of the case summaries were

rated as either inadequate or the level above. These questions were not asked

of the workers in order to reduce the response burden on them.

DISTRIBUTION OF DECISIONS

On Question 1, (family preservation services not available), both workers

and experts would place the child in substitute care in about 56% of thecases,

with experts slightly more likely to give services (37% vs. 34%) and workers

slightly more likely to close cases (10% vs. 8%). Both workers and experts

placed children in a higher percentage of cases than theworkers who actually

investigated the cases. Cases presented to experts were evenly divided

between those that were originally placementcases andthose that were origi-

nally nonplacement cases. Of the cases presented to workers, 44% wereoriginally placement cases. The figures presented here do not take into

account the small differences in the sets of cases presented to experts and




8/21

workers. Calculations taking the differences into account result in very simi-

lar findings.

When FPS was available as an alternative, the decisions made by both

experts and workers changed noticeably. Placement decisions dropped to

44% forworkers and to 37% forexperts. Experts seeminglyhave more confi-

dence in FPS, using it in 44% of the cases, compared to the workers 28%.

Correspondingly, workers made more use of ordinary services (18% vs.

12%) and closed out cases slightly more often (9% vs. 7%).

The overall percentages mask great variations among experts and workers

in their propensities for various options. On Question 1, the proportion of

cases placed by experts ranged from 39% to 87%. On Question 2, experts

placement percentages ranged from 14% to 68% and their family preserva-

tion referrals from 23% to 71%. Ranges for workers were even larger,although they are not strictly comparable because workers were given sub-

sets of thecasespresented to experts. Such wide rangessuggest disagreement

among our respondents on decisions to be made, an issue to which we now

turn in more detail.

AGREEMENTAMONG RESPONDENTS

MEASURINGAGREEMENT

Agreementmay be measured in a numberof ways. A simpleand attractive

wayis to calculatethe percentage of cases on which twopersons agree. How-ever, agreement percentages are generally thought to be an inadequate meas-

ure of agreement because the percentage of agreement does not take into

account the fact that a certain amount of agreement is expected by chance. A

more sophisticated measure adjusts the percentage of agreement by what is

expected by chance. We calculate the chance-expected agreement of two

respondents and adjust the actual percentage of agreement for this expected

agreement. The adjusted measure of agreement is called Cohens kappa

(Cohen 1960), and the formula is kappa = (actual agreement expected

agreement) / (1 expected agreement). It can range from 1 to +1, with 0

indicating actual agreement equal to expected agreement. Negative kappas

indicate that the degree of actual agreement is less than expected by chance

and that the two sets of judgments disagree more than can be expected by

chance. In this data set, there are very few negative kappas, and the ones we

do have are very close to 0.




9/21

Cohens kappa is a measure of agreement between two raters. There are

351 unique pairs that can be formed among the 27 experts and 1,276 unique

pairs among the four groups of workers receiving identical sets of cases. For

each of Questions 1 and 2 we calculated the 1,627 kappas between pairs of

raters, or pairwise kappas. However, it would obviously be desirable to have

some kind of measure of agreementover all27 experts andoverall 103 work-

ers. Another version of kappa is available to measure degree of agreement

among more than two raters (Landis and Koch 1977; Fleiss 1981). It begins

with a measure of agreement among raters on each category. The alternatives

are dichotomized (e.g., placement vs. all other alternatives) and a kind of

analysis of variance analysis is then performed. Consider one of the dichoto-

mous categories to be positive. A within-subjects mean square is calculated

measuringthe variation withincases in thenumber of raters with positive rat-ings. If there is perfect agreement, that is, for each subject all the ratings are

positive or all the ratings are negative, the within-subjects mean square is

zero. The between-subjects mean square reflects the variation in the propor-

tion of positive ratings from the overall proportion of positive ratings (the

total number of positive ratings over all cases divided by the total number of

ratings on all cases). If all cases have the same proportion of positive ratings,

the between-subjects mean square is zero. Kappa is then a simple function of

the within- and between-subjects mean square. This category-wise kappa is

computed for each category and these kappas are then combined in an overall

kappa.The category-wise andcombined overall kappascan range from 0 to 1.

OVERALL FINDINGSON LEVELS OFAGREEMENT FOR QUESTIONS 1 AND 2

The average percentage of actual agreement among the 351 pairs of

experts was 65% on Question 1 and 56% on Question 2. For workers, the cor-

responding percentages were 64% and 48%. These findings indicate that

strong consensus on how to handlecases does not existamong eitherworkers

or experts. Furthermore, agreement percentages are not adjusted for chance

agreement levels.

Kappas for Questions 1 and 2 are shown in Tables 1 and 2. The overall

kappas for Question 1 were .45for experts and .35 for workers, and for Ques-

tion 2 were .32 and .24. All of the kappa values are significantly different

from zero, but they are nothigh. Both experts and workers are far from reach-

ing consensus on how to treat abuse and neglect cases. Experts agree amongthemselves slightly more than workers, but the differences are not large.




10/21

We can also examine agreement on a case by case basis. Collapsing the

alternatives for Question 1 into two categories, placement versus nonplace-

ment, we found that our 27 experts unanimously agreed on the disposition in

23% of the 70 cases they read, whereas in 67% of the cases, 81% or more of

the experts agreed. Thesenumbers are potentially deceptive because any time

we ask a group of respondents to make decisions about a large number of

cases, there will be some cases on which most of the respondents choose a

particular alternative, whereas on other cases, there will be more even splits of

judgments. Nonetheless, the data suggest that there are cases on which there

is a high level of agreement. These cases appear to be those at the extremes,

cases that are very serious or those involving very little risk to children. Of

course, it is expected that agreement would be greater at the extremes.

As with many things in social research, there are no very good standardsfor what a high, low, or adequate kappa isbut generally, kappas of less than

.6 are considered as not very good. Even kappas of .6 can be considered

inadequate when thedecisioninvolved is of some importance, when thedeci-

sion is likely to alter the course of someones life. In any event, the kappas

both for workers and experts are far from .6, suggesting that decision making

in the child protective services has a substantial component of chancethe

same abusing or neglecting family can be treated quite differently by differ-

ent workers or different experts.

Tables 1 and 2 also show kappasfor each of thestates andfor theoreticians

and practitioners separately. Although workers from New York have the

highest overall kappa for Question 1, the Texas workers have the highest

overall kappa forQuestion2. Among experts,practitioners tend to agree with

each other more frequentlyindeed, the highest overall kappa value, .48, isrecorded for practitioners answering Question 1.

Agreement was somewhat higher for the decision to place children than

for the other alternatives. Expert kappas for placement versus nonplacement

(aggregating nonplacement categories) were .54 for Question 1 and .44 for

Question 2, andthe corresponding kappasfor workers were .45 and .39. Kap-

pas for other decision categories (FPS, other services, closing the case) were

lower.

The category-wise and overall kappas were lower for Question 2 than for

Question 1. Because Question 2 involves the introduction of FPS as a choice

fordecisionmakers, lower kappas on Question 2 indicate that thepresenceof

this option (FPS) reduced levels of agreement among experts. Of course, the

addition of new categories tends to reduce agreement simply by increasing

the number of choices.




11/21

Disagreement over decisions can take several forms. One form occurs

when decision makersdisagreeon specific cases but have about thesame dis-

tribution of decisions. For example, all decision makers might place thechild

in about the same proportion of cases even though they disagree about the

cases in which placement should occur. Another form of disagreement

occurs when decision makers vary in their use of the different choices avail-

able. For example, some might take custody in a large proportion of cases,

with others doing so infrequently. In the latter case, there is disagreement

about thresholds, that is, in the propensity to favor one kind of case disposi-

tion over another. Finally, decision makers might disagree about both thresh-

olds and the kinds of cases to be considered for placement.In the present case, much of the disagreement arises because respondents

had widely varying thresholds, as shown in Table 3. On Question 1, although

74% of experts and 52% of workers placed the child in 46% and 65% of their

cases, respectively, 7% of experts and 23% of workers placed less frequently,

and 19% of experts and 24% of workers placed more frequently. What this

threshold variation means is that a familys chances of having a child placed

varies widely according to the person assigned to investigate the case.

Threshold differences are also evident when referrals to FPS are avail-

able as an alternative. The differences between experts and workers on

placement distributions are not statistically significantfor either Question1

or Question 2the groups have threshold distributions that are similar. The

threshold variation within groups is far greater than that between.

As to family preservation thresholds, workers and experts have statisti-callydifferent referral patterns: Experts are much more inclined to refer cases

to FPS, three fourths of them doing so in 36% or more of the cases compared


TABLE 1: Question 1 (no FPS available)Category-Wise and Overall Kappas

for Experts and Workers (standard errors in parentheses)

Question 1 (no FPS) Kappa Values

Group n Overall Place Services Close

All workers 103 .35 (.005) .45 (.007) .27 (.007) .23 (.007)

Michigan 44 .37 (.013) .47 (.016) .27 (.016) .24 (.016)

New York 32 .44 (.019) .53 (.023) .39 (.023) .23 (.023)

Texas 27 .37 (.022) .45 (.027) .25 (.027) .36 (.027)

All experts 27 .44 (.005) .54 (.006) .38 (.006) .37 (.006)

Theoreticians 12 .44 (.012) .51 (.015) .35 (.015) .46 (.015)

Practitioners 15 .48 (.010) .59 (.012) .42 (.012) .28 (.012)

NOTE: FPS = family preservation services.



12/21

to less than one in four of the workers. A substantial proportion of workers,

42%, would refer one quarter or fewer of the cases to family preservation,

compared to only 4% of the experts.

The main findings concerning agreement on decision making are as fol-

lows: There are wide differences among workers and experts in their tenden-

cies to place children in substitute care and to make referrals to FPS. As a

result, similar cases in themidrangeof severity of familyproblemsare treated

quite differently by different experts and workers.

COMPARISONS WITHPREVIOUS STUDIES

There have been a number of other studies of agreement on child welfare

decisions, either as part of larger studies or as a central issue in the investiga-

tion. In some cases, the reports of these studies do not provide data sufficient

to make comparisons with our work. We review here a few studies that do

allow at least rough comparisons.

Phillips, Haring, and Shyne (1972) and Phillips, Shyne, and Sherman

(1971). In this study of decisions in Massachusetts, case records of 25 fami-

lies who were given in-home services and 25 families in which a child was

placed were reviewed by three judges, each of whom had more than 5 years

experience in making decisions regarding placement (Phillips, Shyne, and

Sherman 1971, 69). Selected cases had some element of doubt about

whether or not the child should be placed, and cases with insufficient datawere not included (Phillips, Shyne, and Sherman 1971, 69). Judgments were

made blind, that is, without knowing the actual disposition of the case.


TABLE 2: Question 2 (FPS available)Category-Wise and Overall Kappas for

Workers and Experts (standard errors in parentheses)

Question 2 (FPS available) Kappa Values

Group n Overall Place FPS Services Close

All workers 103 .24 (.004) .39 (.007) .12 (.007) .14 (.007) .23 (.007)

Michigan 44 .25 (.010) .37 (.016) .13 (.016) .13 (.016) .23 (.016)

New York 32 .25 (.015) .39 (.022) .10 (.022) .21 (.022) .20 (.022)

Texas 27 .27 (.015) .41 (.022) .15 (.022) .09 (.022) .40 (.022)

All experts 27 .32 (.004) .44 (.006) .26 (.006) .16 (.006) .36 (.006)

Theoreticians 12 .28 (.010) .37 (.015) .21 (.015) .10 (.015) .49 (.015)

Practitioners 15 .36 (.008) .51 (.012) .30 (.012) .20 (.012) .29 (.012)




13/21

Dichotomous judgments of placement or own home were made on 94 chil-

dren in the 50 families, and data are reported on these 94 children. These

authors say that two of the three associations among the judges (in terms of

theKendall Q, a measure of association in 22 tables) indicate a substantial

degree of agreement and that the levels of agreement are considerably bet-

ter than chance (as indicated by2; Phillips, Shyne, and Sherman 1971, 71).

The data presented allow for the computation of the kappa statistic. For the

three pairs, the actual agreements were 62%, 65%, and 69%, and the kappas

were 0.24, 0.30, and 0.37. Hence, the degree of agreement found appears to

be somewhat less than that indicated by our category-wise placement kappas

for Question 1.

Rapp (1982). A total of538 workers fromIllinoisDCFSweregivenone of

three vignettes of cases that could go either way on the placement decision.

Workers were divided into experimental and control groups. The experimen-

tal group was given descriptions of services and told to assume their unlim-

ited availability, and the control group was not given this information. The

decision was to place or not place. Cohens kappas were computed for each

case for each of the experimental conditions, a total of six coefficients. Coef-

ficients for one case for the experimental and control conditions were 0.58

and 0.46, respectively. Coefficients for the other two cases ranged from 0.02to 0.28. The overall coefficient is 0.26. The data presented allow for estima-

tion of the combined category kappa computed from marginals on the cases.


TABLE 3: Distribution of Decisions (in percentages)

Question 1 Question 2 Question 2 Placement Placement Family Preser vat ion

Proportion of

Placement

Decisions Experts Workers Experts Workers Experts Workers

Less than .26 0.0 1.0 18.5 9.7 3.7 41.8

.26 to .35 0.0 1.9 22.2 21.4 22.2 35.0

.36 to .45 7.4 20.4 33.3 26.2 33.3 14.6

.46 to .55 44.4 19.4 18.5 18.5 11.1 4.9

.56 to .65 29.6 33.0 3.7 13.6 22.2 2.9

.66 to .75 14.8 13.6 3.7 7.8 0.4 0.0

More than .76 3.7 10.7 0.0 2.9 0.0 1.0

Minimum 0.39 0.22 0.14 0.0 0.23 0.0Maximum 0.87 0.83 0.68 0.78 0.71 0.83

n 27 103 27 103 27 103



14/21

This estimated kappa is 0.02. It is evident that in this study, levels of agree-

ment were not much different than chance. The study found little effect of

service availability on decisions.

Craft, Epley, and Clarkson (1980). In this study, 16 cases were presented

to 38 workers who made a determination of whether to recommend court

action in the case. The 16 cases were constructed to vary four dichotomous

characteristics of the case (consistency of explanation with the injury, parent

reaction, severity of injury, and existence of a previous report). From the data

presented, it is possible to compute the overall degree of agreement (71%)

and a combined category kappa, 0.24. Again, this is somewhat lower than the

levels of agreement we found.

Shinn (1969). In this study, 117 workers from various fields of service

were each given three cases out of a pool of six cases. There were five deci-

sion alternatives: definitely not separate; not separate, reserved; separate rou-

tinely; separate immediately; and other actions. The last of these alternatives

occurred in only two judgments. A coefficient said to be Cohens kappa was

computed (0.35, actual agreementof 48%), but it is not clear how it wascom-

puted. For collapsed separate versus not separate categories, the coefficient

was 0.43 (actual agreement of 54%). The data reported allow us to compute a

combined category kappa. For the 5-category item it is 0.04, and for two col-

lapsed categories it is 0.06. Again, these results reflect considerably lower

agreement than that in our data.

UNDERSTANDING REFERRALS TOFPS

Most family preservation programs are designed to prevent unnecessary

and inappropriate placement. Although programs in different sites are imple-

mented in different ways, they share common features. First, they are based

on the assumption that there are a significant number of placements in which

placement is not needed to protect the physical safety of the victimized chil-

dren. Second, it is believed that FPS, consisting of short-term but intensive

casework by FPS workers, can bolster the functioning of families at immi-

nent risk of placement sufficiently to make such substitute care placements

unnecessary. Third, FPS assumes that child protective service investigating

workers can reliably identify families at imminent risk of placement and thatthey will refer many of these families to FPS. Finally, putting FPS in place

can result in considerable cost savings to the state, because long-term




15/21

placement is so much more expensive than FPS services. At the same time, it

is generally recognized that placement will continue to be required in some

cases, so FPS cannot prevent all placements.

Because the experts and the workers were asked how they would make

decisions under the contrasting conditions of FPS being unavailable (Ques-

tion 1) and being available (Question 2), we can explore in some detail how

the criterion of imminent risk is used in referring cases to FPS.

Table 4 shows the cross-classification of Questions 1 and 2 decisions.

Question 1 placement decisions have been divided according to whether the

decision maker intended temporary placement, longer term placement, or

permanentplacement (see thequestionwording above). Theupper half of the

table contains experts decisions and the bottom half contains workers

decisions.At theextremes, both workers and experts tend to usethe same disposition

whether FPS is available or not. Of the decision makers making permanent

placement decisions in Question 1, 95% of the experts and 97% of the work-

ers also placed in response to Question 2. The percentages of those who close

cases inresponseto Question 1 and alsoclose cases inQuestion2 are 97% for

experts and 90% for workers. These percentages show a quite encouraging

consistencyafter all, if a case was judged appropriate for permanent place-

ment or case closing when FPSwas notavailable,the availability of FPSas an

alternative should not affect such decisions. The former can be considered as

cases in which the safety of the child is very much in question and the latter

are incidents that would be generally recognized as not requiring further

intervention by the state.

The fate of the remaining Question 1 decision categories is not as clear-cut. Of the cases experts would place in longer term (but not permanent)

placement on Question 1, 72% would continue in that category when FPS is

available; the corresponding figure for workers is 78%. Both workers and

experts would refer most of the remainder of this category to FPS. The Ques-

tion 2 distributions of the remainingQuestion 1 decisions aremuch more het-

erogeneousof those placed temporarily in response to Question 1, on

Question 1 workers would also place 58% and experts 46%. Most of the

remainder (39% and 54%) would be referred to FPS. Note that experts are

more willing than workers to refer to FPS cases they temporarily placed in

Question 1. Perhaps most interesting of all are the cases referred to ordinary

services in response to Question 1expertswould refer 68%of such cases to

FPS, and workers 50%.

The targeting issue is most directly addressed by looking at the sources of

FPS referralsin Question 1 decision categories. Of cases experts refer to FPS,




16/21

57% would have been provided ordinary services in the absence of FPS. The

corresponding figure for workers is 60%. Very few cases referred to FPS in

Question 2 were designated for permanent placement in Question 1.

In the findings of Table 4, it is clear that both experts and workers do not

consistentlyuse thecriterion of imminent risk of placementin referringcases

to FPS. Although some FPS referrals come from among those cases decision

makers would place when FPS is not present, the bulk come from amongcases they gave services in that condition. A substantial proportion of the

remaining referrals to FPS come from cases the respondents believe would

have been short-term placements. Hence, in practice, the respondents would

rarely use FPS to avoid long-term placement.

CONCLUSIONS

INTERRESPONDENT AGREEMENT

We chose experts prominent in the field, hoping to find consensus among

them strong enough to warrant constructing best practice standards for

making decisions about the treatment of substantiated cases of maltreatment.


TABLE 4: Question 2 Decisions by Question 1 Decisions: Experts and Workers

(in percentages)

Question 1 Decision (FPS not available)

Question 2 Decision Permanent Longer Temporary Ordinary Close

(FPS available) Placement Placement Placement Service Case Total

Experts decisions

Placement 94.6 72.3 45.7 0.0 0.0 36.3

Refer to FPS 5.4 27.5 53.8 68.1 2.8 44.1

Ordinary service 0.0 0.2 0.5 31.8 0.7 12.0

Close case 0.0 0.0 0.0 0.2 96.5 7.4

n 167 480 387 689 142 1,865

Workers decisions

Placement 97.2 77.7 57.8 0.0 0.5 44.1

Refer to FPS 2.4 19.8 38.6 50.3 3.8 28.4

Ordinary services 0.3 2.5 3.2 49.0 6.0 18.4

Close case 0.0 0.0 0.4 0.7 89.6 9.2

n 291 485 251 620 182 1,829




17/21

Our most important (and disappointing) finding was that overall agreement

among experts on such decisions, although better than chance, was below

levels that would support clear and positive identification of standards based

on expert consensus.

We also found that caseworkers were inconsistent in their decisions about

cases. Although workers showed somewhat lower levels of agreement than

experts, the two groups were not far apart. If the experts decisions exempli-

fied current best practices, the workers were fairly close to achieving such

practices.Previousstudies generally showed even lower levels of agreement.

It should also be noted that lack of consensus is not unique to the field of

child welfare. For example, levels of agreement found here are within the

range of those found in studies of major diagnostic classes of the American

Psychiatric Associations Diagnostic and Statistical Manual (1994), inwhich case summarieswere used (Kirk and Kutchins 1992). Variations in the

delivery of medical procedures are well known (Wennberg and Gittelsohn

1973). In fact, it is possible that the results found here are representative of

situations generally in which professionals make judgments about people.

A major source of the lack of interexpert and interworker agreement was

the wide range among those in both groups in their tendencies to place chil-

dren in substitute care. These varying thresholds necessarily mean that on

specific cases, disagreements arise even if there may be agreement on the

kinds of cases that warrant placement consideration.

UNDERSTANDING FAMILYPRESERVATION SERVICES

A key design feature of our research was that it allowed contrasting dispo-

sition decisions in the presence and absence of family preservation services.

The contrast showed that family preservation served two functions. On one

hand, less serious cases that would be disposed of by placing children in the

absence of FPS were referred to those services when that was an available

alternative. On theotherhand, many of thecasesthat hadbeen slatedfor ordi-

nary services were also sent to FPS. Family preservation services diverted

some placement cases, but mainly provided enriched services for families

that the experts would not have placed.

As a consequence, manyperhaps mostof cases referred to FPS could

not be considered as meeting the imminent risk criterion. This result con-

forms to the findings of experimental studies of family preservation pro-

grams suggesting a substantial targeting problem in the provision of theseservices, that is, that these programs often do not serve cases for which they

are intended (cases that would experience placement in the absence of the




18/21

programs) and thus cannot be expected to meet their objectives (placement

prevention) (Schuerman, Rzepnicki, and Littell 1994). The targeting prob-

lem is less severe in these data than in the experiments which found that

between 7% and 20% of cases could be considered to be at imminent risk of

placement (Schuerman, Rzepnicki, and Littell 1994). The results of this

study also suggest that it may be quite difficult to solve the targeting problem

in FPS. There are at least tworeasons for this pessimism. First, these services

are a quite attractive alternative in cases needing services but not involving a

risk of placement, particularly given the paucity of services in the child wel-

fare system for intact families. Second, there are suggestions in the data that

cases requiring placement are quite different from those thought to be appro-

priate for FPS. We pose the hypothesis, which we cannot conclusively prove

in these data, that if a case has the characteristics required for family preser-vation, it is unlikely that that case would be placed. If this hypothesis is cor-

rect, a rethinking of the objectives of these services is required. In particular,

it will be necessary to ground these services on objectives other than place-

ment prevention, perhaps focusing goals on the enhancement of child and

family functioning, a shift that has recently been suggested (Wells and Tracy

1996). Of course, this would require evidence that short-term, intensive fam-

ily preservation services have effects on child and family functioning that are

more than marginal and more than short term.

THEPROBLEM OF INCONSISTENCY OF DECISIONS

Ouranalysessuggest that forcasesthatare notat theextremes (thosewith-

out very severe conditions or relatively insignificant problems), disposition

decisions depend very much on who is making the decision. Of course,

extreme cases are always easythe test of a decision-making process lies in

the middle cases.

We believe this lack of consistency in the handling of cases should be a

significant concern to the field. The question is what to do about it. We

encounterhere a tension between thegoal of makingconsistent decisionsand

the goal of makingright decisions. One cancontemplateactions that promise

greater consistency but we cannot be sure that the resulting decisions will be

optimum in terms of ultimate outcomes. But existing knowledge is inade-

quate to provide rules on how to reach correct decisions, so we believe that

considerations of equity compel us to at least reach for consistency.

Our data indicate that the lack of consistency may be largely a matter ofdiffering decision thresholds held by various decision makers. By and large,

workers appear to take into account the same pieces of information and




19/21

combine these data in similar ways. So the scales they use are similarit is

the cut points that differ. Efforts to standardize thresholds might be under-

taken by adopting rules for the handling of certain categories of cases (Stein

and Rzepnicki 1984). Of course, exceptions to rules must be allowed, but

they should be fully justified. Often, such rules would be codifications of

existing informal practices. For example, we might adopt the rule that cases

with only one or two prior incidents of identified maltreatment would not

result in placement unless the current incident was very serious or there were

other serious aggravating circumstances.

Although the major problem in inconsistency lies in the middle cases, one

could focus ones efforts to attain somewhat more equity on more extreme

cases, on the grounds that the middle cases represent those in which our pres-

entknowledge is simplyinadequate to do much to improve decisions(i.e.,weare unable to make adequate predictions about the response of such cases to

various actions of the child welfare system). Our data contain a number of

cases in which, say, 75% or more of the respondents agree on the action to be

taken. But this means that up to 25% of respondents would take an action at

variance with that chosen by the vast majority. Eliminating such outlier deci-

sionsthrough, for example, rulesfor handling more extreme caseswould

result in at least a marginal increment in equity. Of course, such a strategy

would require that we have reliable rules for determining extreme cases.

Improved training of investigative workers on decision making will be

central to any efforts in improving the outcomes of their work. Training must

deal with the gathering of data, the identification of critical issues affecting

the safety of children and the likely response of families to various services

on how to combine various bits of information in the process of reaching adecision, and on the use of threshold standards (Stein and Rzepnicki 1984).

Case vignettes, such as those used in this study, might be employed in such

training.

The above modest proposals suggest certain directions for further

research. Our common scale, varying thresholds hypothesis is just that, a

hypothesis. It is suggested in ourdata, but is notconclusively proven. It needs

to be further explored. We need further work to specify thefactorsthat govern

decisions and the ways these factors are combined, uncovering the implicit

equations that are used by workers. We also need to explore further how case

characteristics are related to levels of agreement. The field may be relatively

settled as to how to deal with some kinds of cases,whereas other cases maybe

surrounded by uncertainty (e.g., drug-affected infants). Finally, efforts to

improve consistency of decision making need to be carefully evaluated.




20/21

REFERENCES

American Psychiatric Association.1994.Diagnostic and statistical manual of mental disorders.

4th ed. Washington, DC: Author.

Cohen, J. 1960. A coefficient of agreement for nominal scales.Educational and Psychological

Measurement20:37-46.

Craft, J., S. Epley, and C. Clarkson. 1980. Factors influencing legal disposition in child abuse

investigations. Journal of Social Service Research4:31-46.

Fleiss, J. L..1981. Statisticalmethodsfor ratesand proportions. 2d ed.New York:John Wiley.

Kirk, S.,and H. Kutchins. 1992. Theselling of DSM: Therhetoric of sciencein psychiatry. New

York: Aldine de Gruyter.

Landis, J. R., and G. G. Koch. 1977. A one-way components of variance model for categorical

data.Biometrics33:671-79.

Phillips, M. H., B. L. Haring, and A. W. Shyne. 1972.A model for intake decisions in child wel-

fare. New York: Child Welfare League of America.Phillips,M. H.,A. W. Shyne,and E. A.Sherman.1971. Factors associatedwith placement deci-

sions in child welfare. New York: Child Welfare League of America.

Rapp, C. A. 1982. The effect of the availability of family support services on decisions about

child placement.Social Work Research and Abstracts 18:21-27.

Rossi, P. H. 1991.Evaluating family preservation services. New York: Edna McConnell Clark

Foundation.

Rossi,P.H., J. R. Schuerman, andS. Budde.1996. Understanding childmaltreatment decisions

and those who make them. Chicago: Chapin Hall Center for Children.

Schuerman, J.R., T. L. Rzepnicki, andJ. H.Littell.1994. Putting families first: An experimentin

family preservation. New York: Aldine de Gruyter.

Shinn, E. 1969. Is placement necessary? An experimental study of agreement amongcasework-

ers in making foster care decisions. Ph.D.diss.,ColumbiaUniversitySchoolof Social Work,

New York.

Stein, T., and T.L. Rzepnicki. 1984.Decision making in child welfare services: Intake and plan-

ning. Boston: Kluwer-Nijhoff.Wells, K., and E. Tracy. 1996. Reorienting intensive family preservations services in relation to

public child welfare practice.Child Welfare 75:667-92.

Wennberg, J. E., and A. Gittelsohn. 1973. Small area variations in health care delivery.Science

182:1102-8.

John Schuerman i s a professor and deputy dean for Academic Affairs in the School of Social

Service Administration and a faculty associate at the Chapin Hall Center for Children. He has

published books on researchand statistical methods and on programs for children and families,

includingPutting Families First: An Experiment in Family Preservationco-authored with Tina

Rzepnicki and Julia Littell. He is currently coprincipal investigator of the National Evaluation of

Family Preservation Services, a set of randomized experiments on familypreservation and fam-

ily reunification programs financed by the U.S. Department of Health and Human Services.

Peter H. Rossi is currently Rice Professor Emeritus of Sociology and Director Emeritus of the

Social andDemographic Research Institute at the University of Massachusetts at Amherst. He is

past president of the American Sociological Association and was the 1985 recipient of the




21/21

Common WealthAwardfor contributions to sociology. He has received awards from the Evalua-

tion ResearchSociety,the Eastern Evaluation ResearchSociety,and the Policy Studies Associa-

tion for his contributions to evaluationresearch methodology.His recent books include Evalua-

tion: A Systematic Approach (1999, with Howard Freeman and Mark Lipsey), Of Human

Bonding (1990,with Alice S. Rossi), Downand Outin America (1989), Just Punishments (1997,

with R. A. Berk), andFeeding the Poor(1998). He has served as editor of theAmerican Journal

of Sociology andSocial Science Research. Hehas beenelecteda Fellow ofthe American Acad-

emy of Arts and Sciences and of the American Association for the Advancement of Science.

Stephen Budde is a senior research associate at Chapin Hall Center for Children at the Univer-

sity of Chicago. He currently is a coprincipal investigator on Chapin Halls evaluation of the

Edna McConnell Clark Foundations Community Partnerships for Protecting Children initi a-

tiveand a research associateon thefederallyfunded National Evaluation of Family Services. In

addition, Dr. Buddeteaches advanced clinical research and directs a training programfor pub-

lic child welfare staff at the University of Chicago School of Social Service Administration.



Eval Rev 1999 Schuerman 599 618

Documents

Transcript of Eval Rev 1999 Schuerman 599 618