An Evaluation of Volunteers Courts Controversy
Study of appointed advocates for children produces some surprising results and a challenge for the group that asked for it.
By Barbara White Stack

The Court Appointed Special Advocate (CASA) program couldn't sound more apple pie, more thousand points of light.

CASAs are a cadre of 74,000 volunteers trained for dozens of hours, then dispatched to conduct independent investigations of child abuse and to represent the children's interests in courts around the nation. What could be wrong with that?

Virtually nothing, according to past evaluations. A qualitative consumer satisfaction survey of 23 CASA programs last year, commissioned by the National CASA Association, gave glowing reviews.

So, a second, more ambitious evaluation, a national one including a control group, seemed without risk.

That's not how it turned out.

While containing some information for national CASA to brag about - such as judges assigning CASAs to the most difficult cases, then frequently doing exactly what the advocates recommend - the report commissioned by the association delivers some surprisingly damning numbers.

It says CASAs, an overwhelmingly white and female group, spend little time on cases, and even less on those of black children. It says youngsters with CASAs are associated with more removal from parents, less kinship care and less reunification with parents.

CASA critics, including social workers who say problems in child welfare should be addressed by hiring more professionals rather than relying on volunteers, seized on those numbers, contending that they prove CASAs might actually harm children and families.

CASA officials focus on the positive findings and argue that the negative ones are questionable or need more study. Even the evaluation's author, Caliber Associates, has taken the unusual step of responding to CASA critics by stressing that some of the numbers may not prove anything because the controls in the study may have been faulty.

Whatever the truth about CASAs, the organization's experience with the study illustrates an increasingly important point for the youth field: the risks that groups take when complying with mounting demands from government and foundation funders to prove that what they do works.

Initial Findings

Michael Piraino, national CASA's chief executive officer for the past decade, says he sought funding for a thorough CASA evaluation long before accountability became the demand du jour. He recalls that in the mid-1990s, when he was searching for $1 million to pay for two evaluations and a data collection system to support them, funders wondered if they were worth doing. (See "Court Advocate Program Grows, But How Much Does It Help?" June 2000.)

Beginning in 1997 and ending in 2000, Piraino received three grants totaling $1 million from the David and Lucile Packard Foundation of Los Altos, Calif. The grants paid for software, called COMET, to help local CASA organizations track information, such as the hours that volunteers spent on cases and how many recommendations were accepted by judges. The grant also paid for two evaluations: the first, a qualitative study conducted by Pat Litzelfelner of the University of Kentucky College of Social Work, and the recently released quantitative study by Caliber.

In Litzelfelner's satisfaction survey, which began in 2001 and examined responses from 742 judges, lawyers, parents, foster parents and social workers, every group gave CASA a positive ranking on every question. The questions ranged from whether the respondent understood the role of a CASA (which got the highest scores) to whether the CASA visited the children regularly (which got among the lowest).
Litzelfelner notes that her results can't be generalized to all CASA programs because the sites surveyed were not randomly selected.

CASA critic Richard Wexler, executive director of the National Coalition for Child Protection Reform, points to comments from caseworkers that CASAs needed to spend more time on cases, and that many CASAs have middle-class values and do not appreciate the different cultural backgrounds of their clients. Wexler says these comments reinforce his contention that CASA is little more than a bunch of white women with matching shoes and purses telling poor black mothers how to run their households.

The Caliber study, which cost $317,000, was based on COMET data and on the U.S. Department of Health and Human Services' (HHS) National Survey of Child and Adolescent Well-being, which contains statistics on 5,500 children from across the country. Twenty-five of the 915 CASA organizations around the country provided information from their COMET databases.

Among Caliber's findings from COMET: Ninety percent of volunteers were white, and 79 percent were women. Only 8 percent were African-American, a sharp contrast to the children in the nation's child welfare system, 40 percent of whom are African-American, according to HHS statistics. Piraino says that among all CASAs nationwide (as opposed to the 25 CASA organizations studied), 18 percent are African-American, a figure he notes has increased from 13 percent five years ago.

The COMET data also showed that after receiving an average of 43.9 hours of training, volunteers spent 3.2 hours a month working on the case to which they were assigned. When this figure was announced at the CASA national conference in Washington in June, a sucking noise of disbelief was audible, recalls Dennis S. Hockensmith, executive director of the Pennsylvania CASA Association.

CASA directors across the country say their volunteers give much more time than the report says. Caliber says the number in its report may be incorrect because CASAs may not have done a good job of logging their hours on COMET. Volunteers are asked but not required to log their hours, and it is unclear how many of them do it promptly or completely.

The data did show that volunteers spent most of their time with the children, much more than on other activities, such as writing reports for the court and interviewing parents and foster parents. But when the child was black, the amount of time spent per month dropped by more than an hour per volunteer.

On the positive side, the data also showed that in 61 percent of cases, judges accepted every one of the volunteers' recommendations. But there was this oddity, considering the overwhelming number of white women volunteers: Judges were four times more likely to abide by advice from men than from women, and 2.5 times more likely to follow recommendations from African-American volunteers than those of other races.

From there, the findings get more controversial, and the research more tricky.

Out-of-Control Group

National CASA has long suspected that CASA volunteers are assigned the most serious and difficult child abuse and neglect cases that come before the judges. Using data from the federal well-being survey, Caliber confirmed that.

To isolate the impact of CASAs on the youth assigned to them, Caliber attempted to create a control group of children who did not have CASAs, but whose cases were as serious as those who did. Caliber considered factors such as age, sex, race, previous out-of-home placements, and the number of risk factors reported by caseworkers
The results: Children who were assigned CASAs, and their parents, received more services than those without CASAs. Children without CASAs weren't any more likely that those with CASAs to be mistreated again, and they stayed in the child welfare system about the same amount of time.

But CASA kids were more likely to be removed from their parents: 89 percent of the time for CASA kids, compared with 18 percent for non-CASA kids. For children whose cases remained open at the end of the study period, CASA kids were less likely to be reunified with their parents and less likely to have been placed with kin.

Wexler jumped on the findings, saying the study showed that the CASA program "does nothing to actually improve the lives of children and may well make them worse." Part of Wexler's criticism of CASA stems from his conviction that judges, based on recommendations from child welfare caseworkers and CASAs, remove children from their parents far too often.

By contrast, for people like David W. Soukup, who founded CASA when he was a judge in Seattle, these removals usually signal that the system is working to protect abused children.

Policy differences aside, Caliber says Wexler misconstrued its report. But the consultant has also backed away from some of its own findings. Caliber says that when it used a mathematical model to test the control group findings, it discovered that it may have failed to establish comparable groups.

"It doesn't look statistically plausible to me that having a CASA has more of an effect than the type of abuse, what is going on with parents, et cetera," says Caliber senior associate Jennifer Brooks, who worked on the study.

So, the most that can be said about the findings is that there is a relationship between the CASA and other effects, such as more frequent removal from parents. But the study, Caliber researchers say, probably does not show that the CASA caused that effect.

Raymond Kirk, research professor at the school of social work at the University of North Carolina/Chapel Hill, agrees that the comparison group findings should be viewed with caution. "The CASA model should not be over applauded or criticized," he says. "The science part of this report is too soft for that."

The feelings of Susan Cammarata, a Pittsburgh attorney with some expertise in child welfare cases and several years of experience as a CASA volunteer, may represent that perspective. The Caliber evaluation confirmed her inklings. "CASA has its place," she says, "but I would rather see the child welfare system improved."

The Packard Foundation, the funder, declined to comment on the evaluation's findings.

Picking and Choosing

One reason the findings threw CASA officials for a loop is that, although they were participating in a scientific evaluation, what they really wanted was validation. A statement on the National CASA Association website calls the Caliber study part of CASA's commitment to "measuring success in our work."

"We hear good stories about what we do for kids," Piraino says. "But we wanted to document it.

It's not surprising, then, that CASAs were taken aback when the report suggested that what they thought about themselves might not be true. The reaction of CASA organizations to the negative parts of the report has almost uniformly been denial. Kelly Warner, spokeswoman for the Oklahoma CASA Association, says, "I read the study and my reaction was: Who are they talking about? Because this cannot be Oklahoma. I do not think we are doing something wrong."

National CASA has boasted about the parts of the study it liked, while saying the findings that could be considered critical are questionable and in need of further study.

This might be a natural organizational reaction, but it can border on duplicity.
Trudy Strewler, executive director of the Pikes Peak Region CASA, based in Colorado Springs, Colo., is among those who say CASAs should be proud of the results showing that CASA kids and families get more services. The National CASA Association has emphasized that result as well - even though it comes from the same part of the study in which the association rejects the negative findings because of potential control group problems.

As for those negative findings, CASA organizations are calling for more study and improved data collection. Hockensmith says of the racial disparity in the time CASAs spend on cases, "I am sure CASA will look long and hard at that. It needs to be analyzed more deeply." And Warner of Oklahoma is among those who say volunteers must improve data collection by being more diligent about recording their time in the COMET database.

Potential Impact

Faltering on an accountability test can't be all that bad. After all, every state in the nation has failed the federal Child and Family Service Reviews of their child welfare programs, and HHS, the agency holding their purse strings, hasn't done a thing about it.

HHS says states now know what they must do better

The negative findings in the Caliber study didn't prompt CASA to promise changes in practices. "National CASA will not allow inconvenient facts to get in the way of its insistence that CASA is a success," Wexler said in a news release. "This state of denial only compounds the harm done by the program itself."

CASA's attempts to stress the positives may be nothing more than instinctual self-preservation. Negative evaluation findings, followed by an outcry from critics, could hinder a youth group's ability to raise money and expand.

Piraino concedes that the report could hurt CASA. "It depends on how people look at it," he says. If they believe the criticisms sound too simplified, he says, the evaluation won't be a problem.

It can't be much help, however, in places like Cleveland, one of the last major CASA holdouts. Cuyahoga County Juvenile Division Senior Judge Peter J. Sikora says he prefers to have children represented by licensed attorneys who are trained for the work by the bar association and who may be held accountable for their performance in ways that volunteers cannot be, such as loss of license.

Because it would be costly to start and run a CASA program, Sikora says he'd have to be convinced of its value before moving in that direction. "Our court was taking a bit of criticism for not having a CASA program," he says. "But unless you can show me children and families benefit, and can show me a cost benefit, I see no reason to switch from the system where children are represented by licensed attorneys."
The Caliber evaluation won't do that for him.

Maybe another study will. Caliber is recommending that one be done in the same form as a double-blind medical investigation, where some patients get an experimental drug and some get a sugar pill.

Piraino says that's unlikely, because judges want CASAs appointed to all of the most serious cases and won't stand for half not getting the volunteers
He agrees that it might be done in a place where there is no CASA program, so that judges will feel that they're getting something they didn't have before (CASAs), at least for half the serious cases.

Perhaps Piraino's hometown of Cleveland would work.