|
 |
|
 |
|
This Web site is a component of the SAMHSA Health Information Network. |
Outcomes EvaluationOutcomes evaluation is designed to determine the effectiveness of an intervention as compared with a control (no treatment) group, an alternative intervention, or a standard intervention. It measures and assesses a program's effect; that is, the ability of a program to produce changes of the specified type and desired direction in the people who are exposed to it. Ideally, the evaluation should be conducted by an external person or group to avoid bias in data collection and analysis. However, many institutions and programs conduct their own evaluations and such evaluations can be very useful. Program evaluation can provide answers to a number of questions. Did the treatment group show significant change in relation to the comparison group, and can that change be attributed to the intervention? How well does the program work in real-world settings? With which subgroups does it work? What kinds of effects can be expected from the program and at what magnitude? What are the essential ingredients of the program? Obtaining answers to each of these questions requires changing the research strategy. Thus, a program evaluator must have a clear idea about the purpose of the evaluation to develop the appropriate evaluation design. Evaluation IssuesA number of problems can occur when implementing outcomes evaluation studies. These problems can call the validity of the evaluation results into question. The occurrence of one or more of the following problems makes it difficult to determine what is being evaluated, whether the results are valid, or whether the results can be applied to the intended population (Dennis, 1993):
The presence of any of these problems can lead to questions about the validity of the evaluation results. To the extent possible, the program evaluator should take steps before the study commences to reduce the likelihood that these problems will occur. For instance, written protocols and careful documentation make for greater uniformity and delivery of services. Or, a pilot study to examine the flow of clients with the targeted characteristics through a probation department could ensure the availability of an adequate number of study subjects. The implementation also should be carefully monitored to preserve the integrity of the research design. If deviations occur, they should be documented and reported in the final study report. Sample SizeOne issue of particular importance to assessing the effectiveness of treatment programs is sample size. The smaller the sample size, the lower the probability of detecting statistically significant treatment effects. This aspect of the research design, known as power analysis, is often overlooked, but it is important because program evaluations with sample sizes that are not large enough to detect treatment effects can waste resources and may lead to abandoning otherwise promising treatment approaches. The "nothing works" doctrine in the criminal justice field is partly attributable to research studies with small samples that failed to give rehabilitation programs a chance to prove themselves. To correct this problem, the evaluator should determine a sample size that is large enough to show whether the intervention makes a difference in the population and to conclude whether the treatment program did produce an effect. The technical procedure for doing this is beyond the scope of this chapter but can be found in a number of standard texts (Cohen, 1977; Lipsey, 1990). Successful program implementation and management depends on more than a good design and an adequate number of subjects. It also relies on the cooperation of staff and others involved in the intervention. This cooperation can be expected in a research environment. But in "real-world" settings like prisons, jails, probation departments, and community treatment programs, the intervention can place demands on people that they initially may be unwilling to assume. If staff members are not convinced of the need for the study, they can actively undermine the evaluation procedures. An evaluator may use a number of strategies to gain and maintain the cooperation of program staff (Dennis, 1993), including
Research FollowupSimilar considerations apply to gaining a subject's cooperation. Methods must be developed to reduce study attrition, from which every study suffers. The longer the study, the greater the number of subjects who drop out. Attrition can bias the results and, in the worst case, defeat the study. Researchers have developed a number of techniques (Dennis, 1993) that can be used to reduce subject attrition:
Outcomes MeasuresA variety of measures have been used to assess the effectiveness and treatment outcomes for AOD-involved offenders. These outcome measures include changes in
The specific measures selected should include behaviors specified in program goals and objectives. The treatment program designed to reduce substance use, decrease criminal involvement, improve self-concept, and increase job skills must include an evaluation designed to collect data on each of these variables.
Outcomes measures can cover a wide variety of psychological and social behaviors, but the primary purpose of most treatment programs for offenders is to reduce drug use and the criminal activity associated with it. Regardless of the positive benefits programs may produce, criminal justice drug treatment programs that are unable to bring about significant reductions in drug use and recidivism cannot be regarded as effective. Thus, measures that focus on relapse and recidivism are commonly used as indicators of program effectiveness in criminal justice settings. Neither relapse nor recidivism are simple behavioral measures. Both can be used to refer to a variety of behaviors and can be defined in a variety of ways. RelapseRelapse is not clinically regarded as a treatment failure, but as an indication that the treatment plan should be changed to address the cause or circumstances associated with the relapse. Some clinicians distinguish between the degrees of relapse -- from a single "slip," to sporadic use or a return to addiction. For research purposes, however, relapse often is defined as the single use of a specified drug during a given period of time. Various relapse studies have reported that more than 50 percent of people treated for alcohol or other drug dependence relapse within a year after a single treatment episode, and that that percentage increases with longer followup periods (Maddux and Desmond, 1986). As a result, treatment success can be measured as a reduction in the relapse rate when compared with the relapse rate of an untreated group or a group that received a different treatment. Drug use can be measured either by urine tests or by self-report. Some studies rely on self-reported drug use only, while others (Wish and Gropper, 1990) use validated urine test results. Urine test results can be abstracted from client records at little cost for some treatment evaluations. For followup studies, clients can be asked to provide urine specimens following the interview and the cost of testing can be factored into the evaluation budget. The evaluator should carefully consider the drugs that are of greatest relevance to the objectives of the program being studied. A potential problem is that subjects may refuse to provide urine specimens. If specimens are not collected from a large number of clients, the study's purpose can be undermined. However, inadequate collection has not proven to be a major problem. For instance, in the Drug Use Forecasting program, 80 percent or more of those interviewed in jails agreed to provide a urine sample (National Institute of Justice, 1992).
Although self-report results underestimate drug use (Mieczkowski, 1990; Rouse et al., 1985; Wolber et al., 1990), self-reports provide information about use patterns that cannot be obtained from drug testing. Thus, through self-reports, the evaluator can obtain reasonably reliable data on quantity and frequency of use, method of administration, circumstances of use, and reasons for use. These measures may be more useful for assessing the impact of a treatment program than merely determining whether the person tested positive at the 6-month followup interview. RecidivismWaldo and Griswold (1979) use the following definition of recidivism: "an offense committed by a person who has previously been convicted or adjudicated for an offense." This definition focuses on the behavior rather than "tendencies" or "proneness" and includes offenses that appear in official records and those that do not. In studies of treatment effectiveness, recidivism data usually are obtained from official criminal justice records. In addition, studies also may ask subjects about their criminal behavior during the followup period (Weis, 1986). But even if evaluators decide to rely on official records as the measure of recidivism, they must decide which level of contact with the criminal justice system will be used to determine recidivism. Should the measure of recidivism be rearrest, reconviction, reincarceration, or technical violations of probation or parole? Although there may be theoretical or methodological reasons for selecting one measure or a combination of measures, in practice, the choice will depend on the purpose of the study, the access the evaluator has to official records, and budget constraints. In addition to recidivism, official records often permit the evaluator to select other measures of the program's impact on criminal behavior, such as the number of rearrests or technical violations during the followup period, the types and severity of the offense(s) committed, the time to rearrest from program discharge, the length of any sentence imposed, and the annual rate of arrest, controlling for time on the street. The interpretation of recidivism rates is not always straightforward. Although most criminal justice outcome studies use recidivism from official records, researchers have noted (Turner et al., 1992) that "recidivism is actually a product of the offender's underlying criminality and the System's ability to detect that criminality and act on it (e.g., arrest probability)." For example, offenders in a community treatment program are committing fewer crimes than those not in treatment, but the increased supervision and surveillance associated with the treatment program increases the probability that their crimes and technical violations will be detected. Thus, even though the actual rate of the criminal behavior for the treatment group may be lower than for the comparison group, the treatment group's "official" behavior may be the same or worse than that of the comparison group. Collecting self-report data would be one way of clarifying the relationship between the offender's criminal behavior and the criminal justice system's ability to respond to the behavior.
Endnote |
| |||||||||||||||||||||||||||||||||
| Initiatives | Funding | Home | ||||||||
|
Site Map
Help
Account
Cart
Home
