Program evaluation should be a process, not a one-time event,
that begins before the startup of a program. The evaluator should
be selected and the evaluation design developed while the program
is in the design stage; that allows the evaluator and the program
planners to jointly address key evaluation issues, for example,
the articulation of program goals and the identification of appropriate
performance measures, before implementation begins. The collaborative
process, with ongoing communication and exchange of information,
should then continue through the life of the program.
In the real world, arrangements for evaluation often fall
short of the ideal. All too often, evaluators come into the
picture only when implementation is about to begin or, worse,
after it is well under way. The earlier evaluators are brought
into the planning and program design process, the better.
This chapter is not a detailed guide to evaluation strategies
and methods. Rather, it is an overview of ways to evaluate programs
that integrate substance abuse treatment with the pretrial processing
of criminal cases. The following topics are covered in a broad-brush
fashion:
Process evaluations answer questions about how well
a program is meeting administrative and procedural goals and
should generate valuable information about program structure
and operations. Questions typically addressed in a thorough
process evaluation include the following:
How well is the program achieving its goals in terms of provision
of services?
To what extent is it reaching the intended target
population? If the target group is not being reached, why not?
What services (such as detoxification, assessment, treatment,
and aftercare services) are being provided? To which program
participants?
What services not currently provided should be added? What
current services should be dropped or modified? Why?
What problems have been encountered in program implementation?
To what extent, and how, have the problems been overcome?
What are the characteristics of the defendants participating
in the program?
To what extent, and in what ways, do the characteristics
of participants differ from what was anticipated during the program
design stage?
What are the principal treatment and ancillary service needs
of participants?
Do program services reflect the needs of participants?
Does the program provide culturally appropriate services
and staff to address the needs of participants?
Does the program provide treatment services and settings
that can adequately address the needs of the full range of participants?
How does the program affect the work of courts, and other
justice system agencies, and the overall delivery of substance
abuse treatment?
How do program operations affect caseloads and resource
allocation in the court?
How do program operations affect the delivery of treatment
services to clients not involved in criminal cases?
What are the attitudes and perceptions of justice system
leaders and key practitioners?
What are the attitudes and perceptions of treatment providers
and public health officials?
What are the attitudes of community leaders?
How do key stakeholders in the jurisdiction perceive the
effectiveness and value of the program?
Process evaluations will typically use information from a
wide range of sources. The purposes are principally to describe
and analyze actual operations in comparison to initial plans
and to identify key implementation issues for review by program
managers and key stakeholders. The primary audience for a process
evaluation should be the program managers and key stakeholders
in the jurisdiction, but good process evaluations should also
be of interest to policymakers and practitioners in other jurisdictions
that are considering program replication or adaptation. They
can also be valuable in describing the context or environment
in which program outcomes take place, and in that sense are closely
related to outcome evaluations.
Outcome evaluations focus on the goals of the program, both
in terms of its impact on individuals and its impact on collaborating
systems and the larger community. For example, if the program
adapted goals similar to those outlined in Chapter
2, the evaluation design would attempt to measure success
(or at least progress) in relation to those goals.
Experimental designs are the preferred method for outcome
evaluations. Using an experimental design, defendants would
be assigned to one of two groups: (1) an experimental group that
participates in the full range of program activities and is eligible
for all services; or (2) a control group that does not receive
program services or receives only the services that were available
before implementation of the program. The principal advantage
of experimental designs is that a relatively high degree of confidence
can be placed in findings of differences in outcomes between
the two groups.
It is not always feasible, however, to use an experimental
design. First, experimental designs that minimize possible "contamination"
of the experiment can be expensive. Second, they may not be
ethical or legal. For example, a judge, prosecutor, or defense
attorney may object to random assignment of defendants to services,
arguing that it is unfair or a denial of equal protection of
law to arbitrarily withhold beneficial program services from
eligible defendants who have a demonstrated need for treatment.
One strategy that has been used to allay concerns related
to defendants' need for, or rights to, treatment is to assign
defendants to one of several different "categories" of treatment.
For example, defendants might be randomly assigned to a more
intensive group (daily outpatient groups, drug education, and
drug testing), or to a less intensive group (only drug education
and drug testing). This strategy ensures that all defendants
identified as needing substance abuse treatment receive some
services. Experimental designs are generally more tenable when
defendants are not ordered by the court to participate in the
program and when defendants provide a full and informed consent
regarding their involvement in randomization procedures and other
program evaluation activities. Evaluation procedures, including
legal and ethical issues, should be carefully reviewed by judges,
prosecutors, and defense attorneys before an experimental evaluation
design is implemented.
If an experimental design is inappropriate, for whatever reason,
a second approach is to develop a quasi-experimental design that
allows for scientifically rigorous examination of outcomes.
One type of quasi-experimental evaluation is a pre-post design,
in which outcomes obtained following discharge from a drug court
program are compared to those obtained prior to program participation.
For example, defendants' frequency of drug use, frequency of
arrests, or length of time between arrests can be compared before
and after admission to the program. A drawback to pre-post designs
is that they do not take into account other factors that may
contribute to the behavior changes they measure. For example,
a reduction in drug use among program participants might be influenced
by a sudden decrease in the supply of drugs in a community.
Using a pre-post design, this reduction might mistakenly be attributed
to involvement in treatment. In contrast, an experimental design
would demonstrate reductions in drug use among both the "treatment"
and "no-treatment" groups, making it clear what effects on drug
use were attributable to the program.
Another type of quasi-experimental evaluation studies one
or more comparison groups, rather than a randomly assigned control
group. Using this design, results from the experimental and
comparison groups are contrasted. Comparison groups consist
of substance-abusing defendants who do not participate in (or
who do not complete) the drug court program. These groups should
include defendants who have substance abuse problems and who
are otherwise similar to program participants in key areas such
as criminal history, severity of most recent offense, age, gender,
ethnicity, substance abuse history, employment, income, and other
areas used as eligibility criteria in the program. It is often
useful to "match" subjects in experimental and comparison groups
to control for the problem of confounding variables introduced
through nonrandom assignment. Using this approach, both groups'
subjects are selected based on factors expected to affect outcomes,
such as age or educational attainment.
One useful type of comparison group consists of defendants
placed on a program "waiting list." Use of such a group minimizes
differences in motivation for treatment and other potential differences
in program eligibility that may affect evaluation outcomes.
Other comparison groups could include: (1) defendants who are
eligible for the program but elect not to participate; (2) defendants
who are discharged from the program prior to completion; and
(3) other groups of defendants who are similar to program participants
in the areas described above.
In an era of fiscal restraint and close scrutiny of social
services programs, program leaders should expect evaluators to
study the costs and benefits of a program integrating substance
abuse treatment with pretrial case processing. Cost-benefit
analysis builds upon both process and outcome evaluations. The
cost side of the study requires a good understanding of the program's
operating procedures and of the actual costs of those operations
for all the participating organizations. The benefits are the
positive outcomes sought by the program.
Cost-benefit evaluation strategies focus on the economic impacts
of substance abuse treatment before and after involvement in
the program. Cost-benefit analysis must build on a solid foundation
of knowledge about program operations, impact, and costs of the
program as well as criminal justice, health care, and labor market
operations and costs. Such analysis requires time (for follow-up
on participants' post-program behavior), specialized expertise
in justice system and health care operations and economics, and
money to support the research. See Chapter 7
for an example of a cost-benefit analysis.
Evaluations are usually conducted by an outside or "independent"
evaluator -- an individual or group that has no affiliation with
the organizations involved in the program, and is thus less likely
to have a conflict of interest or bias about the merits of the
program. However, it is possible to have objective "internal"
evaluations, and in some jurisdictions there are criminal justice
agencies that have highly competent in-house research departments
capable of evaluating the program.
Regardless of whether evaluators are external or internal,
they must have experience conducting evaluations. Ideally, the
evaluator will be knowledgeable about both justice system operations
and substance abuse treatment. If a potential evaluator has
experience with criminal justice operations but not with substance
abuse treatment, or vice versa, the evaluator should be involved
early, while the program is still in the planning and design
stages, if possible.
The evaluator should be someone in whom the program team has
confidence, because trust and open communications are important
elements of a good evaluation. The evaluator should be willing
to develop strategies that address questions important to program
managers and key stakeholders and to regularly update program
staff on preliminary findings from the evaluation. The evaluator
will need full access to program records and to management information
reports and other documents relevant to program operations and
will need to be able to interview program staff, clients, and
policymakers. There should be full agreement between the evaluator
and the program leaders from the outset, preferably, before initial
implementation of the program, on the goals and objectives of
the program and of the evaluation, on the performance measures
and evaluation methods to be used, and on the procedures, costs,
timelines, and report review procedures to be followed. The
agreement can be formalized in a contract.
The scope, focus, and depth of an evaluation will necessarily
be limited by the funds available. Using a very rough rule of
thumb, evaluation is sometimes budgeted at 5-15 percent of the
overall program budget, but the percentage actually spent varies
considerably depending on the circumstances (and the degree of
interest the policymakers and program managers have in evaluation
of the program). Regardless of the amount of funds available,
close attention should be paid to evaluation priorities and strategies.
The following are some key questions to be asked, by both the
evaluator and the program leaders, in developing evaluation priorities
and strategies:
What are the most important things to know about the program
in terms of operations and/or impact?
What data are needed to answer these questions?
What relevant information is already available?
Given available resources, what additional data can be collected
and, together with data already available, analyzed to produce
answers to the top priority questions?
How will preliminary findings be presented and reviewed,
with opportunity for input from the program team and key stakeholders,
before submission of a formal report?
How can the evaluation findings be used? How can a focus
on these questions, using these research techniques, help shape
program policy and practice?
A range of local, State, Federal and private resources may
be available to help support evaluations of drug court programs.
If funding is limited, assistance from colleges and universities
may be sought. Often, they have social science departments looking
for places to conduct evaluation and research.
Site Map
Help
Account
Cart
Home
