Self-monitoring and evaluation are ways of gaining dynamic, reliable, and up-to-date information
about patient and program characteristics and the frequency and efficacy of services
delivered. This information is necessary to accurately describe patient flow and
service delivery patterns and to examine outcomes. Self-monitoring and evaluation
need not be complicated or excessively time consuming. They do not require
extensive resources.
This chapter is directed not to evaluators and researchers but to personnel
working in the alcohol and other drug (AOD) treatment field who are interested
in applying evaluative tools to their work and moving their programs toward
increased self-monitoring and evaluation.
In the following pages, the concepts underlying effective self-monitoring
and evaluation are explained. A step-by-step guide to evaluation is provided,
which explains how and why each step should be followed. Examples make it clear
how these processes are pertinent to this Treatment Improvement Protocol (TIP)
subject -- matching patients to type and intensity of opioid substitution therapy services.
Instruments are suggested, and a detailed hypothetical example is presented
within the framework that has been outlined. The example demonstrates not only
how an agency can monitor itself on one specific issue (the effectiveness of
treating methadone patients with a high level of psychopathology is the example
used), but also how it can use the information gained from this study to make
changes in its program, and how the effect of those changes can then be evaluated.
A very basic introduction to the concepts of monitoring and evaluation is
outlined here. To understand the distinctions among these various concepts, it
is helpful to look at a continuum of evaluative efforts and what their aims
are. Those wishing a more advanced approach are directed to other sources,
and to another TIP, entitled Developing State Outcomes Monitoring Systems for
Alcohol and Other Drug Abuse Treatment. It provides more detailed discussions
of the principles and goals of outcomes monitoring and their importance in
the current healthcare environment. The National Institute of Drug Abuse (NIDA)
has developed a valuable package, entitled How Good Is Your Drug Abuse Treatment
Program? (National Institute on Drug Abuse, 1993),
which provides practical guidelines.
Self-monitoring, the first point on the continuum, is something a program does to help itself.
It is generally perceived as more modest in goals and execution than self-evaluation
or evaluation done by an external source. Self-monitoring usually refers
to the selection and regular inspection of intermediate indicators of patient
outcomes (which, it is hoped, are predictive of final outcomes).
Self-monitoring employs tools that are nontechnical and easy to use. It utilizes a program's
internal resources to collect and analyze information in a focused area. It incorporates
some kind of feedback system. Optimally, self-monitoring is a continuous part
of a program's operation, with periodic assessment of results and action taken
in response to findings.
Program evaluation is a more elaborate form of monitoring than self-monitoring,
using a more formal methodology. It may be done by internal or external evaluators.
Its objective is also quality improvement.
Evaluation can be done by the State, by a reimburser, by an independent evaluator,
or by a program itself. Evaluation does not have to be an overly ambitious,
expensive, or daunting prospect. It need not be elaborate. It does not have to include
all the elements of an academic research evaluation project, which tests a
hypothesis. Rather, it can be a specific, focused effort to determine whether a program,
or part of a program, is meeting specific goals and objectives.
Evaluation can begin with the basic formulation of a few questions and can then look
for ways to answer them. It can cover a broad range of activities, but persons
wanting to evaluate their program need not be overwhelmed by looking at the full
range. A program can keep it simple, beginning with a small component to evaluate,
then selecting and choosing the questions that -- when answered -- will provide
the most useful information. Any well-conceived effort, no matter how limited,
can contribute to better understanding of treatment practices.
Evaluation is an important and necessary part of the operation of any opioid substitution
therapy program. It gathers information that is useful and based on measurable
data collected by the scientific method, rather than on intuition and impression.
Self-monitoring and evaluation serve to meet patients' needs. Both approaches study active
elements of treatment. By measuring treatment progress and outcomes, self-monitoring
and evaluation can improve the process of patient matching. They can help
identify populations that need specific services and can evaluate the effectiveness
of such services once they are implemented. They can support program and
policy changes and can document the effects of changes on policies and procedures.
They can help with decisions about cost-effectiveness and suggest new
ways of looking at programs. The application of these uses will be illustrated
in the hypothetical example at the end of this chapter.
Regulatory requirements, and questions that arise from them, can serve as triggers
to suggest specific self-monitoring and evaluation needs. In this era of
managed care and limited funding, it is increasingly important to implement a program
of self-monitoring and evaluation. Such a program should be part of an organization's
quality improvement strategy and its accountability to regulators and to the public.
Some external agencies, including reimbursers, seek the demonstration
of effectiveness that evaluation can provide. Evaluation can help programs
satisfy regulatory requirements and document compliance with regulations and standards.
Evaluation can be useful at any point in programming, certainly not only at patient
discharge. This point is particularly pertinent to the long-term, continuing nature
of much of methadone treatment. Periodic behavioral change, not just final
outcomes, can be measured by evaluation.
Most studies collect baseline data and then establish an endpoint that is either
the point of discharge or a point at a specified time after the beginning of
treatment. Such approaches may provide some useful data. However, in a program where
patients may not be discharged or may not complete treatment for a long time, it
is also necessary to look at continual measures of change. This view may
mean revising standard notions of outcomes evaluation, because there are no
outcomes to study other than behavioral change within treatment. It is important
to note that outcome does not necessarily imply finality, even when the patient
leaves treatment.
The results of evaluation can feed directly back into treatment services.
For example, once information has been gathered about baseline rates of drug
use, staff and patients can set goals -- such as the goal of reducing positive
urine tests to a certain level. As data are collected, they can be used to revise
goals as necessary. The goals of evaluation vary greatly among programs. In
setting treatment goals, tradeoffs have to be made between the intensity of service
and the number of patients who can be served. The ideal situation is to provide
services for all those who need them in as intensive an approach as necessary.
However, achieving this goal is usually impossible in practice.
Evaluation also serves an important public education function. Despite the large
body of literature demonstrating the efficacy of opioid substitution therapy,
there remains much public skepticism about whether it works. Sometimes this
skepticism is related to philosophical differences. For example, many people feel
that addiction should be treated only in a drug-free environment. Skepticism
may also be more experiential. Citizens reading in their local newspaper about
drug trafficking in the proximity of a methadone clinic may jump to the unjustified
conclusion that the program is not effective in treating opioid addiction. Thus,
negative opinions are formed.
To the extent that self-monitoring and evaluation can demonstrate that opioid
substitution therapy achieves its goals with a certain number of patients, they can
perform a valuable role in public education. If a study showed, for example, that
of the 200 people served by a methadone program more than 75 percent tested
negative for illicit drug use a year after entering the program, the study would
be a useful demonstration of program effectiveness that could be brought to
the attention of local public officials and the media. Evaluations can be
designed specifically to gather information to present to public officials, legislators,
or State agencies to illustrate specific points.
A primary goal of evaluation in the context of this TIP is to improve the
process of matching patients to opioid substitution therapy services. In many
programs, decisions about matching are often made intuitively. Evaluation can help
to assess the impact of these decisions. It can provide both prospective
and retrospective data to quantify the matching process and can identify patients
needing additional or different services. Evaluation of the matching process can
be valuable in the following areas:
Phases of treatment
Levels of care
Types of services
Identification of specific program elements
Identification of subpopulations.
The public expects certain results from drug treatment in general, and methadone
treatment specifically, including the cessation or reduction of illegal drug use,
the reduction of crime, and improvement in patients' social functioning. Programs
may also want to assess patient progress in treatment. Evaluation is one way
to determine if public expectations are being met and how patients are progressing
in treatment. In some cases expectations may be unrealistic, for example,
that patients will stop all drug use, become abstinent after a certain period,
or remain abstinent once they leave treatment. Evaluation results can provide
a realistic picture of what can be expected from methadone treatment, as
well as demonstrate the extent to which patients are meeting these expectations.
Although the task of establishing evaluation goals may sound straightforward in
the context of opioid substitution therapy programs, it can be quite complex.
Patient characteristics, program philosophy, and State licensing and reimbursement
requirements can influence this process. For example, the duration of treatment may
be prescribed by State policy. In some States, a course of methadone treatment
is limited to 2 years; others support much longer-term therapy that may continue
for a dozen years or more. A program that has time-limited treatment and patient
goals directed toward becoming medication free (methadone free) presents different
issues than a program whose goals focus on achieving a change in lifestyle with
the continued use of methadone (or another opioid substitute). Therefore,
the goal of evaluation must be considered within the context of the goal of
the opioid substitution program.
Evaluations can be used as mechanisms to influence policy change. Data collected can
provide information for reimbursers, who may be inclined to limit treatment, contain
costs, and make services available to more people. When establishing objectives
and designing self-monitoring and evaluation projects, it is critical that
these distinctions be understood and considered and that the goals be clearly
established. This issue is discussed in greater detail in the section of this chapter
on the role of program philosophy in evaluation.
To determine clear objectives, questions that should be asked include
Is the program working as well as it should?
How can it be improved?
How does it compare with similar programs?
Are patients being matched to appropriate treatments?
How satisfied are patients with the services provided?
What happens when changes are made?
Can the program be more efficient and more cost-effective?
One of the sources of the variability discussed above is program philosophy.
Program personnel sometimes do not use known results of research and evaluation
to guide clinical practice because these results support the use of techniques
that differ from those consistent with their own particular program philosophies.
Very strong feelings and emotions are often involved, and the role of
the evaluator is to remain neutral. The effect of program philosophy is one
aspect that should be evaluated and considered.
Issues of program philosophy that must be considered include the following:
Short-term vs. long-term treatment. As discussed above, the length of treatment
relates to the program's ultimate goal. Is the ultimate goal abstinence from any
drug (including methadone), or is it a significant reduction of drug use within
a long-term maintenance model?
Specific goals centered on HIV and AIDS. These goals can involve a number of
areas -- perhaps most critically, the program's discharge policies. For example,
since human immunodeficiency virus/acquired immunodeficiency syndrome (HIV/AIDS)
is such an important factor in treating injection drug users, should patients
be kept in treatment as long as possible with the expectation that this length
of stay will reduce HIV transmission? This question has an increasingly important
impact on evaluation, as well as on treatment decisions and the establishment
of program goals. HIV has tipped the balance of the argument about the duration
of methadone treatment toward liberal retention policies. This factor has
come to play an increasingly important role in program evaluation, particularly
in jurisdictions with a high incidence of HIV infection.
Dosage levels. Dosage levels, which are discussed in greater detail in Chapter 4, are frequently a matter of dispute and controversy. Determination
of dosages may be based as much on philosophy as on clinical guidelines.
This additional philosophical issue must be considered within the context of
evaluation.
Various other program policies and practices may be influenced by philosophy.
In different models, the same data can be interpreted in radically different
ways. For example, consider a study comparing illicit drug use in a program
based on an abstinence model and in a substantial-reduction, long-term maintenance-model
program. In the abstinence program model-program, the finding that sporadic use
occurs in 50 percent of the patients after 2 years of treatment could be interpreted
by some to represent a substantial failure. However, in the long-term maintenance-model
program, the reduction of drug use from 100 percent to sporadic use in 50 percent
of the patients, with the other 50 percent being abstinent, would represent
a major gain.
The evaluation of outcomes and of process are different, requiring different
approaches. Both approaches should be considered since they answer different questions.
Outcomes evaluation focuses on results; process evaluation focuses on
how results were achieved and tends to identify the active ingredients of treatment.
A subset of process evaluation is implementation analysis, which evaluates
the success of an action that has been taken in response to the results of
an evaluation.
Outcomes evaluation focuses on the patient and patient progress (or lack of progress)
during and after participation in a program or some facet of a program. When
looking at outcomes evaluation, terms such as "success" and "failure" should be
avoided; rather, considerations of progress markers and behavioral improvements
should be the guideposts. Even small improvements may be significant, and feedback
can provide important reinforcement to the staff and encouragement to the patient.
For example, a patient outcomes evaluation might measure drug use (as
quantified by the results of urinalysis) in patients with high levels of psychopathology
who are referred for mental health services. This example is discussed in
detail at the end of this chapter.
Researchers measure a variety of variables to assess outcomes of treatment for opioid-addicted
patients. These may include drug use, criminal activity, attitudes, vocational skills,
employment, institutional adjustment, family relationships, and involvement in social
activities. The particular measures chosen should include the behaviors specified
in the goals and objectives of the programs. The evaluation of a treatment
program designed to reduce substance use, decrease criminal involvement, improve
self-concept, and increase job skills must collect data in each of these areas to determine
whether the expected changes have taken place. Other areas can also be measured
to assess the general impact of the program on patients' behaviors or on aspects
of the larger community.
Outcomes evaluation can also measure and assess the impact of a program or an element
of a program, and its ability to produce desired changes in the patients who
are exposed to it. For example, a program might provide bus tokens to its
patients to help them with transportation to and from services. After a certain
period of time during which the bus tokens are provided, an evaluation might be
conducted to determine whether program attendance has improved. This is an example
of a simple evaluation for which only attendance data would need to be collected.
The most reliable evaluation uses a control group for comparison (for
example, a group of patients to whom bus tokens were not provided), but this is
not always a practical approach for programs to take.
New programs provide an opportunity to build in continuous data collection
mechanisms from the start. Programs may choose not to use the data, but evaluation
should be built in. However, it is important not to begin evaluation too early
when implementing a new program or part of a program. The program should become
stabilized before evaluation begins.
Process evaluation. Process evaluation describes what is happening within
the "black box" of the treatment program. "Black box," a commonly used term
in this context, refers to the concept that patients go into a program as
known and described entities and come out with certain measurable outcomes; but
what actually goes on in the program -- or black box -- is not readily apparent.
Process evaluation allows for successful replication of programs that
are achieving their goals, because it looks at the factors that are responsible
for this goal achievement.
By documenting what actually happens over the course of the intervention,
a process evaluation can help interpret the results of outcomes evaluation.
Also, either explicitly or implicitly, the results of a process evaluation
provide an assessment of the strengths and weaknesses of a program and suggest
ways in which the program can be improved, thereby serving as a management tool
for program development. Finally, the documentation produced by a process
evaluation can be used to develop a manual describing the theory and practice of the
treatment program, a helpful guide for others wishing to replicate the program.
Process evaluation has proved to be a useful tool for examining the procedures
of a treatment program in comparison with its stated intent. Through a process
evaluation, the evaluator can determine whether the subjects actually received an intervention
as it was intended to be delivered. The evaluator can also measure the intensity
and duration of the services received by the subjects.
Implementation analysis is a particular type of process evaluation that can be applied
to new programs or new program elements. Implementation analysis should begin
as early as possible in order to fully document the process. It is also useful,
once an evaluation has disclosed that over time, as a result of something new,
certain changes have been taking place -- for example, increased attendance at
counseling sessions since the provision of bus tokens. Implementation analysis describes
how the new program was implemented, considering what the steps were and who
the players were. The analysis looks at such things as planning meetings (when
they were held and who attended), specific problems and barriers that were encountered,
strategies that were used to overcome problems and barriers, and modifications to
the original plan that were necessary.
The benefits and problems of methadone treatment have been studied extensively.
Much can be learned from the evaluation work that has already been done
in this field, including what the greatest needs are for future evaluative
work. Data have been collected to describe effects of opioid substitution therapy
on reductions in injection drug use, other drug use, criminal activity, unemployment,
and other areas. Instruments have been designed to collect these data. A
body of literature focuses on medication levels, treatment program variation,
and factors that are predictive of retention. Some of these studies are discussed
in Chapter 1 and in another TIP State Methadone
Treatment Guidelines.
The existing literature can inform the process of evaluation and program improvement.
It can offer evaluators a sense of what is generally acceptable and examples
of how other programs typically define success rates. The results of earlier
evaluations can suggest ideas about how existing work can be applied or adapted to
particular program needs and how it can guide current evaluative efforts.
There are large gaps, however, in existing work. Only a few studies have been
done of the actual process of matching patients to services. A 1990 report
from the General Accounting Office (GAO) found that none of the methadone maintenance
programs studied "systematically evaluated their effectiveness in treating patients."
Virtually none of the studies have used control groups. Most studies
have high dropout rates (Simpson and Joe, 1993).
Results must be considered within the context of these limitations.
This section offers a general look at evaluation that can be related specifically
to opioid substitution therapy programs and even more specifically to patient
matching within these programs. The methodology that is used in individual studies
depends on the purpose of the specific evaluation, but a number of general principles
are applicable to any evaluation.
The evaluation should not be overly ambitious. A poorly designed evaluation
is likely to be worse than no evaluation at all. The best way of ensuring
quality and achieving goals is to keep the project manageable. The less complex
an evaluation project, the more likely it is to be well planned and implemented.
As discussed in the introduction to this chapter, evaluation can provide data
for both internal and external use. Externally, data can be used to provide
accountability. Internally, data can be a valuable agent for quality improvement. In
the context of the philosophical problems that opioid substitution therapy
programs may encounter and their need to explain their methods and goals, evaluation
can help demonstrate program effectiveness.
Evaluation does not need to be a major undertaking that starts from scratch. Often
data already available can be plugged in to answer the question being considered
for evaluation. Use of available data is discussed in greater detail in the
section of this chapter on the data collection process. One primary fact to remember
in using previously collected data is that the data must be handled consistently
and applied appropriately. In other words, apples shouldn't be compared with
oranges.
In designing and implementing self-monitoring and evaluation projects, it
is desirable to include personnel involved in various aspects of programming
whenever possible. Staff investment in and commitment to the study is critical
for the success of the project. A number of different approaches can be considered.
One is creating a team. Team members can provide a range of expertise and
opinions for the evaluation process, even if no members are actually specialists
in evaluation. An evaluation team might include the members of the treatment
team, including the program director, the medical director, counselors, nurses,
and educators.
Another approach is to contract with an outside agency to perform the evaluation.
Whether the evaluation is done internally or by an outside agency depends
on several factors, including internal resources, available funding, and the
purpose of the evaluation. If it is being done for internal information-gathering
or quality improvement purposes, and program personnel are confident that
the data collection procedure will provide reliable data, an in-house effort
may be sufficient. However, independent (external) evaluations often have
more credibility and may be necessary if an evaluation is being conducted, for
example, to demonstrate effectiveness to payer sources, the State, or other regulators
or interested parties.
Programs located in communities with universities can sometimes gain valuable help
from faculty and doctoral students in designing and implementing research studies.
Ball and Ross (1991) describe four domains that
must be considered in evaluating substance abuse treatment. By considering
each of these four areas for data collection and study, the treatment process
can be broadly conceptualized and insight can be gained into what actually
occurs in treatment. Evaluation strategies may differ for each area, and these
should be considered in a general way before designing an individual evaluation
project. A common pitfall in designing evaluation studies is looking at only one
domain. Once a broad overview is achieved, programs can select the variables that
are most meaningful for their study, assigning primary or secondary values
to them. Each domain is discussed briefly below.
Patients. Information gained from life histories and current health surveys
of opioid-addicted patients has long been valued as a tool for treatment evaluation.
Collection of these data has always been emphasized in substance abuse
treatment, although the focus of treatment has changed somewhat through the years
as addicted patients have been considered increasingly within medical and
psychiatric contexts.
Programs. Ball and Ross (1991) describe the domain
of the treatment program as "conspicuously neglected" in most substance abuse
treatment evaluations. They list four program areas that should be considered:
Setting or environment
Policy
Staff and leadership
Physical facility and other resources.
Services. In looking at the actual provision of treatment and adjunct services,
evaluators should consider
Methadone dosage
Clinic hours
Number and type of counseling sessions
Medical and psychiatric services
Educational and vocational services
Urine screening
Any other services that are provided.
Outcomes. Common outcome criteria that have been studied include measures of drug
use, criminality, treatment retention rates, and employment status. The study
of outcomes is discussed further in the next section of this chapter.
Determining exactly what you want to find out from your evaluation involves a number
of steps in order to refine and focus the evaluation questions as much as
possible:
Arrive at a broad goal statement. Why do you want to know the answers to the
questions you have posed? What is the goal of obtaining this information?
Move from articulating the general goal statement to defining specific objectives.
A general goal might be to improve patient outcomes. This goal must be
narrowed to a specific, feasible project that can be measured through evaluation.
Identify the question as either a process or outcome question. Examples of outcome
questions are, "Will providing childcare reduce program dropouts among parenting
women?" or "How effective is the program in treating patients with high levels
of psychopathology, and how often are these patients referred for mental health
services?" An example of a process question is, "What are the problems and barriers
in providing mental health services for patients with a high level of psychopathology,
and what strategies can be used to overcome these problems?"
Look at what others have learned about this question through previous research.
The National Clearinghouse for Alcohol and Drug Information (NCADI) is
a useful resource for this purpose.
Determine the relevance of previous research to your specific evaluation project.
What you find in existing research may help further define your evaluation
questions and identify instruments that may be used to collect data.
As previously described, questions about methadone therapy have been answered
in a number of studies through the years. For example, it is well documented
that the longer patients stay in treatment, the better they do; likewise, the
more services they receive, the better patient outcomes will be. Instead of
attempting to look at these broad questions that have already been answered, most
evaluators would do well to focus their studies more narrowly and look, for example,
at questions such as
How can patients be retained in the program for the first 90 days (the period
critical to their becoming engaged in treatment)?
What are the reasons given for dropping out of treatment?
What is the range of services that patients receive?
How does the program use the results of urine screens?
Some factors that will help determine what data are needed include whether the
evaluation being done is a process or outcomes evaluation, which domains are being
covered, and whether an implementation analysis is involved.
The data that are collected should be directly relevant to the question and
should consider all domains appropriate to the question. When considering how
well patient matching procedures work, the approach should consider both program
and patient variables. As Ball and Ross (1991)
have pointed out, there is often a tendency to look only at patient data
when evaluating. But it is important to consider the other areas that are appropriate
to the question, such as program setting, services, staffing, and so forth.
The specific variables about which data will be collected will depend on the
question. Variables may be quite diverse and may include patient, program, and outcome
variables. For example, patient variables may include gender and ethnicity; program
variables may include staffing ratio, number of scheduled counseling appointments,
and dosage levels; and outcome variables may include rates of positive urine
tests, patient satisfaction, and compliance with scheduled counseling appointments.
A word of caution about variables: If the number of variables under consideration
for a planned evaluation seems to be approaching an unmanageable level, it
is likely that the project is overly ambitious. This is a clue to reconsider
and reduce the scope of the original question.
Next, it must be specifically determined what data are already available that
pertain to the selected variables. What relevant information is already in program
records? What existing data are available elsewhere, where and in what form, and
how can these data be abstracted? What, if any, are the barriers to obtaining
it? Are there confidentiality issues that affect extracting and collecting
data? Are existing data in a form that can be used for comparative purposes
with the data that will be obtained from this evaluation? One possible starting
point may be to establish a uniform data set, if this does not already exist.
A program may standardize assessment forms or instruments as well as indicators
of ongoing progress.
When possible, data to be used in evaluations should be computerized. Computerization
creates a database that can be used in subsequent projects, and provides a format
for quantifying information for the current project.
Existing resources may include
Program data (for example, patient records, results of urine testing, and administrative
and fiscal information)
The National Drug and Alcohol Treatment Unit Survey (NDATUS)
The next step is to identify the population or process that is to be evaluated.
This identification depends on the evaluation question. For example,
does the evaluation concern new patients coming into a program and how they
will be matched to treatment? Or is it a look at outreach to a critical population
such as addicted persons who are human immunodeficiency virus (HIV) positive,
pregnant women, polydrug abusers, or people with disabilities? Or is it limited
to assessing how existing patients are being matched to new or different services?
It may be a "what" rather than a "who" that is being evaluated. In the case
of a State evaluation, the "whats" may be programs or elements of programs --
for
example, program procedures such as intake or patient matching; program elements
such as specific interventions, staff ratios, and characteristics; administrative
procedures such as payer mix or dollar costs; or patient movement through phases of
treatment.
After these determinations have been made, it is then possible to decide on the
sampling strategy. Factors influencing this strategy include the scope of the research
question, the resources, and the size of a program. A study of a small program may
include everyone in a program. In other evaluations, for a number of reasons such
as resources and manageability of the data, it may be necessary to sample
only a portion of the population to be studied. If an evaluation is being conducted
at the program level, the best procedure may be to sample among programs.
Sampling strategies also depend on the problem to be studied. For example, in a
study focused on the program's success in administering purified protein derivative
(PPD) skin tests for tuberculosis (TB) or on completing a physical examination
on admissions, every patient admitted over a specified period of time might
be sampled. On the other hand, if the study is focused on evaluating the
frequency of opioid- or cocaine-positive urines, a random sample of persons who have
been in the program for less than 6 months might be compared with a random sample
of those who have been in the program for more than 2 years. Or, the rate
of positive urines in the sample of newer patients could be compared with
the overall rate of positive urines in the entire clinic.
It should be determined whether the evaluation will require approval by an
institutional review board concerned with protection of human subjects and if patient
consent is necessary. Care must be taken to ensure that consent is truly informed;
the fact that patients are dependent on the opioid substitution services
might give program staff undue influence over patients. This influence should
not be abused in the evaluative process. Also, confidentiality must be ensured
whether data are stored in a computer or in a paper file. Any other relevant ethical
issues must also be considered.
Data must be collected in an empirical, quantitative form. The first step in
data collection is the identification or development of an instrument to be
used for this purpose. The program may already collect data in a form that
is amenable to the evaluation project, such as the results of urine tests
or attendance records.
The development of a data collection instrument involves several steps. It
is necessary to devise data fields, draw up a draft of the instrument, pilot
test it, and then revise it based on results of the pilot study. Although this
sounds complex -- and is, in some cases -- it can also be quite simple. The instrument
may be just a single sheet with a few boxes on it. Piloting the instrument
may simply involve a counselor's asking questions of a few patients. This
procedure does not need to be a daunting, complicated process.
Certain guidelines must be followed so that the information collected is congruent
with other existing resources. This is part of the challenge of putting together
an instrument, and one of the reasons that all but the most experienced evaluators
should probably use (or adapt) existing instruments. For example, if the instrument
is to include an ethnicity category, it is necessary to be aware of the various
ways to categorize ethnicity. Another example is drug use categories. An instrument
may use the term narcotic to refer to opioids; another may use legal definitions
and include cocaine or even marijuana under the narcotic category. Some consider
alcohol a sedative-hypnotic drug, and others consider alcohol in a separate category.
If questions are developed, categories should be parallel to those used
in other data sources, so that collected data will be comparable with the
other data.
A program developing a complex data-collection instrument should use the
services of evaluation experts and consultants whenever possible. Many existing
instruments may be applicable. For example, the Addiction Severity Index (ASI) or
sections of the ASI may be useful. Several existing personality inventories may
also be useful. Serious consideration should be given to using existing instruments,
or portions of them, if they are appropriate. Sometimes only a few modifications
are necessary to adapt an instrument to a program's particular evaluative purpose.
Adapting an instrument is usually much easier than trying to invent a
new tool.
A wide variety of such instruments is available. NIDA publishes a comprehensive
list of existing instruments in the Diagnostic Source Book on Drug Abuse Research
and Treatment (Rounsaville et al., 1993). Another
useful guide is the DATAR Forms Manual: Instruments Used for Data Collection
(Simpson, 1992). DATAR is the Drug Abuse Treatment
for AIDS-Risk Reduction project, which included an extensive evaluation component.
The project is reviewed in a recent article by Simpson and associates
(Simpson et al., 1995).
Part of designing an evaluation includes deciding with what frequency and over
what duration of time data will be collected. If data are to be collected more
than once, it is possible that more than one instrument will be necessary.
These questions relate to whether the evaluation is a one-time look, or a survey
of changes over time. If it is a one-time look, these questions are not applicable.
Training staff members to perform self-monitoring and evaluation does not have to
be extensive or complicated. However, for data to be useful, a certain amount
of training and staff support is necessary to standardize procedures and assure
quality. Data must be collected using standardized methods in a way that protects
its integrity. A written protocol explaining exactly how data should be collected
is usually the best way for staff to understand the procedures to be used.
Although all staff must participate, it is often best to assign responsibility
for collecting data to one or two staff members.
Another purpose of training is to address concerns of staff members about the use
of the results of the evaluation. There are no "right" or "wrong" results
of evaluation. However, program personnel know that certain findings may
reflect poorly on a program and may even jeopardize funding or accreditation.
Staff members must be taught to use results -- whatever they may be -- in a proactive
way, bringing problems out into the open and addressing them. Evaluation presents
an opportunity for a program to move in a positive direction, perhaps seeking
outside help from State technical assistance services. Programs that incorporate
self-monitoring and evaluation information into a plan for improvement will be looked upon
positively by the State, by regulators, and by reimbursers.
It is also important that procedures be developed to address specific points
when conducting an evaluation. Interviewing is one area that often must be
addressed. When interviewing patients is part of the evaluation process, procedures
for interviewing should be prescribed. To some extent, the structure of the
interview will be determined by the instrument used. Data collected in the interview
should be quantifiable, and staff should receive training in conducting the interview
in a way that will elicit data that can be quantified.
The interview site is another important consideration. The decision of whether
to conduct interviews in a waiting room or in a private office, for example,
could influence the way that patients respond to questions. Another issue to
consider is how to record or account for patients who refuse to be interviewed.
Also, if the work involves abstracting information from existing patient
records, it must be determined when this work can best be done so that records are
still available for staff to use for routine purposes.
When all the data are gathered, standardized procedures are used to code, to
enter, and to store data. These procedures require staff training and monitoring
to ensure consistency and quality. Ever-increasingly, this type of work is
being performed on computers, and the use of a computer is necessary for a large
or complex project. However, it is important to note that computers are not
essential for small-scale self-monitoring and evaluation. For some evaluations,
a simple checklist is all that is necessary. If you are doing a study of
drop-out rates, for example, your data may consist of simple, hand-drawn line graphs.
The results are to be found in the data that have been collected through the
evaluation project. If data have been accurately defined and collected, they should
provide at least some answers to the questions originally posed.
For example, suppose a program wants 100 percent compliance with PPD skin testing
for TB. Investigators review 20 consecutive intakes over a 2-month period
and find that five persons did not receive a PPD skin test. Corrective action
is taken, and the study is repeated in 3 months. Consider another example:
The program wants to be certain that all patients with more than one opioid-positive
urine test a month are receiving adequate methadone doses. Records are reviewed,
and all patients who have been in the program for more than 6 months and who
have had more than one opioid-positive urine a month are identified. The treatment
plan of any patient in this group whose methadone dose is below 60 mg is reviewed
by a physician, the counselor, and the treatment team, and appropriate adjustments
are made.
A data analysis strategy determines how the data should be aggregated and
presented. Generally, results are compiled in an understandable format, which can
vary from a simple percentage, bar, or line graph to sophisticated presentations.
Usually some kind of narrative summary accompanies the graphic display
of the results. Means or percentages must also be generated to make the statistics
meaningful.
The process of understanding evaluation results should involve consultation
with experts and discussion among program staff. Objectives that were described
in designing the evaluation and the interpretation of the statistical data
are both factors in determining what the collected data mean and how the data
will be used.
The first point to be determined is whether the results answer the question
that the evaluation attempted to address. Results may be clear, but just as
often they may not be. Even in well-designed and well-executed evaluations,
results may be inconclusive, which should not be discouraging to evaluators. Such
results may lead to additional questions.
The program may want to involve a staff member or outside consultant with experience
in statistical interpretation to help with interpreting the results, particularly
when they are not clearcut. The sensitivity, background, and experience of
the evaluators who have collected the data and who prepare the presentation
of the material will affect the interpretation and understanding of the results
of an evaluation. Analysis can be a complex process. Even if results are
clear, implications may be less apparent. Questions of statistical and clinical
significance must be considered. Such considerations can be guided, in part, by comparisons
to previous studies that have been conducted on the same or similar topics.
Again, most caveats about interpretation and implications apply primarily to the
more difficult and ambitious evaluation projects. Evaluators should not be
intimidated by possibilities that may seem daunting. Often results are clear and implications
are obvious. For example, consider the hypothetical illustration of distributing
free bus tokens in an attempt to decrease patient dropout rates in the acute
phase of treatment. Baseline data show that before bus tokens were distributed,
the dropout rate for patients in the first 2 months of treatment was 30 percent.
After 3 months of tokens distribution, the dropout rate for patients in
the first 2 months of treatment was 15 percent. The distribution of the bus
tokens was the only change in the program. Without a control group, interpretation
of findings cannot be absolute. But it is fairly obvious that the results
of using bus tokens were encouraging, and it is a practice that the program
would want to continue, if feasible.
If the dropout rate had gone from 30 percent to 25 percent, the results would
have been less clear, and more subjective judgments would have been necessary
about the value of the intervention. Sometimes other factors must also be considered.
For example, suppose that the first 2-month period (before bus tokens
were distributed) was an exceptionally rainy period, while the sun shone nearly
every day after the token distribution began. It is possible that the weather
may have had as much influence on program attendance as the distribution of
bus tokens. Or suppose that another factor had been introduced in the same
time period, such as child care for patients with young children. These are
elements that must be weighed by evaluators and then factored into interpretation
of the study. As well, decisions must be made about any action taken as a
result of the study and the design of future evaluations.
The implications of a more complex study are explored in the section of this
chapter outlining a hypothetical evaluation of the need for mental health services
for methadone patients with a high level of psychopathology.
The final questions of the evaluative process are: How can the data be applied?
How can the program be improved on the basis of this evaluation? Once
it has been decided that evaluation results are valid, the results of evaluation
provide program personnel with a foundation for taking their next step.
Programs are constantly making changes, and the purpose of self-monitoring and evaluation
is to make these changes part of a more conscious and reflective process.
When a change is contemplated, consideration should be given to what the
result will be if that change is made. This is a basic purpose of evaluation
-- make a change, look at what happens when the change is made, decide whether
the results are sufficiently successful to continue, and, if so, institutionalize
the change.
Perhaps the most important point is that evaluation results should not just be
filed away but should be used in some manner. Some results will point to recommendations
for change. In some cases, positive results can be used to publicize the accomplishments
of a program. Other findings will indicate the need for further evaluation.
Some evaluation results can serve as a basis for discussions with funders
or can provide the foundation for funding applications.
Clearly, there are many different options to consider when deciding what steps should
follow evaluation and self-monitoring and what kinds of changes should be made.
A general rule is to begin with the least invasive intervention, which
might be as simple as releasing the data. Other options might include establishing
new policies, changing regulations, applying sanctions, or implementing certain
educational requirements for staff.
Several examples illustrate the "What is next?" step. Suppose a program wants
to find out whether there is a correlation between the amount of group counseling
a patient receives and whether the patient stays in treatment. Once the
data are gathered and analyzed, it becomes clear that increased numbers of group
counseling sessions are correlated with retention in treatment. That information
can provide the foundation for an education program to inform program directors
of the findings and encourage programs to increase their group counseling
sessions. From there, an implementation analysis (a form of process evaluation)
can be conducted to determine if this intervention is actually having an impact
on the amount of group counseling that is available in the various programs.
This hypothetical evaluation of counseling is an example of an evaluation in
which the unit of analysis is the program, not the patient (in contrast to the
bus tokens example). In studying programs, it is important to be aware of
variables that may influence outcomes, such as the size and age of the program, staffing
patterns, or other internal components.
Another example of how evaluation data might be used to move programs in a certain
direction can be seen in the case of a study in a midwestern State that showed that
when methadone dosages were above certain levels, rates of illicit drug use
by patients were lower (as measured by urine screens) than were rates when
lower methadone dosages were used. Knowing that average dosing levels were considerably
lower than the cut-off rate in the study, State officials began a campaign to
educate providers and policymakers that lower rates of illicit drug use were associated
with higher methadone dosing levels. This campaign included distribution of
articles explaining the study, as well as presentation of conferences and workshops
for practitioners to further disseminate the information.
A number of hypothetical examples have been referred to throughout the preceding
discussion of self-monitoring and evaluation. The following example1 is a more complex exercise. Exhibit 5-1 shows
the practical application of the eight steps of evaluation to a specific example
of an inhouse evaluative project. Based on the results of the program then
instituted, certain program changes are implemented for a period of time, and the effectiveness
of the intervention is then evaluated, as shown in Exhibit 5-2.
Based on the results of the first evaluation, this hypothetical program then
instituted the following actions:
Routine screening of all patients on admission and every 6 months, using the psychological
status section of the ASI.
Establishment of formal liaison with a local mental health agency to ensure rapid intake
for methadone patient referrals.
Assignment of case managers to monitor the followup to ensure that mental health assessment
this and treatment actually occur.
Six months after initiating these changes, the program conducted a second study
to evaluate the effectiveness of the intervention.
Self-monitoring and evaluation are important components of any opioid substitution therapy
program. Whether a program employs self-monitoring for regular self-inspection
or uses evaluation to solve a problem or to gain a better look at the effects
of change within a program, provisions for monitoring and evaluation are a
necessity to programs concerned about patient-treatment matching. Self-monitoring
and evaluation allow a program to engage in continuous quality improvement.
Designing and implementing an evaluation component need not be extensive or complex.
In fact, simple and manageable projects that address the interests of
a specific program are generally more successful than projects that are overly
ambitious. Evaluation can be used to identify service needs, assess the effectiveness
of services, support program and policy changes, document the effects of procedural
changes, assist in decisions about cost-effectiveness, and provide information for
public education. Regardless of the program's area of interest, consider a systematic,
step-by-step approach when designing the project. Include the four basic
domains -- patients, programs, services, and outcomes -- to provide an
overview of what actually goes in the treatment program. Use available data and
existing instruments whenever possible. Work as a team; staff investment in and
commitment to the study is a key for a successful project.
Site Map
Help
Account
Cart
Home
