|
This document was developed under the direction of Soledad Sambrano, Ph.D., of the Center for Substance Abuse Prevention (CSAP) through contract #277-95-5002 with EMT Associates, Inc., and ORC Macro (formerly Macro International, Inc.).
The principal authors of this document are J. Fred Springer, Ph.D., of EMT Associates, Inc.; Soledad Sambrano, Ph.D., of the Center for Substance Abuse Prevention; Elizabeth Sale, Ph.D., and Rafa Kasim, Ph.D., of EMT Associates, Inc.; and Jack Hermann, Ph.D., of ORC Macro. Special thanks to Paul J. Brounstein, Ph.D., and Fred Seitz, Ph.D., of CSAP, Charles Turner, Ph.D., Wei Pan, Ph.D., David Cordray, Ph.D., Will Shadish, Ph.D., Chrystalla Ellina, Ph.D., and Mary Nistler, M.P.P.A., for their important contributions.
Special thanks to the CSAP staff who managed the projects and the 48 grantees who participated in the study. Also appreciation to Juana Mora, Ph.D; Craig Love, Ph.D.; and Jane Maxwell, Ph.D. for their review of these documents.
All material appearing in this report is in the public domain and may be reproduced or copied without permission from the Substance Abuse and Mental Health Services Administration. However, this publication may not be reproduced or distributed for a fee without specific, written authorization of the Office of Communications, SAMHSA, U.S. Department of Health and Human Services. Citation of the source is appreciated. Suggested citation:
Substance Abuse and Mental Health Services Administration. The National Cross-Site Evaluation of High-Risk Youth Programs. Center for Substance Abuse Prevention, DHHS Publication No. SMA-003375. Rockville, MD, 2002.
Copies may be obtained, free of charge, from the National Clearinghouse for Alcohol and Drug Information (NCADI). NCADI is a service of the Substance Abuse and Mental Health Services Administration (SAMHSA). For copies of publications, please write or call:
National Clearinghouse for Alcohol and Drug Information (NCADI)This publication can be accessed electronically through the Internet World Wide Web connection listed below.
World Wide Web Internet connection: http://www.samhsa.gov
Originating OfficeThe Center for Substance Abuse Prevention (CSAP) in the Substance Abuse and Mental Health Services Administration (SAMHSA) is the Nation's lead agency for substance abuse prevention. The Center funds community-based organizations, universities, behavioral health providers, and public agencies to identify effective prevention programs and practices and disseminates findings, program models, and other prevention materials to practitioners and policymakers across the country. This document summarizes the findings of CSAP's National Cross-Site Evaluation of High-Risk Youth Programs. This large multiple-site evaluation was designed to assess 48 prevention programs and to identify those program characteristics that are associated with strong substance abuse prevention outcomes. The rigor and relevance of The National Cross-Site Evaluation earned CSAP the 2000 Outstanding Study of the Year Award from the American Evaluation Association.
The National High-Risk Youth Demonstration (funded from 1987 to 1995) has been one of the most ambitious and productive of CSAP's funding initiatives. In its early years, the demonstration focused on identifying promising approaches to prevention. As the demonstration matured, individual site and cross-site research produced knowledge about risk and protective factors related to substance use and helped to identify model programs for effective prevention. Research and experience in the High-Risk Youth Demonstration also contributed to the awareness of the importance of culturally sensitive, age-appropriate, and gender-specific programming. This progress in prevention theory and practice laid the foundation for the research reported in this document, the largest and most comprehensive of CSAP's High-Risk Youth studies.
In addition to individual-level information on substance use, risk, and protection, the evaluation includes detailed information on the nature and amount of prevention services in which each child participated, as well as systematic process information on the study programs. This monograph highlights the study's contributions to the growing evidence that "Prevention Works." More specifically, this document summarizes major findings concerning the effectiveness of prevention programs in reducing rates of substance use among youth at high risk. The analyses identify important characteristics of program setting (e.g., implementing organization, and the setting in which the program is delivered) that condition program effectiveness, and the approaches to program content and delivery method that have contributed most to effective programming in these real-world prevention programs.
This document is part of a series of Points of Prevention publications that document the contribution of CSAP's National Cross-Site Evaluation of High-Risk Youth Programs to prevention knowledge and provide science-based guidance for improved prevention policy and practice.
Charles G. Curie, M.A., A.C.S.W.In the last decade, prevention of substance abuse has become a standard part of the awareness and practice of teachers, youth workers, and community-based organizations. Beginning with the visible outbreak of illicit drug use among youth in the 1960's, awareness of the risk, harm, and reduction of opportunity associated with alcohol, tobacco, and drug use among young people has increased dramatically. Public agencies and private foundations have supported research that has produced important insights into the initiation and prevalence of substance use among youth and into its association with poor educational, behavioral, and social outcomes. Practitioners and evaluators have developed and tested prevention strategies and programs that contribute to positive youth development and reduce substance use. Social institutions and private citizens have made the prevention of substance use an integral part of efforts to support and promote positive future outcomes for America's youth. Progress in providing prevention is evident.
As concerned professionals and citizens have recognized the need for prevention programming, they have also become more concerned about how to make their prevention activities as effective as possible. Many of their questions focus on the design of program content and delivery. What types of learning experiences or activities are most effective in preventing substance use? What are the most effective ways to deliver these activities? Are different organizational arrangements more or less effective in providing prevention services to youth?
Scientific study is important to answering these questions and developing guides to effective programming. However, prevention professionals and volunteers also need to know how to make effective prevention practices work in real community settings, typically with limited resources. Designing and implementing effective prevention programs depends on science-based knowledge tempered and shaped by experience and success in actual community settings. The bulk of prevention research has concentrated on school-based programs for the general youth population (Paglia and Room, 1999). Relatively little information is available to prevention practitioners about effective programming for high-risk youth.
This monograph is one in the Points of Prevention series, which reports findings from a landmark evaluation that is uniquely suited to provide information on the design and implementation of prevention programs for high-risk youth in actual community conditions. The National Cross-Site Evaluation of High-Risk Youth Programs is a large study by the Center for Substance Abuse Prevention (CSAP) within the Substance Abuse and Mental Health Services Administration (SAMHSA). The study includes 48 programs originally funded by CSAP in 1994 or 1995 to implement prevention programs for youth at high risk for substance abuse. These programs were not selected as model programs, nor were they implemented specifically for the purposes of the study. The findings for these programs are not the product of successful studies already published. These are actual programs implemented by a variety of organizations with different resources, staffing, and experience. The programs were implemented in communities at high risk and represent the experience and challenges of programs in these conditions. Not all of the programs succeeded, and of those that did achieve their objectives, some were much more successful than others. In short, this study provides the opportunity to learn about a range of program experience in actual community conditions, and to learn what design and implementation features contribute to effectiveness in reaching prevention objectives.
Designing and Implementing Effective Prevention Programs examines the experience of 48 programs providing prevention services to youth at high risk for substance abuse in diverse communities across the country. The monograph presents information and findings on several major issues that prevention workers must consider in making their programs effective.
The following section briefly introduces CSAP's National Cross-Site Evaluation, focusing on the way in which program effectiveness is measured, and the ways in which comparable information on program design and implementation was gathered. Subsequent sections present major findings and lessons concerning how to design and implement effective prevention programs for high-risk youth.
The National Cross-Site Evaluation of High-Risk Youth Programs has produced a unique collection of information on prevention programs, the youth who participate in them, and the effects of this participation on substance use and factors that protect youth from substance use. The study includes extensive questionnaire information collected from approximately 6,000 youth who participated in prevention programs funded by CSAP, and approximately 4,500 similar comparison group youth who did not participate in the CSAP programs. Data were collected from these youth at four points in time, including program entry, program exit, 6 months after exit, and 18 months after exit. This large data set on individual youth is the focal point of analyses presented in other monographs within the Points of Prevention series.
The National Cross-Site Evaluation also produced extensive and detailed information about the organization of the 48 programs in the study sample, and on the services they provide to youth. This program-level information, plus information about the effectiveness of study programs in achieving intended outcomes, provides the basis for the findings and recommendations on how to design and implement effective prevention programs that are presented in this monograph.
Accurate and detailed information on program organization and content was critical to achieving the National Cross-Site Evaluation's full potential for contributing to knowledge about what works in prevention programming. In single-site evaluations, this program-level information is part of the description of the program being studied. In a multisite study such as the National Cross-Site Evaluation, these measures are potentially important explanations of differences in program effectiveness. The analyst can ask whether certain characteristics of programs are associated with greater effectiveness in achieving intended outcomes. For this study, program-level measures were constructed from systematic data collected and organized in the following ways.
Site Descriptions. The National Cross-Site Evaluation worked closely with each of the 48 program sites to clearly describe their organization, management, and services in comparable ways. Program managers and staff, as well as representatives of the organizations that administered or collaborated with the programs, were interviewed extensively in 2- to 3-day site visits to each site, and in followup interviews by telephone. Program documents such as proposals for funding, lesson plans, and curricula were studied, and program activities were observed when possible. The detailed information from these site visits was used to code more than 400 closed-ended variables profiling the community and organizational environments of programs, management and staffing, program objectives and prevention strategies, planned program services, and participant recruitment. Narrative descriptions, elaborations, and explanations were added to the coded site profiles to capture the unique circumstances and context of each program.(1) To provide an overview of each program site, brief case descriptions were prepared using a common outline.
(1) For a complete description of the site visit protocol and the resulting database, see the National Cross-Site Evaluation of High Risk Youth Programs Final Technical Report, available through the Center for Substance Abuse Prevention.
Service Profiles. The National Cross-Site Evaluation collected information of unprecedented detail and completeness concerning the services received by youth participating in each program site. The process of conceptualizing and collecting these "dosage" data was complex and careful. Working from program proposals and preliminary site visits, six categories of program content (prevention strategy), and four categories of service delivery method were developed by the evaluation team. Utilizing these coding categories, detailed "dosage coding plans" were developed for each program. These coding plans mapped all planned program activities so that each activity was assigned to a prevention strategy and a delivery method category. Intervention activities were organized so that several different interventions could be coded within a program session, allowing detailed documentation of the amount and type of contact each participant youth had with their program. Contact information was coded for more than 217,000 interventions across the 48 programs.
These data provide rich and detailed information that was used to develop the program and site characteristics analyzed below. It is this program-level data that provides the unique content of this monograph. They allow us to assess the ways in which differences in program setting, design and implementation affect the achievement of program objectives. The following section reviews the outcome measures developed to assess the degree to which the programs achieve their major objectives.
The National Cross-Site Evaluation provides an unusual opportunity to assess the effectiveness of a broad range of programs implemented under real community conditions. The cross-site programs were not selected because they were model programs. The programs in the study represent the day-to-day challenges of design and implementation confronted by prevention workers in the field.
Substance Use. While many of these programs had specific objectives that were relatively unique to their design or their target population, they all shared a major objective to reduce the levels of substance use by participating youth as compared to comparison youth who are similar in circumstances, demographics, attitudes, and behaviors, but did not participate in a CSAP prevention program. To assess the effectiveness of the programs in achieving this overall objective, four measures of substance use are employed in the following analyses as program outcomes.
Substance Use Attitudes. While reductions in substance use are the ultimate objectives of the CSAP programs, changes in other factors that are thought to be related to substance use are common intermediate outcomes. Accordingly, many of the study programs were designed to change the attitudes held by youth regarding substance use. The Cross-Site Evaluation captured data on this traditional prevention program objective.
Risk and Protection Factors. In response to the CSAP solicitation for funding proposals, each of the programs in this study was designed with recognition of the syndrome of related circumstances and behaviors in youth referred to as the risk and protection framework. Unlike the common focus on reducing substance use, the programs varied substantially in the specific definition and targeting of the risk and protective factors they intend to impact. It follows that there may be less commonality across programs in impacting risk and protection than in impacting substance use.
Nevertheless, the analysis presented here assesses the degree to which study programs impacted two important protective factors-school connectedness and family connectedness. These variables were selected because analyses of the cross-site data that are reported elsewhere(2) identify these major protective factors as central to creating, supporting, and sustaining opportunities for positive development. These variables, representing the development of positive and rewarding connections to important social environments, were shown to be highly related to other protective factors, and to substance use. One or the other of these major variables was often the direct or indirect target of the 48 programs in the study sites. Because those programs focused their attention on working directly with youth rather than with environmental influences, the primary impacts of the programs on connectedness would come through building more positive orientations to environmental influences (e.g., family and school), and through improving their competencies in behaving in a meaningful and fulfilling manner in these environments.
(2) See Points of Prevention Monograph #2, Understanding Risk, Protection and Substance Use Among High-Risk Youth.
The fundamental question asked in evaluation is, "Did the program work-did it attain its objectives?" For policymakers and prevention practitioners, the important answers are not about a single program but about many programs in a variety of communities, offered by different organizations, and serving different types of people. Policymakers and practitioners know, from the literature on evaluation and from experience, that some programs do attain their objectives and some do not. They are less certain about the portion of existing programs that succeed or why some programs do better than others. This section presents information that will help reduce this uncertainty and provide the type of uniform information over multiple programs that decision makers need.
In the National Cross-Site Evaluation, 48 programs provided services designed to reduce substance use and strengthen internal protective factors in the participating youth. Past research and experience indicates that these programs would experience a range of success in their programming-that some would be more successful than others. By determining a statistical effect size for each program, we can identify the percentage of study programs that are effective with respect to a particular program outcome or combination of outcomes.(3)
(3) As used here, effect sizes are simply the difference between participant group change and comparison group change on the outcome measure for each program, standardized to make it comparable across the different programs. For a complete discussion, see the National Cross-Site Evaluation of High Risk Youth Programs Final Technical Report, available through the Center for Substance Abuse Prevention.
As shown in Figure 1, the proportion of programs that achieved success varies on the different outcome measures. An effect size of 0.20 or above is used to determine success.(4) The length of the bars indicates the percentage of programs with positive effect sizes of 0.20 or above.
(4) While the analyses in this monograph utilize the full range in effect sizes across programs, the standard of .20 to .49 constituting a small effect (but one attributable to program), .50 to .79 constituting moderate effects, and .80 and above constituting a large effect is adopted as cutpoint criterion of effectiveness. The typical experience of prevention research, and the restricted variance in the substance use measures in this study (a large portion of non-users, particularly in the younger ages) justifies the .20 standard as appropriate for important program effects.
The distributions in Figure 1 support several findings.
In review, the distribution of findings on outcomes for youth while they are participating in their respective programs offers positive results. More than one-third (35%) of the programs produce a meaningful positive effect on one of the three substance use outcomes. If the risk and protective factors and attitudes are considered, almost two-thirds (60%) of the programs produce a meaningful positive result in at least one major outcome area.
The findings presented above indicate that programs frequently may have meaningful positive effects in one or a few outcome areas, but not in others. While this pattern suggests that meaningful outcomes may reflect particular program emphases in one area and relative lack of emphasis in others, it does not suggest that success in one area does not tend to go with success in another. For example, although every program that achieves a meaningful change in attitudes toward substance use may not achieve a meaningful change in the effect size for substance use itself, it may be that there is a tendency for use patterns to move in a corresponding positive direction when attitudes change.
The relationships among effect sizes of the six outcome variables are displayed in Table 1. Higher numbers indicate a stronger tendency for the effect sizes to move together.
Note: Effect size variables are non-normally distributed, therefore the
Spearman's rho correlation coefficient was used.
*Correlation is significant at the 0.05 level (1-tailed). **Correlation is significant at the 0.01 level (1-tailed).
The pattern of association among program effects on the different outcomes supports several important observations.
The inconsistency in effect sizes for substance use attitudes with other outcomes has important implications. First, it again highlights the potential fallacy of assuming that behavior follows expressed attitudes. Clearly, professed attitudes changed in many programs without accompanying changes in behavior. Second, these findings suggest that time devoted to activities that target change in attitudes about substance use would be better spent on activities designed to impact risk and protective factors, particularly those related to building the connectedness of youth with meaningful social environments. For this reason, program effects on substance use attitudes are not included in subsequent analyses in this monograph.
The following section introduces a simple organizational framework for thinking about the program setting, design, and implementation factors that shape programs and may have consequences for program effectiveness. Subsequent sections present evidence from the Cross-Site Evaluation concerning those setting, design, and implementation characteristics that are associated with more effective programming.
This monograph focuses on information that is useful to the designers and implementers of prevention programs. In part, this information relates to what is known about different program strategies and activities and their success in achieving specific program objectives. This information directly relates to decisions program designers and implementers may make. These decisions are also shaped by the community and organizational settings in which programs operate and by the implementation capacity of organizations and staff who deliver the program.
One of the important objectives of the Cross-Site Evaluation is to contribute to knowledge about how prevention programs are actually funded, designed, and implemented, as reflected in the experience of the programs in this study. In Figure 2, we provide an overview of major factors that operate to shape a program, and ultimately its degree of success, throughout the processes of design and implementation. In most prevention research, which focuses on individual programs, the close dynamic interaction of these factors is not considered. Evaluators assess the program as it is designed and delivered without the ability to consider the circumstances of setting, or of implementation, that shape the program as received by participants. As conceptualized here, three major sets of factors constitute these circumstances.
Program Setting. While program strategies are a key concern of providers, programs must be designed and implemented within specific organizational and community settings. The implications of setting have often been overlooked in prevention research. The cross-site data allow us to investigate the influences of different community and organizational settings on the design, implementation, and success of study programs.
Intervention Strategy. Before they are implemented, programs are planned. The exact content of intended programming, the ways in which it will be delivered, and the amount of program contact are key elements of the intervention strategy. The strategy is influenced by the planning process itself, including the degree to which design is based on solid rationale or program theory, and the degree to which program elements are pre-specified or made explicit.
Program Implementation. Ultimately, program participants are affected by the program as it is delivered to them each day that they attend. The program that is implemented may be very much like what was planned, or it may depart from the plan in important ways-either intended or unintended. The capacity of the implementing organization to manage the program has a potentially major impact on the way in which the program is experienced by participants.
The analysis presented below assesses the ways in which factors within these three domains have influenced the effectiveness of programs in the National Cross-Site Evaluation. More specifically, this analysis includes the following factors.
Programs in the Cross-Site Evaluation operated in a variety of community, organizational, and service setting contexts. Two measures of program setting were developed-organizational setting and service setting. Each of these factors potentially affects program implementation and, consequently, program outcomes.
Organizational setting refers to distinguishing features of the grantee's organization. Profiling the characteristics of grantees revealed two distinct types:
Service setting characterizes where the program's services are actually delivered to participants. Again, two general types of service settings emerged from analysis of the program data:
The cross-site study includes a broad array of programs with varying content and methods of delivery. Early in the study, the interventions of each program were classified according to six content areas, including: 1) information (Alcohol, Tobaco, and Drugs), 2) information (ATD-Indirect), 3) academic and vocational support, 4) targeted-skills development, 5) positive recreation and enrichment, and 6) emotional and social support. In addition, the delivery method of each intervention strategy was coded as one of four delivery methods, including 1) one-on-one, 2) didactic/classroom-style presentation, 3) support group interaction, and 4) experiential programming. These strategy and delivery categories were originally developed (5) to reflect major categories of interventions that were proposed by the grantees in their grant applications. Over the course of the cross-site study these categories provided useful ways of describing and profiling the delivery of prevention services in the programs, understanding changes between program-as-planned and program-as-implemented, and comprehending the relation slip between strategies and delivery method.
(5) Documentation on the coding procedures used for the intervention and delivery categories is provided in the CSAP National Cross-Site Evaluation of High-Risk Youth Programs Final Technical Report, available through the Center for Substance Abuse Prevention.
For purposes of identifying the characteristics of effective prevention program design, however, these categories required simplification and focus. Drawing on the small but growing literature on generalizable guidelines to effective programming, the following measures of important program design were developed through secondary analysis of program-level and dosage data:
Program content measures were constructed to describe what is delivered to youth, that is, the major program messages. The following four measures differentiate between those programs that emphasize certain content areas and those programs that do not: (6)
(6) Programs were placed in these four measurement categories through the review of data on program interventions and data on the relative amount of contact youth had with each of the intervention types. The measures described the dominant category of programming offered to youth. Life-skills dominant programs are those that focused primarily on targeted skill development, academic and vocational support, or positive recreation and enrichment that focused on skill-development (e.g., wilderness adventures, courses). The exception to this coding decision was in the area of affective self-esteem building programming, which became a new strategy type, in combination with programs that focused on emotional/social support. Knowledge-dominant programming combines the two informational strategy types. Recreational programming then included all non-structured recreation-focused programming.
Program delivery method measures address the ways in which programs are delivered to youth. The study programs ranged in delivery method from primarily classroom-style, didactic programming in which youth received information through lectures, videos, or other similar means, to interactive and experiential programming in which youth were required to actively participate, reflect on the subject at hand, and act upon it. Typically, experiential programming includes role-play activities, team-building projects, outdoor wilderness experiences, or other activities that require youth to be actively engaged. While the desirability of experiential programming as a prevention tool has been consistently supported in recent prevention research, the understanding of exactly what constitutes effective experiential programming, or why it is more effective, has not been clearly developed. The richness of the information on the National Cross-Site Evaluation program interventions allows a more in-depth exploration of the nature of experiential programming and why it is effective. To elaborate on the delivery methods used by the High-Risk Youth Programs, three dimensions of experiential delivery methods were identified.
The effectiveness of an intervention depends on how well it is put into action. Evaluations of prevention and other social programs have clearly demonstrated that many programs did not meet their expectations because they were not delivered as planned. The Cross-Site Evaluation includes information on three dimensions of program implementation.
Service contact measures assess the amount of services provided to youth. Service contact, or dosage, is a fundamental implementation dimension. Two measures of service contact were developed, including:
Program structure measures describe the degree to which programs are clearly and consistently structured. The following two variables were created to assess program structure:
Program management measures assess the extent to which management is able to create a supportive and empowering work environment for staff. Two measures of management were developed.
The central question for this monograph is to determine whether the program characteristics identified above distinguish between more and less effective programs. The findings reported here are from analyses that used data collected when participants entered their program and when they completed the program. These findings are sometimes referred to as immediate program outcomes. The primary outcome measure used in these analyses is an overall measure of substance use, which combines 30-day cigarette, alcohol, and marijuana use. When they are important to help elaborate on the findings, five more specific outcomes are used: 1) 30-day cigarette use; 2) 30-day alcohol use; 3) 30-day marijuana use; 4) school connectedness; and 5) family connectedness. Substance use attitudes are not included in these analyses because prior analysis demonstrated that program effects on substance use attitudes are not related to program effects on other outcomes. In other words, the research showed that increasing negative attitudes toward substance use does not lead to reduced use of alcohol, tobacco, and/or illicit drugs.
Programs operate within particular organizational environments. While these environments clearly create opportunities and constraints that shape the day-to-day opportunities of staff, there is little systematic information on the ways in which a program's setting impacts its design and implementation. Indeed, few studies encompass differences in setting sufficient to support such an analysis. The Cross-Site Evaluation offers an exceptional opportunity because it includes programs in a variety of organizations that are delivered in different settings.
Organizational Setting. The organizational setting of programs may shape the capacity to implement prevention services. The degree of affiliation that programs have with large institutions potentially impacts their resources and may affect the ways in which prevention programs are designed. Programs that have greater access to resources, including additional facilities and staff, may be able to more successfully deliver services. However, freestanding, nonaffiliated programs that are located within the neighborhood of the target population and that have greater familiarity with the issues facing that population may benefit from greater experience with and proximity to the target population.
In the cross-site study, about two-thirds (62%) of participating programs were classified as freestanding, and a little more than one-third (38%) were categorized as affiliated programs. In Figure 3, the differences in resource availability between affiliated and freestanding programs are profiled. Across all resource areas, affiliated programs have greater access to resources than freestanding programs.
Figure 4 displays the differences in effect sizes between affiliated and freestanding programs. (7) The higher the effect size, the greater the positive impact the programs had on reducing substance use among participating youth. As shown in the graph, there is only a small, statistically insignificant difference in 30-day substance (8) use in the effect sizes between programs administered by affiliated and freestanding organizations. Although affiliated programs have more organizational resources at their command, freestanding programs in the community are able to design and implement programs with similar effectiveness for preventing substance use.
(7) Median effect sizes for this and subsequent figures are presented because effect sizes are not normally distributed.
(8) The Wilcoxon test (one-tailed) is used to determine statistical significance in all subsequent figures presented in this report.
Note: Averages are median values.
Service Setting. The optimal setting for delivering prevention services to youth has been a matter of debate among researchers and practitioners. School-based services have clear advantages of access, retention, and potential scale. However, their effectiveness for youth at higher levels of risk has been questioned because school-based services are usually universal approaches geared to the general student population. In the National Cross-Site Evaluation, 15 programs delivered a substantial portion of their prevention messages in school classrooms, and 33 programs delivered services after school. This diversity in setting allows a comparison of effectiveness on substance use and select protective factors. Figure 5 compares average effectiveness on 30-day substance use (9) achieved by programs that were delivered in school and those delivered after school. Differences for 30-day use approached statistical significance, with after-school programs producing more positive effects than in-school programs.
(9) When using substance use outcomes in the analysis, the number of programs is reduced to 46 because two programs were implemented in controlled environments that prevented access to alcohol and other drugs.
Figure 6 shows effect of service setting for the individual substance use outcomes. The difference between effect sizes for in-school and after-school programs is significant for cigarette use, with after-school programs producing a much more positive effect on cigarette use. Also, the pattern of effects suggests that after-school programs were more effective for alcohol use as well.
* Statistically significant at 0.05 level (one-tailed Wilcoxen test).
A contrast in the effectiveness of in-school and after-school programs with respect to school connectedness and family connectedness is shown in Figure 7. While after-school programs have positive effects on family connectedness, the in-school programs show positive effects on school connectedness.
* Statistically significant at 0.05 level (one-tailed Wilcoxen test).
School connectedness is the sole measure for which in-school programs are more successful than after-school programs. Effects are particularly strong in programs with high intensity, high levels of academic support activity, or high levels of one-on-one activity. The findings strongly suggest that the classroom may not be as effective in a setting directly influencing substance use among high-risk youth. However, programming in school does appear to have a positive impact on youth's perceptions of connectedness to school, an important protective factor.
The differential effectiveness of in-school and after-school programs may be attributable to the particular opportunities and constraints that characterize each service setting because the setting can shape the type, style, and amount of service that is offered. Programs offered during school hours are constrained by the periods in which they are offered, while after-school programs have more discretion in their ability to shape program offerings. The nature of the school setting requires planning and scheduling to fit into the classroom routine and environment. The constraints of schedule and competing classroom demands may limit the amount of total service time, restrict peer bonding that might develop in more intensive after-school programming, and be oriented toward less experiential and more didactic delivery methods.
Figure 8 summarizes differences between in-school and after-school programs on several program delivery characteristics. The differences are clear. In-school programs provided less contact with youth than after-school programs, had less orientation to building relations in their activities than after-school programs, and had fewer activities requiring active participation (each of these characteristics is explained more fully in subsequent sections of this report). This pattern is consistent with the opportunities and constraints in each setting and may help explain differential effectiveness across the settings.
* Statistically significant at 0.05 level (one-tailed Wilcoxon test).
** Statistically significant at 0.01 level (one-tailed Wilcoxon test).
When practitioners think about designing prevention programs, many of their questions concern intervention strategy or program content and delivery. What types of learning experiences or activities are most effective in preventing substance use? What are the most effective ways to deliver these services? This section explores each of these questions.
Program Content. The participating study sites used an array of program designs with different emphases and learning methods, thus providing an excellent opportunity to test differences in effectiveness based on program content. As noted above, descriptive program information was used to identify the degree to which study sites used each of four different categories of program content in their activities. Figure 9 displays the number of programs in which each type was emphasized more than others. For example, a program is categorized as knowledge-focused if participating youth were exposed to this prevention strategy more than any of the other three. Any or all of the others, however, may have been used in the program.
When categorized this way, the largest group of cross-site programs (n = 17) was knowledge focused. While these programs often included other types of activities, they emphasized the presentation of information about substance use and other related problem behaviors. These programs often emphasized negative health and social consequences associated with substance use.
The second largest group of programs (n = 14), emphasized life-skills training. These interventions focused on specific attitudes and behavior skills often identified as "protective" in the risk and protection literature. As applied in the study programs, these skills include refusal skills, anger management, communications skills, decision and goal-setting skills, academic and vocational skills, and other social skills. The category also includes "recreation and enrichment" activities that focused on skills development (e.g., wilderness programming, ropes courses, programs of arts with competency emphasis).
The third category (n = 12) placed priority on affective programming. These programs focused on emotive and awareness factors thought to be associated with substance use. Following Tobler (1993), affective programming includes activities focusing on self-esteem, self-awareness, and attitudes, beliefs, and values associated with substance use and other problem behaviors.
Finally, a small number of programs (n = 5) emphasized substance-free recreational programming that focused on alcohol and drug-free leisure time activities, often including organized sports.
Effects of Program Content on Substance Use. A major purpose of the National Cross-Site Evaluation was to test whether the nature of prevention programming makes a demonstrable difference in the effectiveness of programs for reducing substance use among high-risk participants as compared to non-participating comparison youth. Figure 10 clearly indicates that the primary content strategies of programs did make a difference. Programs that focused on delivering life-skills programming were significantly more effective than other types of programs, with recreation programs the second most effective. Programs that focused on providing knowledge about alcohol, drugs, and other attendant issues were significantly less effective than the other types of programs. These findings support prior research on prevention strategies for youth (Tobler et al., 2000) that show the superiority of life-skills content over programming that focuses on other information only, or on affective orientations. Importantly, the National Cross-Site Evaluation has confirmed these findings for a large number of programs targeting high-risk youth.
* Statistically significant at 0.05 level (one-tailed Wilcoxen test).
Effects of Program Content on Risk and Protective Factors. Figure 11 displays differences between the four program types in their effectiveness for strengthening school connectedness. Even though these differences are not significant, the pattern across program content types is the same as that for substance use. The patterns suggest that life-skills programming is a particularly effective strategy. However, the differences between the other three program types are small.
Delivery Method. Information about the Cross-Site Evaluation programs shows that the same program content may be delivered in different ways. Messages about how to resolve conflicts, for example, may be delivered through lectures in which the participant is largely a passive recipient or in more experiential methods such as role plays. Past studies have pointed to the method of service delivery as an important factor in the effectiveness of prevention programming. In particular, delivery methods that involve experiential processes, particularly among the youth themselves, have been shown to be more effective than didactic teaching or adult-led techniques (Hansen, 1992; Tobler, 1986; Tobler and Stratton, 1990; Tobler et al., 2000).
As explained above, the Cross-Site Evaluation data allow delivery methods to be differentiated in several ways including (1) the degree of interactive participation required; (2) the degree of introspection (self-examination) required; and (3) the amount of emphasis placed on connection building. These three concepts help to elaborate experiential programming and to understand why it is an important component of effective prevention programs.
Effects of Delivery Method on Substance Use
Each of the dimensions articulated above represents an elaboration of the ways in which youth may interact with peers and leaders within their programs. Figure 13 presents median effect sizes for 30-day substance use within programs that are higher and lower on each of these dimensions.
* Statistically significant at 0.05 level (one-tailed Wilcoxon test).
** Statistically significant at 0.01 level (one-tailed Wilcoxon test).
Programs that emphasize connection building had significantly higher effect sizes than programs that focused on individually oriented strategies and activities. The extent to which programs helped youth understand their own orientations and behaviors was also important, with significantly higher effects for programs that were oriented toward introspective activities. Active participation was less influential, although patterns of effect sizes for youth in programs with active rather than passive activities suggest that these programs were more effective than programs with a more passive orientation. In sum, programs that actively engage youth in thought-provoking and meaningful activities that encourage building positive connections to peers or adults are most likely to produce positive effects.
The cross-site analysis also explored the degree to which the elements of positive program design reinforce each other. For this analysis, a combined measure of all three types of interactive programming was used. When both program content and method of delivery are considered, programs with both a life-skills orientation and interactive methods are significantly more effective than other programs. The median effect sizes for programs characterized by both life-skills programming and interactive delivery are higher than those for programs that combine non-interactive methods with life-skills development (see Figure 14). In short, programs that engage at-risk youth experientially in skills development activities have the greatest impact on reducing substance use.
* Statistically significant at 0.05 level (one-tailed Wilcoxon test).
Effects of Delivery Method on Risk and Protection
As shown earlier, life-skills development strategies also have a positive effect on school connectedness, which is an important protective factor. When service delivery methods are factored into the analysis we find similar patterns as those shown for 30-day substance use. Figure 15 shows highly significant differences in effect sizes for school connectedness depending on whether programs emphasized introspection, connection building, and active learning. Youth who are asked to participate in programming in meaningful ways clearly improve their own school connectedness. When building connectedness with other youth in prevention programs is a program focus, connectedness to school also improves.
** Statistically significant at 0.01 level (one-tailed Wilcoxon test).
In summary, the Cross-Site Evaluation confirms and extends the accumulating evidence on effective design of prevention programs. The findings specify the types of interventions and delivery methods effective in working with high-risk youth.
The evaluation field has clearly demonstrated that there is often considerable slippage between program planning and the program services that are actually received by participants. The process of implementing program ideas will inevitably determine the exact configuration of the program experienced by youth. It follows that improved understanding of this complex topic is important to improving prevention effectiveness.
This section includes information on three areas of program implementation. First, the importance of the amount and intensity of program contact for prevention effectiveness is assessed. Second, the importance of clear and consistent organizational and conceptual structure of activities for effective programming is considered. Finally, indicators of successful management are related to program effectiveness.
Service Contact. If program activities are to make a difference in the lives of youth, they must be organized and delivered so that youth participate sufficiently to benefit. Programs that do not provide a sufficient dose of service presumably cannot impact youth. For the following analyses, three measures of the contact that youth had with their program are used. First, we consider the total amount of contact (number of hours), next, the length of time in the program, and last, the intensity of contact, or the number of hours per week.
* Statistically significant at 0.05 level (one-tailed Wilcoxon test).
Program intensity, unlike total amount of program contact, has a significant relationship with major outcome variables. More intense programs were significantly more effective in changing alcohol and 30-day substance use patterns. There were no significant differences for the other outcome measures though the pattern suggests the importance of program intensity for school connectedness.
In summary, the intensity of programs, or the number of hours per week of exposure they provide, has a significant relationship to substance use of at-risk youth, but total amount of contact does not. This finding suggests that program impact is facilitated by a concentrated presence in the lives of youth.
Program Structure. Prevention programs vary in their degree of explicit rationale and organization to accomplish defined outcomes. Indeed, one of the objectives of CSAP grant funding procedures has been to increase this explicit program structuring through the use of logic models and similar tools. The Cross-Site Evaluation analysis considers the impact of two types of structuring on program effectiveness-coherence and consistency. Coherence relates to conceptual organization and rationale, and consistency relates to the clear structuring of program time and schedule.
* Statistically significant at 0.05 level (one-tailed Wilcoxon test).
While differences for school connectedness or family connectedness are not statistically significant, the patterns suggest that these protective factors are influenced more positively in programs with higher coherence. This finding reinforces the importance of clear program purpose and logical connection of program activities to that purpose. Funding and technical assistance for prevention programs should encourage consistency, planning, and training that supports this program quality.
In summary, the pattern of program effects displayed in Figure 20 has two implications. First, coherence of program theory and activities is not the same as consistency in organization and schedule. Programs high on one dimension might be low in the other. Second, each quality contributes to effectiveness. When both are present, programs are most effective.
Program Management. Organizations have their own ways of operating that shape program delivery. All organizations develop "routines, or standard operating procedures, to guide workers' actions in different circumstances" (Scheirer, 1987). These routines may be guided by written rules or program guidelines, through strong leadership from supervisors or through the norms and standards about how work is accomplished, that are brought in by staff or are developed in the program work group.
These aspects of the program environment are undoubtedly related to management. The clarity of job functions and responsibilities, the degree to which procedures are explicit and rule oriented, the nature and clarity of leadership, personnel policy, and the opportunities for staff to have input and shape the program are all examples of organizational operations that are strongly shaped by management. The experience of CSAP-funded demonstration programs, and that of other funders of demonstration programs, has amply shown that programs often fail because of organizational failure, breakdowns in the personnel systems (e.g., high turnover, burnout) or organizational operations that prevent implementation of the services proposed by the program. However, management procedure that is effective in one organizational setting or work group may not be effective in another. The cross-site program sample encompasses heterogeneous organizational setting, and management styles that are likely to differ across program settings.
Considering the clear importance of management, the following analysis focuses on two components of organizational operations that are expected to contribute to effective program implementation-positive and supportive work environment and results-oriented management. Both of these components assess effective program management by the degree to which they empower staff to feel supported and effective in carrying out their responsibilities, rather than by a direct assessment of how these management results were achieved.
* Statistically significant at 0.05 level (one-tailed Wilcoxon test).
These findings suggest that both supportive and results-oriented management are components in program excellence, but they cannot substitute for other basic building blocks of intervention strategy and implementation.
An underlying purpose of the National Cross-Site Evaluation of High-Risk Youth Programs is to produce and disseminate practical lessons on how to better contribute to the positive development of youth at high risk. This monograph represents the first step in realizing these objectives. Most generally, the monograph is encouraging in its promise that science-based principles of good programming can be identified across a broad range of programs serving high-risk youth. The findings are consistent with findings from other studies in some areas. The importance of active participation by youth and the comparative efficacy of life-skills program content are examples. The Cross-Site Evaluation has added new lessons in other areas. For example, the study provides new understanding of what makes interactive participation effective by identifying the components of that idea. The study also indicates that it is not exclusively program strategy that matters; other important factors also contribute to effective programming. More specifically, Figure 23 summarizes program characteristics that contribute to significant reductions in 30-day substance use.
Illustrated in the above graph are the following points:
The pattern of findings has important implications for strengthening prevention programming for high-risk youth as implemented in communities across the Nation.
For prevention program designers and implementers, the ultimate test of the utility of science-based knowledge lies in the ability to go beyond identifying what to focus on as program objectives, and to help identify exactly how to work with youth to achieve those objectives. Through collecting detailed data, and relating program characteristics to outcomes achieved by the study programs, the National Cross-Site Evaluation has produced science-based answers to practitioner questions about how to make prevention activities more effective.
Cross-Site Evaluation programs that focused on developing life skills were more effective in reducing substance use than programs that emphasized other program content. Programs that focused on knowledge and information concerning alcohol, drugs, and other related issues were significantly less effective than other programs, as were programs that emphasized affective programming concerned with changing attitudes, self-esteem, and value perceptions. These findings are consistent with recent meta-analyses of prevention programs (Tobler et al., 2000).
Cross-Site Evaluation programs that involved participants interactively were more effective in reducing substance use than programs that relied more on passive, classroom-style learning, confirming similar findings in meta-analytic studies (Tobler and Stratton, 1997; Tobler et al., 2000). The cross-site analysis elaborated on this finding by identifying component styles of interactive programming and assessing their relative contribution to program effectiveness. Among the different types of interactive methods, connectedness-building approaches that focus on activities in which youth work cooperatively with adults or in peer teams made the strongest contributions. This finding is consistent with the connectedness theme running throughout the cross-site findings. Approaches that helped youth understand their own orientations and behaviors were also more effective than methods that simply involved active participation.
The Cross-Site Evaluation team collected detailed information on the duration and amount of participant contact with their program. Analysis of this information demonstrated that programs with higher levels of contact were on average no more successful than programs with less contact. However, more intense programs, measured as programs delivering more hours of programming per week, were more effective in reducing substance use than programs that were less intense. The total number of hours per week spent by youth in a program is more important than the overall amount of time youth are exposed to program services.
The study findings support the conclusion that the classroom is not the best setting in which to provide a substance abuse prevention curriculum. However, the findings also demonstrate the importance of school connectedness as a protective factor, and the greater effectiveness of in-school programs in achieving stronger school connectedness. The latter finding is consistent with the findings of the Child Development Project, a CSAP model program, that demonstrates the many positive benefits of changes in the school environment to make it more of a "caring community" (Battistich et al., 1996) in the eyes of students.
Programs must be effectively implemented if they are to deliver services as planned. In the Cross-Site Evaluation programs, positive management practices appeared to provide a necessary but not sufficient foundation for program effectiveness. Supportive management, measured as management practices that provided staff with resources, training, voice, and work conditions perceived as empowering them to be effective in their job, did not guarantee program success, but programs with poor management rarely produced positive effects, particularly for the risk and protective factors measured in the study. Above this floor, programs that were coherent in relating clear understandings of objectives, planned activities, and the relationship between these two factors to staff, were more effective in achieving outcomes. In sum, programs with coherent planning and management were more effective.
The degree to which youth express more or less accepting attitudes toward substance use has often been identified as a risk and protection factor, and moving attitudes toward less acceptance has often been identified as a program objective. Achieving this desired change in attitudes has been considered a sign of program success. However, the Cross-Site Evaluation findings clearly demonstrate that programs that achieve positive effects on attitudes (i.e., more reports of non-accepting attitudes) are no more likely than other programs to produce positive effects on the other primary outcome variables assessed in the study. In contrast, positive program effects for all of the other primary outcomes tended to co-occur. Furthermore, the kinds of programs that were effective in changing reported attitudes were different than the kinds of programs that were effective in changing the other risk and protection factors and substance use behaviors. The conclusion and the implications for program outcome objectives are fundamental. Program interventions that are more effective for changing attitudes are not more effective in reducing substance use or strengthening important risk and protective factors measured in this study.
This monograph has presented information on program characteristics that contribute to the effectiveness of prevention programs targeting high-risk youth. The findings begin to unravel the complex knot of interconnected influences of setting, organization, and program design that can strengthen or diminish program effects. They also begin to provide important concrete guidance concerning what elements of design or implementation are important to achieve intended effects within a particular setting. By identifying and scientifically demonstrating the building blocks of effective prevention programs, the National Cross-Site Evaluation has provided an important resource for the practice of science-based prevention.
Battistich, V., Schaps, E., Watson, M., and Solomon, D. (1996). Prevention effects of the Child Development Project: Early Findings from an ongoing multi-site demonstration trial. Journal of Adolescent Research, 11 (1), 6-11.
Hansen, W.B. (1992). School-based substance abuse prevention: A review of the state of the art in curriculum, 1980-1990. Health Education Research, 7, 403-430.
Hansen, W.B. (1997). Prevention programs: Factors that individually focused programs must address. In Resource Papers for the Secretary's Youth Substance Abuse Prevention Initiative, pp. 53-66. Rockville, MD: Center for Substance Abuse Prevention, Substance Abuse and Mental Health Services Administration.
Paglia, A., and Room, R. (1999). Preventing substance use problems among youth: A literature review and recommendations. The Journal of Primary Prevention, 20, 3-50.
Scheirer, M.A. (1987). Program theory and implementation theory: Implications for evaluators. In L. Bickman (Ed.), Using program theory in evaluation. New directions in program evaluation, 33. San Francisco: Jossey-Bass.
Sambrano, S., Springer, J.F., and Hermann, J. (1997). Informing the next generation of prevention programs: CSAP's cross-site evaluation of the 1994-95 high-risk youth grantees. Journal of Community Psychology, 25, 375-395.
Tobler, N.S. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16, 537-567.
Tobler, N.S., Roona, M.R., Ochshorn, P., Marshall, D.G., Streke, A.V., and Stackpole, K.M. (2000). School-based adolescent drug prevention programs: 1998 meta-analysis. The Journal of Primary Prevention, 20, 275-336.
Tobler, N.S., and Stratton, H.H. (1997). Effectiveness of school-based drug prevention programs: a meta-analysis of the research. The Journal of Primary Prevention, 18, 71-128.
| |||||||||||||||||||||
| Initiatives | Funding | Home | ||||||||
|
Site Map
Help
Account
Cart
Home
