|
This document was developed under the direction of Soledad Sambrano, Ph.D., of the Center for Substance Abuse Prevention (CSAP) through contract #277-95-5002 with EMT Associates, Inc., and ORC Macro (formerly Macro International, Inc.).
The principal authors of this document are J. Fred Springer, Ph.D., of EMT Associates, Inc.; Soledad Sambrano, Ph.D., of the Center for Substance Abuse Prevention; Elizabeth Sale, Ph.D., and Rafa Kasim, Ph.D., of EMT Associates, Inc.; and Jack Hermann, Ph.D., of ORC Macro. Special thanks to Paul J. Brounstein, Ph.D., and Fred Seitz, Ph.D., of CSAP, Charles Turner, Ph.D., Wei Pan, Ph.D., David Cordray, Ph.D., Will Shadish, Ph.D., Chrystalla Ellina, Ph.D., and Mary Nistler, M.P.P.A., for their important contributions.
Special thanks to the CSAP staff who managed the projects and the 48 grantees who participated in the study. Also appreciation to Juana Mora, Ph.D; Craig Love, Ph.D.; and Jane Maxwell, Ph.D. for their review of these documents.
All material appearing in this report is in the public domain and may be reproduced or copied without permission from the Substance Abuse and Mental Health Services Administration. However, this publication may not be reproduced or distributed for a fee without specific, written authorization of the Office of Communications, SAMHSA, U.S. Department of Health and Human Services. Citation of the source is appreciated. Suggested citation:
Substance Abuse and Mental Health Services Administration. The National Cross-Site Evaluation of High-Risk Youth Programs. Center for Substance Abuse Prevention, DHHS Publication No. SMA-003375. Rockville, MD, 2002.
Copies may be obtained, free of charge, from the National Clearinghouse for Alcohol and Drug Information (NCADI). NCADI is a service of the Substance Abuse and Mental Health Services Administration (SAMHSA). For copies of publications, please write or call:
National Clearinghouse for Alcohol and Drug Information (NCADI)This publication can be accessed electronically through the Internet World Wide Web connection listed below.
World Wide Web Internet connection: http://www.samhsa.gov
Originating OfficeThe Center for Substance Abuse Prevention (CSAP) in the Substance Abuse and Mental Health Services Administration (SAMHSA) is the Nation's lead agency for substance abuse prevention. The Center funds community-based organizations, universities, behavioral health providers, and public agencies to identify effective prevention programs and practices and disseminates findings, program models, and other prevention materials to practitioners and policymakers across the country. This document summarizes the findings of CSAP's National Cross-Site Evaluation of High-Risk Youth Programs. This large multiple-site evaluation was designed to assess 48 prevention programs and to identify those program characteristics that are associated with strong substance abuse prevention outcomes. The rigor and relevance of the National Cross-Site Evaluation earned CSAP the Outstanding Study of the Year 2000 Award from the American Evaluation Association.
The National High-Risk Youth Demonstration (funded from 1987 to 1995) has been one of the most ambitious and productive of CSAP's funding initiatives. In its early years, the demonstration focused on identifying promising approaches to prevention. As the demonstration matured, individual site and cross-site research produced knowledge about risk and protective factors related to substance use and helped to identify model programs for effective prevention. Research and experience in the High-Risk Youth Demonstration also contributed to the awareness of the importance of culturally sensitive, age-appropriate and gender-specific programming. This progress in prevention theory and practice laid the foundation for the research reported in this document, the largest and most comprehensive of CSAP's High-Risk Youth studies.
In addition to individual-level information on substance use, risk, and protection, the evaluation includes detailed information on the nature and amount of prevention services in which each child participated, as well as systematic process information on the study programs. This monograph highlights the study's contributions to the growing evidence that "Prevention Works." More specifically, this document highlights the study's contributions to the understanding of how substance use develops in youth at high risk, and the risk and protective factors that contribute to or protect against substance use. The monograph also summarizes major findings concerning changes in substance use, risk, and protection as youth mature through adolescence, summarizes findings concerning the pathways between external and internal risk and protective factors and substance use during adolescence, and identifies implications for policies and programs designed to prevent substance use.
This monograph is part of a series of "Points of Prevention" publications that document the contribution of CSAP's National Cross-Site Evaluation of High-Risk Youth Programs to prevention knowledge and provide science-based guidance for improved prevention policy and practice.
Charles G. Curie, M.A., A.C.S.W.Substance use is one of the Nation's most pervasive, costly, and challenging health and social problems. The use, and particularly the early use, of tobacco, alcohol, marijuana, and other illicit drugs is intricately entwined with serious personal and social problems including school failure, crime, family violence and abuse, and a host of additional problems that constitute a continuing national tragedy. For over a decade, the Center for Substance Abuse Prevention (CSAP) within the Substance Abuse and Mental Health Services Administration (SAMHSA) has been the Federal agency responsible for providing leadership in preventing the profound negative consequences of substance use. Important components of this responsibility include design and funding of demonstration substance use prevention programs, followed by evaluation of these programs to identify successful community prevention services and by dissemination of science-based guidance for strengthening prevention services in communities across the Nation.
CSAP has made great progress in generating new knowledge about the design, implementation, and effectiveness of prevention strategies and activities. The most recent of these advances is CSAP's completion of its National Cross-Site Evaluation of High-Risk Youth Programs-a study that offers many lessons for prevention policy and practice.
CSAP's High-Risk Youth (HRY) Demonstration Grant Program targeted substance use among at-risk youth. Since HRY began in 1987, CSAP has funded more than 400 High-Risk Youth projects that have created a wealth of knowledge about substance abuse prevention. Demonstration projects have been awarded to community-based organizations, universities, behavioral health providers, and public agencies. These varied groups of grant recipients serve communities that are diverse in location, racial and ethnic composition, and level and nature of risk.
The National Cross-Site Evaluation of High-Risk Youth Programs was a 5-year evaluation of more than 10,500 at-risk youth. Forty-eight HRY demonstration grant recipients across the Nation participated in the study.(1) These grantees were funded by CSAP to implement and assess programs to prevent and reduce the use of alcohol and other drugs among at-risk youth.
(1) CSAP funded 94 programs in 1994 and 1995. Programs were not included in the Cross-Site Evaluation if they served children promarily under the age of 9. Approximately half of the 48 programs that aprticipated in the Cross-Site study were funded in 1994 for 5 years, and the other half were funded in 1995 for 3 years. These programs were located throughout 22 states, including Alaska and Hawaii.
Because the High-Risk Youth Demonstration Program took place over a decade, the grant process and the funded programs were able to mature and focus on promising practices and special prevention needs. Early evaluations of the HRY programs showed that the programs most successful in preventing substance use were those that clearly stated a theory about why the use began and continued; those successful programs often emphasized the concepts of risk elements and protective elements in the "web of influence" in the lives of at-risk youth (CSAP, 1993; CSAP, 1994). The experience gained over time also reinforced research literature that identified special needs of specific populations, particularly girls and young women, who were relatively neglected in the early development of substance abuse prevention programming. As a result of this experience, and as the demonstration program matured, new grant recipients had to clearly outline the basic ideas of their interventions and were encouraged to emphasize risk and protection concepts (Sambrano, Springer, and Hermann, 1997). The 1994 and 1995 High-Risk Youth Demonstration Programs included two special initiatives: the Female Adolescent Initiative ensured that projects addressed the special needs of girls and young women and the Replication Initiative funded promising program models in diverse settings.
The National Cross-Site Evaluation of High-Risk Youth Programs created an unprecedented opportunity to learn more about effective substance abuse prevention for youth at risk. The evaluation was initiated with three major objectives:
The National Cross-Site Evaluation of High-Risk Youth Programs provides strong evidence that substance abuse prevention programs in communities where youth face multiple risk factors do produce lasting reductions in substance use and have important positive effects on the lives of youth at risk. These results are not hypothetical. They represent real accomplishments by real programs operating in these communities. Major findings of the study include the following important outcomes:
Substance abuse prevention programs produce statistically significant reductions in substance use and positively impact the lives of at-risk youth.
CSAP's National Cross-Site Evaluation of High-Risk Youth Programs found reductions in rates (2) of alcohol and marijuana use by youth participating in a CSAP-funded prevention program relative to comparison youth who had limited opportunity to participate in prevention programs in their community. At exit, substance use (3) for participants in CSAP prevention programs was 10 percent lower than for comparison youth. Eighteen months later (Figure 1) (4), substance use was 6 percent lower for participants than comparison youth. These positive results were statistically significant and show that the effects of program participation continue to be felt for approximately 18 months after youth leave the programs. Even within the challenging conditions of this community-based evaluation design, the entire pool of participants across all programs decreased their use of alcohol and marijuana relative to similar youth who did not participate in the programs.
Note. Statistically significant at .05 level.
Calculated for the 23 sites with low comparison group exposure to prevention programs.
(2) Rates are measured in terms of the number of days in which a substance was used in the last 30 days; the numbers of days are combined across cigarettes, alcohol, and marijuana to calculate the substance use rate.
(3) Substance use combines use of cigarettes, alcohol, and marijuana to assess overall change in adolescent substance use for the three most widely used substances in this population.
(4) Youth reported their frequency of substance use by indicating how many days they had used cigarettes, alcohol, or marijuana in the previous 30 days. Response categories were (0), no use; (1), 1 to 2 days; (2) 3 to 5 days; (3), 6 to 9 days; (4), 10 to 19 days; and (5), 20 to 31 days. The numerical values of the categories were averaged for each group and adjusted for age, sex, ethnic background, and baseline 30-day substance use. Analyses were conducted using Hierarchical Linear Modeling (HLM) (Bryk and Raudenbush, 2000). For these estimates of the magnitude of prevention effects the sample includes only those 23 sites in which comparison groups were shown to have limited opportunities for participation in prevention activities other than the study program (see finding #6 below).
Alcohol Use. Alcohol use was the most widespread focal topic of substance use prevention in cross-site programs. Alcohol is more easily accessed and more prevalent than any drug in the study. Study programs produced statistically significant reductions in alcohol use rates for participants relative to comparison youth. Participant use rates were on average 11 percent lower than use rates of comparison youth over the 18 months following the end of the program. While the rate of reduction is small, it is statistically significant, which means that scientists view this as a real program achievement. Unlike cigarettes and marijuana, the reductions in alcohol use faded at the 18-month measurement point.
Marijuana Use. Marijuana use begins at a later age than alcohol use, but rises extremely rapidly, particularly among males. Marijuana use is becoming a major concern in prevention as use proliferates, but its development and patterns of use are less well understood than for alcohol. The study programs produced lasting reductions in the rate of marijuana use by program participants relative to comparison youth. Program participants averaged a consistent 11 percent below comparison youth rates of marijuana use at program exit and 6 and 18 months after exit.
Cigarette Use. Prevention of cigarette use is a well-developed field of knowledge. Although researchers note that cigarette prevention functions independently of alcohol, marijuana, and illicit drug prevention, the cross-site programs produced statistically significant reductions in the rate of cigarette use by program participants. Program participants averaged 7 percent lower use rates than comparison youth over the three points in time when outcomes were measured.
| A true analysis of program effectiveness must consider the fact that non-program youth may be exposed to prevention programming. |
The majority of youth in this study are in the age range when substance use begins. Around three-fourths of all youth in this study (76%) reported no use of substances at baseline, making reduction of their use impossible. Accordingly, the fact that the overall reported use of alcohol and marijuana by participants was lower relative to comparison youth after participating in the program is an important result.
A major objective of the Cross-Site Evaluation was to identify and explain how participation in prevention programs helps prevent substance use. This explanation compares results among youth in the program with results amoung similar youth from the same communities who were not participants. Although the pooled information on substance use across all study programs shows that use rates for alcohol and marijuana were significantly lower for youth who participated in the programs, the size of this reduction is small. There are three factors that help explain why these pooled differences are small.
First, using contrasts with comparison group youth to demonstrate program effectiveness presumes that the non-program youth do not have access to other prevention services. In communities that offer limited or no prevention programming, this presumption may be accurate. However, when the community has additional prevention programs, particularly those that may target youth similar to those in the Cross-Site programs, a true analysis of how effective the programs are must consider the fact that non-program youth may be exposed to as much or more prevention programming as program youth. This is why estimates of the overall magnitude of prevention effectiveness in this study are based on the 23 programs in which comparison group youth were shown to have only limited opportunity to participate in prevention programs other than the study program (see Finding #6).
Second, as discussed above, the large percentage of youth who did not use substances made it impossible to produce dramatic reductions in substance use among the full pooled study sample. Findings presented below (Finding #2) demonstrate that the positive effects of prevention on substance use are much more evident for youth who already had initiated substance use when they entered the study.
Third, the pooled effects of prevention across all the programs are small because the study sample was composed of a broad range of programs implemented in real community conditions. The primary objective of the study was to generate information about what makes programs effective in these real settings. Not all of the programs reduced substance use, though they often had other positive outcomes for the youth. Among those that did show positive effects on use, some were more effective than others. One of the most important outcomes of the study is the identification of program features that made the study programs more effective in reducing substance use among youth at high risk. These findings, summarized later in this overview (Finding #5), demonstrate that, while prevention currently benefits youth significantly, these benefits will be greatly strengthened as programs increasingly adopt the effective science-based practices generated by this study and other prevention research.
In high-risk communities, prevention programming must change behavior in youth who have already begun to use substances, not just deter those who have not yet started. Most of the youth in the Cross-Site Evaluation reported no use of substances in the 30 days before the program began. Still, approximately 25 percent of youth reported some substance use at baseline, and use rates were higher for this high-risk population than for the general population of youth the same age. Participation in CSAP substance abuse prevention programs reduced use of cigarettes, alcohol, marijuana, and the combined use measure relative to comparison youth. Program participation produced significant reductions in substance use patterns and the positive effects continued over the course of the study.
Figure 2 presents the trends in use of cigarettes, alcohol, and marijuana and 30-day substance use for participant and comparison youth who had already begun to use substances. Use of any of the three substances by participants who reported prior drug use was 10 percent lower at exit than comparison youth, and use levels were 22 percent below comparison youth 18 months later. Furthermore, the positive impact of program participation on substance use was consistently strong across the three most commonly used substances among youth. Program participants' rates of use of cigarettes in the last 30 days were an average of 17 percent below the rates for comparison youth at exit and at 6 and 18 months after program exit. For alcohol, participant's rates were an average of 13 percent below rates for comparison youth across the three outcome time points; and participant rates of marijuana use in the last 30 days were an average of 18 percent below those of comparison youth at program exit and 6 and 18 months after exit. These findings indicate that the CSAP-funded prevention programs provided effective interventions for youth who have already been identified as initiating use.
Figure 2 also shows that, overall, participant youth reduced their use of all substances through the full period of the study. Given that widespread substance use typically starts and escalates among youth in the age groups included in this study, programs are called effective when they slow the rate of increased use among program participants. However, this analysis of youth who were already using at baseline demonstrates that program participation can actually reduce use rates. (5)
(5) The slight decrease in comparison group use in these analyses may be attributable to a statistical artifact called regression toward the mean. This is a tendency for a measurement (i.e., substance use) on which subjects are selected for an extreme value (i.e., substance use when the norm for the subject's age is no use) to be more like the population average at a repeated measurement. Some of the youth who reported use in the 30 days prior to the baseline measurement are not habitual users, for example, and may not use in the 30 days prior to a second measurement. Controlling for this tendency is one reason for a comparison group, and the important point here is that reductions in use scores at repeated measurement pints for treatment youth were larger than those for comparison youth.
Among the total sample, for both cigarettes and marijuana, participant youth report slightly higher levels of use at baseline than comparison youth. Furthermore, their use levels decrease more rapidly during the program period so that they are substantially lower than those for comparison youth at program exit. For cigarettes and marijuana, the use gap between participant and comparison youth continues to widen throughout the followup period. However, for alcohol use, the large positive impact on participants between program entry and exit fades substantially by the second followup. Alcohol is a widely available and widely used substance among youth, and the implication is that a continued protective influence in the program environment may be necessary as a deterrent to alcohol use.
Prevention practitioners and researchers have become increasingly aware of the differences between boys and girls in the way substance use develops and how well prevention works. The Female Adolescent Initiative was developed to fund 19 of the Cross-Site Evaluation programs partly because research suggested that current prevention programming may be less effective for girls. The ability to compare boys' and girls' responses to substance abuse prevention participation across a large number of programs over time contributes new knowledge that will be helpful in developing programming for both genders.
Prevention effects over all four points of data collection show a large gap (29%) in substance use rates between participant and comparison group boys at program exit (Figure 3). For girls, the difference between participant and comparison group use is only 3 percent at program completion. However, participant girls do significantly better than comparison girls 6 and 18 months after program exit, with a 9 percent gap between participant and comparison group girls at the final measurement point. In contrast, use rates by participant boys move back closer to the rates of comparison boys after prevention programs end. In the study overall, substance use outcomes were more positive for boys than for girls at program's end, but positive outcomes emerged later and lasted longer for girls.
Despite these differences in the overall effects of program participation on boys and girls, programs that are characterized by multiple science-based program components (see Finding #5) identified in the study produced stronger and longer lasting effects for both boys and girls. Girls participating in the 8 programs with five or more of these science-based components reported substance rates that were 24 percent lower than those reported by comparison girls at program exit, 24 percent lower than comparison girls 6 months after program end, and 13 percent 18 months after their program ended. The pattern for boys is very similar. At the end of their programs, boys participating in strong programs reported substance use rates that were 28 percent lower than those reported by comparison boys, still 27 percent lower 6 months after program end, and 15 percent lower than those reported by comparison boys 18 months after program end. When programs are consistently strong in design and delivery, they produce stronger and similar reductions in the rate of substance use for participants relative to comparison youth for both girls and boys at high risk. In these well-designed and implemented programs, the positive effects of substance use for high-risk boys are still significant 18 months after the program has ended.
| When programs are strong in design and delivery, they produce similar reductions in use rates for both boys and girls. |
The study extended knowledge about substance use, risk and protection, and prevention for boys and girls in additional ways, including the following:
Over the past decade, risk and protective factors have become a dominant part of the theory behind substance use prevention and associated problems among youth. In its most recent funding years, CSAP used the risk-and-protection framework to help document the need for service and to develop program objectives among grant recipients. Risk and protective factors form a "web of influence" for each child, an interwoven backdrop that influences each child's life. A growing body of research has documented this "web of influence" through which community, family, school, and peer factors may put youth at risk for substance use (CSAP, 1999b). The web consists of the forces at work in five domains: family, peers, school, community, and society. The evaluation confirms that family, peers, school, and community are all important influences on substance abuse among highrisk youth. Efforts to strengthen families (by encouraging communication, appropriate supervision, and positive norm setting) and to strengthen schools as caring communities (to improve school bonding and opportunities for meaningful achievement) increase the protection of youth against substance use.
The National Cross-Site Evaluation was designed to develop information that is useful to prevention programs as they are developed and implemented. For this purpose, it is important to distinguish between risk and protection factors that are "external" to youth (in their environment) or "internal" to youth (the way that they process, interpret, and respond to their environment). Program activities designed to reduce risk or promote protection in external environments like community, school, and family differ from prevention strategies that work directly with youth to develop internal protective factors. Analysis of programs in this study indicates a focus on internal risk and protective factors that show up in the attitudes, beliefs, and orientations of youth. Program activities and objectives emphasize strengthening individual characteristics in young people themselves. One of the important contributions of the National Cross-Site Evaluation is to clarify the relationship between external and internal risk and protection, prevention interventions, and substance use.
Types of Risk and Protection
External risk and protective factors can be defined as the characteristics of young people's circumstances that give them the opportunity or influence them to use substances or to avoid them. The study measured four external factors:
| External risk and protective factors refer to characteristics of young people's circumstances. |
Adolescents move from protection toward risk as they get older (Figure 4), with a decided decline in two of these factors: family supervision and school prevention. This pattern is not surprising because youth mature and family supervision diminishes. Involvement in community activities declines least, suggesting an ongoing opportunity for community-based prevention. As youth become more independent, they form associations in the community. The challenge for prevention is to help make these associations positive.
Internal risk and protective factors refer to young people emotional and developmental processes. Internal risk and protective factors shift in terms of their relative importance during the age span included in the National Cross-Site Evaluation population. The study measured six internal risk and protective factors:
(6) For some analyses, school bonding and self-efficacy are combined into a single measure of school connectedness.
| Internal risk and protective factors refer to young people's emotional and developmental processes. |
Around age 12, most high-risk youth begin to move from the protective toward the risk ends of these measures, bottoming out at age 15 (Figure 5). The exception is social confidence, which increases slightly from preadolescence to the mid-teen years. The drop toward greater risk in the remaining orientations during the middle and early high school years indicates that internal protective orientations are also lessening during these years. The change is most pronounced with school and family bonding. The important implications in these patterns are that interventions need to be age-specific and offered at the time during the maturing process when they will have the greatest impact, and that meaningful associations with family and school should be prominent foci of adolescent interventions.
(6) For some analyses, school bonding and self-efficacy are combined into a single measure of school connectedness.
Connectedness and Substance Use
The cross-site programs targeted risk and protective factors because researchers expect these factors will reduce substance use and other negative behaviors. The study findings on the association of individual protective factors and substance use indicates that ties to family and school are central to prevention of substance use (Figure 5). The analysis suggests that individual protective factors that begin and help maintain connectedness with positive external environments (family bonding and school bonding) are more strongly associated with lower levels of substance use than other internal factors measured in the study (belief in self, self-efficacy, social confidence, and self-control). These data, and the more complex findings of the risk and protective factor modeling in the study, are consistent with a recent research review concluding that "closeness to and connectedness with family and school were the most salient, cross-cutting protective factors" (Resnick, 1999).
*Bar length indicates the strength of the negative association between the factor and substance use-in other words, the degree to which the factor protects against substance use. Longer bars indicate stronger protective relationships.
The data also indicate important differences between boys and girls in linking protective factors and substance use. Across all protective factors, the relationship between internal risk and protective factors and substance use is stronger for girls than for boys. This difference is especially true for family bonding and self-efficacy.
What Are the Major Pathways to Substance Use?
To advance understanding of the relationship between risk and protective factors and substance use, the CSAP study team used a statistical technique to estimate the pathways that may link risk and protective factors to substance use (Figure 7). Although the technique does not test cause-and-effect relationships, it does indicate relationship patterns that are consistent with protective influences.
| Ties to school and family are central to prevention of substance use. |
Two new variables were constructed for this analysis:
The resulting model (Figure 7) shows that family and peers are important influences on substance use and that school and community conditions also influence use. The width of the arrow reflects the strength of the association. Numbers along lines indicate the strength of the relationship.
The model provides a plausible understanding of the way risk and protective factors from different domains in the "web of influence" (CSAP, 1999) interact in their association with substance use. These relationship structures, which are highly consistent with the data produced by the National Cross-Site Evaluation, support the following interpretation:
Family and school represent powerful protective factors when they offer youth meaningful involvement: involvement that is challenging, provides recognition, and is rewarding.
| Family and school represent powerful protective factors when they offer youth meaningful involvement: involvement that is challenging, provides recognition, and is rewarding. |
In summary, this model begins to tease out the dynamics of the "web of influence" to explain the relationships between external risk and protection, internal risk and protection, and substance use. Although distinct concepts, external and internal protection are tightly interwoven in the dynamics of behavioral development in youth. Connectedness lies at the core of the model-specifically, the connectedness to family represented in family bonding and the connectedness to school represented in school connectedness. These two external environments represent powerful protective factors when they offer youth meaningful involvement-involvement that is challenging, provides recognition, and is rewarding.
The model also indicates the importance of comprehensive and persuasive prevention activities and resources in the community. Although the cross-site evaluation programs focus on the internal protective factors, clearly their efforts will be more effective if families are strengthened (to provide communication, appropriate supervision, and positive norm setting). Similarly, school bonding will be more meaningful if schools are strengthened as caring communities (Battistich et al., 1996).
The evaluation included 48 programs implemented in communities at high risk. These programs experienced all the challenges of implementing programs in these settings, and they approached these challenges using differing prevention strategies and program management approaches. This diversity was important to the research objectives of the study because it provided an opportunity to learn from the differences among programs. Not all of the programs succeeded, and of those that did achieve their objectives, some were more successful than others.
The evaluation identified and verified six program components that produced statistically significant improvements in the degree to which programs achieved reductions in the rate of substance use by participants relative to comparison youth in each site. Figure 8 identifies the six program characteristics. The height of the bar indicates the relative magnitude of the effectiveness (effect size) for programs with that characteristic; the higher the bar, the stronger the effectiveness of the programs with that characteristic. The identification of these characteristics of effective prevention programs is an important contribution to science-based understanding of how to make prevention programs more effective for high-risk youth.
The figure demonstrates that the following program characteristics are important contributors to effective reduction of the rates of substance use among youth at high risk. These findings are explained and elaborated in Points of Prevention Monograph No. 3, Findings on Designing and Implementing Effective Prevention Programs for Youth at High Risk.
The positive effects of these science-based practices are demonstrated in the substance use rates of youth participating in programs that incorporate all or most of these practices. Figure 9 shows the relative trends for youth who participated in the seven study programs that were characterized by at least five of the practices described above.(7) The programs provide strong prevention benefits that last throughout the 18-month followup period of the study.
(7) An eighth site with 5 positive characteristics was not included in this analysis because it was an outlier with respect to the magnitiude of its positive effects. Including that site would have created a graph unrepresentative of typical program effectiveness.
Through identifying the design and implementation features that characterize more effective prevention in real community settings, the evaluation has contributed important science-based guidance for strengthening future prevention initiatives and programs.
(7) An eighth site with 5 positive characteristics was not included in this analysis because it was an outlier with respect to the magnitude of its positive effects. Including that site would have created a graph unrepresentative of typical program effectiveness.
Since the National Cross-Site Evaluation was studying youth in actual community settings, evaluators considered that some of the youth in comparison groups would have the opportunity to enter prevention programs other than the CSAP program under study. To identify the degree to which this occurred, and the effects on study findings, data were gathered on the availability of prevention programming in the school and community environments of study youth. When study communities were divided into those that offered higher and lower opportunity for participation in prevention programs, an important finding emerged.
Results from the National Cross-Site Evaluation of High-Risk Youth indicated that comparison youth in sites with high opportunity for prevention participation have lower rates of use than youth in sites with fewer available prevention resources. This supports the cross-site evaluation conclusion that "Prevention Works" for high-risk youth. The results indicate that prevention effects in sites with higher availability of prevention services will be underestimated because comparison youth in these field settings also benefit from prevention. To make sure the findings of the study are accurate, estimates of the magnitude of prevention effects on youth are made using the 23 sites in which comparison youth had lower opportunity to participate in prevention programs. This finding also suggests that prevention effectiveness is often underestimated in evaluation studies where comparison youth have opportunities to participate in prevention activities within the community.
| Prevention effectiveness is often underestimated in studies when comparison group youth participate in prevention activities. |
Information on in-school and community prevention programming indicates that many of the HRY programs operate in communities that offer comparison youth opportunities to participate in several different types of prevention programming. When the data were pooled for all youth across the 48 sites, the data showed that there were nearly as many protective factors in place for comparison youth as for HRY program participants. Those protective factors include school prevention programming and involvement in organized community activities such as youth organizations, after-school classes, and organized athletics. In 20 programs, participation in prevention and organized activities was actually higher for comparison youth than for HRY program youth, because programs other than the CSAP-funded program were available in their communities.
To address the degree of comparison group exposure to prevention services, the evaluation researchers divided study sites into two categories. The 23 sites in which comparison group youth had little access to prevention services were termed low-exposure sites. The remaining 25 sites at which comparison group youth had plentiful opportunities for exposure to organized prevention services were termed high-exposure sites. The expectations were, first, that this study might identify the degree of influence of prevention programming on comparison youth in the 25 high-exposure sites. The second expectation was that it would be harder to detect the impact of program participation on participants in high-exposure sites than in low-exposure sites.
Looking at comparison youth only, youth in high-exposure sites reported less substance use from baseline to exit than did youth in low-exposure sites (Figure 10). More specifically, comparison group youth who had greater opportunities to take part in prevention activities reported 57 percent to 68 percent smaller increases in alcohol, marijuana, and cigarette use than the comparison youth who had fewer opportunities for prevention participation. This finding among comparison group youth indicates that widely available prevention activity reduces the degree to which youth begin and increase substance use in the critical adolescent years.
These results have major implications for the interpretation of findings in the Cross-Site Evaluation:
First, the results indicate that comparison youth in sites with high exposure to prevention services benefit from community exposure to those services. This pattern strengthens support for the National Cross-Site Evaluation conclusion that "Prevention Works." Where comparison group youth are exposed to high availability of community prevention, they report smaller increase in substance use as they get older.
Second, the results suggest that the study underestimates the effects of the study programs in sites where exposure to prevention activities is high. In these sites, comparison youth also benefit from community prevention services. To make sure this analysis is as objective as possible, only sites with low comparison group exposure to prevention services are used when the purpose of analysis is to estimate the effectiveness of CSAP-funded programs. Where the focus of the analysis is the relationship of program characteristics to program effectiveness, the full sample of sites is used because the additional information provided by high-exposure sites is important. It is important to recognize, however, that analyses using the full site sample underestimate the effectiveness of programs in high-exposure sites.
For the larger understanding of prevention, this finding reinforces the positive effects of prevention programming in the community. Furthermore, it suggests that prevention's effectiveness has probably been underestimated in research where community prevention programming was available but not measured or considered in interpreting study results.
The CSAP National Cross-Site Evaluation of High-Risk Youth Programs provides science-based guidance for improved prevention policy and practice. For policymakers, this rigorous study of a broad range of programs implemented in actual communities demonstrates the effectiveness of prevention for at-risk youth.
The data re-emphasize that substance abuse prevention programs are a proactive step toward reducing substance use or keeping it from beginning. Among the findings from the HRY evaluation that will help policymakers are:
For prevention practitioners, the study provides lessons on how prevention programming can be refined and strengthened in several areas. The findings have important implications for prevention design and implementation. By contributing to understanding the factors that prevent substance abuse in youth, and producing science-based lessons about effective prevention strategies and program practices, this major research effort holds the promise of more effective prevention and more positive futures for America's youth who are high risk.
For teachers, counselors, and other substance abuse prevention practitioners, this research can help identify "blueprints" of effective practice to help guide their efforts to promote positive youth development. Prevention professionals can use the information generated by CSAP's study to improve their aim at this moving target of approaches. This diverse high-risk study sample provides an excellent proving ground for identifying connective paths among risk and protection factors and substance use. It also provides a basis for assessing the appropriateness of specific prevention objectives and practices in a variety of settings. Among the findings from the High-Risk Youth Evaluation that will help practitioners nationwide are:
The Cross-Site Evaluation implemented a comprehensive and rigorous study design marked by six major features: sample diversity, comparison samples, a common survey instrument, multiple measurement points, program contact data, and program-level measures. Each is explained below.
| The Cross-Site Evaluation design has generated a large and comprehensive data set that supports analyses of prevention effectiveness from a variety of perspectives. |
The Cross-Site Evaluation design has generated a large and comprehensive data set that supports analyses of prevention effectiveness from a variety of perspectives, including analyses that distinguish between program effects that occur while youth are in the program and those that occur after the program ends. The multiple levels of data allow prevention effectiveness to be assessed on both a program and individual basis. Program-level findings permit assessments and comparisons of program effectiveness. Individual-level findings support conclusions about the effects of prevention activities on youth across programs.
As part of the grant application process, CSAP grant recipients describe how they recruit youth who are at high risk for substance use. Most typically, recruitment focuses on communities that are characterized by high levels of risk, such as poverty, crime rates, or ambient substance use rates, that are known to be associated with higher than average levels of substance use in youth. Figure 12 compares use rates of 12- to 17-year-olds in the study sample with those of youth who participated in the 1998 National Household Survey on Drug Abuse (NHSDA), a randomly sampled general population survey of persons age 12 years and older.
| The youth in the cross-site sample reported higher use for all substances within all age groups than did the general population. |
The youth in the cross-site sample reported higher use for all substances within all age groups than did the general population. Also, the use rates for cross-site youth rise more rapidly with age than do those in the general population (Figure 12), a pattern that is particularly noticeable for marijuana. Although the circumstances of the NHSDA and the National Cross-Site Evaluation data collection are not identical, this comparison suggests that the cross-site programs served youth who were at higher risk for initiating substance use when young.
The majority were not substance users at the baseline measurement, which was entry into the prevention program. Figure 13 displays, by age and sex, any self-reported use of cigarettes, alcohol, or marijuana in the 30 days before baseline data collection of youth in the study. Substance use is very low until about age 12 and then rises rapidly through the early teen years. Rates are somewhat lower for girls, particularly after age 14.
| The majority of youth in the sample (76%) report no substance use at baseline, creating a "floor effect" that makes it more difficult to detect positive program effects. |
The majority of youth in the sample (76%) reported no substance use at baseline. This finding indicates that youth are being recruited at an appropriate age for prevention, before they have initiated use, but it also presents an analysis problem because youth who do not use cannot "do better" at later points in time. Such a high occurrence of non-use at the baseline measurement creates a "floor effect" that makes detecting positive program effects more difficult. As noted earlier, this effect is one reason for the small magnitude of measured prevention effects in this study and other studies of programs serving young adolescents. In this study, separate analyses of youth who had already initiated use found larger positive program effects.
Because 19 of the sample programs were funded through the Female Adolescents Initiative, two-thirds of the sample are girls. The cross-site programs targeted a diversity of racial and ethnic groups (Figure 14). More than one-third of the youth are African American; approximately one-fourth are Hispanic; and the remaining youth are relatively evenly distributed among Native American (13%), non-Hispanic White (12%), and Asian or Pacific Islander (11%).
| The diversity of the youth sample strengthens the ability of the study to identify general characteristics of youth at risk and how prevention programming impacts them. |
CSAP's National Cross-Site Evaluation of High-Risk Youth Programs has contributed important new information that documents the effectiveness of prevention programming, and provides specific guidance on how to strengthen prevention programs in communities across the Nation. This monograph has identified and elaborated on six major findings that have implications for prevention policy and practice. Not only does the study demonstrate that "prevention works," but it goes further to identify what works and why. For prevention practitioners, the study provides lessons for refining and strengthening prevention programming in several areas. The evaluation also has produced important implications for prevention design and implementation.
An initial objective of the National Cross-Site Evaluation was to use rigorous research to examine the experiences of prevention programs to extract lessons for improving future programming. This monograph summarizes findings that are important for realizing these objectives. Further analyses in a "Points of Prevention" monograph series describe in more detail the findings of the study and their many implications for prevention practice. The final step belongs to the prevention policymakers and practitioners whose charge it is to put these lessons to work for the benefit of our Nation's youth.
Battistich, V., Schaps. E., Watson, M., and Solomon, D. (1996). Prevention Effect of the Child Development Project: Early Findings From an Ongoing Multisite Demonstration Trial. Journal of Adolescent Research, 11: 12-35.
Bryk, A. S., and Raudenbush, S. W. (2000). Hierarchical Linear Models. Newbury Park, CA: Sage. Center for Substance Abuse Prevention (CSAP). (1993). Signs of effectiveness: In prevention alcohol and other drug problems. (DHHS Pub. No. (SAM) 98-2001). Rockville, MD: Substance Abuse and Mental Health Services Administration.
Center for Substance Abuse Prevention (CSAP). (1999a). CSAP National Cross-Site Evaluation of High-Risk Youth Programs year four technical report. Rockville, MD: Department of Health and Human Services.
Cohen, J. (1988) Statistical power analysis for the behavioral sciences, 2nd edition. Hillsdale, NJ: Earlbaum.
Joreskog, K. G. and Sorbom, D. (1999). Lisrel 8: New Statistical Features. Chicago: SSI. Resnick, M.D. (1999). The impact of caring and connectedness on adolescent health and well-being: Research summary. In getting results: Developing safe and healthy kids, Update 1 (p. 33). Sacramento, CA: California Department of Education.
Sambrano, S., Springer, J.F., and Hermann, J. (1997). Informing the next generation of prevention programs: CSAP's Cross-Site Evaluation of the 1994-1995 High -Risk Youth Grantees. Journal of Community Psychology, 25: 375-395.
Tobler, N.S. (1986). Meta-analysis of 143 adolescent drug prevention programs: Quantitative outcome results of program participants compared to a control or comparison group. Journal of Drug Issues, 16, 537-567.
Tobler, N.S., Roona, M.R., Ochshorn, P., Marshall, D.G., Streke, A.V., and Stackpole, K.M. (2000). School-based adolescent drug prevention programs: 1998 meta-analysis. The Journal of Primary Prevention, 20, 275-336.
Tobler, N.S., and Stratton, H.H. (1997). Effectiveness of school-based drug prevention programs: a meta-analysis of the research. The Journal of Primary Prevention, 18, 71-128.
| |||||||||||||||||||||
| Initiatives | Funding | Home | ||||||||
|
Site Map
Help
Account
Cart
Home
