This Web site is a component of the SAMHSA Health Information Network.

Assessing Costs and Benefits
of Early Childhood Intervention Programs:
Overview and Application to the
Starting Early Starting Smart Program

EXECUTIVE SUMMARY

Lynn A. Karoly
M. Rebecca Kilburn
James H. Bigelow
Jonathan P. Caulkins
Jill S. Cannon
James R. Chiesa

Spring 2001

Contents

Preface

Acknowledgements

Acronyms

Executive Summary

The Cost and Outcome Analysis Framework

Applying the Framework to Early Childhood Interventions

Some Illustrative Analyses

Framing a Policy Scorecard Analysis for a Specific Program

Conclusions

Appendix
A. STARTING EARLY STARTING SMART GRANT SITES
B. SESS PROGRAM ACKNOWLEDGMENTS
C. MISSION STATEMENTS OF THE NATIONAL COLLABORATORS

References

Preface

The increased interest in the potential for early childhood intervention programs to save dollars in the long run has focused attention on the potential for cost-benefit and related analyses to aid decision-makers in their policy choices. The goal of this report is to identify the conceptual and methodological issues associated with the analysis of costs and outcomes of early intervention programs in general and to make recommendations regarding the application of these tools for subsequent demon­stration studies of a particular intervention program: Starting Early Starting Smart (SESS).

SESS is a public-private collaboration designed to test the effectiveness of integrating behavioral health services within primary care and early childhood service settings for children from birth to age seven. The SESS program is an initiative of the Office on Early Childhood, Substance Abuse and Mental Health Services Administration (SAMHSA), and the Casey Family Programs, along with several other federal spon­sors. The program currently operates in 12 sites across the United States and is entering the third year of its first five-year phase. An outcomes evaluation is built into the first phase.

Program sponsors are beginning to plan for a second phase, the design of which they hope will be informed by the first phase. It was during the initiation of this planning process that program sponsors identified a need for cost information to supplement their outcomes information. Recognizing that the literature offered somewhat lim­ited guidance on the specifics of cost considerations in this context, they requested that RAND not only present them with a summary of research bearing on their problem but that we also examine their program and make specific recommenda­tions regarding how cost and outcome analysis could improve their decision­making.

This project began with a meeting of cost and outcome analysis experts held in August 2000, convened by RAND on behalf of the Casey Family Programs and the Office on Early Childhood, SAMHSA. Participants at the meeting included four national experts in cost and outcome analysis with backgrounds in mental health and substance abuse, as well as several RAND staff members with experience in cost and outcome analysis. Also participating were staff from SAMHSA, the Casey Family Programs, the SESS Data Coordinating Center, and two of the SESS program sites. The proceedings from the meeting are summarized in the following document:

Cannon, Jill S., Lynn A. Karoly, and M. Rebecca Kilburn, Directions for Cost and Outcome Analysis of Starting Early Starting Smart: Summary of a Cost Expert Meeting, CF-161-TCFP, Santa Monica, California: RAND, 2001.

Readers interested in more detail are urged to obtain a copy of the full report, AsSESSing Costs and Benefits of Early Childhood Intervention Programs: Overview and Application to the Starting Early Starting Smart Program, by Lynn A. Karoly, M. Rebecca Kilburn, James H. Bigelow, Jonathan P. Caulkins, and Jill S. Cannon, which can be obtained from RAND (www.rand.org), the Casey Family Programs (www.casey.org/projects.htm#SESS), or SAMHSA (www.samhsa.gov).

This research is funded by the Casey Family Programs. The opinions expressed and conclusions drawn in this report are the responsibility of the authors and do not represent the official views of the Casey Family Programs, SAMHSA, other agencies, or RAND.

Acknowledgements

We thank our sponsors at the Casey Family Programs and the Substance Abuse and Mental Health Services Administration (SAMHSA) for their support of this project. We especially acknowledge the valuable guidance we received from Peter Pecora at the Casey Family Programs, and Michelle Basen and Patricia Salomon from the Office on Early Childhood at SAMHSA. We also benefited from discussions with a number of cost and outcome analysis experts who specialize in mental health and substance abuse interventions. In particular, we thank Anthony Broskowski (President, Pareto Solutions), William A. Hargreaves (University of Califor­nia, San Francisco), Brenda Reiss-Brennan (President, Primary Care Family Therapy Clinics, Inc.), and Brian T. Yates (American University). These four indi­viduals served as expert panelists for a two-day meeting convened by RAND in August 2000 to discuss the potential use of cost-benefit analysis or related methods in the analysis of subse­quent knowledge development and application studies of the Starting Early Starting Smart (SESS) program, a public-private initiative funded and directed through SAMHSA and the Casey Family Programs. This document was shaped by the insights they offered during and after the meeting. In addition, we benefited greatly from information about the current SESS evaluation provided by Fred Springer and his colleagues at EMT Associates, which serves as the SESS Data Coordinating Center. The perspectives on the SESS program offered by David Deere and Karen Rossmaier from the Russellville, Arkansas, site, and Miriam Escobar and K. Lori Hanson from the Miami, Florida, site were also insightful. Other participants at the meeting from SAMHSA and the Casey Family Programs also provided useful input.

We are also grateful for the careful technical reviews provided by our RAND colleagues Carole Roan Gresenz and Shin-Yi Wu. Other valuable comments on the draft report were provided by Richard Boyle and Mark Friedman. Finally, we also thank Patrice Lester and Claudia Szabo of RAND for their very able assistance with the assembly and production of the docu­ment.

Acronyms

EC
Early childhood (program)

PC
Primary care (program)

PEIP
Prenatal/Early Infancy Project

SAMHSA
Substance Abuse and Mental Health Services Administration

SESS
Starting Early Starting Smart (program)

Executive Summary

Agency and program administrators and decision-makers responsible for implement­ing early childhood intervention programs are becoming more interested in quanti­fying the costs and benefits of such programs. Part of the reason for this is that foundations and other funders are putting more emphasis on results-based account­ability. At the same time, arguments for the value of early childhood intervention are being made within the public sphere on the basis of published estimates of costs and benefits. Program implementers are naturally attracted by statements that a certain intervention produces $4 in savings for every $1 it costs and would like to make simi­lar statements about their own programs. Meanwhile, decision-makers without par­ticular interest in any given program would like more quantitative decision aids when it comes time to choose among a variety of possible program models or pro­gram improvements to implement.

Our objective here is to offer assistance to decision-makers and program imple­menters considering an assessment of costs and outcomes. We do not offer a specific step-by-step manual, but we discuss the kinds of issues that must be taken into account and why. We do so in enough detail that readers can decide if this type of quantitative analysis is the right course for them and, if so, can knowledgeably inter­act with an expert cost-outcome analyst. While we understand that some readers will want to undertake analysis of costs and outcomes to justify a program in which they have a special interest, we take the viewpoint here of an unbiased allocator of funds. What evidence should such a person want to see before concluding that a particular intervention is a wise investment? That sort of evidence is what the implementer seeking to justify further funding will need to present.

We begin by setting the conceptual framework within which program costs and out­comes may be understood. We then draw out some of the implications of that gen­eral framework for the analysis of early childhood interventions in particular. After reviewing some examples of such analyses, we apply the methodology to an actual case in which a consortium of program funders must decide whether to proceed with an assessment and, if so, what kind of assessment to undertake. The consortium is led by the U.S. Substance Abuse and Mental Health Services Administration and the Casey Family Programs, and the intervention of interest is the Starting Early Starting Smart Program.

The Cost and Outcome Analysis Framework

Decision-makers and program implementers just beginning to think about analyzing costs and benefits are often surprised to learn that several analytic avenues are open to them. Which one or ones they choose will have important implications for what they learn and how much they must spend to learn it. Among the choices are these: [1]

• Cost-benefit analysis (or benefit-cost analysis) entails comparing a program’s benefits to a stakeholder with its costs to that stakeholder. Such a comparison requires putting benefits and costs in comparable terms, and the terms conven­tionally chosen are dollars. Benefits that cannot be expressed in dollar terms cannot be compared in this manner and are included only in associated qualita­tive discussion. Cost-benefit analysis seeks to help in deciding whether a pro­gram is of value to the stakeholder. Often cost-benefit analysis is conducted from the perspective of society at large. [2]

• Cost-savings analysis is restricted to the costs and benefits realized by the government as a whole or a particular funding agency. Only the costs to the government are taken into account, and the benefits are those expressible as dollar savings somewhere in the government. This kind of analysis is used to deter­mine whether a publicly provided program –pays for itself” and is thus justified not only by whatever human services it may render but also on financial terms alone.

• Cost-effectiveness analysis determines how much must be spent on a program to produce a particular outcome (or, what is equivalent, how much of a particular type of benefit will result from a given expenditure). While this can be done for multiple outcomes, no attempt is made to sum the complete array of benefits into a single aggregate measure.

• Cost analysis alone (no measurement of benefits) can be useful to decision-makers for a variety of purposes, for example, discovering which factors need to be considered in replicating a program elsewhere or for informing budget projections.

In deciding which avenues to pursue, the decision-maker or implementer must choose what he or she wishes to learn and consider the funds available for undertaking the analysis. The analyses above are ordered in terms of how much attention must be paid to quantifying outcomes and expressing them in dollar terms (from a lot at the top to none at the bottom). Other variables being equal, the resources and calendar time devoted to the analysis will drop with each successive approach down the list.

As we describe them here, these cost and outcome analysis methods are used only as components within a broader decision support framework that we call policy analysis or policy scorecard analysis (the latter term derives from the use of a tool called the scorecard). [3] Despite the name, it does not pertain only to high-level public policies but also to decisions made regarding specific strategies and programs. Policy score­card analysis offers a framework within which to consider multiple benefits, as required in the first two approaches listed above, and multiple costs, as required by all four. Policy scorecard analysis also entails consideration of alternative programs. This is important for benefit and cost analysis. In trying to determine whether the numbers emanating from these analyses support (further) investment in the pro­gram, funders will be asking, compared with investment in what else? A benefit-cost ratio of 1.5 to one ($1.50 of benefits for every dollar of costs) may not be good enough if an alternative with similar objectives has a ratio of two to one. Decision-makers will thus be considering a range of alternative interventions or at least a choice between funding the program in question and some default course of action (which could be leaving things as they are).

The results of a policy scorecard analysis can be summarized in a simple tool called a scorecard. The scorecard lists benefit and cost categories down the side, together with program design features influencing them, and the alternative courses of action across the top. Thus, each cell in the scorecard gives a particular cost or benefit (or design feature) for a particular program. In identifying the row and column heads and filling in the cells—that is, in conducting the policy scorecard analysis—several guidelines must be kept in mind:

• Designate which benefits and costs accrue to which stakeholders. If you say that a program generates more savings than costs, people will want to know, savings to whom? And costs to whom?

• Define explicitly the period over which the analysis applies. If the purpose of the analysis is to determine whether a program has a favorable benefit-cost ratio or pays for itself in government savings, it is better to look well into the future. No one period or duration is correct, however. The choice depends on the patience of the decision-maker in question, with individuals typically having shorter planning horizons than society as a whole. This distinction makes a difference because the costs of early intervention programs typically accrue over a matter of months or a few years, whereas the benefits are often not fully realized until the participating children age into adulthood. Counting such benefits directly entails long-term follow-up of program subjects, though some future benefits can be predicted on the basis of shorter-term trends.

• Discount future costs and benefits. Although it is important to count future benefits (and costs), they cannot be counted at full, nominal value. People discount future benefits and costs: getting a $1,000 benefit five years in the future does not look as attractive as getting it now; having to pay $1,000 five years in the future does not seem as onerous as having to pay it now. A real annual discount rate of 3 percent to 6 percent is typically applied to future benefits and costs.

• Record cost elements as resource quantities. Until the figures are added up at the end, costs should be recorded in terms of resource quantities—hours of labor, square footage of rental space, etc.—rather than in dollar terms. Prices for these resources can vary from one site to another, and on-budget dollars in particular do not always reflect total costs. A physician may donate time on the weekends, but from society’s point of view, that time is not free; perhaps it could have been put to another, more beneficial use.

• Address uncertainty. Future benefits and costs cannot often be predicted with great confidence. Where a range of values is plausible, that range should be made explicit in the analysis. Likewise, structural uncertainty (e.g., about possible future changes in laws relevant to a program) should also be considered.

The final step in the cost and outcome analysis is to add up all the benefits (or savings) and add up all the costs and compare them across programs. The four methods listed above offer alternative ways for performing this step. Cost-benefit and cost-savings analysis each provide a single measure of merit for each alternative; the alternative with the greatest merit according to this measure is declared the winner. Cost-effectiveness analysis provides multiple measures of merit. They can be combined into a single measure (e.g., the ratio of effectiveness to cost, if a single effectiveness measure dominates), which will be used in the same way as a cost-benefit or cost-savings measure. Or they can be used to define a different kind of selection rule, one that deems best the policy that achieves a specified level of effectiveness at lowest cost (a constant effectiveness analysis) or that achieves the greatest effectiveness for a given cost (a constant cost analysis). [4]

Comparing costs and benefits may not produce a single answer that one program is obviously preferable to another. One program may produce a net benefit to one group of stakeholders, while another benefits a second group. The net benefit of one program may be somewhat higher than that for another, but the uncertainty ranges may overlap so much that the advantage cannot be asserted with high confidence. Some possible change in the institutional environment, e.g., tax reform, could shift benefits and costs enough to change the advantage from one program to another. Such possibilities would not subtract from the value of the cost and outcome analysis. On the contrary, some of the most valuable insights are suggestions for policy changes that reallocate benefits across stakeholder groups so that all of them gain and thus have no incentive to block a program.

In most studies, the majority of the analytical effort will come from learning about the domain, structuring the models of how the intervention works, collecting and cleaning data, etc. In short, filling in the scorecard is challenging. Given that groundwork, computing the summary evaluation metrics is straightforward, whether that metric is a benefit-cost or a cost-effectiveness ratio.

Hence, instead of suggesting that one must choose to implement one of these four approaches, it is more accurate to say that one must choose whether or not to conduct a careful, quantitative summation of the effects of the program. If the answer is yes, then there follows a choice of how one is going to present the results of that analysis to decision-makers, as a benefit-cost ratio, cost-effectiveness ratio, and so on, or some combination thereof.

It is thus important to keep cost-benefit analysis, cost-savings analysis, and other forms of cost and outcome analysis in their place. In any decision, some factors can be resolved only through a decision-maker’s values and subjective judgment or through negotiation among stakeholders. Likewise, the public quantifying of decision factors may occasionally be problematic (e.g., when an auto manufacturer compares the cost of a safety improvement with the dollar-equivalent benefit of the lives that could be saved by that design change). Nevertheless, these methods can provide valuable input to choosing among different programs, demonstrating a program’s worth, improving programs, and replicating them.

Applying the Framework to Early Childhood Interventions

Early intervention programs attempt to improve child health and development by providing young children and their families various social services and supports. Such programs can have effects in four domains: emotional and cognitive development, education, economic well-being (in terms of public assistance, income, and crime), and health. Specific examples of possible benefits within each of these categories are given in Table S.1. Which benefits are measured depends on the purpose of the analysis. Cost-benefit and cost-savings analyses typically seek a comprehensive accounting of the benefits to society or to government (respectively), although many benefits are difficult to express in dollar terms and therefore cannot be aggregated in the cost-benefit asSESSment. While cost-effectiveness analysis can in principle be performed for any outcome, it is often the case in practice that a single benefit or a narrow set receives most of the attention. A full analysis of the benefits of an early intervention program should include collection of data on as many potential benefits as the analyst’s resources permit.

Note that early childhood interventions can benefit parents and other caregivers while simultaneously helping children. It is important to measure benefits to caregivers, because these are often realized over much shorter time periods than are those accruing to children. Ignoring these benefits means underestimating a program’s benefit-cost ratio or its potential net savings to government, particularly over the short term and for some analyses, it will only be feasible to make short-term measurements.

Any analysis of benefits of a program under way must include a comparison group. This is a group of children and caregivers not enrolled in the program but similar in as many ways as possible to the program participants and whose progress along the various benefit measures is tracked. [5] Children in particular have a tendency to improve along various measures of development as they grow. Evaluators must take care to ensure that the program benefits they measure are net of what would have occurred naturally or what children would realize anyway from outside influences without the program. Measurements of the comparison group provide estimates of benefits that would have accrued in the program’s absence.

Data on progress along benefit measures can be collected by survey questionnaires, tests, or other means of direct interaction with the children and their caregivers. For some benefit types (e.g., reductions in involvement with the criminal justice system), administrative data may be available. When only a few years of data collection are feasible, a glimpse into the future can be obtained through mathematical models that can predict future criminal activity or future earnings on the basis of childhood information. (This cannot of course be done with confidence for any given child, but results obtained for a group of children may be sufficiently reliable for the purpose.)

As with benefits, the cost elements to be included in an analysis depend on its purpose. For example, costs that accrue to society but not to a funding agency are included in a societal cost-benefit analysis but omitted from a cost-savings analysis. Regardless of the analysis to be performed, program costs must be estimated as net of those accrued by comparison group children for similar services. For example, if an intervention is intended to increase prenatal care, the analysis should include only the resources devoted to the visits and services received by program participants in excess of what they would have received anyway (i.e., in excess of those received by the comparison group).

Estimation of costs should follow the general guideline given above regarding the need to estimate resource quantities instead of dollars and to account for opportunity costs and other off-budget resource expenditures. Costs borne by participants should also be included, as well as costs borne by other agencies or ser­vice providers. Collecting cost data for the same set of service providers for both the treatment and control groups allows the analyst to detect both cost shifting (e.g., from one payor to another) and cost offsets (e.g., reduced utilization of services in one area as a result of increased service use in another). It may also be useful to dis­tinguish between the fixed costs of implementing a program that are not depen­dent on the number of children served and the variable costs that are. The split between fixed and variable costs will influence the calculation of benefit-cost ratios, net savings, and cost-effectiveness ratios for programs when scaled up to serve larger numbers of children.

Some Illustrative Analyses

Given the challenges and requirements outlined so far, it should not be surprising that not many scientifically sound cost-benefit and cost-savings analyses of early childhood intervention programs with long-term follow-ups have been conducted. Among those recently analyzed or reanalyzed are the following:

• The Perry Preschool program provided center-based classes and teacher home visits for one or two school years to 58 children ages three or four in Ypsilanti, Michigan, from 1962 to 1967. Benefits were tracked for both the participants and the comparison group (65 children) through age 27. Benefits included better school performance, higher employment, less welfare dependence, and lower involvement in criminal activity on the part of participants. The most recent cost-benefit assessment evaluates benefits expressible in monetary terms at $50,000 per child, half of that in the form of savings to government, versus a program cost of $12,000 per child (see Figure S.1).

• In the Prenatal/Early Infancy Project (PEIP) in Elmira, New York, nurses started visiting mothers when they were pregnant and continued until their child was age two. The objective was to improve pregnancy outcomes and parenting skills and link the mother with social services. Between 1978 and 1980 the program reached 116 first-time mothers. They and another 184 in the control group have been followed through age 15 of the first-born child. Benefits for the mothers included better pregnancy behaviors and less child abuse in the short term and lower welfare participation and criminal behavior in the long term. The children benefited as well in several domains. For the higher-risk portion of the sample (unmarried mothers with low socioeconomic status), benefits amounted to almost $31,000 per mother-child pair, with almost half of that in the form of a reduction in welfare received by the mother. For the lower-risk portion of the sample, however, benefits came to only $6,700. Program costs were about $6,100.

• The Chicago Child-Parent Centers have promoted reading and language skills, provided health and social services, and promoted parent involvement for children in preschool through third grade. A cohort of 989 children completing kindergarten in 1986 was tracked to age 20 and compared with a no-preschool group of 550 children. The program resulted in long-lasting educational-achievement benefits. Higher between-grade promotion rates, reduced special-education use, increased earnings expected as a result of better educational performance, and lower involvement with the juvenile justice system translated into about $35,000 in benefits per program participant. The program cost nearly $10,000 per participant.

Figure S.1—Some Early Childhood Interventions Have Been Shown to Have High Benefit-Cost Ratios

These analyses demonstrate that early childhood interventions can generate savings to government and benefits to society that exceed program costs. Indeed, for most of the samples reported above, benefits were a multiple of costs, and all of these programs resulted in benefits that could not be translated into costs and were thus omitted. Therefore, decision-makers and implementers thinking about performing analyses of costs and benefits should not give up merely because they don’t see how some of a program’s principal benefits can be converted to dollar terms.

Two further lessons for cost-benefit analysis may be drawn from these examples. First, many important benefits can only be captured through an extended time horizon. The savings from Perry Preschool, for example, did not accumulate to match the level of program costs until the participants were 20 years old. Some of these benefits can be predicted on the basis of shorter trends, but not all can, and confi­dence in predicted results increases as follow-up periods lengthen.

Second, programs can be beneficial to caregivers as well as to children. In fact, when time is lacking for lengthy follow-ups or when they are not feasible, measuring benefits to caregivers can result in early favorable benefit-cost ratios and net savings. The Elmira program was the only one of those summarized that measured caregiver benefits, and, in that case, savings sufficient to balance costs were tallied within two years of the end of program services.

Framing a Policy Scorecard Analysis for a Specific Program

The Starting Early Starting Smart (SESS) program is intended to test the effectiveness of integrating mental health services and substance abuse prevention and treatment into early childhood education or primary health care for children from birth to age seven. The program is under way at 12 sites nationwide, seven using the early childhood (EC) education model and five using the primary care (PC) paradigm. (See the appendix of the full RAND SESS report for a description of each state.) Most of the sites serve between 100 and 300 children, and comparison groups average out to similar numbers.

By effectiveness, the program means increased access to, use of, and satisfaction with behavioral health services and increased social, emotional, and cognitive functioning on the part of served children. Data on these benefit measures are being collected over an 18-month follow-up period at intervals that average six months (PC sites) or nine months (EC sites). No cost data are being gathered in this first phase of the program, but a second phase is being planned, and part of that planning is to assess the feasibility of cost and outcome analysis.

SESS program implementers are wise to take cost and benefit evaluation issues into account in the planning stage. Too often, evaluation is considered only after program design has been finalized along lines that preclude sound cost and benefit assessment. SESS’s Phase I design raises issues that need to be resolved for Phase II if cost and outcome analyses are to be possible. One issue, for example, is that some sites did not use random assignment (primarily EC sites), which raises concerns about the validity of the treatment group versus comparison group difference as a measure of the true effects of the program. Future demonstration sites should aim for random assignment if at all possible. Another concern is that a few sites are experiencing relatively high dropout rates, which could bias benefit estimates if those who are lost to follow-up are different from those who remain in the study and if they differ in important ways that cannot be observed. Obtaining a consistently high follow-up rate across sites would need to be a priority in Phase II. Also, Phase I has been characterized by between-site variations in services. This is problematic from an evaluation standpoint for a couple of reasons: It complicates interpretation of results, and it complicates the design of comparison groups.

The design of comparison groups for SESS offers lessons for other programs. Because SESS attempts to integrate behavioral health services into existing early childhood and primary-care settings, only the benefits of the new, integrated services plus increases in the dosages of existing services may be credited to SESS, not the full benefits realized from participation in the early childhood program and primary care. Similarly, only the costs associated with these incremental activities should be considered. Therefore, the comparison groups must be designed to isolate the SESS effects by including everything except SESS. The appropriate comparison groups for this evaluation would consist of children involved in early childhood and primary-care programs without the integrated SESS services, not children receiving no services at all.

In the policy analysis scorecard, then, the columns would correspond to the early childhood program without SESS, primary-care program without SESS, and then the integrated EC plus SESS and PC plus SESS interventions, along with whatever variants are retained. The rows would be the program descriptors and cost and benefit categories. The program features reported would be those having implications for costs or benefits, e.g., population served, eligibility criteria, age of children at enrollment, qualifications of program personnel, types and dosages of services rendered, transportation provisions, and so on. In future demonstrations, this information can be collected through site visits and other mechanisms currently being used in the evaluation of Phase I.

Cost estimates would begin with the cost of serving one child (or child’s caregiver) in terms of labor hours expended with the child and in preparing for the session and in terms of materials consumed. These would then be multiplied by dosage per child and number of children served. Fixed costs unrelated to number of children served, such as space rental, would then be identified. Multiplication by unit costs to convert to dollars would be done last. Ultimately, the cost information should be as comprehensive as possible and comparable across demonstration sites.

Benefit measures now being collected for SESS include information on child problem behavior and social skills, child cognitive development, parent-child interaction, caregiver stress and negative or positive behaviors, caregiver mental health prob­lems, caregiver education and employment, and home environment. As discussed above, the emphasis on both child and caregiver benefits will be important to making the short-run benefit tally as complete as possible. Almost all of these measures, however, are within the domain of emotional and cognitive development and are not easily expressed in dollar terms. This makes a formal cost-benefit or cost-savings analysis problematic in that only a limited set of outcomes might possibly be valued in dollar terms to be compared with program costs. Unless the program impact for those outcomes valued in dollar terms is very large and favorable, so that sizable dollar benefits are generated, a cost-benefit analysis would be unlikely to show a favorable outcome for the SESS program based on the information available after two years.

While not the program’s main intent, other benefits could result from it. Some of these benefits, in such areas as physical health, labor market outcomes, and involvement with the criminal justice system, could be more easily expressed in dollar terms than those now being measured. These outcomes could be collected for parents or caregivers in the short term, and with longer-term follow-up, for the participating children. If behavioral changes are large in these areas as a result of the SESS intervention, they can produce sizable dollar benefits that, even when discounted, will be a large offset to the costs of the program. This is especially relevant for changes in parental behavior that can be measured even in the short run. Improvements of adult economic and health outcomes have been demonstrated to produce substantial short-run benefits in other early childhood programs.

Costs and outcomes would be measured for both the participant and comparison groups, with the difference between the two constituting the incremental cost and benefits from implementing SESS. To compare the present values of all costs and benefits, it will be important to predict how they will accrue over time. Costs and benefits should also be categorized according to which groups incur them. It will be of interest, for example, to know how much the intervention costs and benefits participants, the agency implementing the program, other agencies, and society as a whole.

Taking all these steps would be sufficient to support as full a cost-benefit or cost-savings analysis as is likely to be feasible given the current state of the art. If SESS decision-makers wish to be able to say something about the value the program returns to society relative to its costs, the preceding array of evaluation tasks and program design modifications would be required. If they decide it is enough to be able to say how much the program saves the government relative to what it costs, then some elements—costs to participants or losses to crime victims, for example—can be omitted. The overall level of effort required, however, is not likely to change very much.

If SESS funders or implementers would like instead to focus on one or a few promi­nent measures of effectiveness to compare the different SESS variants with each other, a cost-effectiveness analysis should be sufficient. By collecting cost data, along with data on that one or those few benefits, it would be possible to say, for example, how much child problem behavior decreased (relative to no SESS) per thousand dollars spent on SESS plus EC or SESS plus PC. No conversion of the benefit to dollar terms would be necessary.

Finally, if the purpose was to find out how much program modifications or proliferation of sites would cost, no benefit data would be necessary at all. Clearly, program decision-makers may have to make trade-offs between what they might like to achieve and how much of a resource commitment they are willing or able to make.

Conclusions

The recommendations we offer specific to the SESS program may be framed as a set of more-general guidelines for decision-makers considering cost and outcome analysis of an early childhood intervention program. In particular, among the recommendations that can be applied more broadly are the following:

· Regarding the design of a program evaluation and cost and outcome analysis:

· Specify the explicit goals of the cost and outcome analysis to guide the scope of cost and benefit data collection and analysis.

· Identify comparison groups and track the same cost and outcome measures for both comparison and participant groups. If possible, use random assignment to define comparison groups to provide a more valid test of intervention program effects.

· To minimize attrition in a longitudinal study, devote resources to retaining study subjects.

· Collect information on program features through site visits and other mecha­nisms to accurately characterize features of the intervention models as they are implemented and to ensure fidelity to the program model.

· Regarding the collection and analysis of cost data:

· Collect cost information for both treatment and comparison groups at each site where the intervention program is implemented.

· Ensure that the cost information is as comprehensive as possible: Costs borne by various parties should be differentiated, the period in which costs are incurred should be identified, and direct and indirect costs, fixed and variable costs, and goods and services provided in-kind should be measured.

· Plan for proper training and technical support of implementation sites and any cross-site data collection organizations to ensure uniformity in the col­lection of cost data. Collect information on the cost of data collection, training and support, and the related analyses of the data.

· Regarding the collection and analysis of outcome data:

· If cost-benefit or cost-savings analysis is the goal, Include in the outcome data information for parents and other caregivers in the short term and long term and for children in the long term in those domains with outcomes that can be readily evaluated in terms of dollars and can produce large dollar benefits. The choice of specific outcome measures should be guided by findings from related evaluation studies whenever possible.

· Obtain information from participants that facilitates collection of administrative data and allows effective tracking of individuals to increase response rates at later follow-ups.

· Where possible, collect complete histories using retrospective survey questions or administrative data for outcomes that may generate a continuous flow of dollar benefits (e.g., labor market outcomes, social welfare program use, use of costly health or education services).

· When supported by other empirical evidence, project future benefits based on observed outcomes. Consider additional method development that would permit such forecasts for a broader range of outcomes.

While we believe these principles are quite general, ultimately these recommendations should be viewed as guidelines that may need to be tailored to the specific circumstances of a given intervention program and its evaluation design. In the end, the objectives of a program’s decision-makers will dictate the shape of the analysis.

The general policy scorecard analysis tools considered in this report, and those specific to cost and outcome analysis, have great promise for improving decision-making with respect to such investment programs as the early childhood interventions represented by SESS and its counterparts. When used with skill and judgment, the application of these methods to other programs, such as SESS, will further broaden our base of knowledge regarding the value of these investments and aid decision-makers in their choice among program alternatives.

Appendix A
Starting Early Starting Smart Grant Sites

The SESS program is an initiative of the Office on Early Childhood, Substance Abuse, and Mental Health Services Administration (SAMHSA) and the Casey Family Programs, along with other federal sponsors. Patricia Salomon, Director of the Office of Early Childhood at SAMHSA, oversees the SESS program along with project officers Michele Basen, Velva Spriggs, and Jocelyn Whitfield, and staff Shakeh Kaftarian. At the Casey Family Programs, the partnership is overseen by Jean McIn­tosh and Barbara Kelly-Duncan, along with project officers Eileen O’Brien and Peter Pecora.

The SESS program currently operates in 12 sites across the U.S. Table A.1 lists each of the study sites and the associated principal investigator, project director, and local researcher, first for the primary care (PC) sites and then for the early childhood (EC) sites. [1] Information about the Data Coordinating Center and subcontractor is also provided in Table A.1. A brief description of the program at each site is provided in the appendix to the companion report. Further information about the SESS program is available from the Casey Family Programs (www.casey.org/projects.htm#SESS) and SAMHSA (www.samhsa.gov).

Table A.1

SESS Grantees within Primary Care and Early Childhood Groups
(in alphabetical order)

Appendix B
SESS Program Acknowledgments

The families and grantees of Starting Early Starting Smart (SESS) would like to acknowledge: Nelba Chavez, Ph.D., Administrator, SAMHSA, Rockville, Maryland and Ruth Massinga, M.S., President and CEO, Casey Family Programs, Seattle, Washington, along with the Casey Board of Trustees and the three SAMHSA Centers-Center for Substance Abuse Prevention, Center for Substance Abuse Treatment, and Center for Mental Health Services-for their vision and commitment to reaching families with very young children affected by environments of substance abuse and mental disorders. Without their innovative public-private partnership and unprecedented support, this initiative would have been impossible. We further acknowledge the early guidance and program development from Stephania O’Neill, M.S.W., Rose Kittrell, M.S.W., Hildy (Hjermstad) Ayers, M.S.W., Karol Kumpfer, Ph.D., Sue Martone, M.P.A., and Jeanne DiLoreto, M.S. Many thanks to the SAMHSA-Casey team for their tenacious efforts and unprecedented collaboration: Joe Autry, M.D., Acting Administrator, SAMHSA, and Jean McIntosh, M.S.W. Executive Vice President Casey Strategic Planning and Development Pat Salomon, M.D. Michele Basen, M.P.A., Velva Springs, M.S.W., Jocelyn Whitfield, M.A., Barbara Kelley Duncan, M.Ed., Peter Pecora, Ph.D., and Eileen O’Brien, Ph.D.

Appendix C
Mission Statements of the National Collaborators

Substance Abuse and Mental Health Services Administration (SAMHSA)
SAMHSA’s mission within the nation’s health system is to improve the quality and availability of prevention, treatment, and rehabilitation services to reduce illness, death, disability, and cost to society resulting from substance abuse and mental illness. SAMHSA’s mission is accomplished in partnership with all concerned with substance abuse and mental illness. SAMHSA exercises leadership in eliminating the stigma that impedes prevention, treatment, and rehabilitation services for individuals with substance abuse; developing, synthesizing, and disseminating knowledge and information to improve prevention, treatment, rehabilitation services, and improving the organization, financing, and delivery of these services; providing strategic funding to increase the effectiveness and availability of services; promoting effective prevention, treatment, and rehabilitation policies and services; developing and promoting quality standards for service delivery; developing and promoting models and strategies for training and education; developing and promoting useful and efficient data collection and evaluation systems; and promoting public and private policies to finance prevention, treatment, and rehabilitation services so that they are available and accessible. For more information, visit SAMHSA’s Web site at www.SAMHSA.gov .

Casey Family Programs
The mission of Casey Family Programs is to support families, youth, and children in reaching their full potential. Casey provides an array of permanency planning, prevention, and transition services, such as long-term family foster care, adoption, kin-ship care, job training, and scholarships. The program aims to improve public and private services for children, youth, and families impacted by the child welfare system, through advocacy efforts, national and local community partnerships, and by serving as a center for information and learning about children in need of permanent family connections. Casey Family Programs is a Seattle-based private operating foundation, established by Jim Casey, founder of United Parcel Service (UPS), in 1966. The program has 29offices in 14 states and Washington, D.C. For more information, visit our Web site atwww.casey.org.

References

Adler, M. D., and Posner, E. A. (2000). Introduction, to Cost-benefit analysis: legal, economic, and philosophical perspectives, a conference sponsored by the John M. Olin Foundation and University of Chicago Law School, Journal of Legal Studies, 29 (2, part 2), 837–842.

Barnett, S. W. (1993). Benefit-cost analysis of preschool education: Findings from a 25-year follow-up. American Journal of Orthopsychiatry, 63 (4), 500–508.

_____ (1995). Long-term effects of early childhood programs on cognitive and school outcomes. The Future of the Children, 5, 25–50.

Cannon, J. S., Karoly, L. A., & Kilburn, M. R. (2001). Directions for cost and out­come analysis of Starting Early Starting Smart: Summary of a cost expert meeting. Santa Monica, CA: RAND, CF-161-TCFP.

Caulkins, J. P., Rydell, C. P., Everingham, S. S., Chiesa, J., & Bushway, S. (1999). An ounce of prevention, a pound of uncertainty: The cost-effectiveness of school-based drug prevention programs. Santa Monica, CA: RAND, MR-923-RWJ.

Frank, R. H. (2000). Why is cost-benefit analysis so controversial? The Journal of Legal Studies, 29 (2), Part 2, 913–930.

Guralnick, M. J. (Ed.) (1997). Effectiveness of early intervention. Baltimore: Paul Brookes Publishing.

Hargreaves, W. A., Shumway, M., Hu, T.-W., & Cuffel, B. (1998). Cost-outcome methods for mental health. San Diego: Academic Press.

Karoly, L. A., Greenwood, P. W., Everingham, S. S., Houb?, J., Kilburn, M. R., Rydell, C. P., Sanders, M., & Chiesa, J. (1998). Investing in our children: What we know and don’t know about the costs and benefits of early childhood interventions. Santa Monica, CA: RAND, MR-898-TCWF.

Kitzman, H., Olds, D. L., Henderson, C. R., et al. (1997). Effect of prenatal and infancy home visitation by nurses on pregnancy outcomes, childhood injuries, and repeated childbearing: A randomized controlled trial. Journal of the American Medical Association, 278 (8), 644–652.

Olds, D. L., Eckenrode, J. Jr., Henderson, C. R., et al. (1997). Long-term effects of home visitation on maternal life course, child abuse and neglect, and children’s arrests: Fifteen-year follow-up of a randomized trial. Journal of the American Medical Association, 278 (8), 637–643.

Posner, R. A. (2000). Cost-benefit analysis: Definition, justification, and comment on conference papers. The Journal of Legal Studies, 29 (2), Part 2, 1153–1177.

Reynolds, A. J. (2000). Success in early intervention: The Chicago Child-Parent Centers. Lincoln, NE.: University of Nebraska Press.

Richardson, H. S. (2000). The stupidity of the cost-benefit standard. The Journal of Legal Studies, 29 (2, part 2), 971–1003.

Schweinhart, L. J., Barnes, H. V., & Weikart, D. P.(1993). Significant benefits: The High/Scope Perry Preschool Study through age 27. Ypsilanti, MI: High/Scope Educational Research Foundation, Monographs of the High/Scope Educational Research Foundation, No. 10.

Sen, A. (2000). The discipline of cost-benefit analysis. The Journal of Legal Studies, 29 (2, part 2), 931–952.

--------------------------------------------------------------------------------

[1] Terminology in this field has not been standardized, and these terms appear in the literature with a vari­ety of different meanings. We have chosen typical definitions.

[2] Of the four analytic approaches listed here, cost-benefit analysis is subject to the greatest chal­lenges in execution and interpretation. That is because benefits must be denominated in dollars, and that adds another source of uncertainty and potential disagreement over quantities. For some benefits, dollar con­versions are not really feasible. Cost-benefit assessments can thus rarely be comprehensive.

[3] The term policy analysis was originally adopted by RAND analysts and others to describe an approach for quantitatively analyzing management problems. Today, the term is used even more broadly to charac­terize a wide range of quantitative and qualitative approaches to addressing policy issues. Hence, we will employ the more focused term policy scorecard analysis for the remainder of this summary.

[4] The latter is sometimes called a constant budget analysis, but this is only appropriate if all the costs appear in the budget of the agency making the decision. In many programs, costs may be distributed across many stakeholders. They will not all appear in any single party’s budget.

[5] Ideally, one should randomly assign children and caregivers to program participation versus the comparison group. This ensures that the participation and comparison groups are (statistically) identical in both measured and unmeasured characteristics. When the comparison group is selected by random assignment, it is often called a control group. When random assignment is not feasible or desirable, a comparison group can still be chosen, by identifying children and caregivers who are similar in various measured ways to the program participants.

****************

About Starting Early Starting Smart

· Create and test a new model for providing integrated behavioral health services (mental health and substance abuse prevention and treatment) for young children (birth to 7 years) and their families; and to

· Inform practitioners and policymakers of successful interventions and promising practices from the multi-year study, which lay a critical foundation for the positive growth and development of very young children.

The SESS approach informs policymaking for:

· Service system redesign
· Strengthening the home environment
· Using culture as a resource in planning services with families
· Service access and utilization strategies
· Targeting benefits for children
· Working with families from a strengths-based perspective

In October 1997, with initial funding of $30 million, the Substance Abuse and Mental Health Services Administration (SAMHSA) and Casey Family Programs embarked on a precedent-setting public/private collaboration. Twelve culturally diverse grantee organizations were selected. Each provides integrated behavioral health services in community-based early childhood settings—such as Child Care, Head Start and Primary Care Clinics—where young families customarily receive services for children. Critical to this project is the required collaboration among funders, grantees, consumers, and local site service providers. Implicit in the design of this project is sustainability planning for secured longevity of the programs.

The Study Design

The 12 grantees, working collaboratively, designed a study whereby integrated behavioral health services are delivered in typical early childhood settings. Each site has an intervention and comparison group, and each site delivers similar targeted, culturally-relevant, interventions for young children and their families. A collaboratively determined set of outcomes has been established to evaluate project effectiveness:

· Access to and use of services
· Social, emotional, and cognitive outcomes for children
· Caregiver-child interaction outcomes
· Family functioning

The goal of the SESS research is to provide rigorous scientific evidence concerning whether children and families participating in SESS programs achieve better access to needed services and better social, emotional, cognitive, and behavioral health outcomes than do the children and families not receiving these services. SESS programs may also generate information about opportunities, practices, and barriers to sought-after outcomes. This information is critical to achieving effective public policies.

SESS Extended
It was clear from the early days of SESS that whatever effects were uncovered, longitudinal extension of the study would be valuable. In 2001, SAMHSA and Casey Family Programs embarked upon an extension phase, which will increase understanding of the impact of early intervention as young children enter preschool and school years, when babies or toddlers are asked to meet escalating emotional and cognitive demands. This longitudinal extension can validate early methods and findings and assess their durability. It is anticipated that this work will include additional data points of a refined instrument set and intervention package with the addition of study questions related to cost and value, and other special studies. Additional future plans include applying and validating early SESS lessons learned, key concepts, components, and principles to new settings that serve families with young children.

Summation
In sum, SESS reflects the growing acknowledgement that it is important to target positive interventions to very young children. The infant and preschool years lay a critical foundation for later growth and development. Second, successful interventions for very young children must meet the multiple behavioral health, physical health, and educational needs of families. Third, integrated behavioral health services must be made more accessible to families with multiple needs, which are difficult to meet in a fragmented service system.

The SESS Sites
Miami’s Families: Starting Early Starting Smart (Florida)
Raising Infants in Secure Environments (Massachusetts)
Healthy Foundations for Families (Missouri)
Starting Early to Link Enhanced Comprehensive Treatment Teams (New Mexico)
Casey Family Partners (Washington)
National Association for Families and Addiction Research and Education (Illinois)*
Child Development, Inc. (Arkansas)
Asian American Recovery Services, Inc. (California)
Locally Integrated Services in Head Start (Washington, D.C.)
Starting Early Starting Smart Head Start Collaboration Project (Illinois)
Baltimore BETTER Family and Community Partnership (Maryland)
New Wish (Nevada)
Beda?chelh Tulalip Tribes Early Intervention in Tribal and Mainstream Communities (Washington)
Evaluation, Management and Training, Inc.** (California)

*One of the original SESS sites was unable to continue with the study, but it was an important contributor to the original design and implementation of this project. Our thanks to Dr. Linda Randolph and Dr. Ira Chasnoff.

**Data Coordinating Center

For more information about Starting Early Starting Smart
and related SAMHSA-Casey products, contact
http://www.casey.org/ or http://www.csap.gov/ or http://ncadi.samhsa.gov/.

Published 2001 by Casey Family Programs
1300 Dexter Avenue North, Suite 300
Seattle, WA 98109
Telephone: (206) 282-7300; Fax (206) 378-4619;
Internet: www.Casey.org

RAND
1700 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138
1200 South Hayes Street, Arlington, VA 22202-5050
201 North Craig Street, Suite 102, Pittsburgh, PA 15213
Internet: www.rand.org/

To order from RAND, contact Distribution Services:
Telephone: (310) 451-7002; Fax: (310) 451-6915;
E-mail: order@rand.org

RAND is a nonprofit institution that helps improve policy and decisionmaking through research and analysis.

RAND® is a registered trademark.

Any or all portions of this document may be reproduced with proper citation: “Source: Karoly, L., Kilburn, R., Bigelow, J. H., Caulkins, J. P., and Cannon, J. S. (2001). Assessing Costs and Benefits of Early Childhood Intervention Programs: Overview and Applications to the Starting Early Starting Smart Program. Publishers: Seattle: Casey Family Programs; Santa Monica: RAND.”

This report would not have been possible without the contributions of staff from the Office on Early Childhood, SAMHSA, U.S. Department of Health and Human Services, the Casey Family Programs, the Starting Early Starting Smart principal investigators, project directors and researchers, and the parent representatives, who helped design and supervise the data collection. The content of this publication does not necessarily reflect the views or policies of the U.S. Department of Health and Human Services, or the Casey Family Programs, nor does mention of trade names, commercial products, or organizations imply endorsement by the U.S. government. Responsibility for the content of this report, however, rests solely with the named authors.