Appendix 3: Limitations of the Data

I. Identification of the target universe of specialty substance abuse treatment providers

The States identify most providers in the NFR, and differences among States affect the uniformity and completeness of the national listing. Differences include:

1) Funding, licensing, and other regulatory practices. For example, a few States license all specialty providers, including those that receive only private funds, but most do not.

2) "Level" of an NDATUS reporting unit. Reporting units may be agencies or programs that manage multiple facilities at different locations, establishments (sometimes called clinics) where treatment is delivered, or multiple treatment units operating at the same location. Consequently, comparisons in terms of the number of reporting units over time or among States are difficult to interpret.

II. Survey Non-Response

In 1992, 12,331 forms were mailed to previously identified treatment providers. An additional 357 treatment providers reported to NDATUS; 111 had been listed in the NFR as non-treatment providers and 246 had never been listed. One thousand and twenty (1,020) forms were returned with information that the provider had gone out of the treatment business. With these survey results, the NFR universe of active treatment providers was adjusted downward to 11,668. Active providers completed 9,307 NDATUS forms in 1992, and they reported 787,152 one-day census clients.

In order to make unbiased national estimates, a probability sample of 401 non-respondents was surveyed that represented all 2,361 non-respondents. When information about provider size was available from previous NDATUS surveys, the probability of being selected for the sample was made proportional to size. NDATUS forms were completed by telephone. Two hundred and sixty-one (65%) completed at least part of the form; 87 (22%) were no longer in business.

Key characteristics of all non-responding providers and their clients were estimated by assigning analysis weights, to each sample responder, equal to the inverse of the probability of each being selected for the sample. Based upon these results, non-respondents had an estimated one-day census of 157,728 clients, and active providers that did not respond to NDATUS were estimated at 2,009. With these non-response results, the NDATUS response rate was adjusted to 82% and responders accounted for 83% of total NDATUS clients. Furthermore, a broad comparison between responder and non-responder data revealed no major differences in utilization patterns.

The non-response sample was selected to develop estimates at the national level. To estimate numbers of non-respondents and clients at the State and jurisdictional level [see Appendix 7, Tables 2 and 3], certain assumptions were made and the data imputed. These imputations were controlled so that sums over all jurisdictions equalled the weighted national estimates of various parameters.

The crude estimate of the number of non-responding providers (2,364) was computed by simply subtracting the number of respondents from the number of providers who were mailed the NDATUS survey. However, respondents included both those providers continuing to offer treatment and those that indicated they were no longer providing treatment. Thus the crude estimate was adjusted downward, under the assumption that a proportion of the non-respondents similar to that among the respondents was also no longer providing treatment. This number was calculated for each State and jurisdiction, and summed over all to 2,120 providers. This number was slightly more than the 2,009 non-respondents derived from the weighted national estimate. To rectify this discrepancy, the State and jurisdictional estimates were adjusted down to total 2,009, while maintaining the same proportion of total for each jurisdiction.

The number of clients served by non-respondents in each State and jurisdiction was then imputed using a series of four estimates for each jurisdiction. These estimates used the number of non-responding but active providers from the calculation above, as well as data derived from responding providers. The latter included the average number of clients per provider; the average number of clients per provider according to funding source (any public funding vs. private funding only vs. funding source unknown); and ratios of clients according to substance abuse problem.

The first estimate of the number of clients by State and jurisdiction was computed using the average number of clients per provider. The sum of these clients over all jurisdictions was 165,760, compared to 157,728 from the weighted national estimate.

The second estimate allocated the non-respondents into three groups according to funding source (any public funding, private funding only, and unknown source). The total numbers for each of these groups was derived from the weighted national estimates, and the providers allocated according to the jurisdiction's proportion of all non-respondents. The total number of clients was computed using the average number of clients per provider for each funding source, and summing over all funding sources. The sum of these clients over all jurisdictions was 126,429.

The first and second estimates fell on either side of the weighted national estimate. Thus the third estimate was computed by adjusting the second estimate upward for each jurisdiction. The adjustment factor was equal to the difference between the high and low estimates for each jurisdiction times the ratio of the difference between the national estimate and the low estimate, and the difference between the high and low estimates. Using this procedure, the sum of clients across all States and jurisdictions equalled the weighted national estimate.

The final estimate distributed the number of clients in non-responding providers within each jurisdiction according to substance abuse problem (alcohol only, drug only, both). This distribution was based on proportions of these problems according to funding source from the national non-response survey, and summed over funding source. In the final estimate, the sums of clients by State and jurisdiction are equivalent to the weighted national estimates from the non-response survey at the level of funding source and substance abuse problem, as well as aggregates of these.

Prior to 1992, reported survey response rates were based upon the number of potential providers identified at the beginning of the survey and responders included providers that actively refused to submit data. Furthermore, responders and non-responders were not compared based upon a representative sample of non-responders. However, actual response rates and experience with survey administration over the years suggest that response patterns were similar over the period 1980-1992. This conclusion is reinforced by the apparent stability of key ratio indicators that were reported over this period [see forthcoming Main Findings].

III. Item non-response

Item response rates for basic provider characteristics such as ownership, type of facility, and type of treatment exceeded 98% for treatment providers. However, 356 responders (4%) in the main survey and 120 responders (46%) in the non-response sample reported only treatment capacity, not clients. In both the main survey and the non-response sample, missing clients were imputed, by treatment type, as a ratio of reported capacity. These ratios were based upon ratios reported by similar providers, where similarity was defined by source of funds.

IV. One-day census (point-prevalence) estimators for annual provider and client characteristics

One-day census or point prevalence data describe treatment programs, services, and clients on a given day. No significance should be attributed to the particular day except that it is may be characteristic of daily operations during the year. Plausibility is reinforced by the monthly pattern of treatment admissions, reported to CDS for 1992. These admissions occur in a relatively stable pattern throughout the year.

V. Data quality assurance and validation

Provider reporting is facilitated by State agency staff, with training and assistance from a SAMHSA contractor. Experience with survey administration indicates considerable variation among State agencies in funding, staff resources, and policy priorities related to survey implementation. State policies also appear to affect the quality of clinical records maintained by providers. In most States, NDATUS data are collected outside of normal program administration, and thus provider reported data are generally not subject to routine verification and review by program analysts.

VI. Changes in the NDATUS instrument over time

Comparisons are made in this report among NDATUS results from 1980, 1982, 1987, 1989, 1990, 1991, and 1992 because the instruments for these years asked about both drug and alcohol clients and because they asked the same questions about client demographics. Only one comparison was made to the 1984 NDATUS because that survey was reduced to a one-page form.

Comparisons by type of client problem could not be made prior to 1991. Before 1990, clients' substance of abuse was identified only in terms of whether their treatment was funded using alcohol or drug funds (1980 and 1982), or whether they had either a drug abuse problem or an alcohol problem but not both (1987 and 1989). In other words, none of these earlier surveys could have identified clients that abused both alcohol and drugs, and the earlier instrument would have also misclassified drug clients that happened to be treated in facilities funded with alcohol funds, and vice versa. The 1990 survey was the first NDATUS to attempt to identify all three client groups, but a misunderstanding with the States resulted in most clients being classified as having both alcohol and drug problems.

VII. Definition of terms

Outpatient Clients. The practical definition of outpatient clients may be problematic. It is defined as clients who have been admitted but not discharged from treatment, and that have been seen for a scheduled appointment during the past month. In practice, it may be difficult for providers to identify all such clients without an unreasonable effort.

Treatment Capacity. Treatment capacity is difficult to define for outpatient services and for specialty hospital units. Outpatient services have ambiguous limits due to the flexibility of staff caseloads and the general purpose nature of counseling rooms. Inpatient hospital services also have flexible limits to the extent that beds can be reallocated on short notice among patients with different diagnoses.

Detoxification. Unlike rehabilitation, detoxification does not attempt to change behavior. Rather, its purpose is to stabilize physiological systems, often but not always under medical supervision. While detoxification is often the first step toward entering rehabilitation, including both detox and rehab clients in estimates of total clients in "treatment" may be misleading.

Return to default