Introduction
A variety of surveys and data systems other than the National Survey on Drug Use and Health (NSDUH) collect data on substance use and mental health problems. It is useful to consider the results of these other studies when discussing NSDUH data. This document briefly describes one of these other data systems that publish State estimates and presents selected comparisons with NSDUH results. The State-level survey that collects data on substance use discussed in this document is the Behavioral Risk Factor Surveillance System (BRFSS), sponsored by the Centers for Disease Control and Prevention (CDC). Another CDC data system that provides State-level substance use estimates for most but not all States is the Youth Risk Behavior Survey (YRBS). Differences between the YRBS and NSDUH sampling designs, and age groups used in NSDUH small area estimates, imply that comparisons of prevalence rates are not straightforward. However, ignoring these differences and examining estimates at a national level, the YRBS has been generally shown to have higher prevalence rates but similar long-term trends compared with NSDUH (Center for Behavioral Health Statistics and Quality [CBHSQ], 2013).1
When considering the information presented in this document, it is important to understand the methodological differences between these surveys and the impact that these differences could have on estimates of substance use. Several studies have compared NSDUH estimates with estimates from other studies and have evaluated how differences may have been affected by differences in survey methodology (Brener et al., 2006; Gfroerer, Wright, & Kopstein, 1997; Grucza, Abbacchi, Przybeck, & Gfroerer, 2007; Hennessy & Ginsberg, 2001; Miller et al., 2004). These studies suggest that the goals and approaches of surveys are often different, making comparisons between them difficult. Some methodological differences that have been identified as affecting comparisons include populations covered, sampling methods, mode of data collection, survey setting, questionnaires, and estimation methods.
BRFSS is a State-based system of health surveys that collect information on health risk behaviors (including cigarette and alcohol use), preventive health practices, and health care access primarily related to chronic disease and injury. BRFSS is an annual, State-based telephone (landline and cellular telephone) survey of the civilian, noninstitutionalized adult population aged 18 or older and is sponsored by the CDC. In 2012, BRFSS collected data from all 50 States, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, and Guam using a computer-assisted telephone interviewing (CATI) design. More than 450,000 adults are interviewed each year. State estimates are presented on a yearly basis.
In 2011, the BRFSS introduced two methodological changes: (1) the inclusion of cellular telephone-only households in the sample, and (2) the incorporation of iterative proportional fitting (also referred to as "raking") in the production of the final BRFSS weights, replacing the use of poststratification. Cellular telephone-only households were added to improve survey coverage of the telephone population and addressed differences in characteristics found between cellular telephone-only and landline populations. In 2012, in addition to cellular telephone-only households, households where 90 percent or more of their calls were received on cellular telephones also were eligible to participate. Raking incorporates the desired population variables one at a time in an iterative process rather than making adjustments in a single process. As a result, more variables can be used in raking versus poststratification. In addition to the standard demographic variables of age, gender, race, and ethnicity within each State, BRFSS included education level, marital status, renter or owner status, and source of telephone (cellular telephone only, landline only, cellular and landline). As a result of these methodological changes, the CDC has reported small increases in various health risk indicators, including tobacco use and binge drinking. The 2012 BRFSS State prevalence rates and confidence intervals are weighted design-based estimates (i.e., each respondent is weighted in a way that accounts for the survey design).2
Also in 2011, the BRFSS questionnaire underwent some changes in the alcohol consumption and tobacco use sections. In 2010, BRFSS respondents were asked, "During the past 30 days, have you had at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?" The response to this question was used to route respondents to the next question regarding the frequency of alcohol use in the past 30 days. However, only the responses to the first question were used to determine past month alcohol use. In the 2011 BRFSS questionnaire, this question was dropped, and respondents were directly asked, "During the past 30 days, how many days per week or per month did you have at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?" If a respondent answered "1" or higher to this question, he or she was considered a past month user of alcohol. In spite of the questionnaire changes, BRFSS is still producing an estimate of past month alcohol use that can be compared with the NSDUH estimate. There were also minor wording changes in one question, in the tobacco use section, but none of these would affect current cigarette use rates. These newly worded questions were used in the 2012 BRFSS as well.
There are three substance use measures and one mental health measure for which estimates are produced for both BRFSS and NSDUH in each of the 50 States and the District of Columbia:3
- past month alcohol use,
- cigarette use ("past month" use for NSDUH and "current" use for BRFSS), and
- past month binge alcohol use.
Past month alcohol use is defined consistently in both BRFSS and NSDUH as having an alcoholic beverage in the past month. In NSDUH, past month cigarette use is defined as having smoked part or all of a cigarette during the past 30 days. In BRFSS, the cigarette use measure reported is current cigarette use, which is defined as having smoked at least 100 cigarettes during the lifetime and indicating smoking every day or some days at the time of the survey. Because of these subtle but present differences in definitions, the NSDUH cigarette use estimates tend to be higher in that they catch two groups of people that the BRFSS estimates would not: (1) respondents who have not smoked 100 cigarettes in their lifetime but had smoked in the past month, and (2) respondents who had smoked a cigarette earlier in the month but were not smoking at the time of the survey. Both surveys ask about binge alcohol use in the past month. The definition for binge alcohol use in NSDUH is having had five or more drinks of an alcoholic beverage on the same occasion (i.e., at the same time or within a couple hours of each other) on at least 1 day in the past 30 days. In BRFSS, women are asked about drinking four or more drinks on one occasion, whereas men are asked about drinking five or more drinks on one occasion.
Beginning in 2011, the question assessing diagnosed depression was removed from the BRFSS optional anxiety and depression module and placed in the core section of the questionnaire within a group of questions inquiring about various chronic health conditions, such as coronary heart disease and diabetes. Thus, BRFSS estimates for depression are now available for all States. In BRFSS, respondents are simply asked if a doctor, nurse, or other health professional has ever told them that they had a depressive disorder, including depression, major depression, dysthymia, or minor depression. In NSDUH, respondents are considered to have had depression in their lifetime if they answered that a doctor or medical professional has ever told them that they had depression. In the same group of questions asking about depression in both the BRFSS and NSDUH questionnaire, respondents are also asked about heart disease, diabetes, strokes, and asthma. However, because NSDUH State-level small area estimates are not produced for lifetime diagnosed depression or any of these other health conditions, comparisons with BRFSS data cannot be made.
This document presents the findings of the 2012 BRFSS State estimates and the combined 2011-2012 NSDUH State estimates for past month alcohol use and cigarette use ("past month" use for NSDUH and "current" use for BRFSS). In Tables 1 and 2 (shown after this text discussion), the 2012 BRFSS State estimates for adults aged 18 or older are shown alongside the pooled 2011-2012 NSDUH small area estimates for the same age group (by combining the 18 to 25 and 26 or older age groups). Table 1 also includes p values that indicate whether the BRFSS and NSDUH estimates are significantly different from each other for a given State using an exact test as described in the next section. Because the definitions for binge alcohol use in the two surveys are different for women, no comparison of binge alcohol use was done.
Methodology for Comparing BRFSS and NSDUH Estimates
The methodology used to compare BRFSS and NSDUH estimates is similar to what is described in Section B.12 of the "2011-2012 NSDUH: Guide to State Tables and Summary of Small Area Estimation Methodology."4 Here, the null hypothesis of no difference is tested, that is,
(where
is a State-specific BRFSS direct prevalence rate and
is a State-specific NSDUH model-based prevalence rate) or equivalently that the logs-odds ratio is zero, that is,
where
is defined as
, where ln denotes the natural logarithm. An estimate of
is given by
where
and
are the 2012 BRFSS State estimates and the 2011-2012 NSDUH State estimates, respectively (as given in Tables 1 and 2). To compute the variance of
that is,
let
and
then
.
The covariance term can be assumed to be zero because the BRFSS and NSDUH samples are independent.
The quantity
can be obtained by using the 95 percent Bayesian confidence intervals in Tables 1 and 2. For this purpose, let
denote the 95 percent Bayesian confidence interval for a given State-s:
where
The quantity
can be obtained by using the 95 percent confidence intervals in Tables 1 and 2. For this purpose, let
denote the 95 percent confidence intervals for a given State-s. Using the first-order Taylor series approximation, the variance can be calculated as follows:
The p value that is given in Tables 1 and 2 for testing the null hypothesis of no difference (
) is provided by
where
Z is a standard normal random variate, 
, and 
denotes the absolute value of z.
Alcohol Use
As can be seen in Table 1, for past month alcohol use, the NSDUH and the BRFSS estimates for more than half of the States were different (i.e., at the 5 percent level of significance, 27 of 51 States were different). However, these estimates were highly correlated (correlation coefficient = 0.96). Figures 1 and 2, which follow the tables, were created by using State estimates from both BRFSS and NSDUH and categorizing the States into five quintiles similar to the process described on the title page of the "2011-2012 NSDUH National Maps of Prevalence Estimates, by State."5 Note that in Tables 1 and 2, the BRFSS estimates and corresponding confidence intervals are rounded to one decimal place, whereas the NSDUH small area estimates and Bayesian confidence intervals are rounded to two decimal places. Therefore, all of the tables and maps included in this document use that approximation.
As can be seen in Figures 1 and 2, 7 out of 10 States with the highest rates of alcohol use (States shown in red) were the same in the two surveys: Colorado, Connecticut, District of Columbia, Massachusetts, New Hampshire, Rhode Island, and Wisconsin. The lowest rate of past month alcohol use was in Utah for both the BRFSS survey and NSDUH (see Table 1 and Figures 1 and 2).
Cigarette Use
As can be seen in Table 2, the NSDUH estimates of past month cigarette use were always larger than the BRFSS estimates of current cigarette use. Some of this difference is the result of the differences in definitions as discussed earlier in this document; thus, exact tests to examine significant differences between the NSDUH and BRFSS cigarette use estimates are not included. The 2012 BRFSS estimates for current smokers were reported to be higher than in previous years due to the methodological changes introduced (i.e., the addition of cellular telephone-only households and the raking procedure). Although the NSDUH estimates tended to be larger, these two sets of estimates were highly correlated (correlation coefficient = 0.92).
Figures 3 and 4 were created using the same method used to produce Figures 1 and 2. As can be seen in Figures 3 and 4, 6 States with the highest rates of cigarette use (States shown in red) were the same in the two surveys: Indiana, Kentucky, Louisiana, Mississippi, Tennessee, and West Virginia. The lowest rate of cigarette use for both surveys occurred in Utah (see Table 2).
Sample Size Comparisons
The BRFSS estimates are design based, while the NSDUH estimates are model based. Also, the NSDUH estimates are based on the pooled 2011-2012 NSDUHs (2 years of data), whereas the BRFSS estimates are based on the 2012 BRFSS survey (1 year of data). Although the BRFSS estimates are only based on 1 year of data, the BRFSS sample sizes for a given State are in general much larger than the sample sizes for NSDUH over 2 years. The eight "large" States6 have a sample size for those aged 18 or older of approximately 4,800 respondents each for the 2011-2012 combined NSDUH data. In 2011-2012, the 18 or older sample sizes in these States ranged from 4,462 to 4,967.7 For the 2012 BRFSS, all States had larger sample sizes as compared with the NSDUH data. Overall, the BRFSS sample sizes for the eight large States varied from a low of 5,579 respondents in Illinois to a high of 19,958 respondents in Pennsylvania, with a median sample size of 9,814.8 For the remaining 43 small sample States, NSDUH's 18 or older sample size for the combined 2011-2012 data was approximately 1,200 respondents for each State. The BRFSS sample sizes for the small sample States were much larger, varying from a low of 3,827 respondents in the District of Columbia to a high of 21,723 respondents in Massachusetts, with a median sample size of 7,788. Sample size differences of this magnitude explain why the NSDUH Bayesian confidence intervals were generally wider than the corresponding BRFSS design-based confidence intervals.
References
Brener, N. D., Eaton, D. K., Kann, L., Grunbaum, J. A., Gross, L. A., Kyle, T. M., & Ross, J. G. (2006). The association of survey setting and mode with self-reported health risk behaviors among high school students. Public Opinion Quarterly, 70, 354-374.
Center for Behavioral Health Statistics and Quality. (2013). Results from the 2012 National Survey on Drug Use and Health: Summary of national findings (HHS Publication No. SMA 13‑4795, NSDUH Series H-46). Rockville, MD: Substance Abuse and Mental Health Services Administration.
Gfroerer, J., Wright, D., & Kopstein, A. (1997). Prevalence of youth substance use: The impact of methodological differences between two national surveys. Drug and Alcohol Dependence, 47, 19-30.
Grucza, R. A., Abbacchi, A. M., Przybeck, T. R., & Gfroerer, J. C. (2007). Discrepancies in estimates of prevalence and correlates of substance use and disorders between two national surveys. Addiction, 102, 623-629.
Hennessy, K. H., & Ginsberg, C. (Eds.). (2001). Substance use survey data collection methodologies and selected papers [Special issue]. Journal of Drug Issues, 31(3), 595-808.
Miller, J. W., Gfroerer, J. C., Brewer, R. D., Naimi, T. S., Mokdad, A., & Giles, W. H. (2004). Prevalence of adult binge drinking: A comparison of two national surveys. American Journal of Preventive Medicine, 27, 197-204.
