2013-2014
National Survey on Drug Use and Health:
Other Sources of State-Level Data

Introduction

A variety of surveys and data systems other than the National Survey on Drug Use and Health (NSDUH) collect data on substance use problems and mental disorders. It is useful to consider the results of these other studies when discussing NSDUH data. This document briefly describes one of these other data systems that publish state estimates and presents selected comparisons with NSDUH results. The state-level survey that collects data on substance use discussed in this document is the Behavioral Risk Factor Surveillance System (BRFSS), sponsored by the Centers for Disease Control and Prevention (CDC). Another CDC data system that provides state-level substance use estimates for most but not all states is the Youth Risk Behavior Survey (YRBS). Differences between the YRBS and NSDUH sampling designs, as well as the wider range of age groups used in NSDUH small area estimates, imply that comparisons of estimates are not straightforward. However, ignoring these differences and examining estimates at a national level, the YRBS has been generally shown to have higher estimates but similar long-term trends compared with NSDUH (Center for Behavioral Health Statistics and Quality, 2014, 2015).¹

When considering the information presented in this document, it is important to understand the methodological differences between these surveys and the impact that these differences could have on estimates of substance use and mental health. Several studies have compared NSDUH estimates with estimates from other studies and have evaluated how differences may have been affected by differences in survey methodology (Brener et al., 2006; Gfroerer, Wright, & Kopstein, 1997; Grucza, Abbacchi, Przybeck, & Gfroerer, 2007; Hennessy & Ginsberg, 2001; Miller et al., 2004). These studies suggest that the goals and approaches of surveys are often different, making comparisons between them difficult. Some methodological differences that have been identified as affecting comparisons include populations covered, sampling methods, mode of data collection, survey setting, questionnaires, and estimation methods.

BRFSS is a state-based system of health surveys that collect information on health risk behaviors (including cigarette and alcohol use), preventive health practices, and health care access primarily related to chronic disease and injury. BRFSS is an annual, state-based telephone (landline and cellular telephone) survey of the civilian, noninstitutionalized adult population aged 18 or older and is sponsored by the CDC. In 2014, BRFSS collected data from all 50 states, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, and Guam using a computer-assisted telephone interviewing design. About 500,000 adults are interviewed each year, and state estimates are presented annually.

In 2011, BRFSS introduced two methodological changes: (1) the inclusion of cellular telephone-only households in the sample, and (2) the incorporation of iterative proportional fitting (also referred to as "raking") in the production of the final BRFSS weights, replacing the use of poststratification. Cellular telephone-only households were added to improve survey coverage of the telephone population and addressed differences in characteristics found between the cellular telephone-only and landline populations. In 2014, BRFSS respondents who had a cellular telephone were eligible for participation in the cellular telephone survey. In 2013, on the other hand, in order to be eligible to participate in the cellular telephone survey, respondents had to be in either a cellular telephone-only household or a household where 90 percent or more of their calls were received on cellular telephones. Because state-level demographic characteristics of cellular telephone-only households are not available, weighting with the previous method of poststratification was no longer feasible. As a result of these methodological changes, the CDC reported small increases in various health risk indicators, including tobacco use and binge drinking.² The 2014 BRFSS state estimates and confidence intervals are weighted design-based estimates (i.e., each respondent is weighted in a way that accounts for the survey design).³

Also in 2011, the BRFSS questionnaire underwent some changes in the alcohol consumption and tobacco use sections. In 2010, BRFSS respondents were asked, "During the past 30 days, have you had at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?" The response to this question was used to route respondents to the next question regarding the frequency of alcohol use in the past 30 days. However, only the responses to the first question were used to determine past month alcohol use. In the 2011 BRFSS questionnaire, this question was dropped, and respondents were directly asked, "During the past 30 days, how many days per week or per month did you have at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?" If a respondent answered "1" or higher to this question, he or she was considered a past month user of alcohol. In spite of the questionnaire changes, BRFSS is still producing an estimate of past month alcohol use that can be compared with the NSDUH estimate. Also, minor wording changes were made in one question in the tobacco use section, but none of these would affect current cigarette use estimates. These newly worded questions were used in the 2012, 2013, and 2014 BRFSS surveys as well.

In both BRFSS and NSDUH, data are collected on the following four substance use and mental health measures in each of the 50 states and the District of Columbia:⁴

• past month alcohol use,
• cigarette use ("past month" use for NSDUH and "current" use for BRFSS),
• past month binge alcohol use, and
• lifetime doctor-diagnosed depression.⁵

Note that only estimates for the first two of these four measures are compared here because the binge alcohol definition differs in BRFSS and NSDUH and small area estimates of lifetime doctor-diagnosed depression were not produced for NSDUH. The BRFSS and NSDUH questions that were used for the first two measures are shown in the next section.

Past month alcohol use is defined consistently in both BRFSS and NSDUH as having an alcoholic beverage in the past month. In NSDUH, past month cigarette use is defined as having smoked part or all of a cigarette during the past 30 days. In BRFSS, the cigarette use measure reported is current cigarette use, which is defined as having smoked at least 100 cigarettes during the lifetime and indicating smoking every day or some days at the time of the survey. Because of these subtle but present differences in definitions, the NSDUH cigarette use estimates tend to be higher in that they cover two groups of people that the BRFSS estimates would not: (1) respondents who have not smoked 100 cigarettes in their lifetime but had smoked in the past month, and (2) respondents who had smoked a cigarette earlier in the month but were not smoking at the time of the survey. Both surveys ask about binge alcohol use in the past month; however, as noted earlier, each survey defines it slightly differently. NSDUH's binge alcohol use definition for males and females is having had five or more drinks of an alcoholic beverage on the same occasion (i.e., at the same time or within a couple hours of each other) on at least 1 day in the past 30 days. In BRFSS, females are asked about drinking four or more drinks on one occasion, whereas males are asked about drinking five or more drinks on one occasion. Because of this difference in definitions, NSDUH's small area estimates and BRFSS's estimates of binge alcohol use are not discussed here.

Beginning in 2011, the question assessing lifetime diagnosed depression was removed from the BRFSS optional anxiety and depression module and placed in the core section of the questionnaire within a group of questions inquiring about various chronic health conditions, such as coronary heart disease and diabetes. Thus, BRFSS estimates for lifetime diagnosed depression are now available for all states. In BRFSS, respondents are simply asked if a doctor, nurse, or other health professional has ever told them that they had a depressive disorder, including depression, major depression, dysthymia, or minor depression. In NSDUH, respondents are considered to have had depression in their lifetime if they answered that a doctor or medical professional has ever told them that they had depression. In the same group of questions asking about depression in both the BRFSS questionnaire and the NSDUH questionnaire, respondents are also asked about heart disease, diabetes, strokes, and asthma. However, because NSDUH's state-level small area estimates are not produced for lifetime diagnosed depression or any of these other health conditions, comparisons with BRFSS data cannot be made. Although state small area estimates have not been produced in NSDUH for these measures, direct estimates of these health measures could be generated using NSDUH data at the state level and compared with BRFSS estimates. Because the focus here is on small area estimates, however, such comparisons with BRFSS data were not made. Note that NSDUH's state small area estimates are produced for having had a major depressive episode (MDE) in the past year. However, this MDE measure is unrelated to the NSDUH question about being diagnosed with lifetime depression. Instead, NSDUH includes a separate set of questions to assess depression symptoms that are used to measure MDE. Thus, NSDUH's small area estimates for MDE would not be comparable with estimates of the BRFSS depression measure discussed here.

This document presents the findings of the 2014 BRFSS state estimates and the combined 2013-2014 NSDUH state estimates for past month alcohol use and cigarette use ("past month" use for NSDUH and "current" use for BRFSS). In Tables 1 and 2 (shown after this text discussion), the 2014 BRFSS state estimates for adults aged 18 or older are shown alongside the pooled 2013-2014 NSDUH small area estimates for the same age group (by combining the 18 to 25 and 26 or older age groups). Table 1 also includes p values that indicate whether the BRFSS and NSDUH estimates are significantly different from each other for a given state using an exact test as described in the next section. Because the definitions for binge alcohol use in the two surveys are different for females, no comparison of binge alcohol use was done.

NSDUH and BRFSS Questions

The 2014 NSDUH questions that were used to determine past month alcohol use and when alcohol was last used were worded as follows:⁶

AL01

Have you ever, even once, had a drink of any type of alcoholic beverage? Please do not include time when you only had a sip or two from a drink.
1   Yes
2   No
DK/REF

ALLAST3

[IF AL01=1 or ALREF=1] How long has it been since you last drank an alcoholic beverage?
1   Within the past 30 days – that is, since [DATEFILL]
2   More than 30 days ago but within the past 12 months
3   More than 12 months ago
DK/REF

The 2014 BRFSS question that was used to determine past month alcohol use was worded as follows:⁷

10.1 During the past 30 days, how many days per week or per month did you have at least one drink of any alcoholic beverage such as beer, wine, a malt beverage, or liquor?
1 _ _   Days per week
2 _ _   Days in the past 30 days
8 8 8   No drinks in past 30 days
7 7 7   Don't know/Not sure
9 9 9   Refused

The 2014 NSDUH questions that were used to determine past month cigarette use were worded as follows:

CG01

Have you ever smoked part or all of a cigarette?
1   Yes
2   No
DK/REF

CG05

[IF CG01 = 1 OR CGREF1 = 1] Now think about the past 30 days – that is, from [DATEFILL] up to and including today. During the past 30 days, have you smoked part or all of a cigarette?
1   Yes
2   No
DK/REF

The 2014 BRFSS questions that were used to determine current cigarette use were worded as follows:

9.1  Have you smoked at least 100 cigarettes in your entire life?
NOTE: 5 packs = 100 cigarettes
1   Yes
2   No
7   Don't know / Not sure
9   Refused

INTERVIEWER NOTE: "For cigarettes, do not include: electronic cigarettes (e-cigarettes, NJOY, Bluetip), herbal cigarettes, cigars, cigarillos, little cigars, pipes, bidis, kreteks, water pipes (hookahs), or marijuana."

9.2  Do you now smoke cigarettes every day, some days, or not at all?
1   Every day
2   Some days
3   Not at all
7   Don't know / Not sure
9   Refused

Note that these 2014 NSDUH and 2014 BRFSS questions were the same as the questions used in the 2013 NSDUH and the 2013 BRFSS.

Methodology for Comparing BRFSS and NSDUH Estimates

The methodology used to compare BRFSS and NSDUH estimates is similar to what is described in Section B.6 of the "2013-2014 NSDUH: Guide to State Tables and Summary of Small Area Estimation Methodology."⁸ Here, the null hypothesis of no difference is tested, that is, (where is the expected value⁹ of the BRFSS estimate and is the expected value of the NSDUH estimate) or equivalently that the logs-odds ratio is zero, that is, , where is defined as , where ln denotes the natural logarithm. An estimate of is given by , where and are the 2014 BRFSS state-level design-based estimates and the 2013-2014 NSDUH state model-based estimates, respectively (as given in Tables 1 and 2). To compute the variance of , that is, , let and , then . The covariance term can be assumed to be zero because the BRFSS and NSDUH samples are independent.

The quantity can be obtained by using the 95 percent Bayesian confidence intervals in Tables 1 and 2. For this purpose, let denote the 95 percent Bayesian confidence interval¹⁰ for a given state-s:

,     D

where  .

The quantity can be obtained by using the 95 percent confidence intervals in Tables 1 and 2. For this purpose, let denote the 95 percent BRFSS confidence interval for a given state-s, then is given by

.     D

Now, using the first-order Taylor series approximation,¹¹ can be calculated from as follows: .

The p value that is given in Tables 1 and 2 for testing the null hypothesis of no difference () is provided by , where is a standard normal random variate, , and denotes the absolute value of .

Alcohol Use

As can be seen in Table 1, for past month alcohol use, the NSDUH estimates and the BRFSS estimates for more than half of the states were different (i.e., at the 5 percent level of significance, 37 of 51 states had different estimates). However, these two sets of estimates were highly correlated (correlation coefficient = 0.95). Figures 1 and 2, which follow the tables, were created by using state estimates from both BRFSS and NSDUH and categorizing the states into five quintiles similar to the process described on the title page of the "2013-2014 NSDUH National Maps of Prevalence Estimates, by State."¹² Note that in Tables 1 and 2, the BRFSS estimates and corresponding confidence intervals are rounded to one decimal place, whereas the NSDUH small area estimates and Bayesian confidence intervals are rounded to two decimal places. Therefore, all of the tables and maps included in this document use that approximation.

As can be seen in Figures 1 and 2, nine states with the highest estimates of alcohol use (states shown in red) were the same in the two surveys: Colorado, Connecticut, the District of Columbia, Massachusetts, Minnesota, New Hampshire, North Dakota, Vermont, and Wisconsin. Note that Rhode Island was the other state in the top BRFSS group and that Maine was the other state in the top NSDUH group. Eight states with the lowest estimates of alcohol use were the same in the two surveys: Alabama, Arkansas, Kentucky, Mississippi, North Carolina, Tennessee, Utah, and West Virginia. Note that Missouri and Oklahoma rounded out the bottom BRFSS group and that Idaho and Texas were the other states in the bottom NSDUH group. The lowest estimate of past month alcohol use was in Utah for both BRFSS and NSDUH (see Table 1 and Figures 1 and 2).

Cigarette Use

As can be seen in Table 2, the NSDUH estimates of past month cigarette use were always larger than the BRFSS estimates of current cigarette use. Some of this difference is the result of the differences in definitions as discussed earlier in this document; thus, exact tests to examine significant differences between the NSDUH and BRFSS cigarette use estimates are not included. Although the NSDUH estimates tended to be larger, these two sets of estimates were highly correlated (correlation coefficient = 0.89).

Figures 3 and 4 were created using the same method used to produce Figures 1 and 2. As can be seen in Figures 3 and 4, six states with the highest estimates of cigarette use (states shown in red) were the same in the two surveys: Arkansas, Indiana, Kentucky, Louisiana, Mississippi, and West Virginia. Rounding out the top BRFSS group were Michigan, South Carolina, and Tennessee, while Alabama, Missouri, Ohio, and Oklahoma rounded out the top NSDUH group. Seven states with the lowest estimates of cigarette use were the same in the two surveys: California, Connecticut, Hawaii, Maryland, Massachusetts, New Jersey, and Utah. Note that the other three states in the bottom BRFSS group were New York, Texas, and Washington and that the other three states in the bottom NSDUH group were Arizona, New Hampshire, and Oregon.

Sample Size Comparisons

The BRFSS estimates are design based, while the NSDUH estimates are model based. Also, the NSDUH estimates are based on the pooled 2013-2014 NSDUH (2 years of data), whereas the BRFSS estimates are based on the 2014 BRFSS survey (1 year of data). Although the BRFSS estimates are based on only 1 year of data, the BRFSS sample sizes for a given state were in general much larger than the sample sizes for NSDUH over 2 years. In the 2013-2014 NSDUH, the 18 or older sample sizes in the states ranged from 1,258 to 6,015 respondents, with a median sample size of 1,337.¹³ For the 2014 BRFSS, all of the states had larger sample sizes as compared with their counterparts in NSDUH. Overall, the BRFSS sample sizes for the states varied from a low of 3,763 to a high of 22,420 respondents, with a median sample size of 7,950.¹⁴ Sample size differences of this magnitude explain why the NSDUH Bayesian confidence intervals were generally wider than the corresponding BRFSS design-based confidence intervals.

References

American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (DSM-IV) (4th ed.). Washington, DC: Author.

Brener, N. D., Eaton, D. K., Kann, L., Grunbaum, J. A., Gross, L. A., Kyle, T. M., & Ross, J. G. (2006). The association of survey setting and mode with self-reported health risk behaviors among high school students. Public Opinion Quarterly, 70, 354-374. doi:10.1093/poq/nfl003

Center for Behavioral Health Statistics and Quality. (2014). Results from the 2013 National Survey on Drug Use and Health: Summary of national findings (HHS Publication No. SMA 14-4863, NSDUH Series H-48). Rockville, MD: Substance Abuse and Mental Health Services Administration.

Center for Behavioral Health Statistics and Quality. (2015). Section D: Other source of data. In 2014 National Survey on Drug Use and Health: Methodological summary and definitions. Retrieved from https://www.samhsa.gov/data/

Centers for Disease Control and Prevention. (2015, September). Behavioral Risk Factor Surveillance System: Comparability of data BRFSS 2014 (Version #1‒Revised). Retrieved from http://www.cdc.gov/brfss/annual_data/2014/pdf/compare_2014.pdf

Gfroerer, J., Wright, D., & Kopstein, A. (1997). Prevalence of youth substance use: The impact of methodological differences between two national surveys. Drug and Alcohol Dependence, 47, 19-30. doi:10.1016/S0376-8716(97)00063-X

Grucza, R. A., Abbacchi, A. M., Przybeck, T. R., & Gfroerer, J. C. (2007). Discrepancies in estimates of prevalence and correlates of substance use and disorders between two national surveys. Addiction, 102, 623-629. doi:10.1111/j.1360-0443.2007.01745.x

Hennessy, K. H., & Ginsberg, C. (Eds.). (2001). Substance use survey data collection methodologies and selected papers [Special issue]. Journal of Drug Issues, 31(3), 595-808.

Miller, J. W., Gfroerer, J. C., Brewer, R. D., Naimi, T. S., Mokdad, A., & Giles, W. H. (2004). Prevalence of adult binge drinking: A comparison of two national surveys. American Journal of Preventive Medicine, 27, 197-204. doi:10.1016/s0749-3797(04)00121-7

Figure 1. Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2014 BRFSS

     D

Source: Centers for Disease Control and Prevention (CDC), Behavioral Risk Factor Surveillance System Survey, 2014.

Figure 2. Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2013 and 2014 NSDUHs

     D

Source: SAMHSA, Center for Behavioral Health Statistics and Quality, NSDUH, 2013 and 2014.

Figure 3. Current Cigarette Use among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2014 BRFSS

     D

Source: Centers for Disease Control and Prevention (CDC), Behavioral Risk Factor Surveillance System Survey, 2014.

Figure 4. Cigarette Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2013 and 2014 NSDUHs

     D

Source: SAMHSA, Center for Behavioral Health Statistics and Quality, NSDUH, 2013 and 2014.