2016-2017
National Survey on Drug Use and Health:
Other Sources of State-Level Data

Introduction

A variety of surveys and data systems other than the National Survey on Drug Use and Health (NSDUH) collect data on substance use problems and mental disorders. It is useful to consider the results of these other studies when discussing NSDUH data. This document briefly describes one of these other data systems that publish state estimates and presents selected comparisons with NSDUH results. The state-level survey that collects data on substance use discussed in this document is the Behavioral Risk Factor Surveillance System (BRFSS), sponsored by the Centers for Disease Control and Prevention (CDC). Another CDC data system that provides state-level substance use estimates for most but not all states is the Youth Risk Behavior Survey (YRBS). Differences between the YRBS and NSDUH sampling designs, as well as the wider range of age groups used in NSDUH small area estimates, imply that comparisons of estimates are not straightforward. However, ignoring these differences and examining estimates at a national level, the YRBS has been generally shown to have higher estimates than NSDUH has (Center for Behavioral Health Statistics and Quality [CBHSQ], 2016, 2017a, 2018).¹

When considering the information presented in this document, it is important to understand the methodological differences between these surveys and the impact that these differences could have on estimates of substance use and mental health. Several studies have compared NSDUH estimates with estimates from other studies and have evaluated how differences may have been affected by differences in survey methodology (Brener et al., 2006; CBHSQ, 2012; Gfroerer, Wright, & Kopstein, 1997; Grucza, Abbacchi, Przybeck, & Gfroerer, 2007; Hennessy & Ginsberg, 2001; Miller et al., 2004). These studies suggest that the goals and approaches of surveys are often different, making comparisons between them difficult. Some methodological differences that have been identified as affecting comparisons include populations covered, sampling methods, mode of data collection, survey setting, questionnaires, and estimation methods.

BRFSS is a state-based system of health surveys that collect information on health risk behaviors (including cigarette and alcohol use), preventive health practices, and health care access primarily related to chronic disease, injuries, and preventable infectious diseases. BRFSS is an annual, state-based telephone (landline and cellular telephone) survey of the civilian, noninstitutionalized adult population aged 18 or older and is sponsored by the CDC. In 2016 and 2017, BRFSS collected data from all 50 states, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, American Samoa, Palau, and Guam using a computer-assisted telephone interviewing design. More than 400,000 adults are interviewed each year, and state estimates are presented annually.

In 2011, BRFSS introduced two methodological changes: (1) the inclusion of cellular telephone-only households in the sample, and (2) the incorporation of iterative proportional fitting (also referred to as "raking") in the production of the final BRFSS weights, replacing the use of poststratification. Cellular telephone-only households were added to improve survey coverage of the telephone population and addressed differences in characteristics found between the cellular telephone-only and landline populations. Since 2014, BRFSS respondents who had a cellular telephone were eligible for participation in the cellular telephone survey. In 2013, on the other hand, to be eligible to participate in the cellular telephone survey, respondents had to be in either a cellular telephone-only household or a household where 90 percent or more of their calls were received on cellular telephones. Because state-level demographic characteristics of cellular telephone-only households are not available, weighting with the previous method of poststratification was no longer feasible. As a result of these methodological changes in 2014, the CDC reported small increases in various health risk indicators, including tobacco use and binge drinking.² The pooled 2016-2017 BRFSS state estimates and confidence intervals are weighted design-based estimates (i.e., each respondent is weighted in a way that accounts for the survey design).³

Also in 2011, the BRFSS questionnaire underwent some changes in the alcohol consumption and tobacco use sections. In 2010, BRFSS respondents were asked, "During the past 30 days, have you had at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?" The response to this question was used to route respondents to the next question regarding the frequency of alcohol use in the past 30 days. However, only the responses to the first question were used to determine past month alcohol use. In the 2011 BRFSS questionnaire, this question was dropped, and respondents were directly asked, "During the past 30 days, how many days per week or per month did you have at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?" If a respondent answered "1" or higher to this question, he or she was considered a past month user of alcohol. In spite of the questionnaire changes, BRFSS is still producing an estimate of past month alcohol use that can be compared with the NSDUH estimate. Also, minor wording changes were made in one question in the tobacco use section, but none of these would affect current cigarette use estimates. These newly worded questions were used in the 2012 to the 2017 BRFSS surveys as well.

In both BRFSS and NSDUH, data are collected on the following four substance use and mental health measures in each of the 50 states and the District of Columbia:⁴

• past month alcohol use,
• cigarette use ("past month" use for NSDUH and "current" use for BRFSS),
• past month binge alcohol use, and
• lifetime doctor-diagnosed depression.⁵

Note that estimates for only the first three of these four measures are compared in this document because small area estimates of lifetime doctor-diagnosed depression were not produced for NSDUH. The BRFSS and NSDUH questions that were used for the first three measures are shown in the next section.

Past month alcohol use is defined consistently in both BRFSS and NSDUH as having an alcoholic beverage in the past month. Similarly, past month binge alcohol use is defined consistently in the two surveys as drinking five or more drinks (for males) or four or more drinks (for females) on the same occasion (i.e., at the same time or within a couple of hours of each other) on at least 1 day in the past 30 days. In 2014 and prior years, NSDUH's binge alcohol use definition for males and females was having had five or more drinks of an alcoholic beverage on the same occasion (i.e., at the same time or within a couple hours of each other) on at least 1 day in the past 30 days.

In NSDUH, past month cigarette use is defined as having smoked part or all of a cigarette during the past 30 days (i.e., the 30 days prior to the interview). In BRFSS, the cigarette use measure reported is current cigarette use, which is defined as having smoked at least 100 cigarettes during the lifetime and indicating smoking every day or some days at the time of the survey. Because of these subtle but present differences in definitions, NSDUH's cigarette use estimates tend to be higher in that they cover two groups of people that the BRFSS estimates would not: (1) respondents who have not smoked 100 cigarettes in their lifetime but had smoked in the past month, and (2) respondents who had smoked a cigarette earlier in the month but were not smoking at the time of the survey.

Beginning in 2011, the question assessing lifetime doctor-diagnosed depression was removed from the BRFSS optional anxiety and depression module and placed in the core section of the questionnaire within a group of questions inquiring about various chronic health conditions, such as coronary heart disease and diabetes. Thus, BRFSS estimates for lifetime doctor-diagnosed depression are now available for all states. In BRFSS, respondents are simply asked if a doctor, nurse, or other health professional has ever told them that they had a depressive disorder, including depression, major depression, dysthymia, or minor depression. In NSDUH, respondents are considered to have had depression in their lifetime if they answered that a doctor or medical professional has ever told them that they had depression. In the same group of questions asking about depression in both the BRFSS questionnaire and the NSDUH questionnaire, respondents are also asked about heart disease, diabetes, strokes, and asthma. However, because NSDUH's state-level small area estimates are not produced for lifetime doctor-diagnosed depression or any of these other health conditions, comparisons with BRFSS data cannot be made. Although state small area estimates have not been produced in NSDUH for these measures, direct estimates of some of these health measures were generated using NSDUH data at the state level and compared with BRFSS estimates in a methodological study (CBHSQ, 2017b). NSDUH direct estimates of various mental health measures, such as lifetime doctor-diagnosed depression, lifetime doctor-diagnosed anxiety, and past month serious psychological distress, were compared with BRFSS measures. Note that, for most of the measures, BRFSS data came from optional questionnaire modules, meaning that states can choose to use modules according to need, resulting in comparisons with NSDUH that could be done for only a subset of states.

Because the focus here is on model-based small area estimates, however, such comparisons with BRFSS data have not been made. Note that NSDUH's state small area estimates are produced for individuals having had a major depressive episode (MDE) in the past year. However, this MDE measure is unrelated to the NSDUH question about being diagnosed with lifetime depression. Instead, NSDUH includes a separate set of questions to assess depression symptoms that are used to measure MDE. Thus, NSDUH's small area estimates for MDE would not be comparable with estimates of the BRFSS depression measure discussed here. Additionally, BRFSS included an optional module (meaning that states can choose to use it according to need, resulting in a subset of states participating) with a separate set of questions to assess depression based on the Patient Health Questionnaire-8. However, BRFSS data on this measure are not available for all states and are available for only a few years (CBHSQ, 2017b).

This document presents the findings of the combined 2016-2017 BRFSS state estimates and the combined 2016-2017 NSDUH state estimates for past month alcohol use, past month binge alcohol use, and cigarette use ("past month" use for NSDUH and "current" use for BRFSS). In Tables 1, 2, and 3 (shown after this text discussion), the pooled 2016-2017 BRFSS state estimates for adults aged 18 or older are shown alongside the pooled 2016-2017 NSDUH small area estimates for the same age group. Tables 1 and 2 also include p values that indicate whether the BRFSS and NSDUH alcohol use and binge alcohol use estimates are significantly different from each other for a given state using an exact test as described in the next section. Due to definitional differences in the cigarette use measure, no tests of differences between NSDUH and BRFSS estimates were produced.

NSDUH and BRFSS Questions

The 2017 NSDUH questions that were used to determine past month alcohol use and past month binge alcohol use were worded as follows:⁶

AL01

Have you ever, even once, had a drink of any type of alcoholic beverage? Please do not include times when you only had a sip or two from a drink.

1   Yes
2   No
DK/REF⁷

ALLAST3

[IF AL01 = 1 OR ALREF = 1] How long has it been since you last drank an alcoholic beverage?

1   Within the past 30 days - that is, since [DATEFILL]
2   More than 30 days ago but within the past 12 months
3   More than 12 months ago
DK/REF
PROGRAMMER: SHOW 12 MONTH CALENDAR

AL08

[IF ALC30DAY = 1 - 30 OR ALCEST30 = (1 - 6, DK OR REF)] During the past 30 days, that is, since [DATEFILL], on how many days did you have [IF QD01=5 (MALE) THEN FILL 5, IF QD01=9 (FEMALE) THEN FILL 4] or more drinks on the same occasion? By 'occasion,' we mean at the same time or within a couple of hours of each other.

# OF DAYS: _____ [RANGE: 0 - 30]
DK/REF
PROGRAMMER: SHOW 30 DAY CALENDAR

The 2017 BRFSS questions that were used to determine past month alcohol use and past month binge alcohol use were worded as follows:⁸

11.1 During the past 30 days, how many days per week or per month did you have at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?

1 _ _ Days per week
2 _ _ Days in past 30 days
8 8 8 No drinks in past 30 days
7 7 7 Don't know / Not sure
9 9 9 Refused

11.3 Considering all types of alcoholic beverages, how many times during the past 30 days did you have X [CATI X = 5 for men, X = 4 for women] or more drinks on an occasion?

_ _ Number of times
8 8 None
7 7 Don't know / Not sure
9 9 Refused

The 2017 NSDUH questions that were used to determine past month cigarette use were worded as follows:

CG01

Have you ever smoked part or all of a cigarette?

1   Yes
2   No
DK/REF

CG05

[IF CG01 = 1 OR CGREF1 = 1] Now think about the past 30 days, that is, from [DATEFILL] up to and including today. During the past 30 days, have you smoked part or all of a cigarette?

1   Yes
2   No
DK/REF
PROGRAMMER: SHOW 30 DAY CALENDAR

The 2017 BRFSS questions that were used to determine current cigarette use were worded as follows:

9.1 Have you smoked at least 100 cigarettes in your entire life?

NOTE: 5 packs = 100 cigarettes
1   Yes
2   No
7   Don't know / Not sure
9   Refused

INTERVIEWER NOTE: "For cigarettes, do not include: electronic cigarettes (e cigarettes, NJOY, Bluetip), herbal cigarettes, cigars, cigarillos, little cigars, pipes, bidis, kreteks, water pipes (hookahs), or marijuana."

9.2 Do you now smoke cigarettes every day, some days, or not at all?

1   Every day
2   Some days
3   Not at all
7   Don't know / Not sure
9   Refused

Note that these 2017 questions for NSDUH and BRFSS were the same as their 2016 questions.

Methodology for Comparing BRFSS and NSDUH Estimates

The methodology used to compare BRFSS and NSDUH estimates is similar to what is described in Section B.7 of the "2014-2015 NSDUH: Guide to State Tables and Summary of Small Area Estimation Methodology."⁹ Here, the null hypothesis of no difference is tested, that is, (where is the expected value¹⁰ of the BRFSS estimate and is the expected value of the NSDUH estimate) or equivalently that the log-odds ratio is zero, that is, , where is defined as , and where ln denotes the natural logarithm. An estimate of is given by , where and are the 2016-2017 BRFSS state-level design-based estimates and the 2016-2017 NSDUH state model-based estimates, respectively (as given in Tables 1 and 2). To compute the variance of , that is, , let and , then . The covariance term can be assumed to be zero because the BRFSS and NSDUH samples are independent.

The quantity can be obtained by using the 95 percent Bayesian confidence intervals in Tables 1 and 2. For this purpose, let denote the 95 percent Bayesian confidence interval¹¹ for a given state-s:

,     D

where  .

The quantity can be obtained by using the 95 percent confidence intervals in Tables 1 and 2. For this purpose, let denote the 95 percent BRFSS confidence interval for a given state-s, then is given by

.     D

Now, using the first-order Taylor series approximation,¹² can be calculated from as follows: .

The p value that is given in Tables 1 and 2 for testing the null hypothesis of no difference () is provided by , where is a standard normal random variate, , and denotes the absolute value of .

Alcohol Use

As can be seen in Table 1, for past month alcohol use, the NSDUH estimates and the BRFSS estimates for a little less than a fourth of the states were different (i.e., at the 5 percent level of significance, 11 of 51 states had different estimates). However, these two sets of estimates were highly correlated (correlation coefficient = 0.96). Figures 1 and 2, which follow this document's three tables, were created by using state estimates from BRFSS and NSDUH and categorizing the states into five quintiles similar to the process described on the title page of the "2016-2017 NSDUH National Maps of Prevalence Estimates, by State."¹³

As can be seen in Figures 1 and 2, eight states with the highest estimates of alcohol use (states shown in red) were the same in the two surveys: Colorado, the District of Columbia, Massachusetts, New Hampshire, North Dakota, Rhode Island, Vermont, and Wisconsin. Note that Minnesota was the other state in the top BRFSS group and that Connecticut and South Dakota were the other two states in the top NSDUH group. Eight states with the lowest estimates of alcohol use were the same in the two surveys: Alabama, Arkansas, Kentucky, Mississippi, Oklahoma, Tennessee, Utah, and West Virginia. Note that Georgia and Idaho rounded out the bottom BRFSS group and that Hawaii and North Carolina were the other states in the bottom NSDUH group. The lowest estimate of past month alcohol use was in Utah for both BRFSS and NSDUH (see Table 1 and Figures 1 and 2).

Binge Alcohol Use

As can be seen in Table 2, the NSDUH estimates of past month binge alcohol use were significantly larger than the BRFSS estimates for all states. As noted previously, NSDUH and BRFSS used the same thresholds for binge alcohol use among males and females in 2016 and 2017. The use of audio computer-assisted self-interviewing (ACASI) in NSDUH, which is considered to be more anonymous than the use of computer-assisted telephone interviewing (CATI) in BRFSS and yields higher reporting of sensitive behaviors, may explain these findings. Although the NSDUH estimates were larger, these two sets of estimates are moderately correlated (correlation coefficient = 0.80).

Figures 3 and 4 were created using the same method used to produce Figures 1 and 2. As can be seen in Figures 3 and 4, six states with the highest estimates of binge alcohol use (states shown in red) were the same in the two surveys: the District of Columbia, Iowa, Montana, Nebraska, North Dakota, and Wisconsin. Rounding out the top BRFSS group were Colorado, Hawaii, Illinois, and Minnesota, while Massachusetts, New Hampshire, Rhode Island, and South Dakota rounded out the top NSDUH group. Eight states with the lowest estimates of binge alcohol use were the same in the two surveys: Alabama, Arkansas, Georgia, Mississippi, North Carolina, Tennessee, Utah, and West Virginia. Note that the other two states in the bottom BRFSS group were New Mexico and Oklahoma and that the other two states in the bottom NSDUH group were Hawaii and Kentucky.

Cigarette Use

As can be seen in Table 3, the NSDUH estimates of past month cigarette use were always larger than the BRFSS estimates of current cigarette use. Some of this difference is the result of the differences in definitions as discussed earlier in this document; thus, exact tests to examine significant differences between the NSDUH and BRFSS cigarette use estimates are not included. Although the NSDUH estimates tended to be larger, these two sets of estimates were highly correlated (correlation coefficient = 0.93).

Figures 5 and 6 were created using the same method used to produce Figures 1 through 4. As can be seen in Figures 5 and 6, eight states with the highest estimates of cigarette use (states shown in red) were the same in the two surveys: Alabama, Arkansas, Kentucky, Louisiana, Mississippi, Ohio, Tennessee, and West Virginia. Rounding out the top BRFSS group were Indiana and Missouri, while Oklahoma and South Dakota rounded out the top NSDUH group. Eight states with the lowest estimates of cigarette use were the same in the two surveys: California, Hawaii, Maryland, Massachusetts, New Jersey, New York, Utah, and Washington. Note that the other two states in the bottom BRFSS group were Connecticut and Idaho and that the other two states in the bottom NSDUH group were Colorado and Florida.

Sample Size Comparisons

The BRFSS estimates are design based, while the NSDUH estimates are model based. Both sets of estimates are based on 2 years of pooled data (2016-2017). The BRFSS sample sizes for a given state were in general much larger than the sample sizes for NSDUH (both over 2 years). In the 2016-2017 NSDUH, the 18 or older sample sizes in the states ranged from 1,363 to 6,775 respondents, with a median sample size of 1,469.¹⁴ For the 2016-2017 BRFSS, all of the states had larger sample sizes as compared with their counterparts in NSDUH. Overall, the BRFSS sample sizes over 2 years for the states varied from a low of 6,117 to a high of 59,014 respondents, with a median sample size of 13,785.¹⁵ Sample size differences of this magnitude explain why the NSDUH Bayesian confidence intervals were generally wider than the corresponding BRFSS design-based confidence intervals.

References

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (DSM-5) (5th ed.). Arlington, VA: Author.

Brener, N. D., Eaton, D. K., Kann, L., Grunbaum, J. A., Gross, L. A., Kyle, T. M., & Ross, J. G. (2006). The association of survey setting and mode with self-reported health risk behaviors among high school students. Public Opinion Quarterly, 70, 354-374. https://doi.org/10.1093/poq/nfl003 

Center for Behavioral Health Statistics and Quality. (2012). Comparing and evaluating youth substance use estimates from the National Survey on Drug Use and Health and other surveys (HHS Publication No. SMA 12-4727, Methodology Series M-9). Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2016). Section E: Other sources of data. In 2015 National Survey on Drug Use and Health: Methodological summary and definitions. Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2017a). Section E: Other sources of data. In 2016 National Survey on Drug Use and Health: Methodological summary and definitions. Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2017b). State-level comparisons of mental health issues from the National Survey on Drug Use and Health (NSDUH) and the Behavioral Risk Factor Surveillance System (BRFSS). Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2018). Section E: Other sources of data. In 2017 National Survey on Drug Use and Health: Methodological summary and definitions. Retrieved from https://www.samhsa.gov/data/

Centers for Disease Control and Prevention. (2015, September). Behavioral Risk Factor Surveillance System: Comparability of data BRFSS 2014 (Version #1–Revised). Retrieved from https://www.cdc.gov/brfss/annual_data/2014/pdf/compare_2014.pdf

Gfroerer, J., Wright, D., & Kopstein, A. (1997). Prevalence of youth substance use: The impact of methodological differences between two national surveys. Drug and Alcohol Dependence, 47, 19-30. https://doi.org/10.1016/s0376-8716(97)00063-x 

Grucza, R. A., Abbacchi, A. M., Przybeck, T. R., & Gfroerer, J. C. (2007). Discrepancies in estimates of prevalence and correlates of substance use and disorders between two national surveys. Addiction, 102, 623-629. https://doi.org/10.1111/j.1360-0443.2007.01745.x 

Hennessy, K. H., & Ginsberg, C. (Eds.). (2001). Substance use survey data collection methodologies and selected papers [Special issue]. Journal of Drug Issues, 31(3), 595-808. https://doi.org/10.1177/002204260103100301 

Miller, J. W., Gfroerer, J. C., Brewer, R. D., Naimi, T. S., Mokdad, A., & Giles, W. H. (2004). Prevalence of adult binge drinking: A comparison of two national surveys. American Journal of Preventive Medicine, 27, 197-204. https://doi.org/10.1016/s0749-3797(04)00121-7

Figure 1. Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2016 and 2017 BRFSS

     D

Source: Centers for Disease Control and Prevention (CDC), Behavioral Risk Factor Surveillance System Survey, 2016 and 2017.

Figure 2. Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2016 and 2017 NSDUHs

     D

Source: SAMHSA, Center for Behavioral Health Statistics and Quality, NSDUH, 2016 and 2017.

Figure 3. Binge Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2016 and 2017 BRFSS

     D

Source: Centers for Disease Control and Prevention (CDC), Behavioral Risk Factor Surveillance System Survey, 2016 and 2017.

Figure 4. Binge Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2016 and 2017 NSDUHs

     D

Source: SAMHSA, Center for Behavioral Health Statistics and Quality, NSDUH, 2016 and 2017.

Figure 5. Current Cigarette Use among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2016 and 2017 BRFSS

     D

Source: Centers for Disease Control and Prevention (CDC), Behavioral Risk Factor Surveillance System Survey, 2016 and 2017.

Figure 6. Cigarette Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2016 and 2017 NSDUHs

     D

Source: SAMHSA, Center for Behavioral Health Statistics and Quality, NSDUH, 2016 and 2017.