2018-2019
National Survey on Drug Use and Health:
Other Sources of State-Level Data

Introduction

A variety of surveys and data systems other than the National Survey on Drug Use and Health (NSDUH) collect data on substance use problems and mental disorders. It is useful to consider the results of these other studies when discussing NSDUH data. This document briefly describes one of these other data systems that publish state estimates and presents selected comparisons with NSDUH results. The state-level survey that collects data on substance use discussed in this document is the Behavioral Risk Factor Surveillance System (BRFSS), sponsored by the Centers for Disease Control and Prevention (CDC). Another CDC data system that provides state-level substance use estimates for most but not all states is the Youth Risk Behavior Survey (YRBS). Differences between the YRBS and NSDUH sampling designs, as well as the wider range of age groups used in NSDUH small area estimates, imply that comparisons of estimates are not straightforward. However, ignoring these differences and examining estimates at a national level, the YRBS has been generally shown to have higher estimates than NSDUH has (Center for Behavioral Health Statistics and Quality [CBHSQ], 2016, 2017a, 2018, 2019, 2020).¹ Note that comparisons between the state YRBS estimates and the NSDUH small area estimates are not presented because of some of the differences discussed above.

When considering the information presented in this document, it is important to understand the methodological differences between these surveys and the impact that these differences could have on estimates of substance use and mental health. Several studies have compared NSDUH estimates with estimates from other studies and have evaluated how differences may have been affected by differences in survey methodology (Brener et al., 2006; CBHSQ, 2012; Gfroerer, Wright, & Kopstein, 1997; Grucza, Abbacchi, Przybeck, & Gfroerer, 2007; Hennessy & Ginsberg, 2001; Miller et al., 2004). These studies suggest that the goals and approaches of surveys are often different, making comparisons between them difficult. Some methodological differences that have been identified as affecting comparisons include populations covered, sampling methods, mode of data collection, survey setting, questionnaires, and estimation methods.

BRFSS is a state-based system of health surveys that collect information on health risk behaviors (including cigarette and alcohol use), preventive health practices, and health care access primarily related to chronic disease, injuries, and preventable infectious diseases. BRFSS is an annual, state-based telephone (landline and cellular telephone) survey of the civilian, noninstitutionalized adult population aged 18 or older and is sponsored by the CDC. In 2018 and 2019, BRFSS collected data from all 50 states, the District of Columbia, Puerto Rico, the U.S. Virgin Islands, American Samoa, Palau, and Guam using computer-assisted telephone interview (CATI) systems. Note that, in 2019, New Jersey was unable to collect enough BRFSS data to meet the minimum requirements for inclusion in the 2019 annual aggregate dataset. Thus, the estimate shown in this report for New Jersey is based only on 2018 data. More than 400,000 adults are interviewed each year, and state estimates are presented annually.

In 2011, BRFSS introduced two methodological changes: (1) the inclusion of cellular telephone-only households in the sample, and (2) the incorporation of iterative proportional fitting (also referred to as "raking") in the production of the final BRFSS weights, replacing the use of poststratification. Cellular telephone-only households were added to improve survey coverage of the telephone population and addressed differences in characteristics found between the cellular telephone-only and landline populations. Since 2014, BRFSS respondents who had a cellular telephone were eligible for participation in the cellular telephone survey. In 2013, on the other hand, to be eligible to participate in the cellular telephone survey, respondents had to be in either a cellular telephone-only household or a household where 90 percent or more of their calls were received on cellular telephones. Because state-level demographic characteristics of cellular telephone-only households are not available, weighting with the previous method of poststratification was no longer feasible. As a result of these methodological changes in 2014, the CDC reported small increases in various health risk indicators, including tobacco use and binge drinking.² The pooled 2018-2019 BRFSS state estimates and confidence intervals are design-based (direct) estimates (i.e., each respondent is weighted in a way that accounts for the survey design).³

Also in 2011, the BRFSS questionnaire underwent some changes in the alcohol consumption and tobacco use sections. In 2010, BRFSS respondents were asked, "During the past 30 days, have you had at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?" The response to this question was used to route respondents to the next question regarding the frequency of alcohol use in the past 30 days. However, only the responses to the first question were used to determine past month alcohol use. In the 2011 BRFSS questionnaire, this question was dropped, and respondents were directly asked, "During the past 30 days, how many days per week or per month did you have at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?" If a respondent answered "1" or higher to this question, he or she was considered a past month user of alcohol. In spite of the questionnaire changes, BRFSS is still producing an estimate of past month alcohol use that can be compared with the NSDUH estimate. Also, minor wording changes were made in one question in the tobacco use section, but none of these would affect the current cigarette use estimates. These newly worded questions were also used in the 2012 to the 2019 BRFSS surveys.

In both BRFSS and NSDUH, data are collected on the following four substance use and mental health measures in each of the 50 states and the District of Columbia:⁴

• past month alcohol use,
• cigarette use ("past month" use for NSDUH and "current" use for BRFSS),
• past month binge alcohol use, and
• lifetime doctor-diagnosed depression.⁵

Note that estimates for only the first three of these four measures are compared in this document because small area estimates of lifetime doctor-diagnosed depression were not produced for NSDUH.⁶ The BRFSS and NSDUH questions that were used for the first three measures are shown in the next section.

Past month alcohol use is defined consistently in both BRFSS and NSDUH as having an alcoholic beverage in the past month. Similarly, past month binge alcohol use is defined consistently in the two surveys as drinking five or more drinks (for males) or four or more drinks (for females) on the same occasion (i.e., at the same time or within a couple of hours of each other) on at least 1 day in the past 30 days. In 2014 and prior years, NSDUH's binge alcohol use definition for males and females was having had five or more drinks of an alcoholic beverage on the same occasion (i.e., at the same time or within a couple hours of each other) on at least 1 day in the past 30 days.

In NSDUH, past month cigarette use is defined as having smoked part or all of a cigarette during the past 30 days (i.e., the 30 days prior to the interview). In BRFSS, the cigarette use measure reported is current cigarette use, which is defined as having smoked at least 100 cigarettes during the lifetime and indicating smoking every day or some days at the time of the survey. Because of these subtle but present differences in definitions, NSDUH's cigarette use estimates tend to be higher in that they cover two groups of people that the BRFSS estimates would not: (1) respondents who have not smoked 100 cigarettes in their lifetime but had smoked in the past month, and (2) respondents who had smoked a cigarette earlier in the month but were not smoking at the time of the survey.

This document presents findings related to comparing combined 2018-2019 BRFSS state design-based estimates with corresponding NSDUH state small area estimates for past month alcohol use, past month binge alcohol use, and cigarette use ("past month" use for NSDUH and "current" use for BRFSS). In Tables 1, 2, and 3 (shown after this text discussion), the pooled 2018-2019 BRFSS state design-based estimates for adults aged 18 or older are shown alongside the pooled 2018-2019 NSDUH small area estimates for the same age group. Tables 1 and 2 also include p values that indicate whether the BRFSS and NSDUH alcohol use and binge alcohol use estimates are significantly different from each other for a given state using an exact test as described in the next section. Due to definitional differences in the cigarette use measure, no tests of differences between NSDUH and BRFSS estimates were produced.

NSDUH and BRFSS Questions

The 2019 NSDUH questions that were used to determine past month alcohol use and past month binge alcohol use were worded as follows:⁷

AL01

Have you ever, even once, had a drink of any type of alcoholic beverage? Please do not include times when you only had a sip or two from a drink.

1   Yes
2   No
DK/REF⁸

ALLAST3

[IF AL01 = 1 OR ALREF = 1]How long has it been since you last drank an alcoholic beverage?

1   Within the past 30 days – that is, since [DATEFILL]
2   More than 30 days ago but within the past 12 months
3   More than 12 months ago

DK/REF
PROGRAMMER: SHOW 12 MONTH CALENDAR

AL08

[IF ALC30DAY = 1 – 30 OR ALCEST30 = (1 – 6, DK OR REF)] During the past 30 days, that is, since [DATEFILL], on how many days did you have [IF QD01=5 (MALE) THEN FILL 5, IF QD01=9 (FEMALE) THEN FILL 4] or more drinks on the same occasion? By 'occasion,' we mean at the same time or within a couple of hours of each other.

# OF DAYS: _____ [RANGE: 0 - 30]
DK/REF
PROGRAMMER: SHOW 30 DAY CALENDAR

The 2019 BRFSS questions that were used to determine past month alcohol use and past month binge alcohol use were worded as follows:⁹

C10.01

During the past 30 days, how many days per week or per month did you have at least one drink of any alcoholic beverage such as beer, wine, a malt beverage or liquor?

INTERVIEWER NOTE: One drink is equivalent to a 12-ounce beer, a 5-ounce glass of wine, or a drink with one shot of liquor.

1 _ _ Days per week
2 _ _ Days in past 30 days
888 No drinks in past 30 days
777 Don't know / Not sure
999 Refused

C10.03

Considering all types of alcoholic beverages, how many times during the past 30 days did you have X [CATI X = 5 for men, X = 4 for women] or more drinks on an occasion?

_ _ Number of times
88 None
77 Don't know / Not sure
99 Refused

The 2019 NSDUH questions that were used to determine past month cigarette use were worded as follows:

CG01

Have you ever smoked part or all of a cigarette?

1   Yes
2   No
DK/REF

CG05

[IF CG01 = 1 OR CGREF1 = 1] Now think about the past 30 days, that is, from [DATEFILL] up to and including today. During the past 30 days, have you smoked part or all of a cigarette?

1   Yes
2   No
DK/REF
PROGRAMMER: SHOW 30 DAY CALENDAR

The 2019 BRFSS questions that were used to determine current cigarette use were worded as follows:

C09.01

Have you smoked at least 100 cigarettes in your entire life?

1   Yes
2   No
7   Don't know / Not sure
9   Refused

INTERVIEWER NOTE: Do not include: electronic cigarettes (e-cigarettes, njoy, bluetip), herbal cigarettes, cigars, cigarillos, little cigars, pipes, bidis, kreteks, water pipes (hookahs), or marijuana. 5 packs = 100 cigarettes

C09.02

Do you now smoke cigarettes every day, some days, or not at all?

1   Every day
2   Some days
3   Not at all
7   Don't know / Not sure
9   Refused

Note that these 2019 questions for NSDUH and BRFSS were essentially the same as their 2018 counterparts except for slight variations in the formatting of the BRFSS interviewer notes.

Methodology for Comparing BRFSS and NSDUH Estimates

The methodology used to compare BRFSS and NSDUH estimates is similar to what is described in Section B.7 of the "2014-2015 NSDUH: Guide to State Tables and Summary of Small Area Estimation Methodology."¹⁰ Here, the null hypothesis of no difference is tested, that is, (where is the expected value¹¹ of the BRFSS estimate and is the expected value of the NSDUH estimate) or equivalently that the log-odds ratio is zero, that is, , where is defined as , and ln denotes the natural logarithm. An estimate of is given by , where and are the 2018-2019 BRFSS state-level design-based estimates and the 2018-2019 NSDUH state model-based estimates, respectively (as given in Tables 1 and 2). To compute the variance of , that is, , let and , then

.      D

The covariance term can be assumed to be zero because the BRFSS and NSDUH samples are independent.

The quantity can be obtained by using the 95 percent Bayesian confidence intervals in Tables 1 and 2. For this purpose, let denote the 95 percent Bayesian confidence interval¹² for a given state:

,      D

where  .

The quantity can be obtained by using the 95 percent confidence intervals in Tables 1 and 2. For this purpose, let denote the 95 percent BRFSS confidence interval for a given state, then is given by

.      D

Now, using the first-order Taylor series approximation,¹³ can be calculated from as follows: .

The p value that is given in Tables 1 and 2 for testing the null hypothesis of no difference () is provided by , where is a standard normal random variate, , and denotes the absolute value of .

Alcohol Use

As can be seen in Table 1, for past month alcohol use, the NSDUH estimates and the BRFSS estimates for about two fifths of the states were different (i.e., at the 5 percent level of significance, 21 of 51 states had different estimates). However, these two sets of estimates were highly correlated (correlation coefficient = 0.97). Figures 1 and 2, which follow this document's three tables, were created by using state estimates from BRFSS and NSDUH and categorizing the states into five quintiles similar to the process described on the title page of the "2018-2019 NSDUH National Maps of Prevalence Estimates, by State."¹⁴

As can be seen in Figures 1 and 2, nine states with the highest estimates of alcohol use (states shown in orange) were the same in the two surveys: Colorado, Connecticut, District of Columbia, Massachusetts, Minnesota, New Hampshire, Rhode Island, Vermont, and Wisconsin. Note that North Dakota was the other state in the top BRFSS group and that Iowa was the other state in the top NSDUH group. Eight states with the lowest estimates of alcohol use were the same in the two surveys: Alabama, Arkansas, Kentucky, Mississippi, Oklahoma, Tennessee, Utah, and West Virginia. Note that Idaho and North Carolina rounded out the bottom BRFSS group and that Hawaii and New Mexico were the other states in the bottom NSDUH group. The lowest estimate of past month alcohol use was in Utah for both BRFSS and NSDUH (see Table 1 and Figures 1 and 2).

Binge Alcohol Use

As can be seen in Table 2, the NSDUH estimates of past month binge alcohol use were significantly larger than the BRFSS estimates for all states. As noted previously, NSDUH and BRFSS used the same thresholds for binge alcohol use among males and females in 2018 and 2019. The use of audio computer-assisted self-interviewing (ACASI) in NSDUH, which is considered to be more anonymous than the use of CATI systems in BRFSS and yields higher reporting of sensitive behaviors, may explain these findings. Although the NSDUH estimates were larger, these two sets of estimates are moderately correlated (correlation coefficient = 0.82).

Figures 3 and 4 were created using the same method used to produce Figures 1 and 2. As can be seen in Figures 3 and 4, seven states with the highest estimates of binge alcohol use (states shown in orange) were the same in the two surveys: the District of Columbia, Iowa, Massachusetts, Montana, Nebraska, North Dakota, and Wisconsin. Other states in the top BRFSS group were Illinois, Minnesota, and South Dakota, while Colorado, Connecticut, and New Hampshire were the other states in the top NSDUH group. Six states with the lowest estimates of binge alcohol use were the same in the two surveys: Arkansas, Mississippi, Oklahoma, Tennessee, Utah, and West Virginia. Note that the other states in the bottom BRFSS group were Alabama, Maryland, New Jersey, and North Carolina and that the other states in the bottom NSDUH group were Arizona, Georgia, New Mexico, and Washington.

Cigarette Use

Table 3 shows the NSDUH estimates of past month cigarette and the BRFSS estimates of current cigarette use. Exact tests to examine significant differences between the NSDUH and BRFSS cigarette use estimates are not included because the definitions are different, as discussed earlier in this document. Although the NSDUH estimates tended to be larger, these two sets of estimates were highly correlated (correlation coefficient = 0.92).

Figures 5 and 6 were created using the same method used to produce Figures 1 through 4. As can be seen in Figures 5 and 6, eight states with the highest estimates of cigarette use (states shown in orange) were the same in the two surveys: Alabama, Arkansas, Indiana, Kentucky, Louisiana, Mississippi, Ohio, and West Virginia. Other states in the top BRFSS group were Missouri and Tennessee, while Oklahoma and South Carolina were the other states in the top NSDUH group. Eight states with the lowest estimates of cigarette use were the same in the two surveys: California, Connecticut, Hawaii, Maryland, Massachusetts, New Jersey, New York, and Utah. Note that the other two states in the bottom BRFSS group were the District of Columbia and Washington and that the other two states in the bottom NSDUH group were Florida and New Hampshire.

Sample Size Comparisons

The BRFSS estimates are design based, while the NSDUH estimates are model based. Both sets of estimates are based on 2 years of pooled data (2018-2019). The BRFSS sample sizes for a given state were in general much larger than the sample sizes for NSDUH (both over 2 years). In the 2018-2019 NSDUH, the 18 or older sample sizes in the states ranged from 1,362 to 6,929 respondents, with a median sample size of 1,445.¹⁵ For the 2018-2019 BRFSS, all of the states had larger sample sizes as compared with their counterparts in NSDUH. Overall, the BRFSS sample sizes over 2 years for the states varied from a low of 3,090 to a high of 49,999 respondents, with a median sample size of 13,658.¹⁶ Sample size differences of this magnitude explain why the NSDUH Bayesian confidence intervals were generally wider than the corresponding BRFSS design-based confidence intervals.

References

American Psychiatric Association. (2013). Diagnostic and statistical manual of mental disorders (DSM-5) (5th ed.). Arlington, VA: Author.

Brener, N. D., Eaton, D. K., Kann, L., Grunbaum, J. A., Gross, L. A., Kyle, T. M., & Ross, J. G. (2006). The association of survey setting and mode with self-reported health risk behaviors among high school students. Public Opinion Quarterly, 70, 354-374. https://doi.org/10.1093/poq/nfl003 

Center for Behavioral Health Statistics and Quality. (2012). Comparing and evaluating youth substance use estimates from the National Survey on Drug Use and Health and other surveys (HHS Publication No. SMA 12-4727, Methodology Series M-9). Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2016). Section E: Other sources of data. In 2015 National Survey on Drug Use and Health: Methodological summary and definitions. Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2017a). Section E: Other sources of data. In 2016 National Survey on Drug Use and Health: Methodological summary and definitions. Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2017b). State-level comparisons of mental health issues from the National Survey on Drug Use and Health (NSDUH) and the Behavioral Risk Factor Surveillance System (BRFSS). Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2018). Section E: Other sources of data. In 2017 National Survey on Drug Use and Health: Methodological summary and definitions. Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2019). Chapter 5: Other sources of data. In 2018 National Survey on Drug Use and Health: Methodological summary and definitions. Retrieved from https://www.samhsa.gov/data/

Center for Behavioral Health Statistics and Quality. (2020). Chapter 5: Other sources of data. In 2019 National Survey on Drug Use and Health: Methodological summary and definitions. Retrieved from https://www.samhsa.gov/data/

Centers for Disease Control and Prevention. (2015, September). Behavioral Risk Factor Surveillance System: Comparability of data BRFSS 2014 (Version #1–Revised). Retrieved from https://www.cdc.gov/brfss/annual_data/2014/pdf/compare_2014.pdf

Centers for Disease Control and Prevention. (2020, August 31). The Behavioral Risk Factor Surveillance System (BRFSS). Overview: BRFSS 2019. Retrieved from https://www.cdc.gov/brfss/annual_data/annual_2019.html

Gfroerer, J., Wright, D., & Kopstein, A. (1997). Prevalence of youth substance use: The impact of methodological differences between two national surveys. Drug and Alcohol Dependence, 47, 19-30. https://doi.org/10.1016/s0376-8716(97)00063-x 

Grucza, R. A., Abbacchi, A. M., Przybeck, T. R., & Gfroerer, J. C. (2007). Discrepancies in estimates of prevalence and correlates of substance use and disorders between two national surveys. Addiction, 102, 623-629. https://doi.org/10.1111/j.1360-0443.2007.01745.x 

Hennessy, K. H., & Ginsberg, C. (Eds.). (2001). Substance use survey data collection methodologies and selected papers [Special issue]. Journal of Drug Issues, 31 (3), 595-808. https://doi.org/10.1177/002204260103100301 

Miller, J. W., Gfroerer, J. C., Brewer, R. D., Naimi, T. S., Mokdad, A., & Giles, W. H. (2004). Prevalence of adult binge drinking: A comparison of two national surveys. American Journal of Preventive Medicine, 27, 197-204. https://doi.org/10.1016/s0749-3797(04)00121-7 

Figure 1. Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2018 and 2019 BRFSS

     D

NOTE: In 2019, New Jersey was unable to collect enough BRFSS data to meet the minimum requirements for inclusion in the 2019 annual aggregate dataset. Thus, the BRFSS estimate for New Jersey is based on only the 2018 BRFSS data.
Source: Centers for Disease Control and Prevention (CDC), Behavioral Risk Factor Surveillance System Survey, 2018 and 2019.

Figure 2. Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2018 and 2019 NSDUHs

     D

Source: SAMHSA, Center for Behavioral Health Statistics and Quality, NSDUH, 2018 and 2019.

Figure 3. Binge Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2018 and 2019 BRFSS

     D

NOTE: In 2019, New Jersey was unable to collect enough BRFSS data to meet the minimum requirements for inclusion in the 2019 annual aggregate dataset. Thus, the BRFSS estimate for New Jersey is based on only the 2018 BRFSS data.
Source: Centers for Disease Control and Prevention (CDC), Behavioral Risk Factor Surveillance System Survey, 2018 and 2019.

Figure 4. Binge Alcohol Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2018 and 2019 NSDUHs

     D

Source: SAMHSA, Center for Behavioral Health Statistics and Quality, NSDUH, 2018 and 2019.

Figure 5. Current Cigarette Use among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2018 and 2019 BRFSS

     D

NOTE: In 2019, New Jersey was unable to collect enough BRFSS data to meet the minimum requirements for inclusion in the 2019 annual aggregate dataset. Thus, the BRFSS estimate for New Jersey is based on only the 2018 BRFSS data.
Source: Centers for Disease Control and Prevention (CDC), Behavioral Risk Factor Surveillance System Survey, 2018 and 2019.

Figure 6. Cigarette Use in the Past Month among Adults Aged 18 or Older, by State: Percentages, Annual Averages Based on 2018 and 2019 NSDUHs

     D

Source: SAMHSA, Center for Behavioral Health Statistics and Quality, NSDUH, 2018 and 2019.