Why does SAMHSA caution against comparing estimates from 2021 and later years with estimates from prior years?

Starting largely in 2021, NSDUH became a fully mixed-mode survey, with many responses from both web and in-person administration. Repeated analyses showed that the web responses are not directly comparable to the in-person responses and differences are not consistent enough to be fully accounted for with weights or other statistical measures. This means that years with a web response option and years without do not produce comparable estimates. Because of this, data from 2021 should not be compared to estimates from 2019 or earlier.

Estimates from 2021 should also not be compared to estimates from 2020 or even from just quarter 4 of 2020. The reasons are complex. First, the 2020 NSDUH is missing two quarters of data. Tests of data from the years preceding 2020 show that estimates based on just quarters 1 and 4 are not comparable to estimates based on the entire year’s worth of data. This means that 2020 estimates should not be compared to 2021.

Moreover, while the methodology for Quarter 4 of the 2020 NSDUH is mostly the same as for the 2021 NSDUH, data in Quarter 4 of 2020 mostly consists of web interviews. In 2021, the proportion of in-person interviews increased over the year as county-level restrictions related to COVID-19 were gradually lifted, and field interviewers were able to follow up in more areas. These mostly single-mode estimates from quarter 4 should not be compared to the multi-mode estimates from 2021.

A full description of the analyses can be found in chapter 6 of the 2021 Methodological Summary and Definitions report.