Methodology Used to Estimate the Bayes
Significance Level for the Null Hypothesis of
No Change between the 2017-2018 and 2018-2019
State Population Percentages
This discussion focuses on the methodology used to conduct statistical tests of significance for comparing National Survey on Drug Use and Health (NSDUH) state population percentages between 2017-2018 and 2018-2019. Estimates shown in Tables 1 to 33 earlier in this document are based on the small area estimation (SAE) procedure. The moving average state estimates for the overlapping 2017-2018 and 2018-2019 time periods were obtained from independent applications of the survey-weighted hierarchical Bayes (SWHB) methodology; that is, the 2018-2019 models were fit independently of the previously fitted 2017-2018 models. This independent analysis approach was followed because there was no desire to revise the previously published 2017-2018 estimates. The same fixed predictor variables were used in the 2017-2018 and 2018-2019 models, but annual updates were made when more current versions were available.1 The age group-specific fixed predictor variables were defined at five levels (namely, the person level, census block group level, tract level, county level, and state level). Also, each age group model had 51 state-level random effects and 250 "within-state" area-level random effects.1
To determine whether differences between population percentages of a state over two time periods were statistically significant, let 
and 
denote the 2017-2018 and 2018-2019 population percentages, respectively, for state-s and age group-a. The difference between and is defined in terms of the log-odds ratio (
) as opposed to the simple difference because the posterior distribution of is closer to Gaussian than the posterior distribution of the simple difference (
). The is defined as follows:
. D
In this equation, the ln denotes the natural logarithm. The p value given in each of the tables in this document (i.e., Tables 1 to 33) is computed to test the null hypothesis of no difference (i.e., 
or equivalently, 
). An estimate of is given by
, D
where 
are the previously published 2017-2018 state estimates and 
are the 2018-2019 state estimates. Both the 2017-2018 and 2018-2019 state estimates are presented in Tables 1 to 33 above and elsewhere at https://www.samhsa.gov/data/.
Let 
and 
, noting that subscript sa has been dropped from 
and 
in order to simplify the notation. An estimate of the posterior variance of is given by the following formula:
, D
where 
denotes the covariance between 
and 
. This covariance is defined in terms of the associated correlation as follows:
. D
Note that 
and 
used here to calculate 
are the same posterior variances used in calculating the previously published 2017-2018 Bayesian confidence intervals (CIs) and the 2018-2019 Bayesian CIs, respectively.
The correlation between and was obtained by simultaneously modeling the 2017, 2018, and 2019 NSDUH data. Note that these models used the same predictors employed in the 2018-2019 SAE model for all 3 years. This simultaneous modeling approach was adopted based on the results of the validation study2 conducted for measuring change in the 1999-2000 and 2000-2001 state population percentages. For this simultaneous model, four age groups (12 to 17, 18 to 25, 26 to 34, and 35 or older) by 3 years (2017, 2018, and 2019), that is, 12 subpopulation-specific models, were fitted, each with its own set of fixed and random effects. In this case, the general covariance matrices for the state and within-state random effects were 12 × 12 matrices corresponding to the 12 element (age group × year) vectors of random effects. Note that the survey-weighted, Bernoulli-type log likelihood employed in the SWHB methodology was appropriate for this simultaneous model because the 12 (age group × year) subpopulations were nonoverlapping. The correlation 
was approximated by the correlation calculated using the posterior distributions of 
and 
from the simultaneous model. Note, for past year heroin use, the above-mentioned model did not converge. A different model that is based on simultaneous modeling of 2017-2018 and 2018-2019 data where 2018 data are repeated twice is used to obtain the correlations between 2017-2018 and 2018-2019 past year heroin use estimates. For more information about this model, see the "National Survey on Drug Use and Health: Comparison of 2014-2015 and 2015-2016 population percentages" at https://www.samhsa.gov/data/.
To calculate the p value for testing the null hypothesis of no difference (
), it is assumed that the posterior distribution of 
is normal with estimated 
and 
. The Bayesian p value or significance level for the null hypothesis of no difference () is 
, where 
is a standard normal random variate, 
, and 
denotes the absolute value of 
. This Bayesian significance level (or p value) for the null value of , say 
, is defined following Rubin3 as the posterior probability for the collection of the values that are less likely or have smaller posterior density 
than the null (no change) value . That is,
. D
With the posterior distribution of approximately normal, 
is given by the above expression.
