Methodology Used to Estimate the Bayes
Significance Level for the Null Hypothesis of
No Change between the 2012-2014 and 2014-2016
Substate Population Percentages
This discussion focuses on the methodology used to conduct statistical tests of significance for comparing National Survey on Drug Use and Health (NSDUH) substate population percentages between 2012-2014 and 2014-2016. Population percentages shown in Tables 1 to 12 earlier in this document are based on small area estimation (SAE) procedures, and the methodology used to compare these population percentages is the same methodology used to compare the state small area estimates.1 However, because some substate regions within 10 states (Arizona, Colorado, Illinois, Kansas, Minnesota, New Mexico, New York, North Dakota, Rhode Island, and Utah) were defined differently for the two time periods (i.e., 2012-2014 and 2014-2016), significance test results (based on the Bayes posterior probability of no change: p value) could not be produced for these substate regions. In this document's tables, a double hyphen ("--") is used to denote substate regions that were not defined consistently across the two time periods. For more details about the substate regions that had definitions that changed, see the document on the "2014-2016 NSDUH Substate Region Definitions" at https://www.samhsa.gov/data/.
Comparisons of substate population percentages between 2012-2014 and 2014-2016 were made only for individuals aged 12 or older except for the underage alcohol use outcome in Table 5 (where comparisons were made for individuals aged 12 to 20) and for four mental health outcomes on serious mental illness, any mental illness, serious thoughts of suicide, and major depressive episode in Tables 9 through 12, respectively (where comparisons were made for adults aged 18 or older).
To determine whether differences between substate prevalence rates over two time periods were statistically significant, let 
and 
denote the 2012-2014 and 2014-2016 prevalence rates, respectively, for substate-s. The difference between and is defined in terms of the log-odds ratio 
as opposed to the simple difference because the posterior distribution of the is closer to Gaussian than the posterior distribution of the simple difference, 
. The is defined as follows:
. D
In this equation, the ln denotes the natural logarithm. The p value given in each of the tables in this document (i.e., Tables 1 to 12) is computed to test the null hypothesis of no difference; that is, 
or equivalently 
. An estimate of is given by
, D
where the 
are the 2012-2014 substate estimates presented in the "2012-2014 NSDUH Substate Region Estimates: Excel Tables" at https://www.samhsa.gov/data/, and the 
are the 2014-2016 substate estimates presented in the "2014-2016 NSDUH Substate Region Estimates: Excel Tables" at https://www.samhsa.gov/data/.
To compute the posterior variance of 
, that is, 
, let 
and 
, then 
, where 
denotes the covariance between 
and 
. This covariance is defined in terms of the associated correlation, as follows:
. D
Note that 
and 
used here to calculate 
are the same variances used in calculating the 2012-2014 Bayesian confidence intervals (CIs) and the 2014-2016 CIs, respectively. Thus, the posterior variances can be calculated using the published Bayesian CIs, 
and 
, as follows:
, D
where 
and 
.
The correlation between and was obtained by simultaneously modeling the 2012-2013, 2014, and the 2015-2016 NSDUH data using a method similar to obtaining the correlation for producing state-level estimates of .2
To calculate the p value for testing the null hypothesis of no difference (
), it was assumed that the posterior distribution of 
is normal with 
and 
. With the null value of (), the Bayes p value or significance level for the null hypothesis of no difference is 
, where Z is a standard normal random variate, 
, and 
denotes the absolute value of z. This Bayesian significance level (or p value) for the null value of , say 
, is defined following Rubin,3 as the posterior probability for the collection of the values that are less likely or have smaller posterior density 
than the null (no change) value . That is, 
. With the posterior distribution of approximately normal, 
is given by the above expression.
