Methodology Used to Estimate the Bayes Significance Level for the Null Hypothesis
of No Change between the 2008-2009 and 2015-2016 Population Percentages
Estimating P Values
To determine whether the differences between the 2008-20091 and 2015-2016 National Survey on Drug Use and Health (NSDUH) state prevalence rates are statistically significant, let 
and 
denote the 2008-2009 and 2015-2016 prevalence rates, respectively, for state-s and age group-a. The difference between and is defined in terms of the log odds ratio (
) as opposed to the simple difference because the posterior distribution of is closer to Gaussian than the posterior distribution of the simple difference (
). The is defined as follows:
. D
In this equation, the ln denotes the natural logarithm. The p value given in each of the tables in this document (i.e., Tables 1 to 13) is computed to test the null hypothesis of no change (i.e., 
or equivalently, 
). An estimate of is given by
, D
where 
are the previously published 2008-2009 state estimates and 
are the 2015-2016 state estimates presented in Tables 1 to 13 and elsewhere at https://www.samhsa.gov/data/. To compute the variance of 
, that is, 
, let 
and 
, then
, D
where 
denotes the covariance between 
and 
. This covariance is defined in terms of the associated correlation as follows:
. D
Note that 
and 
used here to calculate are the same variances used in calculating the previously published 2008-2009 Bayesian confidence intervals and the 2015-2016 Bayesian confidence intervals given here, respectively. The correlation between and is estimated using generalized correlations,2 which are described in detail in the next section.
To calculate the p value for testing the null hypothesis of no difference (
) between the 2008-2009 and 2015-2016 state estimates, it is assumed that the posterior distribution of 
is normal with 
and 
. With the null value of (), the Bayes p value or significance level for the null hypothesis of no difference is 
,
where 
is a standard normal random variate, 
, and 
denotes the absolute value of 
. This Bayesian significance level (or p value) for the null value of , say 
, is defined following Rubin3 as the posterior probability for the collection of the values that are less likely or have smaller posterior density 
than the null (no change) value . That is, 
. With the posterior distribution of approximately normal, 
is given by the above expression.
Estimating Generalized Correlations
The correlation between and for each substance use outcome measure by state by age group combination was estimated by taking the average of seven correlations for each outcome measure by state by age group, as follows:
- 2002-2003 versus 2007-2008,
- 2002-2003 versus 2008-2009,
- two sets of 2002-2003 versus 2009-2010,4
- 2002-2003 versus 2010-2011,
- 2002-2003 versus 2012-2013, and
- 2002-2003 versus 2013-2014.
The average of these seven correlations is henceforth referred to as a "generalized correlation."
The generalized correlation between and for any mental illness (AMI), serious mental illness (SMI), and suicidal thoughts by state and age group was estimated by taking the average of six correlations for each outcome by state by age group, as follows:
- 2008-2009 versus 2010-2011,
- 2008-2009 versus 2011-2012,
- 2008-2009 versus 2012-2013,
- 2009-2010 versus 2011 2012,
- 2009-2010 versus 2012-2013, and
- 2010-2011 versus 2012-2013.
The generalized correlation between and for the fourth mental health measure—major depressive episode (MDE)—was estimated by taking the average of eight correlations for each outcome by age group by state, as follows:
- 2005-2006 versus 2007-2008,
- 2005-2006 versus 2008-2009,
- 2005-2006 versus 2009-2010,
- 2005-2006 versus 2010-2011,
- 2005-2006 versus 2011-2012,
- 2005-2006 versus 2012-2013,
- 2006-2007 versus 2009-2010, and
- 2008-2009 versus 2010-2011.
Each of these individual sets of correlations was produced by simultaneously fitting 4 years of NSDUH data separately for each outcome measure. For example, to produce correlations between the 2002-2003 and 2007-2008 state estimates for past month marijuana use, four age groups (12 to 17, 18 to 25, 26 to 34, and 35 or older) by two time periods (2002-2003 and 2007-2008), that is, eight subpopulation-specific models, were fitted each with its own set of fixed and random effects. In this case, the general covariance matrices for the state and within-state random effects were 8 × 8 matrices corresponding to the eight element (age group × time period) vectors of random effects.
A study was conducted using 2002-2003 versus 2007-2008 data and 2002-2003 versus 2008-2009 data.5 When the actual (modeled) correlation was replaced by an average correlation of those 2 years, the p values of testing differences between estimates from both sets of comparisons were fairly similar. In most cases, they had the same conclusion of whether the test between 2002-2003 and 2007-2008 and the test between 2002-2003 and 2008-2009 was statistically significant or not. Only a few tests changed their significance6 depending on which correlation (actual or the average) was used. For all of these cases where significance changed, the p values were close to 0.05 (i.e., borderline cases). Given these results, it was decided, starting in 2014-2015 and going forward, that generalized correlations could be used to estimate long-term change.
