SAMHSA 2005 Budget

 

FY 2005 Government Performance and Results Act (GPRA)

    Part V. APPENDIX TO THE PERFORMANCE PLAN

D. Data Verification and Validation

CMHS --Methods for Verification and Validation

Program

Verification and Validation Information

HIV/AIDS Minority Mental Health Services

Data for this program will be obtained from grantee program records and management information systems.

4. Comprehensive Community Mental Health Services for Children and Their Families

The number of children served is obtained from grantees.

The scale used to assess inpatient-residential treatment was an adapted version of the Restrictive of Living Environments Scale and Placement Stability Scale (ROLES) developed by Hawkins and colleagues (1992). An analysis showed that the percentage of agreement between data from the ROLES and data from a management information system in one grantee community was 76%.

Data on children's outcomes are collected from a multi-site outcome study. Delinquency is reported using a self-report survey. Validity analyses were conducted for school attendance and law enforcement contacts. School attendance was found to have a positive relationship with school performance. Children who attended school frequently also had some tendency to receive good grades. The correlation between the two was .313 (p = .000). Data on family satisfaction with services were derived from the Family Satisfaction Questionnaire (FSQ), a measure widely used and recognized for its reliability and validity. Validity analyses indicated that there was a positive correlation of .263 (p = .000) between the FSQ, a care giver-reported instrument, and youth self-reported satisfaction.

Data on clinical outcomes were derived from Reliable Change Index (RCI) scores (Jacobson & Truax, 1991), calculated from entry into services to six months for the Total Problem scores of the Child Behavior Checklist (CBCL, Achenbach, 1991). The Reliable Change Index (RCI) is a standardized method developed by Jacobson and his colleagues to measure change between two data points. The RCI has a clear-cut criterion for improvement that has been psychometrically tested and found to be sound (Jacobson & Truax, 1991).

5. Protection and Advocacy for Individuals with Mental Illness

Data sources for all PAIMI measures are the annual Program Performance Reports and Advisory Council Reports submitted annually by each of the P&A systems as required by the PAIMI Act. The information provided in the annual reports is checked for reliability during on-site PAIMI program visits, annual reviews, and budget application reviews. The information provided in each State's annual Program Performance Reports and Advisory Council Reports is reliable.

6. Projects for Assistance in Transition from Homelessness (PATH)

The source of the information is data submitted annually to CMHS by States, which obtain the information from local human service agencies that provide services. To improve the quality of the data, CMHS has developed additional error checks to screen data and contacts States and local providers concerning accuracy when data is reported outside expected ranges. CMHS has also issued guidance to all States and localities on data collection and monitors compliance with data collection through increased site visits to local PATH-funded agencies. PATH adopted quality control measures have eliminated much double counting of clients and will continue to improve data quality.

10. Community Mental Health Services Block Grant

New measure have been implemented on data reported through the annual Block Grant application. States have been supported by Data Infrastructure Grants, which provide definitions and standards.

CSAP - Methods for Verification and Validation

Program

Information

11. Synar Amendment Implementation Activities

Analyses of compliance rates are performed each year based on data reported in the SAPT block grant applications. The data source is the Synar report, part of the SAPT block grant application submitted annually by each State. States must certify that Block Grant data are accurate. The validity and reliability of the data are ensured through technical assistance, conducting random unannounced checks, and the confirmation of the data by scientific experts, site visits and other similar steps. CSAP is able to provide leadership and guidance to States on appropriate sample designs and other technical requirements, based on scientific literature and demonstrated best practices for effective implementation of Synar. Data sources for the baseline and measures are derived from State project officers' logs and from organizations that were awarded State technical assistance contracts. The analysis is based upon the actual requests/responses received, therefore providing a high degree of reliability and validity.

8. State Incentive Grants (SIGs)

States have agreed to use the same instruments and to collect the same types of data. Data will be collected through several mechanisms: State grantees, local (local community or provider project level) and school and community-based surveys. Data are being sent to a CSAP data retrieval system for entry and analysis. Quality of the data is expected to be high.

States are responsible for local evaluations of a representative sample of these programs. In addition to the States' own evaluations of local programs, over the three years of their grants each State will report data from local subgroups of SIG funds to CSAP on a semi-annual basis for the national cross-site evaluation. The cross-site evaluation team is in the process of completing site visits during which they will evaluate program fidelity, adaptation, and implementation issues.

Working toward performance based budgeting, CSAP is establishing and refining the SIG data collection system to gather information which will directly link cost to program participation. The next reporting cycle will include a measure which links States' SIG expenditures to the number of participants in SIG programs/activities. A baseline will be established for FY 2002.

10. Substance Abuse Prevention and HIV Prevention Initiative Program

It is expected that youth receiving substance abuse prevention services will have an increased perception of risk for substance abuse. These attitudes are expected to result in lower substance use. This program will use the SAMHSA GPRA cross-cutting instrument, which uses measures from reliable and valid instruments. Perception of risk has been shown to have high concurrent validity with drug and alcohol use and other negative behaviors. It is also expected that youth receiving integrated substance abuse prevention and HIV prevention services who have not yet begun sexual activity will delay their first sexual encounter, thus reducing their risk of HIV.

The SAP/HIV program has developed a survey instrument using questions from established instruments to measure this goal. Data is being collected from individual sites on number, types, and quality of services.

9. Centers for the Application of Prevention Technologies (CAPTs)

The national CAPT data collection system reflects a number of critical decisions about the most accurate and effective way to assess the work of the CAPTs. For example, the Technical Assistance (TA) database now focuses on overall TA services provided, and includes selected client ratings (satisfaction with and utility of CAPT service provided). The Event database now allows an examination of participant ratings (satisfaction with event and likelihood of using the information received). In future reports, these client satisfaction data will be provided. The new Systemic Outcomes database captures information on substantive changes that are in some way related to the work of the CAPTs. This redesigned data system represents a significant commitment to tracking the impact of CAPT work. Each CAPT follows a quality control protocol prior to collecting and submitting data, and CSAP has established an external quality control system through a support contractor overseen by CSAP staff.

12. 20% Prevention Block Grant Set-Aside

Data are carefully collected, cleaned , analyzed and reported through a data coordinating center.

CSAT Verification and Validation of Data Section

Program
Information

15. Substance Abuse Prevention and Treatment Block Grant

The number of clients served is a critical measure for the Block Grant program, particularly in light of the national goal to narrow the substance abuse treatment gap. TEDS admissions data have been used as proxy data to set targets and track results. However, the TEDS data represent admissions to treatment, not the total number of individual clients served. A person who presents for treatment twice during the data collection cycle will be included twice in the TEDS data set. TEDS admissions data do not capture either the total national demand for substance abuse treatment or the prevalence of substance use in the general population; data only represents admissions to treatment at facilities within the scope of TEDS data collection. SAMHSA has been working intensively with the Office of National Drug Control Policy to improve estimation methodology for the number of clients served, while efforts with States focus on improving their ability to collect unduplicated client counts. While still developmental, data for the planned Performance Partnership Grant measures will be collected by community-based providers using standard instruments which will be administered to clients by trained interviewers. Data will be forwarded to the SSA's for analysis and subsequent reporting to CSAT, using the Annual Block Grant Application as a reporting vehicle. Adoption by the States of these measures, following further developmental work, is an appropriate current measure for this critical activity.

Customer satisfaction is a good measure of the responsiveness and utility of SAMHSA's technical assistance. CSAT conducts an annual customer satisfaction survey with the States/Territories on the block grant activities. The survey supports service improvements and helps the Block Grant program to be more responsive to customer needs. Reliability and validity were assessed as part of survey development, and implementation, and were determined to be high.

An effective measure of the impact of technical assistance is positive changes that result and are maintained in those systems, programs or practices addressed during the course of the technical assistance activity. Selected measures have been included in a tracking system used with those receiving CSAT TA. The validity and quality of data were assessed in the survey design and development process and found to be high.

One of the statutory requirements for the SAPTBG is that states base their planning for the use of Block Grant funds on needs assessments within the state. Data are collected via the annual Block Grant Application System. A 1998 GAO report identified some problems with the completeness and accuracy of the data reported by the States, and recommended that CSAT develop a plan for making improvements. Validity of the data under this system is reviewed as part of the approval of funding and specific feedback provided to individual States. In addition, reviews of the data are done as part of a cyclical block grant compliance review process required by statute.

Previous Page Table Of ContentsNext Page