Evaluation: MHTP Implementation & Outcomes
Large, multi-component initiatives, such as the Mental Health Transformation Project, are notoriously difficult to evaluate, because of their complexity, the number of proposed pathways from activities to outcomes, and the seemingly infinite number of evaluation activities that could be undertaken.. In the face of these challenges, a common approach to evaluation is to develop a program theory that is used to identify and prioritize the important evaluation questions, and to use multiple methods to answer those questions.
The Mental Health Transformation Project (MHTP) Evaluation Team has partnered with MHTP staff to create a logic model for the MHTP that is used to help explain the goals of the project, the strategies to achieve these goals, and the ultimate outcomes that are proposed to occur as a result. Using this theory-driven approach, several approaches have been developed to evaluate the success of the MHTP.
First, in order to document the success of MHTP implementation, the MHTP staff creates a regular report of strategies and activities, organized by the areas represented in the New Freedom Commission Goals and two additional Washington State goals [
19KB]. These reports, and analysis of their findings, are intended to answer some simple questions:
- What activities are being attempted to "transform" the system?
- Are they being attempted successfully?
- Do they align with stated goals, priorities, and outcomes?
In addition to presenting detail on activities and strategies being used to achieve the MHTP goals, these reports can also present details on the populations of interest being targeted, the type of strategy being employed (e.g., training, policy change, new programming), and the level of funding that has been obtained or leveraged toward the strategy. Regular analysis of the match between proposed MHTP goals and activities can illuminate whether certain target populations or goal areas are being underemphasized in the MHTP, and what holes may need to be filled.
A second major application of the theory-driven evaluation approach is to highlight the proposed interim and ultimate outcomes of a project or initiative. With respect to the MHTP, the Evaluation Team is working with the state’s Division of Research and Data Analysis (RDA), and its integrated database, to elevate the level of monitoring of the proposed outcomes of Transformation, such as:
- Decreased number of individuals with mental illness in the criminal justice system
- Decreased utilization / reduced readmission to psychiatric hospitals
- Increased number of persons served by age, gender, race, and ethnicity
- Increased / retained jobs
- Decreased number of children in out-of-home placements (i.e., juvenile justice residential centers, foster homes, group homes, psychiatric hospitals)
- Increased school enrollment/attendance
- Increased utilization of evidence based practices
- Reduced substance use-related incidents (e.g., emergency room use, arrests, jail bookings)
As of May 2008, the Evaluation Team has initiated its outcomes tracking project by working with the first two outcomes listed above (recidivism of individuals with mental illness into criminal justice and readmission into psychiatric hospitals). A first look at historical trends for these two outcomes is expected by October 2008.
By using a theory of change approach, the Evaluation Team is attempting to make the assumptions behind the MHTP's work more clear and highlight elements of program activity that deserve attention in the evaluation. The approach is intended to facilitate a useful overall evaluation process that can evaluate success in setting priorities, implementing strategies, and improving outcomes for consumers, youth, and families.
