Chapter 1: Introduction

This toolkit has been developed to support users of the MHSIP Consumer-Oriented Mental Health Report Card. The report card itself is one in a line of products that have been developed as a part of the Mental Health Statistics Improvement Program (MHSIP). The purpose of this first chapter is to introduce the reader to MHSIP, to describe the development and design of the report card, to provide a comparison to other mental health report cards, and to describe the national status of report card implementation.

MHSIP began in 1976 as a collaboration between states and the National Institute of Mental Health to develop national data standards for use by state and local governments and individual mental health providers. The aim was to promote uniform collection and reporting of mental health statistical information through the voluntary adoption of data standards by mental health organizations. The first MHSIP product was <FN-8>, published in 1983. The concepts in that report were expanded and refined in 1989 with publication of <FN-10>, Data Standards for Mental Health Dec. Supp. Systems.

Over the past twenty years several additional MHSIP reports have been produced about MH data, its uses and its users. MHSIP has grown so that its mission is now well beyond the task of producing only uniform standards for data collection. MHSIP now carries multiple meanings, including the following:

• MHSIP now represents a set of values centered around the commitment to the use of statistical information for decision support in the mental health service system, and in the inclusion of consumers and other stakeholders in aspects of system processes.

• MHSIP supports an evolving set of guidelines for best practices for the development of mental health data systems, including common definitions of terms and measures and reporting that are accessible to all stakeholders.

• MHSIP is supported, in part by, a Federal funding stream, that includes set-aside funds from the Mental Health Block Grant; the funds have been employed to assist states’ efforts to improve their statistical systems and to initiate task forces that have produced data standards for MH data systems and best practice guidelines for performance indicator and report card development.

• Finally, MHSIP is an informal community of professionals, service recipients, and other advocates committed its goals.

Although MHSIP itself is just over 20 years old, it has roots that go back into the last century. The history of MHSIP is the history of federal reporting of mental health information, beginning with reporting on the numbers of persons in mental health institutions. These data were collected by the United States Census as far back as the 19^th century. When the National Institute of Mental Health was established in 1946, federal efforts to maintain mental health data were incorporated into its functions. Today that responsibility rests with the Survey and Analysis Branch of the national Center for Mental Health Services, a part of the Substance Abuse and Mental Health Services Administration.

The direction of MHSIP is determined, in part, by an ad hoc group that meets several times each year to review and work on current projects, and to initiate new ones. Once limited to federal and state mental health officials, the committee membership has diversified to include recipients of mental health services, local providers of mental health services, staff of the National Association of State Mental Health Program Directors, and officials of other federal agencies.

The MHSIP community meets annually at the National Conference on Mental Health Statistics, usually the week following Memorial Day Weekend. The National Conference--now in its 47th year--provides a forum for discussing major changes in federal and state policies and the direction of the public mental health system, as well as presentations of new ideas about best practices, workshops focussing on how to implement statistical methods, and reports on uses of mental health information.

The MHSIP Policy Group initiated the report card project in the Fall of 1993 with funding support from the national Center for Mental Health Services (SAMHSA, PHS, USDHHS). Anticipating the development of a national managed care report card under the Administration's plan for healthcare reform, the Committee recognized a need to begin planning its mental health component. As events developed, the primary audience for the report card has been individual states proceeding in their own healthcare reform efforts.

The first phase of work was the development of a conceptual definition of the report card. Under the leadership of John Hornik, a task force identified the major domains for evaluation of managed care plans providing mental health services. In May, 1994, at the National Conference on Mental Health Statistics, the task force presented the MHSIP community with a draft report for comment. With broad support the work proceeded into the next phase.

With new leadership in the person of Vijay Ganju, the Phase 2 task force proceeded to develop detailed operational definitions of performance indicators within each domain, including recommended measures and methods of data collection for each indicator. Like the first task force, this group included a wide array of participants including recipients of mental health services and family members, federal and state mental health and substance abuse officials, academic experts in services research, and providers of mental health services. Throughout its18 months of work, the task force formally and informally solicited the views of the groups from which its membership was drawn.

At the May, 1996, annual conference, the MHSIP Consumer-Oriented Mental Health Report Card received an enthusiastic reception from the MHSIP community. In fact, several states had already begun to implement their own versions of the report card, based upon earlier drafts that had been circulated. The strengths of the design included the following:

1. Values. The report card is based upon a well-articulated set of values for the public mental health system. These are represented as specific "concerns" within each of the major report card domains (Access, Appropriateness, Outcomes, Health Promotion/Prevention). Each concern identifies a significant goal of mental health services (e.g., Service recipients experience increased independent functioning). Twenty-six mental health plan goals provide the basis for rating performance; the largest number of goals (13) are in the outcomes domain.

2. Specificity. For each of the 26 concerns, there are between one and four performance indicators to assess how well the plan is performing in that goal area. Recommended measures are presented for each performance indicator. The measures include standardized instruments where the task force judged that these were appropriate and new measures in areas where there were no existing reliable measures.

3. Sources of Data. The design of the report card requires multiple sources of data to construct performance indicators. Most prominent among these is the report of service recipients themselves on their care and treatment outcomes. Clinician assessments, enrollment-encounter data, medical records, and plan financial information are also incorporated.

4. Population Sensitivity. The task force recognized the fact that different populations are served under managed care plans. While a primary focus was on assessing performance of plans serving persons with serious mental illness, the task force identified the subset of performance indicators that applied to other adults and children and to persons with substance abuse problems. It also identified specific performance indicators for children and adolescents with serious emotional disturbance.

Evaluation of mental health programs has a long history, and the MHSIP Consumer-Oriented Mental Health Report Card, like others to be discussed below, borrows liberally from that literature and experience in its design. Program evaluation efforts are typically limited to assessing outcomes for persons who are under treatment within a particular program. What is conceptually different about the MHSIP Consumer-Oriented Mental Health Report Card is that it is focussed, not on a particular person admitted to a particular program, but on the population of persons enrolled in a managed care mental health plan, a plan which will necessarily include a range of programs and services.

The first report card for a managed care plan was published by the Kaiser-Permanente Health Plan of Northern California in the early 1990s. It included two mental health performance indicators: rates of suicide for plan enrollees (in comparison to all persons residing in the same geographic area) and the rate at which persons receiving psychiatric inpatient care for major affective disorders had at least one outpatient visit within 30 days of discharge.

The first report card for a managed behavioral healthcare plan was published by United Behavioral Healthcare of Minneapolis. This report card had approximately a dozen indicators and was based upon data that the company already maintained in its management information system. This included the results of a client satisfaction survey undertaken on a sample of UBH enrollees who had received treatment.

In addition to the MHSIP project, two prominent report cards were developed by the National Committee on Quality Assurance (NCQA) and the American Behavioral Healthcare Association (AMBHA). Each is briefly described below.

NCQA is a national organization organized for the purpose of assessing the quality of care in health care plans. The board of the organization is principally composed of representatives from health care plans and from the major employers who purchase services from them. Beginning in 1992, NCQA began pilot testing its Health Employee Data Information System (HEDIS), a system which required each health plan to produce indicators of plan performance.

The original HEDIS and successive versions have included a limited set of mental health indicators. Among those included were measures of penetration and utilization for the categories of inpatient, ambulatory and day/night services. Rates were further broken down by age and gender groupings, as well as by payor/employer. Readmission rates for inpatient care were also incorporated. Subsequent versions of HEDIS have included a small number of additional indicators in a testing data set, and the addition of a satisfaction survey is also contemplated. This survey may be based upon the MHSIP Consumer Survey or a survey under development as part of a larger health plan evaluation project, the Consumer Assessment of Health Plans Study (CAHPS). Specific HEDIS reporting requirements are issued each year, but changes to behavioral health indicators are typically made less frequently.

In contrast to NCQA, AMBHA is an association of organizations that provide managed behavioral healthcare, not general healthcare (although some of its members are subsidiaries of organizations that offer managed healthcare). Most of the largest such organizations are included in its membership. At the time that its report card project, PERMS, was undertaken, AMBHA members covered over 120 million lives for behavioral healthcare services. The PERMS project was announced by AMBHA in the fall of 1995, and data collection from its members proceeded over the next year. The plan was to produce an industry-wide performance report without identifying differences in performance among companies.

PERMS has a larger number of indicators. Like HEDIS, it includes utilization rates, although breakdown categories include diagnosis and clinician type, as well as age, gender, and treatment setting; and rehospitalization rates within 30 days with and without outpatient follow-up. In addition, PERMS includes expenditure rates by treatment settings, measures of telephone responsiveness, medication management visits for persons with a diagnosis of schizophrenia, and family visits for children under 12. Finally, PERMS incorporates six indicators based upon a consumer satisfaction survey. An updated version of the AMBHA PERMS, version 2.0, was released this summer.

There are several differences between the MHSIP, PERMS, and HEDIS report card designs.

• MHSIP is the most extensive with respect to mental health performance indicators. HEDIS and PERMS were both developed with the limitation that the indicators could be produced with little or no modification of existing data sets among member organizations. As noted earlier, the MHSIP design proceeded from a set of values articulated as formal goals and concerns within the public mental health system.
• The MHSIP design incorporates extensive outcome indicators, while the others do not; all three include indicators of access to and appropriateness of care.
• In the MHSIP design, consumer reports are the most important source of data. PERMS includes a few indicators based upon consumer reports. HEDIS is in the process of incorporating indicators based upon such measures.
• In the MHSIP design, the focus on persons who have serious mental illness is central; HEDIS and PERMS have few indicators that are appropriate for this population.
• The MHSIP design assumes a broad and flexible mental health benefit; in HEDIS and PERMS assumptions about what services are available to the insured population are unclear, but appear to be limited.

It has been assumed that the MHSIP design is the least feasible because it is the most extensive. In fact, both NCQA and AMBHA have experienced very significant problems in obtaining complete data from their member organizations. Despite the MHSIP Consumer-Oriented Mental Health Report Card’s demands, it has gained relatively wide use in public mental health systems.

The MHSIP Consumer-Oriented Mental Health Report Card has already had a wide influence upon state mental health authorities. In the wake of health care reform and the adoption of managed care approaches to public mental health services, approximately 30 states are in the process of adopting or adapting the MHSIP design in part or in whole. The most frequent adoption is of the MHSIP consumer survey, although it is not unusual for states to make local modifications or additions to this instrument. Fewer states have attempted to implement the indicators of the Report Card based on data other than the Consumer Survey. At least one state, New Mexico, did adopt the entire Report Card and did substantial work towards fine-tuning the non-survey indicators. Some states (e.g., Delaware, New York) have also replicated the MHSIP Consumer-Oriented Mental Health Report Card development process, convening stakeholder groups for discussion of values and concerns in the public mental health system, as well as review of proposed performance indicators and measures.

To date, the primary resource for MHSIP Consumer-Oriented Mental Health Report Card users has been the final Task Force report released in April 1996. While this document outlined the domains, indicators and measures necessary to implement the Report Card, the document did not provide guidance on a host of practical decisions that are required in implementing a performance measurement system. Because of this, a number of persons have turned to members of the MHSIP ad hoc Group and to the Evaluation Center@HSRI for technical assistance in implementing the Report Card. To avoid resource-intensive consultations to individuals, the MHSIP ad hoc Group agreed for the Evaluation Center@HSRI to develop a Toolkit for Performance Measurement Using the MHSIP Consumer-Oriented Mental Health Report Card that would provide guidance in a user-friendly, how-to format. The Evaluation Center has worked with a group of experts with experience implementing the Report Card to produce this first version of the Toolkit.

The Toolkit is organized into chapters based on major steps in Report Card implementation. While these major steps are presented in what is most often chronological order, we suggest that users read the entire Toolkit first, and then go back to focus on specific issues in turn. This will help you understand the overall dimensions of a Report Card project before getting bogged down in details. And, reading about steps that occur later in the process may affect your thinking in the early stages.

Wherever possible, we have built on the concrete experience of persons who have implemented the Report Card. In addition, we have tried to identify resources for more detailed information on topics that we address only briefly. Each chapter includes a number of recommendations. These recommendations represent the thinking of one or more of the contributors, and in many cases reflect a group discussion and consensus. For quick reference, we have pulled out the major recommendations within each chapter and displayed these at the close of the chapter.

Finally, we envision this Toolkit as a dynamic resource that will be modified as we learn more about Report Card implementation and its various component steps. Thus, we would like readers to comment on this and future versions in terms of what might improve the Toolkit, particularly including ideas or resources that you may have from your own experience. Please forward your comments to Lawrence Woocher at 617-876-0426 x326 or woocher@hsri.org.