Ronald W. Manderscheid, Ph.D.

U.S. Center for Mental Health Services

Accountability efforts in mental health are relatively recent. Generally, they do not date back more than a decade. The foundation for these efforts can be located in the demise of national healthcare reform, the advent of managed behavioral healthcare in both the public and private sectors, and the continued erosion of resources for behavioral healthcare.

President Clinton’s proposed Health Security Act included specific provisions for the development of health report cards. Such report cards were seen as a vehicle both for enhancing consumer choice among health plans and for promoting competition and accountability. Although the Health Security Act was never passed, the concept of health report cards gained broad acceptance in corporate and governmental circles. The concepts of competition and accountability in the health arena soon pervaded mental health as well.

The advent of managed behavioral healthcare in both the public and private sectors also fostered a new concern with responsiveness to customers. In the private sector, this took the form of satisfaction surveys about plan performance; in the public sector, it took the form of surveys about consumer expectations and problems with plans. These efforts were promoted by a broad-based consumerism in American society and a well-defined consumer movement in public sector mental health.

Beginning at the time of the national health-care reform debate, and continuing to the present, a dramatic decline has been witnessed in the available resources for mental health. Commodification—the progressive transformation of healthcare services into commodities, like corn or crude oil, which are subject to market forces—has led to dramatically lower prices for mental health services. according to the most recent Hay Group report, expenditures on behavioral health benefits decreased 54 percent, from 1988 to 1998, while expenditures on general health benefits saw a modest decrease of 11.5 percent during this time frame (Hay Group, 1999). Elsewhere (Manderscheid, 1998), we have argued that this decline is due to the lack of accountability tools such as practice guidelines, outcome assessments, report cards, and performance indicators available in the mental health field. Without such tools, those negotiating managed care contracts have no basis for "competition based on quality" as opposed to "competition based on cost."

The confluence of all these factors has promoted strong efforts at the national level to develop accountability tools.

In the past, accountability generally referred to financial responsibility; it referred to quality responsibility much less frequently. Further, quality accountability means different things to different people, since it is true that where you sit determines what you see. One can distinguish four different types of accountability.

These different levels of accountability are inter-related. Practice and outcome measures can be aggregated and included in report cards. Report cards can be aggregated and included in broader system performance measures. One always needs to be aware of what aggregations are being incorporated, and the point of view of the intended accountability, to judge potential bias. Report cards and performance indicators are discussed further below.

Like their academic counterparts, report cards for mental health and substance abuse services are intended to provide feedback on achievements and problems. At least three dimensions must be considered when discussing report cards: content, point of view, and intended audience.

Content refers to the topics that are addressed. In school, this would be the courses rated. Generally, report cards for behavioral healthcare services cover one or more of the following domains of care: access, appropriateness, cost, and outcome. The two most common dimensions covered are access and cost.

Point of view refers to the perspective taken. In school, the perspective is that of the teacher. In a behavioral service setting, the perspective might be that of the payer, the managed care entity, the provider, the consumer, or family member. Most frequently, the point of view is that of the managed care entity.

Intended audience can be both explicit and implicit. In schools, the explicit audience of a report card is the parent; an implicit audience might be a future employer. In behavioral service settings, the explicit audience could be the payer, the managed care entity, the provider, the consumer or family member. Most frequently, the explicit audience of a behavioral service report card is the payer; the implicit audience is frequently the media.

In most dialogues about behavioral service report cards, domains are discussed, while point of view and intended audience are ignored.

In 1994, the U.S. Center for Mental Health Services (CMHS) convened key mental health policy leaders at the Carter Center in Atlanta GA, to ask them whether CMHS and the Mental Health Statistics Improvement Program (MHSIP) should undertake the development of a report card for the mental health field, and if so, what should be the point of view. The group responded with a strong affirmation that development effort should be undertaken and that the point of view should be that of the consumer. Subsequently, a task force of technical persons and consumers met on several occasions over about a one-year period to develop the prototype. This prototype was released to the field at a public meeting in April 1996. Later in 1996, CMHS awarded 20 grants to states to begin a pilot test of the report card. In 1997 and 1998, the number of state grants was increased to 40.

The MHSIP report card covers the domains of access, appropriateness, prevention, and out-come. The point of view is that of the consumer; the explicit audience is the healthcare plan, and the implicit audiences are consumers and family members.

The report card consists of two components: administrative data and a consumer survey. The logic is that the administrative data will be drawn from the health plans’ information systems, whereas consumer survey information will be collected during care and after the conclusion of an episode of care. Because the report card is consumer oriented, a major consideration in implementation is to have consumers collect and process report card data. Such report card data could also be supplemented by focus groups of consumers who could aid in interpreting the results.

In the winter of 1999-2000, the MHSIP program will undertake a revision of the report card to incorporate findings that have emerged from the field through the 40 State grants, as well as from other applications being tested. The second generation of the report card will be simplified and more user-friendly.

Report Cards will be a vehicle not only for displaying outcomes, but also for showing contributions (such as, population prevention activities) to the community more generally. Hence, future report cards will need to address not only service-delivery questions, but also population questions. CMHS is currently beginning work on the development of a population-based report card. Results from this project should become available in about 18 months.

Factors both internal and external to the field have led to major indicators. Generally, these performance indicators are designed for large-scale systems and operations. Such large-scale systems can be states, corporations, or national efforts around a particular issue. The point of view can be a legislature, corporate stockholders, the U.S. Congress, or other mental health or substance abuse communities. Indicators selected will depend on the point of view.

External factors influencing the development of performance indicators include efforts to make government accountable, such as the Federal Government Performance and Results Act (GPRA, 1993). This Federal legislation requires that each Federal program have performance indicators in place by Fiscal Year 1999. Some states have similar systems, for example, the State of Texas. Internal factors related to development of performance indicators for behavioral healthcare include declining available resources as discussed above, the transformation of Block Grants into Performance Partnerships, and the call for more accountability from all parts of the field, principally from consumers and family members.

Much of the work on performance indicators in behavioral healthcare can trace its intellectual lineage to the MHSIP Consumer-Oriented Mental Health Report Card. This report card, as discussed above, measures four domains: access, appropriateness, outcomes, and prevention. These domains have provided the initial framework of the National Association of State Mental Health Program Directors Framework for Performance Indicators (NASMHPD, 1998). Likewise, they have provided the initial framework for the indicators developed by the American College of Mental Health Administration (ACMHA, 1998), as well as the work of the National Association of Psychiatric Health Systems and the Association of Behavioral Group Practices (NAPHS & ABSP, 1999). In each of the latter three instances, additional domains have been added. For example, the NASMHPD President’s Task Force on Performance Indicators added a Structure and Management Domain.

Initially, the CMHS work on performance indicators involved a Five-State Feasibility Study testing 28 performance indicators over a nine-month period. Five state mental health agencies collected the data and reported on 28 selected indicators in 1998 (See Table 1 p. 329). Because the initial work showed that it was possible to collect and report on those indicators, CMHS undertook a larger project late in 1998 with 16 states. Input for this pilot project derives from the initial Five-State Feasibility Study and from the NASMHPD President’s Task Force. The net effect is that the pilot project will test 34 indicators, rather than the original 28, although almost all of the original 28 indicators have been included. The performance indicator project is slated for completion at the end of Fiscal Year 2001.

In 1998, CMHS funded a Benchmarking Indicators Survey for the National Association of Psychiatric Health Systems (NAPHS) and the Association of Behavioral Group Practices (ARGP). The purpose of this study was to identify a set of performance measures being widely used in behavioral healthcare settings and determine their feasibility for national implementation. The study consisted of a series of meetings, a literature review, and a mail survey. The domains of measurement included health status, client perception of care, coordination of care, clinical performance, family involvement in child and adolescent treatment planning, and peer review.

Among other findings, the results of the survey (Dewan, et al., 1999) indicate that all levels of care measure performance in multi-dimensional categories; measures of clinical performance and perception of care are most commonly used; and for most measures, definitions were consistent across facilities.

Both the public and private sector work underway to develop common performance indicators has great potential. The prognosis is good for the future because of the positive collaborative relationship that has developed around these endeavors. In the future, we expect that such systems will be operated through web-based technology with both plan and geographic-based reports available.

1. Manderscheid, R. From many into one: Addressing the crisis of quality in managed behavioral health care at the Millennium. Journal of Behavioral Health Services & Research, vol 25(2): pp233-236, 1998.
2. Manderscheid, R. A brief report on report cards. Connection: pp. 6 & 8, 1999.
3. Allness, D. & Knoedler, W. The PACT model of community-based treatment for persons with severe and persistent mental illnesses: A manual for PACT start-up. Arlington, VA: National Alliance for the Mentally Ill (NAMI), 1998.
4. Mental Health Statistics Improvement Program (MHSIP) Report Card Phase II Task Force. The MHSIP Consumer-Oriented Mental Health Report Card. Rockville, MD: Center for Mental Health Services, 1996.
5. Manderscheid, R. Untangling the accountability maze: Developing outcome measures, report cards and performance indicators. Managed Behavioral Health News, April 15, 1999. Pp. 6-7, 1999.
6. Dewan et al. Benchmarking Indicators Survey Report. Washington, DC: National Association of Psychiatric Health Systems (NAPHS), 1999.
7. National Association of State Mental Health Program Directors Research Institute, Inc., (NASMHPD-NRI). Five State Feasibility Study on State Mental Health Agency Performance Measures. Draft final report, June 1998. Prepared for U.S. Center for Mental Health Services. Alexandria, VA: NASMHPD-NRI, 1998.

Taken from:

2000 Behavioral Outcomes & Guidelines Sourcebook

Chapter 7: Using Report Cards and Indicators To Evaluate Performance