This chapter describes methods for evaluating both the process of developing your performance measurement system and the results of this process. These two overarching goals lead to two distinct types of evaluations of a performance measurement system: a process evaluation, and an impact evaluation. The process evaluation has as its overall goal to assess each step in the development and implementation of a performance measurement system in terms of inclusion, efficiency, resource expenditure, etc. The impact evaluation has as its overall goal to assess the degree to which the performance measurement system has achieved its goals, which may include things like assisting consumers in making informed health care choices, informing legislators in making resource allocation decisions, facilitating quality improvement within service provider agencies, etc.

As with any type of evaluation, the planning of an evaluation of a MHSIP Consumer-Oriented Mental Health Report Card project should begin early, concurrent with the planning of the other aspects of the project. The scope and content of the evaluation will, of course, vary with each project. Below we present a structure for thinking about the evaluation purposes, evaluation questions that may be relevant, and some methods that might be used to address such questions.

It is crucial to begin the evaluation process with a clear and shared understanding of the purpose(s) for the evaluation. The defined purposes should shape the scope and content of the evaluation and ensure that evaluation resources are allocated efficiently. We present and discuss several potential purposes of a process evaluation.

This is likely to be the overarching purpose of a process evaluation. This purpose builds on ideas found in concepts like continuous quality improvement (CQI) and formative evaluations (Rossi & Freeman, 1993).

Since stakeholder commitment is so central to a successful PM system, this should be a major goal of your process evaluation. Once again, it should be formative, providing constant feedback to the performance measurement system administrators so that fine tuning can take place before small problems become large. Within this goal, you may want to examine at least two aspects of stakeholder satisfaction: (1) Satisfaction with stakeholder involvement in and ability to influence the process, and (2) Satisfaction with the quality of products.

If performance measurement is based on the tenet that evaluative information can improve service delivery, then the performance measurement system itself should use evaluative information to improve its own process.

It may be useful to make this an explicit evaluation purpose to help remind yourself to take advantage of the thinking and work that has already been done.

Define the Evaluation questions.

The evaluation questions relevant to any particular process evaluation will vary depending on the characteristics of performance measurement system being developed. Nonetheless, below we suggest some questions that might be addressed in a process evaluation.

• Are stakeholders satisfied with their level of involvement in the process?

• Consumers/family members
• Providers
• Payers
• State agencies

• Is the development process efficient?

• Are there clear lines of responsibility for tasks?
• How much time does the process consume? (This should be examined for both research/evaluation staff and other stakeholder groups.)

• Technical assistance: Is sufficient technical assistance provided to persons and organizations required to participate in the performance measurement system? Is it provided to the right individuals/organizations? Is the TA appropriate for its intended audiences?
• Training: Is sufficient training provided? Is it provided to the right individuals/organizations? Is the level of training appropriate for its intended audiences? Are there appropriate mechanisms to ensure that training diffuses through the system?
• Barriers to implementation: What are the barriers encountered in attempting to implement the system? How are these addressed?
• Problem resolution: How effectively does the system resolve problems related to implementation? Are problems addressed at the most appropriate system level?
• Protocol Adherence: Are protocols that have been designed for the performance measurement system followed rigorously?

• Are samples drawn according to the sampling plan?
• Are data collected during the specified time period?
• Are data collected for all of the performance measures included in the planned system?
• Are protocols related to privacy, confidentiality and informed consent followed?

• Have the psychometric properties of the data been assessed? Are the data reliable and valid?
• Do the data meet acceptable standards of completeness, accuracy and timeliness?
• In those cases where data from extant data bases are required, have appropriate quality checks been conducted to ensure that these data are accurate and complete? Have the methods used to integrate data across multiple organizational levels and data sources been checked to ensure that these procedures produce accurate data?

• What resources are required from the central office/providers for:

• Training and TA
• Quality monitoring
• Data collection
• Data entry/management
• Data analysis/reporting

• What are the costs associated with the process of developing the performance measurement system?
• How much burden is placed on various stakeholder groups?
• What burden is placed on staff/consumers?

III. Impact Evaluation

Define The Purpose Of The Evaluation.

The purposes of an impact evaluation should follow from the specific goals of the performance measurement system as articulated early in the process of planning and development. The various purposes of a performance measurement system are described in Chapter 2. Phrased in terms of evaluation purposes, they are:

• Assessing the degree to which the performance measurement project effected changes in service delivery
• Assessing the degree to which the performance measurement project provided useful information to legislators and other policy makers for resource allocation decisions
• Evaluating the utility of information from the performance measurement system for contract monitoring and management
• Evaluating the extent to which consumers used the information produced in the performance measurement system in selecting plans, providers, and/or services

An additional purpose of the impact evaluation, not specifically related to the goals of the performance measurement system, is related to the appropriateness of performance measurements selected:

• Evaluating the association of performance measures with other, typically longer-term, measures of outcome or system effectiveness

This type of evaluation is a kind of convergent validity testing mentioned in Chapter 4. Sequentially, this type of evaluation necessarily follows data collection and analysis. Although not strictly an evaluation of the performance measurement system’s impact, this type of evaluation may shed light on whether some structure or process measures are good indicators. For example, Druss and Rosenheck (1997) evaluated an individual HEDIS measure of continuity of care, percentage of persons who received a follow-up appointment within 30 days of hospital discharge, by evaluating the correlation between this measure and future rehospitalization.

Again, the specific evaluation questions will follow from the intended impact of the performance measurement system. We suggest some more general questions that might apply to many impact evaluations.

• Do various stakeholders find the information generated by the performance measurement system meaningful and relevant?

• Does the information help consumers make choices about plans, providers, and services?

• Is the information produced culturally sensitive?

• Are the reports produced adequate?

• Does the content address the concerns of each stakeholder group?
• Is the format appropriate for the content and the level of expertise of various audiences/
• Are reports generated in a timely manner?

• Does the information result in measurable changes in the system with respect to:

• Program management and improvement
• Contracting
• Ensuring accountability for public funds
• Local planning
• State level planning
• Treatment planning
• System level policy changes

Below, we present a matrix that suggests methods for addressing the questions that might be the focus of an evaluation of a performance measurement system. The section also includes one or more references selected methods for readers who are interested in learning more about the specific methods. In some cases, we also refer the reader to other sections of this Toolkit where these methods are discussed.

We suggest two other strategies, not included in the matrix of methods, for evaluating your performance measurement system. The first is the use of case studies based on specific uses of performance measurement information. This type of qualitative description can provide a widely accessible report that highlights a particular outcome of the performance measurement system. For example, a case study could describe how an individual consumer used Report Card data to select a mental health plan.

The second additional strategy we propose is to conduct comparisons of systems with and without access to performance data. Both qualitative and quantitative comparisons of various aspects of system functioning might be appropriate, using a similar system without a performance measurement system in place as a "control" group. This strategy may highlight advantages of implementing a performance measurement system that are undetectable without an external comparison.

Table 6.1

Suggested Methods for Evaluating a Performance Measurement System, by Type of Evaluation Question

Krueger, R. A. (1988). Focus Groups. A Practical Guide for Applied Research. Newbury Park: Sage.

Morgan, D. L., Krueger, R. A. (1998). The Focus Group Kit. Thousand Oaks: Sage. Vols 1-6.

See Chapter 3

Guenzel, P.J., Berckmans, T. R., Cannell, C. F. (1983). "General Interviewing Techniques." Survey Research Center. Institute for Social Research. The University of Michigan.

Ralph, R. O. (1996). "Research Interviewer Handbook." Edmund S. Muskie School of Public Service. Portland, ME: University of Southern Maine. 1-10

Using Structured Interviewing Techniques. (1991) Transfer Paper 10.1.5. United States General Accounting Office. Program Evaluation and Methodology Division.

See Chapter 3

Salant, P. & Dillman, DA., (1994) How to conduct your own survey. London: Wiley.

Dillman, DA (1978). Mail and telephone surveys: the total design method. New York: Wiley.

See Chapter 4

See Chapters 2 and 4

Arminger G, Clogg CC, Sobel ME. (1995). Handbook of Statistical Modeling for the Social and Behavioral Sciences. New York: Plenum Press.

• Begin the evaluation process with a clear and shared understanding of the purpose(s) for the evaluation

• Consider both process and impact evaluations

• A major goal of the evaluation process should be to assess stakeholder satisfaction because of its importance to the success of the performance measurement system

• Use case studies based on specific uses of performance measurement information

• Conduct comparisons of systems with and without access to performance data