This chapter describes the major steps in planning and organizing your performance measurement system. A common theme throughout the process described here will be the importance of using a "multi-stakeholder approach" (Koch, Lewis, & McCall, 1998) that maximizes collaboration among all constituencies.

The individual steps will be presented in typical chronological order. However, planning and organizing a performance measurement system will not always follow a clearly linear process. Many of the steps will occur simultaneously and others will be repeated several times. In addition, the order of the steps is often influenced by which stakeholder (e.g., state mental health authority (SMHA), state legislature, consumer group) initiates the effort.

Finally, it should be noted that this chapter is written from the perspective of an SMHA planning to implement a performance measurement system statewide, or at least in multiple sub-state areas. However, the same basic steps and guidance are applicable to a private healthcare plan, HMO, or an individual provider organization attempting to establish its own performance measurement system.

Implementation of a performance measurement system may significantly change the operations of a public mental health system. These changes can range from an increase in the availability of information useful for management and quality improvement to the use of performance data for incentives and sanctions of providers. When performance results are made public, they subject the mental health system to greater public scrutiny, and they may change the balance of power between providers and consumers and family members. Implementation of a performance measurement system may also cause significant changes in staff roles at both the state and local levels which may be viewed as threatening. As a result, like all attempts at system reform, resistance is likely to be encountered from key constituencies. The potential benefits of performance measurement for all involved, however, may act to lessen resistance, particularly if potential benefits are effectively communicated.

A successful performance measurement system also requires the investment of substantial organizational resources (Five State Feasibility Study, 1998). As described later in this chapter, implementation of a performance measurement system may require additional resources for information technology and staff to collect, analyze and report performance data. Thus, implementation of a performance measurement system typically requires obtaining new resources and/or reallocating existing resources.

Given the above, prior to beginning the development of a performance measurement system, you must obtain a commitment from those persons/groups who have the authority to allocate resources and the resolve to effectively overcome resistance. Their initial and ongoing support will be critical to the success of your efforts.

This section will describe the critical leaders who must make the commitment and the types of commitments they must make.

Ultimately, the success of your performance measurement system will depend on the support and cooperation of a long list of stakeholders. However, there is a "short list" of stakeholders who are critical to this effort. If their active support is obtained, they have the ability to move the performance measurement system forward. These persons are the ones who control resources and provide leadership and credibility for the system.

• Commissioner and/or top-level administrators. For public mental health system a key person(s) from whom a commitment must be obtained is the commissioner of mental health or other top-level administrators. These such persons have the authority to allocate the existing resources (both dollars and staff) that are needed for the performance measurement system, and they can provide the leadership necessary to obtain support from other stakeholders. Commissioners are generally very receptive to performance measurement systems because such systems provide them with much of the information they need to effectively manage the operation of the public mental health system and advocate for additional resources to enhance services.
  

• Consumers. If a Report Card project has no meaningful stakeholder involvement, particularly involvement of consumers, the project will lack credibility among other critical sources of commitment such as a commissioner. If a project lacks this type of credibility, you will be much less likely to garner support from a commissioner, state legislators, and other system stakeholders.

• Service providers. In most performance measurement systems, providers are critical in supplying data in the form of clinician ratings, administration of surveys, and/or in keeping high quality records (e.g., claims) that can then be used for performance measurement purposes. Since the quality of data is of paramount importance and collecting high quality data will burden providers to some extent, it is important to obtain the commitment of providers and to ideally establish collaborative relationships with them.

• State legislature. Particularly when additional resources are needed to implement the performance measurement system, support from state legislature is critical. Their support is also critical when implementation requires changes in state law (e.g., to permit sharing data across state agencies or between local providers and the SMHA). State legislatures are typically receptive to performance measurement systems because they are viewed as a means of increasing accountability for the taxpayer dollars they allocate to fund services.

To draw on the full range of expertise needed to design and implement an effective performance measurement system and to develop "buy-in" from those who must support this effort, a steering committee should be established that includes representatives from all major stakeholders. This committee should be established at the very beginning of the process to ensure stakeholder input on all critical decisions. This will also help to prevent inaccurate information from being spread among key stakeholder groups that may increase resistance to your efforts.

Several major decisions must be made regarding the steering committee: (1) the role of the committee, (2) its composition and (3) organizational structure. Each of these is discussed below.

The steering committee may play several roles in the development and implementation of your performance measurement system. Several recommended roles are described below, but the adoption of other roles will be influenced by the characteristics and organization of your state mental health system and the desires of the steering committee itself.

Advisory vs. decision making.

The first decision to be made regarding the steering committee is whether it is to be "advisory" or if it is to be empowered to make decisions. There are advantages and disadvantages to each role, and authority may be given to the committee to make certain decisions and not others. The approach that you take will be determined by a number of factors including the relationship between the SMHA and various stakeholders (e.g., does the SMHA contract for community services or does it directly operate local service providers), the current political climate, and the leadership style of the SMHA and its commissioner.

Generally, providing the steering committee with as much decision making authority as possible will result in greater commitment to the success of the system, greater participation, and more willingness to help "sell" the system to others. In any case, it is most important that this aspect of the role of the steering committee be stated clearly when making invitations to serve on the committee. Lack of clarity about limitations in the authority of the committee will lead to frustration and conflict.

Liaison to key stakeholders.

Perhaps most importantly, steering committee members should serve as liaisons between the performance measurement project and key stakeholder groups. In this role, committee members should represent their constituencies at committee meetings, assuring that their unique perspectives are represented in all discussions and decisions. Similarly, committee members must be responsible for sharing the results of meetings and the rationale for particular decisions with their constituencies--they must keep them informed. This latter role is particularly important in preventing inaccurate information and misperceptions from spreading--problems that can create tremendous obstacles to be overcome in generating support for your effort. One mechanism to ensure that the liaison role has been played is for steering committee members to report to the committee names and affiliations of people with whom they have discussed the project.

Define the purpose of the performance measurement system.

As described in the next section, performance measurement systems may serve several purposes (e.g., accountability, quality improvement, and management), and the purposes the system is to serve will largely determine its design and other major decisions. Given this, it is important that the steering committee have input into defining the purpose(s) of the system. Even when the primary purpose of the performance measurement system is externally mandated (e.g., a legislative mandate for performance-based accountability), the steering committee may want to identify other purposes the system is to serve.

Assist in the design of the performance measurement system.

Designing a performance measurement system is a highly complex task requiring a large number of technical decisions regarding such issues as protocols for data collection, management, and analysis; selection of standardized instruments; and development of software applications. How you choose to involve the steering committee in these activities will have to be determined. For example, you may want to establish technical sub-committees or work groups to address these activities, or you may want to have SMHA technical staff do the design work and submit it to the steering committee for review and comment.

Using sub-committees for the technical tasks works well when at least several members have the needed skills and you have more time to design your system. When neither of these is true, you should probably have SMHA evaluation and data processing staff assume major responsibility for designing the performance measurement system and submit draft materials to the steering committee. Even in this role the steering committee can be invaluable in identifying practical and political problems that must be addressed.

Marketing the performance measurement system.

Throughout the development process, but particularly when implementation begins, the steering committee should play an active role in "selling" the performance measurement system to their constituencies and others. Many stakeholders (e.g., associations of private providers and consumer and family member organizations) also have political ties and will be free to lobby state legislators for funds to support the system. If the process has gone well and committee members feel that they had a hand in shaping the system and they now "own" it, they will usually promote it among their constituencies.

Once the major roles of the steering committee have been defined, its composition must be determined. In general, it will be necessary to ensure that all major stakeholders are represented. Given that performance measurement systems are generally intended to have a far- reaching impact on the policies and operation of the mental health system, nearly all stakeholders of the public mental health system need to be represented--it should not be limited to technical staff.

Generally, it is important to include policy makers and those who control the resources needed to make the performance measurement system work, those who must implement components of the system, those who will be directly affected by the system, and those who have the technical skills required to design an effective system. Given this, listed below are the key stakeholders who should be represented on the steering committee:

• Consumer organizations

• public
• private

• State mental health authority

• commissioner
• SMHA management staff

• Technical staff (in house or consultant)

• research and evaluation
• data processing/information technology

Depending on the organization and structure of your mental health system and state politics, you may also want to include representation from several other stakeholder groups. For example, in county-based systems in which public providers are part of local government, you may want to include local government officials. In states where legislative support is critical to the success of the system (e.g., new funds are needed to support its operation), strong consideration should be given to including a key legislator or legislative staff members. Where the performance measurement system will result in new requirements for managed care organizations, these should be represented. And, finally, if it is expected that other state agencies (e.g., Medicaid, education, criminal justice) will serve as the source of performance data, you may want to consider inviting their participation as well.

Finally, much thought should be given to the selection of particular individuals to represent the various stakeholder groups. While it is important to have a committee whose membership generally supports the goal of establishing a performance measurement system, selecting only enthusiastic advocates of performance measurement does have its disadvantages. First, it is important to hear the views and concerns of those who do not entirely support such efforts. They are likely to raise important questions that must be addressed if the system is to be effective--questions that supporters might not think to raise. Second, when involving only those who support the performance measurement system, their mutual enthusiasm can result in the design of a system that is unrealistically large and complex. Therefore, it is advisable to include some "skeptics" on the steering committee who will add an element of realism to the endeavor. If their concerns can be effectively addressed, it is more likely that the system will be successful when implemented in the field.

Once the roles and composition of the steering committee have been defined, you will need to determine how the committee will be structured. Generally, given the need to include policy makers and advocates, you will find that the full steering committee is not the place for detailed discussions about technical issues related to measurement, data processing, and so forth. Such discussions in the full committee are a sure way of discouraging continued participation by many members. Therefore, it is usually desirable to establish one or more technical work groups under the steering committee to do the detail work of reviewing and selecting measures, designing data collection protocols, etc. When possible, it is still desirable to include representation from multiple stakeholders on these work groups, but these may be technical staff from the stakeholder groups rather than the policy-level staff who serve on the steering committee.

Given a structure in which there is a steering committee with multiple technical work groups reporting to it, the steering committee would establish overall direction for the effort, define principles for the development of the system (e.g., stakeholder involvement and consensus), and establish policies for the system (e.g., policies regarding the use of performance data and the protection of consumer confidentiality). The technical work groups would address the technical requirements of the system to ensure that it produces reliable and valid information that addresses the concerns and priorities of the steering committee.

An active steering committee that represents all key stakeholders is critical to the success of your performance measurement system. Their input will result in a higher quality system having stood the test of addressing the concerns raised by persons with different perspectives. In addition, their active support will be a major factor in moving the system off the drawing board and into the field.

A major factor determining the design of a performance measurement system (including content, reporting formats and procedures, etc.) is its purpose; that is, how the information will be used. There are several potential major purposes for performance measurement systems; each is described below.

This is a major use by purchasers of mental health and substance abuse services, be they federal, state, local or private organizations. Performance measures will help hold providers, MCOs and state or local mental health authorities accountable for achieving desired system processes and outcomes. Persons or organizations filling this role will require information from a Performance Measurement system that will point to inadequacies in systems of care, ideally in such a way that links the inadequacy to the person or organization responsible for this function.

The main purpose of performance measurement in quality improvement is to identify areas for potential improvement and monitor progress in remedying the identified problems. Thus, the most useful information within a quality improvement context is "actionable" information. In addition, the prevailing approach to quality improvement today fosters partnership between management and providers and shies away from punitive actions. The application of sanctions based on performance data makes the dissemination of sometimes potentially damaging data a major concern for providers. If quality improvement is the primary use of performance improvement data, a cooperative environment must be established and other possible uses of the data need to recognize this relationship.

In places where the public mental health system is moving towards a truly competitive market with multiple MCOs or service providers competing to enroll consumers in their plans or programs, results of a performance measurement system can be critical in ensuring consumers have the tools to make informed choices. If this is anticipated as a primary use of the performance measurement system results, care should be taken to ensure that the data are presented in a consumer-friendly format and should include some global measures to allow for simple digestion of the information. It is even more important to include indicators judged to be important to consumers.

Policy makers and program planners design programs to address specific unmet needs and solve identified problems. If a performance measurement project can highlight specific unmet needs and problem areas within a service system, this information will be useful in policy and program planning. Planners can use this information to inform decisions about organization of services, selection of particular service types, and focusing attention on existing problems.

Managers of mental health services are likely to be most interested in relative cost-effectiveness. Cost effectiveness is a concept that combines information on the degree to which an intervention, or in this case a service delivery organization, leads to desired outcomes, and the cost of the services leading to those outcomes. Cost effectiveness analysis, thus, requires good measures of outcomes as well as costs.

The public sector is experiencing a trend towards performance contracts, in which managed care organizations and/or provider groups are paid based on meeting certain performance standards. Reimbursement mechanisms vary widely, but can include bonuses for exceptional performance and financial penalties for poor performance. The performance standards within contracts are just one particular use of performance indicators more generally. If managing contracts is anticipated as a major use of your performance measurement system, it suggests certain qualities to look for in selecting performance indicators and measures.

For effective contract management, performance measurements need to be:

1. Specifically quantifiable and unambiguous
2. Obtainable in short time periods
3. Clearly directional, so that "more is better" or "more is worse"

Short of performance contracting per se, a performance measurement system can provide important information for purchasers in negotiating contracts with providers. Kamis-Gould and Hadley (1996) discuss ways in which state mental health authorities might use performance measurement systems in contracting with providers to increase overall system efficiency.

Closely related to accountability, monitoring system change may be an important goal of a performance measurement system. When this is a primary purpose, it is critically important to consider the timing of the performance measurement system implementation with respect to the system changes of interest. If the performance measurement system is only put into place after the system changes of interest, it may be in your interest to select indicators and measures for which baseline data will be available so that post-system change measures can be evaluated against a benchmark of prior system functioning.

While not typically the primary use of a performance measurement system, the collection and analysis of data for a performance measurement system can contribute greatly to the scientific understanding of services and outcomes. Certain conditions must be approximated when this is goal. First, a greater emphasis would need to be placed on documenting the intervention whose associated outcomes would be measured; i.e., the independent variable must be well defined. In some cases, this independent variable may be an organizational or financing arrangement, while in others it may be a particular service modality. Second, given a detailed description of the intervention, a comprehensive implementation analysis would need to be conducted to assure that the intervention under study was actually put in place as planned. Finally, information gained from performance measurement system results is most easily accepted as scientific knowledge if the data were collected within a quasi-experimental or experimental design; i.e., there is a comparison group to compare with the group receiving the intervention of interest.

Most performance measurement systems will have more than one purpose. We suggest that you make primary and secondary purposes explicit early in the planning process. Articulating clearly the purposes of a performance measurement system will help the purposes inform future decisions in a more concrete way.

Another factor greatly affecting the design of a performance measurement system is its "scope" or breadth of coverage. Limiting the scope of your system is one of the easiest ways to reduce its cost, if this is a major consideration. In general, the greater the scope of the performance measurement system, the greater the complexity and the greater the resources required. It is often advisable to initially limit the scope of implementation, focusing on the highest priority areas or populations, and then expanding when the system is operating effectively. Several dimensions of scope are identified and discussed below.

One dimension of scope is the target population for your performance measurement system. For example, you will need to decide if your system will address children and/or adults, persons with the greatest impairment (e.g., children with serious emotional disturbances (SED) and adults with serious mental illness (SMI)) or all consumers, and so forth. In general, this decision must balance the resources available with the purposes of the system and the need for information.

The complexity of the performance measurement system and resources required increase almost exponentially as the diversity of target populations increase. Expanding the scope of the system to additional populations does not merely increase the number of consumers about whom data must be obtained. Often, it requires the addition of a new set of indicators, measures, and data collection protocols. For example, treatment goals and, therefore the outcomes to be measured, will be very different between persons with SMI and a less severely disabled population. Further, measures that are sensitive to change for persons with SMI are likely to exhibit a "ceiling effect" for a less severely disturbed population (i.e., nearly all consumers with less severe mental health problems would score at the highest level of functioning or lowest level of disability). Finally, rates of change differ for different populations and problems. In contrast, change is very slow for persons with SMI, and an appropriate assessment schedule may be every six months for this population. Measuring changes that take longer to occur require longitudinal designs and consequently more complex measurement systems.

Given the complexity and resource requirements added by including different populations, you should be very careful in deciding which populations to include in your system. However, there may also be pressure to include many if not all populations in your system. Advocates for various populations (e.g., children, homeless, etc.) may fear that excluding "their population" will allow them to be ignored, particularly when it comes to obtaining new resources for services. They may also be anxious for an opportunity to "prove" that the services for these populations are effective, or they may desire the information the system will provide to better manage services. All of these are valid concerns and good reasons to include all populations.

In deciding which populations to include, you must be realistic about the resources you have available and your ability to implement and manage the performance measurement system. If limits do need to be placed on the number of populations to be included, several factors should be considered in selecting the target populations. First, many states have identified "priority populations" for state-funded services. Particularly when the primary purpose of the system is to improve accountability for state funds, the priority populations should be included in the system. Second, you may want to focus your efforts on those populations that consume the greatest amount of resources. These may be the populations with the highest average cost per consumer or the populations that, as a group, consume the greatest percentage of resources. Particularly if a purpose of your system is to improve efficiency, this is where you will get the "biggest bang for the buck."

Program types can be identified in several different ways. For example, you could distinguish between community-based versus state hospital services, or you could target a specific type of community service such as case management, residential, or employment services. As in selecting a particular population, selecting one or more specific program types may be an effective way of reducing the cost and complexity of your system, and it would allow you to focus on a particularly high priority program. As in selecting a particular population, one factor that should be considered in selecting the target program is cost. That is, you may want to select the most resource intensive or costly programs. You may also want to select those programs where there are the greatest concerns about efficiency or effectiveness.

One potential advantage of selecting specific program types, is that it provides an opportunity to tailor the performance and consumer outcome measures to the specific goals of the program. Typically, performance measurement systems try to assess a broad range of outcomes, but due to a need to not overburden providers and consumers, do not go into great depth on any one dimension of outcome. However, if the focus of the performance measurement system is an employment program, for example, you could assess multiple facets of employment outcomes and provide a more comprehensive assessment of program effectiveness in this area.

Another dimension of scope is the breadth of geographic coverage of your system. Typically, it will not be desirable to limit coverage to specific sub-state areas. If resources are a factor, another method of controlling costs should generally be considered. However, there may be special circumstances when focusing on certain sub-state areas would be appropriate. For example, one might choose to focus attention on selected areas that appear to be experiencing difficulty; that is, where there are major concerns about the quality of services. Also, in states where only state-level measures of performance are desired, you may want to select a representative group of localities from which you can generalize to the state. Again, however, performance measurement systems will be most useful if they are implemented on a statewide basis.

Finally, you will need to determine if your system will target consumers whose services are paid for by certain funding streams (e.g., Medicaid, state general funds, special categorical state funds, mental health block grant). You may be able to limit your performance measurement system to persons whose services are paid, at least in part, by a certain funding stream. Very often, however, some of the services these persons receive will be paid by another funding stream. This is especially true for the most disabled consumers who receive multiple services. Given this, it will be very difficult to isolate the outcomes attributable to a single funding stream.

In designing your performance measurement system, you will have to decide about the scope of your system on several dimensions, including population, program type, geographic areas, and funding streams. In general, these decisions will have to balance the resources available with the breadth of information needed. One criteria to use in determining which populations and program types to include is the amount of resources consumed; that is, those populations and programs types that are the most costly are excellent targets for performance measurement. A second criteria to consider is the extent to which there are concerns about the quality of services for a given population or provided by a given program.

While the ideal goal is to have the greatest breadth of coverage possible, you may want to first focus on the highest priority populations, programs, and funding streams. Once your system is operating effectively in these areas, you can expand its scope.

Once the purpose and scope of the performance measurement system have been identified, you will have to select and/or create performance indicators. Although this task will be eased if all parties have agreed to adopt the MHSIP Report Card as its guiding framework, you will still need to make decisions about whether to measure all indicators recommended in the Report Card, or whether to select some but not others, and if so, which ones. Our experience to date is that no state or system that decided to implement the Report Card has actually reported results on each and every indicator included in the Task Force report. In addition, many if not most systems that have adopted parts of the Report Card have decided to add indicators from other sources to meet local needs. The process for determining which indicators to measure is, thus, a critical and challenging one.

If you have completed the tasks described above, the process of selecting indicators should be manageable. To further facilitate this process, we recommend the following:

The values inherent in the Report Card clearly imply that whatever specific process is adopted for indicator inclusion, all stakeholders should be involved in meaningful ways to ensure diverse opinions are reflected in the results of the process. A related point is that the indicator selection should flow directly from the statement of values; i.e., each indicator should be a more specific statement stemming from a value of how the service system should perform. A very clear grounding for each indicator in one or more of the consensus value statements is an effective method for ensuring selection of indicators that factor diverse stakeholder opinions, focusing on areas of common values and not divisive ones. This is reflected in the format of the Report Card.

As an illustration, a group might agree on a value that a service system should be responsive to all consumers regardless of race, ethnicity, or cultural background. An indicator stemming from that value might be availability of bilingual providers. The close connection between the value and the indicator is a safeguard against selection of indicators that meet certain constituency needs, but are not reflective of the diverse opinions of system stakeholders.

Take Advantage of the State of the Science

Use Criteria Identified by Experts

It is in your interest to use the best available science on performance measurement. As applied to the specific topic of selecting indicators, this suggests you should use criteria for selecting indicators that have already been identified by experts. A number of persons experienced with the science of performance measurement have proposed sets of criteria, many of which overlap significantly, or are closely related. Table II-1 contains 17 criteria for selecting indicators proposed variously by Elizabeth McGlynn, Viriginia Mulkern, and Edna Kamis-Gould. Along with each criteria we provide a brief explanation.

Table II-1

Select indicators from existing sources

In selecting indicators, avoid reinventing the wheel. Presumably, your performance measurement project will rely heavily on the indicators selected by the MHSIP Task Force and outlined in the MHSIP Consumer-Oriented Mental Health Report Card. As noted earlier, however, you may decide to add other indicators. We strongly recommend that you turn to sources of existing indicators before creating new ones yourself. Hundreds of indicators from dozens of processes already exist, representing a great deal of thought and substantial research and development. Table II-2 cites just a few resources for identifying indicators.

Table II-2
Sources of Candidate Performance Indicators

Select indicators by domain

In selecting indicators it helps to group them into domains. This approach is effective in breaking down the task of selecting indicators into more discrete conceptual boundaries. Much work of classifying indicators into domains and in conceptualizing what different domains entail has been already done. Many different lexicons of domains have been proposed. The MHSIP Consumer-Oriented Mental Health Report Card organizes indicators into the domains of access, appropriateness, outcomes, and prevention—you may want to follow this framework for selecting additional indicators.

Donebedian provides an alternative organizing framework for conceptualizing quality and its measurement, that with a little elaboration works quite well. Table II-3 shows a slightly modified version of Donabedian’s framework.

Table II-3

Donabedian suggests that we consider measuring quality across each dimension: structure, process, and outcome. Some persons might ask why measure process variables if outcome information is available. There are several reasons:

1. Process information may be available long before outcome information is available. With such information, presumably, we can act to prevent bad outcomes.
2. Process information may be a better measure of quality if outcomes are being effected by factors beyond the control of the provider/MCO.
3. Process information permits us to infer what might be causing outcomes which will allow us to find new and, perhaps, less costly ways of providing service, and inform others how to achieve the same outcomes.

The Donabedian framework can be used alone or in combination with other commonly used domain categories such as access, appropriateness, etc., to serve as an additional guiding structure for selecting indicators.

Our experience suggests it is important to separate the steps of selecting or developing indicators from that of selecting or developing measures. The process of selecting indicators is about identifying important areas that reflect overall system performance. This is not a highly technical process, but rather is more focused on identification of important areas of system performance, integrating different stakeholder opinions, and prioritizing certain topic areas over others. The process of selecting indicators should refer back to the values articulated in the previous step more so than looking forward to measures that may or may not be available for indicators under consideration.

This may lead to selection of some performance indicators for which there are not currently measures that meet the criteria for selection of measures, discussed below. This is not a futile exercise, however, but is advisable for three reasons:

• First, it will make explicit and public your belief that this indicator is a critically important topic and may stir the service system to strive towards improvement in this area, even without a specific measure of progress.
• Second, knowing that an indicator has been endorsed will motivate technical experts and to develop an acceptable measure for the indicator as quickly as possible.
• Finally, if a group is selecting indicators for multiple systems or organizations, it may be impractical to select a specific measure due to differences in information systems and routine data collection.

Once you have identified the indicators for your system, you must determine how they are to be measured. That is, you must determine the specific instruments, data sources, and methods that will be used to operationally define the indicators in a manner that produces the highest quality data. In accomplishing this step, you will have to make several choices. For example, you will need to decide the source of the data (i.e., measurement instruments or administrative data sets) and, if using a measurement instrument, whether to select an existing instrument or to develop a new one. In arriving at a decision regarding each choice, the same general process should be used. The steps in this process are:

• Review published literature and field experience
• Establish criteria for measures

• Reliability
• Validity
• Sensitivity to change
• Robustness (i.e., extent to which the measures will maintain good psychometric quality in applied settings)
• Availability of norms and benchmarks for similar populations
• Appropriateness for the identified populations (e.g., reading level)
• Cultural sensitivity
• Burden on staff and consumers (time to administer, score, etc.)
• Availability of training materials and software
• Clinical utility or other ancillary benefit
• Cost

• Identify potential measures based on comparison to criteria
• Field test
• Select

It is worth noting that if you want to compare the results of your performance measurement system with other systems, you should add "current use" as a criteria for selection of measures. If you select different measures, even for the same indicators, cross-system comparison will be difficult at best and impossible at worst.

This section makes recommendations on issues related to confidentiality and IRB approval for conducting consumer surveys within a performance measurement context.

An IRB proposal and approval is necessary for research projects involving human subjects when State, Federal, or other public funds and/or patients, consumers, students or employees are involved. The federal requirements are referenced in the Code of Federal Regulations 45 CFR 46 (applicable only when federal funds are used), 21 CFR 50, and 21 CFR 56. IRBs are responsible for holding the investigator accountable for compliance with these regulations based on its own judgment about the benefits and risks of the project.

These issues will differ somewhat based on the method of survey administration selected (mail, telephone, in-person or clinician office interview, etc.) and how the sample is selected. (See Chapter 3 for discussion of survey administration methods)

Recommendations:

• Become knowledgeable about privacy protections in both federal and state law. Assess both federal and state confidentiality laws and educate all parties involved in the project.

• Include IRB requirements and confidentiality considerations in your planning as early as possible

• First, assess whether an IRB review is necessary. This can be done by consulting with lawyers and/or IRB chairpersons. If a review is necessary, adjust your timeline accordingly to prevent delays in implementing your project.

• IRBs may meet just once a month, and require submission of materials prior to the meeting. There may also be a limit to the number of proposals considered each month on a first-come first-serve basis. Expect that some revisions will be necessary and return to the next month’s meeting may be required prior to approval. An IRB may also require the final versions of instruments and documents such as consumer letters, consent forms, return postcards, etc.

• Decide whether the data will be anonymous—this decision will affect the extent of your planning for processes to ensure confidentiality.

• Clearly articulate the benefits of your performance measurement system to communicate their importance.

• This step is very important to ensure that an IRB assessment of whether the benefits outweigh the risks is based on full information.

• Include rationales that you may have assumed obvious, such as why the consumer perspective is important, how the data can be used for service planning, the benefit of understanding the characteristics of consumers who have optimal and less than optimal outcomes, etc. Remember that the level of expertise and sensitivity related to mental health may vary among board members.

• Establish procedures for reporting information such as harm to self or other, and child abuse.

• Include confidentiality in your training of interviewers and other project staff.

Other information that an IRB may require includes:

• a description of data collection procedures to ensure confidentiality;

• a description of the process to obtain consent including interviewer scripts and forms;

• examples of any materials that will be sent to consumers such as postcards, letters, etc.;

• a description of how the interviewer would access help in an emergency;

• examples of confidentiality forms to be signed by all interviewers and project staff;

• information about how to access mental health services. For example, in Maryland, interviewers are distributing refrigerator magnets with a 800# for accessing the Public Mental Health System; and
  

• a 6^th grade literacy level for all materials (for Medicaid consumers, HCFA requires this as well).
• consider all of HCFA’s requirements

Once indicators and measures have been selected and IRB approval has been obtained, it is time to turn attention to data collection procedures. Procedures need to be defined before IRB submission. If your performance measurement system is to produce reliable and comparable data across different reporting units (e.g., providers, regions, etc.), standard protocols must be developed for data collection, management, analysis and reporting. While some of these protocols must reflect local organizational structures and resource capacity, the following chapters will provide detailed guidance in this area to ensure the greatest possible data quality and consistency. As we will describe below, options exist in each step of design, data collection and analysis. Standardization of protocols employed in data collection, however, is essential regardless of the specifics of the protocol.

The implementation of a fully functional performance measurement system requires the investment of extensive resources. Therefore, prior to full implementation, you should pilot test your system. To the extent possible, the pilot should include a complete test of all aspects of the performance measurement system, including data collection, management, analysis and reporting. The pilot will provide an opportunity to identify problems (e.g., sources of unreliable data) and correct them before investing large amounts of resources. Correcting problems at this stage will be much easier and will increase the chances that full implementation will be successful. A first attempt at full implementation that is unsuccessful will lead to frustration and make it more difficult to obtain stakeholder commitment for continuing your efforts (Koch, Cairns & Brunk, 1998).

A detailed work plan is essential for the successful implementation of any effort as complex as a performance measurement system. The work plan should specify critical tasks or steps to be completed, the person(s) responsible for completing each task, and the beginning and ending dates of each task. The work plan should be continuously reviewed to monitor your progress in achieving critical milestones, and adjustments made as necessary.

An example work plan covering the major steps described in this chapter is presented in Table II-4. The amount of time required to complete each step will vary significantly from system to system. Some of the factors that will influence the time requirements are:

1. The strength of existing collaborative relationships with key stakeholders.
2. The complexity of the proposed system, including the number of distinct purposes it is to serve and its scope.
3. The number of indicators to be measured.
4. How the indicators will be measured; in particular, if measures will be based on existing data or require new data collection, including standardized instruments.
5. Level of development of your data infrastructure.
6. Level of development of your information technology infrastructure.
7. Availability of resources, including the availability of technically qualified staff or consultants.
8. Level of resistance/support among key stakeholders.

The work plan presented below assumes a moderately complicated system requiring some new data collection using standardized instruments. It also assumes general support among key stakeholders for implementing the system, moderately developed data and information technology infrastructures, and the availability of adequate resources.

Table II-4