Chapter 5: Reporting Results

Reporting results is the culmination of all of the planning, data collection and analysis for a performance measurement project. As with many of the steps in the process, the way in which you decide to report results should depend on the primary and secondary purposes of your performance measurement project. Clearly, internal quality improvement and contract management, two potential purposes of a performance measurement system, call for different kinds of reporting, both in terms of process and content; i.e., we envision reporting results to include not only the actual charts, graphs or tables that you will produce, but also to include a process for developing the structure of such reports, an audience for the reports, feedback on ways the reports could be more useful and user-friendly, and finally a plan for the actual release of the reports.

Here are a sample of questions you will need to answer in planning for reporting results:

• Who will results to reported to?
• Will the same report be sent to all audiences or will certain persons or groups receive specialized reports?
• If you are reporting results by provider organizations or managed care plans, will these organizations have the opportunity to review the reports prior to public distribution?
• Are your results subject to immediate release at any time under freedom of information laws or regulations?
• Will you prepare a narrative report or just report data in the form of charts, graphs and/or tables?
• At what level of detail should results be reported?
• What will your policy be regarding requests for reports of additional analyses or variations on standard presentation?
• How will you distribute the reports? Through a meeting or simply by mail?

Below we provide guidance in answering the above questions and others regarding reporting performance measurement results along with illustrations where they are available.

Planning for reporting should be initiated as the first step in the design of the system. Developers should be clear about the actual purpose and use of the data and how it should be reported and disseminated. The process of designing reports imposes a discipline that will result in significant cost containment if followed through. In the absence of defining how each element of data will actually be reported, there is a significant risk to designing a data collection system with significant unnecessary costs.

One thing that will help you focus on essential goals is to identify and engage the groups that will use the information you will provide. It is important that you also establish a method of obtaining feedback from these user groups. You must demonstrate to these groups that you are listening and responding to their recommendations. Work with your groups using well thought out protocols. Focus groups might be useful.

Since performance measurement projects almost always have multiple audiences, it is especially important to design a process and content for reporting results that factors in the particular needs and desires of important audiences. For example, with the advent of consumers, family members, and other lay persons as active members of planning and quality improvement boards and committees, the need for effective reporting of results becomes paramount. Lay persons increasingly demand reports that are clear, accurate, and simplified. The central role of diverse audiences also suggests that it is important to involve these persons in the planning of performance measurement reporting.

Some of the user groups you might identify:

• Consumers
• Family members
• Program managers
• State and local mental health authorities
• Payors, MBHOs, HMOs
• Governor
• Legislature
• Public and mass media

It should be made clear to the various user groups that will use this information that it is their responsibility to indicate changes they require in order to make information useful in improving services and customer satisfaction.

One mechanism for soliciting stakeholder needs and preferences vis-à-vis reporting results is to convene stakeholder focus groups. These groups would be homogenous (i.e., composed only of consumers, or only of providers, etc). These focus groups could serve as "sounding boards" for your ideas on reports. It may be advisable for them to meet periodically as you modify your reporting plans.

Constant feedback is essential. There needs to be a mechanism in place that allows for constant feedback from the user groups as to the quality of the information being received and any changes that may be needed to fine tune the process. Strategies such as the focus groups discussed above may be useful.

It will be very helpful if you develop a flow chart that indicates clearly how data will move through the system. This will help you visualize and organize the structure and tasks you will need for your effort.

Decisions about report content are crucial to the success of a performance measurement initiative. Very few information users will be able to make productive use of undigested information. And it is frustrating for potential users to have to wade through voluminous data tables to get the specific pieces of information they seek. This means that considerable attention should be given to exactly what information should be included in reports intended for different audiences.

Decisions about report content should not be left to the endgame. They should be considered in the design phase of the performance measurement project. Mechanisms that might be helpful in eliciting ideas about useful content include key informant interviews with knowledgeable stakeholders from each group, focus groups of stakeholders, and information interviews with persons from other states who have more experience. In most cases, brief, targeted presentation with clear summaries highlighting major points are likely to be more effective than long, detailed reports.

Decisions about report content should be driven by several considerations including:

• The use to which different groups will put the information (e.g. contract monitoring, CQI, advocacy, etc.)
• The major policy concerns that need to be addressed
• The relevant comparisons that should be featured (e.g. trends over time, performance vis-à-vis standards or benchmarks, relative performance across organizations, etc.)

In all cases, reports should include a discussion of the strengths and limitations of the data presented. This discussion should, at a minimum, include:

• A clear description of the methods used in collecting the data
• A description of sample selection procedures, sample sizes, response rates, and any known or suspected biases
• The degree to which the data are considered generalizable across the state or local area
• A discussion of what level of difference might be considered meaningful for each variable or class of variables presented

As with all phases of this process, it is useful to revisit decisions about report content in a systematic fashion. Periodic discussions with stakeholder groups can provide important feedback on whether the reports are hitting the mark.

Having determined the content of each report, the next task is to design formats for data presentation that help to make your point efficiently. Decisions about format should be based on considerations such as the following:

• Different formats are appropriate for different types of data.
• Different audiences will require varying levels of detail in their presentations. For example, audiences that are interested in highly summarized information, may prefer a few striking bar graphs or pie charts. However, for those stakeholders who are interested in item level analyses, tabular presentations may be a much more efficient way of presenting the data.
• Audiences will vary in the amount of experience they have in interpreting statistical data. Presentations should be tailored to the audience’s level of expertise.
• Stakeholder needs may change over time as policy issues evolve and as various constituent groups gain more experience in understanding statistical information. It is useful to monitor such changes by providing opportunities for stakeholder feedback.

Below are some examples of graphic representations that might be useful:

Bar graphs are a simple chart to read and to make. They are useful for showing sums and averages and are often used to illustrate variable values. Bar graphs use spaces between the bars to emphasize distinct variables. Bar graphs can vary in the amount of space between the bars, a horizontal or vertical display of the bars, and the order of the bars. Bar graphs can also display two or more categories of data by organizing the bars in a group, overlapping, or stacked chart. Bar graphs can be designed to show both numbers and frequencies.

Bar graphs can be displayed vertically or horizontally. Horizontal bar charts are used in the following two occasions:

1. Variable values with long names - a horizontal graph is the only solution when the variable names will not fit under the vertical bar.
2. Many variable values – as few as 6 to 8 variables makes a vertical chart difficult to construct neatly. A horizontal graph easily makes more space for the variables.

The following is an example form the Results of the Adult Mental Health Consumer Survey 1997 conducted by the Office of Strategic Planning and Resource Development, Office of Research and Evaluation, Texas Department of Mental Health and Mental Retardation. This bar graph displays the percentage of consumers with ratings of agree or strongly agree for the items that make up each of the four domains, outcomes, access, quality/appropriateness, and satisfaction.

Example:

       For the Simple Bar Graph example, please see the note at end of this chapter.

Grouped bar graphs are used for displaying two or more categories at a time. Separate bars on the same axis represent the different categories. The bars are often distinguished by different shading or coloring. Grouped bar charts, like normal bar charts, can be drawn either vertically or horizontally.

The following is an example of a grouped bar graph. The graph is from the 1997 Adult Mental Health Consumer Survey: Report Comparing Local Mental Health Authority to State Average in the Domains of Outcomes, Access, Quality/Appropriateness and Satisfaction. The report is by the Office of Strategic Planning and Resource Development and Office of Research and Evaluation in Texas. The grouped graph compares outcomes, access, quality/appropriateness, and satisfaction of the local MHA and the State Overall.

Example:

        For the Grouped/stacked Bar Graph example, please see the note at end of this chapter.

Pie Charts

Pie charts are used when you want to focus on proportions of quantitative variables. The following is an example from the "The Delaware Experience" from the Consumer/Client Satisfaction Survey Pilot Project presented at the National Conference on Mental Health Statistics in May 1998 by Maurice Tippett and Edna Kamis-Gould. This chart displays the relationship of program satisfaction and life satisfaction. If you have several groups that you would like to compare then it is useful to draw a pie chart for each group. In the Delaware example, the groups are persons with high program satisfaction scores and persons with low program satisfaction scores, each displayed on its own pie chart.

Example:

        For the Pie Chart example, please see the note at end of this chapter.

Tables can be used to display performance results of multiple organizations or of a single organization on several domains. The major advantages of tables are their ease of construction, straightforward interpretation, and ability to read actual numbers. Tables are not terribly visually appealing, however, and can be overwhelming to a reader if they are large. In general, tables are most appropriate when the reader is interested in detailed results and/or there is a large number of organizations. Below is an example of results from the MHSIP Consumer Survey access domain from Texas, displayed by provider organization:

        For the Table example, please see the note at end of this chapter.

When you want to display data in a time series it is important to design a chart so that it is easy to make comparisons over time. It is difficult to detect patterns of change over time in tables. Line charts are suitable when you want display changes that are large and when there are turning points. Line charts are easy for the eye to follow and clearly display distinct changes over time.

Example:

        For the Time Series Chart example, please see the note at end of this chapter.

Scatterplots are used to show the relationship between two quantitative variables. The data consist of paired coordinates (x,y) each indicated as a dot on the graph. A curve in the chart is often included to show an estimated regression. This gives an overall picture of the relationship between all the dots on the graph.

Example:

This type of reporting structure is common among ratings of movies, restaurants, colleges and the like because of its simplicity and visually appealing qualities. Simply, this type of reporting is the display of a number of stars or other symbol next to a variable or organization based on the organization’s performance or score on the particular variable. This type of report is familiar to most and is the basis of so-called "five-star" restaurants, resorts, etc. Different symbols can be used to denote different areas of performance. For example, stars may be used to represent overall performance, smiling faces may be used to represent consumer satisfaction, houses may be used to represent housing outcomes, and clocks may be used to represent waiting times. The Reporting Kit from the Consumer Assessment of Health Plans Study (CAHPS), a performance measurement system for health plans proposes the following report:

        For the Stars or Other Symbolic Representations report, please see the note at end of this chapter.

Your will need to:

• Determine how often and when reports are needed: Different user groups may have different needs as far as frequency and timing of reports. A legislator, for example, may be most interested in a report that is concomitant with the fiscal year and is released soon before the budget appropriations period. A program manager, alternatively, may be interested in more frequent reports for quality improvement purposes and in timing of reports that help with preparation of proposals.
• Examine how the timing and frequency of reporting aligns with the planned frequency and timing of data collection: Without careful coordination of data collection and reporting, you may find yourself without any data to report at a time when a report is due.

Plan for production and distribution.

1. Decide what your production schedule is and who is responsible for each area. Here a detailed chart and timeline would be appropriate and helpful.
2. Determine the responsibilities for these areas:

• Data flow
• The report development schedule
• Use of the results

• Program management
• Quality improvement
• Ensuring accountability for public funds
• Local planning
• State planning and policy
• Treatment planning that supports consumer involvement and choice.
• Risks or cautions of reporting. Public information: potential lawsuits.

• Developers should be clear about the actual purpose and use of the data and how it should be reported and disseminated.

• Identify and engage the groups that will use the information you will provide. It is also important to establish a method of obtaining feedback from these user groups.

• Focus groups could serve as "sounding boards" for your ideas on reports. It may be advisable for them to meet periodically as you modify your reporting plans.

• There needs to be a mechanism in place that allows for constant feedback from the user groups as to the quality of the of the information being received and any changes that may be needed to fine tune the process.

• Develop a flow chart that indicates clearly how data will move through the system.

• Decisions about report content should be considered in the design phase of the performance measurement project. Mechanisms that might be helpful in eliciting ideas about useful content include key informant interviews with knowledgeable stakeholders from each group, focus groups of stakeholders, and information interviews with persons from other states who have more experience.

• In all cases, reports should be include a discussion of the strengths and limitations of the data presented.

• Periodic discussions with stakeholder groups can provide important feedback on whether the reports are hitting the mark.

• Design formats for data presentation that help to make your point efficiently.

• Determine how often and when reports are needed.

• Examine how the timing and frequency of reporting aligns with the planned frequency and timing of data collection.

• Decide what your production schedule is and who is responsible for each area.

NOTE:  Copies of graph and report examples may be requested at phone number (405) 522-3966.