Attendees: Carrie Kaufmann, Ron Tremper, Steve Davis, Cindy Hopkins, Judy Hall, Randy Koch, Ted Lutterman, Olinda Gonzalez, David Brown, Maureen Veech, Jack Wackwitz, Mary Smith, Pat Smith

The MHSIP Consumer Survey, a key component of the MHSIP Consumer Report Card, has been increasingly adopted by states and other entities for implementation since it became available in 1996. However, a number of different versions have emerged during the last four years due to attempts by mental health system stakeholders to adapt the survey for local use. Survey items have been reworded and proposals have been made to add new items to address areas not previously "tapped" by the original survey. In addition, there has been a wide range of variability with regard to how the survey has been implemented, the population targeted, the scoring of items, sampling procedures and the reporting of results. The MHSIP Policy Group is undertaking the task of developing and issuing guidelines regarding what constitutes the "Official MHSIP Consumer Survey", and of providing guidance for its implementation.

It should be stressed that these guidelines do not reflect a revision of the Consumer Survey; they are merely a refinement of the original version and an attempt to answer questions that have been raised by various entities. Work on a revision of the full MHSIP Report Card, including the Consumer Survey, will be undertaken later this year in collaboration with a number of mental health stakeholders.

The MHSIP Consumer Workgroup was convened to recommend guidelines for the Survey. The workgroup consists of MHSIP Policy Group members who have implemented consumer surveys or who have an interest in the area, as well as three technical experts who have performed comprehensive analytic work in this area. The Workgroup was convened with the following mandate:

(1) Review specific areas in which work is currently being performed on the Adult MHSIP Consumer Survey, including data collection efforts;

(2) Based on the review, propose basic guidelines for implementing the Survey including the items that comprise the survey, scoring, and data presentation;

(3) Review specific areas in which work is being performed on Consumer Surveys for children and adolescents and their parents/caregivers; and

(4) Identify the next steps for moving toward a formal revision of the consumer survey (to be known as Version 2).

This report, which summarizes the workgroup’s findings and recommendations, will be submitted to the MHSIP Policy Group for review and discussion at the next Policy Group meeting which is scheduled for April 12^th and 13^th. Pending approval of the recommendations, the guidelines are to be disseminated to the field as soon as possible.

In preparation for reaching a consensus on the items that will comprise the recommended version of the MHSIP Consumer Survey, presentations were provided by Caroline Kaufmann, Jack Wackwitz and other workgroup members.

Drs. Caroline Kaufmann and Dan Phillips recently completed and issued (October 1, 1999) a report titled Survey of State Consumer Surveys that provides a summary of the extent to which states have implemented consumer surveys. At the time of the survey, most states were using a short form version (21 items) or some other version of the MHSIP Consumer Survey. Two observations were made on the basis of the report:

1. There is a need for surveys to be piloted by states prior to full implementation---one in four states reported initiation of a full survey process without piloting, and
2. Information provided from the use of the surveys has a positive impact on the mental health service delivery system.

Dr. Kaufmann made the following recommendations based on the report:

1. Any revision of content of the Consumer Survey needs to reflect the broad based input of consumers;
2. State Mental Health Offices of Consumer Affairs should be more involved in the implementation and administration of consumer surveys, and
3. Future work needs to address the translation of the official survey format into the languages of persons served by the mental health service delivery system.

Dr. Wackwitz described the comprehensive analytic work that he has performed on the Consumer Survey using data generated by states participating in the 16 State Pilot Indicator Project.

Observations and Recommendations:

(1) The negatively worded items from the original 40-item MHSIP survey are problematic and minimally should be restated as positively worded items. Analysis is indicative that variability in responses to the negatively worded items are more likely to reflect respondents’ confusion regarding the interpretation of the items as opposed to actual dissatisfaction with services, which was the concept that was originally envisioned as the target for measurement using these items.

(2) A factor analysis of the MHSIP consumer survey with the negative items reworded as positive items generally supports the MHSIP four domain structure of access, appropriateness, outcome and satisfaction, and

(3) An analysis of a sample of surveys drawn from Rhode Island and Colorado databases revealed that the same survey items were likely to have a substantial amount of missing data. This pattern of missing data may be indicative of some methodological problem with the items. This information should be considered when future work is undertaken to revise the survey.

Dr. Wackwitz made the following specific recommendations on the basis of his analyses:

(1) That the twenty items identified through a recent factor analysis of survey data form the initial basis for retention of items to be included in the recommended version of the MHSIP survey. These items offer the best fit in terms of psychometric criteria such as uni-factorialness, factor pattern loadings and minimal missing data. The items also support the underlying MHSIP Survey domain structure of access, appropriateness, outcome and satisfaction and

(2) That six specific additional items representing key or important MHSIP indicators be added to the 20 item set to form a core set of 26 survey items (see attachment I).  Although additional items could be added based on local issues, populations served or other concerns, essentially the 26 items as currently worded would form the heart of the survey.

Additional recommendations and comments were reviewed from the following sources: a communication from the MHSIP Western Users Group, a summary of conversations with various state reform grant holders and MHSIP contacts (Kentucky, Nevada, Georgia, Vermont, Minnesota and Maryland) provided by Dr. Olinda Gonzalez and information provided by representatives of state mental health authorities from Virginia, Texas, Rhode Island, Maryland, Louisiana and Illinois. Information related to the development of the Consumer Assessment of Behavioral Health Services (CABHS) instrument was also considered. Following is a summary of the information provided:

A wide range of consumer survey versions are being used by these sources--a 19 item version, the NCQA short form 21 item version, the original survey (including negatively worded items), the 40 item survey with the negative items reworded to positive items and 40+ item versions. Additionally "home grown" or locally developed instruments (which address some or all MHSIP domains) are also being used by some states.

Surveys are being administered in a variety of ways. Methods for administration include "face-to face" interviews, direct mail, and telephone, as well as various combinations. Similarly, "who" administers the survey varies. Surveys are administered by consumers, treatment staff, support staff and by other contractors specifically hired to conduct consumer surveys.

Response rates vary widely, some are as low as 18%, some states report response rates as high as 60-70%. The amount of missing data, when reported, also varies.

Some states are able to link survey responses back to an individual consumer and some surveys are completely anonymous. Some states ask consumers to provide demographic information as part of the survey process; some do not. The absence of demographic data is problematic in that it is difficult to determine the population to whom the data applies and it is impossible to compare information across states.

A wide range of populations are being surveyed: adult consumers with serious mental illnesses, adult consumers with other mental illnesses, and adult consumers with dual illnesses (mental illness and substance abuse). One state also used the MHSIP Consumer Survey as a basis for developing a survey for children and adolescents.

Surveys are administered at the discontinuation of treatment, a year after treatment has been discontinued, at intervals related to review of treatment plans, once a year at a set point in time and at a variety of other time intervals.

Several states collect qualitative information that is rich in content through the use of open-ended questions, in addition to the consumer survey items.

Different response sets are also used to collect responses to the survey. Some states continue to use the scale developed for the original 40-item MHSIP survey, others have truncated the scale using only three possible responses. One state is using a scale based on a school grading response set (e.g. A, B, C, D, F), and the CABHS instrument utilizes a ten-point response set.

(1) Some concerns were expressed regarding the range of input on survey related issues. It was suggested that a broad range of input needs to be sought from consumers, as well as others who have experience implementing the consumer survey.

(2) There needs to be some consensus and guidelines issued regarding methodological issues associated with the implementation of the survey.

(3) The use of a "core set" of MHSIP consumer survey items, small enough to permit states to add items of specific interest was recommended. This "core set" needs to be promoted as the "Official MHSIP Consumer Survey".

(4) There need to be guidelines developed around the presentation of data and in particular, the best way to present data to different groups of stakeholders so that it is meaningful.

The first task of the workgroup was to make recommendations regarding items that will comprise the "Official MHSIP Consumer Survey", taking into account the recommendations made by Drs. Kaufmann and Wackwitz, as well as recommendations and issues identified by other mental health system stakeholders as described above.

The workgroup agreed to use the following criteria for this selection. The items retained should be:

(1) value driven, that is they should reflect the values that underlie the MHSIP Consumer Report Card;

(2) meaningful to consumers and other mental health system stakeholders and

(3) psychometrically sound.

Additionally, it was decided to review survey items within the following context:

(1) The 40 item version with all positively worded items would provide the starting point for the review.

(2) No attempt would be made to change the wording of these items (beyond the positive wording). Other changes in wording would be addressed in version 2 of the revised survey.

(3) The survey as it stands is directed toward adults served by community mental health agencies.

An initial review of survey items led to a decision to accept Dr. Wackwitz’ recommendation regarding the retention of the 26 items emerging from the analytic work performed on the survey. Each of the remaining 14 survey items was reviewed and discussed in turn by the workgroup.

1. A consensus was reached by the workgroup to retain twenty-eight items (see Table 1). These items should form the core of the Official MHSIP Consumer Survey. Although additional items may be added based on local issues, populations served or other concerns, essentially the 28 items are mandatory and must be used as currently worded by entities who wish to report that they are using the MHSIP Consumer Survey.
2. A number of the items, though retained, were identified as problematic because it is unclear that they measure the concepts that they were initially designed to assess. The items were retained to ensure that the concern was not lost, but these items are targeted for revision and inclusion in Version 2 of the Survey. These specific items are identified in table 1.

The second task undertaken by the workgroup was to discuss and recommend methodological guidelines for implementing the survey. Dr. Judy Hall described the extensive work that she has undertaken in working with the 16 State Pilot Indicator workgroup on consumer surveys. The methodology used across the states (e.g. who administers the survey, when and where it is administered, population surveyed and various methods used for presenting survey results) was discussed, as well as proposals for addressing each issue.

Cindy Hopkins also briefly discussed the results of research that she recently performed in Texas to determine the impact of consumer survey collection methodology on response rate, consumer representativeness, respondent characteristics and cost. The results were indicative that in general "who administered the survey" did not impact consumer responses or response rates, although there was a difference for several sub groupings of participants.

Based on the discussion, Dr. Hall agreed to draft methodological guidelines for the workgroup to review. This document will be incorporated into the full report for review by the MHSIP Policy Group. (A copy of the document is attached—see attachment 2.)

Several weeks prior to this meeting, members of the workgroup conducted a conference call with various key informants to determine the efforts being undertaken to develop and implement consumer surveys for children and adolescents and their caregivers. Two key efforts were identified. Several years ago, a small workgroup developed a child and adolescent survey that was adapted from the MHSIP Consumer Survey. A second more recent effort has been undertaken in relation to the 16 State Pilot Indicator Project. Randy Koch reported on efforts initiated by a workgroup that has been chaired by Dr. Molly Brunk of Virginia. A survey has been created using the Family Satisfaction Questionnaire (FSQ) and the Youth Satisfaction Questionnaire (YSQ) as its foundation. The 16 state workgroup felt that the MHSIP survey was not oriented to the child and adolescent population, and that essentially the FSQ and the YSQ were a better fit. The survey covers the following domains: access/provider availability, cultural sensitivity, involvement in treatment, outcome and satisfaction. Extensive work has been performed on the development of this instrument which is currently in the process of being piloted by a number of states associated with the 16 State Study. It is expected that data from the pilot will be available in fall, 2000.

Dr. Koch also reported on a collaborative effort between the MHSIP Policy Group and the Children’s Outcome Roundtable. The Roundtable has been working on the concept of a consumer survey which has a broader focus than the MHSIP survey in terms of the audience to be surveyed. While the MHSIP Consumer Survey is more geared toward the public mental health system, the Roundtable Survey encompasses other "child serving" systems in addition to mental health. However, the Roundtable membership is very interested in the FSQ/YSQ based survey and has proposed that a collaborative effort be undertaken to complete work on the survey. It is likely that the MHSIP Policy Group and the Children’s Roundtable will form a working partnership that will advance the work in this area.

The work in the area of child and adolescent and caregiver consumer surveys as they relate to the MHSIP Report Card concept is less advanced than the work done with the adult survey. The workgroup recommends:

1. That information be disseminated regarding the 16 State Workgroup pilot, including copies of the surveys developed and
2. That the MHSIP Policy Group form a working collaboration with the Children’s Outcome Roundtable.

Adaptations of the MHSIP adult consumer survey are not recommended for use with children and adolescents or their caregivers.

Ted Lutterman reported on the work that the NASMHPD Research Institute (NRI) Performance Measurement System has undertaken with regard to using the survey to meet JCAHO ORYX requirements. The NRI has recently received approval from JCAHO to utilize the 21-item version of the MHSIP Consumer Survey to provide measures of consumer perceptions of access, appropriateness, outcome and satisfaction.

Dr. Mary Smith also reported on the use of a slightly reworded version of the MHSIP Survey that is being used across ten state psychiatric hospitals in Illinois. The 52-item version (40 original items, plus 12 additional items) was implemented approximately 9 months ago after being piloted across 6 state hospitals. An analysis of data essentially supports the four factor item structure of the MHSIP survey as reported by Dr. Wackwitz.

Dr. Kaufmann also noted work that is being undertaken by Pat Howard from Kentucky on the use of the MHSIP Survey in inpatient settings.

Workgroup members expressed some concerns that important concepts may not be "tapped" by the current surveys being used in inpatient settings—most notably factors such as refusal of treatment and coercive treatment practices. Workgroup members wondered if the NRI has the latitude to modify or add to the survey items approved by JCAHO for the ORYX requirement.

1. A joint working committee should be established comprised of MHSIP Workgroup members and representatives of the NRI Consumer Survey workgroup to review work being undertaken on the inpatient survey and to make recommendations on a possible modification of items (to the extent necessary or possible);
2. That NRI determine the extent to which survey items can be modified or added to without undergoing a full review by JCAHO in the event that recommendations for modification are made by the workgroup and
3. That information regarding consumer surveys is amassed from as many sources as possible for review. Toward that end, MHSIP User Group chairpersons will be contacted to solicit information, as will State Offices of Consumer Affairs. Information is to be forwarded to Cindy Hopkins.

The Consumer Workgroup’s recommendations and guidelines as documented in this report should be reviewed by the MHSIP Policy Group as soon as possible. Once reviewed (modified as necessary) and approved, the guidelines should be disseminated to the field. Possible means of dissemination include: presentations at the National Conference on Mental Health Statistics (May-June, 2000), discussions at MHSIP Users Group Meeting (Spring, 2000), and contact and discussion with state MHSIP contacts.

Two specific sets of recommendations are detailed below. The first set of recommendations pertain to Version 1.1 of the survey, which is essentially a refinement of the original MHSIP Consumer Survey. The second set of recommendations pertain to work to be undertaken during the revision of the Consumer Survey, specifically referred to as Version 2.

(1) The 28-item version of the survey as described in this report should be adopted and promoted as the Official MHSIP Consumer Survey. Essentially any entity that reports using the MHSIP Consumer Survey must utilize all 28 items as worded in this report. Although as noted earlier, items may be added to address local concerns, the 28 core items must be used. Translations should similarly focus on the core 28 items. The goal of comparing and aggregating data across states for benchmarking purposes and for describing the impact of mental health service delivery systems based on consumer perceptions, requires the adoption of a strategy in which a consistent set of survey items are used and reported. This version of the survey will hence be referred to as Version 1.1 which is a refinement of the original MHSIP Consumer-Oriented Consumer Survey.

(2) The need to collect basic demographic information for consumers completing the survey is essential. In the absence of this information it is difficult to interpret data produced from the survey. The workgroup recommends the collection of the following demographic variables: age, sex, and ethnicity. The items contained on this form should be considered to represent "core data elements", not unlike the concept of "core survey items". That is, states and other entities should feel free to collect more data, but these demographic data elements represent a minimum dataset. It would also be useful to include the following statement on these face sheets to explain the collection of the data: "This information will be used for statistical compilation purposes only".

(3) The survey form should identify the time period that consumers are being asked to rate (e.g. rate services received during the last year).

(4) Sampling and other methodological issues are of critical importance and must be addressed prior to conducting a consumer survey. The MHSIP Policy Group acknowledges the importance of these issues and urges persons considering conducting consumer surveys to consult appropriate literature such as the Toolkit on Performance Measurement Using the MHSIP Consumer-Oriented Report Card published by the Evaluation Center at the Human Services Research Institute.

(5) A number of states provide an "introduction to the survey" through the use of a letter which describes the purpose of the survey. This letter is typically endorsed by the State Mental Health Commissioner, the head of the SMHA State Office of Consumer Affairs, Consumer organizations, other key stakeholder groups or some combination thereof. The workgroup felt that this approach is useful in that it may encourage some persons to participate in the survey who may not typically consider participating. In particular, the endorsement by other consumer groups may be particularly helpful.

(6) The workgroup endorses the use of "open ended questions" in addition to the 28 core items comprising the official survey. This information has been found to provide a richness of qualitative information that may not be "tapped" by the standard survey items.

(7) Scoring of surveys --The issue as to how missing data should be handled in the calculation of domain scores is critical. Domain scores (percentages and means for access, appropriateness, outcome and satisfaction) should only be calculated if two thirds of the items comprising any domain are answered.  All "not applicable" responses should be recoded as missing.

(8) Display of survey results - The workgroup felt that the format for data presentation should be driven by the audience for whom the data is intended. In a discussion regarding the first level display of survey data, the workgroup made the following observation. Sometimes the simplest data presentation will meet the needs of many mental health system stakeholders. With this in mind, the presentation of response rates as percentages by domain and by individual item represents the most basic and understandable way in which to communicate survey results. If the need arises for a more complex way of displaying the data this can easily be accomplished. The important thing is that the recommended scoring procedure is followed.

(9) Response rates should be reported any time reports are generated for Consumer Survey data. Reports should include specific information on the amount of missing data, as well as the number of surveys returned that were either blank or that had so much data missing that they were not included in the overall analysis.

(10) Many entities utilize individual survey items for quality improvement purposes. It is recommended that the percentages be calculated for each of these items, regardless of the amount of missing data. The percentage of missing data associated with each item, should however, be reported.

(11) Two survey items have been identified by numerous states as being important indicators on their own. The two items are: I felt comfortable asking questions about my treatment and medication (original survey, item 12); and I, not staff, decided my treatment goals (original survey, item 19). It is recommended that responses be calculated and reviewed individually for each of these items.

(12) Any presentation/publication of survey results should include the following information: the methodology used to administer the survey, the sampling method, the target population, the time period for which data was collected and the amount of missing data by domain and by survey item.

(13) The reader is cautioned against comparing survey results with results obtained by other states or entities. Critical issues such as population targeted, sampling and other methodological issues must be reviewed prior to any such comparison.

(14) Child and Adolescent and Caregiver Surveys - The work in the area of child and adolescent and caregiver consumer surveys as they relate the MHSIP Report Card concept is less advanced than the work done with the adult survey. The workgroup recommends that (a) information be disseminated regarding the 16 State Workgroup pilot, including copies of the surveys developed; and (b) the MHSIP Policy Group form a working collaboration with the Children’s Outcome Roundtable. Adaptations of the adult survey are not recommended for use with children/adolescents or their caregivers. A description of the most recent work undertaken to develop child and adolescent and caregiver survey, as well as copies of the surveys are attached (see attachment 3).

(15) The following recommendations are made with regard to the use of the MHSIP Consumer Survey in Inpatient Settings: 

(a) A joint working committee should be established comprised of MHSIP Workgroup members and the NRI Consumer Survey workgroup to review and make recommendations regarding inpatient survey items;

(b) That NRI determine the extent to which survey items can be modified or added to without undergoing a full review by JCAHO in the event that extensive changes are recommended by the workgroup;

(c) That information regarding inpatient consumer surveys currently being used is amassed from as many sources as possible for review. Toward that end, MHSIP User Group chairpersons will be contacted to solicit information, as will State Offices of Consumer Affairs. Information is to be forwarded to Cindy Hopkins.

(1) A number of the survey items retained in Version 1.1 of the survey need to be reworded. These items were retained because the workgroup felt that it was important to capture some information related to the content area, but essentially the wording of the items is problematic and may be likely not to tap the underlying concept that is intended. Additionally, some of the items are worded using present tense, and some are worded using past tense. This is problematic because some surveys are being implemented while consumers are still receiving treatment, making the use of past tense inappropriate.

(2) Preparation for revising the survey, resulting in version 2, should be based on the broadest input possible. Toward this end, it would be useful to conduct focus groups with consumers and other key stakeholders to determine what areas should be addressed that are not currently included and what items should be dropped. The fact that the survey is designed to "tap" consumer perceptions requires that the revision rely heavily on information generated from the consumer perspective.

(4) Areas that were dropped from the original survey by this workgroup which need to be "revived" and rewritten for Version 2 include: cost of treatment as a barrier to receiving services (original survey, item 4); respect of treatment staff for consumers’ rights (original survey, item 14); competence and knowledge of treatment staff (original survey, item 25); responsiveness of treatment provider to consumers concerns about medication (original survey, item 29); consumers dependence on the mental health system (original survey, item 37); impact of medications on consumers symptoms (original survey, item 38).

(6) Parameters for sampling should be established with recommended methodology.

(7) The wide variety of "lessons learned" from Consumer Survey implementations should be incorporated into efforts to develop Version 2 of the survey.

Addendum

1.  Survey Administration: How are you going to administer the survey? Each method has its own strengths and weaknesses.

         Strengths                                                     Weaknesses

MAIL

-can reach broad group of consumers     -need accurate addresses

-privacy ensured                                         -low initial response rates

-relatively inexpensive                                -need follow-up mailings to

-relatively convenient                                   increase response rates

-sample can be chosen

Phone 

-can reach broad group of consumers     -need accurate phone

-high response rates                                   numbers

-interviewer can assist in survey               -some consumers view as

-sample can be chosen                              intrusive

                                                                     -need to train interviewers

                                                                     -relatively expensive

                                                                     -may require many callbacks

Consumer-Administered 

-can reach broad group of consumers    -need to train interviewers

-consumers may feel more comfortable  -relatively expensive

 responding to survey                                -great deal of field coordination

-interviewer can assist in survey

-high response rates

-sample can be chosen

Provider-distributed

-inexpensive                                               -only reaches people who

-high response rates                                  appear for services

-convenient                                                 -may bias respondents to

-interviewer can assist in survey               give favorable responses

                                                                     -sample may not be representative

Populations to Survey:

1. Re-coding all "not applicable" (N/A) responses to missing responses.
2. Calculating a mean across the items of each domain. Means should only be calculated if at least 2/3 of the items have been answered (are not missing).
3. Results can be reported for each domain as a percentage of individuals agreeing or strongly agreeing with the items in the domain (% with a mean score less than 2.5).