Appendix C

Literature Review

A peer-to-peer surveyor is one approach to administer the consumer survey.  However, questions about the efficacy and potential bias of consumers assisting other consumers in completing MHSIP Consumer-Oriented Mental Health Report Card consumer surveys has been raised.

A review of the literature and a series of interviews with experts in the field of consumer evaluation of services was conducted to determine if there was adequate support for endorsing this methodology.  Twenty-nine individuals were contacted by phone and/or via internet.  The results of this literature search and interview process show significant support for the use of peer-to-peer surveyors as an appropriate and valid methodology.

A summary of the information gathered follows below.

3)  The state of New Jersey utilized consumers as survey interviewers to assess persons with serious mental illness having past long term hospitalizations, currently residing in the community.  The survey focused on consumer perceptions about quality of life in nine domains.  The results of the survey (450 Project) found that clients held positive perspectives about their lives in the community.  The 1996 report on the project states, "A by-product of the survey, perhaps no less significant than the evaluation's results, was the opportunity to utilize consumers as survey interviewers. Consumer participation in this context proved beneficial in a variety of ways and on a number of fronts from the interviewers themselves, to the 450 Project, to the Division of Mental Health Services. Their awareness made it possible to elicit information others with less sensitivity would have found impossible and to complete interviews where others might have failed."

4)  Sinikka McCabe and Rae Unzicker report, "The involvement and input of consumers as evaluators and researchers is another significant aspect of note (in mature mental health systems). . . both New York State and the City of Philadelphia employ consumer/survivors to conduct their studies ("What Makes Mental Health Programs User Friendly?", 1993). Tanzman (1993) reports on eighteen studies that used consumers or former patients to collect data on consumer preferences.  Anagnos, McConnell, Chafetz, and Barto (1993) describe a program in San Francisco that uses community members as well as mental health service consumers to collect data to evaluate clinicai services.   Also, several qualified consumer-researchers on both coasts have begun to influence the research methods and the research agenda of many agencies.  The Center for Mental Health Services has involved consumer/survivor-researchers in the Mental Health Systems improvement project, an effort to improve state level mental health management information systems."

5)  Dr. Jean Campbell of the Missouri Mental Health Institute has conducted research on consumer involvement in assessing services.  In her book Towards Collaborative Mental Health Outcomes Systems she states, "In mental health services research, meaningful inclusion of consumers has enriched the process of good science by illuminating important discrepancies between perceptions of consumers and professionals."  She supports her view about the value of using consumers in research and evaluation by citing Continuous Quality Improvement concepts where "Feedback from customers provides the data necessary for establishing a reiterative process called Continuous Quality lmprovement."  She notes Participatory Action Research (PAR) which involves subjects in becoming co-researchers through a repetitive and cyclical process of diagnosis,  analysis, action, and evaluation with constant feedback loops.  PAR stresses respect for and reliance on the needs and expertise of the people under study from start to finish.  She cites seven research studies to support her statements.  She reviews Constituency-Oriented Research and Dissemination and Pioneer Dialogues as consumer involvement approaches to assess services.

6)  Kathleen Stacy, et. al, in the article "Breathing Life Irto Collaborative Research" cites collaborative researchers (Erlandson, Harris, Skipper, & Allen, 1993) conclusion this type of research "takes responsibility for inviting empowerment, being educative, and facilitating connection."  Stacy concludes, "Collaborative research in mental health also goes beyond many qualitative methods of research which do not capture the richer meanings that fall both within and outside of numbers.  In many qualitative approaches to research, researchers still remain the experts on what is to be researched and what the outcomes then mean about the participants.  In collaborative research, participants as "co-researchers" shape what is most important to be researched and how the research occurs, and then influence what this means about them as people and the experiences, therapy or program in which they have been involved.

7)  The 1996 National Health Law Program study on state's Medicaid Managed Care programs noted the following:

1. The federal Medicaid Act requires three methods of consumer involvement:  consumer representation on medical care advisory committees, the employment of low-income consumers as community-service aides or in the administration of the state plan, and fair hearings before the state Medicaid agency.

2. Regardless of whether consumer involvement mechanisms are legally required or voluntarily initiated, a fairly uniform set of mechanisms are being used to involve Medicaid recipients in managed care.  Each state reports that it uses at least one consumer involvement mechanism.

3. Telephone hotlines and consumer surveys are used in most states and by some health plans.  Interest in consumer surveys and report cards as tools for consumer plan selection is rapidly expanding.  In particular, the need for consumer involvement in the development of the survey instrument is achieving wide-spread recognition.

4. Reports from Colorado's and New Mexico's MHSIP surveys indicate that consumers needed and welcomed assistance with the Survey.  Judy Hall of Colorado stated that Consumer Surveyors "report that many consumers they talk with are hesitant, have difficulty, or are unable to fill out the survey without some assistance."

5. Discussion with numerous researchers about potentiaI bias in the use of consumer surveyors indicated (with one exception*) that bias of that type was not a factor.   Most cited advantages to consumer assisted survey responses.

* Arthur R. Smith, President of Advantage Management, stated that bias is a factor in consumer assisted survey results.  He states, "It truly needs to be an independent evaIuation."  No research was cited to support his opinion.

Adams, N., "The Role and Impact of Mental Health Consumers in Gathering Consumer Based Survey Data," Paper for Conference on Mental Health Statistics, May 1997

Campbell, J. "Towards Collaborative Mental Health Outcomes Systems." In New Directions For Mental Health Services, no.71, Fall 1997, Jossey-Bass Publishers.

Erlandson, D., Harris, E., Skipper, B. & Allen, S. (1993). Doing naturalistic inquiry: a guide to methods. Newbury Park, Ca: Sage.

McCabe,S., Unzicker, R., "Changing Roles of Consumer/Survivors in Mature Mental Health Systems", New Directions for Mental Health Services, No. 66, Summer 1995, Jossey-Bass Publishers

Minsky, J., Riesser, G., Duffy, M. "The Eye of the Beholder: Housing Preferences of Inpatients and Their Treatment Teams." In Psychiatric Services, Vol. 46 No.2, Februrary 1995.

Perkins, J., Olson, K., Rivera, L., Skatrud, J .,"Making the Consumer's Voice Heard in Medicaid Managed Care, Consumer’s Voice Publication: NHeLP, 10/29/97

Polowczyk, D., Brutus, M., Orvieto, A. Vidal, J., Cipriani, D. "Comparision of Patient and Staff Surveys of Consumer Satisfaction." In Hospital and Community Psychiatry, Vol. 44, No.6, June 1993.

Reisser, G. "Consumer Satisfaction in New Jersey's 450 Project: Consumers as Both Interviewers and Subjects," Abstract: NJPRA conference presentation.

Stacey K., Wood, A. et.al., "Look at our change since we’ve been here/Breathing Life Into Collaborative Research", Collaborative Outcomes Research Project Team Paper, September 26,1997.

Researchers/practitioners Contacted

Carol Mowbry, Michigan University, School of Education
Erna Koch, Harrison Institute, Georgetown Unviersity
Paulo Delvechio, Office of Consumer Affairs, Center for MH Services, SAMHSA
Randy Kaplan, Ph.D. Genesis Center, Manchester, CT
Arlene Stanton, Ph.D. Center of Mental Health Policy & Services Research, Phila, PA
Joe Marrone, Technical Assistance, Boston Children's Hospital/Harvard
John Morris, University of South Carolina, School of Medicine
Dan Chandler, Consultant, California Department of Mental Health
Randy Kaplan, Ph.D. Genesis Center, Manchester, CT
Tom Trabin, Ph.D. Insititute for Behavioral Health, Tiburon, CA
Paul Carling, Ph.D. Trinity College of Vermont
Dave Mahrer, Ph.,O., Cultural Action, Columbia, South Carolina
Rhonda Baldwin, Behavioral Services, Phoenix, Arizona
Jean Campbell,Ph.D. Research Assistant Professor, Missouri lnstitute of Mental Health
Geraldine Dallek, Families USA, Wash.,D.C.
Laurie Curtis, Center for Community Change, Madison, Wisconsin
Rob Netucik, Health Care for All, Boston, Mass.
Judy Hall, MHSIP Colorado
Gayle Bluebird, Broward Health Planning Council, Florida
Lorrefla Ferry, Consumer Affairs, Philadelphia , PA.
Pat Carvallo, Behavioral Services, Phoenix, Arizona
Jane Perkins, National Health Law Project, North Carolina
Kristi Olson, National Health Law Project, North Carolina
Lynda Lowe, Georgia Legal Services
Bob Bernstein, Bazalon Center for Mental Health Law
Deborah Torres, Health Action NM
Patty Bear Paw, Health Action NM
Larry Belcher, Director West Virginia Mental Health Consumers Assoc