Because preexisting client characteristics (e.g., employment history, age, physical and mental health status etc.) are significant determinants of employment, variations in employment rates among providers are – to some extent – dependent on the composition of clients served. Meaningful comparisons of provider performance require the use of case-mix adjustment methodologies to isolate the provider effect.

In brief, case-mix adjustment is accomplished through the following steps (The Lewin Group, 1998):

Step 1: Develop a model of the outcome of interest (e.g., employment) using client

characteristics. Where the outcome is dichotomous (as it is in the analyses

described in this paper), logistic regression is used to model the outcome.

Step 2: Identify the "provider effect" by either:

1. Generating predicted outcomes for clients based on the model previously described (see #1 above) and ranking providers by differences between observed and predicted outcomes; or,

 

2. Including a "dummy" variable for each provider into the model based on client characteristics previously describe (see #1 above). The use of "dummy" variables require that a single provider be selected as a reference for comparison (i.e., one of the providers must be selected as the "reference" and the other providers are described in comparison to the "reference" provider). The selection of the "reference" provider is critical to meaningful interpretation of the "dummy" coefficients. Using a provider with "average" performance as the reference provider is attractive because the "dummy" coefficients reflect how much the other providers are better (or worse) than "average". The methodology for identifying an "average" provider to use as reference is discussed in the analytic methods section.

Case-Mix Adjusted Provider Rankings:

Two methods were used to identify exceptional (superior and inferior) outpatient substance abuse treatment providers:

Method 1: Each provider’s observed outcomes were compared to their expected outcomes in the following manner:

Step 1: A logistic model for each employment outcome (e.g., any employment in the year following discharge from substance abuse treatment) was developed using client characteristics as predictors.

4

Step 2: For each individual, the predicted probability of employment in the year following discharge from outpatient substance abuse treatment was generated from the logistic model.

 

Step 3: The predicted probabilities of employment for all clients of a provider were summed to compute the number of each provider’s clients that could be expected to be employed in the year following discharge from outpatient substance abuse treatment.

 

Step 4: Providers for whom there was a statistically significant difference between the observed and expected number of employed clients were identified. A provider was considered exceptional (good or bad) if the provider’s observed outcomes differed from the expected outcome by 1.96 or more standard deviations. A z-score representing the number of standard deviation units by which the observed outcome differed from the expected outcome was calculated as:

 

z=((O-E)-.05) / (sum of p_I(1- p_I))^1/2

 

where O equals the observed number of clients employed in the year following discharge, E equals the number of clients expected to be employed in the year following discharge. A positive sign represents outcomes better than expected while a negative sign represents outcomes worse than expected.

 

Method 2: Each provider’s case-mix adjusted outcome was compared to the "average" providers case-mix adjusted outcome in the following manner:

Step 1: A logistic model of employment in the year following discharge – using client characteristics only – was developed.

 

Step 2: "Dummy" indicators for all providers were introduced into the logistic model described in Step 1 (directly above) with the provider who served the largest number of clients (n=936) arbitrarily selected as the reference provider.

 

Step 3: The "dummy" coefficients were then ranked from lowest to highest. Ideally, the provider associated with the median coefficient would be identified as the "average" provider and used as the reference provider. If the provider associated with the median coefficient served relatively few clients, however, the power to identify other providers that differed from the "average" provider would be low. To maximize statistical power, the number of clients served by providers associated with coefficients that were within 5 ranked positions of the median coefficient were examined. The provider with the largest client base was selected to serve as the "average" provider.

 

Step 4: The logistic model of employment described in Step 1 was recomputed including "dummy" indicators for all providers and using the "average" provider as the reference. Exceptionally good (and bad) providers were identified by examining the statistical

 

5

The Lewin Group and Caliber Associates (1998). Performance Measurement for Substance Abuse Treatment Providers and CSAT Knowledge Development and Application. Rockville, MD: Center for Substance Abuse Treatment, National Evaluation Data and Technical Assistance Center (NEDTAC).

significance of the provider's coefficients. A statistically significant coefficient (p<.05) indicates worse than average performance.

Back to Risk Adjustment Home Page