JOURNAL OF PERSONALITY ASSESSMENT, 76(1), 150–168 Copyright © 2001, Lawrence Erlbaum Associates, Inc.
Predicting DSM–IV Cluster B Personality Disorder Criteria From MMPI–2 and Rorschach Data: A Test of Incremental Validity Mark A. Blais Inpatient Psychiatry Service Massachusetts General Hospital and Harvard Medical School
Mark J. Hilsenroth Department of Psychology University of Arkansas
Frank Castlebury Raleigh Psychiatric Associates
J. Christopher Fowler Erik H. Erikson Institute for Training and Research Austen Riggs Center Stockbridge, Massachusetts
Matthew R. Baity Department of Psychology University of Arkansas
Despite their frequent conjoint clinical use, the incremental validity of Rorschach (Rorschach, 1921/1942) and MMPI (Hathaway & McKinley, 1943) data has not been adequately established, nor has any study to date explored the incremental validity of these tests for predicting Diagnostic and Statistical Manual of Mental Disorders (4th ed. [DSM–IV]; American Psychiatric Association, 1994) personality disorders (PDs). In a reanalysis of existing data, we used select Rorschach variables and the MMPI PD scales to predict DSM–IV antisocial, borderline, histrionic, and narcissistic PD criteria in a sample of treatment-seeking outpatients. The correlational findings revealed a
INCREMENTAL VALIDITY
151
limited relation between Rorschach and MMPI–2 (Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer, 1989) variables, with only 5 of 30 correlations reaching significance (p < .05). Hierarchical regression analyses showed that both the MMPI and Rorschach data add incrementally in the prediction of DSM–IV borderline and narcissistic PD total criteria scores. The findings were less clear for the incremental value of Rorschach and MMPI–2 data in predicting the total number of DSM–IV histrionic PD criteria, which were best predicted by Rorschach data, and antisocial PD criteria, which were best predicted by MMPI–2 data. In addition to providing evidence of the incremental validity of Rorschach data, these findings also shed light on the psychological characteristics of the DSM–IV Cluster B PDs.
The versions of the Minnesota Multiphasic Personality Inventory (MMPI, Hathaway & McKinley, 1943; MMPI–2, Butcher, Dahlstrom, Graham, Tellegen, & Kaemmer, 1989) and the Rorschach (Rorschach, 1921/1942) continue to be among the most popular psychological instruments for both clinicians and researchers interested in measuring personality (Lubin, Larsen, & Matarazzo, 1984; Lubin, Wallis, & Paine, 1971; Piotrowski & Keller, 1984, 1989; Piotrowski, Sherry, & Keller, 1985; Watkins, Campbell, Nieberding, & Hallmark, 1995). Despite their ongoing clinical popularity and the prodigious number of published studies using these instruments (estimated to be more than 16,000; Ganellen, 1996a, 1996b), important questions remain regarding the interrelation between the Rorschach and the MMPI–2 (Archer & Krishnamurthy, 1997; Ganellen, 1996a, 1996b; Meyer, 1993, 1999). Furthermore, the utility of combining data from these tests to predict relevant non-test-based criterion variables, such as treatment outcome or diagnoses, remains unclear. Archer and his colleagues (Archer, 1996; Archer & Gordon, 1988; Archer & Krishnamurthy, 1993a, 1993b, 1997; Krishnamurthy, Archer, & House, 1996) have both reviewed the existing literature and conducted a series of studies exploring the interrelation between the Rorschach and the MMPI and MMPI–2. In their review of the 37 Rorschach and MMPI studies that existed at that time, Archer and Krishnamurthy (1993b) found that 51% of the studies reported nonsignificant findings and an additional 22% yielded only weak associations across the two tests. For the remaining studies (27%) that reported significant findings, the reported correlations tended to be modest (rs = .24–.34), and the positive findings were typically limited to a small subgroup of the total variables studied. The results of this review and his own empirical studies have persuaded Archer to conclude that conceptually similar MMPI scales and Rorschach variables are essentially unrelated (Archer & Krishnamurthy, 1997). Meyer (1993, 1996, 1997, 1999) added to our understanding of Archer’s findings by correctly pointing out that correlating Rorschach and MMPI data represents a form of cross-method correlation and that such correlations are typically modest in nature. In addition, Meyer developed a more sophisticated methodology for investigating Rorschach and MMPI relations. Using this new methodology,
152
BLAIS ET AL.
Meyer (1993, 1997) demonstrated that when Rorschach and MMPI data are grouped according to response style (so that only respondents with similar response styles on both tests are studied), strong cross-test associations are found. Meyer (1999) recently replicated his earlier work, again showing strong cross-test association but this time using new response-style grouping criteria. The results from this series of studies suggests that “the convergence of MMPI and Rorschach constructs is tied to broad stylistic qualities related to how patients interact with both assessment methods” (Meyer, 1999, p. 32). Although Meyer’s work represents a promising evolution in this area of research, his findings require independent replication (Archer, 1996; Archer & Krishnamurthy, 1999), and their clinical application also needs to be explored. The absence of clearly demonstrated relations between Rorschach and MMPI data thought to be measuring conceptually similar constructs does not mean that these data cannot be combined to increase their predictive validity. It is well established that uncorrelated but valid measures combine to provide the best predictors of a criterion variable (Meehl & Hathaway, 1946). Incremental validity has long been recognized as a property necessary for a clinical diagnostic instrument to possess (Sechrest, 1963). However, the question of whether Rorschach and MMPI data can be combined to improve the prediction of a desired outcome variable has been infrequently studied (Archer & Krishnamurthy, 1997; Ganellen, 1996a, 1996b; Meyer, 1996). This is a puzzling omission in the research literature, one that leaves open questions about the true clinical validity of these commonly conjointly used psychological tests (Acklin, 1993). Two incremental validity studies conducted by Archer and Gordon (1988) and Archer and Krishnamurthy (1997) found little evidence of incremental validity for combined Rorschach and MMPI scales in accurately classifying adolescents as either depressed or schizophrenic. For example, Archer and Gordon found that both the Rorschach Schizophrenia Index and MMPI Scale 8 were somewhat successful in identifying adolescents with a clinical diagnosis of schizophrenia (hit rates of .80 and .76, respectively); however, the combined use of these two indexes did not significantly improve the overall classification rate. In a follow-up study, Archer and Krishnamurthy (1997) obtained highly similar findings showing limited incremental validity for the combined use of MMPI and Rorschach data in classifying either conduct-disordered or depressed adolescents. The limited empirical data supporting the combined use of the Rorschach and MMPI were highlighted by Archer (1996) and Wood, Nezworski, and Stejskal (1997) as a significant weakness in the scientific foundation of the Rorschach. The purpose of this study was to add to the existing incremental validity literature by exploring the ability of Rorschach and the MMPI to predict the criteria for the Diagnostic and Statistical Manual of Mental Disorders (4th ed. [DSM–IV]; American Psychiatric Association, 1994) personality disorders (PDs). Specifically, we used
INCREMENTAL VALIDITY
153
multiple-regression analyses to explore the ability of select Rorschach variables and the MMPI–2 PD scales (Colligan, Morey, & Offord, 1994) to predict ratings of DSM–IV Cluster B PD (antisocial personality disorder [ANPD], borderline personality disorder [BPD], histrionic personality disorder [HPD], and narcissistic personality disorder [NPD]) criteria in a sample of treatment-seeking outpatients with PDs. The Rorschach variables were selected for use in this study on the basis of prior research demonstrating their conceptual or empirical relation to one of the Cluster B PDs. Portions of the data presented in this article were previously reported in a series of Rorschach and MMPI studies (see Baity & Hilsenroth, 1999; Blais, Hilsenroth, & Fowler, 1998; Blais, Hilsenroth, Fowler, & Conboy, 1999; Castlebury, Hilsenroth, Handler, & Durham, 1997; Hilsenroth, Fowler, Padawer, & Handler, 1997); however, all of the multiple-regression analyses performed for this study and the majority of the correlations reported in this article are original to this study.1 This study was distinctive in that it represents the first effort to empirically establish the incremental validity of Rorschach and MMPI–2 data for predicting the DSM–IV Cluster B PD criteria.
METHOD Patients The data used in this study were drawn from an archival search of patients’ files at a university-based outpatient psychology clinic.2 This review included approximately 800 case files covering a 7-year period. A number of previous studies have successfully used chart information to retrospectively rate patients for Axis II diagnostic criteria (Fyer, Frances, Sullivan, Hurt, & Clarkin, 1988; McCann, 1991; Morey, 1985). The selection of cases proceeded in three phases. In the first phase, 217 patients were identified as having been diagnosed with a PD. A team consisting of an advanced clinical psychology doctoral student and a supervising licensed clinical psychologist rendered the original clinical diagnoses on completion of the intake assessment procedure of the clinic. In the second phase of selection, the chart material for these 217 patients was rated for the presence or absence of a DSM–IV PD diagnosis. The presence or absence of a PD was determined through a retrospective review of the patient records, which included an evaluation report, session notes from the first 12 weeks of therapy, and 3-month treatment reviews when available. Information regarding 1Correlations between HPD–Overlapping (O), HPD–Nonoverlapping (N), DEN, and T, for N = 62 were reported in Blais et al. (1998). Also, correlations between NPD–O, NPD–NO, REF, and IDEAL, for N = 62 were reported in Hilsenroth et al. (1997). 2 For a more complete review of the sample selection methodology, see Hilsenroth et al. (1997).
154
BLAIS ET AL.
patient identity, diagnosis, and test data (including all Rorschach and MMPI–2 data) were appropriately masked from the reviewers. Raters in this phase of the study were four advanced doctoral students in an American Psychological Association-approved clinical psychology program. The raters had received special training in the diagnosis of DSM–IV Axis II disorders. Interrater reliability was established through independent ratings of a randomly selected pool of 31 patients; the obtained kappa value was .90 for the presence or absence of a DSM–IV PD. Of the 217 patient case files reviewed in this manner, 57 were found to meet DSM–IV criteria for an Axis II disorder as well as to contain a completed Rorschach and MMPI–2 protocol. The Axis II diagnoses of these 57 patients had the following distribution: ANPD = 11, BPD = 18, HPD = 5, NPD = 10, Cluster A PD = 6, and Cluster C PD = 7. The sample (N = 57) was 53% female and had a mean age of 28 years (SD = 8) at the time of their clinic intake. They averaged 14 years of education (SD = 2.5), and their mean Wechsler (1981) full-scale IQ was 106 (SD = 11). Thirty-three patients were single, 8 were married, 15 had been divorced, and 1 was widowed. In the third phase, the records of these 57 patients were again independently rated on all of the DSM–IV Cluster B PD symptom criteria (ANPD, BPD, HPD, and NPD) with the same case material and methodology outlined earlier. Again, interrater reliability was established by independent ratings of a randomly selected pool of 25 patients. Interrater agreement for the presence or absence of each individual DSM–IV Cluster B symptom criterion were as follows: .86 (ANPD), .80 (BPD), .90 (HPD), and .90 (NPD).
Procedure The Rorschach was originally administered and scored according to the procedures of Exner (1986, 1993). All Rorschach protocols were rescored by Mark J. Hilsenroth, who was blind to previous scores and patient diagnoses. Interrater reliability (Weiner, 1991) was obtained by having J. Christopher Fowler, who was blind to the first coder’s scores and patients’ diagnoses, score 20 randomly selected protocols. The interrater agreement for the Structural Summary Rorschach variables and the Rorschach content scales (described later) were above 80%. All Rorschach protocols used in this study were reviewed for validity; none were found to have fewer than 14 responses and a Lambda above 1.0. The MMPI–2 (Butcher et al., 1989) was administered according to standard instructions. All MMPI–2 protocols were reviewed for validity (see Castlebury et al., 1997, for details). The MMPI–2 Cluster B PD (ANPD, BPD, HPD, NPD) scales were scored and included as variables in this study (Colligan et al., 1994). Originally developed by Morey, Waugh, and Blashfield (1985) to assess the DSM–III PDs, the MMPI PD scales have frequently been used in personality re-
INCREMENTAL VALIDITY
155
search (Castlebury et al., 1998; Dubro, Wetzler, & Kahn, 1988; McCann, 1989, 1991; Morey, 1986; O’Maille & Fine, 1995; Trull, 1993; Wise, 1996) and represent the only PD scales currently available for the MMPI. Two versions of the MMPI PD scales, one containing O (overlapping) items and the other having no O items, were developed by Morey et al. The O version of the PD scales allows MMPI items that are relevant to multiple DSM PDs to be scored on multiple PD scales (the scales have O items). The NO version of the MMPI PD scales focus on the core or defining features of the DSM PDs and restricts item scoring to a single scale (these scales have no O items). Across a number of research studies, the MMPI PD scales have demonstrated adequate reliability and validity (see O’Maille & Fine, 1995, for a review of these data). The impact of the MMPI restandardization and item revision on the PD scales was explored by Colligan et al. While also presenting updated norms for the PD scales, Colligan et al. found the MMPI–2 version of the PD scales to be comparable to the original scales. Wise (1996) also found the MMPI–2 PD scales to be comparable to the original scales. Both the O and NO versions of the MMPI–2 PD scales were included in the analyses.
Rorschach Variables All Rorschach variables used in this study were selected, a priori, on the basis of either prior theoretical or empirical linkage to DSM–IV Cluster B PD criteria. Although the Rorschach variables used in this study are briefly reviewed next, all relevant citations are provided, allowing the interested reader to explore the rationale for their selection in greater detail.
ANPD. The Rorschach Comprehensive System (CS; Exner, 1986, 1993) variables T, PER, MOR, and AG, along with Holt’s (1977) primary process aggression variable (A1) and Gacono and Meloy’s (1991) measure of Aggressive Content (AgC) have all been conceptually and empirically associated with DSM–IV ANPD. (See Baity & Hilsenroth, 1999, and Gacono & Meloy, 1994, for interrater reliability and a more complete explanation of the variables, including their scoring.)
BPD. Rorschach measures of aggression, including Holt’s (1977) A1 variable, the Rorschach Oral Dependency (ROD) scale (Masling, Rabie, & Blondheim, 1967), quality of object relations using the Mutuality of Autonomy Scale (MOAS; Urist, 1977), and the Lerner Defense Scale (LDS; Lerner, 1991) defenses of splitting (SPLIT), and devaluation (DEVAL) have all been conceptually and empirically linked to the DSM–IV BPD. In this study, the MOAS–Highest score (MOAS–H) was
156
BLAIS ET AL.
used, which is the single highest or most disturbed MOAS score from a given protocol. (See Blais et al., 1999, for interrater reliability and a more complete explanation of the variables, including their scoring.)
HPD. The Rorschach CS variables (Exner, 1986, 1993) FC + CF + C, T, and the ROD scale (Masling et al., 1967), have all been conceptually and empirically linked to DSM–IV HPD. (See Blais et al., 1998, for interrater reliabilities and a fuller explanation of the rationale underlying the selection of these variables.)
NPD. The Rorschach CS (Exner, 1986, 1993) REF variable and the LDS (Lerner, 1991) variable of Idealization (IDEAL) have been conceptually and empirically associated with DSM–IV NPD. (See Hilsenroth et al., 1997, for interrater reliability and a more complete explanation of the variables, including their scoring.)
Statistical Analyses Before conducting our analyses, we correlated the Rorschach variables with the total number of Rorschach responses (R). For any Rorschach variable that showed a trend toward significant (p = .10) correlation (T, ROD, REF, and AgC), the effect or R was partialed out in all reported analyses. Although multivariate analyses have been shown to be robust to violations normality (Stevens, 1996), following Viglione’s (1997) recommendation we reviewed the descriptive data for each of the Rorschach variables to determine the nature of their distributions. Curran, West, and Finch (1996) suggested that distributions with a skew of 2.0 or more or kurtosis of 7.0 or more should be considered moderately nonnormal in shape. Applying these criteria, we found three Rorschach variables, SPLIT, REF, and MOR, to be sufficiently nonnormal in their distribution to potentially effect the multivariate analyses. We subjected these three variables to a log transformation in an attempt to improve their distributions. The distributions for two of the variables (MOR and REF) were successfully transformed, although the distribution for one variable (SPLIT) was not be sufficiently improved by this process and was dropped from the study. Table 1 presents the descriptive data for all retained Rorschach variables. Following these transformations, correlations (Pearson rs) were obtained for the Rorschach variables and the appropriate MMPI–2 O and NO PD scales. Next, we summed the Cluster B (ANPD, BPD, HPD, NPD) symptom ratings to produce total scores (equaling the total number of criteria met) for each of the Cluster B PDs. This procedure produced a dimensional representation of each of the four Cluster B PDs. Substantial empirical (see Blais & Norman, 1997) and theoretical (see Widiger, 1991) support exists for the use of dimensional PD scores.
157
INCREMENTAL VALIDITY TABLE 1 Descriptive Data for the Rorschach Variables Used in the Multivariate Analyses Rorschach Variable Ta RODa REF a,b A1 AG MORb PER AgC a IDEAL FC + CF + C MOAS–H DEVAL
M
SD
MIN
MAX
Skew
Kurtosis
0.05 0.16 0.12 0.51 0.91 0.35 1.80 2.75 3.50 4.03 4.84 7.17
0.06 0.11 0.21 1.04 1.15 0.28 2.10 1.81 3.73 2.90 1.91 6.47
0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 1.0 0.00
0.25 0.42 0.78 3.0 4.0 1.08 9.0 8.0 14.0 12.0 7.0 23.0
1.28 0.31 1.60 1.14 0.97 0.27 1.41 0.66 0.87 1.20 –0.68 0.87
1.45 –0.54 1.36 0.69 –0.30 –0.79 1.74 0.31 0.25 1.03 –1.04 –0.06
Note. N = 57. MIN = minimum; MAX = maximum; Rorschach variables: T = Texture (Comprehensive System [CS]; Exner, 1986, 1993); ROD = Rorschach Oral Dependency scale (Masling et al., 1967); REF = Reflections (CS); A1 = Primary Process Aggression (Holt, 1977); AG = Aggressive Movement (CS); MOR = Morbid (CS); PER = Personalized (CS); AgC = Aggressive Content (Gacono & Meloy, 1994); IDEAL = Idealization (Lerner Defense Scale [LDS]; Lerner, 1991); FC + CF + C = Form Color + Color Form + Color (CS); MOAS–H = Mutuality of Autonomy scale single highest or most disturbed score (Urist, 1977); DEVAL = Devaluation (LDS). aThe effect of R was partialed out of this variable. bLog-transformed Rorschach variables.
Multivariate analyses. Using the Cluster B PD total symptom scores as target or criterion variables, we sought to explore the incremental validity of Rorschach and MMPI–2 test data using two forms of multiple regression: stepwise multiple regression and hierarchical regression. First, we ran a series of four separate stepwise multiple-regression analyses in which all predictor variables (MMPI–2 scales and Rorschach variables) were available for entry into the equation in a single block. In this form of multiple-regression analysis, the decision concerning which variables are added or subtracted at a given step is determined on the basis of the empirical relation among the variables (Cohen & Cohen, 1983; Licht, 1995). Second, we conducted two series of hierarchical regression analyses to determine the incremental validity offered by each test in the prediction of the Cluster B PD criteria totals. Hierarchical regression analysis has been endorsed as the appropriate statistical method for determining incremental validity (Cohen & Cohen, 1983). Hierarchical regression analysis uses a series of simultaneous multiple-regression analyses in which one or more new predictors are added to those used in the previous analysis; the decision concerning which variables to add at each point in the series is made by the investigator (Cohen & Cohen, 1983; Licht, 1995). The analyses used in this study were modeled after those presented by Ben-Porath, Butcher, and Graham (1991). The hi-
158
BLAIS ET AL.
erarchical regression analyses were performed in the following manner. First, a series of four hierarchical regressions were conducted (one for each DSM–IV Cluster B PD) with the appropriate MMPI–2 PD scales (both the O and NO versions of the target disorder) and the a priori selected Rorschach variables as predictors. The predictor variables were entered into the regression analyses as two separate blocks, with the MMPI–2 scales entered as Block 1, followed by the Rorschach variables, which were entered as Block 2. Thus, all scales that were entered into the regression equation in the second block contributed incrementally to the prediction of the DSM–IV PDs (criteria totals). We then ran these four additional analyses with the order of the blocks switched: The Rorschach variables were entered stepwise in Block 1, and the MMPI–2 scales were entered stepwise in Block 2. In all the regression analyses, we entered the independent predictors (Rorschach variables and MMPI–2 scales) into the regression equation using a forward stepwise method until the point at which they no longer contributed significantly to the predictive power (p < .05) of the equation. Two measures of incremental validity were available from the regression analyses: the coefficient of determination (R2) and the adjusted R2. Both of these coefficients reflect the amount of variance in the dependent variable that can be accounted for by the independent variable. However, the formula for R2 is not designed to include the number of variables, so when variables are added to a regression equation, the value of R2 will invariably increase. Adjusted R2 is a more conservative estimate than R2 because it takes into account the addition of further variables when estimating the variability explained by the independent predictors. Given that the adjusted R2 is a more conservative statistic, we chose to report it in place of R2 in this article.
RESULTS Table 2 shows that five significant correlations (p < .05) were obtained between the matched Rorschach and MMPI–2 PD variables. From the correlations examined between the Rorschach and MMPI–2 variables selected to measure the DSM–IV ANPD criteria totals, AgC was the only significant correlation with both the MMPI–2 ANPD–O and ANPD–NO scales (rs = .31 and .30, respectively). Among the test scores selected to measure the DSM–IV BPD criteria totals, no significant correlations were obtained. One significant correlation was obtained among the test variables selected to tap the DSM–IV HPD criteria ROD with MMPI–2 HPD–NO (r = –.26). Of the variables selected to measure the DSM–IV NPD criteria, the Rorschach REF and IDEAL scores were significantly correlated with the MMPI–2 NPD–NO scale (rs = .31 and .27, respectively). Table 3 provides the results of the four separate stepwise multiple-regression analyses in which all predictors were included simultaneously. Table 3 shows that
INCREMENTAL VALIDITY
159
TABLE 2 Correlations for Select Rorschach Variables and the MMPI–2 PD Scales MMPI–2 Cluster B PD Scale ANPD Rorschach Variable Ta PER MORb AG A1 AgC a RODa MOAS–H DEVAL FC + CF + C REF a,b IDEAL
BPD
O
NO
.02 .24 .15 –.01 .04 .31*
.02 .23 .13 –.03 .02 .30*
O
HPD NO
.16
.04
–.17 .10 .02
–.02 .04 .00
NPD
O
NO
.20
.21
–.22
–.26*
.09
.12
O
NO
.23 .22
.31* .27*
Note. N = 57. MMPI–2 = Minnesota Multiphasic Personality Inventory–2; PD = personality disorder; ANPD = antisocial personality disorder; BPD = borderline personality disorder; HPD = histrionic personality disorder; NPD = narcissistic personality disorder; O = overlapping version; NO= nonoverlapping version; Rorschach variables: T = Texture (Comprehensive System [CS]; Exner, 1986, 1993); PER = Personalized (CS); MOR = Morbid (CS); AG = Aggressive Movement (CS); A1 = Primary Process Aggression (Holt, 1971); AgC = Aggressive Content (Gacono & Meloy, 1994); ROD = Rorschach Oral Dependency scale (Masling et al., 1967); MOAS–H = Mutuality of Autonomy scale single highest or most disturbed score (Urist, 1977); DEVAL = Devaluation (Lerner Defense Scale [LDS]; Lerner, 1991); FC + CF + C = Form Color + Color Form + Color (CS); REF = Reflections (CS); IDEAL = Idealization (LDS). aThe effect of R was partialed out of analyses that employed this variable. bLog-transformed Rorschach variable. *p < .05.
the MMPI–2 NPD–O scale (Step 1 adjusted R2 = .25) and Rorschach REF scores (Step 2 adjusted R2 = .33) were both nonredundant significant predictors of the DSM–IV NPD total criteria. The MMPI–2 BPN–O scales (Step 1 adjusted R2 = .18), the Rorschach variables ROD (Step 2 adjusted R2 = .30) and MOAS–H (Step 3 adjusted R2 = .43) and DEVAL (Step 4 adjusted R2 = .48) were all nonredundant significant predictors of the DSM–IV BPD total criteria. The Rorschach variable ROD was the only negative predictor of the DSM–IV BPD total criteria (standardized β = –.41). The Rorschach variables FC + FC + C (Step 1 adjusted R2 = .16) and T (Step 2 adjusted R2 = .24) were all nonredundant significant predictors of the DSM–IV HPD total criteria. In predicting the DSM–IV ANPD total criteria, the MMPI–2 ANPD–O scale (Step 1 adjusted R2 = .29) and the Rorschach MOR scores (Step 2 adjusted R2 = .33) were both nonredundant significant predictors.
160
BLAIS ET AL.
TABLE 3 Summary of the Stepwise Multiple-Regression Analyses With the MMPI–2 PD Scales and Rorschach Variables Used to Predict DSM–IV Cluster B PD Criteria Totals Dependent Variable (DSM–IV PD)
Predictor Variables (Rorschach and MMPI–2)
ANPD ANPD BPD BPD BPD BPD HPD HPD NPD NPD
ANPD–O MORa BPD–O RODb MOAS–H DEVAL FC + CF + C Tb NPD–O REF a,b
Step
R
Adjusted R2
Standardized β
1 2 1 2 3 4 1 2 1 2
.55 .59 .44 .57 .67 .72 .42 .52 .51 .60
.29 .33 .18 .30 .43 .48 .16 .24 .25 .33
.58 –.23 .34 –.41 .30 .24 .38 .30 .44 .31
Note. N = 57. All multiple-regression analyses were significant at p < .02 or greater. MMPI–2 = Minnesota Multiphasic Personality Inventory–2; PD = personality disorder; DSM–IV = Diagnostic and Statistical Manual of Mental Disorders (4th ed.; American Psychiatric Association, 1994); ANPD = antisocial personality disorder; O = overlapping version; BPD = borderline personality disorder; HPD = histrionic personality disorder; NPD = narcissistic personality disorder; Rorschach variables: MOR = Morbid (Comprehensive System [CS]; Exnter, 1986, 1993); ROD = Rorschach Oral Dependency scale (Masling et al., 1967); MOAS–H = Mutuality of Autonomy scale single highest or most distributed score (Urist, 1977); DEVAL = Devaluation (Lerner Defense Scale; Lerner, 1991); FC + CF + C = Form Color + Color Form + Color (CS); T = Texture (CS); REF = Reflections (CS). aLog-transformed Rorschach variables. bThe effect of R was partialed out of analyses that used this variable.
However, the Rorschach MOR score was negatively associated with the DSM–IV ANPD total criteria (standardized β = –.23). Table 4 provides the results of the first series of four hierarchical regression analyses. In these analyses, the MMPI–2 scales were entered first in Block 1, and the Rorschach variables were entered second, in Block 2. Table 4 shows that for predicting the DSM ANPD criteria totals, the MMPI–2 ANPD–O scale entered from Block 1, and the Rorschach variable MOR were entered from Block 2, adding incrementally to the regression equation. For predicting the DSM–IV BPD criteria totals, the MMPI–2 BPD–O scale entered from Block 1, whereas the Rorschach variables ROD (Step 1), MOAS–H (Step 2) and DEVAL (Step 3) all entered from Block 2. Together these three Rorschach variables accounted for an additional 30% of the variance in the BPD criteria total score. For predicting the DSM–IV HPD criteria, the MMPI–2 HPD–O scale was entered from Block 1, whereas the Rorschach variables FC + F + C (Step 1) and T (Step 2) were entered from Block 2. The Rorschach variables FC + CF + C and T combined to account for an additional 19% of the variance in the HPD criteria totals beyond that accounted for by the MMPI–2 HPD–O scale. In the last analyses, the MMPI–2 NPD–O scale was
161
INCREMENTAL VALIDITY
entered from Block 1, whereas the Rorschach variable REF was entered from Block 2. Table 5 contains data from the second series of hierarchical regression analyses. In these analyses, the order of the variables was switched, with the Rorschach variables entered at Block 1 and the MMPI–2 scales entered at Block 2. Table 5 shows that, for the ANPD criteria, no Rorschach variable was entered from Block 1, whereas the MMPI–2 ANPD–O scale was entered from Block 2 and accounted for 28% of the variance in the ANPD criteria total. For the DSM–IV BPD criteria, three Rorschach variables were entered from Block 1 (ROD, MOAS–H, and DEVAL), whereas the MMPI–2 BPD–O scale was entered from Block 2. The MMPI–2 BPD–O scale accounted for additional 11% of the variance in the BPD TABLE 4 Hierarchical Regression Analyses With MMPI–2 PD Scales Entered Stepwise in Block 1 and the Rorschach Variables Entered Stepwise in Block 2 to Predict DSM–IV PD Criteria
Criterion Variable (DSM–IV PD) ANPD Block 1 Block 2 BPD Block 1 Block 2
HPD Block 1 Block 2 NPD Block 1 Block 2
Predictor Variable (MMPI–2 Cluster B PD Scale and Rorschach)
Step
Adjusted R2
Standardized β
ANPD–O MORa
1 1
.28 .05b
.55 –.23
BPD–O RODc MOAS–H DEVAL
1 1 2 3
.18 .12b .25 .30
.44 –.41 .30 .25
HPD–O FC + CF + C Tc
1 1 2
.08 .14b .19
.22 .33 .26
NPD–O REF b,c
1 1
.25 .09b
.44 .31
Note. N = 57 for all analyses. All multiple-regression analyses were significant at p < .02 or greater. MMPI–2 = Minnesota Multiphasic Personality Inventory–2; PD = personality disorder; DSM–IV = Diagnostic and Statistical Manual of Mental Disorders (4th ed.; American Psychiatric Association, 1994); ANPD = antisocial personality disorder; O = overlapping version; BPD = borderline personality disorder; HPD = histrionic personality disorder; NPD = narcissistic personality disorder; Rorschach variables: MOR = Morbid (Comprehensive System [CS]; Exnter, 1986, 1993); ROD = Rorschach Oral Dependency scale (Masling et al., 1967); MOAS–H = Mutuality of Autonomy scale single highest or most distributed score (Urist, 1977); DEVAL = Devaluation (Lerner Defense Scale; Lerner, 1991); FC + CF + C = Form Color + Color Form + Color (CS); T = Texture (CS); REF = Reflections (CS). aLog-transformed Rorschach variables. bThe adjusted R2 for Block 2 variables is reported as the incremental improvement over the Block 1 values. cThe effect of R was partialed out of analyses that employed this variable.
162
BLAIS ET AL.
TABLE 5 Hierarchical Regression Analyses With Rorschach Variables Entered Stepwise in Block 1 and the MMPI–2 PD Scales Entered Stepwise in Block 2 to Predict DSM–IV PD Criteria
Criterion Variable (DSM–IV PD) ANPD Block 1 Block 2 BPD Block 1
Block 2 HPD Block 1 Block 2 NPD Block 1 Block 2
Predictor Variable (MMPI–2 Cluster B PD Scale and Rorschach)
Step
Adjusted R2
Standardized β
No variable entered ANPD–O
1
.28
.55
RODa MOAS–H DEVAL BPD–O
1 2 3 1
.16 .33 .37 .11b
–.41 .30 .24 .34
FC + CF + C Ta No variable entered
1 2
.16 .24
.38 .30
REF a,c NPD–O
1 1
.16 .18b
.31 .44
Note. N = 57 for all analyses. All multiple-regression analyses were significant at p < .02 or greater. MMPI–2 = Minnesota Multiphasic Personality Inventory–2; PD = personality disorder; DSM–IV = Diagnostic and Statistical Manual of Mental Disorders (4th ed.; American Psychiatric Association, 1994); ANPD = antisocial personality disorder; O = overlapping version; BPD = borderline personality disorder; HPD = histrionic personality disorder; NPD = narcissistic personality disorder; Rorschach variables: ROD = Rorschach Oral Dependency scale (Masling et al., 1967); MOAS–H = Mutuality of Autonomy scale single highest or most distributed score (Urist, 1977); DEVAL = Devaluation (Lerner Defense Scale; Lerner, 1991); FC + CF + C = Form Color + Color Form + Color (Comprehensive System [CS]; Exner, 1986, 1993); T = Texture (CS); REF = Reflections (CS). aThe effect of R was partialed out of analyses that employed this variable. bThe adjusted R2 for Block 2 variables is reported as the incremental improvement over the Block 1 values. cLog-transformed Rorschach variables.
criteria score beyond the Rorschach variables. With the DSM–IV HPD criteria, the Rorschach variables FC + CF + C and T were entered from Block 1, and no MMPI–2 scales were entered from Block 2. For the DSM–IV NPD criteria, the Rorschach variable REF was entered from Block 1, whereas the MMPI–2 NPD–O scale was entered from Block 2. The MMPI–2 NPD–O scale accounted for an additional 18% of the variance in the DSM–IV NPD criteria score beyond the Rorschach variable. DISCUSSION In this study, we sought to explore two related issues: (a) the interrelation between select MMPI–2 scales and Rorschach variables and (b) the incremental validity
INCREMENTAL VALIDITY
163
achieved by combining data from these two assessment instruments for predicting the total number of DSM–IV Cluster B PD criteria (ANPD, BPD, HPD, and NPD). These are two important areas that have not been extensively studied in the assessment literature. The intertest correlational data obtained in this study are quite interesting because they reveal a fairly low degree of association between the matched Rorschach variables and the MMPI–2 PD scales. Only 5 of the 30 correlations were significant (17%), and 1 of these was unexpectedly in the negative direction (ROD with MMPI–2 HPD–NO, r = –.26, p < .04). Our correlational findings are highly consistent with Archer and Krishnamurthy’s (1997) review of the literature, both in terms of the number of significant correlations and their magnitude. Still, the modest magnitude of the correlations is within the range expected, given that they represent cross-method correlations (see Meyer, 1996). With the exception of Meyer’s (1996, 1997, 1999) recent and, in some ways, novel work, our findings are consistent with past research and support the conclusion that MMPI and Rorschach data are generally unrelated or at best modestly related (Archer & Krishnamurthy, 1997). However, the results from the two series of hierarchical regression analyses provide additional important information regarding the diagnostic utility of these two tests. Despite the weak interrelation observed between the two instruments, the multiple- and hierarchical regression analyses clearly indicate that both the Rorschach and MMPI–2 PD test variables are meaningfully related to the DSM–IV Cluster B PD criteria total scores. The multiple-regression analyses (Table 3) showed that, for three of the four target variables (NPD, BPD, and ANPD total criteria scores), when the MMPI–2 and Rorschach variables were presented together (as a single block), a variable from each test was entered into the equation on either the first or second step. Together the combined MMPI–2 and Rorschach data accounted for one third (33% for NPD) to just under one half (48% BPD) of the variance in these two criterion variables. This impressive finding can be seen as supporting the clinical tradition of combining Rorschach and MMPI data in the assessment process. Still, one should not make too much of these findings, because the hierarchical regression analyses provide a more rigorous evaluation of the incremental validity offered by these two instruments. In the first series of hierarchical regression analyses (Table 4), Rorschach variables (entered in Block 2) added significant predictive power to all the DSM–IV Cluster B PD criteria scores beyond that provided by the MMPI–2 PD scales. For example, in the prediction of the DSM–IV BPD total criteria score, three Rorschach variables combined to explain an additional 30% of the variance beyond the MMPI–2 scales. The results from the second series of regression analyses were less clear. When the Rorschach variables were entered first, in Block 1, there was evidence of incremental validity for the MMPI–2 NPD–O and BPD–O scales, which both entered their respective regression equations from Block 2. However, in this series of analyses, no Rorschach variable entered from Block 1 into the equation for predicting the DSM–IV ANPD criteria score, and no MMPI–2 scale
164
BLAIS ET AL.
(MMPI HPD–O and HPD–NO) was entered from Block 2 into the equation for predicting the DSM–IV HPD criteria score. These findings raise questions regarding the strength of the relation of both the Rorschach MOR score and the MMPI–2 HPD scales to their respective target variables. Taken together, the results of these two series of regression analyses provide strong support for the criterion-related validity of three MMPI–2 PD scales (ANPD–O, BPD–O, and NPD–O) and six of the Rorschach variables (ROD, MOAS–H, DEVAL, FC + CF + C, T, and REF) used in this study. Strong support was also obtained for the incremental validity achieved by combining Rorschach and MMPI–2 data in predicting DSM–IV BPD and NPD criteria. Our findings are consistent with the observation that the best data from which to predict a criterion variable are data that are valid but not highly intercorrelated (Meehl & Hathaway, 1946). It is interesting to speculate a little further on the meaning of our findings. For example, both the Rorschach REF response and the MMPI–2 NPD–O scale independently predicted the total number of DSM–IV NPD criteria assessed to patients. The final adjusted R2 for this regression equation was .33, indicating that together these scales accounted nearly one third of the total variance in the DSM–IV NPD criteria assignment. However, as the data in Table 1 show, these scales were not themselves significantly correlated (r = .23). Together these findings indicate that to a substantial degree, material present in a patients’ self-report (the MMPI–2 data) and material that is provided outside of a patient’s awareness (Rorschach responses) reflect important but different components of the DSM–IV conception of narcissism and NPD. The findings for the DSM–IV BPD also show that MMPI–2 and Rorschach data can be combined to improve the prediction of the disorder (or its criteria). These data provide further support for the importance of using multimethod assessment procedures, particularly in the evaluation of personality functioning (Hilsenroth, Handler, & Blais, 1996). One potential limitation to the generalizability of our results is the nature of the sample we used. We used a retrospective chart review to identify potential patients and a comprehensive review of chart material to make the DSM–IV PD diagnoses used as target variables in the study. Although chart review methodology has been successfully used in previous studies of the DSM PDs (Fyer et al., 1988; McCann, 1991; Morey, 1985), such a methodology tends to identify a “prototypic example of PDs” and therefore establishes a “purer” or better defined criterion variable (Zimmerman, 1995). The use of prototypic patients might have enhanced the predictive value and functioning of our test data, making our findings more pronounced than those that might be obtained with a less prototypic sample. However, any enhancement effect would be expected to affect both tests equally. Compared to previous Rorschach and MMPI studies, this study has many unique features. We established a stringent level of reliability both for the Rorschach variables used in the study and the criterion variable (DSM–IV Cluster B
INCREMENTAL VALIDITY
165
PD criteria). This basic requirement has not always been meet by past studies exploring relations between Rorschach and MMPI data (Ganellen, 1996b). Furthermore, the Rorschach variables selected for this study were all either conceptually or empirically associated with the criterion variables. We used MMPI–2 scales that were developed through a conceptual and empirical process to specifically measure the criterion variables, the DSM PDs (unlike the traditional MMPI–2 clinical scales, which have a less clear relation to current psychiatric constructs). Last, our criterion variables, the DSM–IV Cluster B PD criteria total scores, were subject to a rigorous and reliable diagnostic process and therefore may possess greater criterion validity than the more typically used self-report measures. Together these features served to maximize the reliability and validity of all the assessment methods used, increasing our statistical power and allowing underlying relations to reveal themselves. This study provides the first clear support for the incremental validity achievable by combining Rorschach and MMPI–2 data in predicting clinically relevant, nontest, real-world behaviors (the DSM–IV PD criteria). Future work in this area should seek to incorporate the methodological advantages of using reliable and conceptually relevant test data when exploring the incremental validity of these two popular psychological assessment instruments.
ACKNOWLEDGMENTS An earlier version of this article was presented at the annual midwinter meeting of the Society of Personality Assessment, Boston, February 1998. We thank Myra Christensen, Susan Church, Sheila O’Keefe, Karen Toman, and Greta Smith for their various contributions to this study. We also thank the two anonymous reviewers for their constructive and helpful comments.
REFERENCES Acklin, M. W. (1993). Integrating the Rorschach and MMPI in clinical assessment: Conceptual and methodological issues. Journal of Personality Assessment, 60, 125–131. American Psychiatric Association. (1994). Diagnostic and statistical manual of mental disorders (4th ed.). Washington, DC: Author. Archer, R. P. (1996). MMPI–Rorschach interrelationships: Proposed criteria for evaluating explanatory models. Journal of Personality Assessment, 67, 504–515. Archer, R. P., & Gordon, R. A. (1988). MMPI and Rorschach indices of schizophrenic and depressive diagnoses among adolescent inpatients. Journal of Personality Assessment, 52, 276–287. Archer, R. P., & Krishnamurthy, R. (1993a). Combining the Rorschach and MMPI in the assessment of adolescents. Journal of Personality Assessment, 60, 132–140. Archer, R. P., & Krishnamurthy, R. (1993b). A review of MMPI and Rorschach interrelationships in adult samples. Journal of Personality Assessment, 61, 277–293.
166
BLAIS ET AL.
Archer, R. P., & Krishnamurthy, R. (1997). MMPI–A and Rorschach indices related to depression and conduct disorder: An evaluation of the incremental validity hypothesis. Journal of Personality Assessment, 69, 517–533. Archer, R. P., & Krishnamurthy, R. (1999). Reply to Meyer on the convergent validity of the MMPI and Rorschach. Journal of Personality Assessment, 73, 319–321. Baity, M. R., & Hilsenroth, M. J. (1999). Rorschach aggression variables: A study of reliability and validity. Journal of Personality Assessment, 72, 93–110. Ben-Porath, Y., Butcher, J., & Graham, J. (1991). Contribution of the MMPI–2 content scales to the differential diagnosis of schizophrenia and major depression. Psychological Assessment: A Journal of Consulting and Clinical Psychology, 3, 634–640. Blais, M. A., Hilsenroth, M. J., & Fowler, J. C. (1998). Rorschach correlates of the DSM–IV histrionic personality disorder. Journal of Personality Assessment, 70, 355–364. Blais, M. A., Hilsenroth, M. J., Fowler, J. C., & Conboy, C. A. (1999). A Rorschach exploration of the DSM–IV borderline personality disorder. Journal of Clinical Psychology, 55, 1–10. Blais, M., & Norman, D. (1997). A psychometric evaluation of the DSM–IV personality disorder criteria. Journal of Personality Disorders, 11, 168–176. Butcher, J. N., Dahlstrom, W. G., Graham, J. R., Tellegen, A., & Kaemmer, B. (1989). MMPI–2: Minnesota Multiphasic Personality Inventory: Manual for administration and scoring. Minneapolis: University of Minnesota Press. Castlebury, F., Hilsenroth, M., Handler, L., & Durham, T. (1997). Use of the MMPI–2 personality disorder scales in the assessment of DSM–IV antisocial, borderline, and narcissistic personality disorders. Assessment, 4, 155–168. Cohen, J., & Cohen, P. (1983). Applied multiple regression/correlation analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum Associates, Inc. Colligan, R., Morey, L., & Offord, K. (1994). The MMPI/MMPI–2 personality disorder scales: Contemporary norms for adults and adolescents. Journal of Clinical Psychology, 50, 168–200. Curran, P., West, S., & Finch, J. (1996). The robustness of test statistics to nonnormality and specification error in confirmatory factor analysis, Psychological Methods, 1, 16–29. Dubro, A., Wetzler, S., & Kahn, M. (1988). A comparison of three self-report questionnaires for the diagnosis of DSM–III personality disorders. Journal of Personality Disorders, 2, 256–266. Exner, J. (1986). The Rorschach: A comprehensive system: Vol. 1. Basic foundations (2nd ed.). New York: Wiley. Exner, J. (1993). The Rorschach: A comprehensive system: Vol. 1. Basic foundations (3rd ed.). New York: Wiley. Fyer, M., Frances, A., Sullivan, T., Hurt, S., & Clarkin, J. (1988). Comorbidity of borderline personality disorders. Archives of General Psychiatry, 45, 348–352. Gacono, C., & Meloy, J. (1991). A Rorschach investigation of attachment and anxiety in antisocial personality disorder. Journal of Nervous and Mental Disease, 179, 546–552. Ganellen, R. J. (1996a). Exploring the MMPI–Rorschach relationships. Journal of Personality Assessment, 67, 529–542. Ganellen, R. J. (1996b). Integrating the Rorschach and the MMPI–2 in personality assessment. Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Hathaway, S. R., & McKinley, J. C. (1943). The Minnesota Multiphasic Personality Inventory.. Minneapolis: University of Minnesota Press. Hilsenroth, M. J., Fowler, J. C., Padawer, J. R., & Handler, L. (1997). Narcissism in the Rorschach revisited: Some reflections on empirical data. Psychological Assessment, 9, 113–121. Hilsenroth, M., Handler, L., & Blais, M. (1996). Assessment of narcissistic personality disorder: A multimethod review. Clinical Psychology Review, 16, 655–683. Holt, R. R. (1977). A method for assessing primary process manifestations and their control in Rorschach responses. In M. A. Rickers-Ovsiankina (Ed.), Rorschach psychology (2nd ed., pp. 375–420). Huntington, NY: Krieger.
INCREMENTAL VALIDITY
167
Krishnamurthy, R., Archer, R. P., & House, J. J. (1996). The MMPI–A and Rorschach: A failure to establish convergent validity. Assessment, 3, 179–191. Lerner, P. (1991). Psychoanalytic theory and the Rorschach. Hillsdale, NJ: The Analytic Press. Licht, R. (1995). Multiple regression and correlation. In L. Grimm & P. Yarnold (Eds.), Reading and understanding multivariate statistics (pp. 19–65). Washington, DC: American Psychological Association. Lubin, B., Larsen, R. M., & Matarazzo, J. D. (1984). Patterns of psychological test usage in the United States: 1935–1982. American Psychologist, 39, 179–191. Lubin, B., Wallis, R., & Paine, C. (1971). Patterns of psychological test usage in the United States. Professional Psychology, 2, 70–74. Marsh, A., & Viglione, D. (1992). A conceptual validation study of the texture response on the Rorschach. Journal of Personality Assessment, 58, 571–579. Masling, J., Rabie, L., & Blondheim, S. (1967). Obesity, level of aspiration, and Rorschach and TAT measures of oral dependence. Journal of Consulting Psychology, 31, 233–239. McCann, J. (1989). MMPI personality disorder scales and the MCMI: Concurrent validity. Journal of Clinical Psychology, 45, 365–369. McCann, J. (1991). Convergent and discriminant validity of the MCMI–II and the MMPI personality disorders. Psychological Assessment, 3, 9–18. Meehl, P. E., & Hathaway, S. R. (1946). The K factor as a suppressor variable in the MMPI. Journal of Applied Psychology, 30, 525–564. Meyer, G. J. (1993). The impact of response frequency on Rorschach constellation indices and on their validity with diagnostic and MMPI–2 criteria. Journal of Personality Assessment, 60, 153–180. Meyer, G. J. (1996). The Rorschach and MMPI: Toward a more scientifically differentiated understanding of cross-method assessment. Journal of Personality Assessment, 67, 558–578. Meyer, G. J. (1997). On the integration of personality assessment methods: The Rorschach and MMPI–2. Journal of Personality Assessment, 68, 290–330. Meyer, G. J. (1999). The convergent validity of MMPI and Rorschach scales: An extension using profile scores to define responses and character styles on both methods and a reexamination of simple Rorschach response frequency. Journal of Personality Assessment, 72, 1–35. Morey, L. (1985). A psychometric analysis of five DSM–III categories. Personality and Individual Differences, 6, 323–329. Morey, L. (1986). A comparison of three personality disorder assessment approaches. Journal of Psychopathology and Behavioral Assessment, 8, 25–30. Morey, L., Waugh, M., & Blashfield, R. (1985). MMPI scales for the DSM–III personality disorders: Their derivation and correlations. Journal of Personality Assessment, 49, 245–251. O’Maille, P., & Fine, M. (1995). Personality disorder scales for the MMPI–2: An assessment of psychometric properties in a correctional population. Journal of Personality Disorders, 9, 235–246. Piotrowski, C., & Keller, J. (1984). Psychodiagnostic testing in APA-approved clinical psychology programs. Professional Psychology: Research and Practice, 15, 450–456. Piotrowski, C., & Keller, J. (1989). Psychological testing in outpatient mental health facilities: A national study. Professional Psychology, 20, 423–425. Piotrowski, C., Sherry, D., & Keller, J. (1985). Psychodiagnostic test usage: A survey of the Society for Personality Assessment. Journal of Personality Assessment, 49, 115–119. Rorschach, H. (1942). Psychodiagnostics. Berne, Switzerland: Hans. (Original work published 1921) Sechrest, L. (1963). Incremental validity: A recommendation. Educational and Psychological Measurement, 23, 153–158. Stevens, J. (1996). Applied multivariate statistics for the social sciences (3rd ed.). Mahwah, NJ: Lawrence Erlbaum Associates, Inc. Trull, T. (1993). Temporal stability and validity of two personality disorder inventories. Psychological Assessment, 5, 11–18.
168
BLAIS ET AL.
Urist, J. (1977). The Rorschach test and the assessment of object relations. Journal of Personality Assessment, 41, 3–9. Watkins, C., Campbell, V., Nieberding, R., & Hallmark, R. (1995). Contemporary practice of psychological assessment by clinical psychologists. Professional Psychology, 26, 54–60. Wechsler, D. (1981). Wechsler Adult Intelligence Scale–Revised. New York: Psychological Corporation. Weiner, I. (1991). Editor’s note: Interscorer agreement in Rorschach research. Journal of Personality Assessment, 56, 1. Widiger, T. (1991). Personality disorder dimensional model proposed for DSM–IV. Journal of Personality Disorders, 5, 386–398. Wise, E. A. (1996). Comparative validity of the MMPI–2 and MCMI–II personality disorder classification. Journal of Personality Assessment, 66, 569–582. Wood, J., Nezworski, M., & Stejskal, W. (1997). The reliability of the Comprehensive System for the Rorschach: A comment on Meyer (1997). Psychological Assessment, 9, 490–494. Viglione, D. (1997). Problems in Rorschach research and what to do about them. Journal of Personality Assessment, 68, 590–599. Zimmerman, M. (1995). Diagnosing personality disorders: A review of issues and research methods. Archives of General Psychiatry, 51, 225–245.
Mark A. Blais Inpatient Psychiatry Service Massachusetts General Hospital Blake-11 55 Fruit Street Boston, MA 02114 E-mail:
[email protected] Received May 3, 2000 Revised June 19, 2000