The Complexity of Rorschach Random Protocols: A Fundamental Issue with the Dimensional Approach to the Rorschach

Patrick Fontan and Anne Andronikof

        Complexity, the Rorschach 1st factor, is one of the most important components of R-PAS interpretation (Meyer et al., 2011). However, this dimension is problematic on conceptual grounds and has been described as tautological (Meyer, 1992b). In that case, this notion would be unfalsifiable and would appear as unscientific(Popper,1959).In order to study this problem empirically, 98 protocols from the Belgian CS reference sample were submitted to the analysis described in R-PAS manual, replicating Complexity main findings. Protocols were then randomly recorded and submitted to the same analysis which yieldedvery similar results. In addition, the Complexity scores of real and randomly rescored protocols were highly correlated. We conclude that Complexity main findings do not depend on data, and its reliability in clinical practice is questionable. Means to deal with the dimensional approach to the Rorschach are discussed and alternatives are proposed.

        In the psychological assessment field, the clinical - or even projective – approach is generally opposed to the psychometric and dimensional one. The Rorschach test, in particular, is somehow emblematic of the clinical assessment approach, and remains a popular instrument in clinical practice (Piotrowski, 2015) while statistical procedures and methodological approaches are major topical areas of study in personality assessment (Piotrowski, 2019).

        The idea of making these two approaches converge and to apply dimensional models to the Rorschach is not new and has been defended by several authors. The first factor analysis of the Rorschach was carried out as early as 1947, and until 2003 a multivariate analysis of the Rorschach was published every 4 years approximately. (Adcock, 1951; Anderson & Dixon, 1993; Coan, 1956; Costello, 1998; Geertsma, 1962; Hsü, 1947; Mason et al., 1985; Meyer, 1992b; Schori & Thomas, 1972; Shaffer et al., 1981; Wittenborn, 1950; Wood et al., 2003; Zillmer & Vuz, 1995; see Fontan (2014) for a detail review). R-PAS (Meyer et al., 2011) represents a major breakthrough in this field and it is the first Rorschach interpretation system based on a dimensional approach, and specifically on the Complexity model.

        Complexity, also known as the Rorschach 1stfactor, is a dimension which is highly correlated with the total number of Rorschach responses (R) and substantially correlated with most Rorschach variables. This dimension “emerges consistently in factor analytic research” (Meyer et al., 2011, p.319), which means that this result has often been replicated (Meyer, 1992b; Murstein, 1960; Schori & Thomas, 1972; Wittenborn, 1950; Wood, Krishnamurthy, & Archer, 2003). In R-PAS, the Rorschach “1st factor” is defined statistically as the result of a principal component analysis of all count variables (R, W, DQo, FC, H etc.), with a single dimension being extracted. Complexity is also a score which is actually computed in R-PAS and corresponds to the sum of all contents but single animal, all determinants but pure Form responses, ordinary developed responses, two times synthesis between detailed responses and three times synthesis responses in whole or space locations (Meyer et al., 2011, p. 295).This calculation implies that the same weight is given for a simple Whole response on cards I or V (e.g. Bat) and for highly synthetic responses on Card X (e.g. a firework in Paris above the Eiffel Tower), contrary to the CS Z score which takes stimuli properties of the cards into account.

 

Complexity is “the best marker of the first factor”, (Meyer et al., 2011, p.319), the correlation being r=0.95 (Meyer et al., 2011, p. 443). There are calculation differences between the Rorschach 1st factor and Complexity; however, conceptually, these two notions are considered equivalent:

"Because many Rorschach variables are highly correlated with the Rorschach 1st factor, which we call Complexity, we developed a method for adjusting all scores for this factor" (Meyer et al., 2011, p.303, bold types are ours)

According to available evidence, Complexity is described as:

“An overall index of complexity of processing in that it measures differentiation, integration, and productivity at the response level. Associated with age, education, intelligence, and adaptation. Responding to the test with a high Complexity score means that the person has brought a considerable amount of psychological activity and effort to bear in coping with the demands of the test. Doing so in real life should be associated with more success and flexibility in coping, and a preference for more cognitive activity and energy when responding to challenges.” (Meyer et al., 2011, p. 349).

        As the Rorschach 1st factor, Complexity is one of the most important components of R-PAS interpretation, comparable to G on IQ test, or Welsh’s A on the MMPI-2 (Meyer et al., 2011, p. 348). Assessing the level of Complexity of a protocol is “an important prerequisite for all other interpretations” (Meyer et al., p. 324) and when Complexity deviates more than one standard deviation from the mean, it is recommended to adjust all test scores with Complexity (Complexity adjusted scores). Within R-PAS, Complexity appears as a fundamental dimension of personality, in the same way as G is a general dimension of intelligence. However, and despite its importance, this dimension is problematic on conceptual grounds:

“[the Rorschach 1st factor] reflects the somewhat tautological position that frequent responding leads to increased scoring across all determinants categories” (Meyer, 1992b, p.124)

        Tautologies are a form of circular reasoning. According to Wittgenstein (1922), tautologies are propositions which are true for all truth possibilities, they say nothing as they are unconditionally true, they are without sense, and they are not a picture of reality. In simple terms, tautologies are statements which are always true, whatever the data and they have no empirical meaning. Indeed, as the validity of tautologies do not depend on data, these statements cannot have any empirical meaning: they are not a picture of reality. Since, tautologies are true whatever the data; moreover, they are also unfalsifiable.

The statement “frequent responding leads to increased scoring across all determinant categories” (Meyer, 1992b, p. 124) can be generalized to “frequent responding leads to increased scoring across all Rorschach count variables.” From a logical point of view, this statement corresponds to the following implication: if

(A) R increases then (B) the total number of codes in a protocol will increase. The truth table of this implication is the following (Table 1):

Table 1: Implication Truth Table

A (R increases)	B (# codes increase)	A -> B
True	True	True
True	False	False
False	True	True
False	False	True

        This implication is false in only one case: when R increases while the total number of codes in a protocol does not. Concerning the Rorschach, this event cannot happen; this situation is impossible. Every Rorschach response contains at least five codes (Location, Developmental Quality, Determinant, Form Quality, and Content) and consequently each supplementary response mechanically adds at least five codes to the total number of codes in a protocol (which is self-evident, or… tautological). Consequently, the statement “frequent responding leads to increased scoring across all determinants categories”, cannot be proven false: it is unfalsifiable. According to Popper (1959), unfalsifiable statements are unscientific. It follows that Complexity – one of the most important components of R-PAS interpretation – might be unscientific. The objective of this paper is to study this logical problem empirically. Two main hypotheses can be derived from the considerations above:

(H1) If Complexity was indeed tautological and would not depend on data, it should be possible to replicate Complexity findings even when using Rorschach randomly scored protocols (true whatever the data).

(H2) If Complexity was indeed tautological and did not have empirical meaning, the Complexit y of Rorschach protocols should stay equivalent even if these protocols were randomly re-scored (no empirical meaning).

        As mentioned above, the Rorschach 1st factor, or Complexity, “emerges consistently from factor analytic research” (a result frequently replicated). The idea of H1 is to test whether Complexity is a genuine psychological construct depending on data, or if it is rather a statistical artifact. If Complexity was a genuine construct, then the same analysis conducted on real and random data should yield very different results. On the contrary, if Complexity was a statistical artifact, we could expect to replicate findings even in the worst-case scenario by using random data. In addition, if Complexity was a statistical artifact it should not be interpreted as a general personality dimension. The first hypothesis refers to the general structure of Rorschach data as related to sample of participants and is a fundamental issue. In this context, real and random protocols are considered as two separate samples analyzed with the same procedure.

        The second hypothesis is related to the practical issues and clinical implications of the Complexity tautological issue. If Complexity does not depend on data, then we could expect the level of Complexity to be equivalent for two very different scorings of the same protocol. So, at the individual level and for a single protocol, if random scoring yields a very similar Complexity score as real and accurate scoring, then the Complexity score would simply appear as unreliable. In this context, real and randomly rescored protocols are matched, and we computed the correlation between the Complexity scores for the two different scorings.

        If these hypotheses were refuted, Complexity would not appear as tautological but rather as a genuine psychological construct. On the contrary, if these hypotheses could be corroborated empirically, the tautological nature of Complexity and its unscientific status would be confirmed. As this dimension is one of the most important components of interpretation, this would call for a fundamental reconsideration in our understanding of the Rorschach, or more precisely of its dimensional structure.

Method:

Sample Size:

According to Guadagnoli & Velicer, if “components possess four or more variables with loadings above

.60, the pattern may be interpreted whatever the sample size used” (Guadagnoli & Velicer, 1988, p. 274). R-PAS manual reports correlations above .60 with Complexity for the five individually coded variables: number of responses R (.79), multiple determinant response Blend (0.82), synthesis responses Sy (.83), human movement responses M (.68) and human contents SumH (.70) (All protocols – norm sample) (Meyer et al., 2011, p.466).

Participants:

        The Belgian adult non-patient Rorschach CS reference sample was used in this study. Detailed information concerning recruitment procedures, examiners, administration and scoring procedures are available in the original article (Mormont, Thommessen, & Kever, 2007). The Belgian database was used primarily because it is available in CHESSSS (an open-source software for the Rorschach CS) (Fontan et al., 2013), which allowed us to generate random protocols. The original database included 100 participants and 98 were used for this study (one protocol included more than 50 responses and could not be scored in CHESSSS, and one other protocol was found to be a duplicate).

        Participants were recruited by graduate psychology students who administered the Comprehensive System under the supervision of senior staff members at the Clinical Psychology Service of the University of Liège. All protocols were administered and scored according to the CS (Exner, 2001) and none had less than 14 responses. Concerning inter-rater reliability, 25 protocols were blindly rescored, and all kappa coefficients were above 0.75 (Table 2), which is considered excellent (Fleiss, Levin, & Paik, 2013).

Table 2: Belgian adult nonpatient sample interrater reliability statistics

 

Variable	% Agreement	Iota (Kappa)
Location	.99	.98
Developmental Quality	.96	.92
Determinants	.98	.88
Form Quality	.88	.80
Pairs	.99	.97
Contents	.99	.89
Populars	.97	.91
Z Score	.88	.80
All Special Scores	.99	.81

Note. 25 cases of the Belgian sample (25%) were scored independently by two judges

Random Protocols:

        The database used in this study includes 98 participants for a total of 2365 responses which were all coded for the following scoring categories (a) Location and Developmental Quality, (b) Determinants, (c) Form Quality, (d) Pairs, (e) Contents, (f) Popular, (g) Z score and (h) Special Scores.Random protocols were generated as follow:

1. the first protocol (according to participants IDs) is selected from the 98 available,
2. the scoring of the protocol responses is deleted,
3. for the first response, a Location and Developmental Quality is randomly selected among the 2365 available responses in the Belgian database (random sampling with replacement), then this procedure is repeated for the other scoring categories (Determinants, Form Quality, etc.),
4. once the first response is rescored for all scoring categories, the protocol second response is randomly rescored in the same manner and so on until every response of the selected protocol are randomly rescored,
5. these steps are repeated for the second protocol and so on until every protocol in the database are randomly rescored.

This procedure is based on a Bayesian approach in which the probability of occurrence of the different CS codes is not inferred according to a statistical law (e.g., uniform or normal distributions), but rather according to the actual distribution of these codes observed in the Belgian sample (taking into account Rorschach variables distribution issues).

 

Analysis:

        A principal component analysis was conducted on the counts of individually coded variables (i.e., R, W, D, F, P etc.) of the real protocols of the Belgian sample. However, some variables of the contents category have been regrouped into broader categories to simplify the results (e.g., Human Contents, see Table 2). In accordance with R-PAS manual, the first unrotated component was extracted (Meyer et al., 2011), and component scores were saved. The same analysis was replicated on the random protocols. Finally, the correlations between the Complexity score and the Rorschach 1st component (i.e., the so-called “Rorschach 1st factor”) of real and random protocols were computed.

 

Results:

Table 3: Correlations between the 1st Component and Rorschach Variables, Main Findings

Main Findings		Real Protocols	Random Protocols
Correlation between R and 1st Component		.95	.99
Variables significantly correlated with Component (loading>30)	the 1st	75%	81%

 

Variables strongly correlated with the 1st Component (loading>50)

Principal Component Analysis of real protocols:

49% 51%

The first unrotated component of real Rorschach protocols is essentially defined by the total number of responses (R is loaded at 0.95 by this component), and most Rorschach variables are significantly and substantially correlated with this component (Table 3, Annex A). The correlation between Complexity and the Rorschach 1st component of real protocols is r=0.95 (Table 4). R-PAS manual reports a correlation of r=0.95 between these two variables. This result replicates perfectly Complexity main findings.

Principal Component Analysis of random protocols:

The first unrotated component of random Rorschach protocols is essentially defined by the total number of responses (R is loaded at 0.99 by this component), and most Rorschach variables are significantly and substantially correlated with this component (Table 3, Annex A). The correlation between Complexity and the Rorschach 1st component of random protocols was r=0.97 (Table 4). Complexity main findings are replicated even with randomly scored protocols. In these two analyses, real and random protocols were unmatched.

Correlation between the Complexity of real and random protocols:

 

	R	Real Protocols Complexity	1st Comp.	Random Protocols Complexity	1st Comp
R	1
Complexity	.82	1
1st Comp.	.95	.95	1
Complexity	.95	.82	.93	1
1st Comp.	.99	.83	.95	.97	1

Table 4: Correlations between the Complexity and the Rorschach 1st Component of real and matched random Rorschach protocols

Real Protocols

Random Protocols

        In the previous analyses, real and random protocols were treated as two distinct datasets. However, random protocols were real protocols which were randomly rescored, so it is possible to associate them pair wise, and therefore to compute the correlation between Complexity scores of real and matched randomly rescored protocols. It appears that the first component (i.e., the Rorschach “1st factor”) of real and random Rorschach protocols is essentially equivalent (r=0.95) and the Complexity score (which is an estimate of the Rorschach 1st component) of real and random protocols are highly correlated (r=0.82; Table 4).

        Concerns were raised, in professional congresses, about our randomization procedure, so we decided to generate randomly rescored protocols in a second fashion, using uniform distribution, i.e., choosing codes by rolling dices. The advantage of the first randomization method is to include blends and multiple contents in the random scoring. Taking into account Rorschach variables distribution issues, general frequencies of all codes was similar to real scoring; we simply made codes completely unreliable. On the contrary, the second method include only one determinant and content per response and is based on a radical approach to randomization (rolling dices) ; changing dramatically Rorschach count variables distributions. However, the correlation between Complexity scores were found to be identical (r=.82).

 

Discussion:

Complexity & science, the problem of R & nature of Rorschach variables:

        The logical analysis presented in the introduction as well as the evidence we provided demonstrate Meyer’s (1992b) point that the “Rorschach 1st factor” is tautological. It follows that this notion cannot be refuted and thus it is unscientific according to Karl Popper’s criteria (1959), as Complexity findings can be replicated even with random and meaningless data.

        The statement “frequent responding leads to increased scoring across all determinants categories” (Meyer, 1992, p. 124, bold types are ours) includes a form of causation: the tendency to respond to the test would imply higher frequencies among response codes1.This form of causation is also reflected in the terminology “Rorschach 1st factor”. This dimension is not the result of a factor analysis but rather of a component analysis which makes it the “Rorschach 1st component”. This point is important because, in the dimensional approach, factors are thought to be explanatory while components are simply descriptive. So the term “Rorschach 1st factor” represents an attempt to explain a fact which is well known with the Rorschach: R is substantially correlated with most Rorschach variables, which is known as “the problem of R” (Cronbach, 1949; Fiske & Baughman, 1953; Meyer, 1992a; Meyer, 1992b; Murstein, 1960; Perry & Kinder, 1990). However, as it is also well known in statistical sciences, correlation does not imply causation. This explanation could be relevant if R was independent from other Rorschach variables, but it is not the case, and each Rorschach response mechanically adds at least five codes to a protocol by definition. This issue is a basic property of the Rorschach test as it is defined. For example, the Harrower Multiple Choice Rorschach Test does not present the problem of R as a single whole response is recorded for each card (Harrower-Erickson & Steiner, 1945). The issue is simply that R and Rorschach variables are not independent variables because of a part-whole relationship. Each Rorschach human, reflection or cooperative response is by nature a Rorschach response; as Rorschach codes are parts of a whole that we call a Rorschach response. At the protocol level, in statistics, these part-whole relationships are called “multi-colinearity”. There are major multi-colinearity issues in the Rorschach as R equals to the sum of locations, developmental qualities, primary determinants, form qualities, and primary contents2. Neglecting

1Also shown in the sentences: « Complex records tend to trigger elevations of multiple scores […]. Conversely, simplistic records

generate suppressions on multiple scores » (Meyer et al. p. 324, bold types are ours)

2R =W+D+Dd

R = DQ+ + DQo + DQv + DQv/+

R = FQ+ + FQo + FQu + FQ- + FQnone

these part-whole relationships and treating R as any Rorschach variable as if it was independent (i.e., treating a whole as a part of it) leads to circular reasoning, tautologies, as well as irrefutable and unscientific statements. Basically, Rorschach variables multicollinearity issues perfectly explain the problem of R and a factor analysis trying to explain this problem is (a) unnecessary and (b) impossible to make because of the very multicollinearity issue.Strictly speaking, this means that the "Rorschach 1st factor" simply does not exist.

 

Clinical implications: Complexity as a psychological construct in personality assessment:

The apparent robustness of the “Rorschach 1st factor”, its reproducibility, is misleading, based on a statistical artifact and circular reasoning (tautology, problem of R, multicolinearity issues). The “fact” that the number of codes increases when the number of responses increases is not a fundamental dimension of personality, should it be called “Complexity”, but rather simply a basic property of the test as it is defined. Complexity as a construct is not reliable and introduces a confusion between the person who takes the test and the test itself.

        This issue becomes more concerning when this dimension, the “Rorschach 1st factor”, is considered a fundamental psychological construct, “Complexity”, rather than just a problem within the test data. Indeed, with the problem of R, the number of responses given to the Rorschach is considered a confounding variable of the test interpretation. Consequently, with the Complexity model, one of the most important components of R-PAS (and personality…) would be essentially defined by a confounding variable, and the Rorschach should not be considered as reliable. In addition, this potentially fundamental dimension of personality would be sensitive to psychologist manipulations as it is possible to ask for supplementary responses and to pull out cards during administration (in the CS and even more with R-PAS R-Optimized procedure). Finally, and as demonstrated in this study, two psychologists scoring the same protocol might be in strong disagreement and still find similar values of Complexity (i.e., agree concerning one of the most important components of interpretation despite strong disagreement in the scoring).

Basically, our results demonstrate that the Complexity score has little to do with the properties of the responses given by people taking the test.Consequently, we highly recommend for psychologists using R-PAS not to draw any clinical inferences based on Complexity scores.

 

Implications for the CS and R-PAS:

        It might be argued that the problem raised in this paper impacts the CS and R-PAS in equal proportions, but this is not the case. Indeed, the problem of R is the same in the CS and R-PAS. However, the Complexity model, the “Rorschach 1st factor” is not included in the CS. If this dimensional model corresponded to a genuine psychological construct, then potential issues with this construct would apply to the CS and R-PAS equally. However, we demonstrated in this paper that this model is not reliable and that it should not be used to interpret Rorschach data. As the CS was not designed with a dimensional approach and does not include the Complexity model, the issue raised in this paper impacts R-PAS but not the CS. In spite of these considerations, the issues presented in this paper raise a more general question: is the dimensional approach relevant to the Rorschach?

 

Reductionism and under extraction:

The dimensional approach is a very important trend in psychology since the early 20th century (e.g., Spearman, 1904) until today and it is legitimate to wonder if this approach is relevant to Rorschach data.

R = Sum of Primary Determinants R = Sum of Primary Contents

        Basically, this approach aims to analyze a complex pattern of inter-correlation between variables and to reduce it to a “simple pattern” of most significant dimensions (it summarizes the data). This approach has been very substantially developed and many different techniques are available. Therefore, the dimensional analysis of data requires a researcher to make a certain number of methodological choices: extraction method (principal component, principal axis, maximum likelihood, etc.), number of dimensions to retain, rotation method (orthogonal, varimax, oblique, oblimin, promax etc.), cutoff for significant loadings depending on number of significant variables per dimension and sample size, method for score calculation (regression, Bartlett, Aderson-Rubbin). Among these different choices, the number of dimensions to retain appears to be a “key question” (Courtney, 2013), a “critical component” (Hayton et al., 2004), a “crucial decision” (Jelaska et al., 2012) and the “most critical decision” (Zwick & Velicer, 1986). Retaining the wrong number of dimensions leads to underextraction (too few dimensions retained) or overextraction (too many). The impact of over- and underextraction on factor and component analyses has been studied in Monte Carlo studies (i.e., data analyses simulation). Overall, authors agree that (1) the accurate determination of the number of dimensions is critical, (2) though overextraction is not desirable ; underextraction is a much more severe problem than overextraction, and (3) underextraction muddies genuine dimensions with inappropriate loadings (Fava &Velicer, 1992; Fava &Velicer, 1996; Wood et al., 1996).

        With these points in mind, it is very concerning that (1) All Rorschach raw count variables are reduced to a single dimension in R-PAS, which is an extreme form of reductionism and (2) that this decision is not supported by any statistical argument in R-PAS manual while several modern techniques are available to assess the proper number of dimensions to retain, for example: Minimum Average Partial, Parallel Analysis, Optimal Coordinates, Comparison Data (Buja & Eyuboglu, 1992; Horn, 1965; Raîche et al., 2013; Ruscio & Roche, 2012; Velicer, 1976). Given these two points, R-PAS Complexity model very likely represents an extreme form of underextraction, which is a severe problem, even if the tautological issue of Complexity was not addressed.

 

Toward a dimensional approach to the Rorschach:

        R-PAS manual states that the Rorschach 1st factor “emerges consistently in factor analytic research” (Meyer et al., 2011, p.319). While this point is true, it is also an oversimplification of the literature on the dimensional approach to the Rorschach. Indeed, we identified six relatively consistent dimensions in the literature:

1. Productivity, a dimension essentially defined by the total number of responses and which explains most of the variance of Rorschach data, equivalent to Complexity (Meyer, 1992b; Murstein, 1960; Schori& Thomas, 1972; Wittenborn, 1950; Wood, Krishnamurthy, & Archer, 2003).
2. Form Dominance, a dimension composed of form dominant color and shading responses (Costello, 1998; Schori& Thomas, 1972; Shaffer et al., 1981; Zillmer&Vuz, 1995).
3. Color Dominance regroups color and some shading responses with vague or no form (Costello, 1998; Geertsma, 1962; Mason, Cohen & Exner, 1985; Shaffer, Duszynski, & Thomas, 1981; Wittenborn, 1950; Zillmer&Vuz, 1995).
4. Perceptual Accuracy, a bidirectional dimension with ordinary form qualities on the positive pole and poor form qualities on the negative one (Geertsma, 1962; Wood, Krishnamurthy, & Archer, 2003; Zillmer&Vuz, 1995).
5. Kinesthesis, a dimension composed of Human Movement and related variables (Geertsma, 1962; Schori& Thomas, 1972; Shaffer et al., 1981; Wittenborn, 1950).
6. Synthesis, a dimension mainly composed of whole responses (W), perceptually organized responses (Zf ), and synthesis responses (DQ+) on the positive pole, and pure form responses (F, F% or Lambda) on the negative one (Anderson & Dixon, 1993; Geertsma, 1962; Mason et al., 1985; Schori & Thomas, 1972; Shaffer et al., 1981; Wood, Krishnamurthy, & Archer, 2003; Zillmer & Vuz, 1995).
   

 

A careful review of the literature on the factor or component analysis of the Rorschach raises four main problems:

1. how to address the problem of R,
2. choice of variables to include in the analysis (there are many redundancies and multicolinearity issues between Rorschach scores),
3. distribution issues and
4. dimensionality issues (how many dimensions to retain).

Published factor or component analyses of the Rorschach generally address some of these points but not the four of them in a systematic manner, and consequently these studies generally present divergent results (see Fontan, 2014, Fontan et al., 2016 for detailed reviews). In addition, focus on factor/component analyses of the Rorschach has decreased in recent years (Piotrowski, 2017).

        To overcome the difficulties of the dimensional approach to the Rorschach test, we proposed to address these four issues together. Paying attention to the dimensionality issue, we developed a technique which combines resampling techniques and parallel analysis and found 12 dimensions to retain out of Rorschach count variables referred as the Rorschach Component Model (Fontan et al., 2016). While this work is still in progress, it sets the basis for an alternative and consistent dimensional approach to the Rorschach.

 

References:

Adcock, C. J. (1951). A factorial approach to Rorschach interpretation. Journal of General Psychology, 44, 261-272.

Anderson, T., & Dixon Jr, W. E. (1993). The factor structure of the Rorschach for adolescent inpatients. Journal of Personality Assessment, 60(2), 319-332.

Buja, A., & Eyuboglu, N. (1992). Remarks on parallel analysis. Multivariate Behavioral Research, 27(4), 509-540. doi: 10.1207/s15327906mbr2704_2

Coan, R. (1956). A factor analysis of Rorschach determinants. Journal of Projective Techniques, 20, 280-287.

Costello, R. M. (1998). Psychometric definition of Rorschach determinant component structure. European Journal of Psychological Assessment, 14(2), 116-123. doi: 10.1027/1015-5759.14.2.116

Courtney, M. G. R. (2013). Determining the number of factors to retain in EFA: using the SPSS R-Menu v2.0 to make more judicious estimations. Practical Assessment, Research & Evaluation, 18(8).

Cronbach, L. J. (1949). Statistical methods applied to Rorschach scores: a review. Psychological Bulletin, 46(5), 393-429.

Fava, J. L., & Velicer, W. F. (1992). The effects of overextraction on factor and component analysis. Multivariate Behavioral Research, 27(3), 387-415. doi: 10.1207/s15327906mbr2703_5

Fava, J. L., &Velicer, W. F. (1996). The effects of underextraction in factor and component analyses. Educational and Psychol ogical Measurement, 56(6), 907-929.

Fiske, D. W., & Baughman, E. E. (1953). Relationships between Rorschach scoring categories and the total number of responses. The Journal of Abnormal and Social Psychology, 48(1), 25-32.

Fleiss, J. L., Levin, B., & Paik, M. C. (2013). Statistical Methods for Rates and Proportions. New York: John Wiley & Sons.

Fontan, P. (2014). Approchepsychométrique du test de Rorschach [Psychometric approach of the Rorschach test]. Unpublished doctoral dissertation. University of Paris Ouest Nanterre, France.

Fontan, P., Andronikof, A., Mattlar, C.-E., & Mormont, C. (2016). Dimensions of the Rorschach Comprehensive System: parallel analysis and principal component analysis of a European adult nonpatient sample. Rorschachiana, 37(2), 114-146.

Fontan, P., Andronikof, A., Nicodemo, D., Al Nyssani, L., Guilheri, J., Hansen, K. G.,Nakamura, N. (2013). CHESSSS: A free software solution to score and compute the Rorschach Comprehensive System and Supplementary Scales. Rorschachiana, 34(1), 56-82. doi: 10.1027/1192-5604/a000040.

Geertsma, R. H. (1962). Factor analysis of Rorschach scoring categories for a population of normal subjects. Journal of Consulting Psychology, 26(1), 20-25. doi: 10.1037/h0040563

Guadagnoli, E., &Velicer, W. F. (1988). Relation to sample size to the stability of component patterns. Psychological Bulletin, 103(2), 265-275

Harrower-Erickson, M. R., & Steiner, M. E. (1945). Large scale Rorschach techniques: A manual for the group Rorschach and multiple-choice test. Springfield, IL, US: Charles C Thomas Publisher.

Hayton, J. C., Allen, D. G., & Scarpello, V. (2004). Factor retention decisions in exploratory factor analysis: a tutorial on parallel analysis. Organizational Research Methods, 7(2), 191-205. doi: 10.1177/1094428104263675

Horn, J. L. (1965). A rationale and test for the number of factors in factor analysis. Psychometrika, 30(2), 179-185. doi: 10.1007/bf02289447

Hsü, E. H. (1947). The Rorschach responses and factor analysis. Journal of General Psychology, 37, 129-138.

Jelaska, I., Mandić-Jelaska, P., &Lovrić, F. (2012). Experimental comparation of Guttman Kaiser, Plum Brandy, scree and parallel analysis - Monte Carlo criterions in exploratory factor analysis via selected kinesiological research. Acta Kinesiologica, 6(2), 80-84.

Mason, B. J., Cohen, J. B., & Exner, J. E., Jr. (1985). Schizophrenic, depressive, and nonpatient personality organizations described by Rorschach factor structures. Journal of Personality Assessment, 49(3), 295-305.

Meyer, G. J. (1992a). Response frequency problems in the Rorschach: Clinical and research implications with suggestions for the future. Journal of Personality Assessment, 58(2), 231-244.

Meyer, G. J. (1992b). The Rorschach's factor structure: A contemporary investigation and historical review. Journal of Personality Assessment, 59(1), 117-136.

Meyer, G. J., Viglione Jr, D. J., Mihura, J. L., Erard, R. E., &Erdberg, P. (2011). Rorschach Performance Assessment System: Administration, Coding, Interpretation, and Technical Manual. Toledo, OH: Rorschach Performance Assessment System.

Mormont, C., Thommessen, M., & Kever, C. (2007). Rorschach Comprehensive System data for a sample of 100 adult nonpatients from the belgianfrench-speaking community. Journal of Personality Assessment, 89, 26-34.

Murstein, B. I. (1960). Factor analyses of the Rorschach. Journal of Consulting Psychology, 24(3), 262-275. doi: 10.1037/h0041619 Perry, G. G., & Kinder, B. N. (1990). The susceptibility of the Rorschach to malingering: A critical review. Journal of Personality Assessment, 54, 47-57.

Piotrowski, C. (2015). Projective techniques usage worldwide: A review of applied settings 1995-2015. Journal of the Indian Academy of Applied Psychology, 41(3), 9–19.

Piotrowski, C. (2017). Rorschach research through the lens of bibliometric analysis: Mapping investigatory domain. Journal of Projective Psychology & Mental Health, 24(1), 34–38.

Piotrowski, C. (2019). Contemporary research emphasis in personality assessment: A bibliometric analysis mapping investigatory domain (2009–2018). SIS Journal of Projective Psychology and Mental Health, 26(2), 97-103.

Popper, K. R. (1959). The Logic of Scientific Discovery. London: Hutchinson.

Raîche, G., Walls, T. A., Magis, D., Riopel, M., &Blais, J.-G. (2013). Non-graphical solutions for Cattell’s scree test. Methodology:

European Journal of Research Methods for the Behavioral and Social Sciences, 9(1), 23-29.doi: 10.1027/1614-2241/a000051 Ruscio, J., & Roche, B. (2012). Determining the number of factors to retain in an exploratory factor analysis using comparison data of

known factorial structure. Psychological Assessment, 24(2), 282-292. doi: 10.1037/a0025697

Schori, T. R., & Thomas, C. B. (1972). The Rorschach test: An image analysis. Journal of Clinical Psychology, 28(2), 195 -199. doi: 10.1002/1097-4679(197204)28:2<195::aid-jclp2270280222>3.0.co;2-z

Shaffer, J. W., Duszynski, K. R., & Thomas, C. B. (1981). Orthogonal dimensions of individual and group forms of the Rorschach.

Journal of Personality Assessment, 45(3), 230-239. doi: 10.1207/s15327752jpa4503_2

Spearman, C. (1904). 'General intelligence,' objectively determined and measured. The American Journal of Psychology, 15(2), 201 - 293. doi: 10.2307/1412107

Velicer, W. F. (1976). Determining the number of components from the matrix of partial correlations. Psychometrika, 41(3), 321-327. Wishner, J. (1959). Factor analyses of Rorschach scoring categories and first response times in normals. Journal of Consulting

Psychology, 23(5), 406-413.

Wittenborn, J. R. (1950). A Factor Analysis of Rorschach scoring categories. Journal of Consulting Psychology, 14(4), 261-267. doi: 10.1037/h0054657

Wittgenstein, L. (1922). Tractatus Logico-philosophicus. New york: Harcourt, Brace, Inc.

Wood, J. M., Krishnamurthy, R., & Archer, R. P. (2003). Three factors of the Comprehensive System for the Rorschach and their relationship to Wechsler IQ scores in an adolescent sample. Assessment, 10(3), 259-265.

Wood, J. M., Tataryn, D. J., & Gorsuch, R. L. (1996). Effects of under- and overextraction on principal axis factor analysis with varimax rotation. Psychological Methods, 1(4), 354-365. doi: 10.1037/1082-989x.1.4.354

Zillmer, E. A., & Vuz, J. K. (1995). Factor analysis with Rorschach data. In J. E. Exner, Jr. (Ed.), Issues and methods in Rorschach research. (pp. 251-306). Hillsdale, NJ England: Lawrence Erlbaum Associates, Inc.

Zwick, W. R., &Velicer, W. F. (1986). Comparison of five rules for determining the number of components to retain. Psychological Bulletin, 99(3), 432-442.