Rorschach Research through the Lens of Bibliometric Analysis: Mapping Investigatory Domain

Chris Piotrowski

This study presents the results of an exploratory bibliometric „topical‟ analysis with regard to mainly primary research regarding the Rorschach published in journal articles 2000-2016. The major aim is to a) determine the scope and breadth of investigatory areas most emphasized by researchers during this time frame, and b) prompt more advanced bibliometric study of the extant Rorschach literature in order to map the structure of scholarship and research domain regarding this popular assessment method. The database PsycINFO was selected to obtain the pool of references based on a „keyword‟ search of the term Rorschach. The search identified a total of 838 peer-reviewed articles from 2000 to 2016; of these, 747 were determined to be mostly „primary‟ articles and served as the data-set for the analysis. The author tagged each article with a topical descriptor and maintained a scoring template based on frequency counts across categories. The analysis identified 29 topical categories which represented at least 1% of the total distribution. The most prevalent researched topics were (in rank order): Norms, psychotic states, eating disorders, historical aspects, psychosomatic factors, treatment planning/outcome, aggression/hostility, personality, psychodynamic issues, depression, and personality disorders. Neglected areas of research were noted such as, assessment training, differential diagnosis, anxiety states, racial/ethnic differences, and social desirability. The focus on norms/normative comparisons was quite apparent. The potential impact of editorial preference/bias was discussed and limitations of the study were noted. Based on this analysis, it appears that recent Rorschach scholarship is a) not cohesive in character, and b) reflects diverse research domains representing disparate research interests. Hence, despite an extensive repository of literature, the Rorschach remains an emerging area of study with an opaque sense of direction for future research.

Bibliometric analysis is a recognized and valid means of gaining an objective perspective on research trends of the extant literature (De Bellis, 2009). Thus, obtaining a limpid view on the scope of investigatory interest on specific research topics seems a worthwhile scholarly endeavor. Yet despite the vast volume of studies and repository of knowledge on the Rorschach method, little is known about the breadth of research emphasis and structure of published scholarship on this charismatic assessment technique. An online search of the PsycINFO database, conducted July 21 2016, highlights the perennial visibility and extensive volume of extant literature on the Rorschach, i.e., 641 books/book chapters, 1,020 dissertations, and 4,022 articles. This prodigious research interest, to some extent, reflects the continued popularity of the Rorschach over the decades as one of the top assessment techniques used in psychological practice (Piotrowski, 2015a; Wright et al., 2016), as well as interest internationally (e.g., Manickam & Dubey, 2005).

However, bibliometric studies regarding the Rorschach have been sparse. Giegerich (2014) examined how personality assessment

was portrayed in 6 popular introductory psychology textbooks. The analysis indicated notable negative bias with regard to coverage of projective techniques, particularly the Rorschach. Research trends on the Rorschach, from 1950-1985, have been found to show a slight resurgence of scholarly interest by the mid-1980s (Polyson et al., 1986), whereas Reynolds and Sundberg (1976), in a cited reference analysis, reported that the Rorschach was referenced highly, from 1951-1971, in the Buros Mental Measurements Yearbook. Piotrowski and Keller (1993) reported on the reference output for studies on the shading response in Rorschach research.

Since there has been recent research attention devoted to bibliometric study on a myriad of psychological topics in the mental health field (e.g., Cox et al., 1995; Piotrowski, 2012) and across applied psychology disciplines (Cascio & Aguinis, 2008; Miles & Naumann, 2011; Piotrowski, 2016), a contemporary examination regarding the scope of research emphasis on the Rorschach seems warranted. The current bibliometric analysis is the first attempt to identify the breadth of investigatory interest on the Rorschach Test in published research. The focus will be on determining the main areas of research focus, based on individual study aim and design, across the pool of published articles (2000-2016), with a focus on the Rorschach. To that end, the aim is to identify the major „topical‟ areas of investigation in Rorschach research in recent years.

Investigatory Approach:

In order to obtain a comprehensive pool of articles on the Rorschach, a „keyword‟ online search, using the term Rorschach, was performed in the database PsycINFO. This scholarly file is a recognized major repository of published research in the social and behavioral sciences (Perdue & Piotrowski, 1991; Piotrowski et al., 2005). Moreover, PsycINFO indexes journals from allied fields and coverage is international in scope (Garcia-Perez, 2010). Thus, PsycINFO was selected since the Rorschach has garnered academic and professional interest across many disciplines and in many countries worldwide. The search produced 4,022 peer- reviewed articles; of these, 838 were published from 2000 to 2016. In order to target mainly „primary‟ research, references reporting corrections, commentaries/replies, editorial remarks, book and software reviews, and erratum were excluded. Thus, the data- set for this bibliometric analysis comprised 747 articles.

Based on extensive experience conducting bibliometric analyses on various research topics and individual journals, the author reviewed each study and determined the main aim or focus of the article. A categorical template, delineating topical areas, was maintained. Each article was labeled with only

1 topical designation. A running-tab was maintained until all 747 articles were reviewed. Although a number of issues of journals were declared a „special issue‟ on a select major topic/research area, some articles had a major emphasis outside of the intended area of focus.

Readers should be aware that the current analysis does not weigh individual research

findings, aims not to be evaluative, does not side on controversial issues, does not review the psychometric credibility of the Rorschach, nor clarifies the merits of the Rorschach for clinical assessment (see Meyer 2001). Simply, this study is descriptive in nature with the sole intent to present a „snapshot‟ on the scope and breadth of „topical‟ focus regarding scholarly research on the Rorschach, over the past two decades.

Findings:

Table 1 displays, in rank order, the major emphasized topical areas in Rorschach research since the year 2000. Only those topics that represented at least 1% of the dataset are noted. Research on normative data or issues has remained a perennial concern over many decades, particularly with regard to demographic groups, non-U.S. samples, and unique norms for children and adolescent samples (see Shaffer et al., 2007). The influence of psychoanalytic perspectives is evident in investigatory interest in schizophrenia and other psychotic states, psychosomatic formulations, psychodynamic factors, and psychotherapy process and outcome. To some extent, this orientation reflects research presented in psychoanalytically-oriented journals.

In viewing the categorical rankings, a couple of observations are noteworthy. First, the prevalence of studies on various eating disorders. Second, a major focus on forensic psychology issues, such as forensic assessment, sexual offenders, psychopathy, and child abuse. Finally, an appreciation for the historical foundations of the Rorschach is reflected in a myriad of reviews to the present day.

Noteworthy, although the Wood et al. controversy/debate (e.g., Garb et al., 2008; Mihura et al., 2013; Wood et al., 2003) has generated some attention in the literature in the form of primary research (n=10), much of the scholarly debate has been in the form of commentaries/replies and rebuttals (and thus not included in the current analysis). Scholarly

„exchanges‟ reported in journals tend to inflate

the exposure that select, controversial topics receive based to some degree on editorial preference (Moustafa, 2015).

Studies with a major focus on Rorschach variables/indexes are as follows (in rank order): Color (n=10), Movement (n=8), # of Responses (n=5), Ego Impairment Index (n=5), Mutuality of Autonomy (n=5), Oral Dependency (n=5), Texture (n=4), R- Perceptual Thinking Index (n=3).

Neglected Areas of Research: Topical analysis frequently provides evidence on areas of sparse research interest. The current findings, regarding the distribution of major subject categories, identified several salient areas of psychological study largely neglected by Rorschach researchers. Most noteworthy: assessment training/education (n=6), examiner/examinee interaction (n=3), differential diagnosis (n=3), creativity (n=5), anxiety (n=3), race/ethnic differences (n=2), physically disabled (n=2), learning disabled (n=3), aged populations (n=4), pain conditions (n=4), grief states (n=1), social desirability (n=1).

Preferential Editorial Selection: Publication bias entails issues related to editorial selection, non-significant results, and decision-making on the part of researchers regarding submission of their findings (e.g., Banks et al., 2012; Matias-Guiu & Garcia- Ramos, 2011; Torgerson, 2006). Although the current analysis does not directly address publication bias, the issue of editorial selection and preference may be a factor on whether an individual paper „sees the light of day‟. The current findings reveal a disproportional number of articles on norms/normative analysis and psychotic states, and few articles across a myriad of salient clinical issues. From a bibliometric perspective, such findings can reflect bias in the editorial process (see Bornstein 1991; Moustafa, 2015) and potential conflict of interest (Young, 2009). This effect may be particularly evident in studies on aggregated reviews of the literature (Coursol & Wagner, 1986). Interestingly, in a meta-analysis involving the Rorschach, Parker et al. (1988) commented, “The historical association between  the  Journal  of  Personality

Assessment and the Rorschach may create an editorial bias favorable to the Rorschach” (p. 372).

Limitations: First, this analysis reports on only the most recent 20% (2000-2016) of the extant research on the Rorschach presented in articles in the database PsycINFO; if a complete examination of archival articles (n=4000+) in the PsycINFO database was conducted, a more comprehensive, historical review would be evident. Undoubtedly, additional studies would have been identified via searches of adjutant databases in the social sciences and in MEDLINE. Second, this study reports a somewhat basic „topical‟ content analysis of the psychological literature, from the perspective of one examiner. Thus, a multi-rater design would enhance scoring reliability. By design, only 1 designation was permitted. A weighted scoring scheme would surely have produced a more refined, accurate ranking. Furthermore, in order to further our understanding of the structure and influence of scholarship regarding the Rorschach, more advanced bibliometric methods (i.e., citation analysis, citation mapping, co-citation analysis) are required (see Hjorland, 2013; Piotrowski, 2013), particularly as the intellectual structure and volume of published literature in the Assessment Psychology field expands over time (see Adair & Vohra, 2003, for a discussion).

Lastly, this analysis did not include the „Gray‟ literature (Conference papers and proceedings, dissertations, working papers, Google postings), nor the great volume of unpublished papers that never see the light of day. Most of this literature is not archival. Indeed, the oft-referred to „File Drawer Problem‟ contributes to publication bias and limits findings in bibliometric studies (Franco et al., 2014; Rotton et al., 1995). Indeed, Lilienfeld et al. (2000) discussed the File drawer problem in their extensive review of the extant literature on projective techniques. Thus, to what extent „unpublished‟ studies on the Rorschach would have impacted the current results is not discernable.

Conclusions:

1. Despite copious levels of scholarship on inkblots for the past century, Rorschach research, overall, seems a diverse enterprise with disparate intellectual pursuits. From an international perspective, the majority of Rorschach investigations overseas seem disengaged from Rorschach research in the USA. The potent impact of Exner‟s Comprehensive System, over several decades on the assessment field, has now been challenged by the R-PAS. Thus, there seems little cohesion in the field regarding the direction for future research on Rorschach assessment.
2. There has been sparse attention in published articles regarding graduate-level training and internship opportunities with the Rorschach, perhaps reflecting diminishing levels of didactic coursework and clinical experiences which have been noted in the recent literature (Evans & Finn, 2016; Piotrowski, 2015b). Where advanced students and novice practitioners, with an avid interest in gaining competency in the Rorschach can turn, seems a perplexing conundrum.
3. There appears to be a host of critical clinical areas that receive sparse investigatory attention. Whether this is a function of disinterest on the part of investigators or editorial preference/bias is difficult to ascertain.
4. Contemporary Rorschach research tends to highlight competing academic and professional dictates; i.e., training in clinical mandate for in-depth, multimethod assessment protocols appears incongruent with the reality of expedited testing constraints in practice.
5. Despite nearly a century of research, it appears that the Rorschach is an emerging area of study reflecting a myriad of scholarly

-interests and academic orientations.

1. The current findings are exploratory in nature; more advanced bibliometric methods need to be applied to the extant literature in order to elucidate the investigatory structure of Rorschach research.

References:

Adair, J.G., & Vohra, N. (2003). The explosion of knowledge, references, and citations: Psychology‟s unique response to a crisis. American Psychologist, 58, 15-23.

Banks, G.C., Kepes, S., & Banks, K.P. (2012). Publication bias: The antagonist of meta-analytic reviews and effective policymaking. Educational Evaluation & Policy Analysis, 34(3), 259-277.

Bornstein, R.F. (1991). Manuscript review in psychology: Psychometrics, demand characteristics, and an alternative model. Journal of Mind and Behavior, 12(4), 429-467.

Cascio, W.F., &Aguinis, H. (2008). Research in I/O psychology from 1963 to 2007: Changes, choices, and trends. Journal of Applied Psychology, 93, 1062-1081.

Coursol, A., & Wagner, E.E. (1986). Effect of positive findings on submission and acceptance rates: A note on meta-analysis bias. Professional Psychology: Research and Practice, 17, 136-137.

Cox, B.J., Wessel, I., Norton, G.R., Swinson, R.P., &Direnfeld, D.M. (1995). Publication trends in

anxiety disorders research: 1990-1992. Journal of Anxiety Disorders, 9(6), 531-538.

De Bellis, N. (2009). Bibliometrics and citation analysis.

New York: Scarecrow Press.

Evans, F.B., & Finn, S.E. (2016).Training and consultation in psychological assessment for professional psychologists. Journal of Personality Assessment, 98, 785-798.

Franco, A., Malhotra, N., &Simonovits, G. (2014). Publication bias in the social sciences: Unlocking the file drawer. Science, 345(6203), 1502-1505.

Garb, H.N., Wood, J.M., Lilienfeld, S.O., Nezworski, M.T. (2005).Roots of the Rorschach controversy. Clinical Psychology Review, 25, 97-118.

Garcia-Perez, M.A. (2010). Accuracy and completeness of publication records in the Web of Science, PsycINFO, and Google Scholar: A case study for the computation of h indices in psychology. Journal of the American Society Information Science & Technology, 61, 2070-2085.

Giegerich, M. (2014). Personality assessment as portrayed in introductory psychology textbooks: A qualitative content analysis. Dissertation Abstracts International, 74(9-B).

Hjorland, B. (2013). Citation analysis: A social and dynamic approach to knowledge organization. Information Processing and Management, 1313- 1325.

Manickam, L.S., & Dubey, B.L. (2005). Rorschach Inkblot Test in India: Historical review and perspectives for future action. Journal of Projective Psychology & Mental Health, 12(1), 61-78.

Matias-Guiu, J., & Garcia-Ramos, R. (2011).Editorial bias in scientific publications.Neurologia, 26(1), 1-5.

Meyer, G.J. (2001). Introduction to the final special section in the special series on the utility of the Rorschach for clinical assessment. Psychological Assessment, 13(4), 419-422.

Mihura, J.L., Meyer, G.J., Dumitrascu, N., & Bombel, G. (2013). The validity of individual Rorschach variables: Systematic reviews and meta-analyses of the Comprehensive System. Psychological Bulletin, 548-605.

Miles, J.A., &Naumann, S.E. (2011). Research trends in the Academy of Management publications. Journal of Management and Marketing Research, 6, 1-31.

Moustafa, K. (2015). Is there bias in editorial choice? Yes.

Scientometrics, 105, 2249-2251.

Parker, K. C., Hanson, R.K., &Hunsley, J. (1988). MMPI, Rorschach, and WAIS: A meta-analytic comparison of reliability, stability, and validity. Psychological Bulletin, 103(3), 367-373.

Perdue, B., & Piotrowski, C. (1991). Online database use in psychology: A survey of academic libraries. Collection Management, 14, 133-137.

Piotrowski, C. (2016). Mapping the research domain in the field of applied psychology: A bibliometric analysis of the emerging literature. Journal of the Indian Academy of Applied Psychology, 42(1), 11- 17.

Piotrowski, C. (2015a). Projective techniques usage worldwide: A review of applied settings 1995-2015. Journal of the Indian Academy of Applied Psychology, 41(3), 9-19.

Piotrowski, C. (2015b). Clinical instruction on projective techniques in the USA: A review of academic training settings 1995-2014. Journal of Projective Psychology & Mental Health, 22(2), 83-92.

Piotrowski, C. (2013). Bibliometrics and citation analysis for the psychologist-manager: A review and select readings. The Psychologist-Manager Journal, 53-71. Piotrowski, C. (2012). Research areas of emphasis in professional psychology: Past and current trends.

Journal of Instructional Psychology, 39(2), 131-135.

Piotrowski, C., & Keller, J.W. (1993). The Rorschach and shading: Citation output from PsycINFO (1987- 1992). Psychological Reports, 72(2), 690.

Piotrowski, C., Perdue, B., & Armstrong, T. (2005). Scholarly online database use in higher education: A faculty survey. Education, 125, 443-445.

Polyson, J., Peterson, R., & Marshall, C. (1986). MMPI and Rorschach: Three decades of research. Professional Psychology: Research and Practice, 17(5), 476-478.

Reynolds, W.M., &Sundberg, N.D. (1976). Recent research trends in testing. Journal of Personality Assessment, 40, 229-233.

Rotton, J., Foos, P.W., Van Meek, L., & Levitt, M. (1995). Publication practices and the File Drawer Problem: A survey of published authors. Journal of Social Behavior and Personality, 10(1), 1-13.

Shaffer, T.W., Erdberg, P., & Meyer, G.J. (2007). Introduction to the JPA special supplement on international reference samples for the Rorschach Comprehensive System. Journal of Personality Assessment, 89(Suppl 1), S2-S6.

Torgerson, C.J. (2006). Publication bias: The Achilles‟ heel of systematic review? British Journal of Educational Studies, 54(1), 89-102.

Wood, J.M., Nezworski, M.T., &Lilienfeld, S. O. (2003). What‟s wrong with the Rorschach? Science confronts the controversial inkblot test. New York: Wiley.

Wright, C.V., Beattie, S.G., Galper, D.I., et al. (2016). Assessment practices of professional psychologists: Results of a national survey. Professional Psychology: Research and Practice, 47, 617-625.

Young, S.N. (2009). Bias in the research literature and conflict of interest. Journal of Psychiatry and Neuroscience, 34(6), 412-417.