Construct Validity of the Pictogram Test for Diagnosis of Schizophrenia

Objectives: To examine construct validity of the Pictogram Test (PT) which assesses disturbances in thinking in individuals with schizophrenia. The PT was developed in Russia; it was found to be applicable for the English-speaking population of the USA. The variables of the PT were correlated with Minnesota Multiphasic Personality Inventory (MMPI-2). Method: Russian-and English-speaking participants completed the PT and MMPI-2 in their native languages. Results: The PT variables that reflected attribute selection choice of intermediate concepts for memorization and the variables that were geometrical shapes had significant correlations with MMPI-2 scales linked to schizophrenia. This represents evidence of convergent validity. The same PT indices did not significantly correlate with most of the MMPI-2 scales that are not elevated in schizophrenia, representing some discriminant validity of the PT. Conclusions: The less often the participants were able to connect target words with economical intermediate concepts, the higher were the elevations of schizophrenia-related scales. Also, the more abstract and remote their intermediate concepts, the less often they recalled targets. These findings give evidence of the validity of the PT in assessing the thinking of individuals with schizophrenia and related conditions.


Introduction Thinking distortions and schizophrenia
Cognitive disturbances and disordered associations in schizophrenia were outlined by the forefathers of modern psychiatry Eugen Bleuler [1] and Emil Kraepelin [2]. Now, over 100 years later, there are many studies that prove the concept that cognitive deficits precede the development of positive symptoms of schizophrenia (i.e. hallucinations and delusions) [3][4][5]. In his recent article, Kahn outlined the path to new studies in the area of investigating cognition in early adolescence, including using biomarkers, brain imaging, and genetics to understand better the causes of schizophrenia [6]. Finding new and sensitive psychological tests that assess cognition might be a helpful tool for such new studies. Logical memory has many indirect definitions in the literature. The main theme of these definitions is that logical memory involves memory storage and consolidation, which in turn involves meaning. Lezak et al. [7] and Squire [8] described that patients with damage to the medial temporal limbic and the midline diencephalon areas are more likely to have significant memory storage and consolidation problems. Due to their structural brain damage, these patients cannot assign meaning to the materials needed to be remembered. Recent research found that patients with schizophrenia have abnormal brain connectivity in the same areas that precede the development of positive symptoms [9,10].
Developing psychological testing that could tap into associative or logical memorization could aid the early diagnoses of those vulnerable to developing schizophrenia. As logical thinking is problematic in individuals with schizophrenia, such test results could help in the evaluation of treatment efficacy as well as the differential diagnosis of schizophrenia from other psychotic disorders. The most commonly used logical memory test is the Logical Memory (LM) subtest of Wechsler Memory Scale-IV [11]. However, the LM test does not give us information on the mechanism of memorization. The Pictogram Test (PT) taps into associative memory and the meaning the test-taker attaches to the materials needed to be memorized and later retrieved. The construct validity of the PT is investigated in this study by the means of convergent and discriminant validation of the three PT indices with Minnesota Multiphasic Personality Inventory 2nd edition (MMPI-2) scales [12]. Convergent validity was examined by correlating PT indices with MMPI-2 scales that are associated with schizophrenia: Paranoia (Pa), Schizophrenia (Sc), Bizarre Mentation (BIZ), Bizarre Sensory Experiences (Sc6,) and Persecutory Ideas (Pa1). Discriminant validity was examined by the PT indices correlation with MMPI-2 scales elevation, which does not necessarily signify schizophrenia: Hypochondriasis (Hs), Depression (D), Hysteria (Hy), Psychopathic Deviate (Pd), Masculinity-Femininity (Mf), Psychasthenia (Pt), Hypomania (Ms), and Ego Strength (Es).

Brief validity overview
Construct validity is one of the main psychometric properties that reflect a test's ability to measure what it intends to measure. Two of the several sources of evidence of the presence of construct validity are convergent and discriminant validation procedures. Convergent validity shows that a certain test correlates highly with another test that measures the same behavioral/psychological domain. If the test variables measure the same behavioral or psychological domain as the other test variables measure, it is possible to conclude that this test is a valid estimation of specific behavior governed by this psychological domain. Discriminant validity shows that a certain test does not correlate with tests that measure different behaviors.

Materials And Methods
The project was approved by the West Virginia University School of Medicine Institutional Review Board (IRB) with approval number 15795-B for English-speaking subjects and Ethics Committee of the 3rd Psychiatric Hospital in St. Petersburg, Russia, with approval number 942 for Russian-speaking subjects and was in accordance with requirements as outlined in the Declaration of Helsinki.

Data collection
A detailed description of the study participants and method of recruitment is outlined elsewhere [13]. Briefly, the data were collected between 2002-2003, half of the participants were English-speaking from the USA while the other half were Russian-speaking from Russia. There were one-half of inpatients diagnosed with schizophrenia and schizoaffective disorder and one-half normal, control participants in each language group.

Inclusion and exclusion criteria
The diagnosis of schizophrenia in the patients' charts was the criterion for inclusion. We did not include patients who were too unstable or disorganized to complete the assessment. Examples of these were patients actively hallucinating, did not understand the tasks, did not follow instructions, aggressive and hostile. No specific scales for exclusion criteria were administered.

Measures
The Beck Depression Inventory-II (BDI-II) [14], MMPI-2, and PT were the measures used in the original study. For the purpose of the present report, only MMPI-2 scales of interest and the PT data are analyzed. All measures were administered in the participants' native language (Russian or English).

Pictogram test
The detailed description of the PT [15] and its cross-cultural applicability is described elsewhere [13]. Briefly, the PT involves memorization of 16 words and phrases that vary in their degree of abstraction by using examinees' own associative drawings that are supposed to have meaning for the examinees' connections to the words/phrases to be memorized. The PT assessment form is included in the appendices section of this article.
The artistic quality of drawings was not judged. The PT was found applicable for an English-speaking population and found to have high inter-rater reliability and internal consistency [13]. The same study conducted the correlational analysis of PT variables, which allowed examination of the variables in relationship to each other; the number of variables were combined into statistically manageable and conceptually meaningful indices. These indices reflect participants' use of intermediate concepts to memorize abstract materials. The indices are: Concrete Index (CI), Attribute Index (AI), and Geometric Index (GI). CI contains photographic-like drawings of people, usually engaged in some activity. Such drawings may be cumbersome and may or may not be helpful in retrieval. AI contains more concise drawings selected according to the principle of belonging (i.e., Christmas tree for "happy holidays") and is helpful in retrieval. GI is characterized by meaningless/vacuous drawings and is very rarely helpful in retrieval. A previous study revealed that AI was the strongest discriminator, correctly classifying 91% of English-speaking and 86% of Russian-speaking participants as either patients or non-patients, as cited in [13]. This indicates the importance of this PT index when assessing logical thinking disturbance in schizophrenia. Patients with schizophrenia lack the ability to form economic and logically connected intermediate concepts for the productive memorization of abstract materials in their PTs (low AI). GI was also significant in its separation of patients from non-patients. Patients with schizophrenia tended to use geometric forms as mediators, which did not help them with memorization. CI did not discriminate patients from non-patients in the prior study [13].

Minnesota multiphasic personality inventory (MMPI-2)
MMPI-2 [12] test represents a 567-item self-administered yes/no questionnaire. All participants completed all 567 questions; for some patients from the schizophrenia group the questions were read by the examiner and the administration was divided into parts due to fatigue. For Russian participants, the research MMPI-2 Russian version was used [12,16]. The following scales were used for convergent validity: clinical scales 6 (Pa = Paranoia), 8 (Sc = Schizophrenia), content scale BIZ (Bizarre Mentation), supplementary scale Sc6 (Bizarre Sensory Experiences) and Harris-Lingoes subscale Pa1 = Persecutory Ideas. Test-retest reliability of MMPI-2 scales Pa, Sc, BIZ, Sc6, and Pa1 for diagnoses of schizophrenia ranges from 0.58 to 0.89; internal consistency ranges from 0.34 to 0.87. These scales have been found to be associated with the psychiatric diagnosis of schizophrenia, thus showing criterion-related validity [17,18]. The following scales that are not necessarily and consistently elevated in schizophrenia spectrum disorders were used for discriminant validity: We excluded MMPI-2 protocols that had elevated validity scales suggesting invalidation of the test responses. Specifically, we excluded those protocols that had scales L higher than 65, F higher than 100, and K higher than 65. This left us with a total of 87 protocols. Thus, the correlational analysis included 87 MMPI-2 and Pictogram Test protocols, both from patients and non-patients.

Research design, hypotheses, and statistical data analysis
A correlational analysis was used for the present paper. Three PT indices (CI, AI, and GI) were correlated with MMPI-2 scales Pa, Sc, BIZ, Pa1, and Sc6 for convergent validity and with scales 1, 2, 3, 4, 5, 7, 9, 0, and Es for discriminant validity. Pearson correlation coefficients of 0.3 and above were considered as reflective of commonly shared variances of MMPI-2 scales and the PT indices. In the statistical analysis of the data, a hypothesis that a correlation coefficient between two variables is significantly different from zero was tested (p < .05). The practical significance of the correlation coefficient was interpreted. In this study, we used patient and non-patient data to increase the total number of protocols used in the correlational analysis. Adjusted p values are calculated from Benjamini and Hochberg method to control the false discovery rate given multiple comparisons [19].

Convergent Validity
AI and GI had significant correlations with all MMPI-2 scales of interest. Specifically, the AI had significantly high negative correlations with all MMPI-2 scales of interest: Pa, Sc, Pa1, and Sc6; and a significant moderate to high negative correlation with BIZ. These correlations indicate that the higher the Tscores on these scales, the lower the AI scores. The GI had significant moderate positive correlations with MMPI-2 scales Pa, Sc, BIZ, and Sc6, and significant positive low to moderate correlation with Pa1. These correlations indicate that the higher the T-scores on these scales, the higher the GI scores. The CI did not correlate significantly with any of the MMPI-2 scales of interest.

Discriminant Validity
AI had a low moderate negative correlation with scales Hs, D, Mf, Pt, and Si. These correlations mean that the higher the T-scores on these scales, the lower the AI. AI has a highly significant positive correlation with Es. GI had significantly low positive correlations with MMPI-2 scales Hs, D, Hy, Pt, and a highly significant negative correlation with Es. CI had only a weak significant positive correlation with scale Mf. See Table 2 for correlational analysis.

Discussion
This study tested the construct validity of the PT by examining convergent and discriminant validities of the indices created from the PT variables with MMPI-2 scales. The construct validity is the most important psychometric property of any psychological test as it proves that the test is measuring what it is supposed to measure [20]. Our results demonstrate that the AI had the strongest significant correlations with MMPI-2 scales indicative of schizophrenia spectrum illnesses: Paranoia, Schizophrenia, Bizarre Mentation, Persecutory Ideas, and Bizarre Sensory. Specifically, the higher the elevation on these scales, the less likely the drawings were succinct and economical and the less they were chosen based on principles of belonging.
In other words, patients with schizophrenia or schizoaffective disorder rarely created intermediate drawings that were "items" (i.e., shovel for "hard work" or Christmas tree for "happy holiday" or thermometer for "illness") which could be helpful for memorization and recall. Instead, they frequently drew unrelated geometrical shapes that did not help them in recall (GI). GI had a significant positive correlation with scales Pa, BIZ, and Sc6. Thus, both AI and GI correlations with MMPI-2 scales indicative of schizophrenia give evidence of convergent validity of these PT indices.
We examined the discriminant validity of the PT indices. We selected the MMPI-2 scales that are nonnecessarily indicative of schizophrenia. The AI does not have significant correlations with scales Hy, Pd, Mf, and Ma, which give evidence of discriminant validity of AI. Results indicated that AI does correlate significantly, weakly, and negatively with scales Hs, D, Pt, and Si. These negative correlations could be explained by the fact that those who have schizophrenia/schizoaffective disorder could commonly be depressed (D elevation) or/and socially isolated (Si elevation) and have many somatic complaints (Hs and Pt elevation Recently proposed theory on the linguistics of schizophrenia and thought disorder associated with it states that positive symptoms and thought disorganization fall into place as failures in language-mediated forms of meaning [21]. Associated memory with the intermediate concepts that the PT taps into shows exactly the meaning the test-takers attach to memorize the abstract concepts. The finding of the current study that PT has a construct validity with MMPI scales associated with schizophrenia gives evidence of the concept that the PT measures. This concept is that associative memory is deficient in patients diagnosed with schizophrenia and related illnesses. The CI index did not show significant correlations with any MMPI-2 scales except Mf (moderate positive correlation). This could be an incidental finding. Thus, the predominance of concrete intermediate concepts (i.e., a scene with people) although not ideal and economical for memorization, did not signify schizophrenia spectrum illness.

Limitations
We have no data on the severity of schizophrenia symptoms measured by the standardized instruments. There is a chance that co-morbid disorders such as traumatic brain injury or intellectual/learning disability contribute to some patients' performance. We have no explanation at this point of why CI had mild significant correlations with Mf scales (measures of masculinity-femininity). There is a small chance that individuals who have more concrete drawings (CI) had disturbances in memory as well, which prevents them from having more efficient memorization.

Conclusions
In summary, the present analysis indicates that two PT indices, AI and GI, correlated with some relevant MMPI-2 scales and, thus, provide evidence for the PT's convergent and discriminant validity. Both of these validities give evidence for construct validity of the PT with MMPI-2 scales elevated in schizophrenia. With the evidence of the construct validity of the PT, this test could be added to psychological assessment batteries when the clinical question of schizophrenia diagnosis is being raised. Such addition could give an incremental validity to the result of the test battery and, as a result, more diagnostic certainty. With more diagnostic certainty and accuracy, more targeted treatment and remediation of schizophrenia could be designed.
interest in the submitted work. Other relationships: All authors have declared that there are no other relationships or activities that could appear to have influenced the submitted work.