Validity of Psychological Tests

Official syllabusPaper II · Unit 1 · Psychological Measurement of Individual Differences

Characteristics and construction of standardized psychological tests – validity.

Definition

“Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment.”

Messick (1989)

Reliability asks whether a test measures consistently. Validity asks whether it measures what its users take it to measure, and whether decisions built on it are justified.

Topic map of validity of Psychological Tests: What validity is (Kelley (1927) → Messick (1989)); Content and criterion (Sampling and forecasting); Construct (Cronbach & Meehl…

Each box is a different kind of evidence for one claim: that a score means what its user says it means.

The rationale behind the use of psychological tests

  • Anastasi (1976): “A psychological test is essentially an objective and standardized measure of a sample of behavior.” Standardization, objectivity and norms make scores comparable, and a short sample forecasts behaviour that would take months to observe.
  • Meehl (1954) opened the clinical-versus-statistical debate; Grove, Zald, Lebow, Snitz and Nelson (2000), pooling 136 studies, found mechanical prediction about 10 per cent more accurate than clinical judgement on average.
  • The rationale is conditional: each test, wrote Anastasi (1976), “must prove its worth by an empirically demonstrated correspondence between the subject’s performance on the test and in other situations”.

Validity among the characteristics of standardized psychological tests

  • Kelley (1927): “The problem of validity is that of whether a test really measures what it purports to measure.”
  • Validity became a coefficient — Guilford (1946): “in a very general sense, a test is valid for anything with which it correlates.”
  • The APA’s Technical Recommendations (1954) named content, predictive, concurrent and construct validity; the Standards (1966) merged the middle two as criterion-related validity, producing the “trinity”.
  • Cronbach (1971): “One validates, not a test, but an interpretation of data arising from a specified procedure.” Validity belongs to an interpretation, comes in degrees, and is specific to a purpose and population.
  • Loevinger (1957) called construct validity “the whole of validity from a scientific point of view”; Messick (1989) folded content and criterion evidence into one construct-based judgement that also weighs value implications and social consequences.
  • Messick (1995) split unified validity into six aspects — content, substantive, structural, generalizability, external, consequential — and named two threats: construct underrepresentation, a test too narrow for its construct, and construct-irrelevant variance, scores partly driven by something else, such as reading load in mathematics.
  • The Standards (2014) of AERA, APA and NCME define validity as “the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests”.
    • Five sources of evidence: test content, response processes, internal structure, relations to other variables and consequences of testing.
  • Kane (1992) framed validation as an argument-based approach, restated in 2013: the interpretation is a chain of inferences — scoring, generalization, extrapolation, decision — tested at its weakest link.
Kinds of Validity Evidence: ContentLawshe (1975), FaceMosier (1947), ConcurrentCronbach & Meehl (1955), Predictive, Convergent and discriminantCampbell & Fiske (1959), FactorialGuilford (1946), Response processes,…

The old trinity survives as headings, but each type is now one source of evidence feeding a single judgement.

Content validity, and why face validity is not validity

  • Content validity is the degree to which items represent a defined domain; Cronbach and Meehl (1955) held it “is ordinarily to be established deductively, by defining a universe of items and sampling systematically within this universe to establish the test.”
  • The procedure: domain analysis, a test blueprint crossing content with cognitive process, items written in proportion, and subject-matter experts judging relevance and coverage.
  • Lawshe (1975) devised the content validity ratio, CVR = (ne − N/2) / (N/2), where ne of N panellists rate an item “essential”. Twelve of fifteen give .60, above the .49 tabled for that panel; the retained items’ mean is the content validity index.
  • Content evidence carries achievement, job-knowledge and work-sample tests, and Lawshe (1975) confined it to observable behaviour. It describes the test, not the scores: a well-sampled paper still misleads if test-wiseness or translation alters item difficulty.
  • Face validity, dissected by Mosier (1947), is how relevant a test looks; Anastasi (1988) ruled it out: it “is not validity in the technical sense; it refers not to what the test actually measures, but to what it appears superficially to measure.”
  • It aids rapport and acceptance, but on personality inventories high face validity invites faking — which is why empirically keyed scales keep “subtle” items.

Criterion-related validity: concurrent and predictive

  • Criterion-related validity is the correlation of scores with an independent measure of the behaviour they should indicate — the validity coefficient, whose square is the share of criterion variance accounted for.
  • Cronbach and Meehl (1955): in predictive validity the criterion follows testing, as with entrance scores against first-year grades; in concurrent validity both are obtained together, as with a new depression scale against current diagnosis. The contrasted-groups method checks that groups known to differ score apart.
  • Wissler (1901) correlated the “mental tests” of J. McKeen Cattell (1890) with the class standing of over 300 Columbia and Barnard students.
    • Reaction time correlated −.02 and the best test, logical memory, .19, while grades correlated substantially with one another; the sensory-test programme collapsed.
    • Limitation: an all-college sample restricted the range, and a few reaction-time trials made that measure unreliable.
  • The criterion problem: criteria can be deficient, contaminated (rated by people who have seen the scores) or unreliable, and need validating themselves.
  • Range restriction — validating on people already selected depresses coefficients; Sackett, Zhang, Berry and Lievens (2022) argued the usual corrections had been over-applied.
  • Validity generalization: the scattered validities compiled by Ghiselli (1966) were read as situational specificity until Schmidt and Hunter (1977) showed most of the variation was sampling error and artefact, so meta-analytic validity transfers to new settings.
  • Taylor and Russell (1939): if half of applicants would succeed anyway, a test of validity .50 used to take the top 30 per cent lifts success among those chosen from 50 to 74 per cent.

Construct validity and the nomological network

  • Cronbach and Meehl (1955): “Construct validation is involved whenever a test is to be interpreted as a measure of some attribute or quality which is not ‘operationally defined.'”
  • A construct is fixed by its nomological network, “the interlocking system of laws which constitute a theory”. Validation tests its predictions through group differences, correlations and factor analysis, change over occasions and studies of process; a failed prediction indicts the test, the theory, or both.
  • A test-anxiety scale, for instance, should rise before examinations, track arousal and fall after desensitisation.
  • Their rule: “Rationalization is not construct validation.”

Convergent and discriminant validity: the multitrait-multimethod matrix

  • Campbell and Fiske (1959) treated each score as a trait-method unit and required convergent validity, agreement across methods measuring one trait, and discriminant validity: “Tests can be invalidated by too high correlations with other tests from which they were intended to differ.”
  • The multitrait-multimethod (MTMM) matrix crosses two or more traits with two or more methods, giving a reliability diagonal, validity diagonals (same trait, different method), and heterotrait-monomethod and heterotrait-heteromethod triangles.
  • Four criteria: validity values significant and substantial; above the heterotrait-heteromethod values in their row and column; above the heterotrait-monomethod values; and one pattern of trait intercorrelations in every triangle.
  • Inflated monomethod values expose method variance; Campbell and Fiske (1959) cited the halo ratings of E. L. Thorndike (1920), in which a teacher’s intelligence correlated .63 with voice quality.
  • A NIMHANS-led expert committee (report May 2008) judged BEOS brain-profiling, used in a Pune murder conviction that June, to have a “sub-optimal scientific basis”, making no allowance for heart rate, fatigue or drugs — rivals a discriminant check must exclude.

Factorial validity

  • Guilford (1946) defined factorial validity as a test’s loading on a factor — a vocabulary test loading .85 on a verbal factor has factorial validity of .85. Confirmatory analysis and measurement invariance across groups add internal-structure evidence (mechanics in Factor Analysis and Multivariate Techniques, Paper I).
  • Limitation: naming a factor is interpretation, and a clean structure says nothing about what the scores forecast.

Incremental and consequential validity

  • Incremental validity, named by Sechrest (1963), is the gain over predictors already available, read as the change in R². In Schmidt and Hunter (1998) an integrity test (.41), uncorrelated with ability, raised a mental-ability test’s .51 to .65 — more than a work sample (.54) did.
  • Consequential validity is the inclusion, by Messick (1989), of the social effects of use; Messick (1995) counted a harm against validity only if it traces to underrepresentation or irrelevant variance. Popham (1997) objected that this files ethics under measurement.

Convergent and discriminant validity of the Civil Service (Preliminary) Test

  • The examination’s two objective papers make construct claims: Paper I samples general awareness; Paper II, the Civil Services Aptitude Test, includes comprehension, logical reasoning and analytical ability, decision-making, general mental ability and basic numeracy.
  • Traits: T1 reading comprehension, T2 logical-analytical reasoning, T3 general awareness.
  • Methods, as unlike as possible: M1 the examination’s machine-scored multiple-choice items; M2 constructed-response tasks — a précis, short written answers; M3 a structured oral exercise scored by two trained assessors.
  • Procedure: several hundred recent candidates stratified by medium, discipline, gender and region; all nine measures within a fortnight; a reliability estimate for each.
  • Expected pattern:
    • reliabilities highest;
    • validity diagonals substantial — T1 by M1 agreeing with T1 by M2 and by M3;
    • heterotrait-monomethod values lower — if comprehension and reasoning correlate more within multiple choice than comprehension does across methods, a format factor (speed, test-wiseness) dominates;
    • heterotrait-heteromethod values lowest;
    • one trait pattern throughout — T1 and T2 moderately related through general ability, T3 less so.
  • Beyond the matrix: scores should not track medium, engineering degrees or coaching once T1–T3 are controlled, and differential item functioning analysis compares the English and Hindi versions item by item.
  • Analysis: apply the four Campbell-Fiske criteria, fit a confirmatory model with trait and method factors, and rewrite method-heavy or translation-sensitive items. Whether Preliminary scores forecast Mains scores is separate criterion evidence, range-restricted by the selection itself.
Validating the Civil Service (Preliminary) Test: Traits (What the papers claim to measure); Methods (As unlike one another as possible); Campbell & Fiske (1959) (Four criteria to…

Convergence is read along the validity diagonals; discrimination by comparing them with everything around them.

Methods of assessing validity of a personality inventory

  • Content evidence — experts judge item coverage of the defined trait; it ignores candour.
  • Criterion keying and contrasted groups — Hathaway and McKinley (1943) kept an MMPI item only if a diagnosed group answered it differently from 724 “Minnesota normals”; items, wrote McKinley and Hathaway (1944), “are not selected for their content or theoretical import.”
    • Significance: validity is built in, and subtle items resist faking. Limitation: a shared demoralization factor made the scales intercorrelate too highly — the discriminant failure Tellegen and colleagues (2003) corrected with the Restructured Clinical scales.
  • Concurrent and predictive criteria — diagnosis, informant ratings or job performance, which justify practical use; Sackett and colleagues (2022) put conscientiousness at about .19.
  • Convergent and discriminant correlations — high with other inventories of the trait and with peer ratings, a different method; low with other traits and social desirability, so the score is the trait, not self-presentation.
  • Factorial validity across cultures:
    • Lodhi, Deo and Belhekar (2002) adapted the NEO-PI-R into Marathi (214 adults); Procrustes rotation recovered five factors invariant with the American structure, and the NEO-FFI’s correlations with Eysenck’s scales (300 adults) added convergent evidence.
    • Shetty, Thomas and Munoli (2023) gave the BFI-2-S in English or Kannada to 400 adults in Udupi and Mangalore: retest stability was good, yet exploratory factoring did not recover the Big Five.
    • Limitation: varimax-rotated components test a hypothesised structure less strictly than targeted rotation, so method partly decides the verdict.
  • Protocol validity — the L, F and K scales show whether a record is interpretable, as set out in Objective and Self-Report Assessment of Personality (Paper I); incremental evidence asks what the inventory adds to interview or ability data.

Validation in test construction: a new personnel selection test

  • Job analysis — observation, critical incidents and expert panels fix the tasks and the KSAOs (knowledge, skills, abilities and other characteristics) the test must sample.
  • Content evidence — a blueprint weighted by task importance, with expert item ratings and Lawshe’s CVR as the retention rule.
  • Construct evidence — convergence with an established test of the same ability, low correlation with what the job does not require, a factor structure matching the blueprint, and no differential item functioning.
  • Predictive criterion evidence — test applicants, select without the scores, seal them from supervisors to avoid criterion contamination, and correlate with structured performance or training ratings six to twelve months on; a concurrent design on incumbents is quicker but range-restricted.
  • Then cross-validation on a fresh sample, since derivation-sample validity shrinks; incremental validity over the existing interview; utility at the planned selection ratio; equal regression lines across groups for fairness (Standards, 2014); and validity generalization where samples are small.
  • Coefficient sizes are disputed. Sackett and colleagues (2022) held that range-restriction corrections behind Schmidt and Hunter (1998) were applied to mostly concurrent studies, where restriction is small, and re-estimated operational validities:
Selection procedureSchmidt and Hunter (1998)Sackett and colleagues (2022)
Structured interview.51.42
Job knowledge test.48.40
Work sample test.54.33
General mental ability test.51.31
Integrity test.41.31
Unstructured interview.38.19
  • The rank order largely survived, levels fell by .10–.20, and the structured interview moved to first; Oh, Le and Roth (2023) contested the reasoning, so the figures remain live estimates.

The relation between reliability and validity

  • Reliability is the consistency of whatever a test measures; validity is whether that is the intended thing. Random error lowers both; systematic error — bias, response style, a second construct — lowers validity while leaving reliability intact or higher.
  • Hence the asymmetry: an unreliable test cannot be valid, since random error correlates with nothing; a reliable one need not be, since consistency can be consistency of the wrong thing.
  • The ceiling: validity cannot exceed √rxx, the index of reliability, or √(rxx × ryy) with a fallible criterion. rxx = .81 caps validity at .90, and with ryy = .64 at .72; a test with rxx = .49 can still reach .70.
  • Correction for attenuation — Spearman (1904) estimated the true-score correlation as rxy / √(rxx × ryy): .40 with rxx = .80 and ryy = .50 becomes about .63. His own example, .38 with reliabilities of .58 and .22, corrected to about 1.
  • The attenuation paradox — Loevinger (1954) showed validity is not monotonic in reliability: near-duplicate items raise internal consistency but narrow the construct, and can lower validity against a broad criterion.
  • Reliable but not valid:
    • a clock thirty minutes fast;
    • response sets (Cronbach, 1946) — acquiescence and social desirability are stable, adding consistency and construct-irrelevant variance together;
    • the polygraph — in Selvi v. State of Karnataka (2010) the Supreme Court noted that the arousal recorded “could be triggered by nervousness, anxiety, fear, confusion or other emotions”;
    • the Kannada BFI-2-S, stable on retest yet not yielding the Big Five structure (Shetty and colleagues, 2023);
    • a software-cast horoscope, identical at every casting, yet in Narlikar, Kunte, Dabholkar and Ghatpande (2009) 27 astrologers sorting charts of bright and intellectually disabled children averaged 17.25 of 40, no better than chance.
  • Messick (1995) treated reliability as validity’s generalizability aspect; its estimation is in Reliability of Psychological Tests.
Reliability and Validity: Reliable, not valid, Neither, Reliable and valid; The ceiling (Validity ≤ √rxx); Spearman (1904) (Correction for attenuation); Consistent, but measuring the wrong thing (Reliable but not valid)

Reliability fixes how tightly the shots cluster; only validity says whether the cluster sits on the target.

Criticisms and limitations

  • The criterion is the weak link — Sohan Lall (1952), Chief Psychologist to the Ministry of Defence: “In most cases the criterion is either not available at all or if it is, it is unreliable.”
  • Low ceilings — Mischel (1968) coined the “personality coefficient” for the .20–.30 correlations questionnaires reach against criteria in another medium; Rosenthal and Rubin (1982) showed that r = .30 still gives correct dichotomous predictions about 65 per cent of the time.
  • Open-endedness — networks can always be adjusted, so Cronbach and Meehl (1955) warned against “a finespun network of rationalizations” passing as validation.
  • The realist objection — Borsboom, Mellenbergh and van Heerden (2004): “a test is valid for measuring an attribute if and only if (a) the attribute exists and (b) variations in the attribute causally produce variations in the outcomes of the measurement procedure.”
  • Practice lags theory — Flake, Pek and Hehman (2017) found 46 per cent of scales in sampled 2014 social-personality articles cited no prior validation.
    • Hussey and Hughes (2020) found 88 per cent of 26 widely used scales passed internal consistency but only 4 per cent a full structural battery.

Indian context

  • Civil Services Aptitude Test (UPSC; Department of Personnel and Training) — introduced in 2011 on the S. K. Khanna committee (2010) recommendation, replacing the optional paper of the Preliminary examination with a common aptitude paper so that every discipline competed on equal terms.
    • After 2014 protests alleging bias against Hindi-medium and humanities candidates, the Arvind Varma committee (2014) saw no case for change; English comprehension was dropped from merit, and a decision of 13 May 2015 made Paper II qualifying at 33 per cent.
    • Principle: consequential validity and suspected construct-irrelevant variance — questions that differential item functioning and differential-prediction evidence can settle.
  • Services Selection Boards and the Defence Institute of Psychological Research (DRDO) — a Dehradun experimental board (1943) became the Psychological Research Wing (1949) and DIPR (1982).
    • Fifteen Officer Like Qualities are rated independently by a psychologist, a group testing officer and an interviewing officer; agreement between techniques is convergent evidence.
  • Sohan Lall (1952) reported that officer-training wastage, 40 per cent under interview-only selection, fell to 9 per cent after psychological selection — criterion evidence, though uncontrolled for changes in intake and training.
  • Mission Karmayogi (Ministry of Personnel; approved 2 September 2020) — calibrates every post to a Framework of Roles, Activities and Competencies. Principle: job analysis, the basis of content validity for competency-based selection and training.
  • Guidelines for assessing specified disabilities (Ministry of Social Justice and Empowerment, March 2024, replacing those of 2018; RPwD Act 2016) — intellectual disability is assessed with the Vineland Social Maturity Scale plus an IQ test: Binet-Kamat, MISIC, WISC-IV or the NIEPID Indian Test of Intelligence.
    • Principle: validity for a purpose and population, since benefits follow.
  • Public Examinations (Prevention of Unfair Means) Act, 2024 (Ministry of Personnel; in force 21 June 2024) — penalises leaks and tampering in UPSC, SSC, railway, IBPS and NTA examinations. Principle: security protects validity, since leaked items make scores measure access, not ability.

Previous Year Questions

  • 2022Some measures can be reliable but not valid. Illustrate with relevant examples. 10 marks
  • 2019Even though validity often requires reliability, the reverse is not true. Explain. 10 marks
  • 2013How will you ensure that a newly constructed personnel selection test measures that it purports to measure and predicts what it intends to predict? Explain. 10 marks
  • 2009You have been asked to establish convergent and discriminant validity of the Civil Service (Preliminary) Test. How would you do it? 20 marks
  • 2008Enumerate the methods of assessing validity of a personality inventory and highlight their significance. 20 marks
  • 2002Elucidate the nature of individual differences. Describe the methods of validity and reliability.
  • 1995What is the rationale behind use of psychological tests? Critically examine reliability and validity of such tests, and their interdependence.
guest
0 Comments
Oldest
Newest Most Voted