Measurement of Intelligence: Methods and Instruments

measurement of intelligence and aptitudes.

Official Syllabus: Paper I · Unit 10 Intelligence and Aptitude

“Quelques philosophes récents semblent avoir donné leur appui moral à ces verdicts déplorables en affirmant que l’intelligence d’un individu est une quantité fixe… Nous devons protester et réagir contre ce pessimisme brutal.”

— Binet (1909), Les idées modernes sur les enfants, p. 141 — “Some recent philosophers seem to have given their moral support to these deplorable verdicts by affirming that an individual’s intelligence is a fixed quantity… We must protest and react against this brutal pessimism.” The book has no authorised English translation, so the rendering is this note’s own.

Intelligence testing began as an administrative instrument, not a theory. A ministry wanted to know which children could not profit from ordinary schooling; Binet built something that answered that question, and almost everything since has been an argument about how far the instrument may be pushed beyond the purpose it was built for. What the score means — the quotient, its constancy, what it predicts — is taken up separately.

Binet’s contribution

• The commission came first. In 1904 the French ministry of public instruction appointed a commission to identify children who needed special instruction, and Binet and Simon produced the first scale in 1905 — thirty tasks of increasing difficulty, ordered empirically by the age at which most children passed them.
• The decisive idea was the intellectual level, and the term matters. The 1908 revision grouped the tasks by age, so a child’s performance could be expressed as the age at which that performance is characteristic: a child of eight passing the tasks most ten-year-olds pass is performing at the level of ten. 1911 regularised this to exactly five tests per level. The innovation converted a heterogeneous set of tasks into a single developmental scale.
  • Binet never used the term “mental age.” His word throughout is niveau intellectuel — intellectual level. “Âge mental” does not appear in his writing at all; it enters the French literature only in 1912, after his death, in a translated article. Stern (1911) proposed the term, and Goddard and Terman carried it into English. Attributing “mental age” to Binet is the commonest error in this topic, and it misstates his own caution — a level is a rank, an age implies a quantity that grows.
• Binet (1905) chose items for complexity, not simplicity, sampling judgement, comprehension, reasoning and memory rather than sensory discrimination — a break from the Galton tradition, which had measured reaction time and sensory acuity and found them unrelated to anything that mattered.
• Binet’s own cautions are part of the contribution and are routinely omitted. He held that the scale ranked children but did not measure a quantity; that it could not be used to compare children of different backgrounds fairly; and that intelligence was not fixed, proposing “mental orthopaedics” to raise it. His protest against treating intelligence as a fixed quantity is the earliest statement of the objection the construct still faces.
• Terman (1916) produced the Stanford revision, standardised it on an American sample, extended it to adults, and adopted the quotient that made the score portable. From this point the scale leaves Binet’s custody and acquires the uses he warned against.

Wechsler’s contribution, and why it works better than Binet’s

• The problem Wechsler set out to solve was adults. The Stanford-Binet was built on children and extended upward; its adult items were simply harder children’s items, and its scoring assumed a growth curve that adults have left. The Wechsler-Bellevue (1939), and the adult, child and preschool scales descended from it, were standardised on adults from the start.
• The point scale replaced the age scale. Binet’s format groups items by age level and a testee passes or fails at levels. Wechsler grouped items by content into subtests, each scored cumulatively in points. This is the structural difference from which the others follow.
  • It yields a profile of subtest scores rather than a single level, so strengths and weaknesses are visible.
  • It permits separate composites — originally Verbal and Performance IQ, and in current editions index scores for verbal comprehension, perceptual or visual-spatial reasoning, working memory and processing speed.
  • It allows items of one kind to be given across a wide ability range without reorganising the test.
• The performance scale is the practical advance. A substantial non-verbal component makes the test usable with people whose language, literacy or schooling would otherwise depress a verbal score — and makes a verbal-performance discrepancy itself informative in clinical work.
• Deviation scoring was Wechsler’s. It removed the ratio IQ’s arithmetic defects at a stroke.
• The definition drove the design. Wechsler conceived intelligence as a global capacity manifesting in many ways, so a test had to sample several and report both the aggregate and the parts. The instrument is the definition made operational, which is why the subtest profile is not an add-on to it.
• The honest qualification. “More effective” is true for adults, for profile interpretation and for clinical use. The Stanford-Binet retains advantages at the extremes of ability, where its wider range of item difficulty discriminates better than a Wechsler scale whose floors and ceilings are tighter.

The Stanford-Binet fourth edition and its hierarchical model

• The fourth edition (1986) abandoned the age-scale format its predecessors had used since 1908 and adopted a three-level hierarchical model of cognitive ability.
  • Level one — g, a general reasoning factor, reported as the Test Composite.
  • Level two — three broad factors: crystallised abilities, fluid-analytic abilities, and short-term memory.
  • Level three — three group factors: Verbal Reasoning, Quantitative Reasoning and Abstract/Visual Reasoning.
• The four area scores are not the third level, and this is where the model is almost always misdescribed. Level three holds three factors, not four. The four area scores reported by the test are those three plus the Short-Term Memory factor taken from level two — which has no level-three children of its own and so serves as an area score directly.
  • The asymmetry is the mechanism. Crystallised has two level-three children, Verbal and Quantitative Reasoning, and therefore no area score of its own. Fluid-analytic has exactly one, Abstract/Visual Reasoning, so that area score does double duty. Short-term memory has none, so the level-two factor is the fourth area.
  • The consequence is worth stating plainly: the crystallised-against-fluid contrast is never scored by the test at all. It can only be estimated by collapsing the verbal and quantitative areas.
• The fifteen subtests sit under the four areas: Vocabulary, Comprehension, Absurdities and Verbal Relations; Quantitative, Number Series and Equation Building; Pattern Analysis, Copying, Matrices, and Paper Folding and Cutting; Bead Memory, Memory for Sentences, Memory for Digits and Memory for Objects.
• The evidence did not support the middle level. Confirmatory factor analysis of the standardisation sample — 3,354 cases — found a strong general factor, as the model’s apex predicts, but did not support the second level, the grouping of the first-order factors into crystallised against fluid; the crystallised-to-g loading came out above unity with a negative variance, which is a sign the model is misspecified rather than a finding.
  • The wider literature is genuinely divided, which is the honest way to put it: of about a dozen published factor analyses, roughly four support the four-factor structure and five challenge it, with the rest equivocal. Sattler recovered only two factors at ages two to six and three from seven upward, with quantitative reasoning conspicuously absent, and concluded the area scores should not be used for most interpretive purposes.
  • The structure is age-dependent, and this is the sharpest single point: the test’s own technical manual recovered all four factors only from about age twelve. In the preschool years the factors collapse to two and short-term memory dissolves into reasoning — so the model is weakest exactly where the test is most often used for placement.
  • A caution on the names. The best-known re-analysis is by Robert M. Thorndike, a different psychologist from Robert L. Thorndike, the fourth edition’s lead author. It is not a test author re-analysing his own instrument.
• The model’s parentage is not Carroll. Level two is Cattell and Horn, except that short-term memory is explicitly not theirs — it reflects how psychologists had actually been using earlier Binets. Level three resembles Vernon’s (1950) hierarchy. Carroll’s three-stratum synthesis postdates this edition by seven years and belongs with the fifth.
• Not all fifteen subtests are administered to anyone, and that is by design. The test is adaptive. A routing subtest — Vocabulary — is given first, and the testee’s performance on it, together with chronological age, fixes the entry level for the remaining subtests. Which subtests apply is then determined by age, since several are restricted to particular age bands, and testing within each proceeds to a basal and a ceiling rather than through every item.
  • So the answer is no, for three separate reasons: several subtests are not appropriate to the testee’s age at all; adaptive entry means the easy and hard extremes are skipped as uninformative; and abbreviated batteries are explicitly provided for screening, with the full set reserved for diagnostic work.
  • The cost of abbreviating is that a shorter battery yields a less reliable composite and a thinner profile, so the choice is between testing time and the precision the decision requires.

Methods of measuring intelligence

• By number tested. Individual tests — Stanford-Binet, the Wechsler scales, Bhatia’s battery — permit observation of how the testee works, rapport and clinical inference, and are necessary for diagnosis. Group tests, the tradition beginning with the Army Alpha and Beta of the First World War, trade all that for throughput and are the only practicable option for large-scale screening.
• By content. Verbal tests depend on language and schooling; non-verbal tests use figural material with verbal instructions, as Raven’s matrices do; performance tests use apparatus and manipulation and can be administered with minimal language, which is the reasoning behind Bhatia’s battery.
• By what limits the score. Speed tests use easy items under tight time limits; power tests use items of increasing difficulty with generous time. Most practical tests mix the two, and the mixture determines what the score is sensitive to.
• By cultural loading. Culture-fair attempts — Raven’s, Cattell’s Culture Fair Intelligence Test — strip verbal and acquired-knowledge content. They reduce the loading; none removes it, since familiarity with test-taking, with two-dimensional figural conventions and with the demand to work fast are themselves cultural.
• By theory. Process-based instruments such as the Cognitive Assessment System, built on the PASS model, report a profile of cognitive processes rather than a general score, and are designed to yield a teaching implication rather than a placement.
• Non-test methods remain part of a competent assessment: systematic observation, interview with the person and an informant, rating scales of everyday functioning, and school or work records. A single score from a single session is the weakest form of the evidence, not the whole of it.

Problems in assessing children with special needs

• The test may measure the impairment rather than the ability. A child with a motor impairment is slow on timed manipulative subtests, a child with hearing loss on verbal ones, a child with a visual impairment on figural ones — and the composite then reports the disability, not the intelligence.
• The norms do not apply. Standardisation samples exclude or under-represent children with disabilities, so the reference group is wrong in exactly the cases where the decision matters most.
• Standardised administration and necessary accommodation conflict. Extra time, a scribe, sign interpretation or an enlarged format may be essential for the child to show what they can do, and every one of them departs from the procedure under which the norms were obtained.
• Floor effects. Many scales do not discriminate well at the low end, so children with significant impairment cluster at the bottom of the scale and the test cannot distinguish between them — which is precisely the distinction a placement decision needs.
• Behavioural and attentional factors: limited attention span, fatigue, unfamiliarity with the testing situation, and difficulty establishing rapport all depress performance for reasons unrelated to ability.
• Communication is the deepest problem. Instructions must be understood before performance means anything, and a child who has not understood the demand produces a score that looks like inability.
• The labelling risk is a consequence rather than a technical flaw: a score attached to a child shapes placement, expectation and self-concept, and is far harder to remove than to assign.
• What answers these problems, so far as anything does: non-verbal and performance instruments chosen to bypass the impaired channel; adaptive behaviour scales and informant report as a parallel source of evidence; dynamic assessment, which measures what the child can do with help and so indexes learning potential rather than present attainment; multiple sessions; and a multidisciplinary judgement in which the test score is one input.

Merits and demerits of psychometric measures

• The merits are real. Standardised administration and scoring remove the interviewer’s idiosyncrasy; norms make a score interpretable; reliability and validity are quantified rather than asserted; the error of measurement is stated; and the procedure is economical and reproducible. Against unaided clinical judgement, which is demonstrably prone to bias and overconfidence, a test is usually the more defensible instrument.
• The demerits are equally real. Scores are sensitive to motivation, anxiety, rapport, language and test familiarity. Norms are a property of the sample that produced them and mislead when transplanted. Self-report personality measures face faking and social desirability; ability measures face coaching and practice effects. And the central problem is interpretive rather than technical — a number invites treatment as a fixed property of a person when it is one estimate, with error, on one occasion, against one reference group.
• The contrast between personality and intelligence measurement sharpens both. Ability tests are maximum-performance measures with better and worse answers, so reliability is high and faking is limited to faking bad; personality inventories are typical-behaviour measures with no correct answers, so they are more exposed to response sets and distortion. The fuller treatment of personality measurement belongs with the personality topics.

Criticisms and limitations

• Norms are the weakest component of every test, and the one most often transplanted across populations without re-standardisation.
• The instruments measure a narrow band of ability, selected because it was teachable and testable, and are silent on creativity, practical competence and interpersonal skill.
• Test performance is affected by factors the construct does not include — practice, coaching, schooling, motivation, anxiety, stereotype effects, nutrition and health.
• Standardised administration and the accommodation a testee needs are in tension, and every departure from the procedure weakens the norms it is scored against.
• A single session is the weakest form of the evidence, not the whole of it; observation, informant report and records belong in a competent assessment.
• Measurement has a political history that is part of the subject rather than external to it: the use of test results to justify immigration restriction, sterilisation and educational segregation is the strongest argument for keeping inference close to the evidence.