Item Response Theory: Precursors, Models and the Rasch Debate

Item response theory.

Official Syllabus: Paper I · Unit 3 Research Methods

“Statistical theories of mental test scores.”

— Lord and Novick (1968), the volume in which Birnbaum’s logistic models gave item response theory its working form

Classical test theory describes a test; item response theory describes an item. That shift of unit is the whole of it, and everything that follows — invariant item parameters, ability and difficulty on one scale, precision that varies by ability, adaptive testing — is a consequence of modelling the probability of a particular response to a particular item rather than the properties of a total score.

The precursors

The theory did not arrive at once. Each precursor contributed one component of it.

• Classical test theory, from Spearman (1904), modelled an observed score as a true score plus an error component. It is simple, needs only modest samples, and leaves three defects that IRT exists to repair: every item statistic is sample-dependent, since item difficulty is the proportion passing and that proportion depends on who was tested; every person’s score is test-dependent, since a raw score means nothing without knowing which items produced it; and the standard error of measurement is assumed constant across the whole score range, which is plainly false — a test measures precisely in its middle and poorly at the extremes.
• Thurstone (1925) supplied the founding idea. In scaling test items for mental development, he modelled the probability of success on an item as a function of the respondent’s standing, and placed persons and items on the same quantitative continuum. That single continuum is the conceptual origin of the whole theory.
• Lazarsfeld (1950) formalised latent structure analysis, in which an unobserved variable accounts for the pattern of manifest responses. From it comes the assumption of local independence — that once the latent variable is held constant, responses to different items are unrelated.
• Lord (1952), in his doctoral work, introduced the two-parameter normal ogive model and the statistical theory of the item characteristic curve, turning Thurstone’s idea into an estimable model.
• Rasch (1960), in Probabilistic Models for Some Intelligence and Attainment Tests, derived the one-parameter logistic model from the requirement he called specific objectivity.
• Birnbaum, writing the item-response chapters of Lord and Novick (1968), replaced the normal ogive with the mathematically tractable logistic function and added the discrimination and guessing parameters, which made estimation practical and gave the field its standard models. Lord (1980) then set out its application to real testing problems.

Is it a latent trait theory?

Yes, and the earlier literature called it precisely that. The quantity being estimated, conventionally written theta, is an unobservable attribute inferred from observable responses; the models are mathematical statements of how that latent variable governs the manifest data; and the local independence assumption is the formal expression of the claim that the latent variable accounts for everything the items share.

The name changed for two reasons worth knowing. “Item response theory” emphasises that what the model actually specifies is the response to an item, which is observable, rather than making ontological claims about traits. And “trait” carries an implication of stability that the models do not require — theta may be an attainment, a transient state or an attitude just as well as a trait. So the answer is yes in substance, with a terminological caution.

The core idea and the item characteristic curve

The model states the probability of a keyed response as a function of the person’s level on the latent variable and of properties belonging to the item. Plotted against theta, that function is the item characteristic curve — an S-shaped ogive rising monotonically from a lower asymptote to an upper one.

Difficulty shifts the curve, discrimination steepens it, guessing lifts its floor.

Theta is scaled with a mean of 0 and a standard deviation of 1, and in practice runs from about −3 to +3.

Three assumptions carry the model:

• Unidimensionality — one dominant latent variable accounts for the item responses. This is why a factor analysis establishing dimensionality normally precedes an IRT calibration.
• Local independence — given theta, responses to any two items are statistically independent. Violated by item chaining, shared passages and testlets.
• Monotonicity — the probability of the keyed response rises as theta rises.

The mathematical functions and the models

The original function was the normal ogive, the cumulative normal distribution. It was replaced by the logistic function, which is far easier to work with; the scaling constant D = 1.702 makes the two nearly indistinguishable.

• One-parameter logistic model, equivalent to the Rasch model:

`P(theta) = 1 / (1 + e^-(theta – b))` Only b, the item difficulty, varies across items; it is the point on the theta scale at which a candidate has a 50 per cent chance of success. Discrimination is constrained equal for every item.

• Two-parameter logistic model (2PL):

`P(theta) = 1 / (1 + e^-a(theta – b))` Adds a, the item discrimination, which is the slope of the curve at its steepest point. A steep item separates candidates sharply near its difficulty; a flat item separates them hardly at all.

• Three-parameter logistic model (3PL):

`P(theta) = c + (1 – c) / (1 + e^-a(theta – b))` Adds c, the pseudo-guessing or lower-asymptote parameter — the probability that a candidate of very low ability answers correctly anyway. It is indispensable for multiple-choice aptitude tests, where that probability is never zero.

• A four-parameter model adds an upper asymptote below 1, for items that even very able candidates sometimes miss through carelessness.
• Polytomous models extend the logic to items with more than two response categories: the graded response model of Samejima (1969) for ordered categories; the partial credit model of Masters (1982); the rating scale model of Andrich (1978) for Likert-type scales with a common category structure; and the nominal response model of Bock (1972) for unordered options.

Information, and why precision varies

The item information function gives the amount of information an item contributes at each level of theta. It peaks near the item’s difficulty and falls away on either side — an easy item tells you almost nothing about a very able candidate. The test information function is the sum of the item information functions, and the standard error of measurement at any theta is the reciprocal of the square root of the test information there.

This is the practical heart of the theory. Classical test theory reports one reliability coefficient and one standard error for the whole test; IRT reports precision as a function of ability, so a test builder can say where the test measures well and deliberately place it where the decisions are made.

Item response theory compared with classical test theory

The same data, two accounts of what a score means.

The two ways IRT improves on classical theory are worth stating exactly, because they are the same property seen from each end. Item parameters do not depend on the sample used to estimate them, so an item calibrated on one group has the same difficulty when administered to another. And ability estimates do not depend on the particular items administered, so two candidates who took entirely different item sets can be placed on one scale and compared. Neither is available in classical test theory, and almost every application below follows from them.

The Rasch model, critically evaluated

Rasch (1960) derived his model from a requirement rather than fitting it to data, and that is the source of both its strengths and the objections to it. The requirement, which he called specific objectivity, is that the comparison of two persons should not depend on which items were used, and the comparison of two items should not depend on which persons answered them.

What is strong about it

• Specific objectivity is achieved, not assumed. The model is the only one in the family with the property, and Rasch argued it is a necessary condition for measurement in any science.
• The raw score is a sufficient statistic for theta, so scoring is simple, transparent and defensible to candidates — an administrative virtue in public examinations.
• It yields interval measurement on a logit scale with persons and items on one ruler, so a candidate’s chance of succeeding on any calibrated item can be read directly off the difference between their position and the item’s.
• It needs smaller samples than the 2PL and 3PL models, and parameter estimates are correspondingly more stable.
• It supplies fit statistics — infit and outfit — that identify both misbehaving items and misbehaving persons, which is diagnostic information the other models give less readily.

What is open to objection

• It constrains every item to equal discrimination, which is empirically false for most real item sets. Items differ in how sharply they separate candidates, and the Rasch model has no way to represent that.
• The methodology is prescriptive rather than descriptive. Where data do not fit, Rasch practice discards the misfitting items rather than adopting a model that fits the data. Critics, Goldstein (1979) among them, argued that this makes the model effectively unfalsifiable and lets the model dictate what the construct shall be.
• Deleting misfitting items can narrow the construct. It is the same failure as maximising internal consistency by retaining near-duplicate items: the instrument becomes more conformable to the model and less representative of the domain.
• There is no guessing parameter, so the model misfits multiple-choice aptitude items, where low-ability candidates succeed by chance at a rate that cannot be ignored.
• The claim that specific objectivity is a requirement of measurement is a philosophical position, not an empirical finding, and the long-running dispute between the Rasch school and the users of 2PL and 3PL models is at bottom a disagreement about whether a model should be chosen for its properties or for its fit.

Applications, and what they give aptitude testing

• Computerised adaptive testing is the largest practical gain. Each item is chosen to be maximally informative at the candidate’s current ability estimate, so a very able candidate is never detained by easy items and a weak one is not defeated by hard ones. The same precision is reached with substantially fewer items, which cuts testing time and fatigue, improves security because no two candidates see the same sequence, and keeps the test appropriately challenging throughout.
• Item banking and test equating. Items calibrated on a common scale allow different forms, administered in different sessions or years, to produce directly comparable scores without the laborious construction of parallel forms. This matters for any aptitude test given to very large numbers in multiple shifts.
• Targeted precision at the decision point. Where a selection cut-off governs the outcome, items can be selected to maximise information at the cut-off, so classification is most accurate exactly where errors are most costly. Classical construction, which aims at overall reliability, cannot do this.
• Differential item functioning. An item can be tested for whether it behaves differently for two groups of equal ability — by language, sex or region. This is the technical basis of fairness analysis, and it separates genuine group differences on the trait from artefacts of item wording.
• Better item analysis. The discrimination parameter is on an interpretable scale and does not depend on the ability spread of the tryout sample, so item quality can be compared across tryouts that classical indices would make incomparable.
• Pattern scoring. Under the 2PL and 3PL models two candidates with identical raw scores but different response patterns receive different ability estimates, because the items they answered correctly carry different information.
• Scale construction beyond aptitude. The polytomous models have made IRT standard for health and quality-of-life instruments, where short adaptive forms now replace long fixed questionnaires.

Criticisms and limitations

• The assumptions are strong and frequently violated. Strict unidimensionality is rare in psychological data, local independence fails wherever items share a passage or a context, and the consequences of violation for parameter estimates are not always obvious from the output.
• Large samples are required, and increasingly so with each added parameter. The 3PL model needs sample sizes beyond the reach of most research projects, which confines proper IRT work to large testing organisations and leaves ordinary scale development with classical methods.
• Technical opacity carries a political cost. A candidate can understand “you answered 60 of 100 correctly”; a theta estimate derived from a pattern of responses to items nobody else saw is harder to accept as fair, and adaptive testing’s defence rests on a model the examinee cannot check.
• Invariance holds only if the model is correct. The celebrated sample-independence of item parameters is a property of the model, not a finding about data, and it fails to the extent that the model misfits.
• Model selection is itself contested, as the Rasch dispute shows, and the choice between a model chosen for its measurement properties and one chosen for its fit has no technical resolution.
• Indian examinations face exactly the problem IRT was built for. Large aptitude examinations administered across multiple shifts with different question papers must make scores comparable, and the normalisation procedures in general use are percentile-based rather than model-based. Percentile normalisation assumes the candidate groups in different shifts are equivalent in ability; IRT-based equating does not need that assumption, because it places items on a common scale through linking items. The gap between what is available and what is used is one of the clearer practical arguments for the theory.