Quasi-Experimental and Non-Experimental Designs: Validity, Artifacts and Causal Inference

Experimental, non-experimental and quasi-experimental designs.

Official Syllabus: Paper I · Unit 2 Methods of Psychology

“Quasi-experiments are experiments that have treatments, outcome measures and experimental units, but do not use random assignment to create the comparisons from which treatment-caused change is inferred.”

— Cook and Campbell (1979)

Most of what psychology wants to know about cannot be randomised. Schools, hospitals, prisons, factories and whole communities are not allocated by lottery, and the variables that matter most — poverty, caste, bereavement, disaster exposure, brain injury — are given rather than manipulable. The designs below are the methodology of that situation: they recover by argument and measurement what random assignment would otherwise supply automatically, and the discipline of the field consists in knowing exactly how much is lost.

Internal and external validity

Campbell and Stanley (1963) drew the distinction that organises the whole field.

• Internal validity — whether the treatment, and not something else, produced the observed difference. It is a question about this study.
• External validity — whether the finding holds for other people, settings, times and operations. It is a question about everything else. Ecological validity, whether conditions resemble those of ordinary life, is the part most often at stake in psychology.

Cook and Campbell (1979) added two further kinds, and the addition matters because it covers errors a perfectly executed experiment can still make. Construct validity asks whether the operations used actually embody the concepts claimed — whether a questionnaire score is anxiety or merely a willingness to report it. Statistical conclusion validity asks whether the inference drawn from the numbers is sound, covering low power, violated assumptions and the inflation of error rates by repeated testing.

Threats to the validity of experimental research

Each threat below is a rival explanation that a design either blocks or does not.

• History — an event outside the study occurring between measurements, as when a stress-reduction programme is interrupted by examinations.
• Maturation — change within participants from growth, fatigue or the passage of time alone. This is fatal to one-group developmental designs, where children would have improved anyway.
• Testing — the pre-test itself alters later performance, through practice or by alerting participants to the topic.
• Instrumentation — the measuring instrument changes, as when observers become more skilled or more lenient across the study, or a rating scale is revised partway.
• Statistical regression — participants selected because they were extreme move towards the mean on re-measurement. Remedial groups chosen by low scores will therefore appear to improve whatever is done to them.
• Selection — the groups compared differed before the treatment. This is the threat random assignment exists to remove, and the reason the principle is used: allocation by chance does not make groups equal, but it makes the expected difference between them zero and leaves the remaining accidental difference estimable by the error term. Without it, any group difference at post-test is confounded with whatever difference existed at the start.
• Mortality, or attrition — participants drop out differentially, so the groups at post-test are no longer the groups that were assigned. A weight-loss programme loses those for whom it is not working.
• Selection–maturation interaction — the groups were not merely different but were changing at different rates, which no single pre-test can detect.

A design is judged by which of these eight rivals it leaves standing.

The threats to external validity are a shorter list and are routinely neglected: interaction of selection with treatment, where the effect holds only for the kind of people studied; interaction with setting, where it holds only in laboratories; and interaction with history, where it holds only at that moment.

A single-group pretest-posttest design admits history, maturation, testing, instrumentation and statistical regression simultaneously, which is why an apparent gain in such a study supports no causal inference at all. Every quasi-experimental design below is an attempt to close one or more of those doors without the use of randomisation.

Artifacts in the experiment: experimenter effects and demand characteristics

An experiment is a social situation as well as a measuring operation, and the social part generates systematic error. Rosenzweig (1933) raised the point early; two lines of work established it.

• Orne (1962) named demand characteristics — the totality of cues by which a participant works out what the study is about and what is expected. Participants adopt the good-subject role and try to confirm the hypothesis they have inferred. Orne’s demonstration was that participants would perform pointless, tedious tasks for hours without protest once the frame was “an experiment”, showing how far the role itself shapes behaviour.
• Rosenthal and Fode (1963) demonstrated experimenter expectancy: student experimenters told their rats were “maze-bright” or “maze-dull” obtained correspondingly better and worse learning from animals in fact assigned at random. Rosenthal and Jacobson (1968) carried the logic into classrooms, where teachers given false information about which pupils would “bloom” produced measurable gains in those pupils.
• Evaluation apprehension — Rosenberg (1965) showed that participants are concerned to appear competent and well-adjusted, which biases responses independently of any hypothesis they have guessed.
• The Hawthorne effect — the generic tendency of people to change behaviour because they are being observed, named from the Western Electric studies of the late 1920s.

The procedural answers form a graded set, and the reason for each is what distinguishes them:

• Single-blind procedure — the participant does not know which condition they are in. This removes demand characteristics and participant expectancy, and nothing else.
• Double-blind procedure — neither the participant nor the person administering the treatment and recording the outcome knows the assignment. This is the only arrangement that also removes experimenter expectancy, because unconscious differential treatment and biased recording cannot track a condition the experimenter is unable to identify. It is what makes the two bias sources independently controllable rather than confounded.
• Placebo control — an inert condition identical in every respect except the active ingredient, separating a treatment’s specific effect from the effect of receiving a treatment.
• Supporting measures: standardised or recorded instructions, automated administration, an experimenter blind to the hypothesis rather than merely to assignment, deception where justified, and post-experimental enquiry to establish what participants actually inferred.

Only double-blinding reaches the bias that originates with the experimenter.

The limits are real. Double-blinding is impossible where the treatment is obvious — psychotherapy, exercise, meditation, surgery — and participants in drug trials often break the blind by noticing side-effects. Blinding also does nothing against selection, maturation or attrition; it addresses one family of artifact only.

Quasi-experimental designs

A quasi-experimental design manipulates a treatment and compares conditions, but assignment is not random. The prefix means “resembling”: these are experiments missing exactly one of the three defining features.

No row begins with R. The broken series and the staggered start do randomisation’s work.

• Nonequivalent comparison group design — an intact existing group receives the treatment and another intact group serves as comparison, both measured before and after. A school cannot randomly assign children to an innovative classroom, and even if it did they would mix at lunch and recess. The design’s quality rests entirely on the choice of comparison group and on how much is measured about it: socioeconomic status, age, sex, prior attainment, teacher characteristics, school size and curriculum are documented either to establish similarity or to serve as statistical controls. The honest aim is not to prove equivalence but to weaken every plausible rival explanation.
• Interrupted time-series design — repeated measurement of a single unit, with the treatment introduced partway so it “interrupts” the series. The baseline serves as its own control and the inference rests on a change in level or slope at the point of interruption. Practical where no comparison group exists at all, and vulnerable to history and to cyclical fluctuation: counselling referrals rise as examinations approach whatever a service does, so a programme launched in October will appear to work. A long enough baseline is what makes such cycles visible rather than invisible, and effects that are gradual or delayed will be missed by a short series.
• Multiple-baseline design — the same innovation introduced at a different time in each of several units. An outside event affecting everyone simultaneously cannot produce changes that follow the staggered start dates, so history is ruled out by the design itself, and replication across units supplies some external validity. It combines the strengths of the time-series and nonequivalent-group designs, and the units compared remain non-equivalent.
• Regression-discontinuity design — treatment assigned by whether a case falls above or below a cut-off on a continuous measure, as scholarships go to those above a mark. Thistlethwaite and Campbell (1960) showed that a discontinuity in the outcome exactly at the cut-off is hard to explain by anything but the treatment, since cases just either side are alike in everything else. Where the cut-off is strictly enforced the design approaches an experiment in interpretability.
• Ex post facto design — groups identified by a difference that has already occurred, such as brain injury, bereavement or deprivation, and compared on outcomes. There is no manipulation and the direction of causation stays open, which places it at the boundary with non-experimental work.

Whether quasi-experimental designs are more advantageous than experiments depends on which sense of advantage is meant, and the two answers point in opposite directions. On internal validity they are strictly inferior, since every one of them leaves selection open. On feasibility, ethics and external validity they are frequently superior and sometimes the only option: the units cannot be randomised, withholding an apparently beneficial programme by lottery raises genuine objections, and the samples are real populations rather than volunteers. The defensible position is that these are not weaker experiments but a different instrument — they answer “did this programme, as actually delivered here, change anything?” where the experiment answers “can this variable, under control, produce this effect?”

Non-experimental designs

A non-experimental design neither manipulates the independent variable nor assigns participants to conditions. It observes what already varies. The family includes correlational studies, surveys, case studies, naturalistic observation and archival analysis.

These are not simply deficient experiments. Where the variable of interest cannot be manipulated, a non-experimental design is not second best but the only honest option. They typically carry higher external validity, since nothing has been arranged, and they are the natural choice at the exploratory stage, before there is a hypothesis worth testing. Emerging social problems — caste discrimination, dowry, child labour, communal violence, farmer distress, substance use — are of exactly this kind: the determining variables are historical and structural, the affected populations cannot be randomly constituted, and an intervention study presupposes knowing which variables to intervene on. The non-experimental study is what establishes that.

Their characteristic limitation is causal. A correlation between two measures is consistent with three accounts: the first causes the second, the second causes the first, or a third variable causes both.

Each device below attacks one of the three, and none of them attacks all three.

The devices for narrowing this are what a serious treatment of moving from correlational to causal inference consists of:

• Establishing temporal precedence by longitudinal measurement, since a cause cannot follow its effect. The cross-lagged panel design measures both variables at two times and compares the two diagonal correlations.
• Partial correlation and multiple regression, which remove the contribution of measured third variables — limited by the fact that only measured confounds can be removed.
• Structural equation modelling, which tests whether a specified pattern of causal paths reproduces the observed covariances. It can reject a model; it cannot establish that a fitting model is the true one, since other models fit the same data equally well.
• Propensity-score matching, which pairs treated and untreated cases on their estimated probability of having been treated.
• Instrumental variables and natural experiments, which exploit variation in the presumed cause unrelated to the outcome except through it.
• Converging evidence across designs, which is how causal claims in psychology are in practice secured. Smoking and lung cancer were never established by a randomised trial in humans; dose-response gradients, animal experiments, mechanism and consistency across populations did it instead.

Indian context

• Banerjee, Cole, Duflo and Linden (2007) ran two randomised field experiments in urban Indian schools. A remedial education programme placed a balsakhi, a young woman recruited from the community, with the weakest children in municipal schools in Vadodara and Mumbai, and schools were randomly assigned to receive it. Average test scores in treated schools rose by about 0.28 standard deviations; the gain was concentrated in the children actually sent to the balsakhi and at the bottom of the distribution, and faded substantially within a year of the programme ending. The study shows both what randomisation in an Indian field setting can deliver and why follow-up measurement matters — a design that stopped at the immediate post-test would have reported a larger and more durable effect than the one that exists.
• The Indian research programme on deprivation rests largely on non-experimental designs, for the structural reason set out above: poverty cannot be assigned. Sinha, Tripathi and Misra (1982) gathered this work in a volume on deprivation and its psychological consequences, and Misra (1976) reported cognitive processes as a function of prolonged deprivation. The strategy was to construct graded indices of deprivation and relate them to cognitive and motivational measures across naturally differing groups, with an ecological framework specifying which features of the environment carried the effect. Durganand Sinha‘s wider argument was that method had to follow the context rather than be imported with the theory, which is the methodological core of the case for non-experimental work on Indian social problems.

Criticisms and limitations

• Quasi-experimental inference is argumentative, not procedural. Its validity depends on the investigator having anticipated and measured the right rival explanations. There is no equivalent of random assignment to protect against the confound nobody thought of, so a quasi-experimental result is only as good as the imagination of the person who designed it.
• Statistical adjustment cannot repair non-equivalence. Covariates remove only what was measured, measured reliably and modelled correctly; an unmeasured difference between intact groups survives every adjustment, and adjusting on an unreliable pre-test reintroduces regression to the mean.
• The two validities trade against each other. They cannot be jointly maximised, so a design is always an argument about which threat matters most for this question, and treating internal validity as simply better is a misreading of the trade rather than a strict standard.
• Control itself can become an artifact. The apparatus of control and bias-reduction can produce a behaviour that exists only under those conditions. Demand characteristics are not noise to be cleaned away but part of what the participant is responding to, and a finding obtained only under blinded laboratory conditions may not describe anything outside them.
• Correlational designs are routinely over-read in practice. The formal caution is universally taught and almost as universally abandoned at the point of discussion, where “associated with” becomes “leads to”. The remedy is design, not phrasing: without temporal ordering or an exogenous source of variation, no amount of statistical sophistication converts covariation into causation.