Experimental Designs and the Control of Variance: Factorial, Within and Between Groups

Experimental, non-experimental and quasi-experimental designs.

Official Syllabus: Paper I · Unit 2 Methods of Psychology

“Research design is the plan, structure and strategy of investigation so conceived as to obtain answers to research questions and to control variance.”

— Kerlinger (1973)

A design is not the same thing as a method. The method says how observations will be gathered; the design says how they will be arranged so that one explanation survives and the rivals do not. Kerlinger’s phrase “control of variance” is the key: scores vary, and a design is an arrangement that assigns as much of that variation as possible to the variable under study and the rest to identifiable, bounded sources.

Variables and the control of variance

• Independent variable — the condition the investigator manipulates or selects, whose effect is in question. It has at least two levels, one of which is usually a control level.
• Dependent variable — the measured outcome, expected to change as the independent variable changes.
• Mediating variable — the intervening process through which the independent variable produces its effect. Baron and Kenny (1986) set the standard test: the path from cause to outcome must shrink once the mediator is entered. Teaching method raises achievement through time-on-task; time-on-task is the mediator.
• Moderating variable — a variable that changes the size or direction of an effect rather than carrying it. The same incentive raises performance in novices and depresses it in experts; expertise moderates.
• Extraneous variable — anything other than the independent variable that could affect the outcome.
• Confounding variable — an extraneous variable that varies systematically with the independent variable, so that its effect and the treatment’s cannot be separated. Ice-cream sales correlate with drowning deaths; summer heat is confounded with both.

Confounding enters a study in three distinguishable ways:

• Operational confounding — the measure intended to capture one construct also captures another, so the operation itself is impure. This occurs in experimental and non-experimental work alike.
• Procedural confounding — something in the running of the study, such as the room, the time of day or the order of tasks, differs along with the treatment.
• Person confounding — the groups compared differ in who is in them, which is the characteristic failure of designs without random assignment.

Techniques of control

• Elimination — remove the extraneous variable altogether, as sound-proofing removes noise.
• Constancy of conditions — hold it at one unvarying value for every participant where it cannot be removed.
• Randomisation — assign participants to conditions by chance, so that nuisance variables, including those nobody has thought of, distribute themselves equally across groups. Fisher (1935) made this the foundation of experimental method, and it remains the only control technique that works against unknown confounds.
• Matching — pair participants on a known nuisance variable, then split each pair across conditions.
• Counterbalancing — vary the order of conditions systematically so that practice and fatigue fall equally on all treatments.
• Statistical control — remove the nuisance variable’s contribution arithmetically, as analysis of covariance does.

Only randomisation reaches confounds that were never thought of.

Randomisation is the strongest of these and the most frequently misread. It does not make groups equal; it makes the expected difference between them zero and lets the size of the remaining accidental difference be estimated by the error term. That is why a significance test means something after random assignment and much less without it. The limitation follows directly: in any single study an allocation may be badly unbalanced, and with small samples this is likely rather than rare.

Experimental designs

An experiment, in the strict sense, requires three things together — manipulation of the independent variable by the investigator; a comparison condition against which the treated condition is read; and random assignment of participants to conditions. A study with the first two but not the third is quasi-experimental; a study with none of them is non-experimental.

The standard notation, which compresses any design into one line, uses R for random assignment, X for the treatment and O for an observation, read left to right in time.

Only the lines beginning R support a causal claim. Every row above them has a rival explanation built in.

Pre-experimental designs

• One-shot case study — one group, treatment, one measurement. There is no comparison of any kind, so any outcome is equally consistent with the treatment having done nothing.
• One-group pretest-posttest design — the same group measured before and after. This design admits almost every threat at once: the group may have changed on its own, the first test may have taught them something, outside events may have intervened, and extreme scorers drift towards the mean whatever is done to them. A gain from pre-test to post-test is therefore uninterpretable on its own.
• Static-group comparison — a treated and an untreated group, both measured only afterwards. The rival explanation is that the groups differed beforehand and nobody knows.

These are cheap and sometimes all that circumstances allow, but they are classified as pre-experimental rather than weak-experimental deliberately: the failure is structural, not a matter of degree.

True experimental designs

• Posttest-only control group design — random assignment, treatment to one group, both measured once. The cleanest design available, and preferable where a pre-test would sensitise participants.
• Pretest-posttest control group design — adds a pre-test to both arms, which lets change be measured within each participant and lets attrition be examined.
• Solomon four-group design — Solomon (1949) combined the two, adding an unpretested treatment group and an unpretested control. It is the only arrangement that lets the effect of the pre-test itself, and its interaction with the treatment, be estimated rather than assumed away.

A design for a practical evaluation question is assembled from these parts. To test whether a teaching method or a compensatory programme works, the unit of assignment is the class or school rather than the individual child, since children in one room cannot be insulated from each other; assignment is by lottery among comparable units; outcome is a common test administered by someone blind to condition; and measurement continues after the programme ends, because gains that vanish within a year are a different finding from gains that persist.

Banerjee, Cole, Duflo and Linden (2007) ran exactly this design in Indian municipal schools. A remedial programme placed a balsakhi, a young woman recruited locally, with the weakest children, and schools in Vadodara and Mumbai were randomly assigned to receive it. Average test scores in treated schools rose by about 0.28 standard deviations, with the gain concentrated in the children actually sent to the balsakhi and at the bottom of the distribution, and it faded substantially within a year of the programme ending. The last clause is the methodological point: a design that stopped at the immediate post-test would have reported a larger and more durable effect than the one that exists.

Between-groups, within-subjects and matched designs

• Between-groups design — each participant experiences one condition only, and conditions are compared across different people. One group of children learns from a lecture and a different group from discussion, and the two mean scores are compared. Order and practice effects cannot occur, but individual differences fall into the error term, so more participants are needed for the same sensitivity.
• Within-subjects design, also called a repeated-measures design — every participant experiences every condition and serves as their own control. The same children are tested for reaction time under quiet, moderate and loud noise, each person contributing three scores. Individual differences are removed from the comparison, making the design far more powerful with fewer participants. The price is order effects: practice, fatigue, boredom and carry-over from one condition to the next. Counterbalancing is the standard answer, and a Latin square does it economically by arranging conditions so each appears once in every ordinal position.
• Matched-groups design — participants are paired on a variable known to affect the outcome, usually a pre-test, and one of each pair goes to each condition. This looks like the best of both and carries a specific defect: matching on a fallible measure guarantees regression to the mean, so matched extreme scorers move apart on re-testing for purely statistical reasons. Matching also fails silently when the matching variable is the wrong one, and it discards participants who cannot be paired.

Each arrangement removes one problem and inherits another.

The same contrast reappears inside factorial designs. A between-groups factorial assigns each participant to one cell, so every factor is manipulated between people; a within-subjects factorial puts every participant through every cell; and a mixed design manipulates one factor between groups and another within subjects, which is the commonest arrangement in practice because treatment usually cannot be given to the same person twice while time of testing always can.

Factorial designs

A factorial design manipulates two or more independent variables, called factors, at once, so that every combination of their levels is represented. Two factors of two levels each make a 2 × 2 factorial with four cells; the count of numbers gives the number of factors, and each number gives that factor’s levels.

Rows give one main effect, columns the other; unequal row differences across columns are the interaction.

The design yields a main effect for each factor and, more importantly, an interaction — the finding that one factor’s effect depends on the level of the other. Interaction is why factorial designs displaced the practice of running one variable at a time: a method that helps anxious learners and hinders calm ones has no single answer, and a one-variable study would report whichever average its sample happened to produce.

• Simple factorial design — two factors. Complex factorial design — three or more; a 2 × 2 × 2 has eight cells, and the gain is the three-way interaction, at the cost of cells too thin to interpret.
• Completely randomised design — participants assigned to treatments entirely at random, resting on replication and randomisation alone.
• Randomised block design — participants first grouped into homogeneous blocks on a nuisance variable, then randomised within each block. This is the principle of local control: the nuisance variable’s variance is removed from error rather than left in it.
• Latin square design — controls two nuisance variables at once by arranging treatments so each appears exactly once in every row and every column.

Laboratory and field experiments

• Laboratory experiment — conducted in a setting arranged by the investigator, with precise measurement and strong control of extraneous conditions. This is why the experiment is described as observation under controlled conditions: what the investigator does is still observe behaviour, but in a situation where everything except the independent variable has been held constant, removed or randomised, so that what is observed can be attributed. Its salient features are a manipulated independent variable, a comparison condition, random assignment, standardised instructions and apparatus, and the deliberate holding constant of everything else. Internal validity — whether the treatment and not something else produced the difference — is high.
• Field experiment — manipulation and random assignment carried out in a natural setting such as a school, factory or clinic. External validity — whether the finding holds for other people, settings and times — is far higher, and control correspondingly weaker: participants cannot be insulated from each other or from events, attrition is larger, and the investigator often cannot choose the unit of assignment.

The two validities cannot be jointly maximised, and that is the substance of the comparison rather than a caveat on it. Tight control is bought with artificial conditions, atypical samples and tasks nobody performs outside the room; realism is bought with contamination. The choice follows the question: a theoretical claim about a mechanism needs internal validity first, an evaluation of a programme in use needs external validity first.

Criticisms and limitations

• The experiment’s central requirement is often unavailable. Age, sex, language, caste, bereavement, disaster exposure and lesion site cannot be manipulated, and manipulating several of them would be unethical if it were possible. For a large part of psychology’s subject matter the experimental design is simply not on offer, which is why the non-experimental and quasi-experimental families exist rather than being tolerated.
• Statistical control is weaker than it appears. Analysis of covariance removes only confounds that were measured, measured well and specified correctly. An unmeasured confound is untouched, and adjusting for a variable that lies on the causal path removes part of the effect itself.
• Matching is the most over-trusted technique in the list. It intuitively seems stronger than randomisation because the groups visibly resemble each other, while in fact it controls one named variable and guarantees a regression artifact in exchange.
• Factorial designs invite over-specification. Each added factor multiplies cells, so a three- or four-factor design either demands an unreachable sample or yields higher-order interactions resting on a handful of participants each — significant findings that do not replicate.
• Significance testing imports problems of its own. Low power makes real effects invisible, multiple comparisons manufacture false ones, and a significant result says nothing about effect size. The replication difficulties of the last decade are largely failures of this kind rather than of design logic.
• The laboratory’s samples are narrow. Much of the evidence base rests on undergraduate volunteers from industrialised Western populations, which makes external validity a standing rather than occasional worry and makes replication in other cultural settings a substantive scientific question.