“Write short note : Reliability of a sample.” (1998)
- “Reliability” here means something distinct from instrument reliability — the consistency of a measuring tool discussed elsewhere in research methodology — and this distinction must be made explicit before anything else.
- A sample’s reliability is a different concept: it concerns how consistently and dependably the sample itself represents the population it was drawn from, not whether a questionnaire or scale gives repeatable readings.
- The formal vocabulary for this is sampling error — the unavoidable margin of error that exists precisely because a sample, not the full population, was measured — and the standard error of the sampling distribution, which shrinks as sample size grows.
- The systematic use of sampling to represent a larger population in social research has a genuine methodological history running back well over a century, from early social-survey pioneers to today’s large official statistical systems.
Sampling Error and the Standard Error
- Sampling error is not a mistake in the ordinary sense — it arises simply because any single sample, however carefully drawn, is one particular subset out of many that could have been selected, and each possible subset would give a slightly different estimate of the population value.
- The standard error measures how much sample estimates would typically vary across repeated samples of the same size and design; a smaller standard error means the sample is a more dependable — more reliable — stand-in for the population.
- Following the law of large numbers, the standard error shrinks as sample size increases: a larger, properly drawn probability sample gives an estimate that clusters more tightly around the true population value, which is why sample size is one genuine lever a researcher has over a sample’s reliability.
Sampling Error Versus Non-Sampling Error
- Non-sampling error is an entirely different category of problem — it arises from a poorly constructed sampling frame, non-response, interviewer mistakes, leading questions, or coding errors, not from the mere fact of sampling rather than enumerating.
- The critical distinction: non-sampling error does not shrink as the sample grows larger — a badly designed instrument or a systematically biased frame produces the same distortion whether it is applied to five hundred respondents or five hundred thousand.
- This yields a sharp, easily stated point: a huge, badly-drawn sample is not more reliable for being huge — it is simply confidently wrong, since its large size shrinks only the sampling error while leaving whatever non-sampling error is baked into its design fully intact.
- A sample’s genuine reliability therefore depends on getting both halves right — an adequate size to control sampling error, and a sound design, frame, and fieldwork to control non-sampling error — and no amount of the first can compensate for a failure of the second.
Only Probability Sampling Licenses a Calculable Reliability
- A margin of error and a confidence interval can only be calculated where the sample was drawn through random, probability-based selection — because only then is the chance of any unit’s inclusion known and quantifiable.
- Non-probability samples — purposive, quota, convenience, or snowball samples — have no such calculable reliability at all: there is no statistical basis for stating how far the sample estimate is likely to be from the true population value, however large or carefully chosen the sample is.
- This is a distinction professional survey practice continues to emphasise: contemporary discussions of polling accuracy increasingly stress a total survey error framework that treats sampling error as only one component alongside coverage, non-response, and measurement error — a reminder that a calculable margin of error, however precise-looking, describes only part of what makes a sample’s results dependable.
- A sample’s reliability is fundamentally about the stability and dependability of its representation of the population, not about the consistency of any single measuring instrument used within it.
- Larger probability samples reduce sampling error in a mathematically predictable way, but this improvement is entirely separate from, and cannot fix, non-sampling error rooted in poor design or execution.
- The discipline’s oldest and still most practically important lesson in this area is that sample size and sample reliability are not the same thing, and mistaking one for the other remains a common source of misplaced confidence in survey findings today.