Research Methodology
Published August 6, 2026
P-Values and Effect Size: Why Statistical Significance ≠ Practical Importance
When reading preclinical research papers, you will encounter p-values and claims of "statistical significance." However, many researchers misunderstand what these numbers actually mean. A small p-value does not measure the size of an effect or its practical importance — it only measures the probability of observing the data if the null hypothesis were true. This article clarifies the distinction and explains why effect size matters more.
What is a p-value?
The p-value is the probability of observing a result as extreme as (or more extreme than) the one obtained in the experiment, assuming the null hypothesis is true (i.e., assuming there is no real effect).
A p-value of 0.05 (the conventional significance threshold) means: if the null hypothesis were true, there is a 5% probability of seeing data like yours by random chance alone.
Critical misconception: A p-value is NOT the probability that your result is "true" or "real." It does not tell you the magnitude of the effect or whether it matters.
The sample size problem: Why large studies find tiny differences
P-values are deeply dependent on sample size. This creates a paradox:
- Small sample size: Even a large, real effect might not reach p < 0.05 if your group sizes are small (n=3 or n=5).
- Large sample size: Even a trivial, meaningless difference can reach p < 0.05 if n is large enough (e.g., n=1,000).
- Example: A 0.5% improvement in a biomarker with n=50,000 might yield p < 0.001, but that 0.5% may be clinically or biologically irrelevant.
This is why many large-scale biomedical studies report "statistically significant" findings that, in practice, are too small to matter.
What is effect size, and why does it matter?
Effect size quantifies the magnitude of a difference or relationship, independent of sample size. Common effect size measures include:
- Cohen's d: For comparing two group means. d = 0.2 (small), 0.5 (medium), 0.8 (large).
- Correlation coefficient (r): Ranges from −1 to +1; r = 0.1 (small), 0.3 (medium), 0.5 (large).
- Percent change or fold-change: Often reported in biology (e.g., "treatment increased response 2.5-fold").
Effect size allows you to assess whether a statistically significant finding is also practically meaningful. A study might show p = 0.002 (highly significant) but Cohen's d = 0.15 (negligibly small effect).
Practical significance vs. statistical significance
Three categories emerge:
- Statistically significant + large effect: A real, meaningful finding worth investigating further.
- Statistically significant + small effect: A "true" difference, but likely too small to matter in practice. Caution: large sample sizes can produce these.
- Not statistically significant but large effect estimate: Possibly underpowered (too small a sample). More research may reveal a real effect.
Statistical power and sample size
Statistical power is the probability of detecting a real effect if it exists. Power depends on:
- Sample size: Larger samples increase power.
- Effect size: Larger effects are easier to detect.
- Significance level (α): Typically 0.05.
The field recommendation is minimum 80% power (often written as "β = 0.2"). Studies with power < 80% have a high risk of false negatives (missing a real effect) and inflated effect sizes (type M error).
Critical reading of research papers
When you read a peptide research paper, ask:
- Is effect size reported? (If only p-values are shown, be skeptical.)
- What is the sample size (n)? Small n (n < 5) suggests high variability and weak conclusions.
- Is the effect size meaningful? A 2% increase might be statistically significant but biologically negligible.
- Were multiple comparisons made without correction? (e.g., 20 tests at α=0.05 expect 1 false positive by chance).
Research use only. This article is educational information about research methodology. No therapeutic claims are made about any product.
Sources & further reading
Universe Peptide publishes research-focused education for the scientific community. Products are for laboratory research only. See more in our News & research updates.