Via Amy Perfors at the Harvard statistics blog, Social Science Statistics Blog, I learned of the Jeffrey-Lindley Paradox in statistics. The paradox is that if you have a sample large enough, you can get p-values that are very close to zero, even though the null hypothesis is true. You can read a very in depth explanation of the paradox here.
I don’t find this either surprising or worrisome, as Perfors does. While I’d never heard of the paradox before (it’s really pretty cool, if you’re into statistics or Bayesian reasoning), everyone who’s taken a statistics course understands the perils of large sample-sizes. The fact is, if you have two different groups, even from the same population, they are, by definition, two different groups, as they are composed of two different sets of individuals. As a result, where measures that are influenced by random variables are concerned, the means of the groups will be different, and if you get a large enough sample size, that difference will be statistically significant. Since everyone is aware of this, I can’t imagine it’s a problem. If it looks like someone’s using a sample that’s too large, so that any significant differences he or she might find are likely to be theoretically and practically uninteresting, people will pick up on it, either through effect size calculations or through subsequent research.