The Perils of Randomization Checks in the Analysis of Experiments

In the analysis of experimental data, randomization checks, also known as balance tests, are used to indicate whether a randomization has produced balance on various characteristics across experimental conditions. Randomization checks are popular in many fields although their merits have yet to be established. The grounds on which balance tests are generally justified include either 1) the credibility of experimental findings, and/or 2) the efficiency of the statistical model. We show that balance tests cannot improve either credibility or efficiency. The most common “remedy” resulting from a failed balance test is the inclusion as a covariate of a variable failing the test; this practice cannot improve the choice of statistical model. Other commonly suggested responses to failed balance tests such as post-stratification or re-randomization also fail to improve on methods that do not require balance tests. We advocate resisting reviewer requests for randomization checks in all but some narrowly defined circumstances