I recently learned of the famous Lady Tasting Tea hypothesis test experiment. When I read of Rod Sturdivant's account of the experiment, it struck me that the experiment was not designed to maximize the amount of learned information for a fixed number of cups.

The designer of the experiment, R. A. Fisher, was trying to determine whether a lady could tell whether milk was poured before or after tea was poured into a cup. To test, he presented her with four cups poured milk-first and four cups poured tea-first in random order. This results in 70 possibilities and a 1.4%-chance of correctly determining the order given no ability. A single mistake on the part of the lady would make it impossible to reject the no-ability hypothesis with reasonable (p=0.05) confidence.

If Fisher had instead flipped 8 coins, filling milk first if head and tea first if tails, then there would have been 256 possibilities with a 0.4%-chance (`1/256`) of guessing all 8 correct given no ability and a 3.9%-chance (`9/256`) of guessing 7-or-8 correct with no ability. So, with this method, a single mistake would have still allowed one to reject the no-ability hypothesis.

Of course, the coin-flip test is technically more difficult because of the need to make absolute determinations, rather than merely determining an ordering of the cups, as is the case in the permutation test.