Fisher’s exact test calculator

What is Fisher’s exact test?

A test of whether two yes/no variables are related, using a 2×2 table of counts. With the row and column totals fixed, the count in the top-left cell follows the hypergeometric distribution, so the p-value is exact — no large-sample approximation, which makes it the test to use when counts are small.

Example: the lady tasting tea

Fisher’s original example (The Design of Experiments, 1935): 8 cups, 4 with milk poured first. There are C(8, 4) = 70 ways to pick 4 cups, so naming all 4 correctly by pure guessing has probability 1/70 ≈ 0.0143 — the one-sided p-value. With 3 of 4 correct, the one-sided p is 17/70 ≈ 0.2429, as R’s documentation shows.

How is the two-sided p-value calculated?

By adding the probabilities of every possible table with the same totals that is no more likely than the one observed — the method R’s fisher.test and SciPy use. For the table 1, 9 / 11, 3 the observed table has probability 0.001346, the one-sided p is 0.001380 and the two-sided p is 0.002759.

Which one-sided p-value should I use?

P(a ≤ observed) tests whether the odds ratio is below 1 (the first row has fewer “yes” outcomes than expected); P(a ≥ observed) tests whether it is above 1. Choose the direction before you look at the data; otherwise use the two-sided p.

Which odds ratio is shown?

Two: the sample odds ratio ad ÷ bc, and the conditional maximum-likelihood estimate that R’s fisher.test reports. For the table 8, 2 / 1, 5 the sample odds ratio is 20 and the two-sided p is 0.035, matching SciPy’s documentation.

Where do the formulas come from?

The test and the tea-tasting example are from R. A. Fisher’s The Design of Experiments (1935). The two-sided rule and the conditional odds ratio follow the documentation of R’s fisher.test; the worked examples above are from that documentation, Wikipedia’s “Fisher’s exact test” article and SciPy’s fisher_exact documentation.

When should I use Fisher’s test instead of chi-square?

When the counts are small. The chi-square test of independence relies on an approximation, and textbooks such as OpenStax ask that every expected count be at least 5 before you trust it. Fisher’s test has no such requirement, because it computes the p-value from the exact distribution, so it is the safer choice for a 2×2 table with a small cell.

Does the order of the rows or columns matter?

For the two-sided p-value, no. Swapping the two groups leaves it unchanged (0.0698 for both orders in the example below). The one-sided p-values swap places, and the sample odds ratio becomes its reciprocal, so keep track of which group and which outcome is the first row and column.

How to arrange the table

Put the two groups in the rows and the two outcomes in the columns: the first row is a successes and b failures in group 1, the second row is c successes and d failures in group 2. The test treats the row and column totals as fixed and asks how likely it is to see a count in the top-left cell as extreme as a, by chance alone. All counts must be whole numbers; proportions or percentages cannot be entered.

Worked example: 7 of 10 versus 2 of 10

Suppose 7 of 10 treated patients improve against 2 of 10 controls. The table is 7, 3 / 2, 8, with 9 improvements among 20 patients. The sample odds ratio is (7 × 8) ÷ (3 × 2) = 9.33. The two-sided exact p-value is 0.0698, the one-sided p-value for more improvement in the treated group is 0.0349, and the other tail is 0.9973. At α = 0.05 the two-sided p-value is above the cutoff, while the one-sided p-value is below it, which is why the direction has to be fixed before the data are seen. Reversing the group order gives the same two-sided p-value, 0.0698.

Common mistakes

Choosing the one-sided test after seeing which way the data lean doubles the false-positive risk, so use the two-sided p-value unless the direction was fixed in advance. Using percentages instead of counts, or counts from paired observations (the same patient measured twice), also breaks the test: paired yes/no data need McNemar’s test, not this one. Last, a significant result says the association is unlikely to be chance, not that the effect is large; read the odds ratio and the counts as well as the p-value.

Sources: R. A. Fisher, The Design of Experiments (1935); the documentation of R’s fisher.test and SciPy’s fisher_exact for the two-sided rule and the odds ratios; OpenStax Introductory Statistics 2e, section 11.3 (test of independence and the expected-count requirement). Related: chi-square test of independence calculator, relative risk calculator, hypergeometric calculator and odds calculator.