Cohen’s kappa calculator
How well do two raters agree beyond chance? Enter their counts as a table; get kappa with its confidence interval.
Rows are rater A’s categories, columns rater B’s, in the same order. The table must be square, from 2 × 2 up to 5 × 5, with whole numbers. The diagonal holds the items on which they agree.
What does Cohen’s kappa measure?
How much two raters agree beyond what chance alone would produce. If pₒ is the share of items they put in the same category and pₑ is the share expected if each rater assigned categories at random with their own overall frequencies, then κ = (pₒ − pₑ) ÷ (1 − pₑ). A kappa of 1 is perfect agreement, 0 is the level expected by chance, and negative values mean agreement worse than chance. Wikipedia’s example has two readers judging 50 grant proposals: both say yes to 20, both say no to 15, and they split on 15. Then pₒ = 35 ÷ 50 = 0.70, pₑ = 0.50 and κ = 0.20 ÷ 0.50 = 0.40.
Worked example: normal and abnormal readings from McHugh (2012)
McHugh’s open-access paper in Biochemia Medica counts 222 items rated normal or abnormal by two raters: 147 normal by both, 62 abnormal by both, and 13 disagreements (3 and 10). The raw agreement is 209 ÷ 222 = 94.1%, the agreement expected by chance from the totals (157 and 65 against 150 and 72) is 57.3%, so κ = (0.9414 − 0.5728) ÷ (1 − 0.5728) = 0.863. The paper prints κ = 0.85 and an interval of 0.78 to 0.92 for this table; recomputing from the counts gives 0.863 and, with the large-sample standard error of 0.0367, an interval of 0.79 to 0.93. Percent agreement looks excellent here, but it can be misleading when one category dominates, which is why kappa corrects for chance.
How are the standard error and confidence interval calculated?
The interval is κ ± z × SE. This calculator uses the large-sample standard error of Fleiss, Cohen and Everitt (1969), the one reported by R’s vcd::Kappa; it does not treat the chance agreement as fixed. Simpler formulas exist: Cohen’s original SE = √(pₒ(1 − pₒ) ÷ (N(1 − pₑ)²)), which McHugh uses, gives 0.0369 for the table above, close to 0.0367. The z test of κ = 0 needs the variance under the null hypothesis, the version in R’s irr::kappa2, which is larger (0.067 here, z = 12.9), so use that one for the test and the large-sample SE for the interval. For small tables the interval is rough; McHugh suggests at least 30 comparisons as a general heuristic.
What is weighted kappa, and which weights should I choose?
For ordered categories (mild, moderate, severe) a one-step disagreement is less serious than a two-step one, and weighted kappa credits partial agreement: each cell gets a weight, 1 on the diagonal. Linear weights are 1 − |i − j| ÷ (k − 1) (R’s “Equal-Spacing”) and quadratic weights are 1 − (i − j)² ÷ (k − 1)² (“Fleiss-Cohen”), which punish distant disagreements much more. For a 4 × 4 table of 50 patients rated by two doctors on four ordered anxiety levels (rows 11, 3, 1, 0; 1, 9, 0, 1; 0, 1, 10, 0; 1, 2, 0, 10) the calculator gives κ = 0.7335 (SE 0.0752, 95% interval 0.586 to 0.881), linear-weighted κ = 0.7475 (SE 0.0791) and quadratic-weighted κ = 0.7664 (SE 0.0902), matching R’s vcd::Kappa. With only two categories, weights make no difference. Choose the weights before you look at the results, and keep the categories in their natural order.
How should I read a kappa value?
Landis and Koch (1977) labelled values below 0 as poor, 0–0.20 as slight, 0.21–0.40 as fair, 0.41–0.60 as moderate, 0.61–0.80 as substantial and 0.81–1.00 as almost perfect. This is a widely used convention, not a law: the authors gave no evidence for the cut-offs. McHugh (2012) argues it is too lenient where errors matter, since a kappa of 0.41 counts as “moderate”; she proposes none (0–0.20), minimal (0.21–0.39), weak (0.40–0.59), moderate (0.60–0.79), strong (0.80–0.90) and almost perfect (above 0.90), and says that a kappa below 0.60 means inadequate agreement. Which standard is right depends on the stakes, so report the value and its interval, not just the label.
Why is kappa low even when the raters agree most of the time?
Kappa depends on how often each category is used. Wikipedia compares two pairs of raters who agree on 60 of 100 items: when one rater says yes 70 times (45 + 25 of 100, a table of 45, 15, 25, 15) chance agreement is 54% and κ = 0.130; when the yes-rate is skewed the other way (25, 35, 5, 35) chance agreement is 46% and κ = 0.259. When one category is very common, or when the two raters use categories at different rates, kappa falls, and it can never exceed a maximum set by the totals; the calculator shows that maximum. Kappa is also not a measure of accuracy: two raters can agree and both be wrong. If one reading is a trusted reference, use sensitivity and specificity instead; for more than two raters, Fleiss’ kappa; for numeric scores, an intraclass correlation.
For general information only — not medical advice. Talk with a qualified professional about your own situation. Sources: J. Cohen, “A coefficient of agreement for nominal scales,” Educational and Psychological Measurement 20 (1960) 37–46, and “Weighted kappa,” Psychological Bulletin 70 (1968) 213–220; J. L. Fleiss, J. Cohen and B. S. Everitt, “Large sample standard errors of kappa and weighted kappa,” Psychological Bulletin 72 (1969) 323–327; J. R. Landis and G. G. Koch, “The measurement of observer agreement for categorical data,” Biometrics 33 (1977) 159–174; M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia Medica 22(3) (2012) 276–282; documentation of R (vcd::Kappa, irr::kappa2) and scikit-learn (cohen_kappa_score); Wikipedia, “Cohen’s kappa”. Related: sensitivity and specificity calculator, chi-square test of independence calculator and percentage calculator. How we check our calculators.