Two raters label the same items, their category counts form a cross-tabulation, and observed and expected agreement are summarized separately.
A conceptual view of aligned ratings, a category matrix and an agreement summary.
On this page

When Cohen's kappa fits

Use unweighted Cohen's kappa for two raters who independently assign one nominal category to each of the same items. Each line in the calculator represents one item, and the two lists must stay in the same order. Examples include two reviewers classifying abstracts as include or exclude, or two coders assigning labels to interview excerpts.

This calculator treats labels as nominal: it does not assume that one category is closer to another. For ordered ratings where disagreement distances matter, use a method with a justified weighting scheme. For more than two raters, repeated ratings, multiple labels per item, or missing observations, choose a method suited to that design instead of forcing the data into this calculation.

Supporting reading: [1] [2]

How the calculation works

The cross-tabulation count in row i and column j is the number of items rated category i by Rater A and category j by Rater B. Observed agreement pₒ is the diagonal count divided by the total number of paired items. Expected agreement pₑ is the sum, across categories, of each category's Rater A proportion multiplied by its Rater B proportion.

Unweighted kappa is κ = (pₒ − pₑ) ÷ (1 − pₑ). The expected-agreement term uses the raters' observed category frequencies under an independence model; it is not an estimate of every source of disagreement or a literal chance process. The result can be negative when agreement is below that expected by the marginal frequencies.

For a numeric example, suppose two categories produce 8 matches in each diagonal cell and 2 ratings in each off-diagonal cell across 20 items. Observed agreement is pₒ = (8 + 8) ÷ 20 = 0.80. Each rater used each category 10 times, so pₑ = (10 ÷ 20 × 10 ÷ 20) + (10 ÷ 20 × 10 ÷ 20) = 0.50. Therefore κ = (0.80 − 0.50) ÷ (1 − 0.50) = 0.60. This example illustrates the arithmetic; it is not a threshold for acceptable agreement.

Supporting reading: [1] [2]

Interpret the result with the category table

Read the diagonal and off-diagonal cells before summarizing κ. The same kappa value can accompany different category prevalence and disagreement patterns. Report the number of paired items, the categories and codebook, raw agreement, the contingency table, and the method used to select or train raters. Explain how disagreements were handled.

There is no universal cutoff that makes a coding scheme reliable in every discipline or study. Kappa is one descriptive statistic for paired nominal ratings; it does not establish that the categories are valid, the items are representative, the raters were blinded, or the underlying construct was measured well. Interpret it alongside the study design and field-specific standards.

Supporting reading: [2] [3]

Prepare and check aligned ratings

Use one line per item in each box and keep the same item order in both lists. Use the same spelling and capitalization for the same category. If an item is missing or cannot be rated, do not delete its line from just one list; select a missing-data method appropriate to the study and document it before calculating agreement.

For screening or qualitative coding, define the categories before independent rating and preserve the original decisions before resolving disagreements. The calculator runs locally in the browser and does not send the labels to AI Essay. Do not paste identifiable participant data or confidential records into a shared device.

Supporting reading: [2] [4]

Free agreement calculator

Compare two sets of ratings

Enter one category per line. Matching line numbers must refer to the same item.

Example

include
include
exclude
include

Your labels stay in this browser. Do not enter identifiable or confidential records on a shared device.

The example shows 4 aligned items. Edit either list, then calculate.

Questions about this topic

What data do I enter in a Cohen's kappa calculator?

Enter two aligned lists with one nominal category per item in each list. Row 1 in both boxes must refer to the same item, row 2 to the same next item, and so on.

Can Cohen’s kappa be used for more than two raters?

This calculator is for exactly two raters. Multiple raters or repeated ratings call for a method designed for that structure.

Does a higher kappa prove the coding is valid or reliable?

No. Kappa summarizes agreement beyond an expected-agreement term based on the raters’ category frequencies. It does not validate the categories, sample, coding process or construct, and no universal cutoff is applied here.

Why is kappa undefined when both raters use only the same category?

When expected agreement is 1, the denominator 1 − pₑ is zero, so the formula has no defined value. The raw agreement is still shown; the calculator reports kappa as undefined instead of assigning a value.

Are my ratings uploaded?

No. The calculation runs in this browser tab. The labels you enter are not sent to AI Essay.

Sources and editorial notes

The supporting reading below informs our editorial guidance. Exercises and scenarios illustrate a method; they do not report a completed product test or measured learning outcome. Read our editorial policy or send a correction with a supporting source to info@ai-essay.com.

  1. scikit-learn API: Cohen's kappa, expected agreement and undefined cases
  2. McHugh: Interrater reliability: the kappa statistic
  3. Artstein and Poesio: Inter-coder agreement for computational linguistics
  4. Cohen: A coefficient of agreement for nominal scales