On this page
When Cohen's kappa fits
Use unweighted Cohen's kappa for two raters who independently assign one nominal category to each of the same items. Each line in the calculator represents one item, and the two lists must stay in the same order. Examples include two reviewers classifying abstracts as include or exclude, or two coders assigning labels to interview excerpts.
This calculator treats labels as nominal: it does not assume that one category is closer to another. For ordered ratings where disagreement distances matter, use a method with a justified weighting scheme. For more than two raters, repeated ratings, multiple labels per item, or missing observations, choose a method suited to that design instead of forcing the data into this calculation.
How the calculation works
The cross-tabulation count in row i and column j is the number of items rated category i by Rater A and category j by Rater B. Observed agreement pₒ is the diagonal count divided by the total number of paired items. Expected agreement pₑ is the sum, across categories, of each category's Rater A proportion multiplied by its Rater B proportion.
Unweighted kappa is κ = (pₒ − pₑ) ÷ (1 − pₑ). The expected-agreement term uses the raters' observed category frequencies under an independence model; it is not an estimate of every source of disagreement or a literal chance process. The result can be negative when agreement is below that expected by the marginal frequencies.
For a numeric example, suppose two categories produce 8 matches in each diagonal cell and 2 ratings in each off-diagonal cell across 20 items. Observed agreement is pₒ = (8 + 8) ÷ 20 = 0.80. Each rater used each category 10 times, so pₑ = (10 ÷ 20 × 10 ÷ 20) + (10 ÷ 20 × 10 ÷ 20) = 0.50. Therefore κ = (0.80 − 0.50) ÷ (1 − 0.50) = 0.60. This example illustrates the arithmetic; it is not a threshold for acceptable agreement.
Interpret the result with the category table
Read the diagonal and off-diagonal cells before summarizing κ. The same kappa value can accompany different category prevalence and disagreement patterns. Report the number of paired items, the categories and codebook, raw agreement, the contingency table, and the method used to select or train raters. Explain how disagreements were handled.
There is no universal cutoff that makes a coding scheme reliable in every discipline or study. Kappa is one descriptive statistic for paired nominal ratings; it does not establish that the categories are valid, the items are representative, the raters were blinded, or the underlying construct was measured well. Interpret it alongside the study design and field-specific standards.
Prepare and check aligned ratings
Use one line per item in each box and keep the same item order in both lists. Use the same spelling and capitalization for the same category. If an item is missing or cannot be rated, do not delete its line from just one list; select a missing-data method appropriate to the study and document it before calculating agreement.
For screening or qualitative coding, define the categories before independent rating and preserve the original decisions before resolving disagreements. The calculator runs locally in the browser and does not send the labels to AI Essay. Do not paste identifiable participant data or confidential records into a shared device.
Free agreement calculator
Compare two sets of ratings
Enter one category per line. Matching line numbers must refer to the same item.
include
include
exclude
include
Your labels stay in this browser. Do not enter identifiable or confidential records on a shared device.
The example shows 4 aligned items. Edit either list, then calculate.
Agreement summary
Unweighted Cohen’s κ
Observed agreement
Expected agreement
Paired items
Kappa is one summary of these paired nominal ratings. Interpret it with the table, category prevalence, codebook and study design. No universal cutoff is applied.
Questions about this topic
What data do I enter in a Cohen's kappa calculator?
Enter two aligned lists with one nominal category per item in each list. Row 1 in both boxes must refer to the same item, row 2 to the same next item, and so on.
Can Cohen’s kappa be used for more than two raters?
This calculator is for exactly two raters. Multiple raters or repeated ratings call for a method designed for that structure.
Does a higher kappa prove the coding is valid or reliable?
No. Kappa summarizes agreement beyond an expected-agreement term based on the raters’ category frequencies. It does not validate the categories, sample, coding process or construct, and no universal cutoff is applied here.
Why is kappa undefined when both raters use only the same category?
When expected agreement is 1, the denominator 1 − pₑ is zero, so the formula has no defined value. The raw agreement is still shown; the calculator reports kappa as undefined instead of assigning a value.
Are my ratings uploaded?
No. The calculation runs in this browser tab. The labels you enter are not sent to AI Essay.
Sources and editorial notes
The supporting reading below informs our editorial guidance. Exercises and scenarios illustrate a method; they do not report a completed product test or measured learning outcome. Read our editorial policy or send a correction with a supporting source to info@ai-essay.com.