Key facts

• The observers watch at the same time but record independently.
• Their tallies are correlated, often with Spearman’s rho.
• A correlation of +0.80 or above means reliability is good.
• To improve it: operationalise the categories, train the observers in a pilot study, and discuss disagreements.

How to do it

1. Rank each observer’s tallies separately (lowest = 1). Tied scores get the mean of the ranks they would occupy.
2. Work out d = rank 1 − rank 2 for each row, then square it.
3. Add the squares to get Σd².
4. Use r_s = 1 − 6Σd² ÷ N(N² − 1).