Interobserver agreement
Interobserver agreement (IOA) is the degree to which two or more independent observers report the same values after measuring the same events.
To collect IOA, two people observe the same session at the same time, each recording independently with the same operational definition and without looking at each other's data. Their records are then compared and the agreement is expressed as a percentage. A commonly used minimum is 80% agreement, and many supervisors aim higher.
Two simple calculations come up often. Total count IOA divides the smaller total by the larger total and multiplies by 100: if one RBT counts 18 hand raises and the other 20, IOA is 18 ÷ 20 × 100 = 90%. Interval-by-interval IOA compares each interval: the number of intervals both observers scored the same way, divided by the total number of intervals, times 100. Total count IOA can hide disagreement, because both observers could reach the same total while counting different responses; interval-by-interval and other point-by-point methods are stricter.
High IOA shows that the data are believable and that the definition is being applied consistently, but it does not prove that the data are accurate. Two observers can agree and both be wrong, for example if both drift from the definition in the same way. Low IOA is a signal to review the definition and retrain observers. RBTs should collect IOA data honestly and independently; checking a partner's sheet or changing a score to match defeats the purpose.
Example and non-example
| Example | An RBT and a supervisor both record a client's elopement attempts during a 1-hour session; the RBT records 8 and the supervisor 8, so total count IOA is 100%. |
|---|---|
| Non-example | A supervisor checking off whether the RBT followed each step of the token plan is a procedural fidelity check, not IOA. |
Don't confuse it with
| Procedural fidelity | IOA checks whether observers agree on the data; procedural fidelity checks whether the intervention was carried out as written. |
|---|---|
| Reliability | IOA is a way to estimate the quality of data by comparing observers; reliability is the consistency of a measurement over repeated measurement of the same events. |
Test yourself
-
Question 1Why is it relatively easy to check the accuracy of permanent product data?
Show answer
B: The product remains available for a second person to count. Because the product lasts, a second observer can count the same items independently and compare results (interobserver agreement) at any time after the session.
- A: RBTs routinely count products; the supervisor does not always do it.
- C: A definition is still needed to decide what counts as a product, such as a correctly folded towel.
- D: A product count is an actual count, not an estimate, and errors in it do matter.
Task A.3 · 40-hour curriculum section C
-
Question 2What is the main reason target behaviors are given operational definitions?
Show answer
D: So different observers record the same instances. A precise definition lets every observer identify the same instances, which makes the data consistent and believable.
- A: A definition describes the behavior; finding its function takes a functional assessment.
- B: Interventions are designed by the supervisor, not chosen from a definition.
- C: Operational definitions describe behavior for measurement, not for diagnosis.
Task A.5 · 40-hour curriculum section C
-
Question 3Which practice is used to check how reliable an RBT's data are?
Show answer
D: A second observer independently records the same session for comparison. Interobserver agreement (IOA) compares the records of two observers who measured the same behavior at the same time without seeing each other's data. Agreement between independent observers is the usual check on how consistent and believable the data are.
- A: A trend shows the direction of the data, not whether the data were recorded consistently.
- B: Recounting tallies only checks arithmetic; it cannot show whether the tallies matched the behavior.
- C: A caregiver's signature confirms the session took place, not that the measurement was reliable.
Task A.8 · 40-hour curriculum section C
Answer every question to see your score.
On the RBT exam
This term belongs to task A.8 Unreliable data and poor procedural fidelity in content area A, Data Collection and Graphing, which is 13 of the 75 scored questions.