A.8 Unreliable data and poor procedural fidelity
Supervisors make treatment decisions from the data RBTs collect and assume the plan was carried out as written. If the data are unreliable or the procedures are implemented inconsistently, those decisions can be wrong, and the client pays for it in lost time, slower progress, or continued problem behavior. This task asks you to explain those risks and recognize the everyday habits that create them.
What the outline says
Task A.8 in content area A, Data Collection and Graphing (13 of the 75 scored questions): "Describe the risks associated with unreliable data collection and poor procedural fidelity."
Two separate problems: bad data and poor implementation
This task covers two related but different things. Unreliable data collection is a measurement problem: the numbers on the data sheet do not match what the client actually did. Poor procedural fidelity is an implementation problem: the intervention the client experienced is not the intervention written in the plan.
Procedural fidelity (also called procedural integrity or treatment integrity) is the degree to which an intervention is carried out as designed. A supervisor usually measures it by watching a session and scoring each step of the procedure on a checklist, then reporting the percentage of steps done correctly. For example, 17 of 20 steps done correctly is 17 ÷ 20 × 100 = 85%.
Either problem alone can mislead the team. Together, they make it almost impossible to know what is really happening.
Key measurement terms
- Reliability: measuring the same events again produces the same values. Reliable data are consistent.
- Validity: the data measure the behavior and dimension that the goal is actually about. Counting minutes seated is not a valid measure of a goal about completing math problems correctly.
- Accuracy: the recorded values match what really happened.
- Interobserver agreement (IOA): two people record the same session independently and their records are compared. With total count IOA, divide the smaller count by the larger and multiply by 100. If you count 16 and your supervisor counts 20, IOA is 16 ÷ 20 × 100 = 80%.
High IOA shows consistency, not correctness. Two observers who both use an outdated or vague definition can agree perfectly and still record the wrong thing.
What makes data unreliable
- Vague operational definitions. If aggression is defined as "acting aggressively," each observer decides what counts, and counts differ.
- Observer drift. Over time an observer gradually changes how they apply a definition, for example starting to count whines as crying when the definition requires tears.
- Observer expectations. An RBT who expects improvement after a new intervention may score borderline instances as non-occurrences, making the intervention look better than it is.
- Reactivity. The client behaves differently because they know they are being watched, so the data do not represent typical behavior.
- Recording late or from memory. Counts written at the end of a long session are estimates.
- Missed observations. Undercounting while attention is elsewhere makes behavior look lower than it is.
- Changing the measurement procedure without direction. Switching from a frequency count to partial-interval recording, for example, produces numbers that cannot be compared with earlier data: a count of responses and a percentage of intervals with any response measure different things.
- Data-entry errors, such as placing an intervention session in the baseline phase on a graph.
What poor procedural fidelity looks like
Fidelity errors fall into two groups. An error of omission leaves out a required step, such as forgetting to deliver a token after a correct response. An error of commission adds something the plan does not call for, such as saying "please use a quiet voice" when the plan says to withhold attention for screaming.
Treatment drift is a gradual slide away from the written procedure, often unnoticed: an RBT slowly drops a 3-second time delay and starts prompting immediately on every trial. Fidelity can also be higher when the supervisor is watching than on other days, so observed fidelity may overestimate everyday implementation.
The risks to the client and the team
- An ineffective intervention looks effective and is continued. Undercounting makes problem behavior seem to decrease.
- An effective intervention looks ineffective and is dropped. If a plan is implemented inconsistently, poor results may be blamed on the plan.
- The supervisor cannot tell why progress is poor. When fidelity is low, lack of progress could come from the plan or from the implementation.
- Skills are moved to mastery too soon. Recording prompted responses as independent inflates percent correct, and the client later cannot perform the skill alone.
- Problem behavior becomes more persistent. Occasionally giving escape or attention after a behavior that the plan puts on extinction reinforces it intermittently.
- Prompt dependence. Prompts that are not faded as written can leave the client waiting for help.
- Safety risks. Under-recording self-injury or aggression can hide how serious a behavior is and delay protective changes to the plan.
- Ethical and legal problems. RBTs must accurately carry out services and accurately complete required documentation such as client data and billing records (RBT Ethics Code 2.02). Inventing, copying, or altering data is falsification.
What you do as the RBT
- Know each operational definition and measurement procedure before the session. Ask your supervisor when anything is unclear.
- Record behavior as it happens, or as soon as the procedure allows, never from memory at the end of the day.
- Use only the measurement procedure the plan specifies. Raise problems with it instead of switching on your own.
- Implement every step as written, and ask for clarification or a demonstration before using a procedure you are unsure of.
- If data were not collected, document that honestly and tell your supervisor. Never fill gaps with estimates or copied numbers.
- If you discover you made an implementation or recording error, tell your supervisor promptly and follow their direction. The supervisor needs to know in order to interpret the data.
- Welcome IOA checks and fidelity observations. They protect the client and show where retraining is needed.
Examples and non-examples
| Example | Non-example |
|---|---|
| An RBT records each trial's response immediately, codes prompted responses as prompted, and tells the supervisor that the client's caregiver has started giving attention for screaming between sessions. | An RBT writes "about 8 hits" at the end of a 3-hour session from memory and enters it on the graph as a count. |
| An RBT notices that a co-worker counts a burst of hits as one occurrence while they count each hit, and both ask the supervisor to clarify the definition. | An RBT quietly changes how they count so their numbers match a co-worker's, without telling the supervisor. |
| An RBT who forgot to collect data on one target writes "no data collected" for that target and tells the supervisor. | An RBT copies the previous session's numbers because "the client's performance has been stable." |
Common mistakes
- Treating high IOA as proof the data are correct. Agreement shows consistency; both observers can share the same error.
- Confusing observer drift with treatment drift. Observer drift changes how behavior is recorded; treatment drift changes how the intervention is delivered.
- Confusing reactivity with observer bias. In reactivity the client's behavior really changes because they are observed; in observer bias the recorder's expectations shape what gets written down.
- Assuming poor client progress means the plan is wrong. Check fidelity first: the client may never have received the plan as written.
- Fixing an error quietly. Corrections to data or procedures go through the supervisor, so the record stays honest and interpretable.
How it's tested
Expect short scenarios in which an RBT records late, uses a vague definition, changes a procedure, misses steps, or gives attention or escape that the plan withholds. You will be asked to name the problem (observer drift, reactivity, error of omission, treatment drift) or, more often, to identify the consequence: a wrong decision about continuing or stopping a plan, premature mastery, more persistent behavior, or prompt dependence. When the question asks what the RBT should do, the answer almost always involves recording honestly and telling the supervisor rather than fixing things independently. Keep IOA (agreement between observers about the data) separate from procedural fidelity (accuracy of implementation).
Practice questions for task A.8
10 of our 33 questions on this task. The rest are in the Data Collection and Graphing practice sets and the mock exams.
-
Question 1Which practice is used to check how reliable an RBT's data are?
Show answer
D: A second observer independently records the same session for comparison. Interobserver agreement (IOA) compares the records of two observers who measured the same behavior at the same time without seeing each other's data. Agreement between independent observers is the usual check on how consistent and believable the data are.
- A: A trend shows the direction of the data, not whether the data were recorded consistently.
- B: Recounting tallies only checks arithmetic; it cannot show whether the tallies matched the behavior.
- C: A caregiver's signature confirms the session took place, not that the measurement was reliable.
Task A.8 · 40-hour curriculum section C
-
Question 2An RBT is busy during a 3-hour home session and waits until the session ends to write down how many times the client hit, estimating the number from memory. What is the main risk of this practice?
Show answer
C: The count may not match what happened, leading to wrong decisions. Counts reconstructed from memory are estimates, and estimates are easily too high or too low. The supervisor relies on these numbers to decide whether the plan is working, so inaccurate counts can lead to the wrong decision. Behavior should be recorded as it occurs, following the data procedure in the plan.
- A: Nothing about late recording by itself would make the hitting increase.
- B: A rate can still be calculated from any count and time; the problem is that the count may be wrong.
- D: Recording late does not turn a count into an interval or time-sampling procedure; it simply makes the count unreliable.
Task A.8 · 40-hour curriculum section C · RBT Ethics Code 2.02
-
Question 3During discrete-trial teaching, an RBT gives a gestural prompt on many trials but records those responses as independent correct responses. What is the most likely consequence?
Show answer
C: The target may be marked mastered before the client can do it alone. Recording prompted responses as independent inflates the percentage correct. The supervisor may then judge the skill mastered, move on, and later find that the client cannot perform it independently.
- A: Inflated data make the intervention look more effective, not less.
- B: The recording does not change the client's behavior; the risk is a wrong decision based on inflated data.
- D: Counting prompted responses as independent makes performance look better, not worse.
Task A.8 · 40-hour curriculum section C · RBT Ethics Code 2.02
-
Question 4At the end of the day, an RBT realizes they forgot to collect data on one target during a clinic session. What should the RBT do?
Show answer
C: Tell the supervisor and document that no data were collected for that target. Data must reflect what was actually observed and recorded. A gap that is honestly documented is far less harmful than invented data, which is falsification and can mislead every decision made from the graph.
- A: Entering numbers that were never observed is falsifying data.
- B: Recollections from someone who was not recording are not session data and should not be presented as such.
- D: Copied data are not observations of that session and misrepresent it.
Task A.8 · RBT Ethics Code 2.02 · RBT Ethics Code 1.01
-
Question 5In behavioral measurement, what does reliability mean?
Show answer
A: Measuring the same events again produces the same values. Reliability refers to consistency: when the same behavior is measured repeatedly, a reliable measurement system yields the same values. Data can be reliable and still not valid if they consistently measure the wrong thing.
- B: That describes validity, not reliability.
- C: Supervisor review is good practice but does not define reliability.
- D: That describes procedural fidelity of the intervention.
Task A.8 · 40-hour curriculum section C
-
Question 6A client's goal is to increase the number of math problems they complete correctly during independent work. The RBT records only how many minutes the client stays seated at the desk. What is the main problem with these data?
Show answer
B: The measure does not reflect the behavior the goal targets. Valid data directly measure the behavior and dimension the goal is about. A client can sit for many minutes without completing problems correctly, so time seated does not tell the team whether the goal is being met.
- A: Nothing in the scenario shows a gradual change in how a definition is applied.
- C: Duration is a continuous measure; the problem is what is being measured, not how.
- D: No second observer is mentioned; the measure could agree perfectly and still be the wrong measure.
Task A.8 · 40-hour curriculum section C
-
Question 7During an IOA check, an RBT counts 16 instances of a behavior and the supervisor, observing the same session independently, counts 20. Using total count IOA (smaller count divided by larger count, times 100), what is the agreement?
Show answer
C: 80%. Total count IOA = smaller count ÷ larger count × 100 = 16 ÷ 20 × 100 = 80%.
- A: 20% is the 4-count difference divided by 20, which is not the IOA formula.
- B: 44% comes from dividing 16 by the combined total of 36, which is not the formula.
- D: 125% divides the larger count by the smaller one; IOA cannot exceed 100%.
Task A.8 · 40-hour curriculum section C
-
Question 8Two RBTs record a client's elopement and reach 100% interobserver agreement. Later the supervisor finds that both were still using an old definition the team had replaced the month before. Which statement is accurate?
Show answer
D: The records were consistent but did not match the current definition. IOA shows that observers recorded the same thing, not that they recorded the right thing. Two observers using the same outdated definition can agree perfectly and still produce data that do not match what the team now needs to measure.
- A: Agreement does not guarantee accuracy; both observers can make the same error.
- B: High agreement does not fix a measurement of the wrong thing.
- C: IOA is about measurement; fidelity is about how the intervention was carried out.
Task A.8 · 40-hour curriculum section C
-
Question 9A client has made little progress on a skill program. Fidelity checks show the RBTs have been completing only about 60% of the program's steps correctly. Why does this make it hard for the supervisor to evaluate the program?
Show answer
A: The poor results may come from the implementation rather than the plan. When a plan is not implemented as written, the client never really experienced the plan. Lack of progress could be due to the plan or to the implementation errors, and the supervisor cannot tell which until fidelity improves.
- B: Fidelity data say nothing about the function of behavior.
- C: Low fidelity means the plan was not tested as designed, so it cannot show the design was wrong.
- D: Fidelity is about implementation; the client's data can still be recorded and graphed.
Task A.8 · 40-hour curriculum section E
-
Question 10While running a session in a home with several siblings, an RBT misses many of a client's tantrums because they are busy elsewhere in the room, so the recorded count is lower than what occurred. The graph shows tantrums decreasing. Which risk does this create?
Show answer
C: An ineffective intervention may look effective and be continued. Undercounting makes the behavior look as if it is improving. The supervisor may keep an intervention that is not producing change and delay needed adjustments.
- A: Undercounting makes the plan look more effective, not less.
- B: The problem is a falsely low level and a misleading downward trend, not added variability.
- D: Recording or not recording does not change how often tantrums happen.
Task A.8 · 40-hour curriculum section C
Answer every question to see your score.
Key terms
| Procedural fidelity | Procedural fidelity is the degree to which an intervention is carried out exactly as written in the plan; it is also called treatment integrity or procedural integrity. |
|---|---|
| Interobserver agreement | Interobserver agreement (IOA) is the degree to which two or more independent observers report the same values after measuring the same events. |
| Reliability | Reliability is the consistency of measurement: a reliable measurement system gives the same values each time the same behavior is measured. |
| Validity | Valid measurement directly measures the behavior that matters, using a dimension relevant to the question, under the conditions and times that matter. |
| Observer drift | Observer drift is an unintended change, over time, in the way an observer applies the operational definition when recording behavior. |
| Reactivity | Reactivity is a change in a person's behavior that happens because they know they are being observed or measured. |
| Operational definition | An operational definition describes a target behavior in observable, measurable terms so that different observers can agree on when it does and does not occur. |
| Frequency | Frequency is the number of times a behavior occurs during an observation period, recorded by counting every occurrence from start to finish. |
| Intermittent reinforcement | A schedule in which only some occurrences of a behavior are reinforced, as in fixed-ratio, variable-ratio, fixed-interval and variable-interval schedules. |
| Escape extinction | An extinction procedure for escape-maintained behavior in which problem behavior no longer ends, delays or avoids the demand or situation. |
| Mastery criterion | The preset level of correct, independent performance a learner must reach before a skill target is considered learned and moved out of acquisition. |
| Duration | Duration is how long a behavior lasts, measured from the moment it starts (onset) to the moment it ends (offset). |
| Token economy | A reinforcement system in which a learner earns tokens for target behaviors and later exchanges them for backup reinforcers. |
| Attention extinction | An extinction procedure for attention-maintained behavior in which the problem behavior no longer produces attention from others. |
| Differential reinforcement of alternative behavior (DRA) | Reinforcing a specific appropriate behavior that is an alternative to problem behavior while placing the problem behavior on extinction. |
| Trend | Trend is the overall direction in which data points on a graph move over time: increasing, decreasing, or showing no clear change (zero trend). |
| Time delay | A prompt-fading procedure that inserts a pause between the SD and the prompt, starting at 0 seconds, so the learner can respond before the prompt is given. |
| Prompt dependence | A situation in which a learner responds correctly only when prompted and waits for help instead of responding to the natural SD. |
| Prompt fading | Gradually and systematically reducing prompts so that the correct response comes under the control of the natural SD rather than the prompt. |
| Performance feedback | Specific information given to a person about how they performed a skill, including praise for what was done correctly and guidance on what to change. |
| Percentage | A percentage is a ratio expressed as parts per 100: the number of occurrences divided by the total, multiplied by 100. |
| Baseline | Baseline is a condition in which the target behavior is measured repeatedly before the intervention is introduced, giving a standard to compare against. |
| Phase change line | A phase change line is a vertical line on a graph that marks the point where one condition ends and another begins, such as baseline to intervention. |
| Partial interval recording | Partial interval recording scores an interval if the target behavior occurs at any time during that interval, however briefly. |
| Schedule of reinforcement | A schedule of reinforcement is a rule stating which responses will be reinforced, based on the number of responses, the passage of time, or both. |
| Discrete trial teaching | A structured teaching method that breaks skills into short, repeated learning trials, each with a clear instruction, the learner's response, a consequence, and a brief pause. |