What are order and practice effects in association measures?

Imagine investigating the meanings people associate with two fictional employers. The first display pairs employer A with the idea of trust. Later, the response-key assignments reverse. A difference in reaction times appears. Does that mean A is more closely associated with trust? Or have respondents learned a rule during the first block that makes the second block harder to complete?

These are different explanations for the same observation. Sound measurement separates the intended psychological contrast from the procedure that makes the contrast measurable. This is not an argument that response times are useless. It is an argument that they become informative only in the context of a carefully specified task.

The article What do reaction times mean in implicit association research? explains response latency and relative task performance. This second article addresses a narrower question: how can we avoid mistaking order, learning or fatigue for an association difference? For the broader choice of research methods, see How do you measure associations?.

An order effect occurs when the position of a task, stimulus, question or experimental condition changes the observed result. Two participants may sort exactly the same concepts but encounter them in different sequences. One completes A then B; the other completes B then A. If average task performance depends on this order, comparing A and B without considering the sequence is insufficient.

An order effect is not itself an association effect. It may emerge because people learn a particular response rule, because an earlier key assignment interferes with a later one, because a participant changes strategy, or because attention declines. In applied research, preceding questionnaires, instructions and earlier exposures can also influence the next task.

At least three order questions need to be distinguished:

1. Block order: which combined category pairing is presented first?
2. Stimulus and key order: in what sequence do words or pictures appear, and which categories are initially assigned to left or right?
3. Assessment order: when is the association task administered in relation to a questionnaire, intervention or previous assessment?

Each calls for a different design control. Reversing the order of two IAT blocks does not, for example, resolve unequal exposure to an intervention between pretest and posttest.

The conventional IAT asks participants to classify examples from two target categories and two attribute categories. In one combined block, certain categories share a response key. In the other, the pairing is reversed. The comparison concerns relative performance across these combined tasks [1].

The two blocks are not performed under identical learning conditions. Participants become familiar with the instructions, form response habits and establish a temporary task set. When the next combined task starts, an earlier rule may have to be abandoned. One relevant concept is task-set inertia: the influence of an established response rule after the requirements have changed [4, 5].

This does not mean that the entire IAT effect can be explained as an order artefact. It means that a score can reflect several contributing processes. Klauer and Mierke demonstrated that prior IAT blocks could influence performance in subsequent evaluative tasks, providing evidence that task switching contributes to compatibility-order phenomena [5].

Consequently, a research report that simply states that the first pairing was faster is incomplete. We need to know which respondents experienced that pairing first, how much training they received and whether the pattern remains when the order changes.

Counterbalancing means deliberately distributing the possible sequences across participants. Some complete A before B; others complete B before A. Ideally, the sequence is also balanced within each treatment group and, where analytically necessary, within important participant segments [7].

Design group	First combined task	Second combined task	What we assess
Group 1	Employer A + trust	Employer B + trust	Performance under A-B order
Group 2	Employer B + trust	Employer A + trust	Performance under B-A order

This simplified illustration uses fictional categories. An actual conventional IAT includes further category mappings and practice blocks.

If the measured contrast changes substantially when the order is reversed, the procedure may be contributing to the effect. Measuring both sequences makes that influence examinable instead of assuming it away.

A fair distribution requires more than writing 'randomly assigned' in the methodology. Check the number of people who actually completed each sequence. Differential drop-out can undermine a nominally balanced design. When comparing an intervention and a control condition, balance the sequence within both groups. Otherwise a difference attributed to the intervention might instead reflect a procedural difference.

Counterbalancing distributes a possible nuisance influence across conditions; it does not remove the influence from each individual record. Participants may differ in how strongly they retain an old rule, how attentive they are or how easily they switch response mappings.

For group comparisons, balanced orders can prevent one condition from receiving a consistently more favourable sequence. For conclusions about one person, the advantage is more limited: that individual generally completed only one particular sequence, unless the study was deliberately designed with repeat administrations.

The 2022 recommendations for IAT research consider balancing the order of the two combined tasks generally desirable. They also recommend balancing which categories start on the left or right response key [7]. These are design recommendations, not proof that a particular score is free from every source of bias.

In research comparing several audiences, consider whether order is balanced within the segments that matter. If an important subgroup is too small for a meaningful comparison, report the limitation rather than producing a misleadingly precise segment estimate.

A practice effect is a performance change caused by experience with the procedure. With repetition, people learn the category labels, where each response key belongs and how the next trial is likely to unfold. Their responses may speed up even when the target attitude remains unchanged.

In an IAT, practice can operate in more than one direction. Familiarity with the general procedure may make later blocks faster. At the same time, the first key mapping can interfere with the reversed mapping. The resulting pattern depends partly on the amount of practice and the exact task design [3, 4].

'Learning the test' and 'changing one's evaluation' are different psychological possibilities. Both might occur, but the same decrease in response time cannot by itself establish which one happened.

This is especially important when an organisation repeats an association task across projects or waves. Respondents may become experienced test-takers while their actual associations remain relatively stable. Identical words on the screen do not guarantee that two administrations are directly comparable.

The conventional IAT contains practice and combined-task blocks. Before participants perform a reversed pairing, it matters that they understand the changed response mapping. Nosek, Greenwald and Banaji reported in 2005 that additional practice with the reversed categorisation markedly reduced a known nuisance order effect in their IAT procedure [3].

The conclusion is not that a particular number of practice trials always works. Their finding concerns a specified experimental arrangement. Appropriate practice length depends on task structure, stimuli, population and evidence that people can perform the switch. Researchers should record which procedure they used and why its block lengths are suitable.

Too little training can leave a major difference in familiarity between blocks. Yet unlimited training is not automatically better: overly long procedures may introduce fatigue, boredom or selective drop-out. The aim is comprehension and stable performance, rather than maximising the number of trials.

Suppose an organisation conducts an association test before a campaign and again three weeks later. The average IAT score moves closer to zero. It is tempting to declare that the campaign caused the change.

However, administering an IAT repeatedly can itself affect results. The 2022 best-practice article notes that, on average, a first IAT administration may yield a score further from zero than a later administration. If this is ignored, a reduction in score magnitude may be mistaken for an intervention effect [7].

Two safeguards deserve particular attention:

1. A comparison group that also completes both measurement occasions. This allows researchers to examine whether the change in the intervention group differs from change observed without the intended intervention.
2. Standardised prior task experience. Participants can complete a suitable preparatory IAT before the actual baseline assessment, where this is methodologically justified and handled consistently across conditions.

Neither safeguard runs on autopilot. A control group must be appropriate for the intended causal contrast, and pre-exposure should not introduce its own changes in the construct under study. A defensible analysis reports change between conditions, measurement uncertainty and plausible alternative explanations, not only before-and-after numbers.

A respondent who reaches the end of a long study may respond more slowly or make more errors as concentration declines. Someone else might respond faster by becoming less cautious. Average latency cannot reveal which explanation is correct. Accuracy, error handling and task duration therefore belong in the interpretation [2].

Record session length, breaks, the task's position within a longer survey and the hardware or software used. Remote studies may include variation in devices and environments. These differences do not automatically invalidate the results, but they must be considered when judging data quality.

A higher error rate in the second block may indicate difficulty switching rules, unclear instructions or reduced attention. It is not direct proof of a stronger negative association. Examine correct response times, error patterns and task order together.

The scoring algorithm also matters. The conventional IAT's D-scoring procedure addresses some differences in overall response speed and handles errors through defined rules [2]. No statistical adjustment, however, can rescue a task with intrinsically misleading instructions or fundamentally incomparable stimuli.

Order is not limited to which combined block comes first. Which category starts on the left or right and how stimuli appear across trials can also influence performance.

The 2022 recommendations discuss balancing initial response sides, choosing suitable category examples and avoiding excessively long sequences in which the same key is repeatedly correct [7]. Such repetitions can facilitate performance through motor-response priming without changing the target association.

Stimuli also differ in familiarity, length, ambiguity and ease of classification. Translation can alter those properties. A multilingual project must not assume that an identical conceptual label creates identical task difficulty in Dutch, English and German.

The principle is straightforward: when comparing two psychological conditions, avoid systematically changing unrelated task characteristics at the same time. Where control is impractical, randomise, balance, measure or disclose the remaining difference.

A fictional research team gives one hundred participants two pairings in an indirect association task. Half begin with employer A and trust; the others begin with employer B and trust. The numbers below are illustrative only and are not Neurofactor findings.

Presentation order	Illustrative mean latency contrast	Possible interpretation
A pairing first, B pairing second	95 ms	May include an intended association contrast and an order influence
B pairing first, A pairing second	35 ms	The observed contrast is smaller in the reversed order

A superficial report might average the two values and stop. A careful researcher asks why the contrast depends on sequence. Was switch practice too short? Did one pairing systematically require a more difficult response mapping? Were errors comparable? Does the pattern persist in an analysis that includes order?

This example cannot establish whether A or B is more attractive. It demonstrates why the procedural design and allocation of participants must be understood before assigning psychological meaning to response-time differences.

1. Define the research question. Are you estimating a relative association between categories, comparing audience segments or assessing change across time? Each aim can require different design choices.

2. Specify the exact task variant. Document categories, stimuli, response keys, block structure, practice, software and instructions. Where possible, use an appropriately supported existing protocol.

3. Decide counterbalancing in advance. Distribute combined-task orders across participants. Where required, do so separately within treatment conditions and relevant strata.

4. Provide adequate practice. Check that instructions are understood and that participants can adopt the reversed response mapping. Justify the number of training trials.

5. Prespecify scoring and exclusions. State how incorrect responses, implausibly short or long latencies and incomplete sessions will be handled before inspecting desirable effects.

6. Examine order and group patterns. Where appropriate, analyse whether order interacts with the experimental condition. Do not hide an important interaction behind a single aggregate number.

7. Report remaining limitations. Separate the conclusion supported by the task from decisions that require additional evidence about brands, employers or interventions.

A transparent protocol makes the measurement auditable. Readers can evaluate not just the final score, but how the score was obtained.

Start with interpretable breakdowns. Compare response time and error profiles by block and sequence. Check participant counts and whether drop-out differs by condition. Examine whether responses speed up within blocks, whether switching temporarily slows people down and whether anomalies are concentrated in a language version, device type or setting.

Do not remove inconvenient blocks after the fact until the desired effect appears. Alternative analyses can be informative, but they need methodological justification and clear reporting. If exploratory analysis uncovers an unexpected order effect, that is an important finding about the procedure and belongs in the report.

The method-specific variation investigated by Mierke and Klauer and the debate about other processes, such as asymmetries in category salience, underline why an IAT score cannot be reduced to one psychological mechanism [4, 6].

A test can be administered consistently yet remain subject to an unwanted procedural influence. Conversely, a theoretically meaningful contrast may be measured with too much random error to draw a precise conclusion. And a reliably measured change is not by itself evidence of what caused that change.

Reliability concerns consistency and measurement error. Validity concerns whether the task justifies the intended interpretation. Causality concerns the basis for statements about cause and effect. Counterbalancing is relevant to research design, but cannot replace these separate assessments.

Reliability in association measures will be explored separately in knowledge base article NF-AS-KB-009. That forthcoming article should only be linked after publication. The underlying logic of response latencies is addressed in NF-AS-KB-007 [2, 7].

A client needs more than a bar chart showing two averages. A useful report identifies the category labels and example stimuli, the orders that were actually administered, allocation across orders, practice blocks, error-handling rules and the analysis used.

It should also provide participant counts, exclusions, quality criteria, appropriate uncertainty estimates and limitations. If an intervention is evaluated, explain how the comparison group and assessment occasions were organised.

Above all, fit the claim to the evidence. 'In this task and sample, relative performance differed after accounting for block order' is not the same as 'we have established the hidden preferences of all job applicants'. The first might be defensible under an appropriate design; the second would go far beyond the data.

• Giving everyone the same order. This makes it harder to distinguish category differences from the course of the task.
• Skipping switch practice. The second pairing can be unfairly affected by the response rule learned in the first.
• Attributing every pre-post change to the intervention. Previous exposure to the test remains an alternative explanation [7].
• Reporting response times without errors. Accuracy and response caution remain invisible.
• Hiding allocation imbalances in an overall mean. Small subgroups may have different procedural patterns.
• Treating scoring conventions as universal. A validated procedure for the conventional IAT does not automatically validate all modified tasks.
• Claiming more than the evidence supports. A well-run indirect task does not by itself predict an individual's purchase or application.

Strong association research does not start by asking how fast people can press a key. It starts by asking what comparison will generate useful evidence. Task order, practice, repetition and fatigue are not incidental details that can be omitted from the report. They are part of the conditions under which the observed performance occurred.

Counterbalancing, adequate practice and transparent analysis reduce the risk of mistaking a procedural difference for a difference in associations. They do not eliminate all uncertainty. What they can do is make the study more honest, inspectable and useful for decisions.

When evaluating a research proposal, ask three questions alongside the final result: who received which pairing first, how much practice was provided, and what happened when the task was repeated? Those questions help distinguish a persuasive number from persuasive evidence.