How do you test a behaviour change intervention in the real world?

Suppose an organisation wants to encourage employees to supervise interns. In interviews, many colleagues say that passing on knowledge matters to them. They identify with the idea of helping a new generation move forward. A communications team therefore develops a campaign around pride, development and meaningful work.

After the campaign, employees view the topic more favourably. The intranet article is widely read. Yet the number of supervision sessions barely changes. There are two possible sides to the problem: the message appealed to the desire to contribute, but did not address resistance to losing time. Or willingness has increased, but no hours have yet been freed up in the schedule.

That distinction is the difference between evaluating a campaign and evaluating a behaviour change intervention. The first tells you about reach, understanding and appreciation. The second must demonstrate whether the intended behaviour happens more often, better or for longer. People speaking positively about a change is not, in itself, evidence that the change has occurred.

Two layers of research work together in behaviour change. The associative layer investigates what an action or situation means to people. Think of “contributing together”, but also “extra work”, “hassle”, “risk” or “I am not good enough at this”. The behavioural layer investigates what people actually do in the relevant situation.

These two layers must not be conflated. An association can change without people taking action. Behaviour can also change because a process becomes easier, while the underlying association barely shifts. A review of interventions that changed implicit measures found no automatic translation into actual behaviour. The specific findings apply to the interventions and measurement instruments studied, not as a judgement on every form of association research [1].

Every associative hypothesis therefore needs a separate behavioural prediction. Not just: the association with “lost time” becomes weaker. But: among employees who are actually allocated time in their schedules, the proportion completing at least one supervision session within eight weeks increases. The first statement concerns a possible mechanism. The second concerns an observable outcome.

“Increasing buy-in” sounds like an objective, but is not a precise behavioural measure. The same applies to “becoming more sustainable”, “greater ownership” or “better adoption”. Make it concrete using five elements: who does what, where, when and within what period?

In the fictional intern-supervision example, the primary outcome is: the proportion of eligible, invited employees who have completed at least one actual supervision session within eight weeks. Agree in advance how a supervision session will be recorded, when someone counts and how to handle employees who go on holiday or leave the organisation. The numerator is the number of employees who perform the behaviour. The denominator is the predefined group of eligible, invited employees.

Also choose secondary outcomes, such as registrations, introductory sessions attended, intern satisfaction and willingness to do it again later. Make the distinction clear: registering is an intermediate step, not a completed supervision session. This prevents the easiest click to measure from replacing the outcome the organisation actually wants.

An intervention does not begin with a poster, training session or new button. It begins with a diagnosis. Interview those involved, examine the existing process, observe moments when behaviour stalls and investigate relevant associations. Compare people who do perform the desired behaviour with those who do not, without assuming that a single difference is immediately the cause.

For an intern supervisor, positive associations might include passing on professional expertise, being recognised and helping someone progress. Negative associations might include extra administration, making mistakes, having no time or being held accountable for someone else’s performance. These examples are hypotheses, not measured Neurofactor findings.

Then consider three possible, interacting explanations: employees may lack relevant skills or knowledge, lack the practical or social opportunity, or lack motivation. Confidence in one’s ability is not the same as capability itself and may also affect motivation. The COM-B framework distinguishes capability, opportunity and motivation as interacting influences on behaviour [2]. Examine which barriers occur before the behaviour, how often they occur and which groups they affect; do not assume that a single instrument can isolate every mechanism.

It is tempting to change everything at once: a new campaign, fewer forms, a different message from managers and more available time. If things improve afterwards, you still do not know why. So start with the smallest meaningful change that addresses the main hypothesised barrier.

Example hypothesis: employees consider supervision meaningful, but associate it with losing productive time. An hour reserved in advance for supervision lowers the practical and perceived barrier, leading more employees to actually complete a first supervision session.

Write down in advance what should happen if the hypothesis is correct. Which association might change? Which behaviour must change? When will that change become visible? And what alternative explanation could produce the same pattern? This makes the intervention testable, rather than merely the subject of a convincing retrospective story.

The strongest evidence comes not from an attractive graph, but from a fair comparison. Three approaches are useful in practice. Your choice depends on feasibility, privacy, participant numbers and the likelihood that groups will influence one another.

A simple before-and-after measurement is often useful for seeing whether something changes, but insufficient for establishing why it changed. Perhaps a busy internship period began in the meantime, management changed the appraisal system or an external event occurred. Precisely for this reason, guidance on complex interventions emphasises that evaluation is not only a question of effects, but must also consider implementation and context [3].

A caution for teams: if the intervention is applied at team level, you must not treat all employees as statistically fully independent. Cluster research requires an appropriate design, enough clusters and an analysis that accounts for clustering [4]. Comparing two teams side by side is usually an informative pilot, but not a robust causal effect test.

A good field test has a brief protocol. It need not be an academic tome, but it must document the decisive choices before implementation.

1. Primary outcome: which behaviour counts as success, for whom and at what point?
2. Comparison: what does the control group receive, and at what level does allocation take place?
3. Baseline: how often does the behaviour currently occur, measured in the same way in every group?
4. Minimum relevant effect: what improvement is worthwhile for the organisation, irrespective of a p-value?
5. Sample and measurement duration: how many observations and, for team-based tests, how many teams are needed to estimate that effect with reasonable precision?
6. Analysis: will you calculate percentage points, counts, repetition or time to first action; how will you handle missing data?
7. Unintended effects: does workload increase, quality decline or a group become disadvantaged?
8. Implementation: how will you check whether the intervention was actually offered and received?

Documenting these choices in advance matters. Choosing only afterwards which of ten measurements was the “success measure” increases the risk of a misleading positive story. A pilot can first establish that measurements and processes are feasible. A subsequent effectiveness test should follow a documented plan as closely as possible. The TIDieR guideline helps describe the intervention itself precisely enough for replication [5].

If you measure only behaviour, you may know that something works but not yet why. If you measure only an association, you may learn something about meaning but not whether the chosen intervention changes behaviour. A three-part approach is therefore useful.

Depending on the question, additional methods may be considered, including association measures, eye tracking, EEG or real-world observation. Their suitability and interpretation require method-specific evidence and an appropriate protocol; the sources cited here do not independently validate these applications. None replaces a valid measure of the target behaviour and a credible comparison when estimating a behavioural effect.

The following case is entirely fictional and intended to illustrate the study design. It presents no research findings from Neurofactor or a client.

An organisation has enough experienced employees to supervise interns, but too few employees volunteer. Preliminary research repeatedly reveals the same tension: “I think it matters” alongside “I have no room for it”. The team has two possible explanations: the role feels insufficiently valued, or scheduling makes the step too difficult.

Rather than developing only a campaign, the organisation chooses one initial intervention: a fixed, visible supervision slot in the existing work schedule, combined with clear registration instructions. The control condition retains the existing schedule and information. If entire teams share the scheduling change, team-level allocation makes sense, provided there are enough comparable teams and the analysis is designed accordingly. The researcher responsible checks feasibility and the required sample size in advance.

Predefined primary outcome: what proportion of eligible, invited employees has completed one supervision session within eight weeks? Secondary outcomes: registrations, cancellations, repeated behaviour, perceived workload and associations with supervision. Implementation check: were the reserved hours actually available, or were they filled with other work after all?

Suppose, purely for illustration, that 18 out of 100 employees in the control condition complete a first supervision session, compared with 25 out of 100 in the intervention condition. The unadjusted difference is then 7 percentage points (18% versus 25%). That is not the same as saying “39% more employees” without context, and certainly not automatically a proven effect. Drawing a conclusion requires information on allocation, sample size, cluster structure, uncertainty and any missing records. These numbers are invented to show how to interpret the outcome.

The better question after that test is not whether the new approach was popular. It is whether the approach generated enough additional completed supervision sessions to justify the scheduling, resources and any unintended effects involved.

An associative measurement and a behavioural measurement can complement one another, but they need not move in the same direction. It helps to anticipate four outcomes.

Even the first combination does not prove that the changed association caused the behaviour. That requires further investigation of the causal pathway. Nor is a null result proof that an intervention never works: the test may have been too small, too short or poorly implemented.

Alongside the outcome, always present the absolute change, the uncertainty and the practical significance. When conversion rises from 18% to 25%, the absolute difference is 7 percentage points. The relative difference is approximately 39%, but sounds larger without stating the starting point. Both figures therefore belong together, with the counts behind them and preferably a 95% confidence interval.

A confidence interval indicates how precisely the effect size is estimated under the chosen model. An interval compatible with both a relevant negative and a relevant positive effect is no basis for an emphatic success claim. Equally, a statistically detectable but tiny improvement does not automatically make the investment worthwhile. Agree before the test what constitutes a worthwhile effect and calculate the feasible measurement sample size accordingly.

Also examine who did and did not take part. Was the effect visible only among enthusiastic volunteers? Was the hard-to-reach group reached at all? Did recording procedures change halfway through? Were there additional costs, waiting times or quality problems? An intervention can improve a primary KPI while creating new resistance. That outcome must inform the decision.

You cannot hold every factor constant in the field. That makes it all the more important to make those factors visible. Watch for at least these pitfalls:

• Simultaneous changes in circumstances: a new manager, a campaign or seasonal workload coincides with the intervention.
• Contamination between groups: employees in the control group adopt the new schedule.
• Selection: only the most motivated employees participate in measurement.
• Incomplete implementation: the measure exists on paper but is not applied consistently.
• Changing records: after the intervention, the organisation counts registrations instead of completed sessions.
• A premature conclusion: the first week looks different, but the old pattern returns after a few weeks.
• Too many post-hoc subgroups: you keep searching until a strikingly positive difference appears somewhere.

Prevent these errors by documenting the context, keeping existing procedures stable where possible and recording every deviation from the protocol. Alongside whether an intervention works, investigate how it was delivered and for whom it worked [3, 5, 6].

Additional measurements may help investigate a proposed mechanism, but their interpretation depends on the instrument, task and study design. A difference in an association score, gaze pattern or EEG response does not by itself establish resistance, trust, mental effort or the cause of a behavioural effect. These interpretations require separate validation and testing.

The cited research on implicit measures concerns the instruments and outcomes studied [1, 7]; it does not validate EEG, eye tracking or every association instrument used in practice. Use such measurements only where justified by the question and relevant method-specific evidence, and report mechanistic interpretations as hypotheses unless the design supports a stronger conclusion.

For many organisations, the appropriate sequence is therefore: first define the behaviour, then investigate relevant associations and friction, then design a targeted intervention, and finally check in the field whether actual behaviour changes. No instrument automatically qualifies as conclusive evidence.

1. Describe the decision point. Who needs to perform which action, where, and in what choice context?
2. Document the associative tension. Which reward attracts, which meaning or fear holds people back, and what remains uncertain?
3. Choose one initial intervention. Describe precisely what changes and what else stays the same.
4. Specify the outcome and comparison. Define the primary behavioural measure and determine the control group or best available counterfactual comparison.
5. Prepare implementation. Check privacy, consent where relevant, capacity, recording, randomisation and measurement accuracy.
6. Measure and analyse according to plan. Report counts, absolute effects, uncertainty, implementation, associations and unintended effects.
7. Make a decision and test again. Scale up when evidence is sufficient; adapt when an effect is plausible but incomplete; do not scale up if results, safety or feasibility are inadequate.

A field test is not intended to confirm the original idea. It is intended to enable the organisation to make a better choice, even when the original idea is wrong.

An intervention must respect the interests and autonomy of those involved. This is especially important for employees and other groups where power relationships may be unequal. Make the behavioural objective and study design appropriately transparent, avoid undue pressure, provide reasonable alternatives and assess negative side effects.

When personal data are processed, the GDPR requires a valid lawful basis, purpose limitation, data minimisation, appropriate information and security. Consent to participate in research is not automatically the same as the GDPR lawful basis for processing data; seek legal review of that assessment where necessary. Do not process associative, behavioural or neural data more broadly than necessary, and preferably report at aggregate level [8].

Source [8] explains lawful bases. The other obligations mentioned here follow from the GDPR itself, in particular Articles 5, 6, 12–14 and 32, and Articles 9 and 89 where applicable. Consent in employment relationships requires particular care because of the imbalance of power (Recital 43).

A behaviour change intervention can produce a convincing story without a convincing effect. The reverse is also possible: a quiet process adjustment changes behaviour, while barely anything shifts in what people say or associate. The task is therefore not to declare one measurement the winner, but to put the layers in their proper place.

Association research can help investigate what a choice means to people. A well-designed field test can estimate whether an intervention changes behaviour compared with a credible alternative, even without an association measure. Combining the two can add insight into possible mechanisms, but a changed association alone does not establish why behaviour changed.

• Why people believe something is sensible but still do not do it – on the gap between intention and behaviour.
• Communication, process or environment: which behaviour change intervention should you choose? – on diagnosis before intervention.
• Associations in behaviour change – the overview page for the subseries.
• Neurofactor research methods – on association research, real-world measurement and complementary methods.