Source Paper
Behaviour change techniques: the development and evaluation of a taxonomic method for reporting and describing behaviour change interventions (a suite of five studies involving consensus methods, randomised controlled trials and analysis of qualitative data)
Susan Michie, Caroline E Wood, Marie Johnston, Charles Abraham, Jill J Francis et al.
Health Technology Assessment • 2015
Taxonomy Coding Reliability and Validity Testing
Objective: Assessment of taxonomy reliability through intercoder and test-retest measures and validity assessment with 40 trained coders for coding interventions using the Behaviour Change Technique Taxonomy version 1 (BCTTv1)
Protocol Steps
Coder Training
Participants received training in use of the taxonomy through two methods: 1-day workshops and distance group tutorials
Note: Training was provided to 161 participants total, with 40 of these trained coders subsequently evaluated for reliability and validity
View evidence from paper
“training in use of the taxonomy (1-day workshops and distance group tutorials) (n = 161) was evaluated by changes in intercoder reliability and validity”
Intercoder Reliability Assessment
Assessment of agreement between multiple coders when coding the same interventions using BCTTv1
Note: Good intercoder reliability was observed for 80 of the 93 BCTs
View evidence from paper
“evaluating the taxonomy for coding interventions was assessed by reliability (intercoder; test-retest) and validity (n = 40 trained coders)”
Test-Retest Reliability Assessment
Assessment of within-coder agreement by having coders code the same interventions at different time points
Note: Good within-coder agreement was observed after 1 month with statistical significance p < 0.001
View evidence from paper
“Good within-coder agreement was observed after 1 month (p < 0.001)”
Validity Assessment
Assessment of validity through agreement with expert consensus for coded interventions
Note: Validity was assessed for 40 trained coders. Both training methods improved validity (both p < 0.05)
View evidence from paper
“validity (agreement with expert consensus); evaluating the taxonomy for coding interventions was assessed by reliability (intercoder; test-retest) and validity (n = 40 trained coders)”
Competence and Confidence Evaluation
Evaluation of changes in coder competence and confidence in identifying BCTs following training
Note: Training doubled the proportion of coders achieving competence and improved confidence in identifying BCTs in workshops (both p < 0.001)
View evidence from paper
“Both training methods improved validity (both p < 0.05), doubled the proportion of coders achieving competence and improved confidence in identifying BCTs in workshops (both p < 0.001)”