Saddle Research Trust

A Horse Owner’s Guide to Interpreting Research Evidence 

Part 2: Bias, Blinding, and the Placebo Effect



Author: Tracy Bye

In the first instalment of this series we reviewed some of the basic principles that scientists use when designing an intervention-based study. We talked about how we can test the effects of an intervention such as an item of tack or a supplement, and we discussed the importance of having a ‘control group’ which doesn’t have the intervention, so that we have something to compare the intervention group to.

Another important consideration at this point is the placebo effect. The placebo effect is the observed phenomenon of inert or ‘sham’ treatments having a therapeutic effect. As the treatment itself cannot possibly be effective, this is attributed to the psychological effect of the patient believing that they have had a treatment. Obviously if a horse is taking part in a research study, they do not know that they have had an intervention or treatment. However, the caregiver placebo effect has been reported in studies of domesticated animals. This is where owners report an improvement in symptoms in animals who are not receiving treatment. An example of this can be seen in headshaking study, where 32 horses were given either a supplement developed to alleviate headshaking or an inert placebo in a crossover design [1]. Owners were not told which condition their horse was in, and they were asked to fill out a questionnaire about their horse’s headshaking behaviours. Owners reported improvement in headshaking on both the supplement and the placebo, whereas professional assessment of headshaking symptoms showed no significant improvement in either condition. Similarly, a study into osteoarthritis treatment in cats also found a significant improvement in caregiver ratings for mobility and activity for animals on a placebo treatment with no active ingredients [2]. The caregiver placebo effect is one reason why evidence based on owners or riders’ observations of their horses, such as product testimonials, is not as useful as the results of well-designed scientific studies. 

Photo: Eva Tomankova on Unsplash

Owner reported results are not usually considered as outcome measures in intervention studies, but how owners or riders interact with their horses could have an effect on other outcome measures. The is known as placebo by proxy effect. This idea was originally described in the context of humans taking part in drug trials [4]. The authors explain that if a patient’s doctors and family members feel empowered and optimistic about a disease, the patient’s environment can become less stressful and more supportive, which can lead to changes in the patient’s own mental state. The placebo by proxy effect has yet to be thoroughly investigated in horses, but it is reasonable to assume that it would be present in animals in the same way as seen in humans, and to try and control for it. For example, if a rider believes that their usually tense horse has received a calming supplement, they may be more relaxed when riding them, leading to the horse displaying less of the tension associated behaviours which are being used as the outcome measure.

To avoid the placebo by proxy effect impacting on our research, the best approach is to make sure the control group take part in the same testing process as the intervention group, but based on a placebo or sham condition, so they would be the placebo group. The important thing here is that all conditions look the same to the caregiver or rider of the horse involved. This is referred to as blinding. So, in the example of the calming supplement study, the horses would receive the real supplement in the intervention condition, but in the control condition they would receive a placebo which looks the same as the real supplement but has no active ingredients. When the person that is involved with riding or caring for the horses is unaware of which condition is which, this is known as a single-blind study. This means the caregiver cannot know whether their horse is getting the treatment or the placebo, so if the placebo by proxy effect is present it should be the same for both groups, and the difference between the groups is the true effect of the intervention.

The students scored more positive and fewer negative behaviours when told that the hens were from an organic instead of a conventional farm, even though the videos were the same [4]. The students will likely have had a more positive view of organic farming compared to conventional farming, which led to them unconsciously looking for more positive welfare and behaviours in the ‘organic’ videos and more negative welfare and behaviours in the ‘conventional’ videos. 

To alleviate the potential impact of this observer bias we can take the blinding one step further and also make sure that the researchers who are testing for the outcome measure don’t know which condition is which. This is then referred to as a double-blind study. Double blinding can be achieved by having an independent person randomly allocate the horses to the intervention and control (placebo) conditions. When the researcher is conducting their assessment, they do not know which horses are in which condition, so the possible effect of observer bias is removed. The independent person keeps a list of which horses were in which condition, which they provide to the researchers after the study is over so that the results can be grouped and analysed. In a truly randomised study, this sorting into two groups is done in a way that is unsystematic and unstructured, so that neither the caregivers nor the researchers would not be able to work out which horses were in which group. The randomised, double-blind, placebo-controlled trial is therefore considered the gold standard for intervention-based research studies, and this is the approach used in human medical research to investigate the effectiveness of new treatments.

There are some types of study where it is not possible for a rider, caregiver, or researcher to be blinded as to which condition a horse is in, such as when testing new items of tack for example, especially when there is a clearly visible difference between the two conditions. If this is the case, researchers will usually think very carefully about the type of outcome measures they use, making sure to use objective measures (e.g. biomechanical analysis) as opposed to subjective measures (e.g. behavioural observations).

In the next instalment we will investigate these ideas a little more, as we delve into the different ways we can measure the outcome of a research study, and how we can ensure that the results are valid and reliable. 

References

1.    Talbot WA, Pinchbeck GL, Knottenbelt DC, Graham H, McKane SA. A randomised, blinded, crossover study to assess the efficacy of a feed supplement in alleviating the clinical signs of headshaking in 32 horses. Equine Veterinary Journal. 2013 May;45(3):293-7.

2.    Gruen ME, Dorman DC, Lascelles BD. Caregiver placebo effect in analgesic clinical trials for cats with naturally occurring degenerative joint disease‐associated pain. Veterinary Record. 2017 May;180(19):473-.

3.    Grelotti DJ, Kaptchuk TJ. Placebo by proxy. BMJ. 2011 Aug 11;343.

4.    Tuyttens FA, de Graaf S, Heerkens JL, Jacobs L, Nalon E, Ott S, Stadig L, Van Laer E, Ampe B. Observer bias in animal behaviour research: can we believe what we score, if we score what we believe?. Animal Behaviour. 2014 Apr 1;90:273-80.

A new study identifies links between conflict behaviours in Dressage competitions and performance evaluation by judges

Author: Jess Irving

The 2019 BETA market survey identified 27 million people within the UK are interested in the equestrian industry, an industry itself which is worth £4.3 billion, and appears to be increasing. Approximately 26% of equestrian sport participants have competed within the last year, supporting the range and number of performance horses within the country.

Whilst animal-centred sports remain a point of debate within the UK public, governing bodies such as the International Equestrian Federation (FEI) promote training which supports equine welfare. Despite this, there is a risk that young horses may be pushed through training at a young age, in addition to certain training aids, including incorrectly fitted nosebands and strong bits, being utilised alongside poor riding or training methods. Unwanted or agonistic behaviours – often referred to as ‘conflict behaviours’ - in the ridden horse may result from conflicting or confusing signals from the rider, for example leg and hand pressure utilised simultaneously.

Equine conflict behaviours include tail swishing, head movements, incorrect head position and tension in the muscles surrounding the mouth, with the most widely recognised agonistic response being the head position behind the vertical (Kienapfel et al., 2014). Whilst the FEI promotes the head position being slightly in front of the vertical, over the last decade there has been an increase in the number of top performance horses, particularly within dressage, being ridden with the nasal plane behind the vertical, with reports of these horses being awarded higher dressage marks by judges (Lashley et al., 2014).

The lack of precise guidelines in dressage marking have appeared to result in high variability in marks and judging, even at Olympic level, however it is clear that future developments of a more objective system should reward riders demonstrating optimum training methods, with consideration of conflict and discomfort within the dyad. The current study, run by researchers at the University of Edinburgh utilising video footage from British Dressage affiliated competitions, sought to identify whether judge marking within dressage competitions at Preliminary, Novice and Elementary level competitions considered conflict behaviours and horse antagonistic responses.

Seventy-five horse/rider combinations were included in the final study, with 447 movements in total analysed. Video footage of individual tests were analysed using blind testing (where the analyser was not aware of the final test mark) for horse behavioural responses using previously developed descriptions of conflict behaviour (Gorecka-Bruzda et al., 2015) and pain-related behaviour (Hall et al., 2014; Dyson et al., 2018), and these were matched against judge marks. Two or more conflict behaviours were identified within 83% of the total movements analysed – with head position of horses either behind or on the vertical gaining significantly higher judge marks than those in front of the vertical. Despite this, horse whole body movements indicative of conflict did receive lower dressage marks by judges and additionally horses with ears held forwards were awarded higher dressage marks. Interestingly, the type of noseband use was associated with tail and mouth behaviours, and head position. Flash nosebands were associated with increased tail behaviours, grackle nosebands were associated with increased nasal plane angle behaviour, and increased mouth behaviour scores were most often noted with grackle or Micklem nosebands. Level of competition and specific movements additionally impacted the presence of antagonistic behaviours, with increased behavioural scores at Novice level compared to Prelim level, and at working canter.

The study suggested that whilst there was no association between overall presence of conflict behaviours and judge marks, when individual categories of behavioural responses were considered, it was clear that judges were accounting for some, but not all, conflict behaviours during test riding. Mouth, tail and poll to wither behaviour were not associated with judge marks, whereas ear movement, head position and whole body behaviour were, suggesting judges may be more attuned to certain behaviours over others.

Photo: Sue Dyson

It is clear from the results of this study that dressage performance within competitions is currently not being assessed in accordance with FEI guidelines, with certain conflict behaviours being rewarded and many being ignored. The development of a more objective and encompassing judging criteria may mitigate some of these risks, however this would need to be supported by more comprehensive training and professional development of judges. To ensure the perseverance and sustainability of the equestrian performance industry, industry bodies, stakeholders and performance judges should be championing and rewarding those riders who base their training and competitive performances around the welfare of the horse.

References



1. Kienapfel K, Link Y, König v. Borstel U. Prevalence of Different Head-Neck Positions in Horses Shown at Dressage Competitions and Their Relation to Conflict Behaviour and Performance Marks. PLoS ONE. 2014;9(8):e103140. doi:10.1371/journal.pone.0103140
2. Lashley MJJO, Nauwelaerts S, Vernooij JCM, Back W, Clayton HM. (2014). Comparison of the head and neck position of elite dressage horses during top-level competitions in 1992 versus 2008. Vet J. 2014;202(3):462–465. doi:10.1016/j.tvjl.2014.08.028  
3. Górecka-Bruzda A, Kosińska I, Jaworski Z, Jezierski T, Murphy J. (2015). Conflict behavior in elite show jumping and dressage horses. J Vet Beh: Clin App Res. 2015;10(2):137–146. doi:10.1016/j.jveb.2014.10.004 
4. Hall C, Kay R, Yarnell K. (2014). Assessing ridden horse behavior: Professional judgment and physiological measures. J Vet Beh: Clin App Res. 2014;9(1):22-29. doi:10.1016/j.jveb.2013.09.005 
5. Dyson S, Berger J, Ellis AD, Mullard J. (2018). Development of an ethogram for a pain scoring system in ridden horses and its application to determine the presence of musculoskeletal pain. J Vet Beh. 2018;23:47–57. doi:10.1016/j.jveb.2017.10.008 

Share YOUR success stories with us!

Continuing on from the SRT 'Happy Athlete' feature, we'd love to see examples of your horses enjoying their ridden careers. Whether that be happy hacking or competing at the upper echelons of the sport, help us to spread examples of the vast numbers of good equine role models at all levels and aspire to proliferate a new normal of happy, healthy horses in the media!

This month we have Katie Hough and her horse, Vistozo, showing a lovely overall picture; moving well with an attentive yet soft expression, working into a light contact. There is a small degree of body lean and head tilt but they illustrate a harmonious partnership.

Photo credit: Shuttr - Photography by Tom Helme.

Share to your socials using the tag #SRTHappyAthlete or email to [email protected] 

Saddle Research Trust

[email protected]

www.saddleresearchtrust.com