Saddle Research Trust

A Horse Owner’s Guide to Interpreting Research Evidence

Part 3: Measurement, Validity, and Reliability

~ Tracy Bye MSc PhD(c)

In the first two instalments of this series we introduced some of the things researchers consider when they are designing a study, such as how to avoid pitfalls like individual differences, unconscious bias, or the placebo effect impacting on the results. One of the most important things to consider is what we are going to measure, our ‘outcome’ variable.

We can group outcome variables into two broad categories, quantitative, which are numerical values, such as heart rate, angle of a joint, or dressage test score, and qualitative, which are descriptions, such as comments from a dressage judge, or answers in an interview. Qualitative measures are generally subjective, meaning they are based on someone’s opinion and open to interpretation. This doesn’t mean they are an inferior way of conducting research, it just means that the researchers need to be really careful about how they interpret qualitative data and must go through the data systematically so they don’t miss anything. Ideally, more than one person will review and interpret the data and then they will come to an agreement on the main messages. A systematic approach to analysing qualitative data is really important to avoid ‘cherry picking’ the bits which support the researchers’ hypothesis.

Quantitative data can be either subjective or objective, depending on how it is derived. An example of subjective quantitative data is a dressage test score, which is a numerical value. It   is not open to different interpretations like the qualitative data, but it is still based on someone’s opinion, and if a different person did it then it might be different. Objective quantitative measures are values which are not open to interpretation, such as heart rate or joint angles. Often these are measured by some type of equipment to limit the effect of human error, for example using a heart rate monitor rather than taking a pulse. 

Figure 1: Head-neck angle is an example of an objective, quantitative outcome variable. If you were to describe this horse’s behaviour then that would be subjective and qualitative. If you were to give a behaviour score based on an agreed scale that would be subjective and quantitative.  

Photo from Robinson and Bye (2021).

When researchers are deciding on what the outcome variable is going to be, they will also think about the validity and reliability of this measure. Validity is how well the outcome measure actually represents the thing we want it to measure. So, if we want to measure stress, for example, we cannot just ask a horse how stressed it is, but we can measure its behaviour, and physiological things that have been linked with stress, such as heart rate variability [1], blink rate [2] or eye temperature [3] (check out the references if you want to learn more about how these different measures work). A study with good validity controls any other factors which could affect the outcome variable other than the intervention. So, if we wanted to use heart rate variability as the outcome measure to look at the effect of a calming supplement on stress levels, we would control for anything else that could affect heart rate variability, such as exercise.

A study with good reliability makes sure that the measures are taken in the same way each time, so they should always yield the same results. They do this by having a strict experimental protocol to standardise the intervention and the measurement, and making sure to follow it exactly with every horse every time they measure it. This means the only thing that changes is the intervention, and any differences seen can be attributed to that intervention, and not the chance that something has been done differently. 

Photo by Kenny Eliason on Unsplash

You may have already picked up on a problem with all of this standardisation. How do we know that these interventions will work in the real world, with horses that are not in strict experimental conditions? This is the problem of external validity, or how well a study represents the real world. The type of validity we have been talking about so far is also termed internal validity. Internal and external validity are on a continuum, so a study cannot be high in both. The more controlled a situation is, the less true to life it is, and vice versa.

Usually when scientists trial new interventions, we want the internal validity and reliability to be as high as possible, so we will test things in a very controlled situation to start with. For example, we may test a new training aid on a sample of ex-racehorses, all of a similar age, conformation, and training background, working on a treadmill all going at the same speed. This scenario has high internal validity, as any differences we see in the horses’ biomechanics are very likely to be due to the training aid, and not a change in speed or surface, and a homogenous sample like this is less likely to show differing effects based on how the training aid interacts with the horse’s conformation or posture, or to elicit different effects based on how the horse was previously trained. Once something has been tested in a very controlled situation like this, then researchers will tend to try out different and more varied situations to see what happens when specific differences are added in. So, if you are looking at research on a new intervention, you may see different studies looking at it in different situations, with different levels of standardisation in the methodology. This allows us to build up a picture of how the intervention might work in a range of scenarios, but without compromising the internal validity and reliability needed in the early stages of researching a new intervention.



It is easy to understand how we can make objective, quantitative studies valid and reliable, but this needs a little more thinking about when you are considering qualitative data. We have talked a little here about subjective, qualitative data, and observation as a scientific approach, but there is a lot more we can learn about how to do this type of research. Horsemen were using observation to work out how to manage horses for centuries before the scientific method was even developed. In the next instalment in this series we will be talking about where observation turns into observational research, and what we can learn from this. 

In Part 1 of The Gait Analysis Revolution, we looked at the different technologies currently available for examining equine locomotion, as presented by Dr. Serra Bragança at the Saddle Research Trust’s 4th International Conference in 2021. These technologies included the use of IMUs and Mocap for kinematic analyses and sEMG for capturing muscle function data. Here in Part 2, you can find a summary of some exciting research studies supported by these technologies. A number of these projects were ongoing at the time of the conference but have since been published with the findings briefly described here. 

Variation in gait parameters used for objective lameness assessment

Dr. Serra Bragança’s research team undertook an interesting study where kinematic data was collected using the Qualisys motion capture system (Hardeman et al., 2019). The purpose of the study was to see if a sound horse will repeatably move in a symmetrical pattern over time (within variation) and whether there is much variation in this movement symmetry between different horses (between variation). 

Each line represents data from an individual horse for measurements taken on Day 1 (pink), Day 2 (green) and Recheck day (blue). A predicted interval was calculated whereby 95% of the measurements were contained within the upper and lower interval boundaries and from this data range reference values were calculated.

Essentially, for sound horses, between horse variation is much larger than within horse variation, i.e., there is quite a large difference between horses but the same horse itself, has quite a consistent pattern of motion symmetry over time. The researchers were also able to deduce the normal expected variation in movements for sound horses. So, for certain parameters of motion symmetry, it is possible to provide reference values for lameness assessment in a clinical setting. The concept is similar to comparing blood work results to standardised reference values. As Dr. Serra Bragança explained, “We know that if the measurement for our horse is within these reference values, it's considered normal, outside these reference values might be considered abnormal in light of the rest of the clinical examination.”

Timing of Vertical Movements Relative to Footfalls in Different Equine Gaits and Breeds



We know that motion symmetry of the head and pelvis are used to measure lameness in horses at trot and these movements are now well-described. However, further examination of the temporal relationships (relative timing) between normal movements is required to better understand how they change with lameness. 

The study he was referring to, published last year, involved the collection of data from Icelandic, Warmblood and Iberian horses, using IMUs and the EquiMoves system (Rhodin et al., 2022). Limb movements, together with vertical movements and lowest/highest positions of the head, withers and pelvis were calculated, and the relative timing of the events was compared across the breeds.

The study revealed clear differences in both limb and upper body kinematics between the three breeds of horse examined. Compared to Warmblood and Iberian horses, Icelandic horses showed a smaller vertical range of motion of the head, withers and pelvis at trot. This, in combination with higher stride frequency, may partly explain why lameness evaluation in this breed is regarded as so challenging. Further to the Hardeman et al. (2019) study above, which looked to determine reference kinematic values for objective lameness assessment, this study also demonstrated a first step to establishing breed-specific reference values to further standardise gait analysis techniques in the near future.

Adaptations in muscle activity and movement occur during induced lameness

When horses are lame, they attempt to reduce their pain by redistributing load from the painful limb onto the non-lame limb or limbs. This is a compensatory movement resulting from altered or adaptive muscle activity controlling the limb movement. However, our understanding of these neuromuscular adaptations to lameness is limited. Dr. Serra Bragança and his team undertook a study to help fill this gap in knowledge, where both kinematic and sEMG technologies were simultaneously employed to evaluate motion and muscle function respectively (Saint George et al., 2022). 

As the team predicted, muscle adaptations identified by sEMG do occur when lameness is induced and result in altered movement patterns in a horse compared to its sound condition. Increases in the muscle activation signal were observed bilaterally in all muscles examined, except for the forelimb muscles on the non-lame side where muscle activation was decreased. Phasic shifts in the muscle activation patterns were also observed, that is, the time point of the stride when the muscle fires. This reflects the known compensatory movement patterns for reducing weightbearing on the lame limb which horses use to avoid pain.

These findings demonstrate that we can objectively quantify underlying neuromuscular adaptations to lameness which would be impossible to do through human observation alone. With further development, the techniques described here could be optimised to improve future diagnostic and treatments methods.

Prediction of kinetic parameters from IMU data using recurrent artificial neural networks

Dr. Serra Bragança gave an inspiring overview of the concept of Machine Learning (ML) and how it is supporting this field of research by helping us understand and interpret the complex signals and data generated from gait analysis technologies. Simply put, ML is a form of artificial intelligence which allows computers to become better at predicting outcomes. The algorithms used in ML take historical or previously generated data as input to predict new output values.

For example, here an artificial neural network (ANN) is being asked to classify a picture. The input layer is where each tiny pixel of the image is fed into the ANN. The hidden layer mimics the neurons of the brain connected by synapses and the output layer is the resultant classified image. You might require the ANN to classify the input pixels as a person, a bicycle or a car. The neural network is “trained” with data during which many images of people, bicycles and cars are shown so it “knows” how to classify them. The model gets penalised (loss function) for incorrect predictions until it gets closer and closer to the correct classification (reduction of loss function). Network models can also be trained to predict time series or the future of a signal for example with weather predictions or financial models.   

Just this year, Dr. Bragança’s research group published a paper on how ML was used to predict ground reaction forces (GRFs) based on inputting kinematic data and building recurrent neural networks (RNNs) (Parmentier et al., 2023). GRF measurements are used as the gold standard in assessment of weightbearing lameness, however the use of force plates for this type of data collection is cumbersome and not very practical in a clinical setting. 

The researchers were thus able to generate models which successfully obtained GRF curves based on the IMU data. The research also showed an interconnection between kinetics and kinematics and that upper-body movements have a greater influence than limb movements on GRF. This study shows us that kinetic data can be generated using ML in a more user-friendly manner than with force plates, with the added advantage of this technology being transferrable to field-studies. This gives great promise as a valuable asset for equine orthopaedics.

Subjective and objective evaluations of horses for fit-to-compete judgement



As has been established, there is ample evidence to relate lameness in the horse to certain parameters of motion symmetry. This evidence supports the use of quantitative gait analysis in a clinical setting but what about in a competitive setting? Competing horses at FEI events must undergo veterinary inspection to determine whether they are ‘fit-to-compete’. However, subjective assessment of gait is not infallible and judges can sometimes have controversial differing opinions. In this study, Dr. Serra Bragança and his colleagues evaluated intra- (within) and inter- (between) observer agreements of fit-to-compete judgement and compared these with objective gait analysis measurements (Serra Bragança et al., 2020).

As evidenced by the heat map from a live rating, the 4 judges were unanimous in their assessment of most horses, however their agreement is far from perfect. In fact, as the second image shows, the inter observer agreement, that is the agreement between different judges, was only “fair” with a kappa value of 0.39 or 0.53, depending on whether the horse was wearing markers for the motion capture. The kappa value is a statistical measure where 0 is rated as poor and 1 is perfect agreement. The intra observer agreement however, that is, when the same vet judges a horse again, was quite high (0.8 – 1.0). The sensitivity of the kinematic data (head and pelvis symmetry) against a consensus of the 4 judges as a reference was 83.3%. 

Even though fit-to-compete judgements are quite complex matters for which lameness assessment is just one aspect, these findings beg the question, “Should we be using objective measurement of horses’ movements to support the subjective veterinary assessments and help improve decision making?”

The effect of water on kinematics

Dr. Serra Bragança also mentioned two very interesting studies his research team were working on which give an idea of the scope of research for which we can apply these gait analysis technologies. This work has not yet been published. The first looks at the effect of water on kinematics of the horse. 

While treadmills and water treadmills are useful in the research setting, they are also now commonplace in competition yards and therapy centres given their usefulness for provision of controlled exercise with reduced load. This research shows how this kind of technology could be useful in tailoring training or rehabilitation protocols to the needs of individual horses.

The effect of 3D printed horseshoes on kinematics and kinetics



The second study mentioned was research being carried out into the effect of 3D-printed horseshoes. Here, the veterinary school farriers at the University of Utrecht carried out 3D scans of horse hooves using a hand-held device and then used Computer Assisted Drawing (CAD) software to design and print the horseshoes. These shoes are tailor-made and unique to each individual hoof of a horse. Serra Bragança’s team then employed both kinetic and kinematic methods to assess the effect these shoes have on the horses wearing them. Longitudinal monitoring was carried out using force plates to measure GRFs and movement analysis using IMUs (EquiMoves). 

reduced with the 3D shoe indicating a better distribution of pressure with the 3D shoe.  

Although we cannot yet access the full complement of data from this research, it is apparent that the value of the 3D horseshoe is in its configuration to fit a horse’s hoof exactly, considering all specific shapes and angles. Additionally, as one of the farriers involved was quoted in saying, “The 3D printed shoe would allow us to integrate any additional (orthopaedic) features, such as a heel wedge, a wide branch extension, and that, again, with absolute precision.”

The depth and scope of the research discussed here, enabled by these incredible gait analysis technologies, has enhanced our knowledge and understanding immeasurably. Given recent events in the media and the public scrutiny of equestrian activities, the discussion around our social licence to operate is becoming even more of an urgent agenda. With the availability of these technologies, we can support ourselves with data and evidence when questioning whether horses are fit-to-compete or examining the suitability of training regimes. User-friendly and pragmatic technology which allows for the daily monitoring of our horses, is at our fingertips; it is no longer a far-fetched concept of the distant future.

Calling all students: SRT Scientific Article Competition

Could you write a research summary aimed at public engagement? Test your skills for a chance to win!

In your own words, you should summarise either your own research (a great chance to showcase your dissertation findings) or other recent research in the field of welfare or performance of the ridden horse. The winner will be picked from our short-listed entrants and will be featured in July's SRT Newsletter. Plus, you and a friend will get FREE admission to a SRT Equine Safeguarding seminar of your choice.

Submit 1,500 words by 5pm 9th June 2023 to enter.

T&Cs: Entries are welcome from students worldwide. By submitting your article you authorise the SRT to publish your work across the SRT newsletters, website and social media platforms. Articles submitted after the deadline will not be considered. The prize includes 2 admission tickets to the SRT Equine Safeguarding event of your choice and does not include transport to nor refreshments at the event.

Equestrian sports are increasingly being placed under scrutiny from the general public, with concerns from stakeholders around threats to Social Licences to Operate (SLO) and the impact public opinion may have on the future of many equestrian sports. Our previous review provided an overview of current research and opinion around social licencing within the UK and how this may impact those involved within the equestrian community.

Historically, equestrian sports such as horse racing were viewed as a national institution, with little emphasis placed on racetrack safety, horse wastage and fatalities, or horse welfare. Over the last century, increasing emphasis has been placed on improving animal welfare across farming, captive exotic animals, domestic animals, and both performance and pleasure horses. Subsequently, increasing attention has been placed on the risks associated with horse sports such as racing, and changes in public opinion on horse racing over the last decade is evidence of the adjusting attitudes towards horse fatalities on the racetrack. Unfortunately, information accessible to the general public around welfare of horses when in training, and post-racing careers of Thoroughbreds after retirement, varies considerably – with many members of the UK public still believing horses are ‘sent for meat’ after their racing careers are finished. Strong public opinions on horse racing, both positive and negative, in addition to the rise of social media results in the ability for unsubstantiated claims to go ‘viral’; being seen and digested by a global audience. This provides evidence of the risks to horse racing’s SLO within the UK, and across the globe.

Our earlier review highlighted a number of famous examples of the impact of negative media and public attention on equestrianism’s SLO – including the removal of equestrianism from the Olympic modern pentathlon after the 2024 Olympics. Highly publicised equestrian sports, particularly racing, will always be at higher scrutiny from the general public and are ultimately at the highest risk for having their SLO jeopardised.

The introduction of the Horseracing Integrity and Safety Act (HISA) in America is evidence of upcoming changes in equestrian sports. HISA encompasses required improvements in racetrack safety and stricter anti-doping practices across horse racing within the USA. The financial implications of the introduction of the new regulations are yet to be determined, but opinions within the horse racing community appear to be largely negative. However, high rates of injuries, breakdowns, and fatalities, in addition to widespread doping issues across the sport within the USA, suggest the introduction of a stringent measures for medication detection and improvements to racetrack safety should enhance equine welfare across American horse racing.

Following continued public outcry over the number of fatalities and injuries on the track, one of the highlights of the UK racing sector, the Aintree Grand National, run last weekend in Liverpool, underwent widespread changes in 2013. This included modified fence design to improve safety for horses and reductions in the number of runners allowed on the course each year. Modifications to the track have been met with conflicting opinions. Many racing fans and trainers claim it ‘is not the race it used to be’, however the number of fatalities on the course has largely reduced since 2013. Sadly, three fatalities occurred during the 175^th annual Grand National Festival this year, which has stimulated discussions across the public and animal rights campaigners. These include calls for bans on jump-racing, bans on whip use during racing, and the need for an independent regulatory body to oversee improved horse welfare during racing sports. It is clear that there remain significant threats to the UK racing industry’s SLO. Therefore, government intervention in the UK may be unavoidable unless further action is seen to be implemented by racetracks and the Horserace Betting Levy Body.

However, horse racing is not the only industry at risk of negative public opinion and threats to social licences. An overhauling of rules, including on the use of equipment, noseband use and fit, bans on hyperflexion and restrictions on whip use, has been recommended by a study group from the French National Assembly (the lower house of the French Parliament), ahead the 2024 Paris Olympics, with rules intended to cover all equestrian disciplines. The report has bypassed the Fédération Équestre Internationale (FEI) entirely, being issued directly to the Olympic Organising Committee. The report additionally recommends 24/7 surveillance of the Olympic grounds and an Independent Welfare Committee comprised of stewards and vets to be on-site throughout the competition. The Parliamentary involvement in the up-coming Olympics has been encouraged after public and media outrage following the 2021 Tokyo Olympics. This highlights the risk of negative publicity to equestrianism’s SLO, and the ensuing interventions which may occur from government bodies.

The German government was one of the first governmental bodies to introduce stricter guidelines around equestrian sports and to identify practices which could violate animal welfare. Whilst the guidelines were not passed as law, they may be upheld and determined legally valid in a court of law. Free jumping of foals and yearlings, the use of Röllkur, trimming of ear or nose hairs, and the use of draw reins were strongly disapproved of within the 2020 guidelines. These guidelines act as ‘expert advice’ for individuals within the equestrian community. However, with the increasing media attention around the 2024 Paris Olympics, and global outcry over horse racing, it is worth considering whether other European governments, who typically uphold stringent animal welfare laws, will implement similar guidelines for those involved in horse sports. Such implementation would ultimately cumulate in more restrictive prohibitions on ridden activities, equipment use and equestrian sports.

In light of the increasing involvement of governmental and regulatory bodies in the management of equestrian sports, continued negative public and media interest in equestrianism may impact the UK’s equestrian SLO, and the risk of potential restrictions on horse sports. Unless we want to find ourselves in a position where equine activities are highly regulated, or in certain circumstances, banned entirely, all stakeholders, owners, professionals, and riders must take an ethical, evidence-based, welfare-centred approach to horse training and competing. Our practices must be in line with current research, and all research conducted must be upheld to the highest welfare standards. We should be working to ensure equine physical and mental health is supported throughout all stages of care and within competition. Any issues should be addressed immediately, and transparency and positive communications with the general public and the media should be encouraged if we hope to avoid restrictions or bans on equestrian pursuits.  

Share YOUR success stories with us!

Continuing on from the SRT 'Happy Athlete' feature, we'd love to see examples of your horses enjoying their ridden careers. Whether that be happy hacking or competing at the upper echelons of the sport, help us to spread examples of the vast numbers of good equine role models at all levels and aspire to proliferate a new normal of happy, healthy horses in the media!

Saddle Research Trust

admin@saddleresearchtrust.com

www.saddleresearchtrust.com