Today's communication addresses a question from many PATH Intl. members. In this communication the topic is the credentialing council's data-informed decision to verify riding knowledge of PATH Intl. Certified Therapeutic Riding Instructors (CTRIs) as a prerequisite rather than in the form of an in-person riding test. The new process removes bias and subjectivity in the certification testing, a longstanding concern of the membership. Communication from the credentialing council about specifics regarding the prerequisites for the PATH Intl. CTRI certification, including how riding experience will be accounted for via the video demonstration of riding instruction and communication skills, will be sent in the next few weeks.
Background information:
The last time a formal review of the job responsibilities and related competencies of a therapeutic riding instructor was conducted was in the mid-90s for the NARHA Certified Riding Instructor credential. Small adjustments to the knowledge and skill level expectations were made over time based on general assumptions of how the job was perceived to have evolved. In order to continue the process of professionalization of the PATH Intl. Registered Therapeutic Riding Certification as requested by the membership, PATH Intl. recognized the need to initiate a cycle of systematic review to define the actual job responsibilities and competencies of current certified registered-level therapeutic riding instructors.
PATH Intl. selected an industry leader in testing and analytics, Alpine Testing Solutions,* to conduct the 2016 job task analysis (JTA). Alpine is widely recognized as a leader in psychometrics,** advanced psychometric analysis capabilities, test development expertise and CertMetrics™ Credential Management. The team at Alpine are experts in understanding the combination of test development, psychometrics, policy and business issues. The work that Alpine performed for PATH Intl. was the first important step in developing a valid, unbiased and objective certification process. Validity in this context is used for a very specific reason and has a very precise meaning. Validity in testing is the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.1
Alpine provided psychometric consultation and advanced psychometric analysis in the creation of the questions and distribution of the JTA survey. At the completion of the survey process, Alpine psychometricians conducted the statistical analysis of the data from which the JTA was created.
Led by the professional consultants at Alpine Testing Solutions, a group of seven PATH Intl. certified professionals from varied backgrounds and geographic locations were assembled to conduct the JTA. The purpose of the JTA was to define the primary areas of responsibility of a PATH Intl. Certified Registered Therapeutic Riding Instructor and the knowledge and skills pertaining to each area. A survey was then sent to the 6,158 PATH Intl. Certified Therapeutic Riding Instructors with current instructor memberships (as of January 1, 2016) to verify the areas of responsibility, knowledge and skills identified by the subject matter experts. The survey respondents were asked to identify:
- How often they personally performed each of the tasks as part of their expected job duties before, during or after lessons. It is essential to include these three time frame categories in order to capture and account for job expectations outside of the lesson setting.
- The level of risk for negative consequences to any party involved if the task was performed incorrectly or not at all.
The survey respondents were also asked:
- How often are you required to ride equines as part of your TRI job?
- Should a certain number of hours riding equines be required before a candidate may sit for the TRI certification and, if so, how many hours?
The PATH Intl. JTA measured frequency, criticality and percentage of job duties based on data reported by certified instructors to determine the importance and value of job tasks. Taking into account numerous demographic factors (age, level of certification, level of education, years of experience, employment status, geographic location and center size to name a few), the Alpine psychometrician*** conducted a statistical analysis to determine the importance of each individual task related to the overall job by combining the average frequency the job task is performed with the level of criticality that the task is performed correctly. This calculation also represents the weight of the task. The weights define how much of the certification exam needs to cover each area of the job to adequately evaluate the proficiency of candidates.
The subject matter experts identified four tasks specific to the activity of riding that were expected to comprise part of a registered instructor's job duties:
- Use basic skills to groom an equine
- Tack an equine safely and appropriately in preferred discipline
- Perform a walk, trot/jog, and canter/lope in each direction
- Demonstrate safe and appropriate arena etiquette
The data:
When asked to "indicate HOW OFTEN you (personally) perform the listed task as part of your expected job duties before, during, or after a lesson" (to capture and account for job expectations outside of the lesson setting), survey respondents replied:
When asked "If this task were performed INCORRECTLY or NOT PERFORMED AT ALL, what would be the risk of negative consequences to any party involved?" survey respondents replied:
When asked "How often are you required to ride equines as part of your TRI job?" survey respondents replied:
Every week
A few times a month
A few times a year
Never
|
Response Percent
38.7%
20.0%
11.5%
29.8%
|
When asked "Should a certain number of hours riding equines be required before a candidate may sit for the TRI certification?" survey respondents replied:
Yes
No
Yes and No |
Response Percent
74.0%
25.0%
1.0%
|
When asked how many hours of riding should be required, the median answer of survey respondents was 40 hours.
The survey responses to these last two questions in particular support verification of candidate riding knowledge as a prerequisite to certification testing.
The data analysis:
The determination of what to include in the actual test and what could be assessed through the prerequisites was made by professional analysis of the data provided by survey respondents. The survey indicates that nearly 30% of registered therapeutic riding instructors do not ride as part of their job while only 43% reported riding either before, during or after half or more of their lessons. Additionally, 76% of respondents identified slight or no risk of negative consequences if instructors riding performance was incorrect or they did not ride at all. In the field of psychometric analysis, it is considered statistically significant when 30% of the current workforce reports not performing a specific task as part of their job duties. Given that nearly 30% of instructors reported they do not ride as part of their job and 76% do not equate significant negative consequences to not riding, the Alpine psychometrician concluded that testing riding is not justified by the data. The psychometrician recognized that though riding may be part of the job for some instructors, the prevalence of riding as a job duty is not statistically high enough to warrant testing riding as an essential ability to conduct the job of therapeutic riding instructor, but is high enough that it warrants being a required prerequisite. When factored in with criticality, riding did not rank statistically as high in importance as other job duties and therefore the psychometrician determined it is not an essential component to be tested by PATH Intl. in this certification process. To evaluate Alpine's conclusion and the corresponding data through the lens of professionals in the field, PATH Intl. and Alpine convened an additional group of subject matter experts (SMEs) to discuss the topic. The SMEs concluded that while personal riding skill contributes to the ability to provide riding instruction, instructor riding is not a critical job task for delivering an effective therapeutic riding lesson. As an example, the group discussed how many top gymnastic coaches are rarely able to perform to the skills they teach, but nonetheless are fully capable of coaching students to achieve high-level skills. Additionally, the group discussed known professionals who cannot ride due to age, physical maladies, etc., but that fact would bear no impact on their ability to provide an excellent riding lesson. As a result, the SMEs determined that riding should not be assessed as a performance component and that candidates' knowledge of riding skills could be appropriately measured through prerequisite experience with riding theory assessed as part of the certification exam.
Composition of the certification exam:
Given the JTA data and the SMEs' findings, Alpine's psychometric analysis determined that the certification exam content should be comprised as follows:
In the past, understanding of the knowledge, skills and abilities of a PATH Intl. Certified Registered Therapeutic Riding Instructor has been nebulous and inconsistent. The 2016 JTA clearly set a specific bar for what can reasonably be expected. Additionally, given the processes the credentialing council has followed, PATH Intl. will have a statistically validated and legally defensible certification, which is an essential first step in elevating the professionalism of PATH Intl. credentials and the EAAT field. The more instructors who hold validated and accredited certifications, the better for the EAAT field. Accreditation of the certification has the potential to inspire more public confidence and expand funding opportunities. It is also important to note that the PATH Intl. CTRI certification is just the beginning of a very important evolution of PATH Intl. credentials. Ultimately, the credentialing council will determine which additional credentials might be appropriate to develop, allowing for additional verification and recognition of skills. The implementation of the PATH Intl. CTRI certification is just the beginning of additional opportunities for advancement and elevated professionalism.
As always, questions are welcome and appreciated as they help inform future communications. Please contact Bret Maceyak.
*Alpine Testing Solutions is a member of the Association of Test Publishers (ATP), Institute for Credentialing Excellence (ICE), and International Test Commission (ITC). Their employees are members of the National Council on Measurement in Education (NCME), American Educational Research Association (AERA), and Northern Rocky Mountain Educational Research Association (NRMERA). Alpine is also associated with the Council on Licensure, Enforcement and Regulation (CLEAR).
** Psychometrics is the field of science associated with the development of instruments (such as examinations) that measure knowledge, skills and attributes (KSAs).1
***A psychometrician is a professional who practices the science of measurement, or psychometrics. A psychometrician develops assessments such as examinations for educational, employment or professional credentialing purposes. Typically, a psychometrician has an advanced graduate degree from a university, usually either from an educational measurement program, from an industrial organization program or from a quantitative psychology program.2 Psychometricians measure the validity, reliability, and fairness of an exam program and are an integral part in the process of creating valid and reliable language tests.3
Job Task Analysis Subject Matter Experts:
Riding Discussion Subject Matter Experts:
Name
Sarah English
Katy Mead
Amy Vessa
Ainslie Kraeck
1 According to American Educational Research Association, American Psychological Association, National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing. Washington, DC
2 "What is a Psychometrician?" http://www.proftesting.com/test_topics/pdfs/psychometrician.pdf
3 "Psychometricians: What They Are and What They Do" https://www.altalang.com/beyond-words/psychometricians-what-they-are-and-what-they-do/
|
