Introduction: When creating and administering educational tests, reliability and validity evidence needs to be accumulated to ensure appropriate interpretation of test scores.

1. A study from 2013 demonstrated that pharmacy and medical literature does not fully document reliability and validity evidence for educational testing.

2. The goal of the study was to determine how practices have changed since 2013 and evaluate the presence of reliability and validity evidence for educational testing described in pharmacy education literature.  

Methods: Articles published between 2019-2021 in five major pharmacy education journals were analyzed.  Included articles presented the use of educational testing of students’ knowledge, skills, and/or abilities.  Two reviewers independently coded the selected articles.  Consensus was reached for any discrepancies in coding.  Articles were reviewed for reliability and validity evidence sources, including content, response process, internal structure, relation to other variables, and consequences.   Results: Of 1467 journal articles, 326 (22.2%) described the use of educational testing.  Of those, 64.1% described testing of knowledge.  Reliability was described for only 19% of the articles.  The most often described source of validity evidence was content, at 53.6% of articles.  The least described sources of validity evidence were internal structure, other than reliability, described only in 3 papers (0.9%) and response process in 12 papers (3.7%).  One third (32.8%, 107/326) of the articles did not mention any sources of validity evidence or reliability for student assessments.    

Conclusions: Validity evidence is necessary for appropriate interpretation of student assessment scores.  Without adequate validity evidence, assessments in the literature are of no use to other investigators.  It is imperative that educators understand the importance of validity evidence and report it in the literature.