Date of Graduation
5-2025
Document Type
Dissertation
Degree Name
Doctor of Philosophy in Educational Statistics and Research Methods (PhD)
Degree Level
Graduate
Department
Counseling, Leadership, and Research Methods
Advisor/Mentor
Ames, Allison
Committee Member
Turner, Ronna C.
Second Committee Member
Treadwell, Kelli
Third Committee Member
Liang, Xinya
Fourth Committee Member
Clauser, Jerome
Keywords
Educational testing; Psychometrics; Test security
Abstract
Educational assessment policymakers and stakeholders have recently called for assessments that incorporate instruction and learning into the assessment process (e.g., through-course assessment and longitudinal certification assessments). A feature of these assessments is repeated measurements of examinee ability intended to inform learning interventions. Though these longitudinally-administered assessments are often designed to be more flexible for examinees, the increased flexibility presents testing practitioners with unique challenges in maintaining the integrity of exam scores. These assessments, often characterized by on-demand or continuous administration, necessitate repeated use of items over time. Repeating items increases the risk of compromised items that may lead to examinees with item preknowledge, thereby compromising the validity of score interpretations.
Test security, and more specifically, item preknowledge, has received much attention among researchers. Given the recency of longitudinally-administered assessments, however, test security statistics applied to longitudinally-collected assessment data have received little or no attention. Beyond increased security threats, these assessments introduce challenges to item preknowledge detection because of systematic (e.g., change in examinee ability) and unsystematic (e.g., unreliability in proficiency estimation) influences. Nonetheless, longitudinally-collected assessment data may provide additional possibilities for detecting aberrant response behavior not possible in point-in-time assessment contexts.
The purpose of this dissertation was to develop and examine methods for identifying examinees exhibiting item preknowledge in a longitudinal assessment framework. This dissertation comprises three studies that may help testing organizations administering longitudinally-administered assessments develop a suite of test security statistics. By leveraging the information from longitudinal data, the first study extends preexisting test security statistics developed for point-in-time assessments to evaluate for changes in examinee response behavior. The second study extends test security statistics specifically developed to detect sudden changes in examinee response behavior to detect whether and when an examinee begins exhibiting item preknowledge throughout a longitudinal assessment. The third study examines if test security programs should adopt specialized longitudinal statistics, such as those from studies one and two, or if conventional point-in-time methods are sufficient. The three studies comprise simulation studies to assess the test security statistics' performance in realistic testing conditions and applied examples to showcase their practical use and limitations with real data.
Citation
Myers, A. (2025). Detecting Item Preknowledge in Longitudinally-Collected Assessment Data. Graduate Theses and Dissertations Retrieved from https://scholarworks.uark.edu/etd/5678
