This study served as a replication of previous work examining the reliability, validity, and sensitivity to growth of the Basic Skills and the Content-Analysis-Multiple Choice probes in two Iowa schools districts. One hundred five students in grades nine to twelve participated in the study. Data were gathered from February 2006 to April 2006. Over three months of data collection, students completed two Basic Skills probes and two Content Analysis-Multiple Choice probes each month. We examined the alternate form reliability and test-retest reliability for both types of probes. We found that both types of probes possessed adequate levels of reliability, with the Basic Skills probes demonstrating a higher level of reliability than the Content Analysis-Multiple Choice probes. To assess the validity of the probes, we gathered data from a variety of indicators of students’ proficiency in algebra including course grades, teachers’ evaluations of student proficiency and growth, and performance on standardized assessment instruments including the Iowa Test of Education Development (ITED) and the Iowa Algebra Aptitude Test (IAAT). We examined both concurrent and predictive validity. Concurrent validity was supported by finding moderate correlations between both types of probes and criterion measures including teachers’ evaluation of their students’ proficiency and IAAT scores. Predictive validity of both types of probes was supported by finding low or moderate correlations between the relationship of the earliest probe scores to other indicators administered at the end of the course including teacher ratings of growth, students’ end-term algebra grades, IAAT scores taken at the end of the semester, and ITED scores. We also examined the extent to which both types of probes reflect student growth and explored the relationship between student growth on the algebra measures and other indicators of growth. We found that students who did not drop the course grew .42 and 1.52 points each week on Basic Skills and Content Analysis- Multiple Choice probes, respectively. This result suggests that Content Analysis-Multiple Choice probes may be more sensitive to reflecting student growth. We also found that there was a small but significant correlation between teacher ratings of growth and Content Analysis-Multiple Choice probes. No significant correlation existed between Basic Skills probes and other indicators of growth.
Perkmen, Serkan; Foegen, Anne; and Olson, Jeannette R., "A Replication Study of the Reliability, Criterion Validity and Sensitivity to Growth of Two Algebra Progress Monitoring Measures" (2006). Project AAIMS Technical Reports. 11.