This technical report summarizes the results of a study in which we examined the technical adequacy of four revised measures for algebra progress monitoring. The measures investigated included a Basic Skills probe, an Algebra Foundations probe, a Translations probe, and a Content Analysis-Multiple Choice probe. Revisions to the measures included the addition of a sample page prior to the initial administration of each type of probe and changes to the design templates used to create the Content Analysis-Multiple Choice measure. Seventy-eight students (6 of whom were receiving special education services) completed two forms of a Basic Skills measure, two forms of an Algebra Foundations measure, two forms of a Translations measure, and two forms of a Content Analysis-Multiple Choice measure administered over two data collection sessions. Each probe data collection session was then repeated to investigate the test-retest reliability of the measures. In addition, we gathered data on criterion variables including grades, overall grade point average, teacher ratings of student proficiency, and scores on district-administered standardized tests, as well as a measure of algebra aptitude. We examined both test-retest and alternate form reliability, as well as criterion validity, for both single probe scores and aggregated scores (computed by averaging two individual scores). We found that three of the four measures produced effective distributions of student scores, with no signs of floor or ceiling effects. On the Translations probe, students produced nearly half as many incorrect responses as they did correct responses, suggesting a high rate of guessing on that measure. The test-retest and alternate form reliability of single probes was higher than results obtained in previous studies, with correlations for most measures (except Translations) in the .6 to .8 range. Aggregating scores from two probes produced reliability estimates in the .7 to .8 range for all measures except the Translations measure. Criterion validity was examined by correlating students’ scores on the probes with other indicators of proficiency in algebra, including grades in algebra, teacher ratings, scores on the district’s achievment test, and scores on a standardized test of algebra aptitude. Correlation coefficients were higher than those obtained in earlier studies (Foegen & Olson, 2005; Foegen, Olson, & Perkmen, 2005). Correlation coefficients for single probes were generally in the low to moderate range (.3 to .5); small increments in the coefficients were obtained when aggregared scores were used. The strongest relations were obtained for the Basic Skills and the Content Analysis-Multiple Choice measures, followed by the Algebra Foundations measure. Students’ scores on the probes are most related to their teachers’ ratings, their grades in algebra, and their scores on the algebra aptitude test. Issues for future research are identified.
Foegen, Anne; Olson, Jeannette R.; and Perkmen, Serkan, "Reliability and Criterion Validity of Four Revised Algebra Measures in Districts B and C" (2005). Project AAIMS Technical Reports. 14.