Date of Award
Doctor of Philosophy
As English as a second language (ESL) populations in English-speaking countries continue to grow steadily, the need for methods of accounting for students’ academic success in college has become increasingly self-evident. Holistic assessment practices often lead to subjective and vague descriptions of learner language level, such as beginner, intermediate, advanced (Ellis & Larsen-Freeman, 2006). Objective measurements (e.g., the number of error-free T-units) used in second language production and proficiency research provide precise specifications of students’ development (Housen, Kuiken, & Vedder, 2012; Norris & Ortega, 2009; Wolfe-Quintero, Inagaki, & Kim, 1998); however, the process of obtaining a profile of a student’s development by using these objective measures requires many resources, especially time. In the ESL writing curriculum, class sizes are frequently expanding and instructors’ workloads are often high (Kellogg, Whiteford, & Quinlan, 2010); thus, time is at its limits, making the accountability for students’ development difficult to manage.
The purpose of this research is to develop and validate an automated essay scoring (AES) engine to address the need for resources that provide precise descriptions of students’ writing development. Development of the engine utilizes measures of complexity, accuracy, fluency, and functionality (CAFF), which are guided by Complexity Theory and Systemic Functional Linguistics. These measures were built into computer algorithms by using a hybrid approach to natural language processing (NLP), which includes the statistical parsing of student texts and rule-based feature detection. Validation follows an interpretive argument-based approach to demonstrate the adequacy and appropriateness of AES scores. Results provide a mixed set of validity evidence both for and against the use of CAFFite measures for assessing development. Findings are meaningful for continued development and expansion of the AES engine into a tool that provides individualized diagnostic feedback for theory- and data-driven teaching and learning. The results also underscore the possibilities of using computerized writing assessment for measuring, collecting, analyzing, and reporting data about learners and their contexts to understand and optimize learning and teaching.
Stephanie Maranda Link
Link, Stephanie Maranda, "Development and validation of an automated essay scoring engine to assess students’ development across program levels" (2015). Graduate Theses and Dissertations. 14567.