“Comparing the quality of human and ChatGPT feedback of students’ writing” (2024), by Jacob Steiss et al. Social Sciences research paper. Original abstract below, unchanged. No free PDF was found.

Comparing the quality of human and ChatGPT feedback of students’ writing

Jacob Steiss, Tamara Powell Tate, Steve Graham et al. · March 11, 2024 · Social Sciences

Abstract

Offering students formative feedback on drafts of their writing is an effective way to facilitate writing development. This study examined the ability of generative AI (i.e., ChatGPT) to provide formative feedback on students’ compositions. We compared the quality of human and AI feedback by scoring the feedback each provided on secondary student essays (n=200) on five measures of feedback quality: the degree to which feedback (a) was criteria-based, (b) provided clear directions for improvement, (c) was accurate, (d) prioritized essential features, and (e) used a supportive tone. We examined whether ChatGPT and human evaluators provided feedback that differed in quality for native English speakers and English learners and for compositions that differed in overall quality. Results showed that human raters were better at providing high-quality feedback to students in all categories other than criteria-based. Considering the ease of generating feedback through ChatGPT and its overall quality, practical differences between humans and ChatGPT were not substantial. Feedback did not vary by language status for humans or AI, but AI and humans showed differences in feedback based on essay quality. Implications for generative AI as an educational tool are discussed.

Source page DOI: 10.1016/j.learninstruc.2024.101894