Examining the Role of Generative AI in Academic Writing Assessment: A Mixed-Methods Study in Higher Education

Abstract

This study examines the use of generative artificial intelligence (AI) to assist in grading and feedback on undergraduate academic writing. While AI technologies in education have shown considerable potential, their adoption remains controversial due to contrasting evidence about their effectiveness. At a university in Hong Kong, an initiative was launched to use generative AI for providing formative feedback on students’ written drafts prior to final submission. This mixed-methods study evaluates the initiative by comparing AI-generated grades with those assigned by human instructors and analyzing stakeholder perceptions of the AI feedback. Discrepancies between the findings of quantitative and qualitative research may be attributed in part to the research instruments used. Quantitative analysis revealed high inter-rater reliability between AI-generated scores and human-assigned grades, demonstrating the technical accuracy of the system. However, qualitative data from student interviews and instructor reflections revealed significant concerns about the AI’s ability to deliver contextually relevant, personalized, and pedagogically meaningful guidance. Stakeholders expressed scepticism and mistrust, emphasizing the lack of depth, specificity, and adaptability in AI feedback compared to human feedback. By exploring these discrepancies, this study underscores the tension between technical reliability and perceived pedagogical value in AI-assisted assessment. It advocates for stakeholder-informed, hybrid approaches that integrate AI with human oversight to ensure AI tools genuinely support student learning outcomes. Future initiatives should prioritize collaborative design, transparent communication, and stakeholder education to address these challenges effectively.