Assessing the Reading Problem-Solving Abilities of Generative AI Models Using the CSAT English Section
0
조회
0
다운로드
0
인용
0
참고문헌
키워드
초록
This study evaluates the reading problem-solving abilities of four leading generative AI models: OpenAI’s GPT-4o and GPT-o3, and Google’s Gemini 2.5 Flash and Gemini 2.5 Pro. The models were assessed using the English reading sections from the College Scholastic Ability Test for the past three academic years. To ensure reliability, each model was tested across three separate sessions. The findings reveal high consistency in each model’s performance but significant accuracy differences among them. Gemini 2.5 Pro performed best with 97.12% accuracy, while GPT-4o was the lowest at 86.83%. All models excelled at tasks involving context inference and detailed comprehension; however, they demonstrated clear weaknesses in higher-order discourse skills, frequently producing incorrect answers for questions requiring logical construction and referential understanding. These results suggest that while generative AI’s reading capabilities have advanced significantly, critical vulnerabilities remain. The study also highlights the necessity of comparative analysis when selecting AI models for educational purposes.
Abstract
This study evaluates the reading problem-solving abilities of four leading generative AI models: OpenAI’s GPT-4o and GPT-o3, and Google’s Gemini 2.5 Flash and Gemini 2.5 Pro. The models were assessed using the English reading sections from the College Scholastic Ability Test for the past three academic years. To ensure reliability, each model was tested across three separate sessions. The findings reveal high consistency in each model’s performance but significant accuracy differences among them. Gemini 2.5 Pro performed best with 97.12% accuracy, while GPT-4o was the lowest at 86.83%. All models excelled at tasks involving context inference and detailed comprehension; however, they demonstrated clear weaknesses in higher-order discourse skills, frequently producing incorrect answers for questions requiring logical construction and referential understanding. These results suggest that while generative AI’s reading capabilities have advanced significantly, critical vulnerabilities remain. The study also highlights the necessity of comparative analysis when selecting AI models for educational purposes.