ChatGPT: The brightest student in the class

Abstract

This paper presents a research study that evaluated the score ChatGPT would get when summarizing a reading comprehension text from the PISA international tests with a prompt that made it simulate doing this as if it were a 15-year-old student. For this purpose, the text was camouflaged among 30 other summaries made by real 15-year-old students and was evaluated by 30 Spanish language teachers with different profiles in terms of age, professional experience, and gender who were unaware that one of the texts was made by artificial intelligence (AI). The evaluation of the summary, for which a homogeneous rubric is used, is based on two fundamental criteria: content and style. For the data analysis descriptive and inferential statistical techniques were used. The results show that the ChatGPT summary obtained the best marks in terms of content and style, with its respective marks being 3 and 2.5 points higher than those of the students. Therefore, we can deduce that the style and content of the ChatGPT summary greatly exceeded those presented by the students. These results are independent of the ages, levels of professional experience, and genders of the teachers who corrected the summary. The integration of AI tools such as ChatGPT must be based on solid methodological proposals that integrate their use from a creative and critical perspective that allows learning with the support of these tools and not using them as substitutes for the development of basic student competencies.