AI Agents as Academic Assistants in Higher Education: Effects on student’s’ Academic Achievement, Motivation, and Self-Regulated Learning

Abstract

This study focused on how the academic achievement, academic motivation, and self-regulated learning (SRL) of university students were affected using an Artificial Intelligence (AI) agent as a conversational academic assistant. The data were collected using a quasi-experimental pre-test–post-test control group design from the target population of the National University of Sciences and Technology (NUST), COMSATS University Islamabad (CUI) and International Islamic University Islamabad (IIUI) of Pakistan, comprising undergraduate students of three major universities of Islamabad. This research aims to investigate the effect of the use of Artificial Intelligence (AI) agents as conversational academic assistants on university students' academic achievement, academic motivation, and academic self-regulated learning (SRL). To collect data, a target population of undergraduate students from three main universities of Islamabad (Pakistan) with three different names of National University of Sciences and Technology (NUST), COMSATS University Islamabad (CUI), and International Islamic University Islamabad (IIUI) was used. The subjects consisted of 360 students from whom 180 students were selected and assigned to an experimental group (structured academic support using the AI agent), and 180 students were selected and assigned to the control group (traditional academic support). A total of 360 students were selected using a multi-stage stratified cluster sampling method and were divided into two groups: an experimental group (structured academic support with AI-agent) and a control group (traditional academic support). The instruments used in data collection were the Standardized Subject Achievement Test (SAT, α = .88), the Academic Motivation Scale (AMS, α = .86) and the Motivated Strategies for Learning Questionnaire (MSLQ-SRL subscale, α = .89). Paired-sample t-test, independent-sample t-test and Analysis of Covariance (ANCOVA) were used for data analysis.

The results were statistically significant and showed that there were differences between the experimental group and control group for all three dependent measures. Students who used the AI academic assistants scored higher in the academic achievement scale of the post test (F(1,357) = 48.32, p<.001, η_p^2=.119) and higher on the intrinsic academic motivation and extrinsic academic motivation scales. The study shows that pedagogically structured AI agents can serve as effective scaffolding pedagogical tools in higher education to facilitate students' cognitive mastery and student agency. Key recommendations include organizing a workshop for teachers to raise awareness and understanding of AI in their literacy, developing policies that guide the use of AI and leveraging frameworks to provide structured prompt engineering in the university curriculum to minimize reliance on AI and maximize pedagogy.

The students were randomly split into an experimental group (n=180) and a control group (n=180): The experimental group was provided with 8 weeks of intervention using the structured academic support with an AI-agent, while the control group was provided with the traditional academic support. The instruments used in collecting the data were standardized Subject Achievement Test (SAT), Academic Motivation Scale (AMS) and Motivated Strategies for Learning Questionnaire (MSLQ-SRL subscale). Paired sample t-test, independent sample t-test, and analysis of covariance (ANCOVA) were used to analyze the data.

The results showed that the experimental group's scores were significantly higher than the control group's scores on all three dependent variables. The students who used the AI academic assistant had higher post-test academic achievement scores (F(1,357) = 48.32, p< .001, η_p^2 = .119), higher scores on the intrinsic academic motivation score, scores on extrinsic academic motivation, and scores on cognitive and metacognitive self-regulation strategies. The study, however, shows that pedagogically structured AI agents can be a scaffolding tool for improving cognitive knowledge and empowering students in higher education. The recommendations stress the need for implementing institutional policies, offering faculty AI literacy courses, and incorporating a structured approach to AI prompt engineering into the university's curriculum to mitigate overuse and maximize the use of AI in the classroom.