Study Shows Negative Impact of Generative AI on Learning
Researchers at the University of Pennsylvania found that Turkish high school students who used the AI bot ChatGPT to help solve practice math problems performed worse on a math test than students who did not use ChatGPT before the test.
The researchers titled their paper “Generative AI Can Harm Learning,” to make it clear to parents and teachers that AI-based chatbots currently available for free can negatively impact learning.
Study details:
The study included nearly 1,000 students in grades 9 through 11 during the fall of 2023. Students were randomly assigned to three groups to study mathematics using one of three methods:
- With access to ChatGPT.
- With access to an educational AI bot powered by ChatGPT.
- Without any technical assistance at all.
Students in each class were given the same problems to practice, and then tested to see how well they had learned the different topics. The researchers ran four cycles of the study, giving students four 90-minute sessions of practice time on four different math topics to understand whether AI tends to help, hurt, or have no effect on learning.
Those who used ChatGPT solved 48% more of the training problems correctly, but ultimately scored 17% worse on the test on the training material than those who did not use the bot.
A third group of students had access to a special version of ChatGPT that acted more like a teacher. It was designed to offer hints on how to solve a problem without directly revealing the answer. Students who used it performed significantly better on practice problems before the test than students who didn’t get any technical help.
But on the test that followed the training phase, the performance of those students who received training with the help of artificial intelligence did not improve. The students who solved the training problems in the traditional way on their own had the same test scores as those who received assistance from artificial intelligence.
The problem, the researchers believe, is that students rely entirely on the chatbot to provide a final answer. When the researchers analyzed the prompts that students wrote to ChatGPT, they found that most students asked for the final answer, and students were not interested in understanding the details of the solution method.
On the other hand, the errors made by ChatGPT while solving may have been a contributing factor. The bot did not provide correct solutions to all math problems, and its calculations were incorrect 8% of the time. But the main problem was that its method of solving the problems was incorrect 42% of the time. The private version of ChatGPT provided the correct answers and correctly solved the problem in most of the problems.
A draft of the trial paper was published on the SSRN website in July 2024, but has not yet been published in a journal. This is just one trial in one country, and more studies will be needed to confirm its results.