Students want more and faster feedback on the texts they write. A research project in secondary school suggests that students can write better texts if they receive feedback from a computer program based on artificial intelligence.
Written by Ann-Kristin Johansen, Østfold University College
At Østfold University College (HiØ), researchers are working on developing an artificial intelligence program to help students write good texts, and the first prototype has already been tested on secondary school students at three different schools in Halden.
In collaboration with teachers at the pilot schools, the researchers developed a writing process for pupils on the theme of heroes. The pupils were asked to write a text about who they consider to be a hero and argue why. Based on the text they entered into the computer program, they received individual feedback in return. The actual writing process consisted of the students working on three drafts of the text and receiving feedback between each round, with the last draft being submitted to the teacher for a final assessment.
- It was designed as a writing process where students wrote their texts through multiple drafts and received feedback from the technology," explains Irina Engeness, professor and dean at the Faculty of Teacher Education and Languages at HiØ.
The testing of secondary school students is part of the project "Artificial intelligence (AI) for assessment for learning (AfL) to strengthen learning and teaching in the 21st century" (AI4AfL). The project has received NOK 12 million in funding from the Research Council of Norway and is run by HiØ in collaboration with Volda University College, Halden Municipality and Hypatia Learning AS.
The university, the municipality and Hypatia Learning are cluster partners in the Cluster for Applied AI.
Such clear differences
While some of the classes that participated in the writing process were allowed to use the new computer program in their text work, the other classes were to work without this help and instead give each other feedback along the way using the assignment criteria. The preliminary results indicate that the choice of method had a different impact on the final grades the students received.
- "At all three schools, the classes that participated in the writing process using the new technology have improved their grades from the first draft to the final version," says Engeness, who is leading the project.
In the classes that did not have access to the program, there were also changes in grades during the process, but not to the same extent. The researchers found that students who used the technology were more involved in the discussions about the feedback and how to further develop their texts in line with the feedback.
- "We saw that they used the feedback provided by the technology as a starting point for the conversations, and from a pedagogical perspective, the discussions in these classes were more content-rich. At the same time, we saw that the students in the other classes that didn't use the same technology were unsure of how to give each other feedback," says Engeness.
The quality of the feedback that students receive on schoolwork generally varies widely, according to the researchers.
- "Some students find that they only receive feedback such as 'write more', but they don't necessarily understand what else they should write," says Siv Gamlem, a professor at Volda University College.
Gamlem's research includes assessment and feedback and says it is important that students receive feedback as an integral part of the writing process. The preliminary findings from the project also confirm this.
- With this program, we want to give all students an equal opportunity to develop their work," she says and adds:
- Through this program, they will receive input on the content of the text and what they should write more about. This will help them to develop knowledge and understanding of their own writing process and to write their own texts.
Received significantly more feedback
After the writing process, the students were interviewed about what it was like to participate.
- "The students who used the computer program said that they really appreciated the feedback they got from it," says Engeness.
The students who did not get to use the new technology had a slightly different experience of the writing process.
- They stated that it was not easy to participate and found it difficult to find the right form of feedback to give and what to focus on. As a result, they became very cautious with their feedback," she explains.
According to the researchers, the project so far shows that the students who used the technology also received the most and most varied feedback. This finding is not unexpected.
- "It will not be possible to give all students this type of feedback integrated into a writing process if it is done by only one teacher," says Gamlem.
Through the technology, each student received as many as 30-40 pieces of feedback on each draft. According to the researchers, the teachers involved expressed concern prior to the workshop that the computer program would give students too much feedback. However, this was not the case.
- "When we interviewed the teachers afterwards, they were surprised at how carefully the students took in all the feedback. At the same time, they said that they would never be able to give the same amount to each student as they received through the program," says Engeness.
However, the researchers note that there are individual differences between students, and that care must be taken not to give too little or too much feedback. Students must be both able and willing to use the feedback they receive on their work.
Will work on improving the concept
The students who participated in the project stated that they liked the design of the program and found the quick response positive. They also expressed that it was positive to work in a process where they could improve their texts in several rounds.
However, the amount of feedback the program provides is one of the things the researchers will now continue to work on towards the completion of prototype 2 of the program.
- "We need to find a solution to how much feedback should be given so that it is motivating and not too much. We know that students who don't perform well will receive a lot of feedback that highlights mistakes, and this can be demotivating," says Gamlem.
She emphasizes that they must ensure that feedback is given in a type and amount that contributes to further development in the students' text work.
- "If they get too much, they won't be able to use it, which is why it's been important for us to interview the students participating in this experiment. We need to make sure that all students receive feedback that motivates them to develop and helps to improve their work," she says.
No new Chat GPT
The development of the computer program is in full swing, in parallel with the rather intense debate surrounding Chat GPT and similar services based on artificial intelligence. The researchers behind the multi-million euro project AI4AfL emphasize that they are not working on a separate Chat GPT version.
Except for the fact that the program they are now developing is based on AI technology, it has little in common with Chat GPT.
- These are completely different products. Chat GPT is a product that produces a text for the user, while in our case it is about taking an essay or text that the student writes and providing feedback on it. Only the AI technology behind these software products is similar.
This is according to Associate Professor Hasan Ogulat the Faculty of Information Technology, Engineering and Economics at HiØ.
According to Ogul, the first prototype of the program, which focuses on syntactic errors, makes little use of AI technology. The second prototype that will now be developed based on, among other things, the results of the testing done among school students, will make greater use of AI technology because it is about analyzing and understanding the actual meaning of the texts.
- It is important for us to emphasize that we are not developing another Chat GPT technology. The educational perspective is absolutely essential to the technology we are developing. It should offer precise and useful feedback on student submissions that can be addressed in the next draft. Such a writing process can be embraced by students as a good approach to understanding what it means to create a good essay," Engeness explains.
Strengthens the teacher as a guide
After testing the first prototype on secondary school students, the researchers have a comprehensive data set that will be analyzed and used to develop prototype 2. This will be tested on the same students during the fall in a workshop where they will work on both syntax and content. During the test project, the researchers collected film material showing the students at work and how teachers give feedback on their writing. They also collected a large number of essays in three versions from each participating student.
The data will be analyzed from both a pedagogical and a technological perspective. In prototype 2, the researchers hope to strengthen the program so that it can work even better for teachers from a guidance perspective.
- "We are all former teachers, and we know how difficult it is to collect data on students to find out what they are struggling with. This tool provides specific info about each student that the teacher can use to understand what the student actually needs," Gamlem points out.
At the same time, she emphasizes that the program cannot replace the role of the teacher.
- It's important to remember that there's one thing that students need that they don't get from technology, and that's emotional support. If you see students struggling, as a teacher you can show that you see them and offer both academic and emotional help and support. This program will give the teacher time and opportunity to use their skills and knowledge in a different way," she says.