AI composes questions to test knowledge after completing the course

Researchers from NC State and Carnegie Mellon University have developed an AI model that can generate questions to evaluate the course completed, which, according to teachers, are indistinguishable from human-written questions.

The new AI is called QUADL, and it does two things: it identifies key terms and ideas in educational texts, and then composes questions that focus on those terms and ideas.

"We provide QUADL with course content and learning goals, and QUADL can develop questions that will help students achieve these goals," says Noboru Matsuda, associate professor of computer science at North Carolina State University and co-author of an article on this work.

"People are good at designing courses, but during interviews with teachers and curriculum developers, we found that they often have difficulty developing questions that effectively assess students' progress in achieving learning goals for these courses," says Machi Shimmei, a graduate student at NC State and the first author of the paper. "Our research shows that QUADL can be a useful tool for teachers and course developers."

To test the effectiveness of QUADL, the researchers used existing software for online courses called Open Learning Initiative (OLI). The researchers recruited five teachers who use OLI for their classes and asked them to rate a long list of questions. Some of the questions were compiled by QUADL, another part by another AI model Info-HCVAE, and some questions have already been used in OLI courses. The study participants were not told where the questions came from, and were asked to evaluate the pedagogical value of each question.

"The assessments of the pedagogical value of the questions created by QUADL were almost identical to the assessments that teachers gave to questions written by people for use in OLI," says Shimmei. - The questions compiled by Info-HCVAE received lower grades from teachers."

Now the researchers are planning to conduct research in the student audience, during which teachers will use the questions compiled by QUADL to see how the questions compiled by QUADL affect students' learning, and whether they do at all.

"This upcoming work should close the cycle of this technology," says Matsuda. - Hypothetically, QUADL should work. Now we need to see if it will work in practice."

QUADL is part of a large suite of AI technologies that Matsuda and his colleagues are developing called PASTEL. All PASTEL technologies are designed to facilitate the development of training courses.

"These technologies deal with everything from question generation - which is the role of QUADL - to quality control functions used to assess how effectively each element of the learning material helps students learn," says Matsuda. "We are looking for both research partners who will help us develop these generative AI technologies, and teaching partners who are interested in using these AI tools in their courses."