Community posts
Students learn by generating study questions - but the content matters!
by Paul Denny
The University of Auckland
About a year ago, in a large first-year science course at the University of Auckland, students were asked what they felt was most useful about creating, sharing and practicing with their own study questions (the class was using PeerWise, and this was one of several questions students were asked about their experience).
I remember reading through some of the students' responses, and this one always stuck out to me:
"You don't really understand how much or how little you know about a concept until you try to devise a good, original question about it"
It seemed to almost perfectly reflect the old adage that in order to teach something (which is, in essence, what students do when they explain the answers to the study questions they create), you must understand it well yourself. And this seemed to be a common perception in this class - overall, students felt that authoring their own questions was more helpful than answering the questions created by their peers. This is illustrated in the chart below, which compares the responses of the class when asked to what degree they felt authoring and answering was helpful to their learning in the course (rated on a typical 5-point scale from "strongly agree" to "strongly disagree"):
So, at least in this class, students felt that question authoring was most helpful to their learning. The instructors felt this was a positive sign, particularly because PeerWise meant they were able to run this activity in their large class with little moderation. But of course student perceptions of learning, and actual learning, are not the same thing! The central question remains - does this activity actually help students learn?
A good starting point is to look at the relationship between student engagement with the activity and their performance in the course - both of which can be measured in various ways. One measure of student engagement is the PeerWise reputation score (an approximate measure of the value of a student's contributions), and the most obvious measure of course performance is the final mark or grade in the course. The chart below shows this relationship for the surveyed course described above:
To make the relationship clearer in the chart, students have been binned according to the final grade they achieved in the course. At the University of Auckland, there are 12 possible grades: 9 passing (from C- to A+) and 3 failing (from D- to D+). The chart plots the average final course mark, and the average PeerWise reputation score for all students who earned a particular final grade in the course. In this case, students who engaged most actively with PeerWise, tended to perform better in the course.
This relationship between student engagement with PeerWise and exam scores or overall course performance appears to be quite robust. A number of excellent studies have emerged this year, across various academic disciplines, that highlight this link. Hardy et al. report a significant positive correlation between students' use of PeerWise and their exam performance in 5 large science courses, taught across 3 research-intensive universities in the UK, spanning the subjects of physics, chemistry and biology. Galloway and Burns examined the use of PeerWise by first year chemistry students and report a significant correlation between student activity with PeerWise (as measured by the reputation score) and their exam performance. Similar positive correlations have been reported recently by McQueen et al. (in biology), Singh (in computer science), and by Kadir et al. (in medicine):
So the evidence is clear - students who engage more with PeerWise also tend to perform better in their respective courses. While establishing this relationship is a useful first step, it does not answer our central question regarding student learning. The correlations do not imply that the use of PeerWise by the more successful students has caused their superior performance. In fact, it would be quite a surprise if we didn't see this positive link. I think most instructors would probably agree that for any kind of course activity, the better students (who tend to earn the better grades) are the ones who are more likely to participate to a greater extent.
To further explore the impact on learning, we recently conducted a randomised, controlled experiment in a first-year programming course (in which engineering students were learning MATLAB programming) at the University of Auckland. Students were randomly assigned to one of two groups (called "Authoring" and "Non-authoring") to control for their ability. Students in the "Authoring" group were asked to publish 3 study questions on PeerWise prior to a summative mid-semester exam. Students in the "Non-authoring" group could access all of the created questions on PeerWise, but did not author any of their own (NB: for fairness, at a later point in the course we switched these conditions for all students):
This was an "out of class" activity - students participated in their own time, and typically the hours between 8pm and 10pm were when most questions and answers were submitted. This took place over an 11 day period prior to a mid-semester exam that consisted of 10 questions. The activity was almost entirely student-driven - other than setting up the practice repository on PeerWise for their students and setting the exam questions, the instructors were not involved in the activity.
A total of 1,133 questions were authored by students in the "Authoring" group, and a total of 34,602 answers were submitted to these questions by students in both groups as they practiced prior to the exam. So, what was the impact on exam performance?
As a group, the "Authoring" students performed better than the "Non-authoring" students on 9 of the 10 exam questions - a binomial test reveals that this is statistically unlikely to have happened by chance (p = 0.0107). In terms of the average exam scores achieved by each group, there was a difference - but it wasn't particularly large. As shown in the chart below, the "Authoring" students performed about 5% better than the "Non-authoring" students, again a statistically significant result (Wilcoxon test, p = 0.0197):
While the superior performance of the "Authoring" students on this mid-semester exam can be attributed to their use of PeerWise (more specifically, to their authoring of the study questions), this doesn't necessarily mean that there aren't more effective ways for students to study. For one thing, we don't know how the "Non-authoring" students spent their time while the "Authoring" students were creating questions - we certainly can't assume that they spent the same amount of time preparing for the exam.
What happens if we take a closer look at the content of the questions? This is where things get more interesting.
In an article published in 1994, entitled "Student Study Techniques and the Generation Effect", Paul Foos suggests a reason for why we may have seen such a small difference between the average exam scores of each group. He argues that students who prepare for an exam by generating study questions may benefit only if they create questions on topics that are targetted by the exam questions. This certainly makes intuitive sense - some of the students in our "Authoring" group probably created perfectly "good" questions, but these questions did not target the concepts that were examined by any of the 10 exam questions, and thus they didn't benefit as a result.
To explore this, we classified all 1,139 student authored questions according to the main topics that they targetted, and we did the same for the 10 exam questions. For simplicity, when we focussed on questions that targetted a single topic, we discovered that there were 3 core topics that were each targetted by 2 exam questions. For each of these three topics, the students can be classified into three groups:
- the "Authoring" students who created at least one question on the topic
- the "Authoring" students who did not create any questions on the topic
- and the "Non-authoring" students
We see virtually no difference between the performance of the "Non-authoring" students and the "Authoring" students who did not create questions on a topic, when answering exam questions on that topic - precisely as described by Foos' earlier work. Students who did author questions on a particular topic performed far better on the corresponding exam questions. The impact of question authoring on learning also becomes clearer - with effect sizes of between 10% and 20% across the question pairs.
Of course, the story doesn't end here. Although the question authoring activity did have a significant positive impact overall, some of the differences observed between the on-topic and off-topic "Authoring" students may be a result of students choosing to author questions on topics they already knew well, rather than learning much new from the process.
It is hard to say a lot more about this without more data - but like the correlation studies mentioned earlier, this helps to paint a picture of an activity which, with very little instructor involvement, can have a measurable positive effect on student learning!