Community posts

Do badges work?


by Paul Denny
The University of Auckland

Badges everywhere

Have you ever wondered whether some of the "game-like" rewards that are becoming more and more common online actually have a measurable impact on user participation? Does the promise of earning a "Hotel specialist" badge on Trip Advisor motivate travellers to write more reviews? On Stack Overflow, a popular question and answer forum for programmers, do people answer more questions than they otherwise would so that they can increase their reputation score and earn a higher spot on the global leaderboard?

Of course, if you play games these kinds of rewards are nothing new - performance in many games is measured by points, leaderboards have been around since the earliest arcade games, and the Xbox Live platform has been rewarding players with achievements for nearly a decade. Now, in an attempt to motivate users across a broad range of applications, we see these game-like elements appearing more frequently. But do they work?

Badges in PeerWise

PeerWise includes several game-like elements (points have been discussed on this blog before), including badges (or "virtual achievements"). For example, regular practice is rewarded with the "Obsessed" badge, which is earned for returning to PeerWise on 10 consecutive days and correctly answering a handful of questions each time.

Other badges include the "Insight" badge, for writing at least 2 comments that receive an agreement, the "Helper" badge for improving the explanation of an existing question, and the "Good question author" badge, awarded for authoring a question that receives at least 5 "excellent" ratings from other students. A complete list of the available badges can be seen by clicking the "View my badges" link on the Main menu.

As you would expect, some badges are much harder to earn than others. Almost every student earns the "Question answerer" badge - awarded when they answer their very first question. The following chart shows the percentage of students with the "Question answerer" badge that earn each of the other available badges. Only about 1 in 200 students earn the "Obsessed" badge.

The badges in PeerWise can be classified according to the roles that they play (there is a nice article by Antin and Churchill that explores this further):

It is interesting to note that most of the badges awarded for answering questions are of the "Goal setting" variety, whereas those awarded for authoring questions are mainly in the "Reputation" category.

And now back to our original question - do these badges have any influence over the way that students use PeerWise? When considering this question, we must keep in mind that observed effects may not necessarily be positive ones. One of the criticisms levelled at extrinsic rewards, such as game-like elements, is that they have the potential to undermine intrinsic motivation in a task, which is clearly of concern in an educational context. However, this is a somewhat contentious claim, and very recent work by Mekler et al. showed no negative impact on intrinsic motivation in an experiment measuring the effect of using game elements to reward user participation in an online image-tagging activity (although it must be noted that this was a short-term study and motivation was self-reported).

Anecdotal support

There is certainly some anecdotal evidence that the PeerWise badges are being noticed by students in a positive way. Examples of this include public tweets:

as well as responses to a survey conducted in 2012 at the University of Auckland:

"I didn't think I was "badge" type of person, but I did enjoy getting badges (I was the first one to get the obsessed badge - yay!). It did help motivate me to do extra and in doing so, I believe I have learnt more effectively."

"The badges did make me feel as if I was achieving something pretty important, and helped keep Peerwise interesting."

Another example was nicely illustrated in a talk given by James Gaynor and Gita Sedhi from the University of Liverpool in June this year, in which they presented their experiences using PeerWise at a local teaching and learning conference. On one of their slides, they displayed a summary of student responses to the question: "Was there any particular aspect of PeerWise you liked?"

Across the two courses examined, "badges" and "rewards" emerged quite strongly (points, rewards, achievements and rankings were coded as "Other rewards").

However, it should be noted that not all students are so positive about the badges. Other responses to the previously mentioned survey indicate that the effect on some students is fleeting:

"well, it kinda increase my motivation a bit at the beginning. but then i get bored already"

"They don't really affect my motivation now, but they did when I first started."

and others pay no attention to the badges at all:

"I never cared about the badges -> simply because they dont mean anything -> i.e. does not contribute to our grade"

"They did nothing for my motivation."

Controlled experiment

To understand the impact of the badges more clearly, we conducted a randomised, controlled experiment in a very large class (n > 1000). All students in the class had identical participation requirements (author 1 question and answer 20 questions), however only half of the students were able to see the badges in the interface and earn them for their participation. This group was referred to as the "badges on" group, whereas the control group who were not able to see the badges were referred to as the "badges off" group. The experiment ran over a period of 4 weeks in March 2012, and the class generated approximately 2600 questions and submitted almost 100,000 answers.

Students in the "badges on" group, who were able to earn the badges, submitted 22% more answers than students in the control group. The chart below plots the day to day differences over the course of the study - on all but one day, the "badges on" students submitted more answers than the "badges off" students.

The table below summarises the number of questions authored, answers submitted, and distinct days of activity for students in each group.

The presence of the badges in the interface had a significant positive effect on the number of questions answered and the number of distinct days that students were active with PeerWise. Interestingly, although there was no effect on the number of questions authored by students, no negative effects were observed - for example, the increase in the number of answers submitted did not lead to a reduction in the accuracy of those answers.

If you you would like to see additional data from this experiment, as well as a more complete discussion and acknowledgment of the threats to the validity of the results, the full paper is available online (and on the PeerWise Community Resources page). Of course, no experiment is perfect, and this work probably raises more questions than it answers, but it does provide some empirical evidence that the badges in the PeerWise environment do cause a change in the way that students engage with the activity. Perhaps we could see similar effects in other, similar, educational tools?

TLDR: And for those who prefer movies to reading, the conference in which this work was published required a brief accompanying video. If you are really keen, see if you can last the full 40 seconds!

Return to Community posts