Gamification in Information Retrieval: State of the Art, Challenges and Opportunities

Gamification aims at applying game design principles and elements, such as points, badges, feedbacks or leader boards in nongaming environments. An interesting goal of gamification is to combine and exploit the fun factor for targeting other aspects like achieving more accurate work, more cost effective solutions and better retention rates. The application of gamification techniques to IR tasks poses interesting research challenges. In this paper, we propose an analysis of the state of the art in this field and we summarize interesting challenges and opportunities for the near future. 1 Gamification in IR Gamification relates with the use of game thinking and techniques in non-game contexts to enable the engagement of users with solving problems and to increase users’ self contributions [15]. It is defined in [5] as the use of game-play mechanics for non-game applications. For this reason, any application, task, process or context can theoretically be gamified [12]. The main goal of gamification is to increase the engagement of users by using game-like techniques such as scoreboards and personalized fast feedback [7], making people feel more ownership and purpose when engaging with tasks [13]. Gamification also implies a social game and interaction with other participants. It employs game mechanics, i.e., points, levels, challenges, virtual goods, leaderboards, gifting and charity, and aesthetics (motivations), i.e., reward, status, achievement, self expression, competition, altruism to enact such interactions [14]. There are precise gamification effects that one may wish to stimulate and clear game mechanics that address these changes. For example in order to stimulate the sense of achievement, challenges can be proposed to the user. Similarly status can be stimulated through levels, competition through leaderboards and rewards through virtual goods and points. In this rule-bounded, goal-oriented play, valuable content is also an important incentive. According to Groh [9] it is important for a product to offer real benefits and real value rather than just rewards which in the long run may seem less appealing to users. Thus by stripping a site down of its point and badges there is still meaningfull content in it. Gamification is about making an application or a task more fun, rather than actually stimulating playfulness. Gamification can be applied in several domains, typically for increasing the engagement of a user, for teaching, entertaining, measuring, and to improve the perceived ease of use of information systems. Successful examples are Foursquare, Twitter, Stack Overflow, Hacker News. While both Foursquare and Twitter employ gamification for increasing their user engagement, Stack Overflow and Hacker News are powerful platforms for question answering that promotes also authoritativeness of trustworthy users with both reward and punishment mechanisms. Gamification is thus used for stimulating and promoting users to be authoritative and active. In gamification it is important to understand users and create scenarios that appeal to their personality types: explorers, achievers, socializers and/or killers. Gamification can bring various benefits like increased engagement, loyalty, time spent, influence, fun or productivity. A possible benefit for IR is the understanding that the contribution of user will help raise the quality of the service and content, thus indirectly the user satisfaction. Bartle argues that although game design is an art form, gamification is an application of psychology, thus one of the most important aspects of gamifying Information Retrieval (IR) tasks relies on understanding of the human motivation [2]. Relevance Assessment, Feedback and Ranking An interesting application in IR is relevance assessment. Chamberlain introduces a model for rewarding and evaluating users using retrospective validation, with only a small gold standard required to initiate the system [4]. The model is not based on the quantity of tasks performed but rather is focused on an agreement-based reward which prizes the quality of the solutions. The evaluation of the theoretical model indicated that the reward mechanism succeeds in awarding the high quality answers, but in practice it is not such a strong signal for predicting the user performance. Harris exploits groups’ ability to assess relevance of documents and images and also rank their choices [10]. During their experiment participants are divided in two groups, one making judgements based on their own assessments, while the other makes judgements based on the estimate of consensus decision. To motivate participant, financial rewards are offered. When participants use consensus opinion as a guide, relevance assessment are homogenous probably due to the fact that they are more conservative. Another paper that uses player feedback is [3]. Here, authors’ objective is improving image recognition accuracy. The process is divided in three steps: first the systems performs recognition, then the information on which they wish to retrieve feedback is compiled and lastly they retrieve the feedback form the players through a crowd-sourced gamified approach. They also propose a method for leveraging ambigous feedback by introducing a measure of certainty. They conclude that due to the feedback received from the players the accuracy of the recognition systems improves. Fort et al. propose a Game With a Purpose that allows annotation of corpora with dependency syntax [8]. Students in linguistics are trained to annotate certain phenomena whereas a previous pre-annotation step gives an idea of whether there are significant inconsistencies between the two. They do not motivate users by claiming it is useful for research instead they focus on the playful aspect of the game and user personal interests. The 1 http://stackoverflow.com/ 2 https://news.ycombinator.com/ quality of the data implies: the trustworthiness of players and the assessment of the correctness of analyses.