Rate or Trade? Identifying Winning Ideas in Open Idea Sourcing

ion (Shaft and Vessey 2006). The task of evaluating ideas is represented by the mechanism that is provided to the user. Thus, mechanism accuracy is, all else being equal, a function of how well a mechanism supports decision-making of its users. The performance of such decision processes is generally referred to as decision quality, defined as the “correctness” or “goodness” of decisions (Dennis and Wixom 2001). It relates to the effectiveness with which users can solve a problem (Vessey and Dennis 1991). In the domain of evaluating innovation ideas, decision-making effectiveness refers to identifying the ideas of highest quality. Based on our multimethod estimation of true idea quality, we define decision quality as the effectiveness of mechanism use, i.e., the correctness of users’ idea evaluations recorded with the mechanisms used. We apply this to idea evaluation and arrive at the levels of abstraction presented in Table 1. Table 1 Abstraction Levels of an Idea Evaluation Task Levels of abstraction Idea evaluation task Task Evaluate idea quality Representation Use evaluation mechanism Action Record decisions about idea quality 2.3. Idea Evaluation with Rating Scales Behavioral decision-making research suggests that using a rating scale instantiates idea evaluation as a judgment task. Judgment involves making decisions about a set of alternatives in which each alternative is individually assessed. Using rating scales, users evaluate a finite set of alternatives (i.e., ideas) by applying a defined set of criteria. Assigning numerical values to these criteria, rating scales strive to identify an alternative that is closest to a defined optimum (Limayem and DeSanctis 2000). By means of different algorithms, individual ratings can be aggregated to group decisions (Todd and Benbasat 1999). The ideation literature has investigated rating scale-based procedures to evaluate the outcome of IT-enabled idea generation (e.g., Dean et al. 2006). However, this research has often focused on the use of rating scales by small groups of experts. For open idea evaluation, rating scales are frequently used (but rarely investigated) as a tool for eliciting evaluations from a larger group beyond select expert panels (e.g., Leimeister et al. 2009, Riedl et al. 2013, Zhao and Zhu 2014). Whereas rating scales by default do not provide peer feedback, typical applications for idea evaluation include some type of mean rating for users. In this regard, Di Gangi and Wasko (2009) as well as Blohm et al. (2013) have investigated how aggregated, rating-scale-based idea evaluations of customers affect organizational resource allocation decisions. In summary, there is a research gap in the current understanding of the circumstances in which rating scales can be effectively used for open idea evaluation. 2.4. Idea Evaluation on Preference Markets In choice tasks, individuals make one holistic decision from which they select one choice, or a few choices from a larger set of alternatives. Preference markets are a frequently applied instantiation of idea evaluation as a choice task. Preference markets are a special type of prediction markets that are virtual markets for collecting, aggregating, and evaluating dispersed information (Soukhoroukova et al. 2012). In prediction markets, users buy contracts that are bound to future events, e.g., the outcomes of political elections or sporting events. These contracts then have a certain payoff if the event occurs; there is no payoff otherwise. In efficient prediction markets, the market price represents the market’s expectation of the probability that the event will occur, and thus the market price acts as a predictor D ow nl oa de d fr om in fo rm s. or g by [ 13 0. 82 .1 .4 0] o n 03 J un e 20 16 , a t 0 3: 40 . Fo r pe rs on al u se o nl y, a ll ri gh ts r es er ve d. Blohm et al.: Rate or Trade? Identifying Winning Ideas in Open Idea Sourcing 30 Information Systems Research 27(1), pp. 27–48, © 2016 INFORMS Figure 1 Hypotheses and Research Model