Search
  • All Journals
Advanced Search
Search articles by Title, Author, Keywords, DOI
Submit Manuscript
7.7CiteScore
2.0Impact Factor
Advanced Search
MAP  >  The Knowledge Engineering Review
2018 Volume 33
Article Contents
Next Previous
ORIGINAL RESEARCH   Open Access    

A survey of incentive engineering for crowdsourcing

  • 1.

    School of Computer Science

  • 2.

    School of Computer Science

  • 3.

    School of Computer Science

More Information
  • Abstract: With the growth of the Internet, crowdsourcing has become a popular way to perform intelligence tasks that hitherto would be either performed internally within an organization or not undertaken due to prohibitive costs and the lack of an appropriate communications infrastructure. In crowdsourcing systems, whereby multiple agents are not under the direct control of a system designer, it cannot be assumed that agents will act in a manner that is consistent with the objectives of the system designer or principal agent. In situations whereby agents’ goals are to maximize their return in crowdsourcing systems that offer financial or other rewards, strategies will be adopted by agents to game the system if appropriate mitigating measures are not put in place. The motivational and incentivization research space is quite large; it incorporates diverse techniques from a variety of different disciplines including behavioural economics, incentive theory, and game theory. This paper specifically focusses on game theoretic approaches to the problem in the crowdsourcing domain and places it in the context of the wider research landscape. It provides a survey of incentive engineering techniques that enable the creation of apt incentive structures in a range of different scenarios.
  • 加载中
  • Adriaens T., Sutton-Croft M., Owen K., Brosens D., van Valkenburg J., Kilbey D., Groom Q., Ehmig C., Thürkow F., Van Hende P. & Schneider K. 2015. Trying to engage the crowd in recording invasive alien species in Europe: experiences from two smartphone applications in northwest Europe. Management of Biological Invasions 6(2), 215–225.

    Google Scholar

    Babaioff M., Dughmi S., Kleinberg R. & Slivkins A. 2015. Dynamic pricing with limited supply. ACM Transactions on Economics and Computation 3(1), 4.

    Google Scholar

    Benkler Y. & Nissenbaum H. 2006. Commons-based peer production and virtue. Journal of Political Philosophy 14(4), 394–419.

    Google Scholar

    Berg N. & Gigerenzer G. 2010. As-if behavioral economics: Neoclassical economics in disguise? History of Economic Ideas 18, 133–165.

    Google Scholar

    Blum A. & Monsour Y. 2007. Learning, regret minimization, and equilibria. In Algorithmic Game Theory, Nisan, N., Roughgarden, T., Tardos, E. & Vazirani, V. V. (eds). Cambridge University Press, 79–102.

    Google Scholar

    Bosha E., Cilliers L. & Flowerday S. 2017. Incentive theory for a participatory crowdsourcing project in a developing country. SA Journal of Information Management 19(1), 7.

    Google Scholar

    Brabham D. C. 2013. Crowdsourcing. MIT Press.

    Google Scholar

    Camerer C. F. 2003. Behavioural studies of strategic thinking in games. Trends in Cognitive Sciences 7(5), 225–231.

    Google Scholar

    Cameron J., Banko K. M & Pierce W. D. 2001. Pervasive negative effects of rewards on intrinsic motivation: the myth continues. The Behavior Analyst 24(1), 1.

    Google Scholar

    Cooper H. M. 1988. Organizing knowledge syntheses: a taxonomy of literature reviews. Knowledge in Society 1(1), 104–126.

    Google Scholar

    David G., Michael R., Erwin F. & Martin S. 2012. Crowdsourcing information systems: definition, typology and design. In ICIS 2012 The 33rd International Conference on Information Systems, Joey F George (ed.). Association for Information Systems/AIS Electronic Library (AISeL), 3562–3572.

    Google Scholar

    Dayama P., Narayanaswamy B., Garg D. & Narahari Y. 2015. Truthful interval cover mechanisms for crowdsourcing applications. In Proceedings of the 2015 International Conference on Autonomous Agents and Multiagent Systems, 1091–1099. International Foundation for Autonomous Agents and Multiagent Systems.

    Google Scholar

    Drexler K. E. & Miller M. S. 1988. Incentive engineering for computational resource management. The Ecology of Computation 2, 231–266.

    Google Scholar

    Ebden M., Huynh D., Moreau L. & Roberts S. 2015. Incentive engineering through subgraph matching with application to task allocation. HAIDM.

    Google Scholar

    Endriss U., Kraus S., Lang J. & Wooldridge M. 2011. Designing incentives for boolean games. In The 10th International Conference on Autonomous Agents and Multiagent Systems-Volume 1, 79–86. International Foundation for Autonomous Agents and Multiagent Systems.

    Google Scholar

    Ghani E., Kerr W. R. & Stanton C. 2014. Diasporas and outsourcing: evidence from odesk and India. Management Science 60(7), 1677–1697.

    Google Scholar

    Ghezzi A., Gabelloni D., Martini A. & Natalicchio A. 2017. Crowdsourcing: a review and suggestions for future research. International Journal of Management Reviews 0, 1–21.

    Google Scholar

    Groves T. 1973. Incentives in teams. Econometrica: Journal of the Econometric Society 41, 617–631.

    Google Scholar

    Guo B., Wang Z., Yu Z., Wang Y., Yen N. Y., Huang R. & Zhou X. 2015. Mobile crowd sensing and computing: the review of an emerging human-powered sensing paradigm. ACM Computing Surveys (CSUR) 48(1), 7.

    Google Scholar

    Hamari J. 2013. Transforming homo economicus into homo ludens: a field experiment on gamification in a utilitarian peer-to-peer trading service. Electronic Commerce Research and Applications 12(4), 236–245.

    Google Scholar

    Higgins C. I., Williams J., Leibovici D. G., Simonis I., Davis M. J., Muldoon C., van Genuchten P., O’Hare G. M. P. & Wiemann S. 2016. Citizen OBservatory WEB (COBWEB): a generic infrastructure platform to facilitate the collection of citizen science data for environmental monitoring. International Journal of Spatial Data Infrastructures Research 11(1), 20–48.

    Google Scholar

    Howe J. 2006a. Crowdsourcing: a definition. Retrieved 3 June 2006 from http://crowdsourcing.typepad.com/cs/2006/06/crowdsourcing_a.html.

    Google Scholar

    Howe J. 2006b. The rise of crowdsourcing. Wired Magazine 14(6), 1–4.

    Google Scholar

    Kahneman D. 2003. Maps of bounded rationality: psychology for behavioral economics. The American Economic Review 93(5), 1449–1475.

    Google Scholar

    Koutsoupias E. & Papadimitriou C. 2009. Worst-case equilibria. Computer Science Review 3(2), 65–69.

    Google Scholar

    Mao A., Kamar E., Chen Y., Horvitz E., Schwamb M. E., Lintott C. J. & Smith A. M. 2013. Volunteering versus work for pay: incentives and tradeoffs in crowdsourcing. In First AAAI Conference on Human Computation and Crowdsourcing.

    Google Scholar

    McLeod S.A. 2007. BF Skinner: Operant conditioning. Retrieved September 9, 2009.

    Google Scholar

    Morschheuser B., Hamari J. & Koivisto J. 2016. Gamification in crowdsourcing: a review. In 2016 49th Hawaii International Conference on System Sciences (HICSS), 4375–4384. IEEE.

    Google Scholar

    Muller C.L., Chapman L., Johnston S., Kidd C., Illingworth S., Foody G., Overeem A. & Leigh R.R. 2015. Crowdsourcing for climate and atmospheric sciences: current status and future potential. International Journal of Climatology 35(11), 3185–3203.

    Google Scholar

    Myerson R. B. 1981. Optimal auction design. Mathematics of Operations Research 6(1), 58–73.

    Google Scholar

    Myerson R. B. 2013. Game Theory. Analysis of Conflict. Harvard University Press.

    Google Scholar

    Nash J. 1951. Non-cooperative games. Annals of Mathematics 54, 286–295.

    Google Scholar

    Nisan N., Roughgarden T., Tardos E. & Vazirani V. V. 2007. Algorithmic Game Theory, 1. Cambridge University Press.

    Google Scholar

    O’Grady, M. J., Muldoon, C., Carr, D., Wan, J., Kroon, B. & O’Hare, G. M. P. 2016. Intelligent Sensing for Citizen Science - Challenges and Future Directions. Mobile Networks and Applications 21(2), 375–385.

    Google Scholar

    Parkes D. C. & Wellman M. P. 2015. Economic reasoning and artificial intelligence. Science 349(6245), 267–272.

    Google Scholar

    Petersen K., Vakkalanka S. & Kuzniarz L. 2015. Guidelines for conducting systematic mapping studies in software engineering: an update. Information and Software Technology 64, 1–18.

    Google Scholar

    Rabin M. 2002. A perspective on psychology and economics. European Economic Review 46(4), 657–685.

    Google Scholar

    Roughgarden T. 2003. The price of anarchy is independent of the network topology. Journal of Computer and System Sciences 67(2), 341–364.

    Google Scholar

    Roughgarden T. 2010. Algorithmic game theory. Communications of the ACM 53(7), 78–86.

    Google Scholar

    See L., Fritz S., Dias E., Hendriks E., Mijling B., Snik F., Stammes P., Vescovi F. D., Zeug G., Mathieu P.-P., Desnos Y.-L. & Rast M. 2016a. Supporting earth-observation calibration and validation: a new generation of tools for crowdsourcing and citizen science. IEEE Geoscience and Remote Sensing Magazine 4(3), 38–50.

    Google Scholar

    See L., Mooney P., Foody G., Bastin L., Comber A., Estima J., Fritz S., Kerle N., Jiang B., Laakso M., Liu, H.-Y., Milčinski, G., Nikšič, M., Painho, M., Pődör, A., Olteanu-Raimond, A.-M. & Rutzinger, M. 2016b. Crowdsourcing, citizen science or volunteered geographic information? The current state of crowdsourced geographic information. ISPRS International Journal of Geo-Information 5(5), 55.

    Google Scholar

    See L., Schepaschenko D., Lesiv M., McCallum I., Fritz S., Comber A., Perger C., Schill C., Zhao Y., Maus V., Siraj, M.A., Albrecht, F., Cipriani, A., Vakolyuk, M., Garcia, A., Rabia, A.H., Singha, K., Marcarini, A.A., Kattenborn, T., Hazarka, R., Schepaschenko, M., van der Velde, M., Kraxner, F. & Obersteiner, M. 2015. Building a hybrid land cover map with crowdsourcing and geographically weighted regression. ISPRS Journal of Photogrammetry and Remote Sensing 103, 48–56.

    Google Scholar

    Senaratne H., Mobasheri A., Ali A. L., Capineri C. & Haklay M. 2017. A review of volunteered geographic information quality assessment methods. International Journal of Geographical Information Science 31(1), 139–167.

    Google Scholar

    Shaw A. D., Horton J. J. & Chen D. L. 2011. Designing incentives for inexpert human raters. In Proceedings of the ACM 2011 Conference on Computer Supported Cooperative Work, 275–284. ACM.

    Google Scholar

    Shah N. B. & Zhou D. 2015. Double or nothing: multiplicative incentive mechanisms for crowdsourcing. Advances in Neural Information Processing Systems 1, 1–9.

    Google Scholar

    Shah-Mansouri H. & Wong V. W. S. 2015. Profit maximization in mobile crowdsourcing: a truthful auction mechanism. In2015 IEEE International Conference on Communications (ICC), 3216–3221. IEEE.

    Google Scholar

    Shoham Y. 2008. Computer science and game theory. Communications of the ACM 51(8), 74–79.

    Google Scholar

    Singla A. & Krause A. 2013. Truthful incentives in crowdsourcing tasks using regret minimization mechanisms. In Proceedings of the 22nd International Conference on World Wide Web, 1167–1178. International World Wide Web Conferences Steering Committee.

    Google Scholar

    Stewart O., Lubensky D. & Huerta J. M. 2010. Crowdsourcing participation inequality: a SCOUT model for the enterprise domain. In Proceedings of the ACM SIGKDD Workshop on Human Computation, 30–33. ACM.

    Google Scholar

    Tran-Thanh L., Stein S., Rogers A. & Jennings N. R. 2014. Efficient crowdsourcing of unknown experts using bounded multi-armed bandits. Artificial Intelligence 214, 89–111.

    Google Scholar

    Tversky A. & Kahneman D. 1992. Advances in prospect theory: cumulative representation of uncertainty. Journal of Risk and uncertainty 5(4), 297–323.

    Google Scholar

    Vickrey W. 1961. Counterspeculation, auctions, and competitive sealed tenders. The Journal of finance 16(1), 8–37.

    Google Scholar

    Von Neumann J. & Morgenstern O. 1944. Theorem 3.1.18, Theory of games and economic behavior, Princeton University Press.

    Google Scholar

    Wang L., Zhang D., Wang Y., Chen C., Han X. & M’hamed A. 2016. Sparse mobile crowdsensing: challenges and opportunities. IEEE Communications Magazine 54(7), 161–167.

    Google Scholar

    Yang D., Xue G., Fang X. & Tang J. 2012. Crowdsourcing to smartphones: incentive mechanism design for mobile phone sensing. In Proceedings of the 18th Annual International Conference on Mobile Computing and Networking, 173–184. ACM.

    Google Scholar

    Yang J., Adamic L. A. & Ackerman M. S. 2008. Crowdsourcing and knowledge sharing: strategic user behavior on taskcn. In Proceedings of the 9th ACM Conference on Electronic Commerce, 246–255. ACM.

    Google Scholar

    Zhang Y. & Van der Schaar M. 2012. Reputation-based incentive protocols in crowdsourcing applications. In FOCOM, 2012 Proceedings IEEE, 2140–2148. IEEE.

    Google Scholar

    Zhao Y. & Han Q. 2016. Spatial crowdsourcing: current state and future directions. IEEE Communications Magazine 54(7), 102–107.

    Google Scholar

  • Cite this article

    Conor Muldoon, Michael J. O’Grady, Gregory M. P. O’Hare. 2018. A survey of incentive engineering for crowdsourcing. The Knowledge Engineering Review 33(1), doi: 10.1017/S0269888918000061
    Conor Muldoon, Michael J. O’Grady, Gregory M. P. O’Hare. 2018. A survey of incentive engineering for crowdsourcing. The Knowledge Engineering Review 33(1), doi: 10.1017/S0269888918000061
    shu
Download PDF

Article Metrics

Article views(65) PDF downloads(86)

Other Articles By Authors

ORIGINAL RESEARCH   Open Access    

A survey of incentive engineering for crowdsourcing

  • 1.

    School of Computer Science

  • 2.

    School of Computer Science

  • 3.

    School of Computer Science

  • Published online: 17 April 2018
The Knowledge Engineering Review  33,  2018, 33(1)  |  Cite this article

Abstract: Abstract: With the growth of the Internet, crowdsourcing has become a popular way to perform intelligence tasks that hitherto would be either performed internally within an organization or not undertaken due to prohibitive costs and the lack of an appropriate communications infrastructure. In crowdsourcing systems, whereby multiple agents are not under the direct control of a system designer, it cannot be assumed that agents will act in a manner that is consistent with the objectives of the system designer or principal agent. In situations whereby agents’ goals are to maximize their return in crowdsourcing systems that offer financial or other rewards, strategies will be adopted by agents to game the system if appropriate mitigating measures are not put in place. The motivational and incentivization research space is quite large; it incorporates diverse techniques from a variety of different disciplines including behavioural economics, incentive theory, and game theory. This paper specifically focusses on game theoretic approaches to the problem in the crowdsourcing domain and places it in the context of the wider research landscape. It provides a survey of incentive engineering techniques that enable the creation of apt incentive structures in a range of different scenarios.

    HTML

    Acknowledgments

    • The authors would like to acknowledge the support of the European Union’s Seventh Programme for research, technological development, and demonstration under grant agreement number 308513. Thanks go to the anonymous reviewers for their feedback and comments, which significantly improved the content of the paper.

    Rights and permissions
    • © Cambridge University Press, 2018 2018Cambridge University Press
References (58)
  • About this article
    Cite this article
    Conor Muldoon, Michael J. O’Grady, Gregory M. P. O’Hare. 2018. A survey of incentive engineering for crowdsourcing. The Knowledge Engineering Review 33(1), doi: 10.1017/S0269888918000061
    Conor Muldoon, Michael J. O’Grady, Gregory M. P. O’Hare. 2018. A survey of incentive engineering for crowdsourcing. The Knowledge Engineering Review 33(1), doi: 10.1017/S0269888918000061
    shu

    • Catalog

      /

      DownLoad:  Full-Size Img  PowerPoint
      Return
      Return