Acquiring High Quality Non-Expert Knowledge from On-Demand Workforce
نویسندگان
چکیده
Being expensive and time consuming, human knowledge acquisition has consistently been a major bottleneck for solving real problems. In this paper, we present a practical framework for acquiring high quality non-expert knowledge from on-demand workforce using Amazon Mechanical Turk (MTurk). We show how to apply this framework to collect large-scale human knowledge on AOL query classification in a fast and efficient fashion. Based on extensive experiments and analysis, we demonstrate how to detect low-quality labels from massive data sets and their impact on collecting high-quality knowledge. Our experimental findings also provide insight into the best practices on balancing cost and data quality for using MTurk.
منابع مشابه
The Impact of Trade Liberalization on the Ratio of Educated Labor Demand
Today with obsolescence policy of import substitution industrialization, countries are opening their commercial doors to each other. But one of the problems of developing countries is high rate of educated unemployed people. Iran is facing tow phenomena. The first is High rate of educated unemployed people and the second is the increasing in the participation rate. In this survey with informati...
متن کاملOutsourcing Expert Services by State Transportation Departments: A Look at Effects on Cost, Quality, and Changing Employment Levels
Privatization has increasingly become a policy option for government agencies struggling to meet rising demands for services but with fewer resources. In the transportation arena, many state departments of transportation (DOTs) have privatized by outsourcing highway functions to the private sector. But the outsourcing of technical and expert services such as those related to the design and cons...
متن کاملA systematic approach to chronic heart failure care: a consensus statement.
The National Heart Foundation of Australia assembled an expert panel to provide guidance on policy and system changes to improve the quality of care for people with chronic heart failure (CHF). The recommendations have the potential to reduce emergency presentations, hospitalisations and premature death among patients with CHF. Best-practice management of CHF involves evidence-based, multidisci...
متن کاملAddressing the challenges of diagnostics demand and supply: insights from an online global health discussion platform
Several barriers challenge development, adoption and scale-up of diagnostics in low and middle income countries. An innovative global health discussion platform allows capturing insights from the global health community on factors driving demand and supply for diagnostics. We conducted a qualitative content analysis of the online discussion 'Advancing Care Delivery: Driving Demand and Supply of...
متن کاملSoftware Creation: Using Specification and Description Language (SDL) for Capturing and Reusing Human Experts' Knowledge in Software Design
Conventional knowledge engineering techniques for acquiring experts’ knowledge can not produce quality knowledge due to improper knowledge documentation and informal knowledge acquisition method. We propose a method for knowledge acquisition based on documentation using Specification and Description Language (SDL). SDL is used to describe both the target system and the design process. The main ...
متن کامل