CrowdOp is a declarative crowdsourcing system, which is designed to hide the complexities and relieve the user the burden of dealing with the crowd. The user is only required to submit an SQL-like query and the system takes the responsibility of compiling the query, generating the execution plan and evaluating in the crowdsourcing marketplace.
CrowdOp provides an SQL-like query language as a declarative interface to the crowd. An SQL-like declarative interface is designed to encapsulate the complexities of dealing with the crowd and provide the crowdsourcing system an interface that is familiar to most database users. To illustrate a declarative crowdsourcing interface, we consider the three example relations shown in the following figure: the REVIEW table contains car reviews from customers; the AUTOMOBILE table contains car specifications; the IMAGE table contains car pictures.
An example query for finding cars with black color, high-quality images and positive reviews is
For a given query, CrowdOp first compiles the query, generates an execution plan, posts human intelligence tasks (HITs) to the crowd according to the plan, collects the answers, handles errors and resolves the inconsistencies in the answers. While CrowdOp improves the usability of crowdsourcing, it is required to to have the capability to optimize and provide a "near optimal" query execution plan for each query. Compared to the query optimization techniques proposed in recent crowdsourcing systems and algorithms, CrowdOp has the following differences in its design principles:
The architecture of CrowdOp is illustrated in the figure below. An SQL query is issued by a crowdsourcing user and is firstly processed by Query Optimizer, which parses the query and produces an optimized query plan. The query plan is then executed by Crowdsourcing Executor to generate human intelligence tasks (or HITs) and publish these HITs on crowdsourcing platforms, such as Amazon Mechanical Turk (AMT). Based on the HIT answers collected from the crowd, Crowdsourcing Executor evaluates the query and returns the obtained results to the user.
J. Fan, M. Zhang, S. Kok, M. Lu, B.C. Ooi IEEE Transactions on Knowledge and Data Engineering, 2015.