Gloria Re Calegari, Andrea Fiano and Irene Celino.
Abstract: With the rise of linked data and knowledge graphs, the need becomes compelling to find suitable solutions to increase the coverage and correctness of datasets, to add missing knowledge and to identify and remove errors. For this reason, several approaches – mostly relying on machine learning and NLP techniques – have been proposed to address this refinement goal; they usually need a partial gold standard, i.e. some "ground truth" to train automatic models. Gold standards are manually constructed, either by involving domain experts or by adopting crowdsourcing and human computation solutions.
In this paper, we present an open source software framework to build Games with a Purpose for linked data refinement, i.e. web applications to crowdsource partial ground truth, by motivating user participation through fun incentive. We detail the impact of this new resource by explaining the specific data linking "purposes" supported by the framework (creation, ranking and validation of links) and by defining the respective crowdsourcing tasks to achieve those goals.
To show this resource's versatility, we describe a set of diverse applications that we built on top of it; to demonstrate its reusability and extensibility potential, we provide references to detailed documentation, included an entire tutorial which in a few hours guides new adopters to customize and adapt the framework to a new use case.
Keywords: games with a purpose; human computation; crowdsourcing; linked data refinement; knowledge graph refinement; gold standard collection; software framework; open source software