In 2016 nobody needs convincing that crowdsourced work is a solution to many problems from data labeling, to gathering subjective opinions, to producing transcripts etc. Turns out it can also work really well for functional software testing - but it's not easy to get right.

One well-known problem with crowdsourcing is sloppy work - where people perform only the absolute minimum actions allowing them to get paid, without actually fulfilling the intended tasks. In many scenarios this can be counteracted by asking multiple workers to complete the same task, but that dramatically increases cost and can still be error-prone. Detecting lazy work is another way to increase quality of gathered data and we have found a way to do this reliably for quite a large variety of tasks.

In this talk I will describe how we have trained a machine learning model to discriminate between good and sloppy work. The outline is as follows: - describe the specific problem we had (5m) - overview of the solution (2m) - ML model details (10m) - data capture - labelling - balancing the dataset - feature engineering - training, retraining - model itself - model persistence (10m) - productizing the result by putting it behind an HTTP API (3m) - limitations, trade-offs, what could we do better (3m)