How to use M-Turk to code consensus…

There are many things that M-Turk may not be great about. But, M-Turk is very good for “coding consensus”, i.e., establishing some average sentiment about a piece of information.

This is a short walkthrough for how I did this with one of my research projects along with the HTML code so you can do it yourself.

For the following study, I wanted to code a list of non-profit websites for how much they appealed to a person’s empathy or displayed their donation target’s vulnerablity.

First, I put together a .csv file. In this file I included all the data I had (in this case it was the name of the website (web) and the url for the front page of the site (url), both of which I labeled in the first row with their respective names.

Then I posted an HIT to M-Turk. My goal was to get 10 M-Turkers to indicate how empathetic or vulnerable the appeal was. For a task like this I aim to pay M-Turkers about $7.00 an hour. When I did this task myself it took me about 2 minutes to answer the questions. So I paid each worker .12 USD per HIT.

You can use the same set-up

I used a basic 7-point likert scale that assessed empathy and vulnerability.

Then, after the HIT was complete, I averaged these responses to provide an overall measure of the perceived empathy and vulnerbaility.