Updating my M-Turk priors (or we should be better to M-Turkers)

What seems like a million years ago, I wrote a post about the 10 commandments for running studies on M-Turk . I have since updated my views on this and feel it is important to let others know who use M-Turk HITs in their research.

1. I try to always pay above the national minimum wage (about $7.50 an hour) for each survey or experiment. This works out to about 12.5 cents a minute + the M-Turk fee Amazon includes for using their platform. This is a departure from my previous post which claimed we could/should pay M-Turkers $1.50 per hour. Many of my studies were run in India, and I assumed that this was a fair wage. It’s probably not. Also, I made an assumption that US participants were treating M-Turk as a replacement for solitaire. What I have learned in my research on poverty and government benefits, is that many M-Turkers are using the service as an income supplement. While I find that people will accept HITs that are below 12.5 cents a minute, I do find that I get higher quality responses (and more people passing attention checks) when I pay a fairer wage. To the M-Turkers I argued with about this issue, I can admit, I was wrong.
2. In general, I try to keep any survey done on M-Turk to less than 5 minutes. Since many of the M-Turk participants are doing these tasks as an income supplement, and they will do very many during the day. So the shorter, and easier (and more fun) the surveys are, the better the data because people are actually paying attention.
3. In Qualtrics, when you launch a survey, it will give you an estimate of how long it takes to complete it. This seems to be a conservative estimate, and I find most M-Turkers will complete your survey about 15% faster than this number. Use that number to decide on payment.
4. If you’re going to add a screener question in which you “kick people out” of your survey, then you run two risks. First, people may realize they were kicked out because of the screener, and attempt to renter the survey and lie about the screener question, then you end up with “non-beer drinkers in a beer drinker survey”. The other concern is that people go onto on of the M-Turk message boards like Turkopticon and complain about you and your survey, and this leads to worse data.
5. To avoid these risks, whenever I do a screener I always send the people that fail the screener into another survey in which the screener question does not matter.
6. M-Turkers are actual people doing actual work, and we should be respectful of them and their time. Interpret this as you will, but any M-Turkers reading this know that I am trying to be better, and I appreciate the work you do. As someone who relies on your inputs to help science, I sincerely thank you.