How do I validate the survey completion code in mTurk and disallow submission if they don't put in correct code?

Question

I am working on mTurk and qualtrics and sending over mTurk workerID to qualtrics and at the end of the qualtrics survey I display the worker id as a confirmation code. This code the turker should paste back in the textbox in the mTurk HIT. My problem is once he writes back the worker id in the HIT, how do i validate that he is entering the code he saw in the Qualtrics survey and not just entering any number? I am using javacript in the HIT to send workerid to Qualtrics.

Answer 1

A couple of possibilities, with (2) probably being the best:

1. You can generate the completion code using Javascript in the HIT and pass this code to Qualtrics as a URL query string parameter. Then, have Qualtrics extract the completion code and show it to the worker at the end of the survey. When they enter the code, you can have a Javascript function that checks whether the code entered matches the code originally sent to Qualtrics and generate a hidden form field that's called something like correct and enter values of "yes" or "no". You could then approve everyone who matches.
2. You could do everything as in (1), except configure an Assignment-level ReviewPolicy that automatically rejects assignments where correct is equal to "no".
3. You could give everyone in the survey the same code (simply displayed in Qualtrics), and then use an Assignment-level ReviewPolicy to reject based directly on the value of the code. (You can't setup a "dynamic" review policy with a different correct answer for each assignment, so this is probably the least desirable option.)

Using any of these methods, you could simply alert the worker that they have entered the incorrect code. You cannot prevent submission with a HIT created in the Requester User Interface. If you instead setup an ExternalQuestion (hosted on your own server) or an HTMLQuestion, either created via the API, then you can prevent submission by only showing the submit button when the code is correct. Note, however, that this will probably annoy some workers.

Answer 2

I'd strongly warn against using #2. Rejections will get you bad Turkopticon reviews and potentially sink your study.

I'd also strongly warn against #3 as members of certain Turker groups will share the code and not do the survey.

The first option is definitely your best bet, and there are tons of form field validation javascript code snippets out there to get the job done. Along with that, alerting the worker when a bad code has been submitted is also a good idea as it gives them the chance to either get it again from the survey, or take the survey again. I like Thomas' idea of only showing the submit button when the code is correct. As long as it works correctly (it's validating the correct code the right way), then it shouldn't annoy any workers who aren't cheating.

Answer 3

For step-by-step instructions for generating individual confirmation codes, I followed Tyler Burleigh's write-up. It's option 1 of Thomas's answer.

https://tylerburleigh.com/mturk/survey-completion-codes-in-qualtrics/