1

I have been working with Psiturk, jspsych and Pybossa for a while. After working on each of it, I experienced that Pybossa is doing the same work what Psiturk is doing, except that Psiturk (with GUI interface designed using jspsych) can be used to integrate directly with Amazon Mechanical Turk.

I am new to all these open source platforms and wish to perform the evaluation of micro-tasks. What each of it is doing is storing the data, giving platform to manage the users, and handle the workflow; what I need is to evaluate the results from these experiments to calculate what we call the Cohen's Kappa and Fleiss' Kappa coefficients. How can this be done?

Daksh Agarwal
  • 167
  • 2
  • 12
  • You could also consider JATOS (www.jatos.org) - disclaimer: I'm the programmer who wrote it. For what you want to do ('storing the data, giving platform to manage the users, and handle the workflow') it's doing the job and it's easy to handle (has a GUI) and easy to install (Win, Mac, Linux, Unix, Docker). – Kris Dec 05 '16 at 13:05
  • @Kris Thanks for this reply. I am surely gonna try this out. But, can u plz let me know how can I evaluate the results, like calculation of confusion matrix, accuracy of the results, calculation of Fleiss' Kappa etc after getting the output from this job. – Daksh Agarwal Dec 06 '16 at 05:41
  • This is not really an answer but inter-rater reliability coefficients seem to be part of data analysis which will regularly not be part of the psiturk/jatos kind of platform. (I don't know about pybossa). The reason for this is that every different experiment will have different needs in terms of how to calculate the coefficients. If you have your output data, you would probably just have to use your statistical software to calculate them. – elisa Dec 06 '16 at 14:15
  • Hi! I'm the developer of PYBOSSA and you can use it as well ;-) We don't have a GUI because we provide templates so you can adap them. For analysis we provide the library Enki, webhooks, allowing you to do real-time analysis. PYBOSSA uses webhooks to notify a small web server when to do statistics, and produce results. This then can be indexed in our full text search API, allowing you to search within your own data. Take a look at our repository, as all the tools are there. Also, Enki uses Pandas, so you have one of the best statistical analysis tools from the scientific community. – Daniel Lombraña González Dec 21 '16 at 09:19
  • 1
    @DanielLombrañaGonzález Thanks for this info. Definitely Pybossa seems to carry a lot of extra functionality which is missing in others. I have been going thru the documentation of Pybossa but I am not getting how exactly I am going to use Webhooks in real time. I tried by adding SSE=True in settings but what needs to be put in webhook link of the project detail? And how this can then be indexed in full text search API? – Daksh Agarwal Dec 22 '16 at 08:03
  • I think you have asked this question in a different thread, and someone has answered you properly. If you still have issues with it, let me know it. – Daniel Lombraña González Jan 03 '17 at 08:30
  • @DanielLombrañaGonzález Thanks. Yes, I have got the reply to my query in this (http://stackoverflow.com/questions/41344811/issue-in-pybossa-webhooks-how-to-execute-it/41438180#41438180) thread. – Daksh Agarwal Jan 03 '17 at 10:47

0 Answers0