Evaluate exam in moodle with a custom evaluation scheme

Question

I'm trying to implement a simple evaluation scheme using exams, but none of the options seems to do what I'd like:

There are 5 answer options. I want to give 0.2 points for all marked correct answers and all unmarked incorrect answers, and zero points for all unmarked good answers and marked incorrect answers. Therefore, a task can yield 0, .2, .4, .6, .8, or 1 points.

I'm aware that this evaluation scheme may have some shortcomings, but I'm counterbalancing those in other ways.

I was able to implement this when I did scanned exams, because I could use a string distance function to tell how many characters differ in two strings that coded the answers and the solutions. But I want to do this in moodle now, so I cannot control the evaluation.

Here are some examples that I tried:

ee <- exams_eval(partial = TRUE, rule = "all", negative = FALSE)
ee$pointsum("01111", "10000") # should be 0 and returns 0
ee$pointsum("01111", "10001") # should be .2 but returns 0
ee$pointsum("11111", "10001") # should be .4 and returns .4
ee$pointsum("00000", "11001") # should be .4 but returns 0
ee$pointsum("11011", "00011") # should be .6 but returns .5
ee$pointsum("11111", "10101") # should be .6 and returns .6
ee$pointsum("11001", "10001") # Should be .8 but returns .66
ee$pointsum("00000", "00001") # should be .8 but returns 0
ee$pointsum("11001", "11001") # Should be 1 and returns 1
ee$pointsum("00000", "00000") # Should be 1 but returns 0

The previous examples yield the same result when using rule = "false" or rule = "false2", or rule = "true". When using rule = "none", this is the only change:

ee$pointsum("01111", "10001") # should be .2 but returns 0.25

Is there a way to implement the above mentioned evaluation scheme in moodle?

Answer 1

Strangely, there is no combination of exams_eval parameters that would give the desired scheme. One can verify that by checking all 20 combinations. For instance,

combs <- expand.grid(partial = c(TRUE, FALSE), negative = c(TRUE, FALSE),
                     rule = c("false2", "false", "true", "all", "none"),
                     stringsAsFactors = FALSE)
sapply(1:nrow(combs), function(n) {
  ee <- exams_eval(partial = combs[n, 1], negative = combs[n, 2], rule = combs[n, 3])
  ee$pointsum("11001", "10001")
})
#  [1]  0.6666667 -1.0000000  0.6666667  0.0000000  0.6666667 -1.0000000  0.6666667  0.0000000
#  [9]  0.6666667 -1.0000000  0.6666667  0.0000000  0.6666667 -1.0000000  0.6666667  0.0000000
# [17]  0.6666667 -1.0000000  0.6666667  0.0000000

---

That's particularly strange because the scheme is quite simple - counting matches, which we may implement as follows:

pointsum <- function(correct, answer) {
  correct <- as.numeric(strsplit(correct, "")[[1]])
  answer <- as.numeric(strsplit(answer, "")[[1]])
  mean(correct == answer)
}

pointsum("01111", "10000") # should be 0
# [1] 0
pointsum("01111", "10001") # should be .2
# [1] 0.2
pointsum("11111", "10001") # should be .4
# [1] 0.4
pointsum("00000", "11001") # should be .4
# [1] 0.4
pointsum("11011", "00011") # should be .6
# [1] 0.6
pointsum("11111", "10101") # should be .6
# [1] 0.6
pointsum("11001", "10001") # should be .8
# [1] 0.8
pointsum("00000", "00001") # should be .8
# [1] 0.8
pointsum("11001", "11001") # should be 1
# [1] 1
pointsum("00000", "00000") # should be 1
# [1] 1

Answer 2

The R/exams package currently does not support the desired evaluation scheme because Moodle does not support it. Looking at the Moodle docs at https://docs.moodle.org/36/en/Moodle_XML_format#Multiple_choice shows that you can see that partial credit schemes always work in the following way:

• Not marking/clicking an <answer> does not yield any points.
• Marking/clicking an answer option yields a certain fraction of the overall points.

Hence, R/exams handles this by assigning the fraction 1/#correct to marking/clicking a correct answer. The rule argument only controls which fraction is subtracted when marking/clicking an incorrect answer. The default is the "false2" rule that essentially subtracts 1/#incorrect. For example, an item with 2 correct and 3 incorrect answers is processed with:

ee2 <- exams_eval(partial = TRUE, rule = "false2", negative = FALSE)
ee2$pointvec("11000")
##        pos        neg 
##  0.5000000 -0.3333333 

When you use rule = "all" then 100% of the points are removed if an incorrect answer is marked/clicked:

ee <- exams_eval(partial = TRUE, rule = "all", negative = FALSE)
ee$pointvec("11000")
##  pos  neg 
##  0.5 -1.0 

There are learning management systems that support more flexible ways of computing the points (e.g., in QTI this is in principle possible) but I don't think your particular scheme can be implemented in Moodle. (If anyone knows more than the Moodle docs above, let me know!)

(You said you are aware of the drawbacks of your evaluation scheme - which is, of course, fair enough. However, just for the record in case anybody else reads this: I'm personally not very fond of the scheme you are proposing. Even if on average each answer option is correct with 50% probability, students can obtain 50% of the points on average by either always clicking all of the answer options or by always clicking none of the answer options. This can even get higher if the probability for each options deviates from 50%. Hence, this sets strange incentives for some students...at least the business and economics students I'm typically teaching.)