“The trouble with Polly was that she had a mind that asked questions even when she really, really didn’t want to know the answers.”
– Terry Pratchett, Monstrous Regiment

I typically find it takes me longer to design good MCQs in comparison to other forms of assessment; however, with large classes there is a huge saving in time on the marking – far more than the extra time needed to design the assessment. Also, this might be only me, but designing and developing the assessment questions is much more fun than marking over 100 tests.

To design better MCQs let us first have a look at the key parts of a MCQ. Figure 1 shows the generally accepted terms for the parts of an MCQ; the stem is the question we are asking, the options are all the possible answers, while the key is the correct answer and the distractors are the incorrect answers.

Figure 1. Anatomy of a MCQ.

Let us take for example the stem “What is the gas constant (8.31 J K^–1 mol^–1) in ft³ psia ^oR^–1 lb-mol^–1?”. The answer to this question is 10.73 which is calculated from a number of unit conversions. We might automatically think therefore that good distractors would be 107.3 or 1.0725, but if we look at the conversions neither of these would likely appear. In this case some good options could be,

• 42.73 (including the +32 when converting form K to ^oR as would be used if converting ^oC to ^oF)
• 1.00 (converting ft to m instead of ft³ to m³)
• 10725.00 (using kPa instead of Pa in the conversion)

Of course, in this case I have found that if an open assessment this is now a useless question as chatgpt was able to just give me the answer with the conversion steps included. I could find the answer in google, but for me it first appeared on the second page and I only spotted it because I knew the answer. Though the use of AI with assessment is another discussion entirely!

Some other key points for the questions design:

• Try to avoid trivial facts and focus on concepts, application, and critical thinking as much as possible.
• Use simple and precise language to avoid unnecessary complexity or ambiguity in the stem. For example avoid negative phrasing e.g. “Which of the following is NOT a function of a control valve?” is better written as “What is the primary purpose of a control valve in a chemical process?“.
• Make sure one option is the clear correct answer, with the distractors being reasonable, but not absurd. Also avoid things like “All of the above” or “None of the above” as it can sometimes make guessing easier. For example to the last question, bad options would be “Flow control, Banana (Clearly incorrect), Flowing liquid is controlled (Too similar to correct answer), All of the above (clearly can’t be because of banana)” whereas good options would be “Temperature control, Flow control, Density measurement, Heat exchange”.
• Avoid absolute terms as words like always, never, only can make answers obvious.
• Be careful with grammatical clues. For example, “Albert Einstein was a: Physicist, Anthropologist, Astronomer, Chemist” contains an accidental clue that it is likely not anthropologist or astronomer as the determiner for these should be “an” rather than “a”. Therefore, this would be an improvement, “Who was Albert Einstein? An anthropologist, An Astronomer, A chemist, A physicist”.
• Avoid patterns in the answers, it is best to randomize answers to prevent guessing strategies – most MCQ software has a function to randomise the presentation of the answers now anyway.
• Aim for 2 or 3 distractors. I was quite surprised to read in Raymond et al. (2019) that if well designed 2 distractors is often just as good as more. The more distractors we have the more effort it takes to make them plausible, and thus they end up being non-functional (not selected), so essentially pointless being there.

Scoring can also be used to prevent guessing of the answers. Large numbers of strategies have been suggested in the literature. Providing a negative mark, or portion of a negative mark for an incorrect answer can prevent guessing and make the expected score in the test equal to that of the student “true” knowledge. Personally, I tend to not have any negative scoring on an MCQ, but if you want to explore scoring more, then I would recommend Kanzow et al. (2023).

Summary

When designing MCQs try to think about the distractors and how they fit in with the stem, this is key to good questions, also try to remember:

References

M. R. Raymond, C. Stevens, and S D. Bucak (2019) “The optimal number of options for multiple‑choice questions on high‑stakes tests: application of a revised index for detecting nonfunctional distractors”, Advances in Health Sciences Education, 24:141-150.
A. F. Kanzow, D. Schmidt, and P. Kanzow (2023) “Scoring Single-Response Multiple-Choice Items:Scoping Review and Comparison of Different Scoring Methods”, JMIR Medical Education, 9: e44084.