How robust is this as an assessment of mastery in the discipline? As a simple model, let's say that any student definitely knows how to answer N types of questions, but is randomly guessing (uniform distribution over 4 options) for the other questions. Obviously this abstracts out the possibility that some students know parts of certain questions and can eliminate certain choices or guess based on the overall "shape" of the question, but it's a reasonable first-degree model. Then the chance to pass the exam for different levels of knowledge is as follows:
Obviously, if a student really "knows" how to answer 15 or more questions, then they will certainly pass this test (omitted from the table). But even if they only know half of the material in the course, then they will probably pass the test (12 questions known: 67% likely to pass). Of students who only ever know about one-third of the basic algebra content, but retake the class 3 times, about half can be expected to pass based on the strength of random guessing on the rest of the test (9 questions known: 19% likely to pass; over 3 attempts chance to pass is 1-(1-0.19)^3 = 1-0.53 = 0.47).
