I find that students in my statistics class are almost totally bewildered by the logic of hypothesis testing and P-values (for hypotheses based on a population mean), no matter how carefully I try to explain the concepts. Here's an idea for a super-short and simple, concrete demonstration of hypothesis testing. Tell me if you think this would be worth the class time:
- Start with a hand of four cards: {A, 2, 3, 4}
- I'll turn my back and secretly do one of two things:
H0: Leave the Ace in, or
HA: Take the Ace out
- Now shuffle the hand and deal out 3 cards.
Question: Say I get a draw of {2, 3, 4}. What's the chance of this happening if I did
not take out the Ace (H
0)? Note that all possible draws would be {{A,2,3}, {A,2,4}, {A,3,4}, {2,3,4}} so the probability of seeing that would be P = f/N = 1/4 = 0.25.
Conclusion: If I draw {2,3,4} then we have some evidence that I did change the deck (H
A) -- because it's unlikely to see that result if I didn't (P = 0.25).
Now -- You can actually demonstrate this and ask the class if they think I left the Ace in or took it out each time. I'd recommend 3 run-throughs: leave it, leave it, then take it out. (In the latter case, also ask: Is it
possible that I left the Ace in?) In reality, you should probably hold the cards against the otherwise full box, so it isn't obvious if your hand becomes empty in the take-it-out case. (And otherwise practice the prestidigitation in advance so your handwork doesn't give it away.)
Open Question: Should I actually reveal to the class which one I did each time (for confirmation), or leave that as a mystery (modeling real-world usage)?