Bias in a Bag

Why this works

A rule feels trustworthy the moment it’s written down — the writing makes it look finished. This activity separates the two things that are actually independent: whether a rule is well-reasoned, and whether the data it was built from looked like the world it gets applied to. Kids can feel the gap between those two the moment they count their misses.

How it works

1. Before anyone sees the full bag, each group draws a sample of 8 buttons and looks only at those.
2. From the sample alone, each group writes one rule for sorting buttons into “keep” and “set aside” — for example, “keep anything blue.”
3. Reveal the full bag. Each group applies its rule to every button inside it, one at a time.
4. Count how many buttons the rule sorted the way a person would have chosen by hand, looking at the whole bag.
5. Compare scores across groups. The groups whose 8-button sample happened to resemble the full bag will have scored far better — through luck, not a better rule.

Talk about it

• Was your rule a bad rule, or was your sample just unlucky? How would you tell the difference?
• If a school built a rule for everyone from what worked for one small pilot group, who might that rule quietly leave out?
• What would you have needed to see, before writing your rule, to trust it more?