Society & AI — Academy Classroom Kit
societyandai.org/academy/classroom-kits
Unplugged Simulation
Bias in a Bag
Write a sorting rule from a small sample, then watch it misfire on the full bag — a hands-on first encounter with unrepresentative data, using nothing but buttons.
Learning objective
Students can explain, in their own words, why a rule built from a small or unrepresentative sample can fail on the larger group it's meant to describe.
What you need
- A bag or box, not see-through
- About 40 mixed buttons or beads — at least 3 colors, in uneven proportions
- Index cards and pencils
Why this works
A rule feels trustworthy the moment it’s written down — the writing makes it look finished. This activity separates the two things that are actually independent: whether a rule is well-reasoned, and whether the data it was built from looked like the world it gets applied to. Kids can feel the gap between those two the moment they count their misses.
How it works
- Before anyone sees the full bag, each group draws a sample of 8 buttons and looks only at those.
- From the sample alone, each group writes one rule for sorting buttons into “keep” and “set aside” — for example, “keep anything blue.”
- Reveal the full bag. Each group applies its rule to every button inside it, one at a time.
- Count how many buttons the rule sorted the way a person would have chosen by hand, looking at the whole bag.
- Compare scores across groups. The groups whose 8-button sample happened to resemble the full bag will have scored far better — through luck, not a better rule.
Talk about it
- Was your rule a bad rule, or was your sample just unlucky? How would you tell the difference?
- If a school built a rule for everyone from what worked for one small pilot group, who might that rule quietly leave out?
- What would you have needed to see, before writing your rule, to trust it more?
Concept: Data
← All Classroom Kits
Free to copy and reuse for classroom, tutoring, and family use.