I have a soft spot for interesting design systems. That love traces back to two distinct moments of delight, when I had the privilege of witnessing the birth of what I consider the two greatest design systems ever made: the original Material Design and the original Fluent Design. Note that I am strictly specifying the first iteration here, and by Material Design, I definitely do not mean the version Luke Wroblewski later butchered. What made those two systems stand out back then was their brilliant balance of playfulness, visual richness, and accessibility, all while leaving plenty of expressive room for skilled designers, without turning every piece of software on the market into the exact same flavor of instant ramen.

Behind both was a genuinely sublime fusion of technology and the liberal arts.

Let's start with a game. Suppose we want to study the factors that influence a person's body weight. I'll invite you to come up with three possible factors. Here are mine: daily caloric intake, physical activity level, and basal metabolic rate.

This concept has been ruined by almost every statistics teacher.

Here is a simple piece of common sense: a system of three linear equations in three unknowns needs three equations (three conditions) to have any chance of a unique solution. When there are more parameters to solve for than conditions, you end up with infinitely many solutions.

Last term, a UX researcher from Google taught our graduate Research Methods course. She gave us free rein to pick any product we fancied and guided us through a full three-month user experience research cycle from soup to nuts. I thoroughly enjoyed every minute of it, eventually walking away with a clean 100% in a course where the class average hovered in the mid-80s. Rigorous, textbook-style usability research isn’t something you run across every day, so I figured it was worth documenting properly.

I chose Jamovi, an open-source statistical package. Part of it was my long-standing appetite for data analysis, but part of it was my identity as an open-source contributor. Dedicated UX research in open-source scientific software is practically non-existent. Giving back to that ecosystem felt like a genuinely worthwhile endeavor.

Today I want to talk about the problem that the significance testing system is trying to solve.

Please note that this piece is entirely framed within Fisher's vocabulary. I have no intention of dragging the Neyman-Pearson framework into the discussion and complicating everything. I understand that for many readers, making sense of two sets of conditional probabilities is painful, and there is no reason to invite that trouble here. If you want to discuss this piece with an LLM, paste this note in full and strictly instruct your LLM not to introduce Pearson's terminology partway through.