Lean Experiments

A lean experiment is a small, cheap test of one slice of your idea, run before you build the real thing. Not the whole product. Not even the whole prototype. One slice. The riskiest assumption, the piece that decides whether the rest deserves to exist.

Why slice it at all? Money. Building a solution is expensive. Engineers, designers, months of calendar, all spent before a single customer has proven they care. The experiment's whole job is to buy confidence at a discount. You raise your confidence that people will actually use the thing before you pay full price to build it. That's the heart of Eric Ries's The Lean Startup. Stop guessing, start running build-measure-learn loops, and let validated learning, not your gut, decide what's worth building.

If you're arriving from the Opportunity Backlog, you're holding a hypothesis and the discovery loop you've been walking just said "now test it." If you're coming out of a Decision Jam earlier in this part, where a team votes ideas onto a low-effort, high-impact grid, you walked out holding the sticky note that won, with your name on it. And if you landed here cold, no prior page required, you're still in the right room. This page is the map. The recipe for each individual experiment lives one click away in the Lean Experiments catalog.

Start with the hypothesis you already wrote

Every experiment starts with a hypothesis, and you already know how to write one. If we do this thing, for this user, they'll respond in this measurable way. You wrote yours back in the Opportunity Backlog, so go grab it if you need it. I'm not re-teaching the formula here.

But I'll repeat one distinction, because it gets butchered in meetings every week. Someone says "my hypothesis is..." and then puts out an assumption. An assumption is something you believe is true without proof. A hypothesis is testable: this action, this user, this outcome, this number.

If it's not testable, it's just a guess. Guesses don't get experiments. Guesses get rewritten until they're hypotheses.

"But our customers already said they'd love it"

I know. You did the interviews. The survey came back glowing. A roomful of people nodded. Why spend two more weeks proving what everyone already told you?

Because people will tell you one thing and then do another.

That's not me being cynical about your customers. People are polite. People are optimistic on your behalf. People are irrational, emotional beings, you and me included. Stated interest is free, and free signals are worth what they cost.

This is exactly why a survey is the wrong tool when the question is desirability. A survey collects opinions, and opinions are cheap. An experiment changes the question entirely. It never asks "would you use this?" It puts something in front of real people and watches whether they DO a thing instead of SAY a thing. And the doing has to cost them something. A click, an email address, thirty seconds of attention, money. Back in Test & Validate you saw it put plainly. What a person is willing to spend on your idea is the signal. When someone spends, you've learned something. When someone agrees, you've learned they're nice.

Match the experiment to your question

Most people reach for whichever artifact would be the most fun to build instead of the one that answers their actual question, and then quietly resent it when the test tells them nothing.

Don't do that. Figure out which question you're really asking, then pull the cheapest experiment off that shelf. Most ideas carry a stack of assumptions, and the riskiest one is whichever kills the whole idea if it's wrong. For a brand-new idea, that's almost always desirability. Anyone can get excited about building a thing. Evidence that anyone wants it is the rare part, so that shelf comes first.

There are four questions an experiment can answer, and each one points you at a different family of tests:

Those names in the right column (smoke test, fake-door, concierge, Wizard of Oz, and the rest) are just labels for specific recipes. You're not expected to recognize them here. Each one gets defined and walked through in the catalog. The taxonomy, and the deeper catalog of named experiments, comes from David Bland and Alex Osterwalder's Testing Business Ideas. Their book maps out more than forty of these tests by the kind of evidence each one buys you. The full recipe for each one (how to run it, what to measure, what good looks like) lives in the Lean Experiments catalog. This page is just the map that gets you to the right shelf.

One rule sits over all four shelves. Start with low fidelity. Fidelity is just how real the artifact is, paper sketch at one end, working software at the other, and it costs money the more real it gets. Only add fidelity when you need more specific feedback. Don't build an app. Build a paper sketch of the app, and only build more when the cheap version stops answering your question.

Five steps, two weeks

Designing a lean experiment takes five steps. Write them down.

1. Write the hypothesis. You know how. Testable, or it doesn't leave this step.
2. Choose the experiment type. Match it to the question, not to the artifact you'd most enjoy making. That's the table above.
3. Design and launch it. Three decisions before anything goes live. Who's the audience? What's the success criteria, as a number, written down before launch so it can't bend afterward? And what's the end date? An experiment without an end date is a pet project with a cooler name.
4. Track the data. While it runs, not after.
5. Present the results and decide: refine, pivot, or proceed.

One scoping rule holds it all together. Design the experiment so the build and launch fit inside about two weeks, never more than a month. Past a month, you've stopped experimenting and quietly started building the thing you were supposed to be testing. And the part that trips people up is this. Depending on the experiment, it may not need any engineering at all. Say your team runs a standing 25-minute live status meeting and you suspect a short written summary, sent ahead of time, would do the same job for less. The whole idea rests on one assumption. The executive will actually read it. The smallest test of that is to write a two-page version, email it before the next meeting, and see if they show up having read it. An afternoon, not a sprint. Experiments aren't just for software. They're for anything you'd rather test than assume.

This is also the right place to remember what the experiment is for. An MVP, a minimum viable product, is still a product. You ship it, you sell it, you support it. What you're building here is smaller than that, and it has a different name. The smallest thing you can put in front of a real customer just to learn is the minimum viable experience (MVE). You're not shipping a tiny product. You're buying the cheapest piece of evidence that tells you whether the real thing is worth building at all.

Please don't

Two temptations show up every time I teach this.

You're going to want to do a UX/UI evaluation. Please don't. A panel grading your screens produces opinions about polish, and polish isn't your question yet. Whether anyone wants the thing has nothing to do with how pretty the buttons are.

Some lists of lean experiments include a group workshop as an entry. I don't recommend running one either. A room full of people discussing your idea is stated interest with catering. More saying, still no doing.

Both temptations are the same mistake at different sizes. You add apparatus, a panel, a workshop, instead of getting one real user to do one real thing. Every panel you bolt on is one more reason nobody ever runs the test.

Keep it simple, and remember 80/20. Most of the learning comes from a small slice of the effort. Some of the best experiments are the simplest. You're testing the very smallest concept in the quickest, cheapest way possible, and every extra data point you bolt on makes the experiment slower to run and blurrier to read. If your experiment design needs its own kickoff meeting, it stopped being lean. Shrink it.

When the data comes back

Every experiment ends one of three ways. Refine. The signal is real but this version of the solution needs another pass, so fix what's broken and run it again. Pivot. The data says this version of the idea isn't it, so change direction and write the next hypothesis. Proceed. The evidence is strong, and the idea just earned the next, bigger investment.

Read that list again. Only one of the three paths says build. That's not failure, that's the process working. You paid two weeks instead of six months to find out.

There's a fourth ending nobody writes down. The data said stop and the project kept going. Refine and pivot are both the data saying stop building this version. When the data says stop, it means actually stop. There are entirely too many pet projects in the corporate world that keep going because they're somebody's strategic initiative, somebody's portfolio line, somebody's baby. The whole point of writing the success criteria before launch is that you can't renegotiate them after.

Aim that same suspicion at your own numbers. A pile of app downloads with no active users isn't evidence of anything. You met vanity metrics back in the Opportunity Backlog, numbers that look good and prove nothing, and experiments are where they do the most damage. Measure a number that flatters the idea, and the experiment stops testing it and starts defending your ego.

So run one. Take the hypothesis you wrote, or the winning sticky from your last Decision Jam, or the idea that's been stuck in committee for a month. Find the question underneath it: desirability, usability, feasibility, or messaging. Pull the cheapest experiment off that shelf, and open the catalog for the recipe. Write down the audience, the success number, and the end date before you launch anything. Cap the build at two weeks.

Then do what the data says. Even when it says stop. Especially when it says stop.