How It Was Built

Agile, underneath the ceremony, is an empirical methodology, and that's how I like to build. You don't plan the whole thing up front and then run the plan. You start with an objective, build one small piece, test it, and let the result tell you what to build next.

That's how this got made. The objective was clear from the start: writing in my voice, at a level I'd put my name on. The plan for the parts wasn't. I didn't design a system of critics and rules up front. We built one piece, tested it on real work, watched where it broke, and what broke told us what to build or fix next. Then again. And again.

This is that session, annotated. The questions I asked, why I asked them, what we changed, and how we changed it. The point isn't the article. The point is the way of working, so you can run your own.

A note on what's here: client names and private details are scrubbed. Some of my messages had the profanity cleaned out. Everything else is what actually happened.

Fix the system, not the output

The move under everything else. Early on, after I hand-edited an opening myself, I didn't ask for a better draft. I asked:

I want to try and see if we can't adjust how our skills write so we can get closer to how I would write.

That's the whole instinct. A draft is one output. The thing that wrote the draft is the system. When the output is wrong, the cheap reaction is to fix that output. The leverage is fixing the system so it never makes that mistake again.

Almost every change in the session worked this way. A bad pattern showed up in the writing, and instead of just deleting it, we wrote a rule against it, or built a new checker, or recalibrated the standard. The article got better as a side effect. The system got better on purpose.

Takeaway: When you catch the AI doing something wrong, ask what part of your setup let it through, and fix that.

Make it verify, don't let it remember

The AI is a probability engine. It will state things that sound right and aren't. So when a claim could be checked, I made it check.

A critic flagged that a page promised three parts of a document but the example only showed two. I didn't have the AI reason about who was right:

Can you go find the actual [project] document and review it?

It read the real file. The real document had two parts, not three. The page was wrong, so we fixed the page to match the truth, not the other way around. Same move when another work session handed over a summary claiming everything was wired up: I had the AI check the summary against the actual files, and it found a gap the summary missed.

Takeaway: When the system makes a claim it can't back, send it to the source. Don't accept memory, and don't accept another AI's summary, when the real thing is right there.

Calibrate against real samples, not vibes

We decided the writing rubric was missing something about how I actually write. The risk: the AI guesses at "how Brandon writes" and the guess becomes a rule that's subtly wrong. So I made it measure instead of guess.

Can you go benchmark what you just outlined against my two books. My judgment isn't there from document to document to catch it.

It read both books and came back with numbers. One stylistic move I lean on (a certain kind of pause) showed up about once per thousand words, clustered in the dramatic passages and absent from the explaining. That measurement became the rule: not "use this," but "use this much, here, not there." A rule without calibration is a trap, because the AI will either overdo it or underdo it.

Takeaway: Ground a standard in measured frequency from real examples. Quantify how much, not just whether.

Steal the method from another field

The research step was a flat checklist: does this page have a story, a number, a source. It nearly missed real material. I'd been thinking about how a good interview show works:

I rarely listen to podcasts, but when I do it's Diary of a CEO. The host does great research on every person and builds questions that really guide the conversation. I wonder if we can build that mindset into how we review my documents, versus just pure look-for-this-or-that.

So we rebuilt the research agent into a prepared interviewer: it studies the material first, then shows up with real story options and pointed questions instead of a blank checklist. Before changing it, I asked to archive the old version:

Let's make copies and store the old ones, so we can see how we evolved.

Takeaway: The best patterns are usually already proven somewhere else. Borrow them. And version your method before you change it, so you can see the evolution and roll back.

Run the work through its own critics

The first full draft of the article was written with AI, and it read like it.

A lot of this sounds like posts that just get beat up for being written by AI. That's the case here. We wrote this using AI. It's not a knock, but it has tendencies within. So let's run it through the critics.

I pointed at two exact lines that gave it away (throat-clearing transitions that announce what's coming instead of just saying it). The critics didn't just flag those two. They found the whole family of them, about ten, across the draft. We deleted the class, then added the class to the rubric's banned list so it never comes back.

Takeaway: When you spot one instance of a tell, have the system hunt the whole family and ban the class, not just the example.

When a gap keeps slipping through, build a specialist for it

Two checkers (one for voice, one for structure) kept passing writing that still had problems, because the problems were a kind neither was built to catch. So we built new checkers.

One was for analogies and pith that sound sharp but say nothing:

Part of our system should read this like someone with a basic reading level. We try to say something in an analogy, but it misses by being too short-sighted or too simple.

That became a clarity reviewer. On its first run it caught a line that sounded profound and meant nothing.

The other was for assumed knowledge:

We have it read like an 8th grader with a very basic level of knowledge, and have it ask the questions that clear the gaps.

That became a naive reader. It immediately caught a wall of unexplained jargon. Then I ran that same checker on a different AI model, and the second model caught a gap the first one missed. More independent reviewers means more coverage. No single check is enough.

Takeaway: When you keep catching the same class of failure by hand, build a dedicated check for it. Run your checks independently, even across different models.

Trust your gut over a green check

This was the one that mattered most. The article passed every checker. Voice, structure, clarity, the naive reader. All green. And it still didn't sound like me.

That gap is the whole lesson. Every checker we'd built was a violation detector: it hunted for the presence of bad things. None of them checked for the absence of a good thing, the patience of staying with a reader instead of stating a point and moving on. The piece was optimized to pass checks. It wasn't written to mean something.

So we added a new trait to the standard, and we were careful: it had to not re-introduce the transitions we'd just banned, and it had to produce uneven, real elaboration, not a uniform machine-like padding of every point. Before changing the files, I asked:

Could we do a v2 of whatever files we're going to change, so we can test the new versus old?

We snapshotted the old standard, made the change, and set up a comparison.

Takeaway: Passing all your checks is not the same as being right. When the instrument says fine and your gut says no, the gap is a check you haven't built yet. Build it carefully, so it doesn't break the ones you have.

Push back when it overreaches

The AI isn't always right, and it will invent rules or overstate things. Part of running the system is correcting it.

It had decided a story couldn't be reused if it appeared somewhere else. I didn't want a hard block:

We don't need to not use a story if it's been used before. There's nothing wrong with that. We can ask me to confirm.

It had also written that the system "catches what I would catch." That overstated my expertise, and undercut the actual point:

I don't want to say it would catch what I would catch, because I don't have expertise in code. But I know how it should work.

Both got corrected. The reuse rule became surface-and-confirm. The expertise claim became "I know what good looks like and I build the system that enforces it," which is truer and is the thesis of the whole article.

Takeaway: When the system invents an over-rigid rule or overstates a claim, tell it where you actually hold the decision.

How to run your own

None of this took writing code. It took knowing what good looks like, watching for where the output missed, and fixing the system instead of the symptom. The repeatable moves, all visible in the session above:

• Fix the check, not the output. A one-off edit fixes today. Updating the system fixes every time after.
• Name what's wrong specifically, then let the AI generalize. Point at one real instance. Let it find the whole family.
• Make it verify, not remember. Send it to the source. Don't trust its memory, or another AI's summary, when the real thing exists.
• Calibrate against real samples. Measure the trait in real examples. Encode how much, not just whether.
• Steal methods from other fields. The pattern you need is usually proven somewhere else already.
• Build specialist checks, and run them independently. One job each. Run the same check across different models to widen coverage.
• Trust your gut over a green check. A passing check that still feels wrong is a missing check.
• Push back. Correct the rules it invents and the claims it overstates.
• Version the method before you change it. So you can see how it evolved, and roll back if a change backfires.

The article on this site was written by the system this session built. The first draft, also here, is what it produced before any of these fixes. Read the two side by side, then come back to this page. The difference between them isn't a better prompt. It's everything above.