Today, one level out: the harness is how the project is run

You do not tame the model directly. You tame the process around it.

You already do project management: a class project, a paper submission.

You do not YOLO either one.

Skills: a written procedure for one kind of work, loaded when that work comes up

In a research group: the protocol for submitting a paper, refereeing one, or filing a reimbursement.

Without a procedure, the agent invents how to work

I asked ChatGPT to prepare this talk.

I was invited to give a talk in one of the Friday slots of this course:
https://katestange.net/ai/course/
The class attends, and also the whole math department.
Help me prepare it.

What do you think happened?

It planned the whole talk before asking me a single question.

A kid with a hammer starts hammering before asking where the nail is.

How would you prevent it?

chatgpt.com/share/6a972415-0e50-83ea-8e47-d4a9ac4704f9 (GPT 5.6 Pro; the full reply is long)

The model will not learn, so write the procedure down in a skill

The experience is disconcertingly similar to advising a PhD student on a paper. That said, a PhD student would need less handholding on their second paper, but an LLM won't really learn.

Noah Snyder, "An experiment with AI-assisted writing", Secret Blogging Seminar, July 29, 2026.

When you write a new skill, test it on a cheap, fast model. It stands in for a frontier model near the end of its context; the skill must work there.

Standing instructions: the short list of rules that apply to every task

In a research group: house rules, short enough that everyone knows them.

Example: four of my 66 standing rules

From my global CLAUDE.md = AGENTS.md (4 of 66 lines)

- Bug fixes require a regression test that distinguishes fixed from broken code.
- Check for existing helpers before reimplementing.
- Announce the size, then hold to it. Say what you are about to write.
- Do not commit plans, PR bodies, investigation notes, findings, logs, or other working documents.

Standing instructions earn their place by recurring across tasks.

I arrived at my rules after many failures.

Be careful about standing rules

Standing rule: "Always cite your work. Every factual claim needs a fetched source, cited inline."
Prompt: "Give me a rough idea of how large the tides are on Jupiter."

Depends what you mean by "tides on Jupiter":
[...]
- **Jupiter itself has no ocean**, so there's nothing like Earth's sea-level tides. It does get a small solid/fluid tidal bulge from the Sun and its big moons, but it's negligible relative to Jupiter's ~140,000 km diameter - a few meters at most, unmeasurable in practice.
[...]

I couldn't find a source that states an actual bulge height in meters for the tide Io raises on Jupiter itself - the literature reports it via a dimensionless Love number rather than a linear displacement. Here's what's actually documented:
[...]
- **Tide Io raises on Jupiter** (the reverse, much smaller effect since Jupiter vastly outweighs Io): Juno gravity measurements give Jupiter's tidal Love number k2 = 0.565 +/- 0.006, about 4% below the theoretical hydrostatic value, and Jupiter's response departs from simple equilibrium-tide behavior - showing dynamical/resonant effects instead. No paper I found converts this into a bulge height in meters; it's reported and analyzed purely via the Love number formalism. ([Idini & Stevenson, Dynamical Tides in Jupiter as Revealed by Juno](https://iopscience.iop.org/article/10.3847/PSJ/abe715); [AAS Nova summary](https://aasnova.org/2021/04/21/raising-tides-on-jupiter-with-its-moons/))
[...]

claude-sonnet-5, 2026-09-03. Without the rule: 13 s, 25k tokens. With the rule: 72 s, 292k tokens.

Project record: where goal, plan, decisions, and evidence live between sessions

In a research group: the shared draft and the notes, what a coauthor reads to catch up.

The conversation is working memory, not the project record.

Like it wrote the whole first draft in 20 minutes, but then when the paper was 50 pages long, I asked it to switch the order of two paragraphs and it took an hour.

Noah Snyder, "An experiment with AI-assisted writing", Secret Blogging Seminar, July 29, 2026.

As work grows, the agent may lose sight of the objective, the plan, earlier constraints, completed tasks, discoveries, or the next action.

Slide 34 of "Slop is a Skill Issue: The Engineering around AI Agents", MIT EECS/CSAIL Agentic Coding in Practice Seminar Series, talk 2. Chart source as printed: OpenAI GPT-5.4 eval table, MRCR v2 8-needle, March 5, 2026. people.csail.mit.edu/saman/acpss/talk-2/talk-slides.pdf

Give plans, decisions, and evidence a durable home

Once the state is durable, a fresh agent can resume without inheriting every tangent and failed attempt.

A long conversation is a collaborator at 2 am at the end of a very long day: everything was said, little of it is in focus.
The record is what you both read in the morning.

Fresh contexts: a new context that knows only its brief

In a research group: the person you send your draft to for comments.

One context should not carry the whole project

Subagents are not primarily fictional personalities. They are context isolation.

Independent checks: review by something that did not produce the work

In a research group: referees, who did not write the paper.

Plausibility is not reliability, so separate checking from production

Five agents agreeing is not five independent proofs.

Budgets and stopping rules: how much effort, and when to stop, restart, or escalate

In a research group: deadlines, and the call to submit, revise, or withdraw.

Back to the advanced harness: every box answers a failure you saw

1 2 3 4 5 6 7 8

Reproduced from Mohammed Abouzaid, "How do AI systems prove math theorems? First Proof and open-source tools," ICARM 2026, PDF page 19 (printed slide 14/41); annotations added. Code: github.com/1stproof/math-solve-skill-FP.

1. Mastermind owns dag.json, state.json, log.jsonl: durable state, not a context window
2. CAS probe and fetchers: exact tools and fetched evidence, not recall
3. Dreamers and Solver, separate roles: context isolation
4. Checker and Referee, then verdict: production separated from checking, two independent judges
5. root correct, proof.md: a stopping condition with an inspectable artifact
6. Patcher, at most 3: corrective iteration with a budget
7. strategic-failure back to Dreamer: a restart rule, not endless patching
8. The skill itself, 2000 lines: an on-demand procedure, written once

Close: there is no one true harness, so start where you are and try one thing

There is no preferred position.

Different tasks have different failure modes, stakes, and acceptable levels of control.

• Start where you are comfortable.
• Try one existing skill or control on real work.
• Notice what it improves and what friction it creates.
• Adapt it to the task.
• Share successes and failures with your colleagues.

Try one thing. Write down what worked and failed. Share what you learned!

Backup: the reply, top of the page, by GPT 5.6 Pro

chatgpt.com/share/6a972415-0e50-83ea-8e47-d4a9ac4704f9 (GPT 5.6 Pro; the full reply is long)

Backup: where each kind of instruction lives