Today, one level out: the harness is how the project is run

You do not tame the model directly. You tame the process around it.

You already do project management: collaborative work, writing a paper.

You do not YOLO either one.

Skills: a reusable written procedure for one kind of work, loaded when needed

In a research group: the protocol for writing a paper, refereeing one, or filing a reimbursement.

Without a procedure, the agent invents how to work

I asked GPT 5.6 Pro. It worked for twelve minutes.

I was invited to give a talk in one of the Friday slots of this course:
https://katestange.net/ai/course/
The class attends, and also the whole math department.
Help me prepare it.

What do you think happened?

It planned the whole talk before asking me a single question.

A kid with a hammer starts hammering before asking where the nail is.

How would you prevent it?

chatgpt.com/share/6a972415-0e50-83ea-8e47-d4a9ac4704f9 (GPT 5.6 Pro; the full reply is long)

The model will not learn, so write the procedure down in a skill

Noah Snyder on "An experiment with AI-assisted writing", Secret Blogging Seminar, reported:

The experience is disconcertingly similar to advising a PhD student on a paper. That said, a PhD student would need less handholding on their second paper, but an LLM won't really learn.

A way to test how comprehensible a skill is, is to test it on cheap/fast models.

You are the one who learns; the skill carries the lesson into the next conversation.

Standing instructions: the short list of rules that apply to every task

In a research group: house rules, short enough that everyone knows them.

Example: seven of my 40 standing rules

From my standing-instructions file, CLAUDE.md = AGENTS.md (7 of 66 lines)

- Bug fixes require a regression test [a test that fails on the old code] that distinguishes fixed from broken code.
- Check for existing helpers [library functions] before reimplementing.
- Do not comment idiomatic patterns [routine code].
- Be concise in PR bodies [change descriptions], code comments, and tracker comments.
- Announce the size, then hold to it. Say what you are about to write.
- Do not commit [save into the shared history] plans, PR bodies, investigation notes, findings, logs, or other working documents.
- Use Conventional Commits [a fixed format for change summaries]: <type>(<scope>): <description>, scope optional.

I arrived at my rules the hard way.

Be careful about standing rules

Standing rule: "Always cite your work. Every factual claim needs a fetched source, cited inline."
Prompt: "Give me a rough idea of how large the tides are on Jupiter."

Depends what you mean by "tides on Jupiter":
[...]
- Jupiter itself has no ocean, so there's nothing like Earth's sea-level tides. It does get a small solid/fluid tidal bulge from the Sun and its big moons, but it's negligible relative to Jupiter's ~140,000 km diameter - a few meters at most, unmeasurable in practice.
[...]

I couldn't find a source that states an actual bulge height in meters for the tide Io raises on Jupiter itself - the literature reports it via a dimensionless Love number rather than a linear displacement. Here's what's actually documented:
[...]
- Tide Io raises on Jupiter (the reverse, much smaller effect since Jupiter vastly outweighs Io): Juno gravity measurements give Jupiter's tidal Love number k2 = 0.565 +/- 0.006, about 4% below the theoretical hydrostatic value, and Jupiter's response departs from simple equilibrium-tide behavior - showing dynamical/resonant effects instead. No paper I found converts this into a bulge height in meters; it's reported and analyzed purely via the Love number formalism. ([Idini & Stevenson, Dynamical Tides in Jupiter as Revealed by Juno](https://iopscience.iop.org/article/10.3847/PSJ/abe715); [AAS Nova summary](https://aasnova.org/2021/04/21/raising-tides-on-jupiter-with-its-moons/))
[...]

Without the rule: 13 s. With the rule: 72 s, 12 times the cost.

Project record: where goal, plan, decisions, and evidence live between conversations

In a research group: the shared draft and the notes, what a coauthor reads to catch up.

The conversation is working memory, not the project record

Noah Snyder also observed:

Like it wrote the whole first draft in 20 minutes, but then when the paper was 50 pages long, I asked it to switch the order of two paragraphs and it took an hour.

As work grows, the agent may lose sight of the objective, the plan, earlier constraints, completed tasks, discoveries, or the next action.

Context problems, ranked:
incorrect >> missing >> too much.

One model's accuracy at recalling one fact from the conversation: 97% when short, 37% when very long.

A long conversation is a collaborator at 2 am at the end of a very long day: everything was said, little of it is in focus.
The record is what you both read in the morning.

Slides 34 and 37, "Slop is a Skill Issue" (MIT), people.csail.mit.edu/saman/acpss/talk-2/talk-slides.pdf

Give plans, decisions, and evidence a durable home

This allows one to use subagents that are ready to go.

Agents take as fact whatever they read, their own guesses included; label each claim by its support.

Fresh contexts: a new conversation that knows only its instructions

In a research group: the person you send your draft to for comments.

One conversation should not carry the whole project

Independent checks: review by something that did not produce the work

In a research group: referees, who did not write the paper.

Plausibility is not reliability, so separate the checking from the writing

Checks

• a fresh reviewer
• another model family: different models have different strengths and blind spots
• human judgment

Five agents agreeing is not five independent proofs.

Budgets and stopping rules: how much effort, and when to stop, restart, or ask for help

In a research group: deadlines, and the call to submit, revise, or withdraw.