The system is IBM Sterling WMS, a commercial warehouse product. Over the years it was customised into several separate solutions, each with its own codebase. Each solution runs as several installations (development, test, production), and each installation has its own configuration and data.

A typical support ticket isn't "fix this function". It's "the pick screen rejects a carton at one warehouse but not at another", or a new requirement in a spreadsheet written by someone who has never seen the code. To handle it, someone has to find out what the system does today, agree what should change, change the code and configuration, test it, and hand it to the person who deploys it.

I built a set of agents that does that work, with a person approving each important step. The agent that writes code is one of six steps. This post explains the rest:

• how the agents get the knowledge they need;
• how we check what they produce;
• where people make the decisions.

I can't share the client's code or tickets, so this post describes the design and the reasons behind it, in a form you can reuse in your own projects.

Why the usual issue-to-PR approach doesn't fit

Tools like GitHub Copilot's coding agent, Codex and Devin take an issue and open a pull request. That works because GitHub already supplies the rest of the process: the issue describes the work, a person reviews the pull request, the test suite checks it, and merging deploys it. Those tools depend on three things being true. In a customised ERP or WMS system, none of them is.

When those assumptions don't hold, the work doesn't go away. It moves to three places, and the rest of this post is about them:

• Giving the agent the right information: the code, the configuration of the right installation, the data model and the product documentation.
• Checking the output, because there is no test suite to do it.
• Handing over to people, because people approve and deploy.

Surveys show the same shift from the other side. In GitLab's June 2026 survey, 85% of respondents said AI had moved the bottleneck from writing code to reviewing it.

The architecture

The system has three parts. Keeping their jobs separate was the most important design decision.

• A ticket app decides what happens next. It's a small web app with its own database of tickets. For each ticket it records which step the ticket is on, which agent run it is waiting for, and who may approve that step. It also holds the business rules, such as "a defect belongs to a requirement" and "a requirement can't move on while it has open questions".
• SwarmKit runs one step at a time. Each step is defined as a topology: a YAML file listing the agents for that step, the tools each one may use, and the format its answer must have. The app starts a step through SwarmKit's HTTP API, then checks the job until it finishes. SwarmKit saves the run as it goes, so if the app restarts, the run carries on.
• Plain scripts do everything that needs no judgement. They save each step's output in the right folder, turn it into documents a reviewer can read, check the design, build the deployment package and run the tests. The app calls the same scripts a person would run by hand, so both always give the same result.

The six steps are triage, design, code, configuration, test plan and test run. A person approves at two points:

• After triage, a support lead decides what kind of work the ticket is.
• After design, the business owner approves the specification before anyone builds it.

After that, the code and configuration are produced from the approved specification. Their output is a code diff and a configuration document, which a person applies.

At first, the order of the steps was defined inside SwarmKit as a pipeline. We moved it into the app, because deciding what happens next, over days and around people's availability, is application logic. SwarmKit now only runs a single step, enforces its approval and keeps the record of what happened.

Giving the agents knowledge of this system

A model knows Sterling WMS in general from its training. It doesn't know how this team customised it, how production is configured, or which custom libraries have no source code. Most wrong answers came from that gap.

Solutions and installations are kept apart. A solution is a codebase plus its product version and build tools. An installation is one deployment of a solution, with its own database, configuration and data. Questions about code are answered per solution; questions about data and configuration are answered per installation. Every tool requires the solution or installation as an argument. An agent can't answer about the wrong site by forgetting to say which one; it would have to name the wrong one.

The agents read this knowledge through MCP servers: small services that give an agent tools it can call. Each server covers one kind of knowledge, and none of them can change anything.

Two choices are worth copying:

• API definitions come from the vendor's schema files, not from the model's memory. Models often produce API attribute names that look right but don't exist.
• Documentation search uses two indexes. Vector search handles questions in plain language. Full-text search handles exact names, such as a table name, which vector search tends to miss.

Convention cards

Product documentation describes what the product can do. It doesn't say how this team builds things, and that gap caused the most expensive mistakes. So we wrote it down as convention cards: one short Markdown file per kind of artifact, such as a mobile screen or a service.

Each card holds three things:

1. what that kind of artifact is made of here;
2. where this team does it differently from the product's default;
3. links to the detailed documentation.

When one solution does something differently, a separate small file records only the difference, and the tool returns both together. There are 12 cards and 5 solution-specific files, and we wrote each one after an agent got that kind of artifact wrong.

The cards replaced instructions that had been copied into agent prompts. The same guidance about mobile screens had been pasted into three agents' prompts, and the three copies had already drifted apart. A card exists once, is reviewed like code, and is only loaded when it's needed.

Using a different model or harness for each step

In SwarmKit, each agent has an executor: the thing that actually does its work. There are two kinds:

• A model node sends prompts to a model through an API. SwarmKit checks that the answer matches the required format and asks again if it doesn't.
• A harness runs a complete coding agent, such as Claude Code, which has its own tools for reading and editing files. It can work through long tasks, but SwarmKit can only ask it for a format; it can't enforce one.

The executor is set in each agent's YAML file, so moving a step from one to the other is a configuration change, not a rewrite.

We chose these by measuring. The low-cost model handled triage's short form well. On the design step, with a 22-field form plus tool calls, it produced a valid design in only 2 of 7 runs; the others returned partial output, a format it made up, or its tool calls written out as plain text. So design went to Claude Code, and triage stayed on the low-cost model.

Most of the cost savings had nothing to do with which model was cheaper:

• No model where none is needed. Saving files, building the package, checking formats and running tests are done by scripts. They cost almost nothing and can't make the kind of mistakes a model makes.
• Steps with nothing to do are skipped. The configuration step only runs when the design adds a service. Before that check, running it anyway cost about $0.20 and five minutes, only to end in an error.
• Cheap checks run before expensive ones. The design review runs its scripted checks first, and only starts the Claude Code reviewer if those pass.
• Re-running a step requires a reason. Without new input, a re-run repeats the same work at the same price, so the person must say what changed.
• An unchanged output isn't sent for approval again. Approvals are tied to a hash of the output's content, so an identical re-run doesn't ask anyone to approve it twice.
• Long Claude Code sessions run on a subscription, not on pay-per-token API credits.

Making agents call tools in the right order

We wanted every agent to read the relevant convention card before writing anything. Telling it to wasn't enough. Two early designs contained file paths that didn't exist, even though each run's instructions told the agent, many times over, to read the conventions first. Neither run did.

One run showed the cause clearly. The same agent, with the same prompt, called a tool that the system required 4 times, and a tool that the prompt only asked for 0 times. Agents follow what the system enforces, not what the prompt asks for.

So we declared the order in each agent's YAML, and SwarmKit enforces it. A requires: entry says that one tool can't be used until another has been called in the same run:

requires:
  get-estate-reference: [get-build-convention]
  find-wms-column:      [get-build-convention]
  search-sterling-docs: [get-build-convention]
  # … every estate skill waits for a convention card

If the agent calls a tool too early, SwarmKit refuses the call and returns a message telling it what to do: "get-estate-reference requires get-build-convention, which has not been called in this session. Call it first, then retry." The agent calls the convention tool and carries on. The rule applies separately to each agent in each run, every refusal is written to the audit log, and a rule that could never be satisfied (for example, one naming a tool the agent doesn't have) is rejected before the run starts.

Where it doesn't work

The same rule did nothing on the coding agent. Claude Code has its own built-in tools for reading and writing files, so it read the code with those and never called our file tools. A rule on a tool the agent never uses can't trigger, and it gives a false sense of safety.

So on Claude Code agents, we only put rules on tools that have no built-in alternative: the data model, the documentation, the installation's configuration and the reference screens. Those rules do work. Adding one to the design agent took it from never looking up reference material to always doing so before checking its design.

This rule controls order; it isn't a security measure. What keeps the system safe is that none of the agents' tools can change anything.

Making the output good, not just correctly formatted

An agent saying it has finished doesn't prove that it has. Each technique below exists because a run reported success when its output was wrong. One run lost access to its tools halfway through and still reported success.

1. Ask for evidence in the output. The coding agent must say, for every file it writes, which existing file it used as a model. The script that saves its changes flags any file without one. This worked where controlling the agent's tools didn't: in the next run, every file named a real existing screen it had copied the style of.

2. Check the format first, then the content. Before a person sees a design, it passes three checks, which SwarmKit runs in order as a "funnel":

1. Format: the design must match a JSON schema. If a field is wrong, the agent is told which field and asked to fix it.
2. Content: scripts first check facts that have a definite answer, for example that every file and service the design mentions really exists in the installation's configuration, and that every service flow has a start and a reachable end. Only if those pass does a second Claude Code agent review the design for things a script can't decide. If its score is below 0.75, the review comments go back to the design agent to rewrite. After two attempts, the design goes to a person with the comments attached.
3. Approval: the business owner approves it, and can't be the person who wrote it.

3. Don't let an agent mark its own work.

• A separate agent writes the tests from the approved specification, not from the code, so the tests check what was agreed rather than what was built.
• After the tests are written, the app checks them itself instead of trusting the agent's report.
• Whether the tests pass is decided by JUnit. The test-run step has no agent at all.

4. Agents return content; scripts save it. Agents never write files into the project themselves. They return their output, and a script decides where it goes. The script refuses unexpected file names, paths outside the ticket's folder, and documents that cite no sources.

5. Let the coding agent edit files, in a throwaway copy. At first the coding agent wasn't allowed to edit files, so it returned whole files as JSON. Adding nine lines to a 611 KB label file came back as the entire file, which showed up as 27,097 lines removed and 12 added. We then let it edit files directly in a temporary git worktree, a separate copy of the repository that is deleted after the run. The same change became a nine-line diff, and the real project stayed untouched.

6. Turn the output into documents people can review. Agents return JSON, which nobody wants to review. After each step, scripts produce what the reviewer actually reads: the specification in Markdown and Word, a configuration document with flow diagrams, the code diff, and a validation report listing anything to check before approving.

Putting people where their decisions are cheap

Reviewing agent work is slow when a person has to judge a large code change after all the work is done. We moved the important decisions earlier:

• Triage review. A support lead reads the triage and decides what kind of work the ticket is. That choice decides which checks and approvals apply, so we don't let a model make it.
• Specification review. The business owner approves the design before any code is written. Fixing a misunderstanding here means rewriting the specification. Later, it means a code change, another review and another round of integration testing.
• Code and deployment. By then the code implements an approved specification, so the reviewer is checking it, not working out what it should have done.

When a step reaches an approval, it stops until a person decides. SwarmKit saves the run and resumes it after the decision, so nothing keeps running while it waits. Only a person can approve: the permission to approve can't be given to an agent or to an API key.

Three ways work gets redone

Work is redone for different reasons, and we handle each one differently so the history shows which reason it was.

Changes requested. The reviewer's comment is passed to the agent when the step runs again. It's labelled as review feedback from a named person, not as instructions, so text written by people can't override the agent's instructions. Each version of the output is identified by a hash of its content, so an approval of version 1 doesn't count as an approval of version 3.

Reopening. Requirements change after approval: a mapping that wasn't final, a rule nobody mentioned, a field the customer added. The design was right for what was known, but what's known has changed. A person reopens the ticket at the step that's affected:

• Only a person can reopen, because it throws away approved work and costs money to redo.
• The person must write what changed, and that note is given to the agent. Without it, the agent would repeat the same work.
• The old runs stay in the history with their cost, and their output is moved to a numbered archive folder.
• Earlier approvals don't carry over to the new version.

This works better than opening a new ticket, which would lose the triage, the answered questions and the specification to revise.

Defects. A problem found in integration testing doesn't send the requirement back to the coding step. It becomes a new ticket, linked to the requirement, with its own steps:

1. investigate, then a person accepts the diagnosis;
2. propose a fix, then a person accepts the approach;
3. write the code, then a person reviews the change.

Several defects can be worked on at the same time. The requirement stays blocked until the last fix is merged, and then the whole requirement goes through integration testing again.

Why approvals get quicker over time

The approvals never went away, but they became quick. Each time a reviewer objected, we turned that kind of objection into something the system checks or knows, so the same objection didn't come back.

Early specifications needed a reviewer to catch invented file paths and service flows that broke halfway through. Later ones arrived already checked against the real configuration, with a report of anything left to look at. Approving became reading the specification, not searching it for problems.

SwarmKit already does the same thing for permissions. When Claude Code needs a permission it wasn't given, the request goes to a person. After a person approves the same action 5 times, SwarmKit suggests adding it to the agent's permanent permissions, and a single refusal resets the count.

What stays with people is the judgement no check can make: is this the right design for what the business asked for?

Turning open questions into tickets

Requirements usually arrive with gaps. One needs three questions answered before it can be designed; the next needs none. A fixed list of steps can't express that, so open questions are handled before the steps start.

How it works:

1. Every agent's output has a blocked_by field. In it, the agent lists what it couldn't find out, and what would answer it.
2. The app turns each entry into a new ticket that blocks the original one. If a retried step reports the same question again, it isn't created twice.
3. The app picks up questions that are ready to be answered, meaning open tickets that aren't themselves waiting on another question. A question ticket doesn't go through the six steps; it goes to the one agent setup that answers that kind of question, or to a person.
4. The original ticket waits. It can't start its next step until all its question tickets are closed.

Most agents handle a gap by asking in the chat or by guessing. Neither leaves a record, and a guess that the rest of the work is built on is the most expensive kind of mistake. A question ticket has an owner, an answer and a history, and the design agent receives the answers instead of making its own assumptions.

How the knowledge builds up

We learned one rule the hard way: knowledge only counts if something delivers it to the agent. An audit found six places where knowledge had been set up but nothing ever loaded it, including the notes folder, the generated documentation, the judge's scoring guide and the memory reader. One design run rebuilt an approach we had already recorded as wrong, at a cost of $3.63, while the correct fact sat unused in storage.

So we sorted knowledge by when the agent needs it, and gave each kind its own way of reaching the agent:

Two rules keep these apart:

• Conventions are always loaded, never ranked by relevance. Memory only adds the five best matches, so a convention could lose its place to five unrelated facts without anyone noticing.
• Memory holds facts, not conventions. How to build something goes in a card; a fact about one installation goes in memory. A card that contained installation-specific values would be out of date before anyone read it.

This pays off when a fact changes the output. Once the correct fact was in memory and given to the agent, the next design used it instead of repeating the old mistake, and the cost of that design run fell from $3.63 to $1.75.

The knowledge grows as people work

The knowledge isn't fixed when the project starts. It grows from what people write down while working on tickets:

• convention cards, written or corrected whenever an agent gets something wrong;
• a knowledge index: a few lines per card saying what it covers and when to read it;
• confirmed facts, recorded by a person at an approval or with the memory command-line tool;
• notes: a hand-written file per solution.

Writing it down isn't enough on its own. One agent, told to read the conventions first, called the schema tool eight times and the conventions tool zero times, and broke every rule in the card.

So knowledge reaches the agent at three levels:

1. The index is stored in governed memory, which SwarmKit adds to the agent's input before every turn, without the agent having to ask. A script copies the index from its reviewed file into memory and updates existing entries in place.
2. The card is fetched when the index points to it, and the requires: rule makes sure that happens.
3. The detail, such as earlier examples and product documentation, is linked from the card and read when needed.

Notes don't reach the agents yet; for now they are for people.

Two kinds of memory

SwarmKit has two kinds of memory:

• Workspace memory: what a run records about itself, with no review.
• Governed memory: curated facts. Each fact is stored as a subject, an attribute and a value, for example an installation, a setting and what it is set to. When a new fact arrives, it's compared with what's already stored: it may be new, confirm an existing fact, update it, or add detail to it. If it contradicts a fact that's already trusted, it's held for a person to decide instead of overwriting it. Older facts gradually count for less.

In this project, only governed memory is given to the agents, and only people write to it.

The subject records where a fact is true: the product, one solution or one installation. A fact confirmed on production but stored against the whole solution would be given to agents working on the development system, where it may not be true.

Facts are written when something stops being a guess: when a person accepts a triage or a diagnosis, when a fix is agreed, and when a ticket closes as done or rejected. An approach that was rejected is worth recording too.

It's tempting to let every run save what it learned. That looks good in a demo and goes wrong slowly: guesses become facts, agents keep confirming their own earlier guesses, and within weeks the system is confidently wrong, with records that look like learning.

What I'd tell someone building this

• Spend your effort on knowledge, not on the coding agent. Getting the coding agent working was quick. Giving it the right information, checking its output and handing over to people took the time.
• Enforce what matters instead of asking for it in the prompt. Tool order, output format and evidence only became reliable once the system enforced them.
• For a full coding agent like Claude Code, check what it returns rather than which tools it uses. It has its own tools, so rules on yours may never apply.
• Never accept an agent's "done" as proof. Have a script, a test run or a different agent check the work.
• Choose the executor per step. Use a low-cost model for short structured answers, a coding agent or harness for long sessions with many tool calls, and no model when a script can compute the answer. Measure before you switch.
• Ask people to decide earlier. One approval of the specification saves several rounds of reviewing code.
• Handle each reason for redoing work separately. "This output is wrong", "the requirement changed" and "the system is broken" are different events, and the history should show which one happened.
• Keep the order of steps in your application. Let the agent framework run one step well and keep its record; let the app that knows the people decide what happens next.

The open problem is people. Every approval needs someone assigned to it, and an approval with nobody assigned stops the ticket. That's the safe choice, but as volume grows, approvals will become the bottleneck before the agents do. Our next step is to run routine approvals in shadow mode: a person still decides, while we record how often they agree with the automated checks, and only relax approvals for a type of ticket when that record supports it.

Sources

• AI code review hits a wall (GitLab survey, Futurum)
• SwarmKit