Inside Arch: What 16 Procedures Taught Us About Building AI Agents That Actually Work

An SOP is already a spec. The hard part is what it leaves out.

If you've worked with AI in any capacity, you already know it struggles with the implicit. It does exactly what you tell it, but rarely understands the why. That's the gap that decides whether the output is actually useful, and closing that gap comes down to how well you guide the AI toward that why.

So when we set out to build an AI architect that builds agentic projects, we came across the same problem at a much larger and more complex scale.

Every enterprise already has a specification for its agents. It is called the standard operating procedure. SOPs are precise about policy: what must be verified, what may be promised, when a human takes over. They are written for people who share a lot of unwritten context.

Let's take an example:

A wealth-management firm wants to incorporate AI into its business workflow. The objective is simple: help the end users make financial decisions that are informed and suited to their existing portfolio.

The firm already has a service procedure that says what to verify, when to escalate, and what to tell the client at each step.

The procedure is forty-one pages long, and probably nobody on the team has read it start to finish. They know the most important pages: the exception buried on page thirty-one that got added after a complaint escalated past three levels; the margin note that says, essentially, ignore step 12 if the account was opened before the migration. This isn't a flaw in how the document was written. It's what a standard operating procedure actually is, in any enterprise old enough to need one: not a clean specification, but the accumulated scar tissue of a business learning, one incident at a time, how it's actually supposed to behave.

But an AI agent still needs the implicit context made explicit:

• Who answers first, and which specialist owns which request

• Which steps are a prerequisite before an intent is triggered

• What each tool returns, and which values the next step depends on

• When to stop, and what counts as done

Having helped hundreds of clients build AI solutions, we understand the importance of these seemingly small decisions that eventually decide the fate of the project. We refused to build a system that fills these gaps with plausible guesses: that's how you end up with agents that compile, pass their tests, and still fail in front of real customers. We were clear on that from day one.

Arch, the AI architect in Artemis, is built to surface these exact questions. But this is just where we began. To confidently certify Arch, we knew we needed more. So we created 16 different SOPs, based on what we had learned over the years serving our clients. These SOPs span multiple domains and scenarios, built to test the robustness of our AI architect. We tested it against 16 real procedures to see how it actually holds up.

What the 16 procedures showed

Before testing anything, we had to decide what the procedures should actually represent. We centered the set on problems we already knew would trip up a generation system, building it around four categories, four procedures each:

• Complete SOPs with all the information required

• Incomplete SOPs, designed to force Arch to question the implicit

• SOPs full of internal contradictions

• SOPs that combined both problems: incomplete and contradictory at once

From there, we randomly assigned each procedure to a business domain, rather than hand-picking the pairing, to keep the test honest.

Arch didn't arrive at its current design in one attempt. Early on, we tried several different approaches to how it generates a project, testing each against all 16 procedures. No single approach got everything right on its own, so the pipeline Arch runs today combines what worked best across them.

Running it against all 16 procedures produced the below results:

The scores above come from a test suite Arch generates automatically from that same SOP, built to check whether the project actually behaves the way the procedure describes, not just whether it compiles.

Pharmacy and banking procedures consistently scored better; retail and consumer-care lagged furthest behind. Digging into why, the gap traces to the nature of each industry. A retail or consumer-care conversation can go almost anywhere, since a customer might ask for nearly anything, which adds real variability for an agent to handle. Pharmacy and banking, by contrast, operate under rigid compliance rules that narrow both what the procedure allows and what a customer can actually ask for, leaving the agent a tighter, more predictable space to work in. 

The Learnings

The first learning was that tests generated from an SOP share its blind spots. When we tested an earlier pharmacy agent, which had scored ~90% against the test suite generated from its own SOP, against a stricter suite built from production data instead, it scored 36.3%, with zero runtime errors. This shows the importance of testing real-life scenarios that really test the AI agent on the implicit signals we talked about at the beginning of this article. We tend to take the implicit for granted, and it's what really makes the difference between a successful and a failed AI project.

Coming back to the pharmacy agent: it made mistakes like telling a member it couldn't see order details that were sitting in its own trace. Repairs to the generated agent, and to the generator itself, took that score to 56.5% in one Autoloop cycle. That is progress, not a finish line, and it is why we always test agents against conversations the SOP never anticipated.

The second lesson was about the importance of consistency. The same SOP can produce a somewhat different agent structure from one run to the next, even with the model's sampling set to be as consistent as possible. That is why Arch works in accepted stages rather than one shot. This helps create boundaries around a problem that, by its nature, is boundless. In turn, it helps us create more reliable agents.

All of that is possible because of how Arch is actually built. Here's the pipeline, end to end:

What Arch does

Interview & Design

Arch reads the SOP and works through a careful interview process across seven strategic stages. At each stage, it proposes a structure, shows the SOP evidence behind it, and flags any conflicts or alternatives it finds. Only the stages a person accepts become part of the agent. The exchange runs both ways: at any stage, a person can supply context Arch is missing, question a proposal, or redirect it.

The seven stages:

• Source scope and channels
• Agents and responsibilities
• Tools and schemas
• Journeys and routing
• Context, memory, and handoffs
• Guardrails and channel behavior
• Outcomes and eval scenarios

Agent Generation

The accepted design becomes an ABL (Agent Blueprint Language) project. It contains a fully functional router, specialists, handoffs, and tools. Built-in checks make sure agents are built on real evidence, not assumptions. For example, a handoff condition invented from an agent's name rather than a declared intent stops the build.

Testing

Once generation is complete, every project proceeds to automatic test-suite generation, built from the same SOP used in the interview and agent-generation steps. This tests what was built against real conversations, rather than simply relying on the ABL, giving us a more comprehensive picture.

Optimization

As we progress through the testing cycle, we naturally come across issues of many kinds, such as a scenario that didn't get handled properly or a prompt that isn't optimized for task completion and cost efficiency. This is where Autoloop comes in. Autoloop is the optimization engine in Artemis. It analyzes the issue, proposes a fix, and then tests the fix to check whether it resulted in an improvement. It also measures whether fixing one issue introduced regressions elsewhere. Humans get to take precise control of Autoloop, reviewing and approving or rejecting its work.

A fitting end to this article is to leave you with the questions that actually surface these blind spots, the ones worth asking before you commit to any vendor's agentic capabilities in 2027:

1. When the SOP is ambiguous, does the tool ask or guess?
2. What safeguards are in place to surface assumptions that could later risk or delay the project?
3. How will generated agents be tested? What exactly is the framework for testing? Is it comprehensive enough?
4. Does testing run without heavy manual effort? Human review has a role, but it shouldn't be the bottleneck.
5. How do you control the tests, and who authors them? How detailed are they?
6. What human review checkpoints are available? What progresses without human review?
7. Can your agent handle conversations that are complex combinations of intents, ones that aren't as straightforward as defined in the SOP?
8. What mechanism is available to test production-grade scenarios?

If you are looking to build your own usecase, talk to us here.