Harvey's Agent Architecture: Add Capabilities, Not Agents

A legal AI company has more than 25,000 custom agents running on its platform. What holds it together isn’t a zoo of specialist agents. It’s one rule: no team gets to write its own orchestration.

TL;DR

• Goals first, rules second. Harvey’s three architecture principles fall straight out of three goals: quality that improves with the models, features that compose, and no dependence on one central team.
• Tool bundles are the core abstraction. An agent is just “a system prompt plus a set of bundles.” Bundles multiply with features; agents barely do.
• Harvey picked OpenAI’s Agent SDK precisely because it can’t do workflow-style orchestration. It cost them determinism. It bought them composition.
• 60+ jurisdictions and 400+ data sources sit behind one agent with parameterized tools. “One agent per jurisdiction” was considered and rejected.
• The part I’d copy first is the eval gate: the path can vary, the output can’t.

1. First, pin down what “agent” means

Harvey lays out a four-layer stack. I love it, because “agent” has been stretched until it means nothing.

Layer 4: Workflow
  One or more agents working together to produce a concrete work product
  "Agents are the means; workflows are the ends."

Layer 3: Agent
  Model System + Plan + Adapt + Interact
  Breaks down tasks, changes the plan based on intermediate results, asks people for information

Layer 2: Model System
  Multiple models + tools + knowledge sources, in a linear, rigid flow
  Steps are fixed in advance; the path never changes based on intermediate results

Layer 1: Model
  One model: prompt in, response out

Harvey calls model systems “rigid one-way trips.” Agents are “model systems that can plan, adapt, and meaningfully interact with humans.”

So whenever someone tells me they’ve built an agent, I ask two questions:

1. Does it change its path based on what it finds along the way?
2. Does it ever ask you for information on its own?

If the answer to both is no, it’s a pipeline. No matter how many tools it has.

A lot of “agents” are pipelines with tools attached.

2. Three goals, three rules

This is the habit I’d most like to steal: Harvey didn’t pick an architecture and then justify it. They wrote down their goals first.

Goal (paraphrasing Harvey)	The principle it produced
Output quality that gets better over time, using models to their fullest	No Custom Orchestration: hand the flow to the model’s tool-calling loop
New features that work together: retrieval and drafting should compose naturally	Capabilities are Tool Bundles: every new capability ships as a bundle
Minimal involvement from a central team, or feature development can’t scale	Eval Gates: every bundle gets its own offline evals and must pass before launch

Every specific decision below traces back to this table.

Architecture principles aren’t taste. They’re consequences of goals.

3. Tool bundles: the smallest unit of capability

The problem: one chat box, and no way to know the path in advance

Harvey’s Assistant is a single conversation thread: a lawyer asks a question, and retrieval, drafting and review all happen in that one thread. The hard part is that the right move can be completely different from one question to the next. Harvey gives three examples:

• Query a tax law database several times, then check the results against recent news
• Halfway through a long draft, pull material from the client’s Vault to fill in one section
• Add a few columns to a document review table, then aggregate over them

The system can’t know in advance which path a question needs.

At first, the Assistant team wrote Python mixed with LLM calls, built each feature as its own custom flow, and routed between them with modes: need to draft? Switch to Draft mode. Need a knowledge source? Here are some recommendations. Other teams could only contribute retrieval sources. Then they hit three walls:

Where	The wall
Product	People weren’t discovering Draft mode
Engineering	Multiple retrieval calls behind one interface were getting hard to maintain
Collaboration	New features from other teams (like Ask LexisNexis) had no clear path to launch

In mid-2025, Harvey moved to a pure agent architecture: every feature became a tool, and the model decides what to call. A new problem showed up right away. Every new capability needed its own paragraph in the main system prompt explaining how to use it. The first plan was “One team owns the system prompt, one owns the tools.” In practice, every feature waited in line for the prompt team, and engineers editing the same core instructions kept stepping on each other’s toes.

What a tool bundle is

A tool bundle is an interface Harvey designed so a developer can package one capability as a single unit. By Harvey’s definition, the unit holds just two things:

1. One or more tools, which can also be sub-agents
2. A set of instructions that teach the model how to use those tools together. These get injected into the main agent’s system prompt

Harvey’s two examples: a file system bundle is a grep-like search tool, a file-open tool and a semantic search tool, plus instructions on how to combine them; a drafting bundle is an editing tool plus a drafting sub-agent.

And an agent? “An agent at Harvey is composed of a system prompt and a set of Tool Bundles.” Think of the agent as a host program and bundles as plugins, each shipping its own features and its own manual. In the diagram above, every column on the right is an agent config: which model, what reasoning level, which system prompt, which bundles. The two labels on the diagram say it outright: bundles scale with the number of AI features (many); agent configs scale with the number of top-level experiences (few).

One thing is not inside the bundle: evals. Each bundle comes with its own offline evals (datasets, evaluators and alert thresholds), maintained by the bundle’s owner. They run when someone changes the prompt, adds a bundle or gets ready to ship, never while the agent is running (more in section 7).

What it solves

For the product, users stop picking modes. Drafting becomes a capability the agent can call, so a lawyer just says what they need, and capabilities combine: halfway through a draft, the agent can go check the Vault.

For the org, feature teams can write their own instructions into the main system prompt “without needing to make a request of the Assistant team.” And one bundle can plug into several agents, like the three lines coming out of Tool Bundle 1 in the diagram.

Harvey names the most important reason itself: “Most importantly, Tool Bundles allow us to apply leave-one-out validation gating on key benchmarks.” Give capabilities boundaries, and you can take them out one at a time to test.

The alternatives, and the cost

Approach	Upside	Cost
The Assistant team builds everything (a monolith)	One owner, no conflicting instructions	The team becomes the bottleneck, and users can’t find the modes. Harvey rules it out: the goal was for other teams to contribute integrations “rather than relying on our team to integrate it into a monolithic system”
One team owns the prompt, another owns the tools	A clean split	Tried, and it failed: every new capability still needed a prompt change, so the prompt team became the bottleneck
A separate agent per feature	No interference	Not discussed in this post. The knowledge-sources post rejects “an agent per jurisdiction” because handoffs between agents lose conversation history; my read is that the same problem applies here
Tool bundles (the choice)	Teams ship on their own; capabilities compose, get reused and can be tested one at a time	Instructions conflict in both directions, the context fills up, combinations can’t all be tested, and every team maintains its own evals (section 7)

My read: a tool bundle is a way of delegating authority. There’s only one system prompt, and Harvey slices write access to it by capability and hands the slices to teams. The upside is that nobody waits in line. The cost is that the slices interfere with each other, and the only thing separating them is natural language, so no compiler can catch a conflict.

What “from one team to four” means

Harvey’s sentence: after adopting a single agent framework, “in-thread feature development” scaled from one team to four. In other words, the number of engineering teams that build and ship features inside the Assistant thread went from one, the Assistant team, to four.

Which four? The post doesn’t say. It does mention integrations like Ask LexisNexis and iManage and product modes like Deep Research, so a fair guess is the Assistant team plus the teams behind those. But that’s a guess.

Four isn’t a target either. It’s where things stood when the post went up in November 2025. What matters is the direction: from “only the central team can add features” to “several teams add them in parallel,” which is exactly the third goal. It’s also the only scaling number Harvey published; there’s nothing on development velocity.

Authority you hand out has to be reined back in by evals. That’s why tool bundles never travel alone.

4. The constraint is the feature: choosing an SDK that can’t orchestrate

This is the decision I admire most in the whole design.

Harvey chose OpenAI’s Agent SDK because, compared to other frameworks, it “explicitly left out an option to orchestrate code in a more ‘workflow’ type format.”

Most people pick a framework for what it can do. Harvey picked one for what it can’t do.

Option	Upside	Problem
Each team writes its own orchestration (the status quo)	The shortest path to any single product goal	It quickly grows into the same “web of routing and human decision-making,” against goals one and two
Write an in-house agent framework	Full control	Not discussed in the source. My read: one more thing a central team has to own
A framework that supports workflow orchestration	Flexible	Leaves a back door, and teams drift back to hard-coded flows
An SDK with no workflow orchestration	Forces every capability into “call tools in a loop”	Gives up deterministic execution

The cost is real. One concrete case: case law research used to be two hard-coded steps, first query the user’s document, then use the result to investigate recent case law. After moving to the tool loop, Harvey admits “we couldn’t guarantee determinism in execution for the case law product.”

How they judged it worth it. Two things. Quality held up: a tuned prompt still hit high tool recall. And the payoff was immediate: once the product followed the framework, it combined with other knowledge sources and Deep Research right away.

How they rolled it out. One hard rule for new code: “All new product features that live in Assistant are Tool Bundles, and every top-level thread interface is an agent.” No big rewrite for old features: forced retrieval calls became tool calls, new integrations became tool calls, custom editing logic became tool calls, all held together by a system prompt that keeps growing.

An old feature that doesn’t move over can’t combine with the new ones. The ability to compose is what pulls the migration along.

One detail worth noticing: “no custom orchestration” governs features inside Assistant. On the workflow side, Harvey still runs a Workflow Engine (diagram in section 6). My read: workflows are the deterministic ends users ask for, the agent inside Assistant is the freely composable means, and each layer keeps its own rule.

5. One agent, a set of parameterized tools

Harvey’s knowledge sources grew from 6 jurisdictions to more than 60, and from 20 data sources to more than 400.

Each jurisdiction is just a declarative config: which domains to search, how to filter step by step, what permissions are needed, and optional agent instructions. The config becomes a set of parameterized tools for “a single unified reasoning agent.”

Option	Harvey’s verdict
A specialized agent per jurisdiction	The natural design, and rejected: handoffs between sub-agents lose conversation history, which breaks multi-turn legal research
Plain open web search	Rejected: customers care about source authority. A German tax question needs the official guidance, not a blog post that happens to rank well
The old way: onboard each source by hand	Find the repository, write a connector, hand-label test cases, recruit a local attorney to review: weeks every time
One agent + parameterized tools + an automated evaluation pipeline	Onboarding went from weeks to days

The costs are on the table too. Evaluating a single source takes roughly 150,000 tokens. Attorneys still review every source; they’re just faster at it, from 1–2 sources an hour to 2–4.

There’s a detail in the evaluation pipeline I love. At first they tested sources with general questions, and the agent simply answered from the model’s own knowledge without citing anything, which produced a pile of false negatives. So they switched to answer-first: make sure the answer lives in the source, then write the question.

The last step is a Decision Agent. The source puts it in one line: “Ambiguous cases route to human review.” People only look at what the machine isn’t sure about; everything else gets sorted automatically.

You scale with parameters, not with more agents. And you spend human attention only where the machine is unsure.

6. On the product side: merge first, then make it visible

One interface instead of several tools

Separate tools had two costs. The one from earlier: nobody found Draft mode. And lawyers kept switching between drafting and chat. Harvey’s line is blunt: “No more toggling between separate drafting and chat tools.”

The strategy behind it is expand, then collapse: “expand the platform with increasingly specific, high-value tools, and then collapse that complexity into a simple, unified interface.” The sources don’t say what merging cost.

Agents showed up inside workflows first

Harvey is explicit about it: “The first client-facing instantiation of agents in the Harvey platform is in workflows.” The first time clients met an agent, it wasn’t an open-ended chat box. It was a card with a defined output.

Line up the dates: March 2025, agents appear inside workflows. July, the interface merges and Deep Research plugs in. November, the tool-bundle principles get written down. February 2026, knowledge sources pass 400.

My read: prove the agent is useful inside a defined flow first, then hand out autonomy a little at a time. Harvey never calls that a strategy, but it’s exactly how the timeline played out.

Visibility: thinking states, clarifying questions and citations

Harvey’s principle: “A key principle for the Harvey user experience is transparency.”

In the product that means three things:

1. Clarify before working. Before research starts, ask the user for the context it needs.
2. Thinking states. Show what the agent is doing in real time, so the user can follow along and step in at any point. Notice the step in the diagram, “Convert reasoning into thinking states”: what users see isn’t raw reasoning, it’s a translated progress view.
3. One citation system. Deep Research’s citations were merged into Harvey’s existing citation system. In their words: “This is essential for trust and verification in legal applications and also streamlines our development process.”

In a regulated industry, a black box is unusable. Visibility isn’t polish. It’s trust infrastructure.

7. The eval gate, and what they admit isn’t solved

The gate is offline

First, where evals actually act: not while the agent runs, but before a change ships.

Each team maintains datasets, evaluators and alert thresholds for its own bundles. A retrieval dataset, for example, holds a large set of queries, each with the recall it should reach across a set of knowledge sources. In Harvey’s words: “When any change is made to the system, developers can verify that their capability has not regressed.” Bundle changes and system prompt upgrades alike must pass a leave-one-out gate before they can deploy. If a metric drops below its threshold, the change doesn’t ship.

Three risks, one after-the-fact gate

Harvey lists three risks that come with many teams contributing to one agent:

1. The system prompt overrides a bundle: push the reasoning part of the system prompt toward fast decisions, and the model may stop calling certain tools
2. A bundle overrides the system prompt: if one bundle says “always call every tool,” the model starts calling other bundles’ tools too
3. Context rot: a new tool returns lots of content, the context window fills up, and quality drops

For all three, the source offers the same single mechanism, the offline gate above:

Risk	What the source says	A concrete fix?
Conflicting instructions (both directions)	Named as a risk, with the two examples above	Only detection after the fact: a bundle’s metric drops and the gate blocks the deploy. Nothing on how to avoid conflicts or resolve them
Context rot	Named as a risk	No. The post ends by saying they’re working on “context management,” with no detail
Untestable combinations	Listed as open: “how do we smartly test for a combinatorial number of Tool Bundles?”	No. Leave-one-out is the current practice; the post doesn’t explain how it works in detail

When several teams edit the same prompt, there’s no unit test for a conflict. ZenML’s summary nails it: “You’re no longer merging unit-testable code, you’re merging English.” ZenML also names leave-one-out’s cost: subtle regressions can slip through.

A gate can catch a regression. It can’t prevent a conflict.

Still unsolved

Two items on Harvey’s own to-do list speak directly to these risks:

• Best practices for system prompt and tool design. How to write instructions that don’t fight is something they’re still working out.
• Reinforcement fine-tuning to improve tool recall and “reduce reliance on prompt engineering.” My read: this is the route to fewer conflicts at the root. Move “when to call which tool” out of words in the prompt and into the model’s weights, and there are fewer instructions left to fight. But it’s a direction, not a result.

ZenML adds two more criticisms: a deep dependency on one SDK with no stated fallback, and no published numbers on development velocity.

So the full story of this plugin architecture is: conflicts, context bloat and combinatorial blow-up will happen; an offline gate catches regressions before they ship; the real fix is still being built.

The hardest part is people

Harvey’s words: “The hardest part of adopting agents isn’t writing the code — it’s learning, as an engineering org, to share ownership of a single brain.”

Engineers are used to owning their own systems. An agent architecture asks everyone to share one reasoning system.

That’s not a technical problem. It’s a culture problem.

8. How Harvey’s agent differs from Claude Code

This comparison is my own.

	Claude Code	Harvey Agent
Direction	Horizontal	Vertical
Domain	Any codebase	Legal workflows
Users	Developers	Lawyers
Tools	Bash, Read, Write, Grep…	LexisNexis, iManage, legal databases
Knowledge	General programming	Legal reasoning + 60+ jurisdictions + 400+ data sources
Interaction	You tell it what to do	It first asks what it needs
Output	Code, commits, PRs	Legal documents, memos, research with citations

To me the core difference isn’t technical. It’s who leads the conversation:

• Claude Code: you say “fix this bug,” and it goes and does it. You start things; it executes.
• Harvey: you hand it a research task, and it clarifies the scope first, then plans its own path, shows its progress and cites every conclusion. It leads; you supply context and judgment.

That’s not an accident. The heart of legal work is structured reasoning under uncertainty, and a lawyer’s value is judgment, not typing speed.

In a regulated industry, the AI asks and navigates. The human judges and confirms.

9. If you’re building AI for a regulated industry

Here’s what I’m taking with me:

1. Write the goals down before the architecture principles. Every principle should trace back to a goal.
2. Add capabilities, not agents. Package each capability as a self-contained bundle (tools + instructions) with its own offline evals, owned end to end by a feature team. An agent is just a config.
3. Let a framework’s limits discipline the org. Pick one SDK, force every feature through the same pattern, and accept losing some determinism.
4. Handle variants with parameters. Different regions, customers and sources are parameters of the same agent.
5. Spend human review only where the machine is unsure. Let automated evaluation sort most cases and keep people for the ambiguous ones.
6. Prove value inside defined workflows before handing out autonomy.
7. Treat visibility as infrastructure. Clarify first, show progress, cite every finding.

A lot of teams are still competing on how many agents they have. Harvey’s lesson runs the other way:

What actually scales is the one brain everyone shares, and the evals standing guard at its door.

Further reading

Harvey’s engineering blog (in recommended order):

1. Principles That Helped Us Scale Agent Development (Nov 2025): the three goals, the three principles, tool bundles, eval gates
2. Using Agents to Scale Harvey’s Knowledge Sources (Feb 2026): one agent with parameterized tools, the evaluation pipeline
3. Introducing Harvey Agents (Mar 2025): the layered definition of agents, workflows, thinking states
4. Integrating Deep Research into Harvey (Jul 2025): the Workflow Engine, the transparency principle, the citation system
5. A More Unified Harvey Experience (Jul 2025): merging the interface, expand then collapse
6. Harvey Raises at $11 Billion Valuation: the source for the 25,000+ custom agents

Outside analysis:

7. ZenML Case Study: a third-party summary of the first post, with critique

All images are from Harvey’s official blog; copyright belongs to Harvey. Passages marked “my read” are my own analysis, not Harvey’s claims.