Agent Harness Design: Choosing the Right Model

Model routing is an operational control problem, not a leaderboard exercise. Privacy and residency requirements define the admissible model pool first; task success and output quality decide what works within that pool; only then should teams optimize tokens, latency, and cost. Local models can be effective, but they usually need stronger decomposition, retrieval, and deterministic verification.

The Harness Is the Productive Unit

An enterprise agent is not a model with a long system prompt. It is a harness: intake, context assembly, policy gates, model execution, tool access, state, verification, retry logic, observability, and escalation. The model is one replaceable component inside that system.

Model comparisons made outside the harness frequently fail to predict production results. A model may perform well in a benchmark yet struggle with the organization’s formats, permissions, vocabulary, or workflows. Another may appear weaker in open-ended chat but perform reliably when the task is decomposed, evidence is retrieved, and outputs are checked against deterministic constraints.

Choosing “the best model” is therefore the wrong abstraction. We choose an admissible execution path for a defined task. The path may use one model, multiple specialized calls, a local model, a hosted model, or no model at all. It is successful only when the task reaches a verified business state.

The architecture should make this replaceability explicit. A thread-based AI operations layer separates user context from the executor, so routing, state, authorization, and verification remain stable when the model changes.

Gate One: Privacy, Residency, and Contractual Admissibility

Routing begins before quality testing. The first question is whether a model endpoint is allowed to receive the data and perform the action.

We create an admissibility profile for each task class. It covers:

• data classification and presence of personal, confidential, regulated, or export-controlled information;
• required processing and storage regions;
• retention, logging, training-use, and deletion terms;
• subprocessors and cross-border transfer constraints;
• encryption, identity, and access-control requirements;
• audit-log and incident-notification obligations;
• whether tools may read or mutate production systems;
• whether retrieved documents may leave a controlled environment.

These are routing constraints, not a policy paragraph attached after launch. If a task contains restricted data, the router must exclude endpoints that do not satisfy the profile. Redaction can sometimes create a less restricted derivative task, but redaction itself must be dependable, logged, and tested for the formats involved. Replacing a name while leaving a unique account number or commercially sensitive combination of facts is not meaningful anonymization.

Residency also applies to the surrounding harness. Sending prompts to an approved regional endpoint does not solve the problem if traces, vector indexes, error logs, or human-review queues are stored elsewhere. We inspect the full data path: intake, temporary files, retrieval, inference, tool results, observability, backups, and support access.

This gate produces permitted routes; it does not select a winner. A local model may be mandatory for one task, optional for another, and unsuitable for a third. Policy narrows the choice; evidence completes it.

Gate Two: Task Success, Then Quality

Within the admissible pool, the first evaluation question is task success. Did the workflow reach the intended state? For a research task, that may mean a supported answer with resolvable citations. For extraction, it may mean a schema-valid record tied to source spans. For an operational action, it may mean a valid dry-run plan, approved mutation, and successful readback.

Task success is broader than answer quality. An eloquent response that cannot be parsed, cites unavailable evidence, or calls a forbidden tool is a failed execution. Conversely, a terse output can be successful if it completes the process reliably.

We define an evaluation set from representative work. Each case contains the input, permitted evidence, expected state transition, hard failure conditions, and scoring rubric. The set should cover:

• ordinary cases that dominate volume;
• boundary cases near a policy or quality threshold;
• missing, stale, duplicated, or contradictory inputs;
• tool and dependency failures;
• prompt-injection or untrusted-content attempts;
• cases that must abstain or escalate;
• multilingual or format variation where relevant.

We score hard constraints before subjective preferences. Schema validity, correct tool selection, authorization compliance, citation support, and required abstention are pass/fail. Semantic completeness, clarity, and usefulness can use a rubric. A model that produces more polished prose does not compensate for a lower rate of verified task completion.

Domain-specific agent skills help stabilize this evaluation because procedures and required checks are represented as explicit artifacts instead of being rediscovered in every conversation.

Gate Three: Tokens, Latency, and Cost

Only after routes meet the success and quality boundary do we optimize efficiency. The relevant unit is cost per verified task, not price per input token.

A cheaper model can cost more if it needs retries, review, or downstream correction. A fast first response can have poor end-to-end latency if the harness repeatedly repairs invalid output. A larger context window can encourage indiscriminate loading, increasing cost and error surface.

We measure:

Dimension	Harness-level measure
Success	verified completions per eligible attempt
Quality	first-pass acceptance, defect severity, and escape rate
Tokens	input, cached input, output, and retry tokens per verified completion
Latency	median and tail time from trigger to verified state
Cost	inference, retrieval, tools, review, and rework per verified completion
Stability	performance variation across time, formats, and demand classes

Normalized indices make internal comparisons legible without implying false precision. Set the current approved route to Baseline = 100 for quality-adjusted cost and end-to-end latency. A candidate is useful only if its lower cost index does not conceal a worse task-success rate, higher exception load, or increased reviewer time. We keep the underlying counts and definitions for audit even when external reporting uses normalized values.

Optimization then becomes targeted. Prompt caching may reduce repeated policy context. Smaller retrieved passages may reduce distraction. Parallel tool reads may improve latency. A smaller model may handle classification while a stronger route handles ambiguous synthesis. Deterministic code may eliminate an inference call entirely.

Route Tasks, Not Users

Static routing by department wastes capability and money. One person may request a public summary, confidential comparison, reconciliation, and production write—task classes with different admissibility, quality, and authority requirements.

We route on a task envelope:

task_type: extract | classify | synthesize | plan | act | verify
data_class: public | internal | confidential | restricted
residency: approved_regions
authority: read | recommend | dry_run | approved_write
evidence: source_ids and freshness requirements
output_contract: schema or rubric version
risk: consequence and reversibility
latency_budget: interactive | asynchronous | batch

The router should be as deterministic as possible. Data class, residency, and authority are policy decisions. They should not depend on a model guessing whether a request “feels sensitive.” A classifier may assist intake, but uncertainty must choose the safer route or request a specific clarification.

Model choice then occurs inside the permitted branch. The route is recorded with the policy version, model version, prompt or skill version, retrieved evidence identifiers, tool calls, verifier results, and final state. This receipt makes failures diagnosable and comparisons reproducible.

Why Local Models Need a Different Harness

Local inference is often discussed as if moving the same prompt from a hosted endpoint to local hardware were enough. It rarely is. Local models can provide strong privacy control, predictable marginal cost, offline operation, and low latency for bounded tasks. They may also have weaker instruction adherence, tool selection, long-context reliability, or multilingual consistency than the strongest hosted alternatives available at a given time.

The right response is not to hide these differences. It is to change the harness.

Decompose More Aggressively

Give the model one bounded transformation at a time. Separate document selection, extraction, classification, drafting, and verification. Each stage should have a narrow input and output contract. Decomposition reduces the number of simultaneous instructions and creates checkpoints where deterministic controls can stop propagation.

A local route might first identify relevant sections, then extract claims into a schema, then map each claim to a source span, then draft from only the accepted claims. This takes more orchestration than one large prompt, but it is easier to evaluate and repair.

Retrieve Less, but Better

Dumping an entire repository or document collection into context is not retrieval. It transfers the selection problem to the model. Local routes benefit from structured indexes, metadata filters, recency constraints, and short evidence packets. Retrieval should return source identifiers and spans so downstream checks can confirm that the answer is grounded.

The practical patterns in building a local AI assistant with retrieval and web search illustrate why local inference depends on a surrounding tool and evidence layer rather than model memory alone.

Verify Deterministically

Use schemas, parsers, allow-lists, database constraints, calculations, policy engines, and readback checks wherever possible. A model may propose a change; code should confirm that required fields exist, identifiers resolve, totals reconcile, permissions allow the action, and the post-action state matches the approved plan.

For semantic checks that cannot be deterministic, use a rubric and an independent review path. Do not ask the same call to generate, grade, and approve its own output. If a model-based evaluator is used, calibrate it against human judgments and keep it outside the final authority boundary.

Cascades, Fallbacks, and Abstention

A useful router does not send every eligible task to the strongest model. It can use a cascade: start with the least expensive route that has already met the acceptance boundary for that task class, then escalate on observable failure.

Escalation signals should be concrete: invalid schema after repair, missing evidence, conflicting sources, low calibrated confidence, unavailable tool, policy boundary, or verifier failure. Token count alone is not a quality signal. Neither is the model’s verbal claim that it is uncertain unless that claim has been calibrated for the task.

Fallbacks need semantic consistency. If the primary route fails because residency requirements cannot be met, the fallback cannot ignore residency. If a tool call is unavailable, a chat-only route must not pretend that the action occurred. If the evidence set is stale, a stronger model cannot make it fresh.

Abstention is a successful outcome when the task contract requires escalation. We include “correctly refused or escalated” in task-success metrics. Otherwise, routing pressure rewards systems for attempting work beyond their evidence or authority.

State, Versions, and Reproducibility

Models and endpoints change. Even when a model name remains constant, behavior can shift with a new version, serving configuration, system prompt, retrieval index, tool definition, or policy. The harness must version the whole execution contract.

At minimum, we record:

• task and policy version;
• model provider, endpoint region, and model version;
• prompt, skill, and tool-schema versions;
• retrieval query, source IDs, and freshness timestamps;
• sampling and timeout settings;
• verifier versions and results;
• retry, fallback, and human-review events;
• final action and readback evidence.

Operational reproducibility comes from inputs, configuration, external actions, and verification evidence—not unrestricted chain-of-thought. Sensitive traces follow enterprise minimization and retention rules.

The procedural discipline described in tuning a project-specific agent is relevant here: persistent workspaces and reusable procedures are useful only when changes, concurrency, and failure modes are observable.

A Practical Selection Run

We use a staged selection process:

1. Define task classes. Separate extraction, synthesis, planning, action, and verification; do not average them into one score.
2. Apply admissibility gates. Eliminate routes that fail privacy, residency, contractual, or authority requirements.
3. Build the evaluation set. Use representative, boundary, adversarial, and must-escalate cases with frozen evidence.
4. Run routes inside the real harness. Include retrieval, tools, timeouts, retries, and verifiers.
5. Check hard failures. Reject routes that violate schema, evidence, authorization, or abstention requirements.
6. Score task success and quality. Review both average performance and the tail of severe defects.
7. Measure total efficiency. Compare tokens, latency, inference cost, review time, retries, and rework per verified completion.
8. Pilot in shadow or dry-run mode. Observe live demand variation without allowing unverified writes.
9. Approve by task class. Record the route, limits, fallback, and expiration date of the decision.
10. Continuously re-evaluate. Trigger review on model changes, drift, new data classes, repeated exceptions, or material price changes.

The output is a versioned routing policy, not a permanent ranking.

The Decision Rule

The order of decisions is the core of responsible model routing. Privacy, residency, and contractual gates determine what is allowed. Verified task success and quality determine what is capable. Tokens, latency, and total cost determine what is efficient among the routes that remain.

Reversing that order creates predictable failure. Cost-first routing sends sensitive work to inadmissible endpoints or low-quality work into expensive review loops. Model-first routing bends processes around a preferred vendor. Benchmark-first routing optimizes for tasks the enterprise may not actually perform.

We design the harness so models can change without dissolving policy, state, or evidence. Hosted models, local models, and deterministic services each have a role. The right choice is the smallest admissible system that completes the defined task, stays inside the quality boundary, and proves the result.

Related Insights

• Reference Architecture for a Thread-Based AI Operations Layer — routing, typed state, authorization, execution, and verification
• Building AI Agent Skills for Domain-Specific Business Workflows — externalizing procedures and domain-specific checks
• Building a Local AI Assistant with Web Search: MCP + Ollama Setup — a practical local inference, retrieval, and tooling pattern