aistack — from imec

Does your harness
matter more than
your model?

Your harness and model combination can double your token bill.

aistack//~11 min read
The takeawayYour choice of agent harness has less impact than we expected on successful resolution of coding tasks (when you stick to the more popular choices).Your harness and model combination determines whether the cost of a single resolved task doubles from $0.35 to $0.7. Multiply that across a real workload and your harness choice becomes relevant to your AI budget.Anomalies like agents doom-looping and prefix caching errors can have a major impact on your invoice, so keep an eye on your telemetry to prevent unpleasant surprises.
In this piece

People you should contact for more details.

A single bright amber beam entering from the left and fanning out into dozens of thin rays across a dark blue-grey background.

As more and more organizations adopt AI agents to put an extra layer of intelligence into the capable hands of their teams, new questions arise: Do we stick to frontier-lab models? When does it make sense to invest in our own GPU infrastructure? Should we switch to the latest and greatest open-weights model? How do we get the most intelligence out of our agents?

01

Intro

In our first blog post, we looked into the actual cost and performance of leading AI models deployed on a few reference hardware setups when used to solve real coding tasks. We kept things vanilla when it came to further optimizing these setups, but there are plenty of meaningful optimizations one can do across the stack. We promised a closer look at those, so here’s a first one that deserves an aistack deep-dive: the agent harness.

“The harness can often be the distinguishing factor that makes one LLM work better than another.”S. Raschka · Ahead of AI · [1]

A harness is the crucial layer that turns an LLM into a smart and capable AI agent. This is what you’d actually install and run, e.g., Claude Code, Codex, or OpenCode. While the model does the thinking, it’s the harness that decides what it thinks about and how, which tools it can reach for, and when to stop.

Harness— what you install and runsystem prompt & context managementtool orchestrationprompt formatting · output parsingerror recoverywhen to stopdecides how and what the model thinks aboutTaskrepo · issue · testsOutcometask resolved or notpromptscompletionscallsresultsModelQwen3.8-27B (FP8)does the thinkingInterfacesterminal · file editortest runner · web search
Fig. 01

Where the harness sits

Unlike models, which can be benchmarked across many flavors of tasks so you can pick the right one for your needs, harnesses are still mostly chosen on vibes, GitHub stars and X threads. Yet increasingly, your choice of harness seems to matter just as much, if not more than the model choice itself [1][2][3][8]. Thankfully, efforts to evaluate harnesses are on the rise [4]. The reported difference when swapping them can “often be the distinguishing factor that makes one LLM work better than another” [1]. Bold statement, so our interest was piqued and we started running the numbers to find out if that is really the case.

02

Our approach

Coding agent harnesses operate by giving the model access to relevant tools (a terminal, file editors, test runners, web search, etc.) and handle context management, prompt formatting, error recovery and output parsing.

In ‘How many devs can you fit on a GPU?’ we held the harness fixed and varied the hardware. This time we are swapping only the harness, comparing 3 of them across 2 models running on a suitable hardware setup. To figure out whether it’s worth swapping harnesses at all when deciding on your own AI stack, we’re looking at 3 things of note:

  • The token efficiency of the 3 harnesses (this will drive cost, time and concurrency)
  • The prefix cache hit rate (often overlooked, but a good measure of smart or bad context engineering and a big cost driver)
  • And, of course, accuracy (token counts don’t matter if those tokens don’t resolve the task you’re trying to get done)

For this deep dive, our aistack team picked two of the latest models at the time of writing:

  • Qwen3.8-27B (HuggingFace card) in FP8 precision, a solid smaller model that fits on a single NVIDIA H200. It scores 52 on the Artificial Analysis Intelligence Index [6] and 51 on their Agentic Index, ahead of models several times its size.
  • The freshly revealed GLM-5.3-Flash in FP8 (known previously as the mysteriously hyped “Ox alpha”) (HuggingFace card), running on 4 NVIDIA H200s.

As always, realism is key for us. We fully simulate the real SWE development workflow: each developer in our evaluation framework is a separate harness worker with its own environment, resources and access to the web.

We reused the same benchmark of 64 coding tasks we curated from SWE-Bench Pro for our first article.

While there are many options to pick from, we compared 3 well-known coding harnesses: Claude Code, Codex, and Pi. The selection was based on popularity within agentic coding and what our developers currently use, but you need only look at GitHub to find many more. We excluded harnesses that aren’t coding-specific, but as promised, we’ll come back to other agentic tasks in future posts. We ran each harness with its default settings and kept the inference engine configuration identical across all three.

We didn’t apply any customisation to the harnesses and evaluated them as-is.

We did our best to get as close to comparing apples-to-apples as it can get, though in our next post you’ll see that our experiment took a slight detour when we zoomed in on what the agents were actually doing. (Yes, they were naughty.) More details on what we observed and how the team handled it will follow in a couple of days.

For now, here’s what our harness comparison can tell you about how to optimize your stack. Let’s dig in.

03

Results

Resolve rate: surprisingly not where the harness matters

All three harnesses land at a 44–53% resolve rate for the coding tasks we threw at them across both models. Surprisingly, the harnesses we selected barely make a difference to accuracy in our tests, with a variation of about 2 to 3 tasks resolved across our runs (see the figure below). If you are picking a stack based on resolve rate alone, save yourself the benchmarking and analysis paralysis – at least for now. Go with what you prefer.

Your mileage may vary with the long (long) tail of other harnesses out there, of course. (Let us know if there’s a standout one that’s bound to make a difference here.)

0%25%50%75%100%48%31/6452%33/64Codex52%33/6450%32/64Claude Code44%28/6450%32/64Pitasks resolved out of 64Qwen3.8-27B · 1×H200GLM-5.3-Flash · 4×H200
Fig. 02

Resolve rate per harness and model, on a 0–100% axis

Tokens are where we start seeing the real difference

When looking at token efficiency, both input and output tokens show variations worth addressing. While most harness/model combinations generate about 3–4M output tokens on our benchmark, Codex with GLM-5.3-Flash generates only around 2.1M output tokens for about the same accuracy. This is important because, compared to the relatively fast processing of input tokens, generating these output tokens is a considerable part of your total wall time. This means each session will hold its GPU slot longer, so the same hardware will resolve fewer tasks per hour (and thus burn pricey GPU hours, but we’ll get to that).

0M1.25M2.5M3.75M5M3.33M2.1MCodex3.72M3.97MClaude Code3.72M3.79MPioutput tokens generated per sweepQwen3.8-27B · 1×H200 ($4.54/hr)GLM-5.3-Flash · 4×H200 ($18.16/hr)
Fig. 03

Output tokens per 64-task sweep, per harness and model

Input tokens are worth a look as well. On Qwen3.8-27B, Claude Code pushes around 445M input tokens through the model, whereas Codex pushes only 330.5M. That is 1.35× as much context to solve just one more task. On GLM-5.3-Flash, the gap is even bigger (480M vs 192M). Claude Code consistently creates a higher context volume, and in our benchmark, it does not translate into more solved tasks. Talking more doesn’t make you smarter, Claude.

0M125M250M375M500M330.5M192.1MCodex445.8M480.2MClaude Code348.5M366.4MPiinput tokens pushed through the model per sweepQwen3.8-27B · 1×H200 ($4.54/hr)GLM-5.3-Flash · 4×H200 ($18.16/hr)
Fig. 04

Input tokens per 64-task sweep, per harness and model

The lesson here is that throwing more context at the model doesn’t mean it solves more tasks. It just makes each solve more expensive. Clever context engineering can have a significant impact on the cost.

From your infrastructure’s perspective, that larger context means a bigger KV cache and therefore more pressure on your GPU’s available memory. That matters when you’re aiming at higher concurrency or higher token throughput.

HarnessModelResolvedWall timeAgent timeInput tokensOutput tokenstasks/hresolved/h
Claude CodeQwen3.8-27B33/64 (52%)2.26 h16.6 h445.8 M3.72 M27.013.9
CodexQwen3.8-27B31/64 (48%)2.07 h14.5 h330.5 M3.33 M31.115.0
PiQwen3.8-27B28/64 (44%)2.24 h15.5 h348.5 M3.72 M27.412.0
Claude CodeGLM-5.3-Flash32/64 (50%)2.48 h15.25 h480.23 M3.97 M25.812.9
CodexGLM-5.3-Flash33/64 (52%)1.26 h6.6 h192.14 M2.10 M50.526.0
PiGLM-5.3-Flash32/64 (50%)2.48 h15.9 h366.4 M3.79 M25.912.9
tasks per hour01530456031.150.5Codex2725.8Claude Code27.425.9Piresolved tasks per hour07.51522.5301526Codex13.912.9Claude Code1212.9PiQwen3.8-27B · 1×H200GLM-5.3-Flash · 4×H200
Fig. 05

Throughput: tasks per hour and resolved tasks per hour, per harness and model

KV cache utilization for Qwen3.8-27B with the three harnesses on 1×H200 (concurrency 8)

metricCodexPiClaude CodeΔ Pi vs CodexΔ Claude Code vs Codex
KV cache used, mean22.2%24.6%32.4%+10.8%+46.3%
KV cache used, p5022.2%24.1%32.6%+8.2%+46.7%
KV cache used, p9034.9%40.5%49.9%+16.0%+42.9%
KV cache used, max59.8%72.1%78.8%+20.5%+31.6%
share of run at ≥90% KV0.0%0.0%0.0%0.0%0.0%
share of run at ≥98% KV0.0%0.0%0.0%0.0%0.0%

Utilisation as a share of the KV-cache pool; the Δ columns are relative to Codex.

0%25%50%75%100%22.2%34.9%59.8%Codex24.6%40.5%72.1%Pi32.4%49.9%78.8%Claude Codeshare of the KV-cache pool in use, Qwen3.8-27B on 1×H200 (concurrency 8)meanp90max
Fig. 06

KV cache in use on Qwen3.8-27B: mean, p90 and peak per harness

According to our measurements, with the same inference engine configuration, Codex puts less pressure on the KV cache: peak utilization reaches ~60% of the total KV cache pool, which is noticeably less than Pi (72%) or Claude Code (79%).

So while resolve rate stays roughly constant, your choice of harness has a significant effect on the token efficiency of your stack, which matters if you care about total wall time and developer experience. Read on to discuss what that means for your bill.

If you need a refresher on input & output tokens and the intricacies of KV cache, see our earlier article “What is your GPU waiting for?”.

That wall time is what feeds your invoice

When you rent or buy your infrastructure, the GPUs stay running whether the model is reasoning or tool calls are being executed. So that total wall time ends up costing you money either because you’re paying the GPU rent by the hour or because the TCO of those GPUs you bought is heavily dependent on how many useful tasks they resolve for you.

On Qwen3.8-27B (1×H200 at $4.54/hr on Modal), Codex wraps the full 64-task sweep for about $9. Using the Claude Code harness instead costs $10.26. Pi costs $10.16. No major differences, unless we’re talking scale here. But look at GLM 5.3 Flash on those 4×H200s ($18.16/hr on Modal) and the differences get uncomfortable pretty fast. Using Codex you’d pay $22.8, but using Claude Code as a harness that increases to $45. Pi: $45 as well. That’s nearly double as costly for a very similar intelligence, just based on switching your harness. All still considerably cheaper than paying for API pricing of course. Remember from our earlier post that the same workload would cost you around $98 based on API token prices via Anthropic.

$0$12.5$25$37.5$50$9$22.8Codex$10.26$45Claude Code$10.16$45PiGPU cost per 64-task sweep (wall time × hourly rate)Qwen3.8-27B · 1×H200 ($4.54/hr)GLM-5.3-Flash · 4×H200 ($18.16/hr)
Fig. 07

GPU cost per 64-task sweep, per harness and model

So yes, picking the right harness and model mix can have a considerable impact on your total cost, whether you’re renting or buying.

Anomalies can get costly

Throughout our tests, we bumped into two anomalies that ended up impacting the performance and thus the cost of our runs significantly.

Doomlooping – when agents get stuck

First up, we noticed a strange recurring anomaly using the Pi harness during our experiments. One or two worker-agents constantly got stuck in a so-called ‘doom loop’, trying to solve their tasks. We observed the same behaviour for Qwen3.8-27B (xhigh) and GLM5.3-Flash (max) models, the tasks where the agents got stuck each time were different.

Eventually, the models collapsed, generating repetitive gibberish. The agent workers were killed by our orchestrator system after reaching the 4-hour duration limit. To be fair to Pi and assuming a fix is incoming, we corrected for this and all the measurements for Pi in this article are based on the mean estimates and the runs where we didn’t observe this behaviour. The runs where we did see this behavior would end up significantly increasing your total wall time.

LOGS
timum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\nAuto packing the repository in background for optimum performance.\nSee \"git help gc\" for manual housekeeping.\n
Fig. 08

A Pi worker in a doom loop: the same two sentences, repeated until the orchestrator killed the run

When prefix caching fails

A second anomaly pertains to prefix caching, which is a crucial inference optimization technique that directly impacts latency, token throughput, and your final bill. Prefix is the static part of the context, the past conversation history. Instead of recomputing the KV cache for the long agentic trajectories over and over again, we can cache it. The higher the prefix cache hit rate, the better for inference efficiency and end-user experience.

However, this mechanism is easy to break. Imagine a dynamic element at the very beginning of the model context, usually the system prompt. This could be a date, a constantly changing UUID or anything else and now your prefix is no longer static, which means nothing after it can be cached.

Sounds like an edge case, but that’s exactly the problem we observed in one of the recent versions of the Claude Code harness. After updating to the newer version that came out during testing, we noticed severe deterioration in time to first token (TTFT), generation throughput and most important — prefix cache hit rate, which is a clear sign that the caching is broken (github.com/anthropics/claude-code/issues/87227).

0%25%50%75%100%97.36%Claude Code · patched releasestable29.84%Claude Code · v2.1.233broken cachingmean prefix-cache hit rate on Qwen3.8-27B (p10–p90: 95.9–98.6% vs 23.4–37.8%)
Fig. 09

Mean prefix cache hit rate on Qwen3.8-27B: patched Claude Code vs. v2.1.233

Prefix cache hit rate measurements

HarnessModelmean%p10p50p90p99lohi
Claude Code v2.1.233Qwen3.8-27B-FP829.8423.3928.7037.8153.3617.4566.50
HarnessModelmean%p10p50p90p99lohi
CodexQwen3.8-27B-FP896.5895.3597.3098.3098.8250.2398.92
PiQwen3.8-27B-FP895.5091.2797.3898.4599.190.0099.31
ClaudeQwen3.8-27B-FP897.3695.9297.8498.5798.8984.1198.96
HarnessModelmean%p10p50p90p99lohi
CodexGLM-5.3-Flash94.1791.8494.8296.5197.8171.1597.90
PiGLM-5.3-Flash94.4390.3896.5397.9898.610.0098.84
ClaudeGLM-5.3-Flash96.7295.2397.3898.3598.9070.6899.01

We didn’t observe a significant prefix cache hit rate difference between the evaluated harnesses, except for the anomaly we spotted using version 2.1.233 of Claude Code. The Claude Code numbers reported in tables #2 and 3 are for the patched version of Claude Code.

Anomalies like the Doom Loops we observed when using Pi and the Prefix Caching issues are clearly temporary as they get fixed over time. But this does mean that blindly updating to newer versions can considerably impact your invoice at the end of the month, and not just by a few %. When you own your own stack, this is something to be very mindful of and monitor.

04

Conclusion

We started this deep dive expecting resolve rate to be the headline of the blog post. Everyone loves a good story where there’s a clear winner. Turns out it’s the less flashy metrics, like wall time, input and output token counts and harness stability, are where the real costs hide, halving your costs in some cases. If you are picking a harness for a self-hosted setup, be prepared to check how your specific harness/model setup holds up for these metrics.

Of course your dev team may not be willing to switch from one harness to the other every week, as a harness impacts developer experience. But as you start automating your software development workflow, switching around the harness that is driving your automated work is easier and impactful. Just make sure to keep an eye out for those anomalies as you upgrade from version to version (as you would with any good software stack) by looking at the right metrics.

Speaking of keeping an eye out: This is a snapshot. Today that Codex running GLM5.3 Flash combo did raise our eyebrows when it comes to the amount of intelligence you get for that lower cost. But both models and harnesses evolve fast, so next month the winning combo may be an entirely different one. But you know where to look for now. And we’ll be updating our numbers too.

Before we end it here, two things to look forward to:

While we’re pretty confident about the numbers in this article, the truth is we almost weren’t. When we looked closer at what the agents were actually doing, we found behavior that made us rerun the entire experiment. And again. What we found, how we fixed it, and why it changes how you should read most benchmarks out there is coming in the next post early next week.

And finally, harnesses were our first look into how to optimize your stack. Next up? What difference does your inference engine make. Stay tuned for this one in a week or two.

Make sure to subscribe if you’d like to get a heads-up when these new articles launch.

References

  1. S. Raschka, “Components of a Coding Agent,” Ahead of AI, Apr. 2026.
  2. P. Winder, “A Comparison of AI Agent Harnesses in 2026,” Winder.AI, Aug. 2026.
  3. htek.dev, “All Agent Harnesses: The Live Comparison,” May 2026.
  4. Y. Yao et al., “Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows,” arXiv:2605.27922, May 2026.
  5. swyx et al., “Zawinski’s Law of MultiAgents,” Latent Space AI News, Aug. 2026.
  6. Artificial Analysis, “Qwen3.8-27B,” accessed Aug. 2026.
  7. D. Vaughan, “Context Compaction Deep Dive,” Codex Knowledge Base, Apr. 2026.
  8. Aikido Security, “How Aikido finds more vulnerabilities than Mythos at half the cost,” Aug. 2026.

Join the conversation
on LinkedIn

We publish every aistack insight on LinkedIn — add your read, push back on the benchmarks, or follow along for the next one.

Discuss on LinkedIn

Read next

A high-end AI accelerator board on a dark surface, its silicon die glowing warm amber-gold

How many devs
can you fit on a GPU?

A reality check on self-hosting the model behind your coding agent — what 64 real coding tasks reveal across GPUs, rentals and APIs.

Full story › Extreme close-up of a silicon die: dense circuitry glowing warm orange on one side and cooling to blue on the other.

What is your GPU
waiting for?

A back-of-the-napkin guide to why one chat barely registers and a few agentic sessions make your GPU sweat.

Full story ›