Harness your
expectations

In the last few months our aistack team has been on a quest to get a grip on what it takes to own your own AI stack. We’ve looked into the differences in cost and performance when using APIs, renting or buying GPUs, and started identifying the most promising ways to optimize your stack. While we’re far from done in that regard, today’s post is about something different, but we felt it important enough to report back on. In the previous post we did a deep dive on what happens if you switch to different coding harnesses while keeping your AI model and underlying hardware the same. As we tasked different model/harness combinations to solve long-horizon coding problems, we found we couldn’t make sense of our initial experimental results and started being suspicious about our setup.

Turns out, we were suspicious for good reason. When we looked at what the agents were actually doing, instead of writing their own code, we caught them ‘cheating’. They were pulling solution commits from git history, fetching upstream PRs from GitHub and one even proudly citing the original patch author by name. It turned out our benchmark was measuring how fast the model could find the answer key and coincidentally, whether it felt bad about it. (Some did. Briefly. We kid you not.)

So before we could trust any further model/harness ranking results we had to even out the playing field by patching our evaluation environment first. We did what was necessary and banged our heads against the wall (technically speaking, against our keyboards) so you don’t have to. Read on to learn more on how to evaluate your model’s actual skills to resolve the tasks that matter to you and not just its ability to shortcut its way to the top of your rankings.

Methodology

As we mentioned in our previous blog post, we fully simulate the SWE workflow. Each ‘developer’ in our setup is a fully autonomous agent with its harness, environment and access to the web. We tell these developers to solve a subset of 64 curated SWE-Bench Pro long-horizon coding tasks.

We instantiate one agent per task. Inside the sandbox, the agents have access to all kinds of tools, they can search the web, write and execute unit tests, etc. The agents get retired when the task is reported as solved. We deterministically evaluate the solution against the golden unit tests once the task is reported as solved by the agent.

In our initial setup, the instructions these agents got were simply the unmodified version of the instructions that already come as part of each of the SWE-Bench Pro tasks.

As this was a first exploration for the team on what impact an agent harness has on performance & cost of your stack, we kept things simple with 2 models evaluated across 5 popular harnesses.

Models

• Qwen3.8-27B-FP8 (HuggingFace card) on a single H200.
• GLM5.3-Flash-FP8 (HuggingFace card) on 4 x H200.

Harnesses

Codex, Pi, Claude Code, OpenCode, DeepSeek. All set to xhigh reasoning effort. Only the first three let us modify the system prompt.

Variance measure

Each harness swept the full task set 3 times. Results reported here are from the latest calibrated sweep (late August 2026).

Monitoring

All runs were tracked in Benchy, our internal benchmarking platform, with full task-level traces, token counts (input, output, cached), inference engine telemetry (throughput, KV-cache usage, TTFT, prefix cache hit rate, etc.) and GPU telemetry (utilization, memory usage).

While inspecting agent traces for the newly released models, we noticed something was off. Some of the agents were making HTTP requests to GitHub mid-task. We quickly realized they were looking for ways to pull upstream PR diffs for the exact repo and issue they were working on. We went back and checked the Qwen 3.6 traces we did for our first post on harnesses, an older model, and found no sign of such behavior. So this was something new.

We first caught the behavior with GLM-5.3-Flash: across all five harnesses, 2/3 of all executions (213 out of 320) had located and used the gold fix commit (the commit that merged that PR). The fix to each task was sitting right there in the sandbox’s git history (the Docker image shipped the full repo, fix commit included), and the agents found it through git log --all, opened it with git show and naturally, applied it. One DeepSeek execution even credited the answer key’s author by name:

Three routes to the answer

Specifically, we found three different pathways to retrieving a solution instead of creating one from scratch:

1. The git history shipped inside the task image
2. The internet (GitHub, PyPI, upstream project pages)
3. The model’s own training data

We closed the first two (the third is an inherent part of the model) and observed a considerable effect on the results.

“Is this cheating?”

When we first observed this behavior, we started having nearly philosophical discussions internally. Is this behavior cheating, or just exploiting the execution environment? Fun fact: it wasn’t just us having these discussions. Our agents asked themselves the very same question, sometimes agonizing over the choice to access the answer, other times justifying it. Read for yourself (these are direct quotes from the models' reasoning traces):

To settle the question, it helps to go back to what the agent was actually told to do. It seems the task instructions (cf. SWE-Bench Pro) explicitly encourage looking around before writing code:

Under that reading, the agent isn’t breaking any rules, it’s following them. This instruction was clearly well-intentioned by the SWE-Bench Pro folks at the time and was meant to instruct the agent to gather much-needed context to write the right code by looking at related files. It’s just that newer models now seem to interpret that as (and are more capable of) also looking up the benchmark task solution itself. Let’s look at what we did to prevent this from happening.

Because we use benchmarks like SWE-Bench Pro as proxies to emulate real-life workloads and their impact on your AI stack, it’s important to us that these behave predictably across different models/setups and allow us to compare apples to apples on equal grounds. So we set out to snuff out this behavior to ‘use shortcuts’. (The numbers in our harness blog post are already based on these fixes, by the way.)

Fix 1: scrubbing git history

The original task images of SWE-Bench Pro contain a full local copy of the git history, including “future” commits and thus the solution patch. While this information is often needed to grade the solution provided by the agents, it should of course not be accessible during its work. We have created new images of these tasks, ensuring the presence of the necessary git commits only during verification. Similar solutions are applied in newer benchmarks such as DeepSWE and harbor-index (which adapts some SWE-Bench Pro tasks) by explicitly separating out the verification step.

We rebuilt the task images to strip the solution from the git history and reran the experiments. Here is a snapshot of the results (tasks resolved of 64).

Qwen 3.8 27B — the git scrub cost ~12 percentage points:

GLM-5.3-Flash — the git scrub mattered less (except for Codex):

Fix 2: the prohibition prompt

Scrubbing git history was the most effective single fix for Qwen 3.8 (cutting ~12pp), but insufficient for GLM. We know 67% of GLM executions found and used the fix commit in the original environment, yet removing that route barely dented the resolve rate. This looked at first like the most surprising result of this investigation… until we fixed the second issue and the GLM mystery was solved.

To restrict the harness agents from fetching the correct solutions from git history and the web, we appended a concise instruction to the harnesses’ original system prompts:

We were only able to apply this fix to 3 out of 5 harnesses (Claude Code, Codex and Pi), giving us the following results (tasks resolved of 64).

Qwen 3.8 — the prompt cost another ~17 percentage points:

That is roughly 29 percentage points from the baseline to the fully patched one. An interesting fact is that the Qwen 3.8 27B fully obeyed the new instructions and stopped fetching the answer from elsewhere:

GLM-5.3-Flash — the prompt clearly makes the difference:

Remember the git scrub barely moved GLM’s numbers? That is because the model was rerouting through internet fetches and training data recall. As we can see in the scores, the system prompt is what successfully seems to constrain those routes (for now). The drop is substantial with ~33 percentage points on average. GLM-5.3-Flash is the bigger model, and on the original environment it was comfortably ahead of Qwen 3.8. Once the shortcuts are closed, the two land in the same ballpark, converging to roughly 40–53% resolution rate. One way to look at it is that the gap between them on the original environment was largely a gap in exploit finding ability rather than in engineering ability, and that this subset does not separate them once the exploits are removed. We don’t want to over-interpret 64 tasks, but it is a reminder that a resolve rate on a leaky environment measures something different from a resolve rate on a clean one, and that the two can rank models differently.

Compliance, and the one exception

The agents clearly registered the prompt: they quoted the ban verbatim in at least 37 executions while declining to fetch. Once given the right instructions, our models follow these pretty well (as they should). Here are some of our favorite quotes from their reasoning traces:

Amusing, no? It’s hard not to anthropomorphize these agents at times, but it’s the right thing not to do so.

Adjusting the prompt worked in almost all the cases, with one single task where Claude Code on GLM lawyered its way to an exception. As seen in its reasoning:

It then issued 8 webfetch calls to Microsoft’s history pages, retrieved the exact KB numbers that make up the gold patch’s payload and was the only harness to pass the task.

And then the leak you can’t close: “I remember...”

That one exception aside, the models complied with the prohibition on external routes. But are we now really sure that each solved task is resolved without prior knowledge of the right answer? Sadly no, because some of our models might have had the reference answers all along… in their weights. And this one is harder to adjust for.

In 94% of executions where reasoning was visible (Claude Code and Pi include thinking blocks but Codex unfortunately suppresses them), the model attempted to recall the upstream fix from training data. In a very specific, obvious manner as seen in the quotes below. It would identify the repository, name the date of the fix, cite a PR number and sometimes reconstruct the patch code verbatim. The model knew repository meta history (“matrix-react-sdk was merged into element-web in December 2023,” “the repo at the time was markphelps/flipt”). The agent even cited specific PR numbers in 57/64 Claude Code, 58/63 Pi, and 17/64 Codex executions:

GLM did the same thing once the prompt closed its internet route. The recall attempts got increasingly desperate:

Those recalls weren’t always accurate. We get it, we all remember we had dinner but not all of us can tell what was on our plate given sufficient time has passed. The model frequently self-corrected (“PR #11151 — hmm, no”) and sometimes, to our surprise, it explicitly abandoned it: “Let me stop relying on memory. The PR description here is the spec.” But the strategy was consistently retrieval-from-memory, not engineering from a spec across all 3 harnesses. We cannot state the model was directly trained on these benchmarks, but the benchmarks are based on real problems from existing repos that models have definitely seen during training. All these observations remind us again: when it comes to gauging intelligence, take all benchmark results with a grain of salt.

Across all the experiments we ran, recurring behaviors appeared across models and harnesses. When the obvious shortcuts to the answer are unavailable, the model doesn’t just return to “solving the bug”. It becomes more cautious, spending longer reasoning about hidden tests and gathering context before committing to a solution. The following behaviors show up across every harness and every newer model we examined.

Evaluation Awareness in Reasoning

Some of our captured trajectories consist of multiple instances of reasoning about what exactly the hidden tests are, instead of getting a ‘correct’ solution/fix. Even though the objectives overlap, a developer aims for correctness and maintainability, not for maximizing hidden-test pass rate. The model tends to maximize benchmark success probability instead. The harness doesn’t cause this, but it does modulate how much of it we see. Pi is more reserved in doing this, usually confining such reasoning to a single block, while Claude Code and Codex exhibit the pattern more explicitly. This pattern has a name in the AI alignment literature, and it’s worth introducing before we move to examples.

One hypothesis is that this behavior reflects evaluation awareness, models recognize the structure of a benchmark or a harness and adapt their reasoning to what they infer the evaluator will reward. AI alignment research [1] shows that frontier models can distinguish evaluation from deployment contexts and often infer what an evaluation is testing. Anthropic has reported a similar phenomenon in its own models: Claude Sonnet 4.5 showed higher rates of evaluation awareness than previous Claude models and was more likely to recognize that a scenario appeared to be a test [2]. They caution that if such awareness changes model behavior, evaluations may become less predictive of real-world behavior. The underlying mechanism remains an open question. A plausible contributor is reward-hacking pressure from reinforcement learning combined with the composition of modern training pipelines, where repeated exposure to benchmark-shaped tasks may teach models to recognize benchmark-shaped inputs. Regardless of the mechanism, the concern is direct: if models behave differently when they detect an evaluation, then benchmark scores may tell us more about models’ performance under evaluation than their performance in deployment.

The trajectories themselves show what this looks like in practice:

Agents reason around the rule “don’t modify tests” again and again

We found in trajectories that the model tries to update the test cases so that the solution passes the test suite but then gets into a conundrum as it is specifically instructed to not update the test cases. The model doesn’t attempt the forbidden edit, but it also can’t move on immediately. Instead it re-reads the constraint, reweighs the options and revisits the same deliberation later. The following is a sample of reasoning thoughts while attempting to solve a task:

Over-verification before committing

Across trajectories, the phase between “I think I have the fix” and “I am ready to submit” is the longest part. And it gets longer when the model has clocked that it’s being tested. This shows up in three ways: re-reading files already read, re-running tests against unchanged code and the tool usage ratios overall. In one of the Claude Code trajectories, the same source file was opened three times including two back-to-back reads at steps 43 and 44. In one of the Pi trajectories, pytest was invoked 13 times across 95 total calls, so 14% of the session was verification of the same unchanged code. Across the trajectories, we observed that information gathering calls outnumber the modification calls by roughly five to one.

We went one step further and split the trajectories into two piles: ones where the CoT contains any hint of eval-awareness (reasoning about hidden tests, calling it a benchmark task, that kind of subtle hinting) and ones where it doesn’t. The second pile is small. Keeping only trajectories where the classification was clear-cut, 260 of 278 (93.5%) showed some eval-awareness. In Codex, aware trajectories verified noticeably longer with both models: a median of about 22 vs 6 on GLM, and about 55 vs 29 on Qwen. In Claude Code and Pi, the gap was small or went the other way. This comes from one sweep with confounds we haven’t chased down yet, so treat it as a lead for our next research insights rather than a finding. But it’s the first spot in our data where eval-awareness shows up as a number instead of a quote from a trace.

Shell access extends the harness

One more habit that runs through nearly every trajectory is that when the harness provides a dedicated Read tool, the model still opens files with cat, head and tail. With Claude Code, even though it exposes a Grep tool, you can find instances in trajectories where the model still pipes find into xargs grep. Piping shell commands combines what would be several tool calls into one, which is efficient. But it also means the shell effectively extends the tool inventory of a harness. The tools the harness declares don’t completely define what the model can do.

None of these behaviors affect the resolve rate, which is why they are easy to miss. Two trajectories that both pass can look nothing alike. One spends its reasoning guessing at the hidden tests instead of thinking about what the code should do. Another loops on the same instruction for several steps, re-runs tests against unchanged code, or reaches past the harness’s declared tools through the shell. The score counts both as a win. Evaluation awareness is the worst version of this: if a model recognizes it’s being tested and adjusts its behavior, the score partly reflects how it acts under evaluation rather than its actual engineering capabilities.

• The agents exploited the sandbox environments and retrieved the correct solutions from the environments’ git history. 2/3 of GLM executions pulled the solution commit straight from git history. One even credited the original author by name.
• We closed the git route. The models shrugged and rerouted through internet fetches and training data recall. We closed those too by appending one sentence to the system prompt. Resolve rates dropped from 75%+ (Qwen 3.8 27B) and 90%+ (GLM-5.3-Flash) to 40–53%. Eval leakage is devastating, no joke.
• Turns out you don’t need to block egress to stop the fetching, asking nicely does most of the job. The agents quoted our ban verbatim in 37 executions while declining to fetch. One Claude Code run read the sentence very carefully, decided that Microsoft’s KB numbers are domain data and not “the solution”, fetched them anyway and was the only harness to pass that task. We gave it the point.
• In 94% of visible reasoning traces, the model tried to recall the fix from its pre-trained memory: citing PR numbers, naming dates, reconstructing patches verbatim. Sometimes right, often wrong, always confident. We can’t close this one. The benchmarks are built from real repos and the models have read them.
• GLM-5.3-Flash is the bigger model and beats Qwen 3.8 in benchmarks comfortably. On our subset, both landed at 40–53% once we took the shortcuts away. So the gap we measured at the start was mostly a gap in how good each model is at finding the answer key. Same tasks, different environment, different ranking.
• Once the shortcuts are closed, the models often don’t go back to “fix the bug”. They reason about what the hidden tests want, re-read files they already read, run pytest 13 times in one session and pick whatever is least likely to break a test they can’t see. If models behave differently when they detect an evaluation, your benchmark score is measuring their evaluation behavior, not their deployment behavior.
• Two trajectories that both pass can look nothing alike inside, and the score counts them the same. When we split traces by whether the model had clocked it was being tested, 260 of 278 showed some eval-awareness.

To round things up: if you’re benchmarking coding agents internally, your eval environment becomes part of your result. A single fix won’t do it. Strip the git history, constrain the prompt, block egress if you want to close the last gap, pin your harness version, and actually read the trajectories. Do all the fixes for an airtight environment or the number you report is measuring something other than what you think.

What’s next from here for team #aistack at imec? Now that we understand this agent behavior much better, we can adapt our own benchmarking pipeline and setup so that our upcoming deep dives keep on reflecting actual agentic usage as accurately as possible. As mentioned, this was already reflected in our previous deep dive on agent harnesses. In our coming few deep dives we’re continuing our focus on getting even more out of your AI stack by identifying and measuring the most impactful optimization techniques out there. Tune in soon to learn more about what optimizing your inference engine can bring to your table (and your token invoice!). If you’d like a heads-up when that is published, make sure to register here (we won’t spam you, pinky promise).

1. Needham et al., “Large Language Models Often Know When They Are Being Evaluated,” arXiv:2505.23836, May 2025.
2. Anthropic, “Claude Sonnet 4.5 System Card,” Transparency Hub, accessed Sept. 2026.