GrantTap field guide · · 15 min read
Evaluate a coding agent on work you can actually verify
A practical evaluation method for your own repository: comparable starting conditions, meaningful acceptance, human intervention, observable evidence and a specific routing decision.
Choosing a coding agent is easiest to discuss when the conversation starts with a particular piece of work. A public ranking can help you discover a tool, but it cannot tell you whether that tool can repair your project's queue, understand its conventions, preserve unrelated changes, and leave a result you can review. The useful question is narrower: under conditions you can describe, does this complete agent route finish the task you actually need? That route includes the model, the installed client, its permissions, the computer, the workspace, available tools, and the person who reviews the result.
This is a practical method for building your own small evaluation. GrantTap can help you follow the task, read available activity, inspect recorded changes, and make decisions from another device. It does not turn those observations into an independent benchmark or guarantee equivalent integrations across providers. The examples below are proposed experiments, not claims about completed tests or provider superiority. The article occupies an October editorial slot; its instructions and sources were prepared and checked on November 5, 2026.
Begin with the failure you want to stop seeing
Write down one real moment in which your workflow disappoints you. Perhaps a message looks delivered while the agent never starts, a generated change fixes one screen but breaks another, or a long task ends without a usable handoff. Describe the action, expected result, observed result, and environment. Replace an abstract goal such as improve agent reliability with a scenario: after a reconnect, one queued instruction reaches its original conversation exactly once, and the interface distinguishes a computer receipt from agent activity. This wording gives the evaluator something concrete to inspect. It also makes a failed attempt informative instead of producing another general argument about model intelligence.
Keep the initial task small enough that a reviewer can understand the affected behavior. A repository-wide modernization asks the agent to make many independent decisions and makes comparison difficult. A bounded repair can still involve several files, but its acceptance should fit into a short description. State which existing behavior matters and which changes require additional discussion. In GrantTap, keep the Task associated with that objective so its later Executions do not become unrelated experiments. Record the original defect before providing hints about an implementation. Otherwise you may end up measuring how well the agent copies your proposed patch rather than whether it can diagnose the problem.
Define acceptance before watching an attempt
Create a short acceptance record before running the agent. Include a functional check, an important regression, and a review condition. For the queue example, the functional check might observe one operation through reconnect. A regression might preserve the draft when storage rejects the submission. The review condition might require the agent to identify the source of each receipt. These are suggested criteria, not tests GrantTap silently supplies. Decide how each criterion will be observed: a meaningful automated test, a controlled interface interaction, a file inspection, or a combination. A green test name without an explained scenario is not sufficient acceptance evidence.
Include an honest unresolved category. If a device test cannot run, the attempt can be partially verified instead of being forced into either perfect success or total failure. Distinguish inability to execute a required check from a check that ran and failed. Also decide whether unnecessary changes, disabled assertions, or a weakened permission boundary disqualify the result. An agent could satisfy a visible example by removing the behavior that made the problem hard. Writing the review conditions first reduces the temptation to accept that shortcut because the demonstration looks convincing. The criteria should guide the experiment, while remaining open to corrections when the original expectation itself proves wrong.
Freeze the starting conditions that affect the answer
Give comparable attempts a known starting revision and a defined workspace. Record local modifications that are intentionally part of the task, and keep unrelated work out of the experiment. A separate worktree or disposable copy can help, provided the project supports it and its dependencies are prepared consistently. A folder without Git can still be evaluated, but you need another reliable description of its starting contents. Do not assume that two folders with the same name contain equivalent inputs. Dependencies, generated files, configuration, and access to local services can change the task substantially even when the visible source file looks identical.
Record the client version, selected model or alias, supported effort setting, authentication state, tool access, and relevant operating environment. You do not need to expose account identifiers or secrets. You do need enough information to recognize that a later run used different conditions. GrantTap's Devices view and the selected conversation can supply parts of this inventory; a computer's reported running MCP version is different from a newer version merely installed on disk. If the provider updates during your evaluation, record that change. A result obtained before the update should not be presented as evidence about an identical route afterward.
Separate readiness failures from reasoning failures
Before scoring an agent's solution, check whether it could begin the intended task. A missing CLI, expired sign-in, unavailable workspace, or unsupported tool is a readiness problem. It matters to the practical choice of route, but it does not demonstrate that the model reasoned poorly about the code. Use a small reversible request to establish that the chosen computer receives work and that the provider produces activity in the intended conversation. Inspect the source of any failure instead of repeatedly pressing Send now. The same operation's delivery state is more informative than several new copies of the prompt.
Track readiness separately because it changes what improvement is useful. If the agent never saw the repository, a more detailed coding instruction is unlikely to fix the situation. If it saw the repository and misunderstood the desired invariant, you need a better task description, different context, or another approach. A complete evaluation can report both: the route was difficult to start, and the eventual solution met or missed particular criteria. That is more useful than a single winner label. For GrantTap, do not treat Saved on relay, Received by computer, Agent activity received, and a completed response as interchangeable evidence.
Make human assistance visible
Choose how much help the reviewer may provide. One experiment might allow clarification of product intent but no implementation hints. Another might deliberately test an iterative collaboration, where the person reviews the first patch and gives focused feedback. Both can be valuable, but they answer different questions. Write down meaningful interventions: supplying an overlooked file, explaining a hidden requirement, authorizing a necessary tool, or correcting a mistaken assumption. A mobile approval is part of the interaction, not proof that the agent independently chose a safe path. Preserve the reason for the decision when it affects the interpretation of the run.
Do not measure a heavily coached attempt against an unassisted one without acknowledging the difference. The number of turns is also an incomplete measure: one clarification may prevent an expensive incorrect rewrite, while several superficial messages may add no useful guidance. Observe what the assistance changed. If a provider asks for access that another provider already had, that is a difference in environment or permission workflow. If an agent asks a genuinely necessary product question, stopping to answer can be desirable behavior. The evaluation should reward the kind of collaboration you actually want instead of treating every request for human input as failure.
Collect a compact record of observable events
Keep enough evidence to explain the result without copying an entire private transcript into a report. Record the Task, the relevant Execution, the starting revision, major decisions, changed files, meaningful checks, and the final unresolved items. GrantTap's available transcript sources and request-scoped Files view can help locate these events, but the sources have different coverage. A missing child-agent conversation may mean that the provider did not report it. An installed skill is not proof that it was used. Distinguish an observed call, a reported success, a recorded write, and a reviewer-confirmed result.
Keep timing equally explicit. Time to relay storage, computer acceptance, first provider activity, and a verified patch are different intervals. If a reviewer spent ten minutes answering a question, do not describe the whole interval as model latency. If a build waited for a dependency download, note that environment cost. You can still care about total time to a usable result, because it reflects your actual workflow. The point is to keep its explanation available. Avoid inventing precision when the source only provides an approximate timestamp or an incomplete history. A small accurate timeline is more useful than a detailed fictional one.
Review the patch as software
Read the changes against the acceptance record. Look for the actual mechanism that protects the required behavior, then examine nearby paths that could bypass it. A queue fix might protect the ordinary send button while Retry still creates a new operation. A file repair might update the visible component while leaving an alternate entry point unchanged. Ask whether the change handles the relevant failure, not whether it contains familiar words from your prompt. In Files, confirm which request the displayed changes belong to. A historical patch from an earlier turn should not be credited as a new result merely because it appears in the same conversation.
Run the checks that matter to the change. Use existing tests when they exercise the behavior meaningfully, and add a focused regression where the defect requires one. Avoid tests that simply repeat the implementation's current output without checking its intended invariant. Review any modified tests as carefully as production code. If the agent changes expectations, ask why the old expectation was wrong. A passing suite can coexist with a bad patch when the suite misses the scenario or the change weakens its assertions. Conversely, a pre-existing unrelated failure should be reported separately rather than quietly attributed to the new work.
Interpret repeated attempts without overclaiming
A single attempt is a case study. Repetition can reveal variation, but only if you track how conditions changed. Use a few representative tasks rather than repeatedly running one problem whose answer is already embedded in the conversation. When comparing prompts, begin from equivalent state and decide which history is visible to each attempt. Otherwise later runs may benefit from a discovered fix, a warmed cache, or a previous agent's notes. Those benefits can be legitimate in a continuing workflow, but they should not be mistaken for an independent comparison of starting performance.
Report the denominator. If two of three selected cases met all criteria and one could not start because authentication was missing, say exactly that. Do not turn a tiny convenience sample into a universal percentage of reliability. Keep the cases that failed; excluding inconvenient results makes the method less useful for choosing your next route. You can also distinguish classes of task: a client may handle isolated refactors well while struggling with a device-specific delivery issue. This does not require a single global ranking. It helps you decide which work to assign, what context to supply, and when to involve a person earlier.
Compare cost and resources only where sources support it
Provider-reported tokens, estimated context, elapsed time, process CPU, memory, and billed currency describe different things. A short response can involve substantial hidden work, while a long explanation may be inexpensive relative to a full build. Use the provider's actual billing information for monetary conclusions. GrantTap usage observations can inform your understanding, but an estimated context count is not an invoice. If a process measurement is attributed across several activities, retain that label. Unknown historical resource use should remain unknown instead of borrowing a value from a nearby sample.
Choose the practical budget that matters for the task. You may prefer a route that needs fewer interventions, preserves a quiet computer, or reaches a verified result sooner, even if its answer is not the shortest. Record the tradeoff in ordinary terms. For example, a particular trial required an extra authorization step but produced a smaller reviewable patch. That observation is more actionable than claiming one provider is always cheaper or more efficient. Do not fabricate a comparative chart from illustration geometry. The images in this article explain the evaluation method; their objects are not measured scores or evidence of a winning model.
Run one deliberately boring pilot
Start with a reversible maintenance issue in a project you understand. Prepare the repository, write the acceptance record, check sign-in, and confirm the computer route. Send one clear instruction. Observe the first receipt and the first meaningful provider activity separately. Let the agent inspect what it needs within the granted scope. Answer necessary questions without supplying the entire solution. When it finishes, read the patch, run the relevant checks, and compare the result with the original criteria. If the outcome is unclear, write the missing evidence before asking for another broad round of work.
The pilot should also test your reporting habit. Try handing its record to a colleague who did not watch the conversation. Can that person identify the initial problem, understand the important decisions, find the changed files, and see which checks actually ran? If not, improve the record while the details are fresh. That improvement makes later provider comparisons easier, but it also helps ordinary development. The durable value of a small evaluation is a better way to explain and verify work. It gives future Task transfers a concrete starting point instead of another declaration that everything was handled.
Turn the result into a routing decision
Use the evaluation to choose a next action with a scope and an expiry. You might keep the same provider for a certain class of change, update its tools, narrow the permissions, improve the acceptance scenario, or assign a difficult review to a different route. Preserve the reason and the conditions. Revisit the decision when the client, model catalog, project, or required tools change materially. You are evaluating a working arrangement at a particular time, not conferring a permanent title on a model. A small repeatable record makes that distinction easy to maintain.
The underlying principle is consistent with the importance of environmental feedback described in Anthropic’s published work on effective agents: a system needs evidence from the environment to assess progress. The detailed workshop here is our own proposed practice for local coding work, not an Anthropic benchmark and not a claim that GrantTap supplies a universal evaluator. For the product controls, consult the GrantTap guide on transcript sources, task identity, delivery, and provider readiness. Keep the experiment modest, the evidence readable, and the conclusion specific enough to guide the next piece of work.
A sample record that another person can challenge
Imagine the objective is to preserve an unsent draft when an upload fails. The starting record names a particular source revision, an isolated sample account, the relevant client versions, and the expected draft text. The acceptance record says that a forced storage rejection must leave the text editable, must not create a confirmed delivery receipt, and must allow a later deliberate submission. The reviewer also checks an ordinary successful submission so the repair does not retain every draft forever. These details are examples of what to specify, not observations about a test we have completed for this article.
After the attempt, the record can be equally concrete. It names the files changed, identifies which check exercised rejection, and reports whether the normal path was also checked. If the agent only added a warning label while still clearing the draft, the functional criterion failed even though the screen looks improved. If the implementation is plausible but the available environment cannot force the rejection, the result remains unverified for that condition. A second reviewer can disagree with the criteria, inspect the mechanism, or request the missing scenario without reading every message. This is the level of precision that makes a small evaluation useful beyond a single conversation.