Deterministic first: why the agent is a tiebreaker, not the engine

There is an obvious way to build a triage tool in 2026: hand the finding and the surrounding code to a language model and ask it whether the finding is real. It demos beautifully. We built the other thing, and this post is the argument for why.

The short version: a model is a poor decision engine and an excellent tiebreaker, and the difference is not about capability. It is about what kind of artifact a verdict has to be before an engineer can act on it.

The pipeline

Three stages, and the ordering is the whole design.

The input is SARIF your scanner already produces — a rule identifier, usually a CWE, and a location. Everything after that is ours.

1. Extract. Take the finding and pull structured facts out of the repository around it: the call path from an entry point to the flagged line, which sanitizers sit on that path and for which sink class, what the framework does with the annotated parameter, which configuration values gate the sink, whether the file is test-only. Every fact carries a file and a line. This stage makes no decisions.
2. Decide by rule. Run rules over those facts. A rule is a stated condition over stated evidence: if the tainted value is constrained to an enum before the sink, close and record the constraint site as the decisive signal. If a path exists with no sanitizer for that sink class, promote and record the path. If the facts satisfy no rule, do not guess — leave the finding open with the specific missing fact named.
3. Break ties. Only for findings where the evidence is extracted, present, and genuinely ambiguous — the evidence supports more than one reading — hand that evidence to an agent. It works from the extracted facts, its output is recorded as a judgment rather than a fact, and it is labelled as such in the queue.

Note where the model is not. It does not decide what the evidence is. It does not get consulted when the rules already resolve the case. And it never converts an absence of evidence into a verdict — that is what the open queue with a stated proof obligation is for.

Same input, same verdict

Determinism here means something narrow and testable: run the same scanner output against the same commit twice and get byte-identical verdicts, with identical reasons. Not similar. Identical.

This is unglamorous and it buys four things that are hard to get any other way.

• A diff means something. If today’s run disagrees with last month’s, the cause is a code change, a scanner change, or a rule change — three things you can look up. When the engine is non-deterministic, a changed verdict tells you nothing, so nobody investigates, so regressions hide in the noise.
• Disputes converge. Someone says a close is wrong. We can reproduce their exact verdict, look at the rule that fired and the facts it fired on, and either fix the rule or explain the evidence. With a sampled model in the loop, the first step of that conversation is often “I can’t reproduce it,” and the dispute dies there.
• A rule change is auditable. Rules are edited, not retrained. When one changes, you can re-run the archive and see exactly which verdicts moved. That is a code review, not a vibe check.
• It is cheap and it is local. Rules over extracted evidence run on a laptop in seconds and cost nothing per finding. An agent call is slow, metered, and — depending on how it is wired — a conversation with your legal team about what left the building. Both of those pressures push the same direction: use the model where it adds something, not as the default path.

The first of those is testable in about a minute, and it is the test we would want run against us.

$ ictx triage scan.sarif --out run-a.json
$ ictx triage scan.sarif --out run-b.json
$ diff <(jq -S . run-a.json) <(jq -S . run-b.json) && echo IDENTICAL
IDENTICAL

$ jq -r '.verdicts[0] | "(.verdict)  (.rule)  (.signal)"' run-a.json
close  constrained-input  value constrained to enum SortSpec at SortSpec.java:22

Reproducibility is not a performance optimisation. It is what makes a verdict a claim someone can argue with.

Why this matters more when a model wrote the code

Our mission page makes the larger version of this argument, and it is the reason the architecture is shaped this way rather than an aesthetic preference.

The trajectory is that most code will be produced by models, reviewed in volumes nobody reads line by line, and changed faster than it is currently changed. Scanners will run over all of it and produce more findings, not fewer.

Now consider what happens if the triage layer is also a model. One system generates the code. Another generates a verdict about it. Neither produces a derivation. The engineer is handed two outputs and no basis for either, and the only remaining way to establish anything is to go read the code — which is the work the whole stack was supposed to shorten. You have not compressed the review loop. You have moved the unreviewed material one layer up and made it harder to see.

Evidence is what breaks the regress, because evidence is checkable without trusting whatever produced it. A call path is either in the code or it is not. A config value is either set or it is not. You can verify those in a minute regardless of what wrote the code or what proposed the verdict. That property does not degrade as more of the stack becomes generated — which is precisely why it should be the load-bearing part.

What the agent is genuinely good at

This is not a position against models, and it would be dishonest to pretend the deterministic path covers everything.

There is a real class of finding where the facts are all extracted and still do not resolve: a sanitizer whose semantics depend on how a caller uses the result, a validation routine that is correct for one sink class and not another, a path that is only reachable under a configuration nobody can find documentation for. A rule that tried to cover these would either be wrong often or be so narrow it never fires.

That is where an agent earns its place — reasoning over evidence someone else established, on cases explicitly marked as judgment calls, with its reasoning recorded next to the facts it used. The output is a recommendation in a queue a person reviews, not a close.

Limitations

• Deterministic does not mean correct. A rule can be confidently, reproducibly wrong, and it will be wrong the same way every time. What determinism buys is that the error is findable and fixable once, not that errors do not happen. The safeguard is the stated decisive signal, which is the thing a reviewer can disagree with.
• Rules are only as good as extraction. Everything upstream in the reachability post applies here — unindexed languages, unresolved dynamic dispatch, framework entry points the extractor does not model. A rule over missing evidence does not fire, which is why “no rule matched” has to route to the open queue rather than to a quiet close.
• The tiebreaker inherits the model’s properties. Where the agent runs, that verdict is not reproducible in the strict sense, and we do not claim otherwise. It is labelled, it is scoped to the ambiguous set, and it does not close findings on its own.
• Coverage is a real cost. A rules-first engine simply declines to decide a great many findings, and that shows up as a larger open queue than a model-first tool would report — on top of a scanner category OWASP already describes as producing high false-positive volume. We think a smaller, honest set of decisions beats a complete set of unverifiable ones. If your constraint is queue size rather than defensibility, that trade goes the other way.

The test we would apply to anyone

Run it twice on the same input and diff the output. Then take one close and ask what fact it rests on and where you would go to check it. Those two questions separate a decision layer from a summarisation layer, and they take about ten minutes.

How it works shows the extraction and rule stages against real output, and the directory lists the runs we have published — including how many of their findings needed the agent at all.