A verdict you can't check is a verdict you can't trust

The worst hour of my week used to be Friday afternoon, working down a list of findings someone else had already marked as probably fine. Probably. Every one of them carried a number — 0.82, 0.91, “low confidence” — and none of them carried a reason.

So I did the only thing available: I opened the repository and redid the work. Traced the parameter back to its caller. Checked whether the validator ran before the query or after it. Read the deployment config to see whether the flag that guards the sink is ever off in production. Then I closed the finding, and my name went in the ticket.

That name is the whole thing. If a pentest turns up something I closed, nobody is going to ask what the model scored it. They are going to ask what I checked.

A score is not a reason

A confidence score compresses an argument into a scalar and throws the argument away. That is fine when the cost of being wrong is small and the volume is large — spam filtering, ranking, autocomplete. It is a bad trade when a single wrong close is a security incident and a person is accountable for it.

Look at what the score actually denies you. You cannot tell which fact moved it. You cannot tell whether the tool saw the sanitizer or merely failed to see the sink. You cannot tell whether 0.9 on an injection finding means the same thing as 0.9 on a hardcoded credential — usually it does not, because the underlying evidence is a different kind of thing in each case. And you cannot re-run it next month against a changed codebase and diff the answer, because you never had an answer, only a position on a dial.

The question is never “how sure is the tool.” It is “what did the tool see, and would I close this if I had seen the same thing.”

What the fear is actually asking for

The fear of closing a real vulnerability is not irrational and it is not a training problem. It is a correct response to being asked to sign off on a conclusion whose derivation is unavailable. The engineer is being made accountable for a judgment they cannot inspect.

There is a shape of artifact that fixes this, and it predates all of us: a decision record. State the claim, state the evidence the claim rests on, state what would falsify it. Security teams already write these by hand, in ticket comments, one finding at a time — that is what the Friday afternoon was. The problem was never that the format is wrong. The problem is that it does not survive contact with a queue of several hundred.

So the useful question for any triage tool is not how accurate it is. It is: when it closes something, what does it hand me? If the answer is a number, I have to redo the work to trust it, and the tool has saved me nothing. If the answer is the call path, the sanitizer it found, the config value it read, and the file and line for each — I can check it in a minute instead of an hour, and disagree with it precisely.

Concretely, the difference is the difference between these two things arriving in your queue.

# what a score hands you
FINDING  CWE-89  sqli   ReportController.java:52
VERDICT  likely false positive          confidence 0.91

# what a record hands you
FINDING  CWE-89  sqli   ReportController.java:52
VERDICT  close
  decisive signal   value constrained to ^[a-z_]{1,32}$
  evidence          ReportRequest.java:31   @Pattern
  enforced at       ReportController.java:52  @Valid on parameter 0
  assumption        constraint stays on the field; removal reopens this

The second one takes about forty seconds to check and about ten seconds to argue with. The first one takes an afternoon, because the only way to engage with it is to redo it.

The three things a close has to carry

Concretely, before I will let anything close a finding on my behalf, the record has to answer three questions.

1. What was the decisive signal? Not “multiple factors.” The one fact that did the work — this input is constrained by an enum, this path is unreachable from any entry point, this sink is behind a flag that is off in every shipped profile.
2. Where do I go to check it? A file and a line I can open. If I cannot navigate to the evidence in one jump, the record is decorative.
3. What would change the answer? The assumption the close depends on, stated plainly, so that when it stops being true the finding comes back rather than staying quietly closed forever.

A finding that cannot produce all three should not be closed. It should stay open with the missing piece named — “confirm that no caller passes an unvalidated value here” — which is a much smaller task than the one you started with, and one you can hand to someone else.

This is also the difference between suppression and triage. Marking a finding as a false positive in a scanner config records that someone once decided something. It does not record what they decided it on. Six months later, after a refactor, the suppression is still there and the reasoning is gone.

Prior art worth reading

None of this is a new observation. The finding formats themselves are built for it and mostly go unused: the SARIF 2.1.0 specification has structures for code flows, thread flows, and suppression provenance — places to put exactly this kind of reasoning, which most tools emit empty. CWE gives every finding class a stable identity, so “we close this pattern for this reason” is a statement you can hold a tool to. And OWASP’s own page on source code analysis tools lists “high numbers of false positives” under Weaknesses, in the same breath as the strengths. The volume is a known property of the category, not a failure of your particular scanner.

Limitations of this argument

Three places where I think the position above is weaker than it sounds.

• Evidence can be wrong too. A stated call path is checkable, which is not the same as correct. A tool can find a sanitizer and be wrong about what it sanitizes. The gain is that the error is locatable — you can point at the assumption that failed — not that errors stop happening.
• Scores are not useless. For ordering a queue you intend to review by hand anyway, a ranking is genuinely helpful. My objection is narrow: a score is not a basis for closing something without review. Those are different jobs and the industry blurs them.
• Evidence has a cost. Extracting call paths and config context is slower and more brittle than emitting a number, and it fails on code that resists static analysis. A tool that promises evidence has to be equally loud about the findings where it could not produce any.

What we are building

ICTX is the tool I wanted on those Friday afternoons. It reads the SARIF your scanner already produces, attaches evidence from your code — call paths, sanitization, configuration, framework context — and decides with rules over that evidence: close with the decisive signal stated, promote with reasons, or leave open with the one thing left to verify. It runs locally. You can read how it works, and the runs we have published are in the directory, each with its own stated limitations. If you find a close you disagree with, that is the most useful thing you could send us.