We triaged Keycloak's scanner output and published every verdict. Tear it apart.

We ran ICTX over unmodified Opengrep OSS output from Keycloak on Sep 2, 2026 and published every verdict with the evidence behind it. The input was SARIF straight from the scanner — no rule tuning, no suppressions applied first. This post is the write-up. The limitations come first, because you would find them anyway and we would rather you found them here.

What was wrong with this run

• The scanner reported partial success. Not a clean run. Some files were not analysed, which means the finding set we triaged is not the finding set a complete scan would have produced. Everything below is scoped to what the scanner actually emitted.
• 218 parse warnings. Each one is a file or fragment the analysis could not fully read. Parse failures do not create false paths — they hide real ones. A missing file produces the same silence as a safe file, so treat every “no path found” here as conditional on coverage we cannot fully characterise.
• Java and Python are indexed; JavaScript and TypeScript are not indexed. Findings in unindexed languages get no deep evidence, and they are not closed on weak signals — they stay open. If you care specifically about Keycloak’s JavaScript, this run says nothing useful to you.
• The commit is not recorded. We have the run and the scanner output; we do not have the exact revision pinned in the published artifact. That is a reproducibility gap and it is ours, not the scanner’s. Runs after this one pin the commit.
• The corpus skews easy. A large share of these findings are workflow, CI and supply-chain classes, which are among the most tractable to decide deterministically. Read the close rate below with that in mind — it is not a claim about dataflow findings in application code, which are the hard class.

The numbers

454 findings scanned, 427 after output collapse, 439 investigations completed locally in 69 seconds. Agent invocations: 0. Every verdict below came from rules over extracted evidence.

RUN     keycloak/keycloak
SCANNER Opengrep OSS          DATE  Sep 2, 2026
INPUT   454 scanned → 427 after output collapse
WORK    439 investigations / 69s local / 0 agent calls

  closed  (likely noise, decisive evidence)   179   41.9%
  review  (promoted, reasons attached)        114   26.7%
  policy  (CI / supply-chain, owner decides)  103   24.1%
  high    (high-confidence review)             16   3.7%
  routed  (out of scope for this repo)         15   3.5%
                                              ---
                                              427

The five queues, and why they are five

A single “triaged” percentage would hide the only thing that matters here, which is that these findings need five different next actions from five different people.

• Closed — 179 (41.9%). Likely noise, each with a decisive signal stated and a file and line to check it against. Nobody needs to look at these unless they want to audit us, which they should.
• Promoted for review — 114 (26.7%). Evidence supports a real concern. These carry the path or the missing sanitizer as the reason, so a reviewer starts from a claim rather than from a rule name.
• Policy queue — 103 (24.1%). CI and supply-chain findings — workflow permissions, action pinning, build configuration. These are not decided by reading application code. They are decided by whoever owns the pipeline, against a policy, and they get their own queue rather than being averaged into a close rate.
• High-confidence review — 16 (3.7%). The subset of promotions where the evidence is strongest. If you have one afternoon, this is the queue.
• Routed out — 15 (3.5%). Findings that belong to someone else — vendored code, generated artifacts, upstream dependencies. Not closed, because they are not resolved. Routed, because the owner is not this repository.

One worked example: why open, and why not closed

The interesting verdicts are not the closes. Here is a finding that stayed open, in the form the run records it.

FINDING  workflow secret reaches a run step
  .github/workflows/*.yml

EVIDENCE
  source        workflow_dispatch input: inputs.admin-password
  path          input → env → run: step (shell interpolation)
  guard         none found in this workflow file
  callers       n/a — workflow entry point, not a call graph node
  indexed       yaml + java + python;  javascript / typescript not indexed

WHY NOT CLOSED
  no constraint on the input value was found in the indexed set,
  and repository/organization settings that gate who may dispatch
  this workflow are not in the repository.

WHY NOT PROMOTED AS EXPLOITABLE
  exploitability depends on dispatch permissions we cannot read.
  claiming it is exploitable would be a guess about configuration
  outside this checkout.

PROOF OBLIGATION
  confirm callers constrain inputs.admin-password, or pass it via
  a secret in env rather than through shell interpolation.

This is the shape of verdict we think is undersupplied. The finding is not closed, because nothing decisive was found. It is not escalated as confirmed, because the deciding fact — who is permitted to dispatch that workflow — lives in repository settings that are not in the checkout. What it has instead is one specific question, answerable in under a minute by someone with access, which is a dramatically smaller task than the raw finding presented.

A tool that only emits “true positive” or “false positive” has to guess here. We would rather name the gap.

What this run does not show

Beyond the limitations above, three claims we are deliberately not making.

• This is not a security assessment of Keycloak. A close is a statement about a scanner finding, not about the project. Keycloak is a mature codebase with an active security process; we picked it because its scanner output is large, public, and reproducible by strangers. Report anything real to them, not to us.
• 41.9% is not an accuracy claim. It is the share of this queue that had decisive evidence, on this corpus, with the skew noted above. Our disputed-close rate is the number that would matter, and we do not have an independent audit to publish yet. When we do, it goes up with its sample and method.
• 0 agent calls is a property of this run. It shows the deterministic path carried this corpus end to end. It does not promise that every corpus will need none.

Tear it apart

The verdicts and their evidence are published. The run summary is on the home page, the pipeline that produced them is on how it works, and every repository we have published a run for is in the directory, each with its own limitations stated next to its numbers.

The ask is specific: find a close you think is wrong. Not a stylistic objection, not the corpus skew we already conceded above — one finding, one verdict, and the reason our stated signal does not hold. Send it to hello@ictx.sh. We will respond in public, credit you if you are right, and fix the rule. A disputed close is worth more to us than a hundred installs, and the dispute log is going to be the most useful page on this site.

The shareable version

If you want to put this in front of a community that will actually argue with it, here is the short form — limitations included, because a version that drops them is not the same claim.

We triaged Keycloak's SAST output and published every verdict — tear it apart

We built a triage layer that sits on top of existing scanners (reads SARIF;
doesn't scan, and isn't an agent roaming the repo). For each finding it
extracts method-level evidence — call paths, sanitization, config, framework
context — then applies deterministic rules: close with the decisive signal
stated, promote with reasons, or leave open with an explicit proof obligation
("confirm callers constrain inputs.admin-password or pass via env"). An LLM
agent exists as a tiebreaker for genuinely ambiguous evidence.

Results on Keycloak (Opengrep OSS output, Sep 2, 2026):
427 findings after output collapse, 41.9% closed as likely noise on
decisive evidence, 26.7% promoted for review, CI/supply-chain findings
routed to a policy queue. Agent invocations: 0.

Known limitations of this run, before you find them: java and python are indexed; javascript and typescript are not indexed.
The scanner reported partial success with 218 parse warnings, and the
corpus skews toward workflow/CI finding classes, which are among the easier
to close deterministically. The commit is not pinned in this artifact.

If you dispute any close, we'll respond publicly and credit you if you're
right — the dispute log is the roadmap.