Building a Purple-Team CI Gate for Agentic-AI Apps

The scene — a chat assistant that forgets its job

A user opens the chat window of a production finance app and asks: "Who is Donald Trump?" The assistant — built on a current-generation LLM, with a system prompt that explicitly says "you are a personal finance assistant; only answer questions about the user's own money" — answers with a three-paragraph political biography.

No secret was leaked. No bank account was drained. The app still logs the user in fine tomorrow. And yet something has gone wrong that every traditional security scanner misses: the agent abandoned its scope. The same failure mode, in a different shape, leaks generic investment advice ("RRSPs are generally a good idea for anyone over 30") from a product that has explicit legal reasons not to offer investment advice. It leaks the system prompt itself to a user who asks politely enough. It answers questions about other users' transactions when the prompt is crafted to probe for them.

These are not edge cases. They are the first-class bugs of the agentic-AI era, and the class of tool that catches them did not exist eighteen months ago.

Why traditional SAST does not see these bugs

Traditional static-analysis tooling was designed around a worldview that is perfectly sensible — for the 2010s. Code is deterministic. Inputs have types. A function's behaviour is a function of its source. If you want to know whether a route is authenticated, you read the decorators. If you want to know whether a SQL query is safe, you trace the taint path from the HTTP handler to the database call.

None of that applies to the agentic surface. The "route" is a prompt template. The "decorator" is a sentence in English inside a system message. The "taint path" passes through an LLM that was not trained to honour your scope rules. There is no grep for "the model decided to be helpful".

The first three figures are order-of-magnitude, drawn from the current state of the OSS security-scanner ecosystem as of this writing. The last one is specific: there is published research showing that if your answer to "how do we check the model's output" is "ask a second LLM", that second LLM is itself injectable ~73% of the time when the attacker controls its input. So the problem is not "just add a judge model". The problem is adding one that cannot be hijacked.

The other problem — the supply chain you were trusting

While the AI-security community was busy writing prompt-injection papers, a different class of incident kept recurring in plain sight: the CI tooling itself was being weaponised.

A security Action for agentic apps that does not take the supply chain of itself seriously is building on sand. The defensible choice is a Docker container action, not a composite or JavaScript one — every scanner pinned by SHA inside the image, the image itself signed with Sigstore, provenance attested, SBOM attached to every release, and a README that tells consumers to pin the Action by image digest and verify the attestation before first use. That is the bar. Anything less and your security scanner is just another potential compromise vector.

Design — red team, blue team, purple gate

"Red team" and "blue team" are old security-industry terms — the red team attacks, the blue team defends. The purple team is the modern combination of both, run inside the same organisation, with a feedback loop. purplegate takes that model literally and turns it into a single CI gate.

The red team covers both layers: the agentic-AI surface (prompt injection, MCP configs, system-prompt leakage) and the traditional surface every CI already needs (secrets, SAST, deps, IaC, workflow hardening). Every finding is mapped to a framework ID — OWASP LLM Top 10 v2025, OWASP Agentic 2026, MITRE ATLAS v5.4.0 — so downstream SIEM + GRC tools can filter by regulation.

The blue team is the defender side. It walks the same repo looking for positive defense signals — @limiter.limit() decorators, llm_guard imports, guardrails validators, allowlist entries. For each red-team finding, it checks whether a defense is already in place, and lowers severity accordingly. It never raises severity above the red-team baseline — the red team has the ceiling. A Critical secret on a route that already has rate limiting and an allowlist entry drops to Low. A prompt-injection finding in a file that imports llm_guard drops from High to Medium. The original severity is preserved on the finding for audit.

This two-sided design matters because every security tool produces false positives, and false-positive fatigue is what kills adoption. A tool that says "hardcoded secret in tests/fixtures/stripe.py" on every run, even after you explained three months ago that it's the public Stripe test key, will eventually be turned off. A tool that says "we see the rate limiter, we see the allowlist — noting this as Low and moving on" gets used.

The judge — and how we keep it from being hijacked

The most interesting probe is prompt injection. This is where the Apr 2025 research comes in — an LLM judge receiving attacker-controlled input is itself an injection target. A naïve implementation looks like this, and is a bad idea:

The purplegate judge is structured very differently:

• Wrapped delimiters. Attacker input and response are placed inside explicit <attack_payload> / <app_response> tags, with instructions to the judge to treat everything inside as data, not commands.
• Sanitized input. Code fences, base64 blobs, and null bytes are stripped or decoded before the judge ever sees the attack string. This is a belt-and-suspenders layer — the judge is instructed to treat input as data regardless, but we also reduce the attack surface.
• Isolated context. The judge has no tool access, no knowledge of other probes, no system context beyond its versioned rubric. It literally cannot exfiltrate anything it is not directly asked about.
• Structured output. The judge emits a JSON object with four boolean fields. No free-form explanation that can carry hidden instructions downstream.
• Three-rep agreement. Each finding requires ≥ 2/3 of three repeated judge calls (at temperature 0, identical prompt) to agree before it is reported. Flaky judgements are dropped.
• Graceful degradation. Judge errors open — the probe logs and continues. Nightly reruns catch transient failures. The tool never blocks a build because the judge API had a bad day.

What purplegate catches that others don't

Severity gating and the case against "block on everything"

The default gate is fail-on: high. Critical and High findings block the merge; Medium and Low report only. The temptation when you build a security tool is to default to fail-on: medium or even fail-on: low — "take security seriously!" — and the temptation is wrong. The actual effect of over-gating is that teams disable the tool within a sprint, or file seven hundred allowlist entries, or build a parallel shadow pipeline without the gate. High as the default is where the blocker-to-signal ratio is sustainable.

For teams with tighter regulatory profiles — custodial finance, health, critical infra — the input flips to fail-on: medium. For early-stage greenfield, fail-on: critical is the right concession for a sprint while the backlog burns down. That's a configuration choice the tool exposes cleanly; it is not a value judgement baked into the gate.

Allowlist entries are first-class, visible, and dated. Every entry needs a reason of at least twenty characters, an acknowledged_by human, and an expires date within 365 days. Expired entries fail the gate with a specific message; the tool refuses to lose track of a suppression. This is explicit design to keep the allowlist from becoming the place where security goes to die.

Why we open-sourced it

Every Kardoxa Labs product is built on a stack that includes an agentic-AI layer. Every one of them needs this gate. We could have kept it closed and billed for it; plenty of commercial AI-security products exist. We chose open source for three reasons, and they are all self-interested.

First, we want the probes to be auditable. The whole pitch of the tool is "you can verify the scanner itself, not just the scan". That pitch only works if the code is visible.

Second, the threat corpus needs community contributions. Lakera Mosscap and Gandalf together ship ~300k prompt-injection attempts — but the novel bug class next month will not be in either corpus yet. An open-source repo is how new payloads land in the tool within days of a public disclosure instead of months.

Third, it is a trust signal for our paid work. Kardoxa Labs builds voice-first financial AI for SMBs and consumer products. The customers who care about that work the most are also the customers who care whether we take CI security seriously for ourselves. An open-source tool whose design invites the Scorecard, signed-image, attested-SBOM scrutiny is a more persuasive reference than a slide deck — and the discipline of actually getting those badges green is itself the exercise that keeps our production stack honest.

What it caught the first week we used it on our own apps

Running purplegate against our own consumer-app repo (in addition to the chat-scope bug that started this whole thread) surfaced things that no single tool we already had would have caught:

• Eleven internal database tables had RLS enabled but no explicit policy. Functionally the access was correct — only the service role could read them — but only by absence of policy, not by written-down rule. Adding explicit FOR ALL USING (false) deny policies makes the intent self-documenting; future "loosen this" PRs now have to delete a deny policy explicitly, which shows up in code review.
• Three log statements in the bank-aggregator service were dumping unbounded third-party API response bodies into our logs on error. Could have leaked vendor-side data on a bad day. Now bounded to 200 chars and routed through a redacting logger.
• Four CI workflow tokens were persisting longer than they needed to (actions/checkout without persist-credentials: false). Cosmetic until something else in the pipeline turns out to be compromised; then it isn't.
• Two intentionally public routes were correctly flagged as auth-free, with no way to know they were intentional. Now they're in .purplegate/allowlist.yml with written justifications and 365-day expiry — the suppression itself is reviewable.

None of these would have been caught by a traditional SAST scan. None of them are dramatic. They're the kind of finding that compounds quietly until something forces you to look.

What's not done yet

Worth being explicit about: the tool is alpha, not v1.0. The Scorecard sits at roughly 6/10 right now. Three of the remaining 0-scoring checks are structural — solo-maintainer Code-Review, the project being <90 days old, and the OpenSSF Best-Practices badge we haven't yet applied for. The prompt-injection probe's recall on novel attack patterns is somewhere around 80%, which is honest about as good as the field gets today; the threat model documents this explicitly. We're aware of the gaps. The roadmap to v1.0 is in the README.

Where we'd love help

If you're building agentic apps in production, the most useful thing you can do right now is run purplegate against your own repo and tell us what it got wrong. False positives, missed findings, weird edge cases in your stack we didn't think about — every one of those is a GitHub issue we want to see. PRs that add probes, tighten judge rubrics, or extend the allowlist policy all welcome. Novel prompt-injection corpora especially welcome — the corpus is what becomes obsolete fastest, and it's exactly the kind of thing that an open repo curates better than a closed one.