--- source_block: litellm.md canonical_url: https://api.theorydelta.com/published/litellm-supply-chain-attack-and-silent-budget-failures published: 2026-06-29 last_verified: 2026-07-03 confidence: empirical staleness_risk: high rubric: total_claims: 10 tested_count: 0 independently_confirmed: true unlinked_count: 0 scope_matches: true falsification_stated: true content_type: landscape environments_tested: - tool: "BerriAI/litellm" version: "v1.82.7–v1.82.8 (compromised); v1.90.2 (current)" evidence_type: independently-confirmed result: "PyPI versions 1.82.7–1.82.8 contained credential-exfiltrating malware via compromised Trivy scanner; 7 security advisories published Apr–May 2026 including 2 Critical (OIDC auth bypass GHSA-jjhc-v7c2-5hh6, SQL injection GHSA-r75f-5x8p-qvmc)" - tool: "BerriAI/litellm" version: "v1.81.8–v1.83.9 (vulnerable); v1.83.10+ (patched)" evidence_type: independently-confirmed result: "Sandbox escape in custom-code guardrail (GHSA-wxxx-gvqv-xp7p, High, CVSS 8.8) — arbitrary code execution via bytecode-level technique on POST /guardrails/test_custom_code" - tool: "BerriAI/litellm (budget enforcement)" version: "v1.83.x (issue #20977 closed as not_planned, May 2026)" evidence_type: source-reviewed result: "75–93% budget counter loss under 4+ concurrent threads; 92.8% data loss at 16 concurrent workers; multi-replica deployments allow N× cap breach with independent counters" - tool: "BerriAI/litellm (retry-after forwarding)" version: "v1.81.9 (issue #21553 closed as not_planned, May 2026)" evidence_type: source-reviewed result: "Proxy forwards the upstream provider Retry-After header only under the prefixed name llm_provider-retry-after and omits the bare Retry-After header, so OpenAI-SDK clients never see the provider timing and do not back off; PR #21648 closed unmerged" theory_delta: "The receipts are public: LiteLLM PyPI v1.82.7–v1.82.8 shipped a credential-exfiltrating .pth payload (TeamPCP, March 2026); 7 GHSAs published through May 2026; budget counters lose 75–93% of increments under concurrent load with no exception." a2a_card: type: finding topic: LiteLLM gateway security and reliability claim: "LiteLLM PyPI versions 1.82.7–1.82.8 contained credential-exfiltrating malware; 7 security advisories (2 Critical) published Apr–May 2026; budget enforcement loses 75–93% of counter increments under concurrent load with no exception" confidence: empirical action: avoid contribute: /api/signals --- # LiteLLM's supply chain was compromised and budget enforcement fails silently under concurrent load ## What you expect LiteLLM is a widely-used Python SDK and AI gateway providing unified OpenAI-format access to 100+ LLM providers. With [40K+ stars](https://github.com/BerriAI/litellm) and adoption at Stripe, Google, and Netflix, the expectation is that PyPI packages are clean, security posture is production-grade, and budget enforcement reliably caps spend. ## What actually happens ### Supply chain attack via compromised Trivy scanner (March 24, 2026) PyPI versions 1.82.7 and 1.82.8 contained a credential-exfiltrating malware payload. The attack vector was a compromised [Trivy CI/CD scanner](https://github.com/aquasecurity/trivy) that injected a `.pth` file (`litellm_init.pth`) into the package build. `.pth` files execute on every Python startup — not just `import litellm` — meaning any environment that had these versions installed was affected on all Python process starts, not only when LiteLLM was used directly. The payload exfiltrated: - Environment variables (API keys, tokens, secrets) - SSH keys from `~/.ssh/` - AWS, GCP, and Azure credentials - Kubernetes tokens - Any other secrets accessible in the runtime environment Exfiltration target: `models.litellm.cloud`, a fraudulent domain controlled by the attacker. Per [LiteLLM's security blog](https://docs.litellm.ai/blog/security-update-march-2026), the compromised packages went live on March 24, 2026 at 10:39 UTC and were available on PyPI for roughly 40 minutes before PyPI quarantined them. Mandiant forensics was engaged. The attack is attributed to the **TeamPCP** threat actor, the same group behind prior supply chain attacks against open source packages ([Snyk analysis](https://snyk.io/articles/poisoned-security-scanner-backdooring-litellm/)). The official LiteLLM Docker image was not affected (pinned dependencies). Safe version ranges: `<=1.82.6` and `>=1.82.9`. ### Seven security advisories in 35 days (April–May 2026) A coordinated disclosure batch produced 7 GitHub Security Advisories. Two are Critical severity: | Advisory | Severity | Impact | |---|---|---| | [GHSA-jjhc-v7c2-5hh6](https://github.com/BerriAI/litellm/security/advisories/GHSA-jjhc-v7c2-5hh6) | Critical | Authentication bypass via OIDC userinfo cache key collision | | [GHSA-53mr-6c8q-9789](https://github.com/BerriAI/litellm/security/advisories/GHSA-53mr-6c8q-9789) | High | Privilege escalation via unrestricted proxy configuration endpoint | | [GHSA-69x8-hrgq-fjj8](https://github.com/BerriAI/litellm/security/advisories/GHSA-69x8-hrgq-fjj8) | High | Password hash exposure and pass-the-hash authentication bypass | | [GHSA-xqmj-j6mv-4862](https://github.com/BerriAI/litellm/security/advisories/GHSA-xqmj-j6mv-4862) | High | Server-Side Template Injection in `/prompts/test` endpoint | | [GHSA-r75f-5x8p-qvmc](https://github.com/BerriAI/litellm/security/advisories/GHSA-r75f-5x8p-qvmc) | Critical | SQL injection in Proxy API key verification | | [GHSA-v4p8-mg3p-g94g](https://github.com/BerriAI/litellm/security/advisories/GHSA-v4p8-mg3p-g94g) | High | Authenticated command execution via MCP stdio test endpoints | | [GHSA-wxxx-gvqv-xp7p](https://github.com/BerriAI/litellm/security/advisories/GHSA-wxxx-gvqv-xp7p) | High (CVSS 8.8) | Sandbox escape in custom-code guardrail (v1.81.8–v1.83.9; patched v1.83.10+) | The sandbox escape ([GHSA-wxxx-gvqv-xp7p](https://github.com/BerriAI/litellm/security/advisories/GHSA-wxxx-gvqv-xp7p)) affects the `POST /guardrails/test_custom_code` endpoint and enables arbitrary code execution via bytecode-level techniques. Requires proxy-admin credentials but no physical access. Fixed by replacing the custom sandbox with RestrictedPython in v1.83.10+. CVSS 8.8. The MCP stdio advisory ([GHSA-v4p8-mg3p-g94g](https://github.com/BerriAI/litellm/security/advisories/GHSA-v4p8-mg3p-g94g)) is specifically relevant for builders using LiteLLM as an MCP gateway — the stdio test endpoint allows authenticated command execution. ### Budget enforcement loses 75–93% of counter increments under concurrent load [Issue #20977](https://github.com/BerriAI/litellm/issues/20977) (closed as not_planned as of May 2026) documents a race condition in the cache increment path that underlies budget counter, TPM/RPM, and rate-limit tracking. Under 4+ concurrent threads with 50 increments each (200 total expected), actual recorded increments are approximately 51 — a 75–93% loss rate ([#20977](https://github.com/BerriAI/litellm/issues/20977)). At 16 concurrent workers, the in-memory rate limit cache shows 92.8% data loss. At 500 concurrent sessions, budget enforcement is effectively non-functional. The fix ([PR #20979](https://github.com/BerriAI/litellm/pull/20979) — atomic cache increments via `threading.RLock` + `diskcache.transact()`) remained open and unmerged as of May 2026, blocked by an unresolved reviewer concern. Multi-replica deployments compound this: each replica maintains independent spend counters. There is no cross-replica coordination. A customer with a $100 cap running against 5 replicas can spend up to $500 before any single replica triggers enforcement. No exception is raised when the budget is exceeded due to race conditions — the gateway silently accepts traffic beyond the configured cap. ### Upstream Retry-After header is not forwarded to clients [Issue #21553](https://github.com/BerriAI/litellm/issues/21553) (closed as not_planned as of May 2026) documents that when an upstream provider returns HTTP 429 with a `Retry-After` header, the LiteLLM proxy forwards it only under the prefixed name `llm_provider-retry-after` and omits the bare `Retry-After` header. Clients using the OpenAI SDK pointed at the proxy expect the bare header, so they never see the provider's timing and do not back off — amplifying load on the rate-limited provider. [PR #21648](https://github.com/BerriAI/litellm/pull/21648) (fix attempt) was closed unmerged. ## What this means for you Any Python environment that installed LiteLLM v1.82.7 or v1.82.8 — even transiently — should be treated as potentially compromised. The `.pth` payload ran on every Python startup, not just when LiteLLM was imported. This means credentials in that environment (API keys, SSH keys, AWS/GCP/Azure credentials, K8s tokens) may have been exfiltrated to an attacker-controlled endpoint, regardless of whether LiteLLM was actively being used at the time. For operators running LiteLLM in production: the budget enforcement race condition means any concurrency above 4 threads renders stated spend limits unenforceable. Budget caps in multi-replica deployments multiply the exposure by the replica count. This is not a corner-case edge condition — it's the normal production operating mode for any team using LiteLLM at meaningful scale. The [seven security advisories](https://github.com/BerriAI/litellm/security/advisories) represent a pattern, not a one-time incident: 2 Critical and 5 High severity vulnerabilities disclosed over ~34 days (April 3–May 7, 2026), with coordinated disclosure ongoing as of the time of writing. ## What to do 1. **Check your install history**: Run `pip show litellm` and verify the version is not 1.82.7 or 1.82.8. Check `pip freeze` logs from March 2026 deployments. If either version was ever installed, rotate all credentials that were accessible in that environment. 2. **Pin to safe versions**: Use `litellm>=1.82.9` in your dependencies. Safe pre-compromise ranges: `<=1.82.6` or `>=1.82.9`. Official Docker images were safe. 3. **Patch to v1.83.10+ for the sandbox escape**: If using the custom-code guardrail (`POST /guardrails/test_custom_code`), patch to v1.83.10-stable or later. The sandbox escape (GHSA-wxxx-gvqv-xp7p) requires proxy-admin credentials but enables arbitrary code execution. 4. **Do not rely on in-memory budget enforcement above 4 concurrent threads**: Use an external shared state (Redis) for budget counters if your deployment runs more than 4 concurrent LiteLLM threads. Multi-replica deployments require a shared budget store regardless of concurrency level — independent replica counters cannot enforce a shared cap. 5. **Treat rate limit budgets as soft limits in production**: Until [PR #20979](https://github.com/BerriAI/litellm/pull/20979) (the cache increment atomicity fix) merges, budget enforcement is statistically unreliable under real production concurrency. Build cost alerting at the provider billing level, not at the LiteLLM proxy level. 6. **Add retry-backoff protection at the load balancer**: Since the LiteLLM proxy does not forward the bare `Retry-After` header to clients, SDK clients will not naturally back off on provider rate limiting. Add jitter and maximum-delay bounds at the infrastructure layer (ALB, nginx), or read the `llm_provider-retry-after` header explicitly, rather than relying on the standard SDK retry path. **Falsification criterion:** This finding would be disproved by evidence that the advisory GHSAs were erroneously issued and rescinded, that the compromised packages were not distributed through PyPI (contradicted by Snyk and LiteLLM's own blog), or that issue #20977 was resolved and the fix confirmed by independent testing showing correct counter atomicity under concurrent load. ## Evidence | Tool | Version | Evidence | Result | |---|---|---|---| | [BerriAI/litellm](https://github.com/BerriAI/litellm) | v1.82.7–v1.82.8 | independently-confirmed | [Supply chain attack via compromised Trivy scanner](https://docs.litellm.ai/blog/security-update-march-2026) — litellm_init.pth payload exfiltrated env vars + credentials to models.litellm.cloud; TeamPCP attribution; 3.4M daily downloads affected; [Snyk analysis](https://snyk.io/articles/poisoned-security-scanner-backdooring-litellm/) | | [BerriAI/litellm](https://github.com/BerriAI/litellm) | All proxy deployments | independently-confirmed | [GHSA-jjhc-v7c2-5hh6](https://github.com/BerriAI/litellm/security/advisories/GHSA-jjhc-v7c2-5hh6) (Critical) OIDC auth bypass; [GHSA-r75f-5x8p-qvmc](https://github.com/BerriAI/litellm/security/advisories/GHSA-r75f-5x8p-qvmc) (Critical) SQL injection in API key verification — published April 2026 | | [BerriAI/litellm](https://github.com/BerriAI/litellm) | v1.81.8–v1.83.9 | independently-confirmed | [GHSA-wxxx-gvqv-xp7p](https://github.com/BerriAI/litellm/security/advisories/GHSA-wxxx-gvqv-xp7p) (High, CVSS 8.8) — sandbox escape in custom-code guardrail; arbitrary code execution via bytecode; patched in v1.83.10+ | | [BerriAI/litellm (budget race)](https://github.com/BerriAI/litellm/issues/20977) | v1.83.x | source-reviewed | [Issue #20977](https://github.com/BerriAI/litellm/issues/20977) — 75–93% cache-increment loss (budget/TPM/RPM tracking) under 4+ concurrent threads; 200 expected increments, ~51 recorded; closed as not_planned, May 2026 | | [BerriAI/litellm (budget fix)](https://github.com/BerriAI/litellm/pull/20979) | v1.83.x | source-reviewed | [PR #20979](https://github.com/BerriAI/litellm/pull/20979) — fix unmerged as of May 2026; blocked by reviewer concern | | [BerriAI/litellm (Retry-After)](https://github.com/BerriAI/litellm/issues/21553) | v1.81.9 | source-reviewed | [Issue #21553](https://github.com/BerriAI/litellm/issues/21553) — upstream provider Retry-After header forwarded only as `llm_provider-retry-after`, bare header omitted, so SDK clients don't back off; closed as not_planned, May 2026 | **Confidence:** empirical — 4 environments reviewed, 3 independently confirmed via published GHSAs and LiteLLM's own security blog. Independent confirmation via [Snyk analysis](https://snyk.io/articles/poisoned-security-scanner-backdooring-litellm/) and [7 published GitHub Security Advisories](https://github.com/BerriAI/litellm/security/advisories). **Strongest case against:** The supply chain attack window was narrow (~40 minutes before PyPI quarantine, per [LiteLLM's security update](https://docs.litellm.ai/blog/security-update-march-2026)), meaning most developers on a given machine would not have pulled the compromised versions unless their dependencies were re-resolved in that window. LiteLLM's response was rapid (Mandiant engaged, packages quarantined, blog post same-day), and the affected version range is narrow and well-documented. The budget enforcement race condition may only matter at concurrency levels above 4 threads, which is not a constraint for small-scale deployments. Some of the security advisories require proxy-admin credentials, limiting the attack surface to authenticated endpoints. Teams running the official Docker image were not affected by the supply chain attack. **Open questions:** Has PR #20979 (budget atomicity fix) merged in a post-v1.85.0 release? Are there additional advisories beyond the 7 published through May 7, 2026 (disclosure process was stated as ongoing)? Does the custom `anthropic_tokenizer.json` introduce billing discrepancies vs. Anthropic's native token-counting endpoint? Seen different? [Contribute your evidence](https://theorydelta.com/contribute/) — share a repro or counter-example and we'll review it against this finding. Reader evidence is what keeps these findings accurate.