{
  "tool": {
    "name": "LiteLLM",
    "repo": "https://github.com/BerriAI/litellm",
    "version_at_verification": "v1.90.0",
    "version_date": "2026-06-27",
    "last_verified": "2026-06-28",
    "dossier_version": "2026-06-28",
    "decay_class": "volatile"
  },
  "scorecard": {
    "claims_sampled": 13,
    "as_labeled": 3,
    "partial": 8,
    "not_as_labeled": 2,
    "not_implemented": 0,
    "open_deltas": 11,
    "verified_closed": 4,
    "closed_partial": 0,
    "closure_trend": "format-translation and release-shippable fixes close within weeks; concurrency, budget-enforcement, and fallback deltas are not closing — fix PRs sit unmerged or were closed while the backing issues were stale-bot closed unfixed"
  },
  "verdict": "The unified-API and provider-breadth claims hold as labeled, but the production-control claims — budgets, rate limits, fallback, and the README's 8ms-P95-at-1k-RPS figure — are contradicted by issue-traced behavior under exactly the concurrent multi-tenant conditions the proxy is marketed for, with the key fixes unmerged and the backing issues stale-bot closed unfixed.",
  "claims": [
    {
      "id": "1",
      "claimed": "Call 100+ LLMs using the OpenAI Input/Output Format (docs.litellm.ai landing page, captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/",
      "observed": "Source-reviewed: the corpus confirms a unified OpenAI-compatible API surface across 100+ providers, and records LiteLLM as the dominant open-source abstraction layer in the gateway category. No corpus evidence contradicts the provider count.",
      "status": "as_labeled",
      "evidence": [
        "https://docs.litellm.ai/docs/providers"
      ]
    },
    {
      "id": "2",
      "claimed": "Drop-in OpenAI compatibility — swap providers without rewriting your code (BerriAI/litellm README, captured 2026-06-11)",
      "claimed_source_url": "https://github.com/BerriAI/litellm",
      "observed": "Source-reviewed: the input/output translation layer is the product's core and works as labeled for mainstream chat-completion use; the corpus's adoption evidence (Stripe, Google, Netflix recorded as adopters) rests on this capability. Cross-provider edge cases are scored separately under the consistent-output row.",
      "status": "as_labeled",
      "evidence": [
        "https://docs.litellm.ai/"
      ]
    },
    {
      "id": "3",
      "verified_at": "2026-07-12",
      "verified_by": "probe-ci",
      "probe": "probes/litellm/claim-3/",
      "claimed": "Router with retry/fallback logic across multiple deployments (e.g. Azure/OpenAI) (docs.litellm.ai landing page and README, captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/",
      "observed": "Source-reviewed: fallback fires as documented for RateLimitError and AuthenticationError, but NotFoundError (404) unconditionally bypasses the entire fallback chain (router.py L5285-86; #21377 stale-bot closed not-planned 2026-06-01, fix PR #21378 still open unmerged as of 2026-06-11); mid-stream fallback injects a default English system prompt over the caller's persona with no disable option (#18229, closed not-planned 2026-03-27); multimodal fallback drops image data (#15803, closed not-planned 2026-01-28); when primary and fallback are both unhealthy the router attempts a nonexistent fallback-for-the-fallback (#17729, closed not-planned 2026-03-17).",
      "status": "partial",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "https://github.com/BerriAI/litellm/issues/21377",
        "https://github.com/BerriAI/litellm/issues/18229",
        "https://github.com/BerriAI/litellm/issues/15803",
        "https://github.com/BerriAI/litellm/issues/17729"
      ]
    },
    {
      "id": "4",
      "verified_at": "2026-07-12",
      "verified_by": "probe-ci",
      "probe": "probes/litellm/claim-4/",
      "claimed": "Track spend & set budgets per project (docs.litellm.ai landing page; 'Budgets, Rate Limits' docs page offers personal, team, team-member, and key-level budgets; captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/docs/proxy/users",
      "observed": "Source-reviewed: a $50 configured budget produced $764.78 actual spend via the AzureOpenAI client — a 15x overshoot with no alert (#12977, closed not-planned 2025-10-31, no fix); user budgets are skipped entirely for virtual keys belonging to teams (#12905, closed not-planned); pass-through routes are not budget-tracked at all (#10750, closed not-planned); the underlying in-memory counters lose 75-93% of increments under 4+ concurrent threads (#20977 — stale-bot closed not-planned 2026-05-20 while the atomic-increment fix PR #20979 remains open unmerged, re-checked 2026-06-11); multi-replica deployments keep independent counters, allowing an N-replica deployment to breach a cap N times.",
      "status": "not_as_labeled",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "https://github.com/BerriAI/litellm/issues/12977",
        "https://github.com/BerriAI/litellm/issues/12905",
        "https://github.com/BerriAI/litellm/issues/10750",
        "https://github.com/BerriAI/litellm/issues/20977",
        "https://github.com/BerriAI/litellm/pull/20979"
      ]
    },
    {
      "id": "5",
      "verified_at": "2026-07-12",
      "verified_by": "probe-ci",
      "probe": "probes/litellm/claim-5/",
      "claimed": "Set tpm limits (tokens per minute), rpm limits (requests per minute), and max parallel requests per key, user, team, or model ('Budgets, Rate Limits' docs page, captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/docs/proxy/users",
      "observed": "Source-reviewed: 5 concurrent requests against a 100 TPM limit consumed 663 tokens — 6.6x the limit — because tokens are deducted after completion, not reserved upfront (#18730, open and reopened as of 2026-06-11; the token-reservation fix PR #23775 was closed unmerged); the in-memory rate-limit cache shows 92.8% counter loss at 16 concurrent workers. Limits bind at low concurrency; under the concurrent multi-tenant load the proxy is marketed for, they do not. UPDATE 2026-06-28: pre-call atomic TPM token reservation shipped in v1.85.0 (PR #27001, merged 2026-05-17) — predating the prior verification, which tracked only the failed PR #23775; the reservation binds TPM under concurrency for Redis deployments. But v1.90.0 added an opt-out (LITELLM_TPM_TOKEN_RESERVATION_ENABLED=false, PR #30211) due to Redis Lua latency, and the in-memory counter race persists for non-Redis deployments (#18730 still open). Limits now bind for the default Redis path; they do not for the in-memory or opt-out paths — hence partial.",
      "status": "partial",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "https://github.com/BerriAI/litellm/issues/18730",
        "https://github.com/BerriAI/litellm/pull/23775",
        "https://github.com/BerriAI/litellm/pull/27001",
        "https://github.com/BerriAI/litellm/pull/30211"
      ]
    },
    {
      "id": "6",
      "claimed": "Guardrails out of the box; per-project customization (logging, guardrails, caching) (README 'Production-ready gateway' bullet and docs.litellm.ai comparison table, captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/",
      "observed": "Source-reviewed: guardrails exist and run in some configurations, but multiple configuration paths had documented non-enforcement windows: GUI/API-defined guardrails never invoked (#15584, vendor-closed completed 2025-10-17), model-level guardrails not taking effect (#18363, vendor-closed completed 2026-01-16), passthrough post_call guardrails skipped (#20270, vendor-closed completed 2026-04-27) — vendor closures not TD-re-verified; intermittent content-filter failures were stale-bot closed without fix (#19637, not-planned 2026-05-01); the custom-code guardrail test endpoint had a sandbox escape (GHSA-wxxx-gvqv-xp7p, High, affects v1.81.8-v1.83.9, patched v1.83.10+). The reported Bedrock guardrail bypass (#22949) was re-verified by TD as a reporter configuration error, not a code defect.",
      "status": "partial",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "https://github.com/BerriAI/litellm/issues/19637",
        "https://github.com/BerriAI/litellm/issues/15584",
        "https://github.com/BerriAI/litellm/issues/18363",
        "https://github.com/BerriAI/litellm/issues/20270",
        "https://github.com/BerriAI/litellm/security/advisories/GHSA-wxxx-gvqv-xp7p"
      ]
    },
    {
      "id": "7",
      "claimed": "Per-project caching (docs.litellm.ai comparison table, captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/",
      "observed": "Source-reviewed: caching works but its savings are unverifiable from the gateway's own telemetry: the cache_hit metric reports 0 even when the upstream provider returns cached_tokens (#6229, closed not-planned); Azure gpt-5.2 prompt caching returned 0/5 hits through the proxy vs 4/5 calling Azure directly (#18219, vendor-closed completed 2025-12-29, not TD-re-verified); default load balancing scatters requests across deployments and defeats provider-side cache warmup (#6784 — a prompt-caching-aware routing strategy was later added, closed completed 2025-02-25).",
      "status": "partial",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "https://github.com/BerriAI/litellm/issues/6229",
        "https://github.com/BerriAI/litellm/issues/18219",
        "https://github.com/BerriAI/litellm/issues/6784"
      ]
    },
    {
      "id": "8",
      "claimed": "8ms P95 latency at 1k RPS (BerriAI/litellm README, captured 2026-06-11)",
      "claimed_source_url": "https://github.com/BerriAI/litellm",
      "observed": "Source-reviewed corpus evidence contradicts the number: a production post-mortem measured P99 latency of 18 seconds at 310 RPS and OOM at 350 RPS sustained over 2 hours — a ~300 RPS Python GIL ceiling that horizontal scaling does not relieve. Caveat stated plainly: the post-mortem's source article has been removed (HTTP 404 since 2026-04-22), so the contradicting measurement is retained at secondary-research backing; no independent benchmark reproducing 8ms P95 at 1k RPS exists in the corpus in either direction.",
      "status": "not_as_labeled",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures"
      ]
    },
    {
      "id": "9",
      "verified_at": "2026-07-12",
      "verified_by": "probe-ci",
      "probe": "probes/litellm/claim-9/",
      "claimed": "Centralized API gateway with authentication & authorization; virtual keys for secure access control (docs.litellm.ai comparison table, captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/",
      "observed": "Source-reviewed: the auth layer exists and functions, but its 2026 security record materially qualifies 'secure': 7 GitHub security advisories published 2026-04-03 through 2026-05-07, including a Critical OIDC authentication bypass (GHSA-jjhc-v7c2-5hh6) and a Critical SQL injection in the API key verification path (GHSA-r75f-5x8p-qvmc) — both patched; a JWT signature-verification bypass report was closed not-planned with no fix merged (#20500, 2026-02-06); PyPI releases 1.82.7-1.82.8 carried credential-stealing malware via a compromised CI scanner (March 2026 supply-chain incident, vendor-disclosed). UPDATE 2026-06-28: an 8th Critical advisory — Host Header Injection (GHSA-4xpc-pv4p-pm3w, CVSS 9.5, disclosed 2026-05-28) — was patched in v1.84.0; current stable v1.90.0 is safe.",
      "status": "partial",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "https://github.com/BerriAI/litellm/security/advisories/GHSA-jjhc-v7c2-5hh6",
        "https://github.com/BerriAI/litellm/security/advisories/GHSA-r75f-5x8p-qvmc",
        "https://github.com/BerriAI/litellm/issues/20500",
        "https://docs.litellm.ai/blog/security-update-march-2026",
        "https://github.com/BerriAI/litellm/security/advisories/GHSA-4xpc-pv4p-pm3w"
      ]
    },
    {
      "id": "10",
      "claimed": "Consistent output - same response format regardless of which provider you use (docs.litellm.ai landing page, captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/",
      "observed": "Source-reviewed: holds for mainstream chat completions; documented divergences at the edges: non-compliant tool-call IDs are not normalized, breaking cross-provider multi-turn tool use (#22317); the thinking.summary field was silently dropped routing Anthropic requests to the OpenAI Responses API (fixed in v1.82.3-stable, TD re-verified present through v1.83.14-stable.patch.3 on 2026-05-17); VertexAI streaming returned the wrong finish_reason for tool calls, looping agents (fixed in v1.82.1+).",
      "status": "partial",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "https://github.com/BerriAI/litellm/issues/22317",
        "https://github.com/BerriAI/litellm/issues/20998"
      ]
    },
    {
      "id": "11",
      "claimed": "Exception handling with OpenAI-compatible errors (docs.litellm.ai comparison table and README, captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/",
      "observed": "Source-reviewed: the exception mapping exists, but asyncio.CancelledError bypasses all three of LiteLLM's exception-handling layers — a DNS cancellation yields a bare 500 with no retry, no fallback, and no log entry (#22100, still open as of 2026-06-11); pass-through paths erase upstream exception types (a provider 401 surfaces as a gateway 500); upstream Retry-After headers are not forwarded to clients under the standard header (#21553, stale-bot closed not-planned 2026-05-29; fix PR #21648 closed unmerged).",
      "status": "partial",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "https://github.com/BerriAI/litellm/issues/22100",
        "https://github.com/BerriAI/litellm/issues/21553",
        "https://github.com/BerriAI/litellm/pull/21648"
      ]
    },
    {
      "id": "12",
      "claimed": "Multi-tenant cost tracking and spend management per project/user (docs.litellm.ai comparison table, captured 2026-06-11)",
      "claimed_source_url": "https://docs.litellm.ai/",
      "observed": "Source-reviewed: cost tracking works in the common path; Azure cost tracking fails on model-name mismatch — Azure returns 'gpt-4' while the deployment is configured as 'azure/gpt-4-1106-preview' and cost calculation uses the response model name (multiple closed issues; the architectural fix request #15851 is marked not-planned); Claude token counts come from LiteLLM's own anthropic_tokenizer.json, whose accuracy against Anthropic's native tokenizer is unverified; multi-replica deployments keep independent spend counters. UPDATE 2026-06-28: a new regression in v1.87.0 pins end_user in SpendLogs to the first request's user for all subsequent requests on a shared virtual key, silently breaking per-user attribution (#31441, open 2026-06-26).",
      "status": "partial",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "https://github.com/BerriAI/litellm/issues/15851",
        "https://docs.litellm.ai/docs/completion/token_usage",
        "https://github.com/BerriAI/litellm/issues/31441"
      ]
    },
    {
      "id": "13",
      "verified_at": "2026-07-12",
      "verified_by": "probe-ci",
      "probe": "probes/litellm/claim-13/",
      "claimed": "Load balancing out of the box; application-level load balancing (README 'Production-ready gateway' bullet and docs.litellm.ai comparison table, captured 2026-06-11)",
      "claimed_source_url": "https://github.com/BerriAI/litellm",
      "observed": "Source-reviewed: router load balancing across deployments is confirmed functional in the corpus; its one documented adverse interaction — scattering requests defeats provider-side cache warmup — was addressed by a prompt-caching-aware routing strategy (#6784, closed completed 2025-02-25). The caching side of that interaction is scored under the caching row.",
      "status": "as_labeled",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/6784"
      ]
    }
  ],
  "deltas": [
    {
      "id": "LL-01",
      "title": "In-memory budget/rate counters lose 75-93% of increments under concurrent load; atomic-increment fix unmerged",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/20977",
        "https://github.com/BerriAI/litellm/pull/20979",
        "Issue stale-bot closed not-planned 2026-05-20 with PR #20979 still open unmerged — re-checked 2026-06-11",
        "Re-verified open 2026-06-28 against v1.90.0"
      ],
      "opened": "2026-03-01",
      "status": "open",
      "falsification_criterion": "Atomic increment fix merged and a concurrent increment test (4+ threads, 50 increments each) yields the expected counter value on a current release, TD re-verified"
    },
    {
      "id": "LL-02",
      "title": "Concurrent requests bypass TPM limits 6.6x — tokens deducted after completion, not reserved; reservation fix closed unmerged",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/18730",
        "https://github.com/BerriAI/litellm/pull/23775",
        "Issue open and reopened; PR #23775 closed without merge — re-checked 2026-06-11",
        "UPDATE 2026-06-28: pre-call token reservation merged in v1.85.0 (PR #27001); #18730 still open and reporter-unconfirmed; v1.90.0 added an opt-out (PR #30211) — falsification criterion partially met, awaiting TD execution",
        "https://github.com/BerriAI/litellm/pull/27001",
        "Re-verified open 2026-06-28 against v1.90.0"
      ],
      "opened": "2026-03-01",
      "status": "open",
      "falsification_criterion": "Concurrent requests against a TPM limit cannot collectively exceed it (pre-call token reservation or equivalent), TD re-verified on a current release"
    },
    {
      "id": "LL-03",
      "title": "Budget enforcement bypassed via AzureOpenAI client (15x overshoot), team virtual keys, and pass-through routes — all closed not-planned without fixes",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/12977",
        "https://github.com/BerriAI/litellm/issues/12905",
        "https://github.com/BerriAI/litellm/issues/10750",
        "Re-verified open 2026-06-28 against v1.90.0"
      ],
      "opened": "2026-03-01",
      "status": "open",
      "falsification_criterion": "A configured budget binds across the AzureOpenAI-client, team-key, and pass-through routes in a TD re-test on a current release"
    },
    {
      "id": "LL-04",
      "title": "NotFoundError (404) unconditionally bypasses the fallback chain; fix PR open unmerged since Feb 2026",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/21377",
        "https://github.com/BerriAI/litellm/pull/21378",
        "Issue stale-bot closed not-planned 2026-06-01; PR #21378 still open — re-checked 2026-06-11",
        "Re-verified open 2026-06-28 against v1.90.0"
      ],
      "opened": "2026-03-01",
      "status": "open",
      "falsification_criterion": "A 404/NotFoundError from one deployment triggers failover to an available alternative deployment, TD re-verified on a current release"
    },
    {
      "id": "LL-05",
      "title": "Mid-stream fallback injects a default English system prompt over the caller's persona; no disable option (closed not-planned)",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/18229",
        "Re-verified open 2026-06-28 against v1.90.0"
      ],
      "opened": "2026-03-01",
      "status": "open",
      "falsification_criterion": "Mid-stream fallback continuation preserves the caller's system prompt, or a configuration option disables the injected prompt; TD re-verified"
    },
    {
      "id": "LL-06",
      "title": "Multimodal fallback OpenAI→Gemini drops image data via in-place message mutation (closed not-planned)",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/15803",
        "Re-verified open 2026-06-28 against v1.90.0"
      ],
      "opened": "2026-03-01",
      "status": "open",
      "falsification_criterion": "OpenAI-to-Gemini fallback delivers image content (non-zero image tokens in usage metrics), TD re-verified"
    },
    {
      "id": "LL-07",
      "title": "README's 8ms P95 at 1k RPS contradicted by measured ~300 RPS GIL ceiling (P99 18s at 310 RPS; OOM at 350 RPS sustained)",
      "evidence": [
        "https://api.theorydelta.com/published/llm-gateway-silent-failures",
        "Contradicting post-mortem article removed (HTTP 404 since 2026-04-22); measurement retained at secondary-research backing",
        "Re-verified open 2026-06-28 against v1.90.0"
      ],
      "opened": "2026-03-01",
      "status": "open",
      "falsification_criterion": "An independent load test sustains 1k RPS with ~8ms P95 on a vendor-documented configuration, or re-measurement fails to reproduce the ~300 RPS ceiling; either result TD-verified"
    },
    {
      "id": "LL-08",
      "title": "Gateway cache metrics unreliable — cache_hit reports 0 while upstream bills cached_tokens; savings unverifiable from gateway telemetry",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/6229",
        "Re-verified open 2026-06-28 against v1.90.0"
      ],
      "opened": "2026-03-01",
      "status": "open",
      "falsification_criterion": "Gateway cache_hit metrics match upstream provider cached-token billing in a TD re-test"
    },
    {
      "id": "LL-09",
      "title": "Upstream Retry-After header not forwarded under the standard header; clients retry immediately, amplifying provider rate-limit pressure",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/21553",
        "https://github.com/BerriAI/litellm/pull/21648",
        "Issue stale-bot closed not-planned 2026-05-29; fix PR #21648 closed unmerged — re-checked 2026-06-11",
        "UPDATE 2026-06-28: fix PR #21648 auto-closed without merge 2026-06-03; v1.90.0 PR #27687 enriches rate-limit error fields but does not forward Retry-After — no fix path",
        "Re-verified open 2026-06-28 against v1.90.0"
      ],
      "opened": "2026-05-17",
      "status": "open",
      "falsification_criterion": "Upstream Retry-After is forwarded to clients under the standard header and respected by router backoff, TD re-verified"
    },
    {
      "id": "LL-C1",
      "title": "Deferred-HTTP providers (Vertex AI, Bedrock, Predibase, Codestral) bypassed streaming fallback entirely",
      "evidence": [
        "https://github.com/BerriAI/litellm/pull/22375"
      ],
      "status": "closed_verified",
      "closed_by": "PR #22375, merged 2026-02-28 — sync streaming fallback plus 429 handling for all streaming paths",
      "reverified_date": "2026-03-28"
    },
    {
      "id": "LL-C2",
      "title": "thinking.summary silently dropped routing Anthropic requests to the OpenAI Responses API",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/20998",
        "https://github.com/BerriAI/litellm/pull/21441"
      ],
      "status": "closed_verified",
      "closed_by": "PRs #21441/#21491 (merged 2026-03-05), shipped in v1.82.3-stable (2026-03-17); TD re-verified present through v1.83.14-stable.patch.3",
      "reverified_date": "2026-05-17"
    },
    {
      "id": "LL-C3",
      "title": "Reported Bedrock guardrail bypass (original input returned instead of blocked output)",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/22949"
      ],
      "status": "closed_verified",
      "closed_by": "TD re-verification of #22949: root cause was reporter configuration (full guardrail ARN corrupting the URL path instead of the short ID) — not a LiteLLM code defect; the finding's row is corrected, not vendor-fixed",
      "reverified_date": "2026-05-17"
    },
    {
      "id": "LL-C4",
      "title": "API keys logged in plaintext in proxy log output (pre-v1.82.4)",
      "evidence": [
        "release-reviewed: suppression fix recorded in v1.82.3-stable.patch.2 / v1.82.4 release notes"
      ],
      "status": "closed_verified",
      "closed_by": "v1.82.4 — plaintext key logging suppressed; not backported, mitigation on older versions is upgrade",
      "reverified_date": "2026-04-10"
    },
    {
      "id": "LL-10",
      "title": "Guardrail fail-open cluster — 8 implementations silently fail open (Lakera category_thresholds disables the blocked verdict, llm_as_a_judge defaults overall_score to 100, OpenAI/Azure moderation scans only the trailing turn, Presidio fails open on analyzer error)",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/30727",
        "https://github.com/BerriAI/litellm/issues/30732"
      ],
      "opened": "2026-06-18",
      "status": "open",
      "falsification_criterion": "Each guardrail blocks the policy-violating input/output it is configured for under analyzer-error and threshold edge cases, TD re-verified on a current release"
    },
    {
      "id": "LL-11",
      "title": "SSE error codes outside the 100-599 range silently discarded — router never fails over or cools down for ZhipuAI/DashScope upstream errors (#31284)",
      "evidence": [
        "https://github.com/BerriAI/litellm/issues/31284"
      ],
      "opened": "2026-06-25",
      "status": "open",
      "falsification_criterion": "A non-standard upstream SSE error code triggers router fallback/cooldown rather than being silently dropped, TD re-verified"
    }
  ],
  "methodology": {
    "sampling_rule_ref": "Theory Delta Method §2 material-claim sampling rule (theorydelta-policy strategy/method.md): a claim is sampled iff it appears on the vendor's own label and would plausibly influence an adoption decision. Label surfaces enumerated and captured 2026-06-11: the docs.litellm.ai landing page, the BerriAI/litellm README, and the 'Budgets, Rate Limits' docs page (docs.litellm.ai/docs/proxy/users). 13 material claims extracted; all sampled (full dossier, 10-claim floor met).",
    "verification_depth": "source-reviewed",
    "verification_statement": "Evidence is drawn from LiteLLM's public issue tracker, pull requests, release notes, GitHub security advisories, the vendor's own security disclosure, and the Theory Delta corpus blocks backing them; the open/closed state of every cited issue and fix PR was re-checked against the GitHub API on 2026-06-11 and again on 2026-06-28 (v1.90.0). Theory Delta has not executed LiteLLM in its own environment for this dossier. Star counts are excluded from assessment as unreliable quantitative metrics.",
    "backing_confidence": "medium",
    "executed_by_td": false,
    "strongest_case_against": "Many backing issues were stale-bot closed ('not planned') rather than triaged, so some behaviors may have been fixed incidentally in LiteLLM's roughly weekly releases since the finding's last full verification (2026-03-22) without an issue or PR recording it; the ~300 RPS ceiling contradicting the 8ms-P95 claim rests on a production post-mortem whose source article has been removed (HTTP 404 since 2026-04-22) and has not been re-measured; several guardrail issues were vendor-closed as completed and TD has not re-verified those fixes; and release velocity (v1.88.1 shipped 2026-06-09) can outpace any source-review window."
  },
  "provenance": {
    "source_block": "theorydelta-blocks/blocks/llm-gateway-routing.md (last_researched 2026-03-22, depth-verified 2026-03-28)",
    "source_blocks": [
      "theorydelta-blocks/blocks/llm-gateway-routing.md",
      "theorydelta-blocks/blocks/litellm.md",
      "theorydelta-blocks/blocks/litellm-gateway.md"
    ],
    "additional_sources": [
      "published/llm-gateway-silent-failures.md (published 2026-03-01, last_verified 2026-03-22; issue states re-checked 2026-06-11)"
    ],
    "derived_date": "2026-06-28"
  }
}