LiteLLM Proxy / Gateway

• Kind: OpenAI-compatible proxy — routes one OpenAI API surface to 100+ backend providers
• Official docs: https://docs.litellm.ai/docs/proxy/configs
• Cyberine adapter: LoopAgentAdapter (ids loop-gateway-a, loop-gateway-b)
• Researched/verified: 2026-06-07 — confidence medium (config structure confirmed; some defaults/aliases inferred).
• DeepSeek capability metadata re-verified 2026-08-24 (T686) — see DeepSeek ids (V4) and the vision gap below.

Base URL & auth

Cyberine runs it via docker-compose up; default GATEWAY_API_KEY=sk-litellm-master-key.

Endpoints (OpenAI-compatible passthrough)

Configuration (litellm.config.yaml)

model_list:
  - model_name: anthropic/claude          # alias the client sends as `model`
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022  # real provider/model upstream
      api_key: os.environ/ANTHROPIC_API_KEY        # NOTE the slash syntax
  - model_name: openai/gpt
    litellm_params:
      model: gpt-4o
      api_key: os.environ/OPENAI_API_KEY
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

• model_name = alias the client sends; litellm_params.model = real upstream id. Independent strings.
• litellm_params: model, api_key, api_base, api_version (Azure), rpm, tpm.
• router_settings: routing_strategy (simple-shuffle/least-busy/usage-based-routing/latency-based-routing), num_retries, fallbacks, model_group_alias.
• litellm_settings: drop_params, num_retries, request_timeout, success_callback, fallbacks, caching.

Models Cyberine routes through it

These two defaults are placeholders that signal "unconfigured" — override them with real upstream ids. Underlying provider docs: openai.md, anthropic.md.

DeepSeek ids (V4) and the vision gap — re-verified live 2026-08-24

LiteLLM's own maintained model_prices_and_context_window.json (the file its Router reads for capability/cost metadata) was fetched raw via curl https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json and inspected as JSON (not a summarized fetch, so no field got dropped or mangled). It has these DeepSeek V4 entries, all under "litellm_provider": "deepseek":

(There are also openrouter/deepseek/deepseek-v4-pro[-0813] entries under "litellm_provider": "openrouter" — those describe LiteLLM proxying to OpenRouter as a backend, a different configuration from the direct deepseek-* rows above; not what Cyberine's own litellm.config.yaml example wires up today.)

Known limitation of this path: searching the same file for a DeepSeek entry with vision in its key returns zero results — there is no deepseek-v4-flash-vision-exp row at all, dated or otherwise. Every DeepSeek row that does exist is explicitly marked supports_vision: false. So even though the underlying model accepts images (confirmed on openrouter.md and deepseek.md), LiteLLM's own capability metadata gives a router no correct way to learn that any DeepSeek id on this path can take an image — an image dispatch routed through the LiteLLM gateway path has no metadata-backed vision route today. Do not configure litellm.config.yaml to promise vision support for a DeepSeek model_name alias until this file gains a vision-capable DeepSeek row upstream.

Notable features

• OpenAI-compatible REST — any OpenAI SDK works by pointing base_url at the proxy.
• Streaming/tools passed through to backends.
• os.environ/VAR syntax in any config value (resolved at startup).
• Virtual keys (scoped sub-keys, budgets), router strategies, fallback chains.
• Spend tracking + budgets (needs PostgreSQL), Redis-backed routing, logging callbacks (langfuse/prometheus/otel/datadog), secret-manager and semantic-cache integrations.

Gotchas

• Endpoint path is /chat/completions with no /v1 prefix internally, but /v1/... is also accepted. Set SDK base_url=http://localhost:41400 (the Cyberine host port; not .../v1) to avoid double-prefix — Cyberine's GATEWAY_BASE_URL includes /v1, which the proxy tolerates.
• Config var syntax is os.environ/VAR (forward slash), not ${VAR} / $VAR.
• master_key must start with sk-.
• CLI default port is 4000; the server package's PORT default is 8000 — don't confuse them.
• drop_params: true silently strips unsupported params before forwarding — convenient but can mask errors.
• Virtual keys are in-memory (lost on restart) without DATABASE_URL.

Pricing

Open source (MIT) — no proxy cost; you pay the underlying providers. Hosted/enterprise: https://www.litellm.ai/pricing

Cyberine integration notes

• loop-gateway-a / loop-gateway-b (read-only Q&A/compare adapters) register when ANY GATEWAY_* or LOOP_MODEL_* env var is explicitly set — including to the documented defaults, since a live local gateway at http://localhost:41400/v1 with the anthropic/claude + openai/gpt aliases IS a real gateway (fixed 2026-06-07; the old heuristic read defaults as "unconfigured" and skipped a working gateway). This is the only path that needs Docker; the DeepSeek/Kimi/OpenRouter loop workers bypass it entirely.
• WRITING fleet: loop-claude-build (routes LOOP_MODEL_A) and loop-gpt-build (routes LOOP_MODEL_B) are sandbox-pinned LoopAgentAdapter implementers backed by the gateway — add them to CYBERINE_BUILD_FLEET to race Claude/GPT against the key-only DeepSeek/Kimi workers on a real implement task. They need the gateway up (docker compose up -d).
• The gateway is what gives Cyberine access to OpenAI and Anthropic models without writing provider-specific code (provider independence via baseURL swap).

Sources

• https://docs.litellm.ai/docs/proxy/configs
• https://docs.litellm.ai/docs/proxy/quick_start
• https://docs.litellm.ai/docs/proxy/virtual_keys
• https://docs.litellm.ai/docs/proxy/config_settings
• https://docs.litellm.ai/docs/providers/anthropic
• https://docs.litellm.ai/docs/providers/openai
• https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json (fetched raw 2026-08-24)