Proxy Server¶

The liter-llm binary ships with a production proxy that speaks the OpenAI REST API on top of any of the 143 supported providers. It terminates Bearer auth, routes by model name, applies the full Tower middleware stack (cache, budget, rate limit, cooldown, health, fallback), and exposes OpenTelemetry spans.

Point any OpenAI SDK at the proxy URL and it works unchanged.

Quick start¶

Start the server against an auto-discovered liter-llm-proxy.toml:

liter-llm api

Start with an explicit config and a master key from the environment:

export LITER_LLM_MASTER_KEY="sk-proxy-master-$(openssl rand -hex 16)"
liter-llm api --config ./liter-llm-proxy.toml

The default bind address is 0.0.0.0:4000. Override with --host and --port.

A minimal config that exposes one OpenAI model:

[general]
master_key = "${LITER_LLM_MASTER_KEY}"

[[models]]
name = "gpt-4o"
provider_model = "openai/gpt-4o"
api_key = "${OPENAI_API_KEY}"

Send a request:

curl http://localhost:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITER_LLM_MASTER_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-4o",
    "messages": [
      {"role": "user", "content": "Summarize the CAP theorem in one sentence."}
    ]
  }'

See Proxy Configuration for every config field.

Command-line flags¶

CLI flags take precedence over config file values, which take precedence over defaults.

Endpoints¶

The proxy exposes 23 API routes with 26 endpoints (verb+path combinations). All /v1/* routes require a Bearer token. Health and OpenAPI endpoints are public.

LLM operations¶

Files¶

Batches¶

Responses¶

Health and discovery¶

Request lifecycle¶

Every authenticated request passes through this pipeline:

flowchart LR
    A[HTTP client] --> B[CORS]
    B --> C[Body limit]
    C --> D[Sensitive header strip]
    D --> E[Bearer auth]
    E --> F{Model allowed?}
    F -->|no| G[403 Forbidden]
    F -->|yes| H[Service pool dispatch]
    H --> I[Tower middleware stack]
    I --> J[Provider client]
    J --> K[Response]

The Tower stack composes: tracing, cost tracking, cache, budget, rate limit, cooldown, health check, fallback. Layer order is set in service_pool.rs when the pool is built.

Deployment¶

Docker¶

A 35 MB statically-linked image is published on every release:

docker run --rm -it \
  -p 4000:4000 \
  -v "$PWD/liter-llm-proxy.toml:/etc/liter-llm-proxy.toml:ro" \
  -e LITER_LLM_MASTER_KEY \
  -e OPENAI_API_KEY \
  ghcr.io/kreuzberg-dev/liter-llm:latest \
  api --config /etc/liter-llm-proxy.toml

Docker Compose¶

services:
  liter-llm:
    image: ghcr.io/kreuzberg-dev/liter-llm:latest
    command: api --config /etc/liter-llm-proxy.toml
    ports:
      - "4000:4000"
    volumes:
      - ./liter-llm-proxy.toml:/etc/liter-llm-proxy.toml:ro
    environment:
      LITER_LLM_MASTER_KEY: ${LITER_LLM_MASTER_KEY}
      OPENAI_API_KEY: ${OPENAI_API_KEY}
      ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY}
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:4000/health/readiness"]
      interval: 10s
      timeout: 3s
      retries: 3

Kubernetes¶

Point the liveness and readiness probes at the dedicated health endpoints:

livenessProbe:
  httpGet:
    path: /health/liveness
    port: 4000
  initialDelaySeconds: 5
  periodSeconds: 10

readinessProbe:
  httpGet:
    path: /health/readiness
    port: 4000
  initialDelaySeconds: 2
  periodSeconds: 5

HTTP behaviour¶

CORS is wide open by default so the proxy works from any browser app during development. Restrict it to a known origin list before shipping to production.

Verify a running instance¶

curl -fsS http://localhost:4000/health/readiness && echo "ready"
curl -fsS http://localhost:4000/v1/models \
  -H "Authorization: Bearer $LITER_LLM_MASTER_KEY"

If the readiness probe returns 200 and /v1/models lists the expected models, the proxy is serving traffic.