Skip to content

liter-llm

liter-llm

A universal LLM API client with a Rust core and native bindings for 14 languages. One surface across 143 providers — chat, streaming, embeddings, rerank, image generation, speech, transcription, OCR, search, files, batches, moderation — plus an OpenAI-compatible proxy server and a Model Context Protocol server, both shipped in the same binary.

Quick Start Installation Providers Join our Community

Why liter-llm

  • 143 Providers

OpenAI, Anthropic, Google, Bedrock, Vertex, Azure, Mistral, Cohere, GitHub Copilot, and 135 more — one client, one model-prefix routing scheme.

  • 14 Native Bindings

Rust, Python, TypeScript, Go, Java, Kotlin, C#, Ruby, PHP, Elixir, Dart, Swift, Zig, WebAssembly — plus a C FFI surface for everything else.

  • Full Endpoint Coverage

Chat, streaming, tools, structured outputs, embeddings, rerank, images, speech, transcription, OCR, search, files, batches, moderation — all behind a single LlmClient trait.

  • Proxy & MCP Server

Drop-in OpenAI-compatible proxy with virtual keys, budgets, fallbacks, and observability. Same binary exposes a Model Context Protocol server for AI agents.

  • Cloud-Native Auth

Azure AD, AWS Bedrock SigV4 with STS/IRSA, Vertex AI service-account OAuth2, GitHub Copilot — automatic token caching, refresh, and rotation.

  • Routing & Fallback

Round-robin, weighted, latency-based, cost-based, and ordered-fallback strategies. Per-request override or proxy-level config.

Language Support

Language Package Docs
Rust cargo add liter-llm API Reference
Python pip install liter-llm API Reference
TypeScript / Node npm install @kreuzberg/liter-llm-node API Reference
WebAssembly npm install @kreuzberg/liter-llm-wasm API Reference
Go go get github.com/kreuzberg-dev/liter-llm/packages/go API Reference
Java Maven Central dev.kreuzberg.literllm:liter-llm API Reference
Kotlin Maven com.github.kreuzberg_dev:liter-llm-kotlin API Reference
C# dotnet add package LiterLlm API Reference
Ruby gem install liter_llm API Reference
PHP composer require kreuzberg/liter-llm API Reference
Elixir {:liter_llm, "~> 1.4.0-rc.27"} API Reference
Dart / Flutter dart pub add liter_llm API Reference
Swift Swift Package Manager API Reference
Zig zig fetch --save from GitHub API Reference
C (FFI) Shared library + header API Reference
CLI cargo install liter-llm-cli Proxy Server
Docker ghcr.io/kreuzberg-dev/liter-llm Proxy Server

Quick Example

src/main.rs
use liter_llm::{ChatCompletionRequest, ClientConfigBuilder, DefaultClient, LlmClient, Message};

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ClientConfigBuilder::new(std::env::var("OPENAI_API_KEY")?).build();
    let client = DefaultClient::new(config, None)?;

    let request = ChatCompletionRequest::builder("openai/gpt-4o-mini")
        .add_user_message("Summarize liter-llm in one sentence.")
        .build()?;

    let response = client.chat(request).await?;
    println!("{}", response.choices[0].message.content_text().unwrap_or(""));
    Ok(())
}
main.py
import asyncio
import os
from liter_llm import create_client

async def main():
    client = create_client(api_key=os.environ["OPENAI_API_KEY"])
    response = await client.chat({
        "model": "openai/gpt-4o-mini",
        "messages": [{"role": "user", "content": "Summarize liter-llm in one sentence."}],
    })
    print(response["choices"][0]["message"]["content"])

asyncio.run(main())
index.ts
import { createClient } from "@kreuzberg/liter-llm-node";

const client = createClient(process.env.OPENAI_API_KEY!);

const response = await client.chat({
  model: "openai/gpt-4o-mini",
  messages: [{ role: "user", content: "Summarize liter-llm in one sentence." }],
});

console.log(response.choices[0].message.content);

Part of kreuzberg.dev

Document intelligence — text, tables, and metadata from 91+ file formats with optional OCR.

Managed document-extraction API with SDKs, dashboards, and observability built in.

High-performance web crawling and scraping with always-on HTML→Markdown and headless-Chrome fallback.

Fast, lossless HTML→Markdown engine — Rust core, the same conversion used by Kreuzcrawl.

306 tree-sitter grammars and code-intelligence primitives.

Join the Kreuzberg community for help, roadmap discussion, and announcements.

Explore the Docs

  • Get Started

Install liter-llm for your language, set an API key, and make your first call.

Installation

  • Guides

Chat, embeddings, media, search, fallback routing, authentication, and the proxy/MCP servers.

Chat & Streaming

  • Concepts

Architecture, feature flags, tokenizer model, and cost-estimation pipeline.

Architecture

  • Reference

Per-language API docs, the configuration schema, type catalogue, and error matrix.

References

  • Providers

Browse all 143 supported providers, model prefixes, auth modes, and endpoint coverage.

Provider Registry

  • Proxy & MCP

Run the OpenAI-compatible proxy and the Model Context Protocol server from one binary.

Proxy Server

Getting Help