Embedding the Proxy¶

The liter-llm-proxy crate exports ProxyServer as a library, so you can mount the proxy inside a larger Rust binary and control authentication and usage tracking directly.

Quick start¶

Build and serve the proxy with custom authentication:

use std::sync::Arc;
use liter_llm_proxy::{ProxyServer, config::ProxyConfig};
use liter_llm::tenant::InMemoryKeyResolver;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let config = ProxyConfig::from_file("liter-llm-proxy.toml")?;

    // Create a custom key resolver (in-memory for this example)
    let resolver = Arc::new(InMemoryKeyResolver::with_entries(vec![
        (
            "sk-my-api-key".to_string(),
            liter_llm::tenant::ResolvedKey {
                tenant_id: "tenant-123".into(),
                allowed_models: vec!["openai/gpt-4o".to_string()],
                monthly_budget: None,
                currency: None,
                metadata: std::collections::HashMap::new(),
                active: true,
            },
        ),
    ]));

    // Build and serve with the custom resolver
    ProxyServer::new(config)
        .with_key_resolver(resolver)
        .serve_with_shutdown(None)
        .await?;

    Ok(())
}

Builder methods¶

`with_key_resolver`¶

Override the default KeyStore-backed key resolver:

pub fn with_key_resolver(
    mut self,
    resolver: Arc<dyn KeyResolver>
) -> Self

When set, the supplied resolver is used instead of the built-in KeyStore constructed from ProxyConfig.keys. Use this to plug in database-backed or remote resolvers without forking AppState.

`with_usage_sink`¶

Attach a custom usage sink:

pub fn with_usage_sink<S: UsageSink>(
    mut self,
    sink: Arc<S>
) -> Self

When set, HooksLayer is pushed outermost in the Tower middleware stack so every completed request (success or error) emits a UsageEvent to the sink. The sink receives request metadata, token counts, latency, and cache state. Default behaviour (no sink) is unchanged when unset.

Wiring both builders¶

Combine key resolver and usage sink overrides:

use std::sync::Arc;
use liter_llm_proxy::ProxyServer;
use liter_llm::observability::LoggingUsageSink;
use liter_llm::tenant::InMemoryKeyResolver;

let config = ProxyConfig::from_file("liter-llm-proxy.toml")?;

let resolver = Arc::new(InMemoryKeyResolver::new(my_keys));
let sink = Arc::new(LoggingUsageSink);

ProxyServer::new(config)
    .with_key_resolver(resolver)
    .with_usage_sink(sink)
    .serve_with_shutdown(None)
    .await?;

When to use embedding¶

Embed the proxy when you:

Control a larger Rust microservice and want the proxy to run in-process
Need fine-grained control over auth without implementing a separate HTTP service
Want to emit usage events to a custom backend or observability system
Already have credential and storage infrastructure to wire in

For standalone deployments, run liter-llm api as a separate binary.

Default behavior¶

When neither builder is called, the proxy uses:

Built-in KeyStore loaded from ProxyConfig.keys (static or etcd-synced)
No usage sink (requests complete without emission)

Calling the builders is always optional.

Edit this page on GitHub