Observability¶

Liter-llm emits OpenTelemetry-compatible tracing spans for every LLM request via two Tower middleware layers: TracingLayer and CostTrackingLayer. Spans follow the OpenTelemetry GenAI semantic conventions.

Feature flags¶

Enable in Cargo.toml:

[dependencies]
liter-llm = { version = "...", features = ["tracing"] }
# Add "otel" to export spans to an OTEL collector:
liter-llm = { version = "...", features = ["tracing", "otel"] }

Span attributes¶

Each request creates a gen_ai span. The following attributes are populated according to the GenAI semantic conventions:

Enabling tracing¶

TracingLayer runs inside the Tower middleware stack. In Rust, build the stack via ManagedClient with a ClientConfig that turns tracing on; the proxy server enables tracing through its [general] tracing = true flag. The current language bindings do not expose Tower-stack toggles directly — to use tracing from a binding, run requests through the proxy.

use liter_llm::{ClientConfigBuilder, ManagedClient};

let config = ClientConfigBuilder::new(std::env::var("OPENAI_API_KEY")?)
    .tracing(true)
    .build();

let client = ManagedClient::new(config, None)?;

For the proxy, set the flag in liter-llm-proxy.toml:

[general]
tracing = true

Exporting spans with OpenTelemetry (Rust)¶

The otel feature re-exports tracing_opentelemetry and opentelemetry at liter_llm::tower::tracing::otel. Wire a subscriber that sends spans to an OTEL collector:

use liter_llm::tower::tracing::otel::{
    tracing_opentelemetry::OpenTelemetryLayer,
    opentelemetry,
};
use opentelemetry_otlp::WithExportConfig;
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};

// Build an OTLP exporter sending to localhost:4317.
let tracer = opentelemetry_otlp::new_pipeline()
    .tracing()
    .with_exporter(
        opentelemetry_otlp::new_exporter()
            .tonic()
            .with_endpoint("http://localhost:4317"),
    )
    .install_batch(opentelemetry_sdk::runtime::Tokio)?;

// Attach the OTEL layer to the tracing subscriber.
tracing_subscriber::registry()
    .with(OpenTelemetryLayer::new(tracer))
    .with(tracing_subscriber::fmt::layer())
    .init();

// Now construct the client with tracing=true.
let config = ClientConfigBuilder::new("sk-...").tracing(true).build();
let client = DefaultClient::new(config, None)?;

Any OTEL-compatible backend accepts these spans: Jaeger, Tempo, Honeycomb, Datadog, etc.

Cost tracking¶

CostTrackingLayer records estimated USD cost as gen_ai.usage.cost on the active tracing span after each successful response. It looks up pricing from the embedded pricing registry (crates/liter-llm/schemas/pricing.json). Models not in the registry produce no attribute.

Enable cost tracking independently of tracing by composing the layer manually in Rust, or by setting cost_tracking = true in the proxy [general] section:

use liter_llm::tower::{cost::CostTrackingLayer, service::LlmService};
use tower::ServiceBuilder;

let inner = LlmService::new(client);
let service = ServiceBuilder::new()
    .layer(CostTrackingLayer)
    .service(inner);

The cost value is also accessible directly on successful response objects via estimated_cost():

let resp = client.chat(req).await?;
if let Some(cost_usd) = resp.estimated_cost() {
    println!("cost: ${:.6}", cost_usd);
}

The pricing registry lives at crates/liter-llm/schemas/pricing.json. Models not in the registry produce no gen_ai.usage.cost attribute.

Proxy trace context forwarding¶

When running behind the proxy server, incoming traceparent and tracestate headers are forwarded to the upstream provider request. The proxy creates a child span for each routed request, which allows distributed traces to span the client, proxy, and provider in a single trace tree.

Enable tracing on the proxy by setting tracing = true in the [general] section of the proxy configuration file. See Proxy Configuration for the full field reference.

Tower layer composition¶

TracingLayer and CostTrackingLayer are standard Tower layers and compose with any Service<LlmRequest>. The recommended order wraps CostTrackingLayer inside TracingLayer so the cost attribute is recorded on the same span:

use liter_llm::tower::{CostTrackingLayer, LlmService, TracingLayer};
use tower::ServiceBuilder;

let service = ServiceBuilder::new()
    .layer(TracingLayer)          // outer: opens the gen_ai span
    .layer(CostTrackingLayer)     // inner: records cost inside the open span
    .service(LlmService::new(client));

Usage events¶

Every request (success or failure) emits a UsageEvent after completion. Use UsageSink implementations to capture token counts, latency, cache state, and cost for billing, dashboards, or analytics.

UsageEvent fields¶

Effective_model¶

Differs from model when routing or fallback rewrites the request. For example:

• Request asks for "openai/gpt-4o", provider echoes "openai/gpt-4o-2024-08-06"
• Request asks for "claude-3-5-sonnet", fallback chain retries via "claude-3-5-opus" and provider echoes the actual model

Set to None for response variants that do not carry a model field: streaming responses, speech, image generation, transcription, rerank, list-models. Also None on error paths where no response body is available.

Cache state¶

CacheState values describe the cache layer outcome:

pub enum CacheState {
    Miss,        // No cache entry; provider was called
    ExactHit,    // Exact-match cache hit; provider not called
    SemanticHit, // Semantic-similarity hit; provider not called
    StaleHit,    // TTL expired but stale entry served
    Bypass,      // Cache lookup skipped (default; streaming, etc.)
}

Task-local cell CACHE_STATE_CELL is written by CacheService and cache_singleflight. HooksService reads it after the inner service resolves and populates the event.

UsageSink trait¶

Implement UsageSink to consume events:

pub trait UsageSink: Send + Sync + 'static {
    fn emit(&self, event: UsageEvent)
        -> impl Future<Output = Result<(), UsageSinkError>> + Send;
}

Implementations should be cheap on the hot path — defer heavy I/O to a background task or channel. Errors from emit are logged but do not propagate to the LLM request caller.

LoggingUsageSink¶

Built-in sink that emits each event as a structured tracing INFO event on the gen_ai.usage target:

use liter_llm::observability::LoggingUsageSink;

let sink = LoggingUsageSink;
// Use sink...

Useful in development and as a smoke-test default. No I/O is performed; always cheap.

MultiUsageSink¶

Fan-out multiple sinks concurrently:

use liter_llm::observability::{LoggingUsageSink, MultiUsageSink};

let multi = MultiUsageSink::from_sinks(vec![
    Arc::new(LoggingUsageSink),
]);

// Or mix heterogeneous types via from_erased:
let multi = MultiUsageSink::from_erased(vec![
    Arc::new(LoggingUsageSink) as Arc<dyn UsageSinkErased>,
    Arc::new(MyCustomSink::new()) as Arc<dyn UsageSinkErased>,
]);

Individual sink errors are logged but do not cause emit to return Err.

Custom sink example¶

use liter_llm::observability::{UsageEvent, UsageSink, UsageSinkError};

struct MetricsSink;

impl UsageSink for MetricsSink {
    async fn emit(&self, event: UsageEvent) -> Result<(), UsageSinkError> {
        // Record to your metrics backend
        println!(
            "request_id={} model={} cost=${:.6} latency_ms={}",
            event.request_id, event.model, event.cost_usd, event.latency_ms
        );
        Ok(())
    }
}

Wiring sinks¶

Tower stack (Rust)¶

Add HooksLayer to your tower stack with a sink:

use liter_llm::tower::hooks::HooksLayer;
use tower::ServiceBuilder;

let sink = Arc::new(LoggingUsageSink);
let service = ServiceBuilder::new()
    .layer(HooksLayer::new().with_usage_sink(sink))
    .service(inner);

HooksLayer wraps the inner service and emits UsageEvent after every completed request.

Proxy (embedded or standalone)¶

Inject a sink via ProxyServer::with_usage_sink:

use std::sync::Arc;
use liter_llm_proxy::ProxyServer;
use liter_llm::observability::LoggingUsageSink;

let sink = Arc::new(LoggingUsageSink);

ProxyServer::new(config)
    .with_usage_sink(sink)
    .serve_with_shutdown(None)
    .await?;

See Embedding the Proxy for a complete example.

Observability integration¶

Export events to your observability backend by implementing UsageSink. For example:

• Send to a message bus (Kafka, NATS, Pub/Sub) for downstream analytics
• Write to a time-series database for dashboarding
• Stream to a billing system for usage-based pricing
• Log to an audit trail for compliance

The UsageEvent is self-contained; no additional context is needed.

Edit this page on GitHub