Go API Reference

Go API Reference v1.9.0-rc.1¶

Functions¶

CreateClient()¶

Create a new LLM client with simple scalar configuration.

This is the primary binding entry-point. All parameters except api_key are optional — omitting them uses the same defaults as ClientConfigBuilder.

Errors:

Returns LiterLlmError if the underlying HTTP client cannot be constructed, or if the resolved provider configuration is invalid.

Signature:

func CreateClient(apiKey string, baseUrl string, timeoutSecs uint64, maxRetries uint32, modelHint string) (DefaultClient, error)

Example:

result, err := CreateClient("value", "value", 42, 42, "value")
if err != nil {
    return err
}

Parameters:

Returns: DefaultClient

Errors: Returns error.

CreateClientFromJson()¶

Create a new LLM client from a JSON string.

The JSON object accepts the same fields as liter-llm.toml (snake_case).

Errors:

Returns LiterLlmError.BadRequest if json is not valid JSON or contains unknown fields.

Signature:

func CreateClientFromJson(json string) (DefaultClient, error)

Example:

result, err := CreateClientFromJson("value")
if err != nil {
    return err
}

Parameters:

Returns: DefaultClient

Errors: Returns error.

EncodeDataUrl()¶

Encode bytes as a base64 data URL: data:<mime>;base64,<b64>.

mime defaults to IMAGE_PNG when nil.

Signature:

func EncodeDataUrl(bytes []byte, mime string) string

Example:

result := EncodeDataUrl([]byte("data"), "value")

Parameters:

Returns: string

DecodeDataUrl()¶

Decode a base64 data URL into DecodedDataUrl.

Returns nil for:

• Non-data URLs (strings that do not start with "data:").
• Malformed prefixes (missing ";base64," marker).
• Invalid base64 payloads.

The returned MIME string is extracted verbatim from the URL prefix — it is not validated or normalised.

Signature:

func DecodeDataUrl(url string) *DecodedDataUrl

Example:

result := DecodeDataUrl("value")

Parameters:

Returns: *DecodedDataUrl

RegisterCustomProvider()¶

Register a custom provider in the global runtime registry.

The provider will be checked before all built-in providers during model detection. If a provider with the same name already exists it is replaced.

Errors:

Returns an error if the config is invalid (empty name, empty base_url, or no model prefixes).

Signature:

func RegisterCustomProvider(config CustomProviderConfig) error

Example:

if err := RegisterCustomProvider(CustomProviderConfig{}); err != nil {
    return err
}

Parameters:

Returns: No return value.

Errors: Returns error.

UnregisterCustomProvider()¶

Remove a previously registered custom provider by name.

Returns true if a provider with the given name was found and removed, false if no such provider existed.

Errors:

Returns an error if the custom-provider registry cannot be updated.

Signature:

func UnregisterCustomProvider(name string) (bool, error)

Example:

result, err := UnregisterCustomProvider("value")
if err != nil {
    return err
}

Parameters:

Returns: bool

Errors: Returns error.

Capabilities()¶

Return the capability flags for a named provider.

Performs an O(n) linear scan over the embedded registry (143 entries). Returns an owned value so bindings can pass capability data without borrowing registry internals.

For unknown provider_name values the function returns an all-false sentinel so callers never need to handle Option.

Signature:

func Capabilities(providerName string) ProviderCapabilities

Example:

result := Capabilities("value")

Parameters:

Returns: ProviderCapabilities

AllProviders()¶

Return all provider configs from the registry.

Useful for tooling, documentation generation, or runtime enumeration. Returns the public ProviderConfig slice (without capability flags). To query capability flags for a specific provider use capabilities.

Signature:

func AllProviders() ([]ProviderConfig, error)

Example:

result, err := AllProviders()
if err != nil {
    return err
}

Returns: []ProviderConfig

Errors: Returns error.

ComplexProviderNames()¶

Return the set of complex provider names.

Complex providers require custom auth/routing logic beyond simple bearer tokens (e.g. AWS Bedrock SigV4, Vertex AI OAuth2).

The returned reference points into the static registry — no allocation.

Signature:

func ComplexProviderNames() ([]string, error)

Example:

result, err := ComplexProviderNames()
if err != nil {
    return err
}

Returns: []string

Errors: Returns error.

CompletionCost()¶

Calculate the estimated cost of a completion given a model name and token counts.

Returns nil if the model is not present in the embedded pricing registry. Returns Some(cost_usd) otherwise, where the value is in US dollars.

When an exact model name match is not found, progressively shorter prefixes are tried by stripping from the last - or . separator. For example, gpt-4-0613 will match gpt-4 if no gpt-4-0613 entry exists.

Signature:

func CompletionCost(model string, promptTokens uint64, completionTokens uint64) *float64

Example:

result := CompletionCost("value", 42, 42)

Parameters:

Returns: *float64

CompletionCostWithCache()¶

Calculate the estimated cost of a completion, accounting for cached (cache-hit) prompt tokens billed at the provider's discounted rate.

cached_tokens is the count of prompt tokens served from the provider's prompt cache. It must be <= prompt_tokens (cached tokens are a subset of the prompt). The non-cached portion is billed at input_cost_per_token and the cached portion at cache_read_input_token_cost when the model has cache pricing; otherwise the entire prompt is billed at the regular input rate.

Returns nil if the model is not present in the embedded pricing registry, mirroring completion_cost.

Signature:

func CompletionCostWithCache(model string, promptTokens uint64, cachedTokens uint64, completionTokens uint64) *float64

Example:

result := CompletionCostWithCache("value", 42, 42, 42)

Parameters:

Returns: *float64

Clear()¶

Remove all guardrails from the global registry.

Primarily useful in tests to reset state between test cases.

Panics:

Panics if the global registry lock is poisoned.

Signature:

func Clear()

Example:

Clear()

Returns: No return value.

CountTokens()¶

Count tokens in a text string using the tokenizer for the given model.

The tokenizer is resolved from the model name prefix (e.g. "gpt-4o" maps to the Xenova/gpt-4o HuggingFace tokenizer). Tokenizers are cached after first load.

Errors:

Returns LiterLlmError.BadRequest if the tokenizer cannot be loaded (e.g. network failure on first use) or if tokenization itself fails.

Signature:

func CountTokens(model string, text string) (int, error)

Example:

result, err := CountTokens("value", "value")
if err != nil {
    return err
}

Parameters:

Returns: int

Errors: Returns error.

CountRequestTokens()¶

Count tokens for a full ChatCompletionRequest.

Sums tokens across all message text contents plus a per-message overhead of ~4 tokens (for role, separators, and formatting metadata). Tool definitions and multimodal content parts (images, audio, documents) are not counted — only textual content contributes to the token total.

Errors:

Returns LiterLlmError.BadRequest if the tokenizer cannot be loaded or if tokenization fails for any message.

Signature:

func CountRequestTokens(model string, req ChatCompletionRequest) (int, error)

Example:

result, err := CountRequestTokens("value", ChatCompletionRequest{})
if err != nil {
    return err
}

Parameters:

Returns: int

Errors: Returns error.

CheckBound()¶

Assert that current_len + incoming does not exceed limit.

Call this before appending incoming bytes to any buffer that must stay below limit. Returns Err(LiterLlmError.Streaming) on overflow and emits a tracing.warn! with context.

Signature:

func CheckBound(context string, currentLen int, incoming int, limit int) error

Example:

if err := CheckBound("value", 42, 42, 42); err != nil {
    return err
}

Parameters:

Returns: No return value.

Errors: Returns error.

EnsureCryptoProvider()¶

Install the ring crypto provider as the rustls process default, idempotently.

rustls 0.23+ removed the implicit default provider. This function installs ring once per process. Subsequent calls are no-ops. Calling it after another rustls crypto provider has already been installed is safe: the Err from install_default() is silently ignored.

Called automatically by every internal reqwest.Client constructor (auth providers, default HTTP client). Bindings and downstream consumers reach those constructors transitively, so no manual init is required.

WASM builds are exempt — the WASM target uses the browser/Node.js fetch API instead of rustls, so no crypto provider is needed.

Windows builds use native-tls (SChannel) via reqwest, so rustls is not present and no crypto provider installation is needed.

Signature:

func EnsureCryptoProvider()

Example:

EnsureCryptoProvider()

Returns: No return value.

EnsureCryptoProvider()¶

No-op on Windows: reqwest uses native-tls (SChannel), so no rustls provider installation is needed. All callers use the same call site regardless of platform.

Signature:

func EnsureCryptoProvider()

Example:

EnsureCryptoProvider()

Returns: No return value.

Types¶

AssistantMessage¶

Assistant's response to a user message.

Methods¶

Text()¶

Return the assistant's textual response, concatenating all Text parts if the content is structured.

Returns nil for Refusal-only or OutputImage-only responses.

Signature:

func (o *AssistantMessage) Text() *string

Example:

result := instance.Text()

Returns: *string

RefusalText()¶

Return the refusal message, if the model declined to respond.

Checks both the top-level refusal field and any Refusal parts inside a structured content.

Signature:

func (o *AssistantMessage) RefusalText() *string

Example:

result := instance.RefusalText()

Returns: *string

OutputImages()¶

Return all AssistantPart.OutputImage parts in the response.

Signature:

func (o *AssistantMessage) OutputImages() []ImageUrl

Example:

result := instance.OutputImages()

Returns: []ImageUrl

OutputAudio()¶

Return all AssistantPart.OutputAudio parts in the response.

Signature:

func (o *AssistantMessage) OutputAudio() []AudioContent

Example:

result := instance.OutputAudio()

Returns: []AudioContent

AudioContent¶

Audio content part for speech-capable models.

AuthConfig¶

Auth configuration block.

BatchListQuery¶

Query parameters for listing batches.

BatchListResponse¶

Response from listing batches.

BatchObject¶

A batch job object.

BatchRequestCounts¶

Request processing counts for a batch.

BudgetConfig¶

Configuration for budget enforcement.

Methods¶

Default()¶

Signature:

func (o *BudgetConfig) Default() BudgetConfig

Example:

result := BudgetConfig.Default()

Returns: BudgetConfig

CacheConfig¶

Configuration for the response cache.

Methods¶

Default()¶

Signature:

func (o *CacheConfig) Default() CacheConfig

Example:

result := CacheConfig.Default()

Returns: CacheConfig

ChatCompletionChunk¶

A streamed chunk of a chat completion response.

ChatCompletionRequest¶

Chat completion request (compatible with OpenAI and similar APIs).

ChatCompletionResponse¶

Chat completion response from the API.

ChatCompletionTool¶

A tool the model can invoke (currently, all tools are functions).

Choice¶

A single completion choice.

ChunkMiddleware¶

A per-chunk transformation in the StreamPipeline.

Each middleware receives a typed chunk and returns Ok(Some(chunk)) to pass it through (optionally modified), Ok(None) to drop the chunk, or Err(e) to propagate a stream error.

The trait is object-safe so multiple middleware implementations can be chained inside StreamPipeline.

Methods¶

Process()¶

Process a single chunk.

• Ok(Some(chunk)) — emit (possibly transformed) chunk.
• Ok(None) — drop this chunk silently.
• Err(e) — propagate as a stream error.

Signature:

func (o *ChunkMiddleware) Process(chunk ChatCompletionChunk) (*ChatCompletionChunk, error)

Example:

result, err := instance.Process(ChatCompletionChunk{})
if err != nil {
    return err
}

Parameters:

Returns: *ChatCompletionChunk

Errors: Returns error.

CreateBatchRequest¶

Request to create a batch job.

CreateFileRequest¶

Request to upload a file.

CreateImageRequest¶

Request to create images from a text prompt.

CreateResponseRequest¶

Request to create a structured response.

CreateSpeechRequest¶

Request to generate speech audio from text.

CreateTranscriptionRequest¶

Request to transcribe audio into text.

CustomProviderConfig¶

Configuration for registering a custom LLM provider at runtime.

DecodedDataUrl¶

Result of decoding a data: URL — MIME type and the decoded byte payload.

Named struct (rather than a tuple) so polyglot bindings can extract decode_data_url with a typed return rather than a sanitized scalar.

DefaultClient¶

Default client implementation backed by reqwest.

Sends requests to 143 LLM providers with automatic provider detection and per-request routing. The provider is resolved at construction time from model_hint (or defaults to OpenAI), but individual requests can override the provider via model name prefix (e.g. "anthropic/claude-3-5-sonnet" routes to Anthropic regardless of construction-time setting).

When the model prefix does not match any known provider, the construction-time provider is used as the fallback. This enables seamless migration between providers by changing only the model name.

The provider is stored behind an Arc so it can be shared cheaply into async closures and streaming tasks. Pre-computed auth headers and extra headers are cached at construction to avoid redundant encoding on every request.

Methods¶

FetchBatchForPolling()¶

Signature:

func (o *DefaultClient) FetchBatchForPolling(batchId string) (BatchObject, error)

Example:

result, err := instance.FetchBatchForPolling("value")
if err != nil {
    return err
}

Parameters:

Returns: BatchObject

Errors: Returns error.

WaitForBatch()¶

Poll a batch until it reaches a terminal status (Completed, Failed, Expired, Cancelled).

Uses exponential backoff with configurable initial interval, maximum interval, and backoff multiplier. Optionally supports a timeout that aborts polling if exceeded.

Errors:

Returns BatchWaitError.Failed if the batch reaches a failure terminal status. Returns BatchWaitError.Timeout if the configured timeout is exceeded. Returns BatchWaitError.Client for underlying client errors.

Signature:

func (o *DefaultClient) WaitForBatch(batchId string, config WaitForBatchConfig) (BatchObject, error)

Example:

result, err := instance.WaitForBatch("value", WaitForBatchConfig{})
if err != nil {
    return err
}

Parameters:

Returns: BatchObject

Errors: Returns error.

DeleteResponse¶

Response from a delete operation.

DeveloperMessage¶

Developer message (system-like message for Claude models).

DocumentContent¶

PDF/document content part for vision-capable models.

EmbeddingObject¶

A single embedding vector.

EmbeddingRequest¶

Embedding request.

EmbeddingResponse¶

Embedding response.

FileListQuery¶

Query parameters for listing files.

FileListResponse¶

Response from listing files.

FileObject¶

An uploaded file object.

FunctionCall¶

Function call details.

FunctionDefinition¶

Function definition exposed to the model.

FunctionMessage¶

Deprecated legacy function-role message body.

HealthChecker¶

Abstraction over a health probe strategy.

Implementors issue a lightweight probe against upstream (typically a provider base URL or named identifier) and report HealthStatus.

Methods¶

Check()¶

Probe upstream and return its current HealthStatus.

The parameter is taken by value (String) so that implementations can move it into the returned future without a clone, making the 'static + Send bound on the future trivially satisfiable.

Signature:

func (o *HealthChecker) Check(upstream string) HealthStatus

Example:

result := instance.Check("value")

Parameters:

Returns: HealthStatus

Image¶

A single generated image, returned as either a URL or base64 data.

ImageUrl¶

An image URL reference with optional detail level for processing.

ImagesResponse¶

Response containing generated images.

IntentPrototype¶

An intent prototype: (intent_name, prototype_embedding, target_model_id).

JsonSchemaFormat¶

JSON Schema specification for constrained output.

ModelObject¶

A model available from the API.

ModelsListResponse¶

Response listing available models.

ModerationCategories¶

Boolean flags for each moderation category.

ModerationCategoryScores¶

Confidence scores for each moderation category.

ModerationRequest¶

Request to classify content for policy violations.

ModerationResponse¶

Response from the moderation endpoint.

ModerationResult¶

A single moderation classification result.

OcrImage¶

An image extracted from an OCR page.

OcrPage¶

A single page of OCR output.

OcrRequest¶

An OCR request.

OcrResponse¶

An OCR response.

PageDimensions¶

Page dimensions in pixels.

PromptTokensDetails¶

Breakdown of tokens used in the prompt portion of a request.

cached_tokens is included in Usage.prompt_tokens — it is not an additional charge on top of the prompt token count. When pricing supports a cache_read_input_token_cost, the cached portion is billed at the discounted rate and the remainder at the regular input rate.

ProviderCapabilities¶

Static capability flags for a provider.

Each flag indicates whether the provider's models generally support that feature. For providers that aggregate many underlying models (e.g. Bedrock, OpenRouter, vLLM) the flags reflect the superset of available model capabilities — a flag being true means at least one model supports the feature, not every model.

All flags default to false so that newly added providers are safe.

Access via the crate-level capabilities function:

ProviderConfig¶

Static configuration for a single provider entry in providers.json.

This struct deliberately does not include capability flags or streaming format, which are accessed via the capabilities function.

RateLimitConfig¶

Configuration for per-model rate limits.

Methods¶

Default()¶

Signature:

func (o *RateLimitConfig) Default() RateLimitConfig

Example:

result := RateLimitConfig.Default()

Returns: RateLimitConfig

RerankRequest¶

Request to rerank documents by relevance to a query.

RerankResponse¶

Response from the rerank endpoint.

RerankResult¶

A single reranked document with its relevance score.

RerankResultDocument¶

The text content of a reranked document, returned when return_documents is true.

ResponseObject¶

Response from a structured response request.

ResponseOutputItem¶

A single output item from the response.

ResponseTool¶

A tool available for the response request.

ResponseUsage¶

Token usage for a response.

SearchRequest¶

A search request.

SearchResponse¶

A search response.

SearchResult¶

An individual search result.

SingleflightResult¶

The value broadcast from a singleflight leader to all followers.

The error value is shared so every follower receives the same upstream failure without cloning the underlying error.

SpecificFunction¶

Name of the specific function to invoke.

SpecificToolChoice¶

Directive to call a specific tool.

StreamChoice¶

A streaming choice with incremental delta.

StreamDelta¶

Incremental delta in a stream chunk.

StreamFunctionCall¶

Partial function call details in a stream.

StreamOptions¶

Options for streaming responses.

StreamToolCall¶

A streaming tool call being built incrementally.

SystemMessage¶

System message guiding model behavior for the entire conversation.

ToolCall¶

A tool call the model wants to execute.

ToolMessage¶

Tool execution result returned to the model.

TranscriptionResponse¶

Response from a transcription request.

TranscriptionSegment¶

A segment of transcribed audio with timing information.

Usage¶

Token-usage accounting returned by the provider on each completion / embedding call.

UserMessage¶

User message in the conversation.

WaitForBatchConfig¶

Configuration for polling a batch until terminal status.

All time values are in seconds as f64 so the struct bridges across FFI boundaries without requiring a Duration shim.

Methods¶

Default()¶

Signature:

func (o *WaitForBatchConfig) Default() WaitForBatchConfig

Example:

result := WaitForBatchConfig.Default()

Returns: WaitForBatchConfig

Enums¶

Message¶

A chat message in a conversation.

UserContent¶

User message content as either plain text or a list of multimodal parts.

ContentPart¶

A single content part in a user message — text, image, document, or audio.

ImageDetail¶

Image detail level controlling token cost and processing.

AssistantContent¶

Content shape for assistant messages.

#[serde(untagged)] means providers returning a plain scalar string for the content field still deserialise correctly into AssistantContent.Text(_). Providers returning an array of typed parts (e.g. after an image-generation or audio-synthesis request) deserialise into AssistantContent.Parts(_).

AssistantPart¶

One part of a structured assistant response.

#[serde(tag = "type", rename_all = "snake_case")] matches OpenAI's parts-spec discriminator ("type": "text", "type": "output_image", …).

ToolType¶

The type discriminator for tool/tool-call objects.

Per the OpenAI spec this is always "function". Using an enum enforces that constraint at the type level and rejects any other value on deserialization.

ToolChoice¶

Tool usage mode or a specific tool to call.

ToolChoiceMode¶

Tool choice mode.

ResponseFormat¶

Wire format for the chat completions response_format field.

Provider mapping¶

• OpenAI (and OpenAI-compatible providers): emitted verbatim as {"type": "json_schema", "json_schema": {...}} per the chat-completions spec.

• Gemini / Vertex AI: translated to generationConfig.responseMimeType = "application/json" and generationConfig.responseSchema = <schema>. The name, description, and strict fields are dropped — Gemini's structured-output API does not consume them.

• Anthropic: no native JSON mode. A system instruction is prepended asking the model to respond with valid JSON. strict is advisory only; callers should still validate the returned JSON if the schema is load-bearing.

StopSequence¶

Stop sequence(s) that cause the model to stop generating.

Modality¶

Output modality requested from the model.

Passed as modalities: ["text", "audio"] (OpenAI) or translated to generationConfig.responseModalities (Gemini / Vertex AI).

FinishReason¶

Why a choice stopped generating tokens.

ReasoningEffort¶

Controls how much reasoning effort the model should use.

EmbeddingFormat¶

The format in which the embedding vectors are returned.

EmbeddingInput¶

Text or texts to embed.

ModerationInput¶

Input to the moderation endpoint — a single string or multiple strings.

RerankDocument¶

A document to be reranked — either a plain string or an object with a text field.

OcrDocument¶

Document input for OCR — either a URL or inline base64 data.

FilePurpose¶

Purpose of an uploaded file.

BatchStatus¶

Status of a batch job.

AuthHeaderFormat¶

How the API key is sent in the HTTP request.

StreamFormat¶

The streaming wire format a provider uses for its response stream.

Most providers use standard Server-Sent Events (SSE). AWS Bedrock uses a proprietary binary EventStream framing.

Deserialized from the streaming_format JSON field via serde.

AuthType¶

Auth scheme used by a provider.

Enforcement¶

How budget limits are enforced.

CacheBackend¶

Storage backend for the response cache.

CircuitState¶

Observable state of a circuit breaker.

HealthStatus¶

The result of a single health probe.

Errors¶

LiterLlmError¶

All errors that can occur when using liter-llm.

Edit this page on GitHub