Configuration Reference

Configuration Reference¶

This page documents all configuration types and their defaults across all languages.

SystemMessage¶

System message guiding model behavior for the entire conversation.

Field	Type	Default	Description
`content`	`UserContent`	`UserContent.TEXT`	Instructions or context that apply throughout the conversation. Accepts either a plain text string or an array of content parts, mirroring `UserContent` so that `Message.system_with_parts` works.
`name`	`str \\| None`	`None`	Optional name for the system message source.

UserMessage¶

User message in the conversation.

Field	Type	Default	Description
`content`	`UserContent`	`UserContent.TEXT`	Message content as plain text or array of content parts (text, images, documents, audio).
`name`	`str \\| None`	`None`	Optional name for the user.

ImageUrl¶

An image URL reference with optional detail level for processing.

Field	Type	Default	Description
`url`	`str`	—	URL of the image (data URI or HTTP/HTTPS URL).
`detail`	`ImageDetail \\| None`	`None`	Detail level: low (512x512), high (2x2 tiles), or auto (model-selected).

DocumentContent¶

PDF/document content part for vision-capable models.

Field	Type	Default	Description
`data`	`str`	—	Base64-encoded document data or URL.
`media_type`	`str`	—	MIME type (e.g., "application/pdf", "text/csv").

AudioContent¶

Audio content part for speech-capable models.

Field	Type	Default	Description
`data`	`str`	—	Base64-encoded audio data.
`format`	`str`	—	Audio format (e.g., "wav", "mp3", "ogg").

AssistantMessage¶

Assistant's response to a user message.

Field	Type	Default	Description
`content`	`AssistantContent \\| None`	`None`	The assistant's response: plain text, structured parts, or absent. `None` is valid when the model replies with tool calls only.
`name`	`str \\| None`	`None`	Optional name for the assistant.
`tool_calls`	`list\[ToolCall\] \\| None`	`\[\]`	Tool calls the model wants to execute, if any.
`refusal`	`str \\| None`	`None`	Refusal reason, if the model declined to respond per safety policies.
`function_call`	`FunctionCall \\| None`	`None`	Deprecated legacy function_call field; retained for API compatibility.

ToolMessage¶

Tool execution result returned to the model.

Field	Type	Default	Description
`content`	`str`	—	Result of the tool execution.
`tool_call_id`	`str`	—	ID of the tool call this result responds to.
`name`	`str \\| None`	`None`	Optional tool/function name.

DeveloperMessage¶

Developer message (system-like message for Claude models).

Field	Type	Default	Description
`content`	`str`	—	Developer-specific instructions or context.
`name`	`str \\| None`	`None`	Optional name for the developer message source.

FunctionMessage¶

Deprecated legacy function-role message body.

Field	Type	Default	Description
`content`	`str`	—	The extracted text content
`name`	`str`	—	The name

SpecificToolChoice¶

Directive to call a specific tool.

Field	Type	Default	Description
`choice_type`	`ToolType`	`ToolType.FUNCTION`	Tool type (always "function").
`function`	`SpecificFunction`	—	The specific function to invoke.

SpecificFunction¶

Name of the specific function to invoke.

Field	Type	Default	Description
`name`	`str`	—	Function name.

JsonSchemaFormat¶

JSON Schema specification for constrained output.

Field	Type	Default	Description
`name`	`str`	—	Name of the schema (must be unique in the request).
`description`	`str \\| None`	`None`	Description of what the schema represents.
`schema`	`dict\[str, Any\]`	—	JSON Schema object defining the output structure.
`strict`	`bool \\| None`	`None`	If true, enforce strict schema validation.

Usage¶

Token-usage accounting returned by the provider on each completion / embedding call.

Field	Type	Default	Description
`prompt_tokens`	`int`	—	Prompt tokens used. Defaults to 0 when absent (some providers omit this).
`completion_tokens`	`int`	—	Completion tokens used. Defaults to 0 when absent (e.g. embedding responses).
`total_tokens`	`int`	—	Total tokens used. Defaults to 0 when absent (some providers omit this).
`prompt_tokens_details`	`PromptTokensDetails \\| None`	`None`	Breakdown of tokens used in the prompt, including cached tokens served at the provider's discounted cache-read rate. Absent when the provider does not return prompt-token details.

PromptTokensDetails¶

Breakdown of tokens used in the prompt portion of a request.

cached_tokens is included in Usage.prompt_tokens — it is not an additional charge on top of the prompt token count. When pricing supports a cache_read_input_token_cost, the cached portion is billed at the discounted rate and the remainder at the regular input rate.

Field	Type	Default	Description
`cached_tokens`	`int`	—	Cached tokens present in the prompt. Defaults to 0 when absent.
`audio_tokens`	`int`	—	Audio input tokens present in the prompt. Defaults to 0 when absent.

ChatCompletionRequest¶

Chat completion request (compatible with OpenAI and similar APIs).

Field	Type	Default	Description
`model`	`str`	—	Model ID (e.g., `"gpt-4o-mini"`, `"claude-3-5-sonnet"`).
`messages`	`list\[Message\]`	`\[\]`	Conversation history from oldest to newest.
`temperature`	`float \\| None`	`None`	Sampling temperature in `\[0.0, 2.0\]`. Higher increases randomness. Defaults to 1.0.
`top_p`	`float \\| None`	`None`	Nucleus sampling parameter in `\[0.0, 1.0\]`. Lower is more focused.
`n`	`int \\| None`	`None`	Number of chat completions to generate. Defaults to 1.
`stream`	`bool \\| None`	`None`	Whether to stream the response. Managed by the client layer — do not set directly.
`stop`	`StopSequence \\| None`	`None`	Stop sequence(s) that halt token generation.
`max_tokens`	`int \\| None`	`None`	Max output tokens. Different from max_completion_tokens in some providers.
`presence_penalty`	`float \\| None`	`None`	Presence penalty in `\[-2.0, 2.0\]`. Positive discourages repeated topics.
`frequency_penalty`	`float \\| None`	`None`	Frequency penalty in `\[-2.0, 2.0\]`. Positive discourages repeated tokens.
`logit_bias`	`dict\[str, float\] \\| None`	`{}`	Token bias map. Uses `BTreeMap` (sorted keys) for deterministic serialization order — important when hashing or signing requests.
`user`	`str \\| None`	`None`	User identifier for request tracking and abuse detection.
`tools`	`list\[ChatCompletionTool\] \\| None`	`\[\]`	Tools the model can invoke.
`tool_choice`	`ToolChoice \\| None`	`None`	Tool usage mode (auto, required, none, or specific tool).
`parallel_tool_calls`	`bool \\| None`	`None`	Whether the model can call multiple tools in parallel. Defaults to true.
`response_format`	`ResponseFormat \\| None`	`None`	Output format constraint (text, JSON, JSON schema).
`stream_options`	`StreamOptions \\| None`	`None`	Streaming options (e.g., include_usage).
`seed`	`int \\| None`	`None`	Random seed for reproducible outputs. Provider support varies.
`reasoning_effort`	`ReasoningEffort \\| None`	`None`	Reasoning effort level (low, medium, high) for extended-thinking models.
`modalities`	`list\[Modality\] \\| None`	`\[\]`	Output modalities to request from the model. For OpenAI audio models, pass `\["text", "audio"\]`. Vertex AI / Gemini translates these to `generationConfig.responseModalities` (uppercase).
`extra_body`	`dict\[str, Any\] \\| None`	`None`	Provider-specific extra parameters merged into the request body. Use for guardrails, safety settings, grounding config, etc.

StreamOptions¶

Options for streaming responses.

Field	Type	Default	Description
`include_usage`	`bool \\| None`	`None`	If true, include token usage in the final stream chunk.

ChatCompletionResponse¶

Chat completion response from the API.

Field	Type	Default	Description
`id`	`str`	—	Unique identifier for this response.
`object`	`str`	—	Always `"chat.completion"` from OpenAI-compatible APIs. Stored as a plain `String` so non-standard provider values do not break deserialization.
`created`	`int`	—	Unix timestamp of response creation.
`model`	`str`	—	Model used to generate the response.
`choices`	`list\[Choice\]`	`\[\]`	List of completion choices.
`usage`	`Usage \\| None`	`None`	Token usage statistics.
`system_fingerprint`	`str \\| None`	`None`	Fingerprint of the system configuration (OpenAI-specific).
`service_tier`	`str \\| None`	`None`	Service tier used (OpenAI-specific).

Choice¶

A single completion choice.

Field	Type	Default	Description
`index`	`int`	—	Index of this choice in the choices array.
`message`	`AssistantMessage`	—	The assistant's message response.
`finish_reason`	`FinishReason \\| None`	`None`	Why the model stopped generating (stop, length, tool_calls, content_filter, etc.).

ChatCompletionChunk¶

A streamed chunk of a chat completion response.

Field	Type	Default	Description
`id`	`str`	—	Unique identifier for this stream.
`object`	`str`	—	Always `"chat.completion.chunk"` from OpenAI-compatible APIs. Stored as a plain `String` so non-standard provider values do not fail parsing.
`created`	`int`	—	Unix timestamp of chunk creation.
`model`	`str`	—	Model used to generate the chunk.
`choices`	`list\[StreamChoice\]`	`\[\]`	Streaming choices (delta updates).
`usage`	`Usage \\| None`	`None`	Token usage (typically only in the final chunk).
`system_fingerprint`	`str \\| None`	`None`	Fingerprint of the system configuration (OpenAI-specific).
`service_tier`	`str \\| None`	`None`	Service tier used (OpenAI-specific).

StreamChoice¶

A streaming choice with incremental delta.

Field	Type	Default	Description
`index`	`int`	—	Index of this choice in the choices array.
`delta`	`StreamDelta`	—	Incremental update to the message (content, tool calls, etc.).
`finish_reason`	`FinishReason \\| None`	`None`	Why the stream ended (present only in final chunk).

StreamDelta¶

Incremental delta in a stream chunk.

Field	Type	Default	Description
`role`	`str \\| None`	`None`	Role (typically present only in the first chunk).
`content`	`str \\| None`	`None`	Partial content chunk (e.g., a few words of the response).
`tool_calls`	`list\[StreamToolCall\] \\| None`	`\[\]`	Partial tool calls being streamed.
`function_call`	`StreamFunctionCall \\| None`	`None`	Deprecated legacy function_call delta; retained for API compatibility.
`refusal`	`str \\| None`	`None`	Partial refusal message.

StreamToolCall¶

A streaming tool call being built incrementally.

Field	Type	Default	Description
`index`	`int`	—	Index of this tool call in the tool_calls array.
`id`	`str \\| None`	`None`	Tool call ID (typically in the first chunk for this call).
`call_type`	`ToolType \\| None`	`None`	Tool type (typically "function").
`function`	`StreamFunctionCall \\| None`	`None`	Partial function name and arguments.

StreamFunctionCall¶

Partial function call details in a stream.

Field	Type	Default	Description
`name`	`str \\| None`	`None`	Function name (typically in the first chunk).
`arguments`	`str \\| None`	`None`	Partial JSON arguments chunk.

EmbeddingRequest¶

Embedding request.

Field	Type	Default	Description
`model`	`str`	—	Model ID (e.g., `"text-embedding-3-small"`).
`input`	`EmbeddingInput`	`EmbeddingInput.SINGLE`	Text or texts to embed.
`encoding_format`	`EmbeddingFormat \\| None`	`None`	Output format: float (native) or base64.
`dimensions`	`int \\| None`	`None`	Requested embedding dimensions (if supported by the model).
`user`	`str \\| None`	`None`	User identifier for request tracking.

CreateImageRequest¶

Request to create images from a text prompt.

Field	Type	Default	Description
`prompt`	`str`	—	Text description of the image to generate.
`model`	`str \\| None`	`None`	Model ID (e.g., `"dall-e-3"`). Optional; API may use default if unset.
`n`	`int \\| None`	`None`	Number of images to generate. Defaults to 1.
`size`	`str \\| None`	`None`	Image size (e.g., `"1024x1024"`, `"1792x1024"`).
`quality`	`str \\| None`	`None`	Image quality: `"standard"` or `"hd"`.
`style`	`str \\| None`	`None`	Style: `"natural"` or `"vivid"` (DALL-E 3 only).
`response_format`	`str \\| None`	`None`	Response format: `"url"` or `"b64_json"`.
`user`	`str \\| None`	`None`	User identifier for request tracking.

ImagesResponse¶

Response containing generated images.

Field	Type	Default	Description
`created`	`int`	—	Unix timestamp of image creation.
`data`	`list\[Image\]`	`\[\]`	List of generated images.

Image¶

A single generated image, returned as either a URL or base64 data.

Field	Type	Default	Description
`url`	`str \\| None`	`None`	Image URL (if response_format was "url").
`b64_json`	`str \\| None`	`None`	Base64-encoded image data (if response_format was "b64_json").
`revised_prompt`	`str \\| None`	`None`	The final prompt used to generate the image (DALL-E 3).

DecodedDataUrl¶

Result of decoding a data: URL — MIME type and the decoded byte payload.

Named struct (rather than a tuple) so polyglot bindings can extract decode_data_url with a typed return rather than a sanitized scalar.

Field	Type	Default	Description
`mime`	`str`	—	MIME type extracted from the URL prefix (verbatim, not normalised).
`data`	`bytes`	—	Decoded base64 payload.

CreateSpeechRequest¶

Request to generate speech audio from text.

Field	Type	Default	Description
`model`	`str`	—	Model ID (e.g., `"tts-1"`, `"tts-1-hd"`).
`input`	`str`	—	Text to synthesize into speech.
`voice`	`str`	—	Voice name (e.g., `"alloy"`, `"echo"`, `"fable"`, `"onyx"`, `"nova"`, `"shimmer"`).
`response_format`	`str \\| None`	`None`	Audio format (e.g., `"mp3"`, `"opus"`, `"aac"`, `"flac"`, `"wav"`, `"pcm"`).
`speed`	`float \\| None`	`None`	Playback speed in `\[0.25, 4.0\]`. Defaults to 1.0.

CreateTranscriptionRequest¶

Request to transcribe audio into text.

Field	Type	Default	Description
`model`	`str`	—	Model ID (e.g., `"whisper-1"`).
`file`	`str`	—	Base64-encoded audio file data.
`language`	`str \\| None`	`None`	Language ISO-639-1 code (e.g., `"en"`, `"fr"`, `"de"`). Optional; model auto-detects.
`prompt`	`str \\| None`	`None`	Optional text to guide the model (improves accuracy for domain-specific terms).
`response_format`	`str \\| None`	`None`	Output format (e.g., `"json"`, `"text"`, `"vtt"`, `"srt"`, `"verbose_json"`).
`temperature`	`float \\| None`	`None`	Sampling temperature in `\[0.0, 1.0\]`. Higher increases variability. Defaults to 0.

TranscriptionResponse¶

Response from a transcription request.

Field	Type	Default	Description
`text`	`str`	—	The transcribed text.
`language`	`str \\| None`	`None`	Detected language (ISO-639-1 code).
`duration`	`float \\| None`	`None`	Total audio duration in seconds.
`segments`	`list\[TranscriptionSegment\] \\| None`	`\[\]`	Detailed segment-level transcription (if response_format is "verbose_json").

TranscriptionSegment¶

A segment of transcribed audio with timing information.

Field	Type	Default	Description
`id`	`int`	—	Segment index (0-based).
`start`	`float`	—	Start time in seconds.
`end`	`float`	—	End time in seconds.
`text`	`str`	—	Transcribed text for this segment.

ModerationRequest¶

Request to classify content for policy violations.

Field	Type	Default	Description
`input`	`ModerationInput`	`ModerationInput.SINGLE`	Text or texts to check.
`model`	`str \\| None`	`None`	Model ID (e.g., `"text-moderation-latest"`). Optional; API uses default if unset.

ModerationCategories¶

Boolean flags for each moderation category.

Field	Type	Default	Description
`sexual`	`bool`	—	Sexual content.
`hate`	`bool`	—	Hate speech.
`harassment`	`bool`	—	Harassment.
`self_harm`	`bool`	—	Self-harm content.
`sexual_minors`	`bool`	—	Sexual content involving minors.
`hate_threatening`	`bool`	—	Hate speech that threatens violence.
`violence_graphic`	`bool`	—	Graphic violence.
`self_harm_intent`	`bool`	—	Intent to self-harm.
`self_harm_instructions`	`bool`	—	Instructions for self-harm.
`harassment_threatening`	`bool`	—	Harassment that threatens violence.
`violence`	`bool`	—	Non-graphic violence.

ModerationCategoryScores¶

Confidence scores for each moderation category.

Field	Type	Default	Description
`sexual`	`float`	—	Sexual content score.
`hate`	`float`	—	Hate speech score.
`harassment`	`float`	—	Harassment score.
`self_harm`	`float`	—	Self-harm content score.
`sexual_minors`	`float`	—	Sexual content involving minors score.
`hate_threatening`	`float`	—	Hate speech that threatens violence score.
`violence_graphic`	`float`	—	Graphic violence score.
`self_harm_intent`	`float`	—	Intent to self-harm score.
`self_harm_instructions`	`float`	—	Instructions for self-harm score.
`harassment_threatening`	`float`	—	Harassment that threatens violence score.
`violence`	`float`	—	Non-graphic violence score.

RerankRequest¶

Request to rerank documents by relevance to a query.

Field	Type	Default	Description
`model`	`str`	—	Model ID (e.g., `"cohere/rerank-english-v3.0"`).
`query`	`str`	—	The search query.
`documents`	`list\[RerankDocument\]`	`\[\]`	Documents to rerank.
`top_n`	`int \\| None`	`None`	Return only the top N results. Optional.
`return_documents`	`bool \\| None`	`None`	Include the document content in results. Defaults to false.

SearchRequest¶

A search request.

Field	Type	Default	Description
`model`	`str`	—	The model/provider to use (e.g. `"brave/web-search"`, `"tavily/search"`).
`query`	`str`	—	The search query string.
`max_results`	`int \\| None`	`None`	Maximum number of results to return.
`search_domain_filter`	`list\[str\] \\| None`	`\[\]`	Domain filter — restrict results to specific domains.
`country`	`str \\| None`	`None`	Country code for localized results (ISO 3166-1 alpha-2, e.g., `"US"`, `"FR"`).

OcrRequest¶

An OCR request.

Field	Type	Default	Description
`model`	`str`	—	The model/provider to use (e.g. `"mistral/mistral-ocr-latest"`).
`document`	`OcrDocument`	`OcrDocument.URL`	The document to process (URL or base64).
`pages`	`list\[int\] \\| None`	`\[\]`	Specific pages to process (1-indexed). `None` means all pages.
`include_image_base64`	`bool \\| None`	`None`	Whether to include base64-encoded images of each processed page.

ModelsListResponse¶

Response listing available models.

Field	Type	Default	Description
`object`	`str`	—	Always `"list"` from OpenAI-compatible APIs. Stored as a plain `String` so non-standard provider values do not break deserialization.
`data`	`list\[ModelObject\]`	`\[\]`	List of available models.

ModelObject¶

A model available from the API.

Field	Type	Default	Description
`id`	`str`	—	Model ID (e.g., `"gpt-4o"`, `"claude-3-5-sonnet"`).
`object`	`str`	—	Always `"model"` from OpenAI-compatible APIs. Stored as a plain `String` so non-standard provider values do not break deserialization.
`created`	`int`	—	Unix timestamp of model creation (or release date).
`owned_by`	`str`	—	Organization or entity that owns the model.

CreateFileRequest¶

Request to upload a file.

Field	Type	Default	Description
`file`	`str`	—	Base64-encoded file data.
`purpose`	`FilePurpose`	`FilePurpose.ASSISTANTS`	Purpose for the file.
`filename`	`str \\| None`	`None`	Optional filename to associate with the upload.

FileObject¶

An uploaded file object.

Field	Type	Default	Description
`id`	`str`	—	Unique file ID.
`object`	`str`	—	Object type (always `"file"`).
`bytes`	`int`	—	File size in bytes.
`created_at`	`int`	—	Unix timestamp of file creation.
`filename`	`str`	—	Filename.
`purpose`	`str`	—	File purpose.
`status`	`str \\| None`	`None`	Processing status (e.g., `"uploaded"`, `"processed"`).

FileListResponse¶

Response from listing files.

Field	Type	Default	Description
`object`	`str`	—	Object type (always `"list"`).
`data`	`list\[FileObject\]`	`\[\]`	List of file objects.
`has_more`	`bool \\| None`	`None`	Whether more results are available.

FileListQuery¶

Query parameters for listing files.

Field	Type	Default	Description
`purpose`	`str \\| None`	`None`	Filter by file purpose (e.g., `"batch"`, `"fine-tune"`).
`limit`	`int \\| None`	`None`	Maximum number of results to return. Defaults to 20.
`after`	`str \\| None`	`None`	Pagination cursor: return results after this file ID.

DeleteResponse¶

Response from a delete operation.

Field	Type	Default	Description
`id`	`str`	—	ID of the deleted resource.
`object`	`str`	—	Object type.
`deleted`	`bool`	—	Confirmation that the resource was deleted.

CreateBatchRequest¶

Request to create a batch job.

Field	Type	Default	Description
`input_file_id`	`str`	—	ID of the uploaded input file (JSONL format).
`endpoint`	`str`	—	API endpoint (e.g., `"/v1/chat/completions"`).
`completion_window`	`str`	—	Completion window (e.g., `"24h"`).
`metadata`	`dict\[str, Any\] \\| None`	`None`	Optional metadata to attach to the batch.

BatchObject¶

A batch job object.

Field	Type	Default	Description
`id`	`str`	—	Unique batch ID.
`object`	`str`	—	Object type (always `"batch"`).
`endpoint`	`str`	—	API endpoint (e.g., `"/v1/chat/completions"`).
`input_file_id`	`str`	—	ID of the input file.
`completion_window`	`str`	—	Completion window (e.g., `"24h"`).
`status`	`BatchStatus`	`BatchStatus.VALIDATING`	Current job status.
`output_file_id`	`str \\| None`	`None`	ID of the output file (present when completed).
`error_file_id`	`str \\| None`	`None`	ID of the error file (present if some requests failed).
`created_at`	`int`	—	Unix timestamp of batch creation.
`completed_at`	`int \\| None`	`None`	Unix timestamp of completion (if completed).
`failed_at`	`int \\| None`	`None`	Unix timestamp of failure (if failed).
`expired_at`	`int \\| None`	`None`	Unix timestamp of expiration (if expired).
`request_counts`	`BatchRequestCounts \\| None`	`None`	Request processing counts.
`metadata`	`dict\[str, Any\] \\| None`	`None`	Metadata attached to the batch.

BatchRequestCounts¶

Request processing counts for a batch.

Field	Type	Default	Description
`total`	`int`	—	Total requests in the batch.
`completed`	`int`	—	Completed requests.
`failed`	`int`	—	Failed requests.

BatchListResponse¶

Response from listing batches.

Field	Type	Default	Description
`object`	`str`	—	Object type (always `"list"`).
`data`	`list\[BatchObject\]`	`\[\]`	List of batch objects.
`has_more`	`bool \\| None`	`None`	Whether more results are available.
`first_id`	`str \\| None`	`None`	First batch ID in the result set (for pagination).
`last_id`	`str \\| None`	`None`	Last batch ID in the result set (for pagination).

BatchListQuery¶

Query parameters for listing batches.

Field	Type	Default	Description
`limit`	`int \\| None`	`None`	Maximum number of results to return. Defaults to 20.
`after`	`str \\| None`	`None`	Pagination cursor: return results after this batch ID.

CreateResponseRequest¶

Request to create a structured response.

Field	Type	Default	Description
`model`	`str`	—	Model ID.
`input`	`dict\[str, Any\]`	—	Input data to process (e.g., a document to extract from).
`instructions`	`str \\| None`	`None`	Instructions for processing the input.
`tools`	`list\[ResponseTool\] \\| None`	`\[\]`	Available tools the model can use.
`temperature`	`float \\| None`	`None`	Sampling temperature in `\[0.0, 2.0\]`. Defaults to 1.0.
`max_output_tokens`	`int \\| None`	`None`	Maximum output tokens.
`metadata`	`dict\[str, Any\] \\| None`	`None`	Optional metadata.

ResponseTool¶

A tool available for the response request.

Field	Type	Default	Description
`tool_type`	`str`	—	Tool type (e.g., "extractor", "search").
`config`	`dict\[str, Any\]`	—	Tool configuration (flattened into the object).

ResponseObject¶

Response from a structured response request.

Field	Type	Default	Description
`id`	`str`	—	Unique response ID.
`object`	`str`	—	Object type (e.g., `"response"`).
`created_at`	`int`	—	Unix timestamp of response creation.
`model`	`str`	—	Model used to generate the response.
`status`	`str`	—	Status (e.g., `"succeeded"`, `"failed"`).
`output`	`list\[ResponseOutputItem\]`	`\[\]`	Output items from the response.
`usage`	`ResponseUsage \\| None`	`None`	Token usage.
`error`	`dict\[str, Any\] \\| None`	`None`	Error details (if status is "failed").

ResponseOutputItem¶

A single output item from the response.

Field	Type	Default	Description
`item_type`	`str`	—	Output type (e.g., `"text"`, `"object"`, `"error"`).
`content`	`dict\[str, Any\]`	—	Output content (flattened into the object).

ResponseUsage¶

Token usage for a response.

Field	Type	Default	Description
`input_tokens`	`int`	—	Input tokens used.
`output_tokens`	`int`	—	Output tokens used.
`total_tokens`	`int`	—	Total tokens used.

WaitForBatchConfig¶

Configuration for polling a batch until terminal status.

All time values are in seconds as f64 so the struct bridges across FFI boundaries without requiring a Duration shim.

Field	Type	Default	Description
`initial_interval_secs`	`float`	`5`	Initial interval between polls, in seconds.
`max_interval_secs`	`float`	`60`	Maximum interval between polls (backoff plateau), in seconds.
`backoff_multiplier`	`float`	`1.5`	Exponential backoff multiplier (e.g., 1.5 increases delay by 50% each poll).
`timeout_secs`	`float \\| None`	`None`	Optional timeout in seconds — polling fails if this duration is exceeded.

CustomProviderConfig¶

Configuration for registering a custom LLM provider at runtime.

Field	Type	Default	Description
`name`	`str`	—	Unique name for this provider (e.g., "my-provider").
`base_url`	`str`	—	Base URL for the provider's API (e.g., `<https://api.my-provider.com/v1>`).
`auth_header`	`AuthHeaderFormat`	—	Authentication header format.
`model_prefixes`	`list\[str\]`	—	Model name prefixes that route to this provider (e.g., `\["my-"\]`).

ProviderCapabilities¶

Static capability flags for a provider.

Each flag indicates whether the provider's models generally support that feature. For providers that aggregate many underlying models (e.g. Bedrock, OpenRouter, vLLM) the flags reflect the superset of available model capabilities — a flag being True means at least one model supports the feature, not every model.

All flags default to False so that newly added providers are safe.

Access via the crate-level capabilities function:

Field	Type	Default	Description
`vision`	`bool`	—	The provider accepts image input in chat messages.
`reasoning`	`bool`	—	The provider supports extended-thinking / reasoning tokens.
`structured_output`	`bool`	—	The provider supports JSON-mode or `response_format` structured output.
`function_calling`	`bool`	—	The provider supports tool / function calling.
`audio_in`	`bool`	—	The provider accepts audio as input.
`audio_out`	`bool`	—	The provider can generate audio / TTS output.
`video_in`	`bool`	—	The provider accepts video as input.

ProviderConfig¶

Static configuration for a single provider entry in providers.json.

This struct deliberately does not include capability flags or streaming format, which are accessed via the capabilities function.

Field	Type	Default	Description
`name`	`str`	—	Provider identifier (matches the entry key in providers.json).
`display_name`	`str \\| None`	`None`	Human-readable provider name shown in UIs.
`base_url`	`str \\| None`	`None`	Base URL used as the default for this provider's HTTP client.
`auth`	`AuthConfig \\| None`	`None`	Authentication scheme metadata (auth type + env var holding the key).
`endpoints`	`list\[str\] \\| None`	`None`	Supported endpoint kinds (e.g. `chat`, `embeddings`).
`model_prefixes`	`list\[str\] \\| None`	`None`	Model-name prefixes claimed by this provider (e.g. `\["gpt-", "o1-"\]`).
`param_mappings`	`dict\[str, str\] \\| None`	`None`	Parameter key renaming for this provider. Each entry maps an OpenAI-spec field name (e.g. `"max_completion_tokens"`) to the name this provider expects (e.g. `"max_tokens"`). Applied automatically by `ConfigDrivenProvider.transform_request`.

AuthConfig¶

Auth configuration block.

Field	Type	Default	Description
`auth_type`	`AuthType`	—	Auth scheme classification.
`env_var`	`str \\| None`	`None`	Name of the environment variable that holds the API key (e.g. `"OPENAI_API_KEY"`). Holds the variable name, never the secret value.

BudgetConfig¶

Configuration for budget enforcement.

Field	Type	Default	Description
`global_limit`	`float \\| None`	`None`	Maximum total spend across all models, in USD. `None` means unlimited.
`model_limits`	`dict\[str, float\]`	`{}`	Per-model spending limits in USD. Models not listed here are only constrained by `global_limit`.
`enforcement`	`Enforcement`	`Enforcement.HARD`	Whether to reject requests or merely warn when a limit is exceeded.

CacheConfig¶

Configuration for the response cache.

Field	Type	Default	Description
`max_entries`	`int`	`256`	Maximum number of cached entries.
`ttl`	`float`	`300000ms`	Time-to-live for each cached entry.
`backend`	`CacheBackend`	`CacheBackend.MEMORY`	Storage backend to use.

RateLimitConfig¶

Configuration for per-model rate limits.

Field	Type	Default	Description
`rpm`	`int \\| None`	`None`	Maximum requests per window. `None` means unlimited.
`tpm`	`int \\| None`	`None`	Maximum tokens per window. `None` means unlimited.
`window`	`float`	`60000ms`	Fixed window duration (defaults to 60 s).

Enums¶

AssistantContent¶

Content shape for assistant messages.

#[serde(untagged)] means providers returning a plain scalar string for the content field still deserialise correctly into AssistantContent.Text(_). Providers returning an array of typed parts (e.g. after an image-generation or audio-synthesis request) deserialise into AssistantContent.Parts(_).

Variant	Description
`Text`	Plain text response (the common case for text-only models). — Fields: `_0`: `String`
`Parts`	Structured parts — text, refusals, output images, output audio. — Fields: `_0`: `Vec<AssistantPart>`

AuthHeaderFormat¶

How the API key is sent in the HTTP request.

Variant	Description
`Bearer`	Bearer token: `Authorization: Bearer <key>`
`ApiKey`	Custom header: e.g., `X-Api-Key: <key>` — Fields: `_0`: `String`
`None`	No authentication required.

AuthType¶

Auth scheme used by a provider.

Variant	Wire value	Description
`Bearer`	`bearer`	Standard `Authorization: Bearer <key>` header.
`ApiKey`	`api-key`	`x-api-key: <key>` header (also handles `"header"` and `"x-api-key"` aliases).
`None`	`none`	No authentication header required.
`Unknown`	`unknown`	Unrecognised auth scheme — falls back to bearer.

BatchStatus¶

Status of a batch job.

Variant	Wire value	Description
`Validating`	`validating`	Validating the input file.
`Failed`	`failed`	Job failed.
`InProgress`	`in_progress`	Job is running.
`Finalizing`	`finalizing`	Finalizing results.
`Completed`	`completed`	Job completed successfully.
`Expired`	`expired`	Job expired before completion.
`Cancelling`	`cancelling`	Job is being cancelled.
`Cancelled`	`cancelled`	Job has been cancelled.

CacheBackend¶

Storage backend for the response cache.

Variant	Wire value	Description
`Memory`	`memory`	In-memory LRU cache (default). No external dependencies.
`OpenDal`	`open_dal`	OpenDAL-backed storage. Supports 40+ backends (S3, Redis, GCS, local FS, etc.). — Fields: `scheme`: `String`, `config`: `HashMap<String, String>`

EmbeddingFormat¶

The format in which the embedding vectors are returned.

Variant	Wire value	Description
`Float`	`float`	32-bit floating-point numbers (default).
`Base64`	`base64`	Base64-encoded string representation of the floats.

EmbeddingInput¶

Text or texts to embed.

Variant	Description
`Single`	Single text string. — Fields: `_0`: `String`
`Multiple`	Multiple text strings (batch embedding). — Fields: `_0`: `Vec<String>`

Enforcement¶

How budget limits are enforced.

Variant	Description
`Hard`	Reject requests that would exceed the budget with `LiterLlmError.BudgetExceeded`.
`Soft`	Allow requests through but emit a `tracing.warn!` when the budget is exceeded.

FilePurpose¶

Purpose of an uploaded file.

Variant	Wire value	Description
`Assistants`	`assistants`	File for use with Assistants API.
`Batch`	`batch`	File for batch processing.
`FineTune`	`fine-tune`	File for fine-tuning.
`Vision`	`vision`	File for vision/image tasks.

FinishReason¶

Why a choice stopped generating tokens.

Variant	Wire value	Description
`Stop`	`stop`	Stop
`Length`	`length`	Length
`ToolCalls`	`tool_calls`	Tool calls
`ContentFilter`	`content_filter`	Content filter
`FunctionCall`	`function_call`	Deprecated legacy finish reason; retained for API compatibility.
`Other`	`other`	Catch-all for unknown finish reasons returned by non-OpenAI providers. Note: this intentionally does not carry the original string (e.g. `Other(String)`). Using `#\[serde(other)\]` requires a unit variant, and switching to `#\[serde(untagged)\]` would change deserialization semantics for all variants. The original value can be recovered by inspecting the raw JSON if needed.

ImageDetail¶

Image detail level controlling token cost and processing.

Variant	Wire value	Description
`Low`	`low`	Low detail: scales image to 512x512, uses fewer tokens.
`High`	`high`	High detail: processes up to 2x2 grid of tiles, higher token cost.
`Auto`	`auto`	Auto: model chooses low or high based on image dimensions.

Message¶

A chat message in a conversation.

Variant	Wire value	Description
`System`	`system`	System — Fields: `_0`: `SystemMessage`
`User`	`user`	User — Fields: `_0`: `UserMessage`
`Assistant`	`assistant`	Assistant — Fields: `_0`: `AssistantMessage`
`Tool`	`tool`	Tool — Fields: `_0`: `ToolMessage`
`Developer`	`developer`	Developer — Fields: `_0`: `DeveloperMessage`
`Function`	`function`	Deprecated legacy function-role message; retained for API compatibility. — Fields: `_0`: `FunctionMessage`

Modality¶

Output modality requested from the model.

Passed as modalities: ["text", "audio"] (OpenAI) or translated to generationConfig.responseModalities (Gemini / Vertex AI).

Variant	Wire value	Description
`Text`	`text`	Text output (the default for all providers).
`Audio`	`audio`	Audio / speech output.
`Image`	`image`	Image output (Gemini Imagen, gpt-image-1).

ModerationInput¶

Input to the moderation endpoint — a single string or multiple strings.

Variant	Description
`Single`	Single text string. — Fields: `_0`: `String`
`Multiple`	Multiple text strings (batch moderation). — Fields: `_0`: `Vec<String>`

OcrDocument¶

Document input for OCR — either a URL or inline base64 data.

Variant	Wire value	Description
`Url`	`document_url`	A publicly accessible document URL. — Fields: `url`: `String`
`Base64`	`base64`	Inline base64-encoded document data. — Fields: `data`: `String`, `media_type`: `String`

ReasoningEffort¶

Controls how much reasoning effort the model should use.

Variant	Wire value	Description
`Low`	`low`	Low
`Medium`	`medium`	Medium
`High`	`high`	High

RerankDocument¶

A document to be reranked — either a plain string or an object with a text field.

Variant	Description
`Text`	Plain text document content. — Fields: `_0`: `String`
`Object`	Document with explicit text field (may include metadata). — Fields: `text`: `String`

ResponseFormat¶

Wire format for the chat completions response_format field.

Provider mapping¶

OpenAI (and OpenAI-compatible providers): emitted verbatim as {"type": "json_schema", "json_schema": {...}} per the chat-completions spec.
Gemini / Vertex AI: translated to generationConfig.responseMimeType = "application/json" and generationConfig.responseSchema = <schema>. The name, description, and strict fields are dropped — Gemini's structured-output API does not consume them.
Anthropic: no native JSON mode. A system instruction is prepended asking the model to respond with valid JSON. strict is advisory only; callers should still validate the returned JSON if the schema is load-bearing.

Variant	Wire value	Description
`Text`	`text`	Plain text output (default).
`JsonObject`	`json_object`	Output must be valid JSON object (no schema validation).
`JsonSchema`	`json_schema`	Output must conform to the specified JSON schema. — Fields: `json_schema`: `JsonSchemaFormat`

StopSequence¶

Stop sequence(s) that cause the model to stop generating.

Variant	Description
`Single`	Single stop sequence. — Fields: `_0`: `String`
`Multiple`	Multiple stop sequences. — Fields: `_0`: `Vec<String>`

ToolChoice¶

Tool usage mode or a specific tool to call.

Variant	Description
`Mode`	Predefined mode: auto, required, or none. — Fields: `_0`: `ToolChoiceMode`
`Specific`	Force a specific tool to be called. — Fields: `_0`: `SpecificToolChoice`

ToolType¶

The type discriminator for tool/tool-call objects.

Per the OpenAI spec this is always "function". Using an enum enforces that constraint at the type level and rejects any other value on deserialization.

Variant	Wire value	Description
`Function`	`function`	Function

UserContent¶

User message content as either plain text or a list of multimodal parts.

Variant	Description
`Text`	Plain text content. — Fields: `_0`: `String`
`Parts`	Array of content parts (text, images, documents, audio). — Fields: `_0`: `Vec<ContentPart>`

Edit this page on GitHub