models Package

A find action to search text within a page.

An open page action.

A web search action.

A search action source URL.

Configuration for the agent.

Configuration for animation outputs including blendshapes and visemes metadata.

An assistant message item within a conversation.

Echo cancellation configuration for server-side audio processing.

Configuration for input audio transcription.

Configuration for input audio noise reduction.

Configuration for avatar streaming and behavior during the session.

Azure avatar voice sync configuration. Uses personal voice synthesis with avatar character.

Azure custom voice configuration.

Azure personal voice configuration.

Azure semantic end-of-utterance detection (default).

Azure semantic end-of-utterance detection (English-optimized).

Azure semantic end-of-utterance detection (multilingual).

Server Speech Detection (Azure semantic VAD, default variant).

Server Speech Detection (Azure semantic VAD, English-only).

Server Speech Detection (Azure semantic VAD).

Azure standard voice configuration.

Base for Azure voice configurations.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: AzureAvatarVoiceSyncVoice, AzureCustomVoice, AzurePersonalVoice, AzureStandardVoice

Defines a video background, either a solid color or an image URL (mutually exclusive).

Details of output token usage.

A voicelive client event.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ClientEventConversationItemCreate, ClientEventConversationItemDelete, ClientEventConversationItemRetrieve, ClientEventConversationItemTruncate, ClientEventInputAudioClear, ClientEventInputAudioTurnAppend, ClientEventInputAudioTurnCancel, ClientEventInputAudioTurnEnd, ClientEventInputAudioTurnStart, ClientEventInputAudioBufferAppend, ClientEventInputAudioBufferClear, ClientEventInputAudioBufferCommit, ClientEventOutputAudioBufferClear, ClientEventResponseCancel, ClientEventResponseCreate, ClientEventSessionAvatarConnect, ClientEventSessionUpdate

Add a new Item to the Conversation's context, including messages, function calls, and function call responses. This event can be used both to populate a "history" of the conversation and to add new items mid-stream, but has the current limitation that it cannot populate assistant audio messages. If successful, the server will respond with a conversation.item.created event, otherwise an error event will be sent.

Send this event when you want to remove any item from the conversation history. The server will respond with a conversation.item.deleted event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

Send this event when you want to retrieve the server's representation of a specific item in the conversation history. This is useful, for example, to inspect user audio after noise cancellation and VAD. The server will respond with a conversation.item.retrieved event, unless the item does not exist in the conversation history, in which case the server will respond with an error.

Send this event to truncate a previous assistant message's audio. The server will produce audio faster than voicelive, so this event is useful when the user interrupts to truncate audio that has already been sent to the client but not yet played. This will synchronize the server's understanding of the audio with the client's playback. Truncating audio will delete the server-side text transcript to ensure there is not text in the context that hasn't been heard by the user. If successful, the server will respond with a conversation.item.truncated event.

Send this event to append audio bytes to the input audio buffer. The audio buffer is temporary storage you can write to and later commit. In Server VAD mode, the audio buffer is used to detect speech and the server will decide when to commit. When Server VAD is disabled, you must commit the audio buffer manually.

The client may choose how much audio to place in each event up to a maximum of 15 MiB, for example streaming smaller chunks from the client may allow the VAD to be more responsive. Unlike made other client events, the server will not send a confirmation response to this event.

Send this event to clear the audio bytes in the buffer. The server will respond with an input_audio_buffer.cleared event.

Send this event to commit the user input audio buffer, which will create a new user message item in the conversation. This event will produce an error if the input audio buffer is empty. When in Server VAD mode, the client does not need to send this event, the server will commit the audio buffer automatically. Committing the input audio buffer will trigger input audio transcription (if enabled in session configuration), but it will not create a response from the model. The server will respond with an input_audio_buffer.committed event.

Clears all input audio currently being streamed.

Appends audio data to an ongoing input turn.

Cancels an in-progress input audio turn.

Marks the end of an audio input turn.

Indicates the start of a new audio input turn.

Client request to clear the avatar output buffer.

Send this event to cancel an in-progress response. The server will respond with a response.cancelled event or an error if there is no response to cancel.

This event instructs the server to create a Response, which means triggering model inference. When in Server VAD mode, the server will create Responses automatically. A Response will include at least one Item, and may have two, in which case the second will be a function call. These Items will be appended to the conversation history. The server will respond with a response.created event, events for Items and content created, and finally a response.done event to indicate the Response is complete. The response.create event includes inference configuration like instructions, and temperature. These fields will override the Session's configuration for this Response only.

Sent when the client connects and provides its SDP (Session Description Protocol)

for avatar-related media negotiation.

Send this event to update the session's default configuration. The client may send this event at any time to update any field, except for voice. However, note that once a session has been initialized with a particular model, it can't be changed to another model using session.update. When the server receives a session.update, it will respond with a session.updated event showing the full, effective configuration. Only the fields that are present are updated. To clear a field like instructions, pass an empty string.

Base for any content part; discriminated by type.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ResponseAudioContentPart, RequestAudioContentPart, RequestImageContentPart, RequestTextContentPart, ResponseTextContentPart

The item to add to the conversation.

Base for any response item; discriminated by type.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: FunctionCallItem, FunctionCallOutputItem, MCPApprovalResponseRequestItem, MessageItem

Top-level union for end-of-utterance (EOU) semantic detection configuration.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: AzureSemanticDetection, AzureSemanticDetectionEn, AzureSemanticDetectionMultilingual

Standard error response envelope.

A file search result entry.

A function call item within a conversation.

A function call output item within a conversation.

The definition of a function tool as used by the voicelive endpoint.

ICE server configuration for WebRTC connection negotiation.

Input audio content part.

Input text content part.

Details of input token usage.

Base model for interim response configuration.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: LlmInterimResponseConfig, StaticInterimResponseConfig

Configuration for LLM-based interim response generation. Uses LLM to generate context-aware interim responses when any trigger condition is met.

A single log probability entry for a token.

A request item that represents a response to an MCP approval request.

The definition of an MCP server as used by the voicelive endpoint.

Represents a mcp tool definition.

Base for any message content part; discriminated by type.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: InputAudioContentPart, InputTextContentPart, OutputTextContentPart

A message item within a conversation.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: AssistantMessageItem, SystemMessageItem, UserMessageItem

OpenAI voice configuration with explicit type field.

This provides a unified interface for OpenAI voices, complementing the existing string-based OAIVoice for backward compatibility.

Output text content part.

Details of output token usage.

An audio content part for a request. This is supported only by realtime models (e.g., gpt-realtime). For text-based models, use input_text instead.

Input image content part.

Extended RequestSession that tracks explicitly set None values.

A text content part for a request.

The response resource.

An audio content part for a response.

Details for a cancelled response.

Create a new VoiceLive response with these parameters.

Details for a failed response.

A response item that represents a file search call.

A function call item within a conversation.

A function call output item within a conversation.

Details for an incomplete response.

Base for any response item; discriminated by type.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ResponseFileSearchCallItem, ResponseFunctionCallItem, ResponseFunctionCallOutputItem, ResponseMCPApprovalRequestItem, ResponseMCPApprovalResponseItem, ResponseMCPCallItem, ResponseMCPListToolItem, ResponseMessageItem, ResponseWebSearchCallItem

A response item that represents a request for approval to call an MCP tool.

A response item that represents a response to an MCP approval request.

A response item that represents a call to an MCP tool.

A response item that lists the tools available on an MCP server.

Base type for message item within a conversation.

Base for session configuration in the response.

Base for all non-success response details.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ResponseCancelledDetails, ResponseFailedDetails, ResponseIncompleteDetails

A text content part for a response.

A response item that represents a web search call.

Configuration for avatar's zoom level, position, rotation and movement amplitude in the video frame.

A voicelive server event.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ServerEventConversationItemCreated, ServerEventConversationItemDeleted, ServerEventConversationItemInputAudioTranscriptionCompleted, ServerEventConversationItemInputAudioTranscriptionDelta, ServerEventConversationItemInputAudioTranscriptionFailed, ServerEventConversationItemRetrieved, ServerEventConversationItemTruncated, ServerEventError, ServerEventInputAudioBufferCleared, ServerEventInputAudioBufferCommitted, ServerEventInputAudioBufferSpeechStarted, ServerEventInputAudioBufferSpeechStopped, ServerEventMcpListToolsCompleted, ServerEventMcpListToolsFailed, ServerEventMcpListToolsInProgress, ServerEventOutputAudioBufferCleared, ServerEventResponseAnimationBlendshapeDelta, ServerEventResponseAnimationBlendshapeDone, ServerEventResponseAnimationVisemeDelta, ServerEventResponseAnimationVisemeDone, ServerEventResponseAudioDelta, ServerEventResponseAudioDone, ServerEventResponseAudioTimestampDelta, ServerEventResponseAudioTimestampDone, ServerEventResponseAudioTranscriptAnnotationAdded, ServerEventResponseAudioTranscriptDelta, ServerEventResponseAudioTranscriptDone, ServerEventResponseContentPartAdded, ServerEventResponseContentPartDone, ServerEventResponseCreated, ServerEventResponseDone, ServerEventResponseFileSearchCallCompleted, ServerEventResponseFileSearchCallInProgress, ServerEventResponseFileSearchCallSearching, ServerEventResponseFunctionCallArgumentsDelta, ServerEventResponseFunctionCallArgumentsDone, ServerEventResponseMcpCallCompleted, ServerEventResponseMcpCallFailed, ServerEventResponseMcpCallInProgress, ServerEventResponseMcpCallArgumentsDelta, ServerEventResponseMcpCallArgumentsDone, ServerEventResponseOutputItemAdded, ServerEventResponseOutputItemDone, ServerEventResponseTextDelta, ServerEventResponseTextDone, ServerEventResponseVideoDelta, ServerEventResponseWebSearchCallCompleted, ServerEventResponseWebSearchCallInProgress, ServerEventResponseWebSearchCallSearching, ServerEventSessionAvatarConnecting, ServerEventSessionAvatarSwitchToIdle, ServerEventSessionAvatarSwitchToSpeaking, ServerEventSessionCreated, ServerEventSessionUpdated, ServerEventWarning

Returned when a conversation item is created. There are several scenarios that produce this event:

The server is generating a Response, which if successful will produce either one or two Items, which will be of type message (role assistant) or type function_call. The input audio buffer has been committed, either by the client or the server (in server_vad mode). The server will take the content of the input audio buffer and add it to a new user message Item. The client has sent a conversation.item.create event to add a new Item to the Conversation.

Returned when an item in the conversation is deleted by the client with a conversation.item.delete event. This event is used to synchronize the server's understanding of the conversation history with the client's view.

This event is the output of audio transcription for user audio written to the user audio buffer. Transcription begins when the input audio buffer is committed by the client or server (in server_vad mode). Transcription runs asynchronously with Response creation, so this event may come before or after the Response events. VoiceLive API models accept audio natively, and thus input transcription is a separate process run on a separate ASR (Automatic Speech Recognition) model. The transcript may diverge somewhat from the model's interpretation, and should be treated as a rough guide.

Returned when the text value of an input audio transcription content part is updated.

Returned when input audio transcription is configured, and a transcription request for a user message failed. These events are separate from other error events so that the client can identify the related Item.

Returned when a conversation item is retrieved with conversation.item.retrieve.

Returned when an earlier assistant audio message item is truncated by the client with a conversation.item.truncate event. This event is used to synchronize the server's understanding of the audio with the client's playback. This action will truncate the audio and remove the server-side text transcript to ensure there is no text in the context that hasn't been heard by the user.

Returned when an error occurs, which could be a client problem or a server problem. Most errors are recoverable and the session will stay open, we recommend to implementors to monitor and log error messages by default.

Details of the error.

Returned when the input audio buffer is cleared by the client with a input_audio_buffer.clear event.

Returned when an input audio buffer is committed, either by the client or automatically in server VAD mode. The item_id property is the ID of the user message item that will be created, thus a conversation.item.created event will also be sent to the client.

Sent by the server when in server_vad mode to indicate that speech has been detected in the audio buffer. This can happen any time audio is added to the buffer (unless speech is already detected). The client may want to use this event to interrupt audio playback or provide visual feedback to the user. The client should expect to receive a input_audio_buffer.speech_stopped event when speech stops. The item_id property is the ID of the user message item that will be created when speech stops and will also be included in the input_audio_buffer.speech_stopped event (unless the client manually commits the audio buffer during VAD activation).

Returned in server_vad mode when the server detects the end of speech in the audio buffer. The server will also send an conversation.item.created event with the user message item that is created from the audio buffer.

MCP list tools completed message.

MCP list tools failed message.

MCP list tools in progress message.

Returned when the output audio buffer has been cleared.

Represents a delta update of blendshape animation frames for a specific output of a response.

Indicates the completion of blendshape animation processing for a specific output of a response.

Represents a viseme ID delta update for animation based on audio.

Indicates completion of viseme animation delivery for a response.

Returned when the model-generated audio is updated.

Returned when the model-generated audio is done. Also emitted when a Response is interrupted, incomplete, or cancelled.

Represents a word-level audio timestamp delta for a response.

Indicates completion of audio timestamp delivery for a response.

Returned when an audio transcript annotation is added to a response.

Returned when the model-generated transcription of audio output is updated.

Returned when the model-generated transcription of audio output is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Returned when a new content part is added to an assistant message item during response generation.

Returned when a content part is done streaming in an assistant message item. Also emitted when a Response is interrupted, incomplete, or cancelled.

Returned when a new Response is created. The first event of response creation, where the response is in an initial state of in_progress.

Returned when a Response is done streaming. Always emitted, no matter the final state. The Response object included in the response.done event will include all output Items in the Response but will omit the raw audio data.

Returned when a file search call has completed.

Returned when a file search call is in progress.

Returned when a file search call is searching.

Returned when the model-generated function call arguments are updated.

Returned when the model-generated function call arguments are done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Represents a delta update of the arguments for an MCP tool call.

Indicates the completion of the arguments for an MCP tool call.

Indicates the MCP call has completed.

Indicates the MCP call has failed.

Indicates the MCP call running.

Returned when a new Item is created during Response generation.

Returned when an Item is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Returned when the text value of a "text" content part is updated.

Returned when the text value of a "text" content part is done streaming. Also emitted when a Response is interrupted, incomplete, or cancelled.

Returned when avatar video frame data is streamed.

Returned when a web search call has completed.

Returned when a web search call is in progress.

Returned when a web search call is searching.

Sent when the server is in the process of establishing an avatar media connection and provides its SDP answer.

Returned when the avatar switches to idle state.

Returned when the avatar switches to speaking state.

Returned when a Session is created. Emitted automatically when a new connection is established as the first server event. This event will contain the default Session configuration.

Returned when a session is updated with a session.update event, unless there is an error.

Returned when a warning occurs that does not interrupt the conversation flow. Warnings are informational and the session will continue normally.

Details of the warning.

Base model for VAD-based turn detection.

VoiceLive session object configuration.

Configuration for static interim response generation. Randomly selects from configured texts when any trigger condition is met.

A system message item within a conversation.

Overall usage statistics for a response.

The base representation of a voicelive tool definition.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: FunctionTool, MCPServer

The representation of a voicelive tool_choice selecting a named function tool.

A base representation for a voicelive tool_choice selecting a named tool.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: ToolChoiceFunctionSelection

A transcribed phrase with timing information.

A time-stamped word in the transcription.

Top-level union for turn detection configuration.

You probably want to use the sub-classes and not this class directly. Known sub-classes are: AzureSemanticVad, AzureSemanticVadEn, AzureSemanticVadMultilingual, ServerVad

A user message item within a conversation.

Defines a video crop rectangle using top-left and bottom-right coordinates.

Video streaming parameters for avatar.

Resolution of the video feed in pixels.

Error object returned in case of API failure.

Specifies the types of animation data to output.

Output timestamp types supported in audio response content.

Avatar config types.

Avatar config output protocols.

Union of all supported Azure voice types.

Client event types used in VoiceLive protocol.

Type of ContentPartType.

Threshold level settings for Azure semantic end-of-utterance detection.

Input audio format types supported.

Interim response configuration types.

Triggers that can activate interim response generation.

Indicates the processing status of an item or parameter.

Type of ItemType.

The available set of MCP approval types.

Type of MessageRole.

Supported modalities for the session.

Supported OpenAI voice names (string enum).

Output audio format types supported.

PersonalVoice models.

Photo avatar base modes.

Constrains effort on reasoning for reasoning models. Check model documentation for supported values for each model. Reducing reasoning effort can result in faster responses and fewer tokens used on reasoning in a response.

Specifies an image's detail level. Can be 'auto', 'low', 'high', or an unknown future value.

Indicates the processing status of a response item.

Terminal status of a response.

Server event types used in VoiceLive protocol.

Options for what additional data to include in session responses.

The available set of mode-level, string literal tool_choice options for the voicelive endpoint.

The supported tool type discriminators for voicelive tools. Currently, only 'function' tools are supported.

Type of TurnDetectionType.