Engine Application

Engine Components

Beyond the pipeline components the engine has a set of components in engine/component/ that the pipeline and the services build on: the realtime models that actually converse, speech recognition management and a number of smaller helpers.

Realtime Models

Summarized here; the contract, the session object and the element vocabulary are documented in Realtime Models.

A realtime model is what conducts the conversation for one session. It receives the system prompt and the tool box, and returns a RealtimeSession producing realtime elements — transcripts, response text, audio and tool calls. RealtimeSessionFactory selects the implementation whose name matches the provider of the resolved realtime properties.

Component Backing service

Component	Backing service
`OpenAIRealtimeModel`	OpenAI realtime API. Shares the protocol handling in `OpenAIBaseRealtimeModel`.
`OpenAILiveRealtimeModel`	OpenAI GPT-Live, a speech-to-speech model without turns, over its own WebSocket protocol.
`AzureOpenAIRealtimeModel`	The same protocol against an Azure OpenAI deployment.
`GeminiLiveRealtimeModel`	Google Gemini Live, with sessions created by `GeminiLiveAsyncSessionFactory`.
`ElevenLabsRealtimeModel`	ElevenLabs conversational realtime API.
`HumeAIRealtimeModel`	Hume AI realtime API.
`InternalRealtimeModel`	The engine’s own realtime pipeline: a streaming chat model resolved through the LLM factory, combined with the speech recognition and synthesis components. This is the implementation used when the conversation is assembled from separate STT, LLM and TTS providers rather than delegated to a speech-to-speech service.

OpenAIRealtimeModel

OpenAI realtime API. Shares the protocol handling in OpenAIBaseRealtimeModel.

OpenAILiveRealtimeModel

OpenAI GPT-Live, a speech-to-speech model without turns, over its own WebSocket protocol.

AzureOpenAIRealtimeModel

The same protocol against an Azure OpenAI deployment.

GeminiLiveRealtimeModel

Google Gemini Live, with sessions created by GeminiLiveAsyncSessionFactory.

ElevenLabsRealtimeModel

ElevenLabs conversational realtime API.

HumeAIRealtimeModel

Hume AI realtime API.

InternalRealtimeModel

The engine’s own realtime pipeline: a streaming chat model resolved through the LLM factory, combined with the speech recognition and synthesis components. This is the implementation used when the conversation is assembled from separate STT, LLM and TTS providers rather than delegated to a speech-to-speech service.

The realtime configuration of an agent — provider, model and voice — is resolved through RealtimeConfigurationService, and can be overridden per connection with realtimeConfigurationId.

Speech Recognition

Component Responsibility

Component	Responsibility
`SpeechRecognitionManager`	Owns recognition for a session: starts and stops recognizers around turns, disconnects a recognizer that receives no audio (configurable through `promethist.pipeline.SpeechRecognitionManager.*`), supplies conversation context in push-to-talk mode and reports recognition time to the EDR service.
`SpeechRecognizerCascadeFactory`	Builds a cascade over the available recognizer components: audio is buffered (up to `promethist.pipeline.SpeechRecognizerCascadeFactory.buffering.maxBytes`) so that when a provider fails, the next one can be started with the audio received so far. When every recognizer in the cascade fails, recognition fails as a whole.
`TritonTextPhonemizer`	Phonemization through a Triton-served model, used for animation data where the synthesizer does not provide it.

SpeechRecognitionManager

Owns recognition for a session: starts and stops recognizers around turns, disconnects a recognizer that receives no audio (configurable through promethist.pipeline.SpeechRecognitionManager.*), supplies conversation context in push-to-talk mode and reports recognition time to the EDR service.

SpeechRecognizerCascadeFactory

Builds a cascade over the available recognizer components: audio is buffered (up to promethist.pipeline.SpeechRecognizerCascadeFactory.buffering.maxBytes) so that when a provider fails, the next one can be started with the audio received so far. When every recognizer in the cascade fails, recognition fails as a whole.

TritonTextPhonemizer

Phonemization through a Triton-served model, used for animation data where the synthesizer does not provide it.

Citation Materialization

Component Responsibility

Component	Responsibility
`MaterializationRegistry`, `MaterializationRegistryStore`, `CitationMaterializer`, `CitationToolCallbackDecorator`	Track which retrieved knowledge was actually used in an answer, so responses can carry citations back to their source.

MaterializationRegistry, MaterializationRegistryStore, CitationMaterializer, CitationToolCallbackDecorator

Track which retrieved knowledge was actually used in an answer, so responses can carry citations back to their source.

Support Components

Component Responsibility

Component	Responsibility
`ProcessorManager`	Assembles the input normalization chain for a session — see Processors.
`EmotionBlender`	Blends the user’s and the agent’s emotional sets into the state the client renders.
`ThumbnailComposer`	Composes avatar and environment images into thumbnails and resizes them, behind the visual endpoints.
`SensitiveInformationMasker`	The masking rules applied before logging text.
`Summarizer`	Summaries of a session’s conversation: the rolling summary of its older part that the relational planning prompt carries instead of the full transcript, and the summary of the whole session made when it ends.
`DeviceCommonTools`, `DeviceWithAudioTools`, `DeviceWithoutAudioTools`	Tool sets offered to the model depending on what the connected device can do.
`SkillTools`, `HandoverTools`	Tools for invoking an agent skill and for handing the conversation over to another agent.
`WebSocketClient`	The outbound WebSocket client used by components that talk to external realtime services.
`EngineLoader`	Startup loading of engine resources.
`healthcheck/LLMChecker`, `healthcheck/RealtimeModelChecker`, `healthcheck/SpeechRecognizerChecker`, `healthcheck/TextSpeechSynthesizerChecker`	Probe the configured providers behind the health endpoints — see Observability.

ProcessorManager

Assembles the input normalization chain for a session — see Processors.

EmotionBlender

Blends the user’s and the agent’s emotional sets into the state the client renders.

ThumbnailComposer

Composes avatar and environment images into thumbnails and resizes them, behind the visual endpoints.

SensitiveInformationMasker

The masking rules applied before logging text.

Summarizer

Summaries of a session’s conversation: the rolling summary of its older part that the relational planning prompt carries instead of the full transcript, and the summary of the whole session made when it ends.

DeviceCommonTools, DeviceWithAudioTools, DeviceWithoutAudioTools

Tool sets offered to the model depending on what the connected device can do.

SkillTools, HandoverTools

Tools for invoking an agent skill and for handing the conversation over to another agent.

WebSocketClient

The outbound WebSocket client used by components that talk to external realtime services.

EngineLoader

Startup loading of engine resources.

healthcheck/LLMChecker, healthcheck/RealtimeModelChecker, healthcheck/SpeechRecognizerChecker, healthcheck/TextSpeechSynthesizerChecker

Probe the configured providers behind the health endpoints — see Observability.