Interaction Pipeline

Interaction Pipeline Components

Pipeline components are the pluggable parts of a conversation. Each implements one of the PipelineComponent sub-interfaces defined in the common library; the engine’s implementations live under engine/pipeline/component/.

Components are ordinary Spring beans. Pipeline collects all of them and sorts them by type at construction, so declaring a bean is all that is needed to add one — see Adding a Component to the Engine.

The component diagram in Realtime Models shows how these categories relate to each other and to the realtime model.

Interaction and Planning Managers

An interaction manager decides what the agent does in a turn: it composes the system prompt, provides the tool box and reacts to the conversation’s lifecycle. A planning manager maintains the conversational plan — what the agent is trying to achieve and how far it has got.

Component Role

Component	Role
`RelationalAgentInteractionManager`	Interaction manager for relational agents — goal and relationship oriented conversations, backed by the planning manager and memory aggregation. See Relational Management.
`BasicAgentInteractionManager`	Interaction manager for basic agents. Builds the system prompt from the agent’s identity, purpose and guardrails, and drives tool calling. Serves both turn-based and turnless models.
`TestAgentInteractionManager`	Interaction manager for test agents, used by the automated testing flow. Triggers session aggregation on disconnect when the agent has memory.
`BasicInteractionManager`	Minimal manager used by the `realtime` setup.
`PreviewInteractionManager`	One-shot voice preview for the `realtime-preview` setup.
`RelationalPlanningManager`	Produces the conversational plan from the agent’s business objectives, the transcript and the previous plan, and steers the speaking model with the instructions it emits. Extends the abstract `PlanningManager`, which defines the prompt scaffold. See Per-Turn Planning.
`TestingPlanningManager`	Planning manager emitting predefined instructions, used for testing.
`TurnPipelineManager` and `TurnlessPipelineManager`	Not interaction managers but the orchestrators that drive them, for a conversation with and without turns — see Realtime Conversation Lifecycle.
`InitialPlanGenerator`	Generates the opening plan of a session, before any user turn exists — see Initial Planning.

RelationalAgentInteractionManager

Interaction manager for relational agents — goal and relationship oriented conversations, backed by the planning manager and memory aggregation. See Relational Management.

BasicAgentInteractionManager

Interaction manager for basic agents. Builds the system prompt from the agent’s identity, purpose and guardrails, and drives tool calling. Serves both turn-based and turnless models.

TestAgentInteractionManager

Interaction manager for test agents, used by the automated testing flow. Triggers session aggregation on disconnect when the agent has memory.

BasicInteractionManager

Minimal manager used by the realtime setup.

PreviewInteractionManager

One-shot voice preview for the realtime-preview setup.

RelationalPlanningManager

Produces the conversational plan from the agent’s business objectives, the transcript and the previous plan, and steers the speaking model with the instructions it emits. Extends the abstract PlanningManager, which defines the prompt scaffold. See Per-Turn Planning.

TestingPlanningManager

Planning manager emitting predefined instructions, used for testing.

TurnPipelineManager and TurnlessPipelineManager

Not interaction managers but the orchestrators that drive them, for a conversation with and without turns — see Realtime Conversation Lifecycle.

InitialPlanGenerator

Generates the opening plan of a session, before any user turn exists — see Initial Planning.

Speech Recognizers

Streaming speech-to-text implementations. Each takes a Flux<Binary> of audio and produces a Flux<Transcript> of interim and final results; provider failures are mapped to a SpeechRecognitionException.

Component Provider

Component	Provider
`AmazonSpeechRecognizer`	AWS Transcribe streaming.
`GoogleSpeechRecognizer`	Google Cloud Speech-to-Text streaming (gRPC).
`DeepgramSpeechRecognizer`	Deepgram realtime API over WebSocket.
`SonioxSpeechRecognizer`	Soniox realtime transcription API.

AmazonSpeechRecognizer

AWS Transcribe streaming.

GoogleSpeechRecognizer

Google Cloud Speech-to-Text streaming (gRPC).

DeepgramSpeechRecognizer

Deepgram realtime API over WebSocket.

SonioxSpeechRecognizer

Soniox realtime transcription API.

Speech Synthesizers

Text-to-speech implementations. Whole-text synthesizers extend TextSpeechSynthesizer; providers supporting incremental synthesis also implement TokenSpeechSynthesizer and consume a stream of text tokens. Each declares the audio file types it can produce and, per locale, the animation data it can deliver (visemes or blend shapes) for avatar lip-sync.

Component Provider

Component	Provider
`ElevenLabsSynthesizer`	ElevenLabs, including the streaming endpoint with word timestamps.
`GoogleSynthesizer`	Google Cloud Text-to-Speech.
`MicrosoftSynthesizer`	Azure Cognitive Services Speech SDK.
`AmazonSynthesizer`	AWS Polly.
`OpenAISynthesizer`	OpenAI speech API.
`HumeAISynthesizer`	Hume AI streaming TTS.
`MamaAISynthesizer`	MamaAI voice service.
`TritonVoiceSynthesizer`	Self-hosted voice model served by Triton.
`ExperimentalSynthesizer`	Internal endpoint used to try out new voice models.
`DummySynthesizer`	Returns empty audio. Used in tests and when synthesis is disabled.

ElevenLabsSynthesizer

ElevenLabs, including the streaming endpoint with word timestamps.

GoogleSynthesizer

Google Cloud Text-to-Speech.

MicrosoftSynthesizer

Azure Cognitive Services Speech SDK.

AmazonSynthesizer

AWS Polly.

OpenAISynthesizer

OpenAI speech API.

HumeAISynthesizer

Hume AI streaming TTS.

MamaAISynthesizer

MamaAI voice service.

TritonVoiceSynthesizer

Self-hosted voice model served by Triton.

ExperimentalSynthesizer

Internal endpoint used to try out new voice models.

DummySynthesizer

Returns empty audio. Used in tests and when synthesis is disabled.

The synthesizer used for a session is resolved from the agent’s configuration, with defaults from promethist.pipeline.tts.*. SpeechService wraps the chosen synthesizer with output normalization, chunking and — when useCaching is enabled — the audio cache in resource storage.

Processors

Processors normalize the element stream before it reaches the conversation logic. ProcessorManager assembles the active chain per session from ProcessorProperties and the session’s configuration, so each one can be switched off per deployment or per connection.

Component Effect

Component	Effect
`IgnoreShortAudioBurstProcessor`	Drops audio bursts shorter than the configured threshold, suppressing clicks and background noise.
`IgnoreAfterTurnStartAudioProcessor`	Drops client audio for a short window after a turn ends, so the agent does not hear the tail of its own speech through an open microphone.
`IgnoreFirstTurnAudioProcessor`	Drops client audio until the first turn has finished, unless `useAudioFromStart` is set.
`IgnoreSyntheticSilenceProcessor`	In push-to-talk mode, drops the synthetic silence clients send to keep the stream alive.
`TokenNormalizerProcessor`	Trims and concatenates text tokens into a transcript, applying the sentence separator rules.
`OutputTranscriptProcessor`	Emits the transcript-related output elements derived from recognition results.

IgnoreShortAudioBurstProcessor

Drops audio bursts shorter than the configured threshold, suppressing clicks and background noise.

IgnoreAfterTurnStartAudioProcessor

Drops client audio for a short window after a turn ends, so the agent does not hear the tail of its own speech through an open microphone.

IgnoreFirstTurnAudioProcessor

Drops client audio until the first turn has finished, unless useAudioFromStart is set.

IgnoreSyntheticSilenceProcessor

In push-to-talk mode, drops the synthetic silence clients send to keep the stream alive.

TokenNormalizerProcessor

Trims and concatenates text tokens into a transcript, applying the sentence separator rules.

OutputTranscriptProcessor

Emits the transcript-related output elements derived from recognition results.

Extractors

Extractors run a language model over a turn or a session and store structured data into scoped memory. Some extend MetricExtractor, which runs a prompt, parses the result into a typed DTO and writes it as extractor data; the model used by each is configured under promethist.ai.llm-components.{ExtractorName}.

Component Extracts

Component	Extracts
`FactualDataExtractor`	Hard facts about the user (names, preferences, circumstances).
`UserProfileEnhancementExtractor`	Additional profile enrichment, written to data memory as the user’s bio.
`SystemExtractor`	Not an LLM extractor — signals whether audio was received in the turn.

FactualDataExtractor

Hard facts about the user (names, preferences, circumstances).

UserProfileEnhancementExtractor

Additional profile enrichment, written to data memory as the user’s bio.

SystemExtractor

Not an LLM extractor — signals whether audio was received in the turn.

AnalyticalJudges holds the shared judge prompts used to validate extraction results.

Filters

Filters wrap the element stream and are applied around the conversation logic.

Component Effect

Component	Effect
`EmotionalFilter`	Interpolates emotional expression values over time from an `EmotionRecognizer` and emits the visual state the client renders. Enabled with `promethist.pipeline.emotional`.
`RecordingFilter`	Records the session’s audio and elements into resource storage, making the recording available through the recording endpoints. Enabled with `promethist.pipeline.recording` and the project’s recording setting.
`LiveAvatarFilter`	Bridges speech and turn events to the external LiveAvatar service over REST and WebSocket (session token, keep-alive, speak, interrupt, listening).

EmotionalFilter

Interpolates emotional expression values over time from an EmotionRecognizer and emits the visual state the client renders. Enabled with promethist.pipeline.emotional.

RecordingFilter

Records the session’s audio and elements into resource storage, making the recording available through the recording endpoints. Enabled with promethist.pipeline.recording and the project’s recording setting.

LiveAvatarFilter

Bridges speech and turn events to the external LiveAvatar service over REST and WebSocket (session token, keep-alive, speak, interrupt, listening).

Emotion input itself comes from an EmotionRecognizer; ConstantEmotionRecognizer is the default implementation, selected by promethist.pipeline.emotional.recognizer.

Action Mappers and Element Filters

Action mappers turn the #name tokens a client sends into an Action the language model understands, resolving the wording per locale. Element filters intercept elements without producing an action.

Component Handles

Component	Handles
`IntroActionMapper`	`#intro` — the agent’s greeting.
`ReadyActionMapper`	`#ready` — the client is ready to interact.
`SilenceActionMapper`	Silence detected on the client side.
`YesActionMapper`, `NoActionMapper`	`#yes` and `#no` confirmations.
`ResumeActionMapper`, `DeclineActionMapper`	Resuming a conversation and declining an offer.
`TranscriptActionMapper`	`#action:transcript` — a final transcript supplied as an action; falls back to silence when blank.
`RatingActionMapper`	`#action:rating` — parses the rating payload and stores it on the session.
`MultimodalActionMapper`	`#action:multimodal` — the user’s response to a displayed multimodal interaction.
`ProjectActionMapper`	Custom actions defined by the project, matched by ref.
`TestedAgentActionMapper`	`#tested-agent:ref=…` — the automated test harness switching the agent under test.
`SessionAttributeFilter`	`#session-attribute:name=…,value=…` — stores an attribute on the session.
`SessionSensoryFilter`	Captures sensory elements (device and environment data) onto the session.
`SessionTraceFilter`	`#session-trace:traceId=…,spanId=…` — correlates the session with the client’s distributed trace.

IntroActionMapper

#intro — the agent’s greeting.

ReadyActionMapper

#ready — the client is ready to interact.

SilenceActionMapper

Silence detected on the client side.

YesActionMapper, NoActionMapper

#yes and #no confirmations.

ResumeActionMapper, DeclineActionMapper

Resuming a conversation and declining an offer.

TranscriptActionMapper

#action:transcript — a final transcript supplied as an action; falls back to silence when blank.

RatingActionMapper

#action:rating — parses the rating payload and stores it on the session.

MultimodalActionMapper

#action:multimodal — the user’s response to a displayed multimodal interaction.

ProjectActionMapper

Custom actions defined by the project, matched by ref.

TestedAgentActionMapper

#tested-agent:ref=… — the automated test harness switching the agent under test.

SessionAttributeFilter

#session-attribute:name=…,value=… — stores an attribute on the session.

SessionSensoryFilter

Captures sensory elements (device and environment data) onto the session.

SessionTraceFilter

#session-trace:traceId=…,spanId=… — correlates the session with the client’s distributed trace.

Aggregators

Aggregators condense a finished conversation into memory.

Component Produces

Component	Produces
`SessionAggregator`	At session end, generates the conversation summary and stores it as the session’s `SessionData` in session memory (start, end, summary) and as the application-scoped insight the next session’s initial plan reads.

SessionAggregator

At session end, generates the conversation summary and stores it as the session’s SessionData in session memory (start, end, summary) and as the application-scoped insight the next session’s initial plan reads.