Interaction Pipeline

Realtime Conversation Lifecycle

A realtime conversation is driven by one of two pipeline managers, chosen from the model’s properties — TurnPipelineManager for a conversation delimited into turns and TurnlessPipelineManager for a model that speaks continuously, see Pipeline Managers. This page describes how the conversation starts, which is common to both, then the turn-based lifecycle: how the element stream is cut into turns, when planning and extraction run, and what happens when a turn or the session ends. What a turnless conversation does instead is described in Turnless Conversations.

The provider side is described in Realtime Models; this page is about the lifecycle around it and the InteractionManager hooks that plug into it.

The Interaction Manager Contract

interface InteractionManager<C : RealtimeContext> : PipelineComponent {
    fun getSystemPrompt(context: C, model: RealtimeModel<RealtimeProperties>): Mono<SystemPrompt>

    fun getToolBox(context: C): ToolBox = EmptyToolBox
    fun onSessionStart(context: C): Mono<Void> = Mono.empty()
    fun onSessionDisconnect(context: C): Mono<Void> = Mono.empty()
    fun onChatEvent(context: C, event: ChatEvent): Mono<Void> = Mono.empty()
}

interface TurnInteractionManager<C : RealtimeContext> : InteractionManager<C> {
    val planningManager: RealtimePlanningManager<C>

    fun process(context: C, input: Input): Flux<RealtimeElement> = Flux.empty()
    fun turnEnd(context: C): Mono<Void> = Mono.empty()
}

interface TurnlessInteractionManager<C : RealtimeContext> : InteractionManager<C> {
    fun onChatMessage(context: C, chatMessage: ChatMessage): Mono<Void>
}

interface TurnlessDelegatingInteractionManager<C : RealtimeContext> : TurnlessInteractionManager<C> {
    fun onSessionDelegation(context: C, text: String): Flux<RealtimeInstruction>
}

InteractionManager holds what every conversation needs and only getSystemPrompt must be implemented; every hook defaults to doing nothing. TurnInteractionManager adds the planning manager and the hooks that only make sense at a turn boundary, TurnlessInteractionManager the one hook a conversation without turns has, and TurnlessDelegatingInteractionManager lets the manager answer when the model asks for help. One class may implement both a turn and a turnless contract, which is what lets an agent run on either kind of model: BasicAgentInteractionManager does.

The manager is selected per pipeline setup — see Pipeline Setups. For the agent setup there is one manager per agent type (relational, basic and test), all registered for the setup with the agent class they serve, and the one matching the agent is used; there is no proxy in between.

Session Start

A conversation is started by PipelineService and RealtimeSessionFactory, before a pipeline manager takes over:

  1. The context service builds the context: agent, user, project (with the permission and subscription check), memory, the resolved RealtimeConfiguration and, for a relational agent, the shared agents.

  2. As soon as the project passes those checks, AgentContextService saves the session, so userId, agent, projectId, tenantId and projectRef reach the stored document at the start of the conversation rather than only at session end. The document created in Session Lifecycle predates agent resolution and carries none of them; without this save, anything that reads the session by id while it is running — after-session feedback specifications above all, which resolve their definitions through the session’s project — would race the final write.

  3. PipelineService resolves the interaction manager for the context, and RealtimeSessionFactory finds the realtime model whose name matches the provider of the resolved properties.

  4. interactionManager.onSessionStart(context) runs first — before the system prompt is built and before the provider session exists. Everything an agent needs warmed up (memory aggregation, pre-session webhook, initial plan generation) happens here, which is why a heavy onSessionStart directly delays the first response.

  5. getSystemPrompt(context, model) is called, and the chat history is loaded in parallel — from the conversation memory for a turn-based model, from the session’s stored chat messages for a turnless one.

  6. Reconnect handling. If the chat history is not empty, the session is a reconnect: the instruction template id gets a .reconnect suffix, the variable is_reconnect is set and the history is passed in as chat_history. The agent therefore resumes instead of greeting again.

  7. getToolBox(context) is called — wrapped to record the tool calls for LLM logging, except for the InternalRealtimeModel, which records its own — and model.process(…​) constructs the RealtimeSession. The model is told whether the interaction manager takes delegations (is a TurnlessDelegatingInteractionManager), which only OpenAI GPT-Live makes use of. The rendered prompt is recorded for LLM logging.

  8. PipelineService picks the pipeline manager from the model’s properties, and its output begins with a ready command carrying the resolved provider, sample rate and temperature.

The turn-based conversation is a Flux.usingWhen in TurnPipelineManager whose resource is this context — releasing it ends the session.

Input Handling

Before input reaches the model — with either pipeline manager — it passes through, in order:

  1. resampling of raw audio to the provider’s sample rate (and images are stashed on the context);

  2. the pipeline’s ElementFilters;

  3. ActionMapper resolution, turning #name tokens into actions.

The turn-based manager then wraps the model in the pipeline’s BlockElementFilters, which wrap the model rather than preceding it — this is how recording and emotional interpolation surround a turn. The turnless manager applies none.

In the turn-based manager the model’s output is merged with a side channel (Sinks.Many) that carries tool events, so tool lifecycle commands interleave with the conversation. Synthesized audio is resampled back to the connection’s output sample rate when they differ.

Turn Windowing

Turns are derived from two coupled streams: a stream of contexts and a stream of element windows. Each context corresponds to exactly one window, and therefore one turn. The first context is the one created at session start; every subsequent context is pushed by the previous turn, which simultaneously closes the old window and opens the next.

Within a turn the manager accumulates the response text, the final transcript and the set of multimodal interactions shown. The turn’s back-end processing is triggered by a specific condition: both a final (non-interim) transcript and a ResponseEnd have been seen. At that point:

  1. the turn’s input and output are recorded, the transcript is added to memory as a user message and the response as assistant messages;

  2. an Input is created through pipeline.createInput(…​), which selects the extractors annotated for the current setup;

  3. three legs start in parallel — the tool event sink, interactionManager.process(context, input) (extraction) and planningManager.process(…​) (planning);

  4. when those legs finish, the next turn’s context is emitted.

Barge-in is the cross-cutting concern here. A cached "next interim transcript" signal is derived per turn, and both extraction and planning are cancelled by it — if the user starts speaking again, the work for the previous turn is abandoned rather than completed against a stale turn. Speech post-processing is dropped and restarted the same way.

Planning and Instruction Injection

Planning runs after the model has already produced its immediate reaction, in parallel with extraction. Its output steers what comes next rather than the current utterance, which is what keeps response latency independent of planning latency.

Each element the planning manager emits is handled as follows:

  • it is forwarded to the client (so planning progress and instructions are observable);

  • if it is a RealtimeInstruction, it is recorded for LLM logging and injected into the live session with realtimeSession.process(instruction);

  • a contextual instruction (SUPPORTING_CONTEXT) ends there — it only enriches the model’s context;

  • a non-contextual instruction (INSTRUCTION or COMMENTARY) additionally waits for the model’s next ResponseEnd and then emits a response-instruction-end command, so a caller can tell when the instructed utterance has completed.

Tool Calls

Tools are executed inside the provider session, not by the orchestrator — the orchestrator only supplies the ToolHandler. That handler turns the callbacks into client-visible commands: tool-start (with the tool name and parameters), tool-success and tool-error. Two handlers exist per conversation: one session-scoped for the model’s own tools and one per turn for the planner’s.

Providers additionally emit tool-call commands for dispatch and completion. A failing tool does not break the conversation: the provider is sent a structured error result so the model can recover and explain itself, and the failure is reported as a command.

Turn End

Every turn’s output ends with an #exit command carrying the turn, session and user IDs, so a client always has an unambiguous turn boundary.

Cleanup runs only if the turn actually emitted something — an empty turn is neither persisted nor reported. When it did emit:

  1. the timing log (last interim transcript, final transcript, first and last text, first and last audio) is written onto the turn;

  2. pipeline.saveContext(context) persists the turn — failures are logged and swallowed rather than failing the conversation;

  3. interactionManager.turnEnd(context) runs, which is where an agent injects per-turn context and triggers per-turn evaluation;

  4. the turn counter metric is incremented.

Session End

The usingWhen release phase runs two things concurrently, each with errors logged and swallowed:

  • finalize(context) — saves the session and writes the usage record (EDR.Realtime) with the session’s active duration and accumulated token usage, tagged by provider, project, agent and test mode;

  • interactionManager.onSessionDisconnect(context) — the agent’s own teardown: cache clearing, end-of-session extraction and memory aggregation.

Nothing closes the provider connection explicitly: it is torn down as a side effect of the output stream terminating, whether that is completion, error or cancellation. PipelineService provides the backstop — once client input has completed, the output is force-closed after promethist.pipeline.sessionEndTimeoutMs (20 seconds by default). See Session Lifecycle for what the surrounding session state machine then does.

Turnless Conversations

A turnless model — OpenAI GPT-Live, isTurnless — speaks continuously and has no response that ends, so there is nothing to cut into turns. TurnlessPipelineManager leaves out everything built on a turn: the turn windows, the #exit command, the response end, per-turn extraction and planning, turnEnd, the block element filters and the per-turn saving of the context. Its input is prepared as described in Input Handling and no engine speech recognition is involved, as the model recognizes speech itself.

What it does instead:

  • The model’s elements pass straight through to the client. Whatever the engine does about them runs beside that stream and is merged back when it is ready, so the model’s speech never waits for it — the "hang on, let me check" a model says precisely because it delegated would otherwise be held back until the answer was found.

  • The user’s final transcript and every sentence the model says are saved as chat messages and passed to onChatMessage. The user’s transcript does not drive any planning; the model decides for itself when it needs help. How the model’s text is cut into sentences without a response end is described in What the Base Class Adds.

  • When the model asks for help, the delegation, which carries the text of what the user said, goes to onSessionDelegation if the interaction manager is a TurnlessDelegatingInteractionManager; the instructions it returns join the output stream as realtime-instruction commands. Otherwise the backend the model delegates to answers, see OpenAILiveRealtimeModel.

  • The model emits audio continuously, digital silence included. Silent frames are forwarded only until they add up to an eighth of a second; the frame that takes the count past it is marked as the last of its burst and the silent frames after it are dropped until the model speaks again.

  • There is no turn to persist, so the context is saved once, when the client disconnects. After that the session is finalized and onSessionDisconnect runs, as in the turn-based session end.

Chat Events

Not every event belongs to a running conversation: a time moment or an incoming webhook, say, may have to produce a message for the chat, a push notification or another channel. POST /api/pipeline/{key} accepts such a ChatEvent for a session without opening a model session. PipelineService builds the context, resolves the interaction manager and calls its onChatEvent, then saves the session in the IDLE state. The default does nothing and no interaction manager in the engine overrides it yet.

Interaction Manager Implementations

Manager Behaviour

RelationalAgentInteractionManager

The full turn-based implementation — overrides every hook. See Relational Management.

BasicAgentInteractionManager

Builds the system prompt from the agent’s identity, purpose and guardrails (with a legacy fallback when none are set), offers the basic, memory, multimodal, knowledge, MCP and web tool boxes, injects per-turn supporting context in turnEnd, and saves the session on disconnect. It does not override process, so no extractors run for basic agents. Planning is blank. It is also a TurnlessInteractionManager, so a basic agent can run on a turnless model: it then uses a prompt without the finish tool section, which a model with no tools of its own could not follow, and onChatMessage only logs. It does not take delegations, so a turnless model’s backend answers them.

TestAgentInteractionManager

The simulated user used by automated testing: builds a test prompt with role, vision and scenario sections (or the raw agent prompt when the connection sets noSystemPrompt), plus a memory section from previous session summaries. Offers device and memory tools. Aggregates the session on disconnect when the agent has memory.

BasicInteractionManager

Smoke-test manager for the realtime setup: a minimal prompt, no tools, and a single delayed command per turn.

PreviewInteractionManager

Voice preview for the realtime-preview setup: instructs the model to speak one short sentence in the requested locale and accent, then stop. No tools, no lifecycle hooks.