Interaction Pipeline

Relational Management

A relational conversation is run by two models at once:

Realtime model

Speaks. It holds the identity, the guardrails and the opening plan, and it reacts to the user immediately. Its tool box is deliberately small.

Planner

Thinks. It owns the business process, retrieval (knowledge, memory, MCP, web) and strategy, and it never speaks — it steers the realtime model by injecting instructions into the live session.

That split is what keeps the conversation responsive while still following a process: the user always gets an immediate reply, and the planner’s conclusions arrive as steering for the following turns.

This page covers both the agent definition the feature is configured with and the components that execute it. The orchestration around them — session start, turn windowing, instruction injection — is documented in Realtime Conversation Lifecycle.

Components

Component Role

RelationalAgentInteractionManager

The TurnInteractionManager for relational agents. Owns session start, the system prompt, the realtime tool box, per-turn context injection and session teardown.

RelationalPlanningManager

The per-turn planner.

InitialPlanGenerator

Produces the opening plan once per session.

RelationalToolBox

The realtime model’s tool box, wrapped so that a call to a planner-owned tool degrades gracefully.

SessionAggregator, AnalyticalJudges

End-of-session processing.

The Agent Definition

A relational agent is a RelationalAgent document in the shared domain model — see Domain Model for where it sits among the other entities. Its fields exist to feed the two models described above, and it is worth knowing which model sees what.

Fields

RelationalAgent is a TemplatableAgent<RelationalAgentTemplate> and Evaluable, so it carries everything from Agent plus:

Field Default Meaning

identityDescription

null

Who the agent is — its persona. Goes into the realtime system prompt only; the planner never sees it.

purpose

null

What the agent is for, in business terms. Goes into both the realtime prompt and the planning prompts.

businessProcessSteps

null

The process the agent should walk the user through. Goes into the planning prompts only — the speaking model is never given the process, it is steered through it by the planner one instruction at a time.

initialPlan

null

A static opening plan, used as the fallback when initial plan generation fails and no previous plan is available.

guardrails

null

Hard rules. Injected into both the realtime prompt and the planning prompts.

businessImpact

1.0

Weights how much the planner should prioritize the business purpose over relationship building. It is bucketed for the prompt: below 0.33 is low, below 0.66 is medium, otherwise high. Low and medium enable the relational objectives in the planning prompt; high tells the planner to focus exclusively on the purpose.

sharedMemoryAgentRefs

empty

Refs of other agents whose session transcripts and evaluations this agent may read — see Shared Memory.

llmConfiguration

LLMConfiguration()

Which models do the planning — see below.

hasMemory, inherited from Agent, gates a great deal of relational behaviour: memory tools, the user information sections of the prompts and the end-of-session aggregation.

Planning Model Configuration

data class LLMConfiguration(
    val initialPlanning: ai.promethist.ai.LLMConfiguration? = null,
    val relationalPlanning: ai.promethist.ai.LLMConfiguration? = null,
)

initialPlanning configures the model that produces the opening plan, relationalPlanning the per-turn planner. Both fall back to a reasoning model default when unset.

This nested configuration does not configure the speaking model — that comes from the agent’s realtimeConfigurationId (see Realtime Models). And because the planning components pass their configuration explicitly, the agent’s llmConfiguration is the effective override point for planning; the promethist.ai.llm-components.* entries for the planning components only supply the component name used in the invocation log.

Templates

RelationalAgentTemplate lets a project define a reusable agent shape. Each templatable field is paired with a *Locked flag:

Template field Locked flag

purpose, businessProcessSteps, guardrails, identityDescription

purposeLocked, businessProcessStepsLocked, guardrailsLocked, identityDescriptionLocked

businessImpact, hasMemory, cameraPreset

businessImpactLocked, hasMemoryLocked, cameraPresetLocked

When a template is applied, a locked field takes the template’s value and an unlocked one keeps the agent’s own. Locking is therefore how a project enforces a field across all agents built from the template while leaving the rest editable.

Template variables are declared on the template (as free text or an enumeration) and their values are stored on the agent in templateVariables. Substitution replaces {name} placeholders, and it runs on the result of the lock decision — so variables are substituted into the agent’s own text as well as into locked template text. An unresolved placeholder is left as written.

The template is applied when the agent is loaded for a conversation; a pinned templateRevision selects an exact template revision, otherwise the published one is used.

Shared Memory

sharedMemoryAgentRefs lets several agents of a project pool what they know about a user, so a handover between them does not start from zero. The refs are resolved at context creation into a map of ref to agent name; a ref that cannot be resolved is warned about and skipped.

There is one implicit rule: when sharedMemoryAgentRefs is empty and the agent is the project’s featured agent, it shares with every other published agent in the project.

The resolved set has two effects:

  • the memory tool box exposes a shared-agent transcript and evaluation tool — and because memory tools belong to the planner, only the planner can call it;

  • the previous evaluation and the most recent shared-agent transcript are injected into the realtime prompt’s context payload and into the initial planning prompt.

The Plan

RelationalPlan is the planner’s own state, stored in data memory:

data class RelationalPlan(
    val description: String = "",   // the full planner output of the turn
    val currentModule: String = "", // where in the business process the conversation is
    val workingFocus: String = "",  // what is currently being worked on
    val gapType: String = "",       // knowledge | process | mindset
)

description is the raw planner output; the three structured fields are parsed out of the state object the planner emits last in its output, and degrade to empty strings if that parse fails. There are no plan steps — the process lives in businessProcessSteps and the plan records where the conversation currently is within it.

The same type is stored under two different memory scopes, which is what makes the two plans behave differently:

Plan Scope Lifetime

Initial plan

Application

Per user and agent, surviving across sessions. Generated at session start, reused on reconnect, and used as the fallback when generation fails.

Per-turn plan

Session

Rewritten by the planner at the end of every turn and read back as the previous output on the next turn. Each write appends a revision, so the plan’s evolution through a conversation is preserved.

Session Start

onSessionStart runs before the provider session exists, in two phases.

First, four initialization tasks run concurrently, each timed and each with its errors swallowed so that no single failure prevents the conversation from starting:

  • the pre-session webhook is executed and its result stored for the planning prompts;

  • user facts are refreshed;

  • the previous session’s summary is loaded;

  • the URLs of the multimodal tool specifications are seeded into the link hallucination guard, so links the model later produces can be checked against something.

All but the last are conditional on hasMemory.

Then the opening plan is generated by InitialPlanGenerator and stored in application-scoped data memory. Two cases skip generation:

  • Reconnect — when the session already has more than one turn, generation is skipped entirely and the stored plan is reused;

  • Failure — the previous session’s plan is used when it is non-empty and the agent has memory, otherwise the agent’s static initialPlan is used with project mentions substituted.

Because the whole session waits on this phase, initial planning is the dominant cost of the first response.

The System Prompt

getSystemPrompt assembles the realtime prompt from the agent, the user and the resolved initial plan. Which template is used depends on the model: the internal realtime model gets its own variant, every provider model gets the standard one.

What the speaking model receives: the agent’s name, purpose, identity, language and gender, the user’s name and gender, the profile biography (when memory is enabled), the guardrails, the names of the available knowledge sources, MCP tools and multimodal elements, and the initial plan description.

What it does not receive: the business process steps, the per-turn plan, skill content, or any raw tool output. Knowledge and MCP are listed only so the model knows they exist — the planner is what actually queries them.

The Realtime Tool Box

getToolBox gives the realtime model only what it needs to act immediately: the basic tools (finishing the conversation, handover when the project defines it), the multimodal display tool, and the request_help tool.

Everything heavier belongs to the planner. To stop the speaking model from telling the user it cannot help when it tries a tool it does not have, the box is wrapped in RelationalToolBox, which:

  • passes specifications through unchanged — it advertises nothing extra;

  • detects a "tool does not exist" result (or the equivalent exception) from the wrapped box;

  • records a tool call hint with the attempted tool name and parameters into session-scoped memory;

  • answers the model with Tool <name> will return results in a few seconds. when the name matches one of the planner’s tools, and returns the original error otherwise.

The hint closes the loop: the next planning turn reads it, renders it into the planning prompt as a tool call hint, and clears it. The planner is told the attempt failed and that it should call the tool itself and deliver the outcome — which is what makes the promise the model just made to the user come true, one planning round later.

request_help is the explicit counterpart to that accidental path. When the model knows it is missing a fact, number, exact wording, threshold, lookup or external result, it calls request_help with a concrete question instead of reaching for an unrelated tool. The tool records a HelpRequest into session-scoped memory and answers the model with an instruction to tell the user it is checking and to wait. The same planning turn reads the request, renders it into the planning prompt, resolves it with the planner’s own tools (knowledge, MCP, memory) and clears it.

Session End Approved by the Planning

Relational agents on a turn-based OpenAI realtime model or the internal realtime model end the session through the planning. Basic agents (no planning) and other realtime models, such as GPT-Live whose tools are delegated or Gemini Live, keep the original behaviour: the realtime model says goodbye and calls finish_session, and the session ends after the goodbye audio. There is no setting; the engine derives the mode from the agent type, the agent’s session end support and the realtime model (Context.sessionEndMode in Extensions.kt). An agent with session end support disabled never ends the session through the planning either.

Through the planning, the planning decides when the session ends, because the planning of a turn finishes only seconds after the realtime model’s reply and would otherwise lose anything the agent should still say or ask. The realtime model sees the same finish_session tool, but in this mode FinishTools requests approval instead of ending it:

  1. Request: the call records a SessionEndRequest in session-scoped data memory and answers the model with a localized instruction to say only one or two neutral sentences (for example „Rozumím. Dej mi chvilku, jen si projdu, jestli jsme probrali všechno, co jsem pro dnešek měl v plánu.“) and wait. The session stays open; the realtime model starts no further response after that sentence.

  2. Decision: the planning of that turn renders an End request section into its prompt and decides, from its whole context, whether to end now or first ask or tell the user something. Either way it outputs results with what the agent should say; to end, the same JSON object also carries "endSession": true. The engine then stores a SessionEndApproval and marks the goodbye instruction for runtime-owned session termination after the response completes. Only the first nonblank, successfully persisted instruction of a planning pass answers the request. The flag is ignored on supportingContext. Without a request the planning may also end the session itself (its prompt then has an Ending the session section), with the same flag and flow.

  3. Goodbye: the runtime ends the session after the approved goodbye response completes. If the user starts speaking while OpenAI is generating the goodbye, the pending end is cancelled and the session remains open for the new input. The end instruction is supported for relational agents on turn-based OpenAI realtime and the internal realtime model; basic agents, turnless OpenAI/GPT-Live, and other realtime providers retain their existing behavior. The model is told not to call finish_session for this goodbye. The stored approval supports a model that still calls it before speaking: the tool reminds it to say the approved goodbye instead of requesting planning again. An unused approval is cleared when the next planning pass starts. If an approved OpenAI response already spoke the goodbye and also called finish_session, it ends without requesting another response. Both realtime models emit sessionShouldEnd before ResponseEnd, so the command stays in the goodbye turn and precedes that turn’s exit, as required by Twilio. OpenAI response and interruption decisions are isolated in PlannedSessionEnd.

Like a pending request_help question, the end request is offered to one planning pass only: the pass clears it as soon as it reads it. When the user says something new before that pass answers, the new input cancels the pass, and the next pass treats the new input as an ordinary turn without the End request section. The realtime model then receives a localized system note right before the new user message (FinishTools.endRequestSupersededInstruction, tracked by PlannedSessionEnd in both realtime models): the conversation has not ended, react normally to what the user just said and do not call finish_session; if the user only confirms again that they want to stop, call it and say only a short neutral reaction. A pass that answers nothing — it emits no results, fails or times out, including during prompt or tool preparation — ends with a localized fallback instruction to say a short goodbye with endSession. Blank results do not answer a request. If persisting the fallback fails, it is sent directly to realtime. The per-pass decisions live in SessionEndGate.

The realtime prompts switch their Finish tools rules through the session_end_mode template variable and are unchanged for the other agents.

Per-Turn Planning

RelationalPlanningManager.process runs in parallel with extraction, after the realtime model has already answered.

What Goes Into the Prompt

The planning prompt (planning.relational) is assembled by PlanningPromptTemplateService from:

Group Content

Business objectives

purpose, the business process steps (with exact-formulation markers materialized), the bucketed business impact and guardrails.

Tools

The planner’s callable tools, plus the descriptions of the UI tools it may not call, plus whether memory and shared agents are available.

Skills

The project’s skills, the active ones and their resources.

User and memory

Gender, biography, the pre-session webhook result and the time elapsed since the last visit.

Transcript

The last 30 messages plus the utterance that just arrived, and the multimodal elements already shown.

Previous plan

The description of the session-scoped RelationalPlan from the previous turn.

Steering

The current date and time, sensory information, the per-turn evaluation insights, any tool call hint, any pending request_help question and any pending session end request.

The planner’s own tool box is knowledge, memory (when enabled), skills, MCP and web page retrieval.

What Comes Out

The planner streams a mixture of reasoning prose and JSON objects. The stream is split as it arrives:

  • prose between objects becomes realtime-planning commands — observable, but not acted on;

  • a JSON object becomes a RealtimeInstruction, with the property name deciding its type: results produces a non-contextual INSTRUCTION, supportingContext produces SUPPORTING_CONTEXT;

  • every turn ends with a fixed contextual short-reaction instruction, which tells the model to react only from information it actually has.

This distinction is the latency design of the whole feature: results forces the model to say something more in the same turn and is deliberately discouraged by the prompt, while supportingContext silently updates the model’s working context for what comes next.

Before an instruction is injected it is citation-materialized, given a header ( Instruction: or Supporting context:), has its URLs registered with the link hallucination guard, and is appended to memory as an InstructionMessage — so it shapes later prompts but is filtered out of the plain conversation transcript.

Timing and Failure

  • Planning is not awaited before the user hears a reply.

  • It is cancelled on barge-in. A user who speaks again quickly can leave the plan un-updated for that turn.

  • The plan is written last, after all instructions have been emitted.

  • A planning timeout is silent: the turn produces no instructions and the plan is not updated, so the next turn plans against the older plan.

  • The next turn cannot begin until planning finishes or is cancelled.

  • Planning runs on the engine’s bounded elastic scheduler because tool calling in the underlying framework is blocking.

Initial Planning

InitialPlanGenerator uses its own template (planning.initial) and differs from the per-turn planner in what it can see and do:

  • no transcript, no previous plan, no tool call hints, no evaluation steering and no sensory data;

  • instead the last session’s summary, the previous evaluation and the most recent shared-agent transcript;

  • its tool box has no web page retrieval, and it runs without a tool handler, so no tool lifecycle commands reach the client during initial planning;

  • the output is collected in full rather than streamed, and the results value is extracted from it as the plan description.

The result is rendered into the realtime prompt, where the non-reconnect template instructs the model to open the conversation according to it. On reconnect, the template tells the model to continue from the chat history instead.

Turn End

turnEnd injects a supporting-context instruction carrying the current date and time plus the session’s sensory information, then triggers per-turn evaluation. Those evaluation insights become the next turn’s steering — which is how an evaluation result influences the conversation while it is still running.

Session Disconnect

onSessionDisconnect runs three legs, with failures logged rather than propagated:

  • clearing the cached link URLs;

  • the analytical judges;

  • end-of-session extraction, then SessionAggregator — strictly in that order.

SessionAggregator condenses the session’s extractor records into session data — averaged metrics, boolean flags, topic aggregations and expression histogram — plus an LLM-generated summary, stored in the session memory index and as an application-scoped insight. It then hands over to UserMemoryAggregator, which folds the session into the single per-user record: a re-summarized profile, cumulative session, turn and time counters, and every numeric trait updated as a weighted average.

Note what is not aggregated: the per-turn plan simply stops being read at the end of a session, while the initial plan persists and is reused next time.

One Turn, End to End

  1. The realtime model streams transcripts, response tokens, audio and possibly multimodal interactions, ending with ResponseEnd.

  2. If it calls a planner-owned tool, RelationalToolBox answers with the "results shortly" message and records a hint; if it calls request_help, the question is recorded for planning the same way.

  3. Once both the final transcript and ResponseEnd exist, the turn is recorded into memory and three legs start: tool events, extraction and planning.

  4. Planning builds its prompt, streams the planner, and injects each resulting instruction into the live session; contextual instructions land silently, a hard instruction blocks until the model’s next ResponseEnd and then reports completion.

  5. Both legs complete (or are cancelled by barge-in), the plan is persisted, and the next turn’s context is created.

  6. The turn is saved, turnEnd injects date, time and sensory context and starts per-turn evaluation.