AI Interfaces

AI Assistant

The AI Assistant (the copilot, /dashboard/aiassistant) is a second user interface of the Studio: a chat with a language model that can do what the user could do by hand — build and edit agents, add knowledge and integrations, design evaluations and test suites, explain analytics — while the panels next to the chat show and edit the same objects. The backend runs the model’s tool loop; the browser renders what the backend tells it to.

Connection

The assistant talks over Socket.IO, not over the REST API:

Aspect Value

Server

netty-socketio inside the backend (SocketIOConfiguration), port 9311, started by SocketIoServeLifeCycle when promethist.socket-io.enabled is on. In deployments the ingress routes /api/socket and /api/socket.io to it.

Path and namespace

Socket.IO path /api/socket/io, namespace /api/socket/io/chat. The Studio connects with the WebSocket transport only.

Authentication

The Keycloak access token in the handshake’s auth.token, validated once at connect time (The AI Assistant Connection). The Studio reconnects whenever its access token changes.

Messages

Every server event is emitted on the Socket.IO event message as JSON {"type": "...", "payload": {...}}. The client emits each event under its type name, for example user.message. Types are listed in dto/chat/EventType.kt (enum names with _ turned into .) and mapped to classes in dto/chat/Event.kt.

On connect the backend resolves the user, sends the latest 50 messages of the user’s chat history as a history event, and clears approvals of dangerous actions left from an earlier connection. SocketIOHandler processes each connection’s events in order with the user’s identity in the reactive security context.

Events

Direction Events

Client → server

user.message (text and optional file attachments, Base64), cancel.message, confirmation.response, navigation events (navigate.*) whenever the user moves in the UI, context.restore after a reconnect, seek.history for older history, and notifications of what the user changed by hand (agent.field.update, knowledge.created, evaluation.updated, test.suite.started, …​).

Server → client

response.stream.text (streamed reply chunks, with a final complete message), response.text, response.function.call (a tool step with its status: running, completed, failed or canceled), render.* (open a view or an object in the side panel), context.set, loading.start/loading.end, confirmation.request/confirmation.cancel, agent.ai.updated (drives the edit animation), history, error and session.expired.

The UI-driving render.* events are what make the panel follow the conversation: when the model opens an agent, the backend emits render.agent.detail and the Studio shows it.

A Turn

ClientEventHandler dispatches each incoming event; a user.message goes to PlatformAssistantService.chat:

  1. Prepare — a user with no history enters the onboarding (REGISTRATION) state; otherwise the message is appended to the model memory. Before it, the backend adds the current date and the user’s session state (open account, project, agent, page) when they changed, and folds in a summary of what the user did by hand since the previous message. Attached files are parsed with the knowledge parsers and added as text.

  2. Prompt — the system messages are built for this turn: the general assistant prompt, the package-specific guidance of the open project, the section prompt of the page the user is on, the user’s name and email, and a snapshot of the object open on the page.

  3. Tools — the tool set is assembled per turn: NavigationTools, CoreManagementTools and AdditionalTools always, plus the tools of the current section (for example KnowledgeTools on the knowledge pages, EvaluationTools on the evaluation pages, TestSuiteTools in the Test section), account and project creation when enabled by configuration, and the test-agent tools for staff.

  4. Loop — LLMService.reactiveCallLLM streams a chat completion with these tools, executes the tool calls the model makes, feeds the results back and continues until the model answers without a tool call. Text is streamed to the client as it arrives.

Navigation state lives in SessionStateService, in memory, keyed by user. NavigationStatus enumerates the places the assistant knows — from the account list down to a single evaluation or test suite — and the map navigationContexts in PlatformAssistantService assigns each a prompt and a tool set.

Tool Loop Safeguards

  • A call identical to the ones right before it in the same turn (same tool, same arguments) is no longer executed once it has been repeated three times; the model is told so, and if it insists the turn ends with an explanation.

  • A turn that ends with a failed step it did not explain, or with no reply at all, gets a fixed reply so the user is never left with an empty bubble (ToolFailureTracker, the replies in LLMService).

  • Optional tool parameters are presented to the model as optional (llm/AssistantToolCallbacks.kt), so it does not fill them with empty values that the edit tools would then store.

  • The stream fails after 180 seconds without a chunk — enough for the long reasoning phase before the first token.

  • Tool references generated into agent fields (@knowledge:...;) are repaired when the model drops the closing semicolon.

The Model

LLMService uses Spring AI’s OpenAiChatModel on top of an openai-java client configured for Azure OpenAI (llm/AzureOpenAiClientFactory.kt, promethist.ai.azure-open-ai.*):

Setting Value

CHAT_MODEL

gpt-5.5 — the Azure deployment name. Newer models refuse function tools combined with reasoning on the chat-completions API this loop is built on.

Reasoning effort

xhigh for the conversation, high for nested calls a tool makes (generating an agent, reviewing quality), which the user waits through.

Verbosity

low for the conversation only, so generated agent content keeps its length.

These are constants in LLMService, not configuration properties.

Memory and History

Two stores serve two purposes:

Store Collection Content

Model memory

chat-messages

What the model sees: a window of up to 100 messages per user (SmartReactiveMessageWindowChatMemory). Older messages are summarized by a language model instead of being dropped, and for a few tools — getAgentDetail, createAgent, editAgent — only the latest result is kept. When a request exceeds the model’s context length, the memory is summarized and the turn retried once.

Chat history

agenticChatMessage

What the user sees: user, assistant and selection messages with a running number, sent on connect and paged with seek.history.

Staff can send RESET to clear their model memory.

Confirmations

Tools that delete, revert, clone or promote run through DangerousToolRunner. Before the action the backend emits confirmation.request (title, warning, the target object) and waits:

  • The user answers with confirmation.response carrying approved and a scope: ONCE, SESSION (the same action on the same kind of resource is approved for the rest of the connection) or ALL_TASKS (every dangerous action). Deleting an output data element, deleting a group and deleting a whole evaluation are three kinds: the first two edit a Draft, the last removes every revision.

  • Only the connection that received the request can resolve it, and an unanswered request is declined after five minutes.

  • Cancelling the turn or disconnecting declines every pending request.

  • A declined action tells the model it was not performed; the tool card shows "canceled".

Cancelling and Interrupting

cancel.message disposes the running turn, dismisses its open confirmations and ends the loading state. A new user.message while a turn runs replaces it; in both cases the model is told on the next turn that its previous answer was cut off. The Studio’s composer holds new messages back until a turn ends, so interruption is the exception.

Reconnecting

Session state lives in the memory of the backend process that holds the connection. When the connection drops — a redeploy, a reconnect that lands on another replica — the Studio replays its context with context.restore (account, project, agent, open object and page). The backend re-applies it and answers context.set, without starting a turn. Events that arrive before the user is resolved wait for it instead of being dropped.

Streaming

With promethist.assistant.streaming.enabled (the default), reply chunks are batched — at most 50 ms or 64 chunks per event — and every assistant message gets its own stream id, so text after a tool call starts a new bubble instead of replacing the previous one.

Voice Input, Web Search and Documentation

Feature Implementation

Dictation

POST /api/v1/assistant/transcribe (TranscriptionService): the recorded audio (at most 25 MB) is transcribed by OpenAI’s gpt-transcribe by default. The prompt carries a vocabulary of product terms, the names of the open project’s agents and the end of the text written so far, which keeps the transcription in the speaker’s language (promethist.ai.transcription.*).

Web search

The webSearch tool makes one call to the Azure OpenAI Responses API with the provider’s web search and returns a cited report (WebSearchService, promethist.ai.web-search.*, five-minute timeout).

Documentation and website

The documentation and promethistWebsite tools search https://docs.promethist.ai and https://promethist.ai, so whatever the public documentation site publishes becomes part of what the assistant can answer.

Adding a Tool

  1. Add a method annotated with Spring AI’s @Tool to the tool class of its area in component/, with @ToolParam descriptions. A nullable Kotlin parameter becomes optional for the model.

  2. Take a ToolContext parameter when the tool needs the caller: it carries the authentication, the user id and the Socket.IO session id. Calls into services run with the caller’s identity, so @PreAuthorize applies as for the REST API.

  3. Route destructive operations through DangerousToolRunner.

  4. Emit render.* events through ChatEventService when the result should open in the panel, and keep the section prompt in llm/constants/Prompts.kt consistent with what the tool does.

  5. If the tool belongs to a new section, register its prompt and tools in navigationContexts.

The hosted MCP server exposes similar capabilities through a separate set of tools (Hosted MCP Server and Claude Plugin); a behavior change usually belongs in both.