Engine Application

AI Layer

The abstractions — LLMFactory, LLMConfiguration, ToolBox, ToolSpecification — are defined in the common library. The engine provides their implementations: the chat models, the tool boxes agents can call, prompt assembly and invocation logging.

Language Model Wiring

SpringAiLLMService implements SpringAiLLMFactory and builds the Spring AI chat model for a component and a context. Only streaming use is supported — a conversation streams tokens as they are produced.

Supported providers:

Provider Notes

Provider	Notes
OpenAI	`OpenAiChatModelConfiguration`, and `OpenAiReasoningModelConfiguration` for reasoning models (fixed temperature, reasoning effort instead).
Azure OpenAI	`AzureOpenAiChatModelConfiguration` and `AzureOpenAiReasoningModelConfiguration`. Clients are built by `AzureOpenAiClientFactory`, which also works around a streaming issue where Azure content filter chunks arrive without a delta. Several deployments can be used round-robin through `AzureOpenAIRoundRobinSpringChatModel`, which spreads load and quota across them.
Google GenAI	`GoogleGenAiChatModelConfiguration` — Gemini models, with optional search grounding and thinking level.
Anthropic	`AnthropicChatModelConfiguration`.

OpenAI

OpenAiChatModelConfiguration, and OpenAiReasoningModelConfiguration for reasoning models (fixed temperature, reasoning effort instead).

Azure OpenAI

AzureOpenAiChatModelConfiguration and AzureOpenAiReasoningModelConfiguration. Clients are built by AzureOpenAiClientFactory, which also works around a streaming issue where Azure content filter chunks arrive without a delta. Several deployments can be used round-robin through AzureOpenAIRoundRobinSpringChatModel, which spreads load and quota across them.

Google GenAI

GoogleGenAiChatModelConfiguration — Gemini models, with optional search grounding and thinking level.

Anthropic

AnthropicChatModelConfiguration.

Triton-served models are reached separately, through the inference model abstraction rather than this factory.

Per-Component Configuration

Every place that calls a model is a component, identified by its class name, and configured under promethist.ai.llm-components.{ComponentName}:

promethist.ai.llm-components.FactualDataExtractor.provider=AzureOpenAi
promethist.ai.llm-components.FactualDataExtractor.azure-open-ai.deployment-name=gpt-4.1-mini
promethist.ai.llm-components.FactualDataExtractor.azure-open-ai.temperature=0.3
promethist.ai.llm-components.FactualDataExtractor.azure-open-ai.json-output=true

provider selects the provider; the nested block carries that provider’s options. This is how each extractor, judge, aggregator, planning manager and summarizer gets its own model and parameters. LLMConfigurationService resolves the effective configuration, including reasoning model detection and replay overrides, and an agent may carry its own LLMConfiguration that takes precedence.

Embeddings

EmbeddingModelFactory builds the reactive embedding model for the provider selected by promethist.embedding.defaultProvider — Triton (the default), OpenAI or Azure OpenAI — with the model name and vector dimension configured per provider. The scoped memories declared in MemoryConfiguration use it for indexing and search.

Tool Boxes

A tool box exposes callable tools to the model. ToolBoxService assembles the box for a turn from the agent, its project and the connected device; MultiToolBox merges the parts and prefixes tool names with their box name, so tool calls are routed back unambiguously.

Tool box Tools

Tool box	Tools
`MemoryToolBox`	Retrieval over the user’s session memory: similar past sessions, the last N session summaries, sessions in a time interval, full conversation transcripts, and transcripts and evaluations of agents that share memory with this one.
`KnowledgeToolBox`	Semantic search over the project’s knowledge base, with a search timeout guard.
`MultimodalToolBox`	Displays UI to the user — a web page, an image, an input prompt, choices or a rating — with link hallucination guarding applied to any URL the model produces.
`WebPageSearchToolBox`	Web search and page retrieval through the crawler service.
`McpToolBox`	Wraps one Model Context Protocol server connection as a tool box, proxying calls to the external server and closing the client when the session ends. Connections come from the project’s MCP connector configuration, with per-user OAuth tokens where the server requires them.
`HandoverToolBox`	Hands the conversation over to another agent, two tools per handover: `{ref}_search` finds the matching agents and returns only their ref, name, title and description, without handing over; `{ref}` then hands over to the chosen refs (one ref redirects, several show a picker), at most `count` of them. When the handover lists fixed agents, the search is optional and only describes them, and `{ref}` without refs offers all of them, always in the handover’s order. Refs outside the handover’s agents, or of unpublished agents, are rejected.
`RelationalToolBox`	A decorator rather than a box of its own: when the wrapped tool is already part of the current plan it returns an interim "results will follow" response instead of calling it again.

MemoryToolBox

Retrieval over the user’s session memory: similar past sessions, the last N session summaries, sessions in a time interval, full conversation transcripts, and transcripts and evaluations of agents that share memory with this one.

KnowledgeToolBox

Semantic search over the project’s knowledge base, with a search timeout guard.

MultimodalToolBox

Displays UI to the user — a web page, an image, an input prompt, choices or a rating — with link hallucination guarding applied to any URL the model produces.

WebPageSearchToolBox

Web search and page retrieval through the crawler service.

McpToolBox

Wraps one Model Context Protocol server connection as a tool box, proxying calls to the external server and closing the client when the session ends. Connections come from the project’s MCP connector configuration, with per-user OAuth tokens where the server requires them.

HandoverToolBox

Hands the conversation over to another agent, two tools per handover: {ref}_search finds the matching agents and returns only their ref, name, title and description, without handing over; {ref} then hands over to the chosen refs (one ref redirects, several show a picker), at most count of them. When the handover lists fixed agents, the search is optional and only describes them, and {ref} without refs offers all of them, always in the handover’s order. Refs outside the handover’s agents, or of unpublished agents, are rejected.

RelationalToolBox

A decorator rather than a box of its own: when the wrapped tool is already part of the current plan it returns an interim "results will follow" response instead of calling it again.

Device-dependent tools (DeviceCommonTools, DeviceWithAudioTools, DeviceWithoutAudioTools), skill invocation (SkillTools), agent handover (HandoverTools) and the realtime response model’s request for planning help (RequestHelpTools) are contributed as annotated tool objects — AnnotatedToolBox derives their specifications from the annotated functions by reflection, so a tool is added by writing a described function.

Tool availability is also serialized into the prompt, so the model knows what knowledge and tools exist before it calls anything.

Prompts

Element Role

Element	Role
`AgentPromptService`	Builds the prompts of an agent: the system prompt from its identity, purpose, guardrails and skills, and the prompts of the metric extractors. Records prompt invocations so they can be shown in the debugger.
`prompts/planning/PlanningPromptTemplateService`	Templates of the planning prompts.
`prompts/responseGeneration/*`	Response generation prompts, with an internal variant used by the internal realtime model.
`TextTemplateService`, `TextTemplateRenderer`	Named templates rendered with Pebble, with the context available as variables.
`src/main/resources/prompts/`	Prompt files shipped with the engine, including the judge prompts used to validate extraction and to compare replayed responses.

AgentPromptService

Builds the prompts of an agent: the system prompt from its identity, purpose, guardrails and skills, and the prompts of the metric extractors. Records prompt invocations so they can be shown in the debugger.

prompts/planning/PlanningPromptTemplateService

Templates of the planning prompts.

prompts/responseGeneration/*

Response generation prompts, with an internal variant used by the internal realtime model.

TextTemplateService, TextTemplateRenderer

Named templates rendered with Pebble, with the context available as variables.

src/main/resources/prompts/

Prompt files shipped with the engine, including the judge prompts used to validate extraction and to compare replayed responses.

The noSystemPrompt connection parameter suppresses the engine’s own system prompt so an agent’s instructions can be evaluated in isolation — this is what the testing setups use.

Invocation Logging

LLMLoggerService implements the LLMLogger contract and records every model invocation as an LLMLog document: the rendered prompt, the response, tool calls and their results, the model and its parameters, and token usage. Text is passed through SensitiveInformationMaskerService first.

These logs are what the session endpoints expose to managers and what the session debugger reads and replays.