Engine Application

Observability

Actuator Endpoints

management.endpoints.web.exposure.include=health,prometheus,metrics exposes:

Endpoint Use

/actuator/health/liveness, /actuator/health/readiness

The probes the Kubernetes deployment uses — see the engine chart. The Elasticsearch health indicator follows promethist.indexer.enabled.

/actuator/prometheus

Metrics in Prometheus format. Scraped by the ServiceMonitor the chart renders.

/actuator/metrics

Individual meters, for ad-hoc inspection.

All metrics carry an application tag with the application name.

Metrics

Meters are declared in engine/metrics/ and tagged per pipeline (withPipelineTags adds the pipeline key, locale, device type and similar dimensions), so they can be sliced per agent or channel.

PipelineMetrics:

Meter Meaning

active_sessions, active_websocket_connections

Gauges of currently open conversations, backed by the transport handlers' session maps.

sessions, turns, users, devices

Counters of created sessions, turns, users and devices.

input_elements, output_elements, output_errors

Element throughput, tagged additionally by element type, and error count.

binary_samples, speech_item_samples

Audio volume in and out, in samples.

realtime_input_elements, realtime_output_elements

Element throughput between the engine and the realtime model.

a2x_frames, a2x_samples

Animation data produced for avatar rendering.

first_text_response_since_last_interim_transcript, last_text_response_since_last_interim_transcript, first_audio_response_since_last_interim_transcript, last_audio_response_since_last_interim_transcript

Response latency measured from the last interim transcript — how quickly the agent reacts to speech still in progress.

first_text_response_since_final_transcript, last_text_response_since_final_transcript, first_audio_response_since_final_transcript, last_audio_response_since_final_transcript

The same latencies measured from the final transcript.

planning_processing, initial_planning_processing, preplanning_initialization_processing, interaction_processing, planning_instructions

Timing of the planning and interaction phases of a turn.

LLMMetrics counts input_tokens, output_tokens and total_tokens; NettyMemoryMetrics reports the Netty allocator’s memory use, which is the resource that matters most when many audio streams are open.

The latency timers are recorded from the per-turn timing log when a turn is saved, so they reflect what the user experienced rather than internal call durations.

Tracing

The engine is instrumented with the OpenTelemetry Java agent, attached through JAVA_TOOL_OPTIONS and pointed at a collector with OTEL_EXPORTER_OTLP_ENDPOINT. In a deployment the chart sets both, along with the service name, resource attributes and logback MDC capture, when app.serviceMonitor is enabled. Locally, see Telemetry.

Reactive chains are annotated with Reactor checkpoint() calls at the pipeline boundaries (Pipeline/{sessionId}, SessionFetch/{sessionId}, PipelineEgress/{sessionId} and the processor chain), which is what makes a stack trace from deep inside a reactive operator attributable to a session. Reactor scheduler metrics are enabled at startup.

Two logging switches exist for expensive diagnostics and are off by default — logging.level.r for reactive chain logging and logging.level.SpeechRecognizer for recognition logging. Neither should be enabled in production.

Log statements carry structured attributes, most importantly session.id, so a conversation can be followed across components.

Provider Health Checks

Beyond the Actuator health of the application itself, the engine can probe its external providers: /api/health/textSpeechSynthesizer/, /api/health/speechRecognizer/, /api/health/realtimeModel/ and /api/health/llm/ (see Internal Controllers). Each checker performs a minimal real request against every configured component, or against one named component, and reports a HealthCheckResult per provider — which is how a provider outage is distinguished from an engine problem.

The LLM checker works per model provider rather than per component: OpenAi, OpenAiReasoning, AzureOpenAi, AzureOpenAiReasoning, Anthropic and GoogleGenAi, each with its default promethist.ai.llm.* configuration and that configuration’s timeout. A probe succeeds on the first non-blank streamed text. It calls the provider model directly instead of through the wrapper of SpringAiLLMService, so probes produce no LLM logs.

Usage Records

EDRService writes event detail records (EDR documents) capturing active users and active conversation duration. They are the basis for usage metering and, together with SubscriptionService, for enforcing a tenant’s seat and minute allowance.

LLM Logs

Every model invocation is stored as an LLMLog document by LLMLoggerService: prompt, response, tool calls, model parameters and token usage, with sensitive information masked. Each log carries the sessionId, turnId, and the session’s projectRef and agentRef. Logs expire after 30 days. They are readable per session through /api/session/{sessionId}/log/llm and are what the debugger works with.

Session Debugger

The debugger is an internal tool for inspecting and re-running what a conversation actually did. Its API is under /api/debug and it is backed by:

Service Role

LLMDebuggerSessionService

Transforms the raw LLM logs of a session into a turn and component timeline.

LLMDebuggerReplayService, LLMDebuggerStandardReplayOrchestrator

Re-run a recorded invocation against a model, optionally with a different configuration, so a change can be evaluated against a real conversation.

LLMDebuggerReplayToolCallbackFactory

Supplies tool callbacks during replay, so recorded tool calls are simulated rather than executed again.

LLMDebuggerOpenAIRealtimeReplayService

Replays recorded realtime sessions against a realtime provider.

LLMDebuggerRealtimeReplayJudgeService

Judges a replayed response against the original with a judge prompt, which is how regressions are detected in bulk.

InternalRealtimeCompanionMatcher

Pairs the pipeline manager’s log entries with the internal realtime model’s entries, using several anchors rather than timestamps alone.

LLMDebuggerAuthorizationService, LLMDebuggerAuthConfigurationService

Authorize access against the session’s project ACL, and provide the front end’s OAuth bootstrap configuration (promethist.debugger.auth.*).

The debugger front end is served at /agent-debugger/index.html and has two modes: Sessions, the timeline and replay of one session, and Tool Calls, described below.

Tool Call Analysis

The Tool Calls mode shows how the agents of a project call their tools, so agent designers can judge whether tools, in particular handovers, are called when they should be. It starts by opening a project: GET /api/debug/projects/{projectRef} confirms the project exists and that the caller may debug it. The open project and section are kept in the URL, as ?view=tool-calls&tab=criteria|search&projectRef={projectRef}, so a link reopens the same view. The project has two sections:

  • Criteria — the project’s criteria, with the state of their revisions, and editing, publishing, discarding a draft and deleting them.

  • Search — the statistics. The scope is the sessions of the project started within a period, optionally narrowed to an agent and revision, to LLM components and to tools. Component and tool filters take comma-separated names in which matches any characters, so handover_ selects every tool of the handover tool box. Test-mode sessions are excluded unless requested, and at most 5,000 newest sessions are analysed at once.

LLMDebuggerToolCallService selects the sessions on session and reads their llmLog documents through an aggregation that keeps only the offered tool names (tools[].name) and the tool trace (toolExecutionTrace), with arguments and responses cut to their first 2,000 characters. The unit of analysis is a turn and a tool: a tool is offered in a turn if any log of the turn offered it, and called if any log of the turn called it. A log without a turn id forms a turn of its own.

Reading the logs is the expensive part, so the sessions and log digests of a project and period (with its agent, revision and test-mode choice) are kept in memory for five minutes: drilling down, paging, filtering by component or tool and classifying reuse them, while Show reads them again (refresh=true). The logs are found through the sessionId_created index of llmLog.

For every tool the statistics give the number of calls, the sessions it was called in (and the share of all sessions in scope), the turns it was called in out of all turns of those sessions, failed calls, loops and, when criteria exist, the relevance verdicts.

  • Outcome of each call is derived by ToolCallOutcome: UnknownTool when the tool box answered that no such tool exists or the name was not offered, Failed when the call threw or returned a known failure response (a missing skill, an invalid URL), Empty for an empty result, otherwise Succeeded. Only Failed and UnknownTool count as failures.

  • Loop is a turn in which the same tool was called three or more times.

  • Relevance compares whether the tool was called with whether it should have been, judged against ToolCallCriteria: true positive, false positive (called but should not have been), false negative (should have been called but was not) and true negative. The false positive rate is computed over evaluated calls, the false negative rate over evaluated turns that needed the tool.

Every number opens the turns behind it, each with the user input, the calls and their outcome, and the judge’s verdict and reasoning; a turn opens in the Sessions mode on the component that called the tool.

Criteria and Classification

ToolCallCriteria is the specification of when a tool should and should not be called, for a tool name or pattern in a project, optionally narrowed to one agent. Criteria are versioned by ref and revision and edited through a draft, managed by ToolCallCriteriaService:

  • saving writes the single Draft revision, created one above the latest revision and overwritten by later saves (POST /api/debug/tool-call-criteria);

  • publishing turns the draft into the one Published revision and archives the previous one (POST /api/debug/tool-call-criteria/{ref}/publish); only the published revision is used for classification, so a draft takes effect once it is published;

  • discarding drops the draft (DELETE /api/debug/tool-call-criteria/{ref}/draft); criteria that were never published disappear with it;

  • deleting drops the draft and adds an Archived revision on top (DELETE /api/debug/tool-call-criteria/{ref}), which removes the criteria from the project and from classification while keeping their history and stored classifications.

Each criteria has a call scope:

  • Per turn (the default) judges every turn on its own: should the tool have been called in response to that turn’s user input?

  • Per session judges the whole session at once: should the tool have been called at least once in the conversation? It suits tools that have to happen once at the right moment, such as a handover. The unit is a session and a tool — offered if any of its turns offered the tool, called if it was called at least once — and the judge reads the session transcript from the turn collection (the user input and the agent response of every turn, as the platform’s other conversation-level judgements read it). A session that is still open is not judged until it ends.

When several published criteria of the same scope match a tool, criteria narrowed to the session’s agent win over project-wide ones, an exact name wins over a pattern, and a longer pattern wins over a shorter one. Each scope is resolved on its own, so a tool can be judged per turn and per session at the same time, and sessions of different agents can be judged by different criteria. The statistics therefore show one line per criteria in the false positive, false negative and evaluated columns, as X (n.n %) of Y classified turns or … classified sessions — Y being every turn (session) classified against that criteria — each naming its criteria and linking to it; the drill-down of a line lists the turns, or the sessions, that criteria judged. Under each false positive and false negative line, the count is split by the judge’s confidence (high X (n.n %) · medium … · low …), each percentage of all units classified against the criteria.

Classification is run on demand. ToolCallClassificator is the contract: it receives one turn, or one whole session — the conversation, the offered tools and the calls — together with the criteria of each tool, and returns a verdict per tool. It does not depend on llmLog, so it can later run inside the pipeline on a live turn. LLMToolCallClassificator implements it with an LLM judge: all tools of a turn are decided in one call, the judge only decides whether each tool should have been called, and the verdict follows from that decision and whether the tool was called. The judge model is set by promethist.debugger.tool-call-classification.model (gpt-5.1); it is engine configuration and the same for all criteria, so verdicts are comparable across tools and projects. Judge calls are not written to llmLog.

pipeline.classificator.tool-call selects the implementation: llm (the default) is the LLM judge above, jev is JevToolCallClassificator, backed by TypeSafe AI Jev, a hosted model that answers typed questions instead of generating text and is much faster and cheaper per turn. It needs spring.ai.typesafe.api-key. Each turn is one Jev call whose state is the turn’s conversation, ending at its last user message (a session’s is its whole conversation), and whose questions are one yes/no question per tool, with the criteria’s should call and should not call as the conditions for yes and no. The question carries the same rules as the LLM judge: should not call takes precedence when both apply, and a situation the criteria do not cover warrants the call only when the need is unambiguous. The answer, a value between 0 and 1, decides should call against pipeline.classificator.jev.threshold (0.5, strictly between 0 and 1), and its distance from the threshold gives the confidence. Jev returns no explanation, so the stored reasoning states the decision and the score.

Each verdict is stored as a ToolCallClassification for the turn, the tool and the exact criteria revision, which acts as a cache: a turn already classified against the current revision is not judged again, and a new revision leaves older verdicts stale but stored. Delete Classification on a tool removes the verdicts its row counts — those of the sessions in the current scope, against the published criteria revisions — so they are judged again (POST /api/debug/tool-calls/classifications/delete). A run classifies at most 5,000 pending turns, newest first.

The results of an older revision stay viewable: Show results on a revision in the criteria history recomputes the statistics and the drill-down from the verdicts stored for that revision (filter parameter criteriaRevision={criteriaRef}:{revision}, which puts that revision in place of the published one; drafts have no results). The view is read-only — classifying and deleting are refused (409) — and Back to current returns to the published revisions.