Project Overview

Technology Stack

Languages and Build

Technology Version Role

Technology	Version	Role
JDK	21	Toolchain and JVM target for every module. The engine container image is based on Temurin 21.
Kotlin	2.4	Implementation language of all modules.
Gradle	9.6 (wrapper)	Build tool. Always build through `./gradlew` so the pinned wrapper version is used. Dependency versions are centralized in `gradle/libs.versions.toml`; shared build logic lives in `buildSrc/`.
KtLint	1.8	Formatting and lint, wired as a pre-commit hook — see Code Conventions.

JDK

Toolchain and JVM target for every module. The engine container image is based on Temurin 21.

Kotlin

2.4

Implementation language of all modules.

Gradle

9.6 (wrapper)

Build tool. Always build through ./gradlew so the pinned wrapper version is used. Dependency versions are centralized in gradle/libs.versions.toml; shared build logic lives in buildSrc/.

KtLint

1.8

Formatting and lint, wired as a pre-commit hook — see Code Conventions.

Frameworks

Technology Version Role

Technology	Version	Role
Spring Boot	4.1	Application framework for the engine, bridge and relay. The engine and relay use the reactive stack; the bridge runs as a servlet application because it drives blocking WebDriver sessions.
Spring WebFlux / Project Reactor	3.8	The engine is reactive end to end — a conversation is a `Flux` of elements from the client transformed into a `Flux` of output elements.
Spring Security	via Spring Boot	OAuth2 resource server (JWT), plus API key and Basic authentication filters.
Spring Data (reactive)	via Spring Boot	Repositories for MongoDB and Elasticsearch.
Spring AI	2.0	Chat model abstraction over the language model providers, plus MCP client support.
Jackson	3.1	JSON serialization. A second, MessagePack-backed mapper provides the binary wire format.
Netty Socket.IO	2.0	Socket.IO server used by the engine (port 9011) and the relay (port 9012).
gRPC / Protobuf	1.80 / 4.34	Client for Triton Inference Server (embeddings and self-hosted models).
JavaCV / FFmpeg	1.5 / 7.1	Audio and video processing: WebP encoding of visuals, session video recording, media remuxing. The engine image also carries a native `ffmpeg` binary for recording conversion.
Appium / Selenium	10.1	Used by the bridge to drive real mobile devices and browsers — see Bridge.
Kotest / JUnit 5 / MockK / Testcontainers	—	Test stack.
Springdoc	3.0	Generates the OpenAPI document served at `/api/schema.json`.

Spring Boot

4.1

Application framework for the engine, bridge and relay. The engine and relay use the reactive stack; the bridge runs as a servlet application because it drives blocking WebDriver sessions.

Spring WebFlux / Project Reactor

3.8

The engine is reactive end to end — a conversation is a Flux of elements from the client transformed into a Flux of output elements.

Spring Security

via Spring Boot

OAuth2 resource server (JWT), plus API key and Basic authentication filters.

Spring Data (reactive)

via Spring Boot

Repositories for MongoDB and Elasticsearch.

Spring AI

2.0

Chat model abstraction over the language model providers, plus MCP client support.

Jackson

3.1

JSON serialization. A second, MessagePack-backed mapper provides the binary wire format.

Netty Socket.IO

2.0

Socket.IO server used by the engine (port 9011) and the relay (port 9012).

gRPC / Protobuf

1.80 / 4.34

Client for Triton Inference Server (embeddings and self-hosted models).

JavaCV / FFmpeg

1.5 / 7.1

Audio and video processing: WebP encoding of visuals, session video recording, media remuxing. The engine image also carries a native ffmpeg binary for recording conversion.

Appium / Selenium

10.1

Used by the bridge to drive real mobile devices and browsers — see Bridge.

Kotest / JUnit 5 / MockK / Testcontainers

—

Test stack.

Springdoc

3.0

Generates the OpenAPI document served at /api/schema.json.

External Services

The engine needs the following services at runtime. In development they are provided by the local infrastructure stack (see Development Setup); in a deployment they are managed services or in-cluster deployments.

Service Required Purpose

Service	Required	Purpose
MongoDB	Yes	Primary operational data store. Holds tenants, projects, agents and their revisions, sessions and turns, access control lists, secrets, evaluation definitions and results, OAuth tokens and chat memory. The engine creates its indexes on startup.
Elasticsearch	Yes	Semantic memory and search. Stores the scoped memory units (session, short-term) together with their embedding vectors and serves vector similarity queries — this is what makes agent memory and knowledge retrieval work. Also the index for technical reports and bridge test runs.
Keycloak	Yes for authenticated use	Identity provider and JWT issuer. The engine validates access tokens against its issuer and maps the token’s realm roles to engine authorities. Also the integration point for enterprise identity providers.
Object storage	Yes	Resource storage for generated and recorded content: synthesized speech cache, session recordings, composed avatar thumbnails. Three backends are supported — S3-compatible (MinIO locally), Azure Blob Storage and the local filesystem — selected with `promethist.storage.type`.
OpenTelemetry collector	Optional	Receives traces, metrics and logs from the engine. Behind it the local stack runs Prometheus, Tempo, Loki and Grafana; a deployment additionally exposes Prometheus metrics through the actuator endpoint.
PostgreSQL	Only with Keycloak	Database used by Keycloak itself. The platform services do not use it.

MongoDB

Yes

Primary operational data store. Holds tenants, projects, agents and their revisions, sessions and turns, access control lists, secrets, evaluation definitions and results, OAuth tokens and chat memory. The engine creates its indexes on startup.

Elasticsearch

Yes

Semantic memory and search. Stores the scoped memory units (session, short-term) together with their embedding vectors and serves vector similarity queries — this is what makes agent memory and knowledge retrieval work. Also the index for technical reports and bridge test runs.

Keycloak

Yes for authenticated use

Identity provider and JWT issuer. The engine validates access tokens against its issuer and maps the token’s realm roles to engine authorities. Also the integration point for enterprise identity providers.

Object storage

Yes

Resource storage for generated and recorded content: synthesized speech cache, session recordings, composed avatar thumbnails. Three backends are supported — S3-compatible (MinIO locally), Azure Blob Storage and the local filesystem — selected with promethist.storage.type.

OpenTelemetry collector

Optional

Receives traces, metrics and logs from the engine. Behind it the local stack runs Prometheus, Tempo, Loki and Grafana; a deployment additionally exposes Prometheus metrics through the actuator endpoint.

PostgreSQL

Only with Keycloak

Database used by Keycloak itself. The platform services do not use it.

AI and Third-Party Providers

Which providers are actually used is per-deployment configuration; credentials are supplied through the deployment’s secret store.

Category Providers

Category	Providers
Language models	OpenAI, Azure OpenAI (including reasoning models and round-robin over several deployments), Anthropic, Google GenAI (Gemini). Configured per engine component in `promethist.ai.llm-components.*`.
Embeddings	Triton (self-hosted, default), OpenAI, Azure OpenAI.
Speech to text	Amazon Transcribe, Google Speech-to-Text, Deepgram, Soniox.
Text to speech	Amazon Polly, Google, Microsoft Azure, ElevenLabs, OpenAI, Hume AI, MamaAI, Triton.
Realtime (speech to speech)	OpenAI-compatible realtime models, and an internal realtime pipeline composed of the STT/LLM/TTS components above.
Avatars and animation	Triton-served animation models and the external LiveAvatar service.
Telephony	Twilio (Media Streams) — see Twilio Telephony Integration.
Messaging and notification	Firebase Cloud Messaging (push), Slack (agent in a channel), Mailgun (email).
Tooling	Model Context Protocol servers, providing external tools to agents; ClickUp for user-submitted bug reports; BrowserStack for the bridge’s device farm.

Language models

OpenAI, Azure OpenAI (including reasoning models and round-robin over several deployments), Anthropic, Google GenAI (Gemini). Configured per engine component in promethist.ai.llm-components.*.

Embeddings

Triton (self-hosted, default), OpenAI, Azure OpenAI.

Speech to text

Amazon Transcribe, Google Speech-to-Text, Deepgram, Soniox.

Text to speech

Amazon Polly, Google, Microsoft Azure, ElevenLabs, OpenAI, Hume AI, MamaAI, Triton.

Realtime (speech to speech)

OpenAI-compatible realtime models, and an internal realtime pipeline composed of the STT/LLM/TTS components above.

Avatars and animation

Triton-served animation models and the external LiveAvatar service.

Telephony

Twilio (Media Streams) — see Twilio Telephony Integration.

Messaging and notification

Firebase Cloud Messaging (push), Slack (agent in a channel), Mailgun (email).

Tooling

Model Context Protocol servers, providing external tools to agents; ClickUp for user-submitted bug reports; BrowserStack for the bridge’s device farm.