Test Suites

It drives the engine’s public pipeline socket, the WebSocket {promethist.mcp.engine-url}/socket/pipeline/agent:{ref}[.{revision}] (wss:// for an https:// engine URL), with text input and JSON output: each turn is one text frame, and the tool reads the elements until the turn’s exit — the only reliable end of a turn, because the engine still plans and may speak again after the reply. A turn has a 90-second budget.

The engine endpoint itself asks for no credentials, so the tool performs its own authorization: the caller needs Viewer on the agent’s project, and a plain ref runs only the Published revision.

Sessions are anonymous, because the engine does not accept the MCP token. For projects the engine closes to anonymous sessions — sign-in required, or external roles enforced with an identity provider listed, the same rule the test-suite check uses — the tool signs in with the engine test identity (promethist.testing.engine-basic-identity). The engine runs a Draft for no anonymous session, so drafts are tested after promotion to Preview.

When the identity does not get a turn through, the error says why. An engine from 2026.42.1 on answers credentials it does not know with 401, reported as an unrecognised identity: the configured value is not one this engine has. An identity the engine knows but does not admit to the project gets a refusal, reported as not enabled for the project. A configured value that is not HTTP Basic credentials at all is never sent, and the refusal says so.

A conversation keeps one connection (TestAgentConversations). The engine holds the agent’s planning in its live realtime session; a new connection per turn would lose it after every line and make the engine treat every line as a disconnect and a new session start. The socket a first turn opens stays open, and the engine session id travels back to the caller as session_id (sessionId=<uuid>); the next call with it sends its line over the same socket. A conversation belongs to its caller and its agent: another caller’s or another agent’s call with its id is refused, and so is a line sent while a turn is still running.

The connection closes after four minutes without a turn (below the ingress’s 300-second read timeout on /socket), after a turn that did not end with its own exit (cut off, failed, or the call cancelled), when the agent ends the conversation, and on shutdown; a caller keeps at most five conversations open, and the pod 200, closing the least recently used idle one when a new one needs room. Closing is a plain socket close, never #exit: the engine sees the client hang up and about 30 seconds later runs the session’s end-of-conversation work (post-session integrations, evaluations at the end of the conversation) — once per conversation, up to about five minutes after the last turn unless the agent ended it.

A session_id with no live connection — closed, or held by another pod or a restarted one, since the registry lives in the pod’s memory — reconnects to the same engine session, which keeps its messages but not the earlier turns' planning, and the result says so. A conversation that closed on this pod is remembered: its reconnect is still refused to another caller or agent, and keeps the locale, simulated user, time zone and run label of its first turn.

On the socket a line starting with # is an engine action (#ready:key= would start a session with any agent past the tool’s guard, #exit would end the session), so #intro, the agent’s opening, is the only such line the tool sends.

The engine decodes the socket’s query twice (the decoded query is split on & and decoded again), so a value carrying an encoded & would arrive as parameters of its own, testMode=false among them. The locale must be one language tag (de_DE is sent as de-DE), the simulated user and the run label may use only letters, digits and . _ : @ -, and a time zone with ` in it is refused, since ` arrives as a space.

device_ref is who the simulated user is, and conversations sharing one are the same engine user coming back; time_offset, an ISO-8601 duration, backdates a whole conversation through the engine’s timeOffset, which the engine reads only because the tool sends testMode=true. Only a negative offset is accepted: the engine adds it to its own clock, so the positive one a caller writes for "ten days ago" would date the session in the future instead and say nothing about it. Like the locale and the simulated user, the offset is fixed when the conversation connects.

The result reports when the agent has ended the conversation.

Live tests run real engine sessions and count towards usage like any other conversation.

Restoring is the engine’s own delete-then-write: it clears the user’s memory before loading the state, so whatever the sessions since the snapshot left behind is gone. The tool is confirmation-gated.

The engine’s /api/memory endpoints are permitAll and take a bare userId, so what a caller can reach is bounded by the platform instead. agent_ref authorizes the call — the caller needs Viewer on that agent’s project, as test_agent requires before it drives one — and every device_ref is forced under the mcp: namespace before it is resolved, the same prefix test_agent stamps on its own. A ref written as another client’s (client:, bridge:) therefore names a device of this connector’s, not the one it looks like.

A session is aggregated about 30 seconds after its connection closes, so a snapshot taken straight after the last turn can hold nothing. The result reports how many memory units it holds and says so when there are none.