Analytics Suite

Sessions come from the session collection, filtered by the project’s ref, the window, the agent selection, the state of the agent revision the session ran on and the user, without the sessions left out under Who Is Counted. Session and user counts skip sessions in which no turn completed; the engine’s turnCount runs one ahead of the stored turns, which TurnMath in util/Functions.kt accounts for.

A conversation’s length is its connected time: the sum of the duration of the session’s Realtime records in the engine’s edr collection — one per realtime connection, written when the connection closes, and the records billing charges for. A user who left and came back days later adds only the time actually connected. A session without such a record, usually one still connected, has no length yet: it is left out of the total and of the average’s denominator instead of counting as zero, while its turns still count towards the average turn count. EdrIndexMigration ensures a partial {sessionId, duration} index on edr that answers the lookup from the index alone; the cost is one lookup per session. Exports report the same per-session figure, empty while none is recorded.

The end shown in exports is the created time of the session’s newest stored turn plus a fixed allowance for that turn — a timestamp only, which is why the length is not end minus start. The newest turn is looked up by sessionId, not by the session’s lastTurnId, which runs one turn ahead of what is stored, because the engine opens the next turn as soon as a turn’s output completes. TurnIndexMigration ensures the {sessionId, created} index on turn that makes the lookup an index seek.

The lifetime variants cover every session the project ever had, so they are computed once per filter and cached for ten minutes. A request waits at most five seconds for them and otherwise answers without them (or with the previous value) while the computation finishes in the background for the next request.

Only evaluations that run at the end of a conversation and are admin-facing reach the analytics. The featured-outcome card reads, for the users with a featured result in the window, their results over their whole history — someone who passed in June and came back this week counts as passed. Agent evaluations (testMode) are left out, and so are the results of the sessions and users left out under Who Is Counted. EvaluationResultIndexMigration ensures the indexes these reads use.

The Conversations panel shows the selected conversation’s results in place of its transcript (GET /api/v1/project/{projectId}/sessions/{sessionId}/evaluations in SessionController, behind the transcript’s gates; SessionEvaluationService): the latest conversation-end run of each of the evaluations above, its elements in the evaluation’s order and worded by their definitions. A result is matched to the evaluation’s current element by id, then by ref, because saving an evaluation recreates its elements. A session or user left out under Who Is Counted gets no results. While the engine has yet to mark the session evaluated (evaluationResultsState still NOT_READY, the session changed within the last 15 minutes), the answer is pending, and the Studio asks again every ten seconds while it is.