Engine Application
Codebook and Development Profile
Two mechanisms that turn a project’s evaluation output into long-term coaching for one person. The codebook is per project: a small, slowly growing vocabulary of the patterns that project’s evaluations keep describing. The development profile is per user: a coach’s note and a list of skills, rebuilt from that person’s coded occurrences after a conversation.
Both are gated by feature flags — promethist.codebook.enabled and
promethist.development-profile.enabled — and both are best-effort. Every failure path ends in an
empty Mono after a logged error, so neither can break the evaluation flow that triggered them.
When They Run
EvaluationService
fires both after a conversation’s evaluations finish, in order: classification first, then the
profile that reads what classification produced. The chain is subscribed on its own, not awaited by
the caller.
Only user-facing evaluations that run ON_CONVERSATION_END feed the codebook. An insight whose
value is null never becomes an occurrence, which is how the evaluation layer’s "cannot be decided
from the transcript" contract keeps unusable material out.
The Codebook
Classification
For each session,
CodebookClassificationService
collects two kinds of subject — the insight results and the featured outcome — and asks one LLM call
to file every one of them under an existing code. Subjects already classified are filtered out
first, so replaying a session costs nothing and cannot double-count.
A name the model returns that does not match a code exactly is resolved semantically before being
given up on; anything still unmatched is stored with a null codeId, which is what the growth pass
looks for. The occurrence’s created is copied from the evaluation result it came from, not from
the moment of classification.
Growth
Once a project has accumulated GROWTH_THRESHOLD (10) unclassified occurrences,
CodebookGrowthService
claims a batch of up to 50 of them and asks the model to propose codes that cover them, then assigns
every text in the batch.
The claim is a per-document atomic update with a 5-minute staleness timeout, so two sessions ending at once cannot grow from the same texts. They can still coin two differently-named codes for the same pattern: each pass sees the codebook as it was when it started, and the unique index on (project, name) only catches identical names. Semantic duplicates are the known residual risk.
New codes are written in the language of the texts that produced them, so a project’s codebook takes the language of its evaluations.
The Development Profile
The Evidence
DevelopmentProfileService
renders three blocks into the prompt, all scoped to the same 30-day window and the same 7-day
comparison bucket:
| Block | What it carries |
|---|---|
Codebook |
Per code: the lifetime count, and a day-bucketed line comparing the last 7 days with the 7 before them — see Evaluated Days. Plus its meaning and up to 3 example excerpts, collapsed to one line and capped at 240 characters each. |
Co-occurrence |
Code pairs that fired in the same session, counted across the window. Only pairs seen in 2+ sessions are surfaced; with nothing recurring yet, the per-session code sets are passed instead. |
Metrics |
A |
Evaluated Days
Counts alone cannot separate a habit that stopped from a week nobody practised, so every bucket is reported against the number of days in it on which this person was evaluated at all — "12× on 3 of 4 evaluated days". That denominator is computed across all of the user’s occurrences, so it is the same for every code and the codes stay comparable with each other. "None in 4 evaluated days" is a real quiet spell; "no evaluated days" is an absence of evidence, and the prompt forbids reading it as progress.
Computed Movement
The direction of a metric is decided in Kotlin, not by the model. A move is only called when both buckets rest on at least 3 records and the change clears 5% of the metric’s declared range, or 10 points for a rate; otherwise the bucket pair is labelled flat or not comparable. Measuring against the range rather than against the previous value is what stops a metric sitting at zero from turning every flicker into a trend.
The range comes from the insight definition’s minValue/maxValue, and travels with the aggregation
so the prompt can state it. Without it a reader — human or model — cannot tell 1.0 out of 5 from 1.0
out of 100.
The Regeneration Gate
The three evidence blocks are hashed into an evidenceFingerprint stored alongside the content. When
the fingerprint matches the stored one, the model is not called at all and nothing is written: it
would be handed the same prompt and could only reword the profile it already wrote. A profile that
changes is then a signal to the reader rather than noise.
Two deliberate exceptions:
-
a profile flagged
needsReviewis retried even on identical evidence; -
when the evidence moved but the model rewrote nothing, only the fingerprint is recorded —
updatedAtis left alone, since nothing a reader sees has changed — so the next run is gated instead of paying for the same call again.
The previous profile is not part of the hash. Including it would change the fingerprint after every regeneration and the gate would never fire.
Authored Content and Computed Content
The model authors everything a reader sees. The service computes advisor — name, ISO dates,
session and evaluated-day counts — after the loop ends, and strips it, along with generatedAt and
model, from the profile the model is shown. The human-readable rendering of the same period lives
in header.periodSummary, which the model writes, because plural forms and date order follow the
language and cannot be assembled from a template.
Language and Stable Labels
A profile is written in the language of the codebook entries it was built from, which is whatever
language the project’s evaluations use. Section headings and direction labels are therefore authored
too, and carried in labels.
Because they are display furniture rather than findings, they are re-used verbatim while language
holds, whatever the model wrote this run — a heading that drifts from one synonym to another between
two sessions is noise the reader has to re-read for nothing. A direction the previous profile never
used has no carried label, and that one is taken from the current run. The prompt asks the model not
to rewrite them either; the carry-over is the guarantee.
Enumerations stay stable tokens in every language, so a client drives colour and ordering from
direction and sentiment and takes the words from labels.
Validation
DevelopmentProfile.validate
holds what the Kotlin types cannot: a note title and text that are present and under 1400 characters
together, 2–8 skill rows with distinct snake_case ids, a label for every direction the rows actually
use, and an action that is required on a Risk row and rejected on every other one.
finishProfile returns these errors to the model so it can fix them before it stops. If the content
is still invalid afterwards, the last valid profile is kept and flagged needsReview; on a cold
start nothing is saved at all. Valid content that arrived without finishProfile — a tool-loop step
cap, say — is saved, also flagged.
Replaying
CodebookReplayService
runs both mechanisms over a user’s existing sessions, oldest to newest, and snapshots the codebook,
the profile and the rendered prompt blocks after each one. It reuses the live path rather than
imitating it, so what it shows is what production would do.
A replay is scoped to one project and one user, and asOf is each session’s own timestamp so the
day buckets are measured from the point being replayed. On a dry run it deletes only the rows it
created — diffed against a snapshot taken before the run — and restores the previous profile, so it
is safe against a user who already has data.
The prompt blocks in each snapshot are what makes the output reviewable: the profile can be judged against the evidence it was actually given, rather than against the evidence one assumes it had.