Engine Application

Codebook and Development Profile

Two mechanisms that turn a project’s evaluation output into long-term coaching for one person. The codebook is per project: a small, slowly growing vocabulary of the patterns that project’s evaluations keep describing. The development profile is per user: a coach’s note and a list of skills, rebuilt from that person’s coded occurrences after a conversation.

Both are gated by feature flags — promethist.codebook.enabled and promethist.development-profile.enabled — and both are best-effort. Every failure path ends in an empty Mono after a logged error, so neither can break the evaluation flow that triggered them.

When They Run

EvaluationService fires both after a conversation’s evaluations finish, in order: classification first, then the profile that reads what classification produced. The chain is subscribed on its own, not awaited by the caller.

Only user-facing evaluations that run ON_CONVERSATION_END feed the codebook. An insight whose value is null never becomes an occurrence, which is how the evaluation layer’s "cannot be decided from the transcript" contract keeps unusable material out.

The Codebook

Classification

For each session, CodebookClassificationService collects two kinds of subject — the insight results and the featured outcome — and asks one LLM call to file every one of them under an existing code. Subjects already classified are filtered out first, so replaying a session costs nothing and cannot double-count.

A name the model returns that does not match a code exactly is resolved semantically before being given up on; anything still unmatched is stored with a null codeId, which is what the growth pass looks for. The occurrence’s created is copied from the evaluation result it came from, not from the moment of classification.

Growth

Once a project has accumulated GROWTH_THRESHOLD (10) unclassified occurrences, CodebookGrowthService claims a batch of up to 50 of them and asks the model to propose codes that cover them, then assigns every text in the batch.

The claim is a per-document atomic update with a 5-minute staleness timeout, so two sessions ending at once cannot grow from the same texts. They can still coin two differently-named codes for the same pattern: each pass sees the codebook as it was when it started, and the unique index on (project, name) only catches identical names. Semantic duplicates are the known residual risk.

New codes are written in the language of the texts that produced them, so a project’s codebook takes the language of its evaluations.

The Development Profile

The Evidence

DevelopmentProfileService renders three blocks into the prompt, all scoped to the same 30-day window and the same 7-day comparison bucket:

Block What it carries

Codebook

Per code: the lifetime count, and a day-bucketed line comparing the last 7 days with the 7 before them — see Evaluated Days. Plus its meaning and up to 3 example excerpts, collapsed to one line and capped at 240 characters each.

Co-occurrence

Code pairs that fired in the same session, counted across the window. Only pairs seen in 2+ sessions are surfaced; with nothing recurring yet, the per-session code sets are passed instead.

Metrics

A EvaluationInsightAggregationSeries — the same aggregations cut into consecutive buckets, so the model sees movement rather than one period average. Fetched once for the whole window and bucketed in memory.

Evaluated Days

Counts alone cannot separate a habit that stopped from a week nobody practised, so every bucket is reported against the number of days in it on which this person was evaluated at all — "12× on 3 of 4 evaluated days". That denominator is computed across all of the user’s occurrences, so it is the same for every code and the codes stay comparable with each other. "None in 4 evaluated days" is a real quiet spell; "no evaluated days" is an absence of evidence, and the prompt forbids reading it as progress.

Computed Movement

The direction of a metric is decided in Kotlin, not by the model. A move is only called when both buckets rest on at least 3 records and the change clears 5% of the metric’s declared range, or 10 points for a rate; otherwise the bucket pair is labelled flat or not comparable. Measuring against the range rather than against the previous value is what stops a metric sitting at zero from turning every flicker into a trend.

The range comes from the insight definition’s minValue/maxValue, and travels with the aggregation so the prompt can state it. Without it a reader — human or model — cannot tell 1.0 out of 5 from 1.0 out of 100.

The Regeneration Gate

The three evidence blocks are hashed into an evidenceFingerprint stored alongside the content. When the fingerprint matches the stored one, the model is not called at all and nothing is written: it would be handed the same prompt and could only reword the profile it already wrote. A profile that changes is then a signal to the reader rather than noise.

Two deliberate exceptions:

  • a profile flagged needsReview is retried even on identical evidence;

  • when the evidence moved but the model rewrote nothing, only the fingerprint is recorded — updatedAt is left alone, since nothing a reader sees has changed — so the next run is gated instead of paying for the same call again.

The previous profile is not part of the hash. Including it would change the fingerprint after every regeneration and the gate would never fire.

Authored Content and Computed Content

The model authors everything a reader sees. The service computes advisor — name, ISO dates, session and evaluated-day counts — after the loop ends, and strips it, along with generatedAt and model, from the profile the model is shown. The human-readable rendering of the same period lives in header.periodSummary, which the model writes, because plural forms and date order follow the language and cannot be assembled from a template.

Language and Stable Labels

A profile is written in the language of the codebook entries it was built from, which is whatever language the project’s evaluations use. Section headings and direction labels are therefore authored too, and carried in labels.

Because they are display furniture rather than findings, they are re-used verbatim while language holds, whatever the model wrote this run — a heading that drifts from one synonym to another between two sessions is noise the reader has to re-read for nothing. A direction the previous profile never used has no carried label, and that one is taken from the current run. The prompt asks the model not to rewrite them either; the carry-over is the guarantee.

Enumerations stay stable tokens in every language, so a client drives colour and ordering from direction and sentiment and takes the words from labels.

Validation

DevelopmentProfile.validate holds what the Kotlin types cannot: a note title and text that are present and under 1400 characters together, 2–8 skill rows with distinct snake_case ids, a label for every direction the rows actually use, and an action that is required on a Risk row and rejected on every other one.

finishProfile returns these errors to the model so it can fix them before it stops. If the content is still invalid afterwards, the last valid profile is kept and flagged needsReview; on a cold start nothing is saved at all. Valid content that arrived without finishProfile — a tool-loop step cap, say — is saved, also flagged.

Replaying

CodebookReplayService runs both mechanisms over a user’s existing sessions, oldest to newest, and snapshots the codebook, the profile and the rendered prompt blocks after each one. It reuses the live path rather than imitating it, so what it shows is what production would do.

A replay is scoped to one project and one user, and asOf is each session’s own timestamp so the day buckets are measured from the point being replayed. On a dry run it deletes only the rows it created — diffed against a snapshot taken before the run — and restores the previous profile, so it is safe against a user who already has data.

The prompt blocks in each snapshot are what makes the output reviewable: the profile can be judged against the evidence it was actually given, rather than against the evidence one assumes it had.