Project Management
Evaluations
|
What are Evaluations for? An evaluation is a set of criteria you write once. A model then reads the whole conversation and returns one value per criterion: a score, a yes or no, a short piece of reasoning. That gives you two things reading transcripts yourself cannot:
An evaluation measures one topic. A second topic belongs in a second evaluation. It also does nothing on its own: it runs only for the agents you attach it to, and it returns only what its output data elements define. See Attaching Evaluations to Agents. |
You find evaluations under Performance → Evaluations in the left sidebar.
|
Evaluations that score your agent during automated tests are a separate list, under Test → Agent Evaluations. See Agent Evaluations. This page covers the evaluations under Performance, which assess the user’s side of real conversations. |
A Worked Example
The rest of this page points back to one evaluation, so that each field has something concrete behind it.
The scenario. Sales reps practise the opening conversation at a trade-show stand. The agent plays a prospect who is busy, slightly distracted, and will walk away in three minutes. The person being evaluated is the rep, not the agent.
The one question the evaluation answers. Did the rep earn a concrete next step, and if not, what cost them?
Everything else exists to explain that answer. The evaluation has seven output data elements and one featured output:
| Output data element | Type | Why it is there |
|---|---|---|
The prospect’s verdict |
Pass / Fail |
The outcome. Labelled |
Asked before pitching |
Pass / Fail |
Did the rep ask about the prospect’s situation before describing the product. |
Qualified the team size |
Pass / Fail |
Did the rep find out how many people the solution would cover. |
Kept it short |
Pass / Fail |
Short alternating turns, or a monologue. |
Stayed inside the claims rules |
Pass / Fail |
Did the rep avoid a retracted figure, an invented number, a named reference customer and a price. |
Strongest moment |
Text |
One moment that worked, quoted, for the debrief. |
Biggest gap |
Text |
One moment that cost them, quoted, with what a stronger version would have said. |
The featured output is a Number from 0 to 5 that combines those seven into one headline result. See Featured Output.
Three things about that shape are worth copying:
One outcome element, several behaviour elements. The verdict says whether it worked. The four Pass / Fail elements say why. Without the behaviour elements a failed conversation tells the trainee nothing they can act on.
Two free-text elements for coaching. Scores rank people. Quoted moments teach them. Both free-text elements ask for a specific moment from the transcript, not a general impression.
Nothing measures two things at once. "Kept it short" is only about turn length. Mixing length and content into one element gives a value nobody can interpret.
The Evaluations List
The list shows every evaluation in the project. Click a row, or the pencil icon, to open one.
Name |
The evaluation name. Click a column header to sort. |
Description |
The start of the description. The full text is shown when you point at it. |
Version |
The state and number of the evaluation’s newest version, for example Draft:5 or Published:4. See Versions. |
Agents using |
The agents the evaluation is attached to. |
Analytics |
Yes when Admin Facing is on, so results appear in the Analytics Suite. |
User insights |
Yes when User Facing is on, so end users see their results in the app. |
Actions |
Edit, Clone and Delete. |
A search field appears once the project has more than five evaluations.
|
You need the Editor role or higher on the project to create, clone or delete evaluations. With the Viewer role you can open an evaluation, but you can’t change it. |
Creating an Evaluation
-
Click Create.
-
Enter an Evaluation Name and a Description.
-
Choose the Evaluation Purpose. See Evaluation Purpose.
-
Write the Evaluation Prompt.
-
Add at least one output data element. See Output Data Elements.
-
If you selected Post-Conversation Analysis, choose where the results go. See Where Results Go.
-
Click Submit. The new evaluation is saved as a Draft.
Agents only run a Published version, so publish the evaluation once it is ready. See Versions.
Reset clears your changes. If you try to leave with unsaved changes, Studio asks you to confirm.
|
While you fill in the form, the right side of the page previews where the results will appear: the real-time steering configuration, the Analytics Suite tiles, the end-user Session Insights screen and the webhook payload. Previews use sample values, not real results. |
Evaluation Purpose
Select one or both modes:
Real-time Steering |
Runs during the conversation and gives the active agent context, so it can adjust on the spot. For example, if the conversation is going off-track, the agent can course-correct without waiting for the session to end. The result stays inside the conversation: it never appears in the Analytics Suite, in the app or in a webhook. |
Post-Conversation Analysis |
Runs after the conversation ends, for analytics, feedback and coaching. You need it for every destination (Where Results Go) and for the featured output. |
Evaluation Prompt
The evaluation prompt is the shared framework. It is given to the model once for every output data element, so put in it whatever applies to all of them: who is being evaluated, what the scenario is, what rules the conversation is judged against.
What each element returns is defined by that element’s own extraction prompt, not here.
|
Always name who is being evaluated, in capitals, in every prompt you write. In a roleplay the subject is the user, the trainee, not the agent. If you do not say so, the model will often score the agent instead, because the agent does most of the talking. Real evaluations repeat it in every single element, for example: Judge the USER's performance, not how the prospect behaved. This is the single most common reason an evaluation returns results that look plausible and measure the wrong person. |
Output Data Elements
Each output data element, also called an insight, is one value the evaluation returns. Every evaluation needs at least one. Click Add insight to add one, or Add group to add a group (see Grouping Output Data Elements). Use the arrows to change the order and the bin icon to remove an element.
Every element has these fields, in this order:
Type |
Text, Number, List or Pass / Fail. Changing the type clears the fields that belong to the old type. |
Output name |
A short label for the value. It is shown in the Analytics Suite, in exports and in the agent’s context. |
Description |
What the value captures, in plain words. It is shown next to the value in analytics and helps the model interpret it. |
Extraction prompt |
How to get the value from the conversation, for example what counts as a pass or how to compute the score. |
Extra fields by type:
Text |
No extra fields. A free-text answer. |
Number |
Min value and Max value: the bottom and top of the scale, as whole numbers. They do not have to start at 0: |
List |
Limit to specific items: a comma-separated list of the values the model picks from, for example |
Pass / Fail |
Pass label and Fail label: the wording shown for each outcome, for example |
Writing Output Data Elements That Work
The form will accept any element you type into it. These are the rules that separate a scorecard somebody acts on from one nobody reads.
Measure one thing per element. An element covers either what the user did or how the agent performed, never both, and never two behaviours at once. If you cannot say in one sentence what a value means, split it.
Name the subject in the extraction prompt, every time. See Evaluation Prompt.
Build the set from three kinds of element. A scorecard that teaches anything has:
One outcome element |
Did the thing work. Usually Pass / Fail, phrased as the other party’s own verdict, for example "would this prospect actually agree to a meeting". This is the result. |
A few behaviour elements |
Did they do the specific things this scenario trains. These explain the outcome and are what a person can actually practise. |
One or two free-text elements |
A strongest moment and a biggest gap, each quoting the transcript. Ask for a specific moment and for what a stronger version would have said, otherwise you get a bland summary. |
Anchor every number scale. Say what each band means, in the extraction prompt. Add "resist inflating scores" if the top of the scale should be rare, because models drift upward without it.
Make list values distinct and exhaustive, so exactly one is always right. Overlapping values produce results you cannot total.
|
Scoring correctness needs source material. There is no ground truth for "was the trainee right" unless the project holds the playbook, rubric or product sheet the answer is judged against. Upload it to the project’s Knowledge first, then write the element. Without it the model invents a standard and scores against that. |
Grouping Output Data Elements
By default, each element is scored on its own (Standalone). Elements in a group are scored together in one call and share the same reading of the conversation, which keeps linked values consistent with each other.
To create a group, click Add group, enter a Group name and an optional Group description, then click Add insight to group. Once a group exists, each element shows a dropdown where you can move it between Standalone and any group. You can also drag an element by the handle in its header into a group, out of a group or to another place in the list, and drag a group to reorder it. Deleting a group keeps its elements; they become standalone.
Group elements that belong together, for example a score and the reasoning behind it. Keep elements standalone when they measure something independent, such as a detected scenario type or the language used.
|
A standalone element can’t see the results of other elements. If one value is calculated from others, for example an overall score, put it in the same group as those elements or make it the featured output. |
Formatting Output as Markdown
You can instruct an evaluation to return its output as Markdown. The result is then rendered with formatting (headings, bold text, lists, and emoji status marks) in Session Insights, instead of as plain text.
To do this, describe the structure you want in the output’s Extraction prompt. For example:
OUTPUT FORMAT — respond in clean Markdown only, in exactly this structure and order: **Score:** `X / 6` **Result:** 🟢 **PASS** (use 🔴 **FAIL** instead if the score is below 3) ### Criteria - 🟢 **Register match** — one or two sentences of reasoning with a short quoted snippet from the transcript. - 🔴 **Visible willingness to act** — reasoning with a short quoted snippet. - (…one bullet per criterion, in the order listed above, each beginning with 🟢 for pass or 🔴 for fail…) Replace the example marks with the real verdicts. Do not add any preamble, closing remarks, tables, or text outside this structure.
Adapt the structure to whatever your evaluation needs.
Featured Output
Available when Post-Conversation Analysis is selected.
A featured output is one headline result for the whole evaluation, for example an overall score or an overall pass/fail. It doesn’t read the conversation itself: it receives the results of all the other output data elements and combines them.
Under Featured evaluation output, turn on Enable featured output and fill in:
Result type |
Pass / Fail (default) or Number. |
Featured prompt |
How to combine the other results into the headline result. This is separate from the main Evaluation Prompt. |
Pass label / Fail label |
Pass / Fail only. The wording for each outcome. |
Max value |
Number only. The top of the scale. The scale starts at 0. |
The featured output appears at the top of the end-user Session Insights screen (with User Facing on) and in the Featured Evaluations card of the Analytics Suite (with Admin Facing on).
|
If your evaluation has an overall score, make it the featured output rather than another Number element. And if there is also a pass/fail rule, do not add a separate Pass / Fail element for it: write the featured prompt so it states the verdict alongside the number, for example |
Writing the Featured Prompt
The featured prompt receives the other elements' results, not the transcript. Say so in the first line, or the model will try to re-read a conversation it was never given.
Three things the worked example does, which are worth copying:
Say which elements carry the most weight. "The prospect’s verdict decides the band, because a booked meeting is the only real outcome at a trade-show stand. The four behaviour elements separate quality inside the band."
Describe the bands, then say the top one is rare.
Decide the band first, then discriminate inside it. 0 to 1, inadequate: the verdict is negative and at least two behaviour elements failed. 2 to 3, the expected range: either they earned the meeting but the process was sloppy, or they ran a decent conversation that ended without a next step. Most real attempts land here. 4 to 5, exceptional and rare: the verdict is positive and every behaviour element passed. Give a 5 only when there is no weakness. Resist inflating scores.
Add a hard cap when one failure should override everything else.
Hard cap: if "Stayed inside the claims rules" is negative, never score above 2, even if they booked the meeting. Quoting a retracted figure is a reputational risk, not a technique slip, and a booked meeting does not offset it.
|
With User Facing on, the trainee reads this text. Tell the model how to write it: second person, supportive and factual, one thing that worked and one thing to focus on next, never mocking. Without that instruction the wording reads like an assessment report, which is the wrong register for someone practising. |
Where Results Go
Available when Post-Conversation Analysis is selected.
| Option | Section | Where the result goes |
|---|---|---|
Session agent |
Agent Memory & Context |
To the same agent, as context for its next conversation. |
Featured agent |
Agent Memory & Context |
To the project’s Featured Agent, as context for its next conversation. See Sending Results to Another Agent. |
User Facing |
Analytics & Visualization |
To the end user, in the web client and mobile apps: the Session Insights screen after the conversation and the agent’s Insights tab. In Empower projects these results also build the user’s Profile. Signed-in users only. See Insights. |
Admin Facing |
Analytics & Visualization |
To the Analytics Suite (Evaluations Metrics, Featured Evaluations) and its exports. |
Webhook integration |
3rd Party systems |
To an external system. Turn it on and enter the URL in 3rd Party systems (Webhook integration). The results are sent when the conversation ends. |
|
|
The Featured agent option is described in Studio as "Updates the main Project Lead agent with new insights." This means the project’s Featured Agent. |
Attaching Evaluations to Agents
An evaluation runs only for the agents it is attached to.
-
Open the agent and scroll to Evaluations, the last section of the form.
-
Click the search field and select an evaluation. Only Published evaluations are listed. The evaluation appears as a card under Added to this agent. You can add several.
-
Save the agent and publish it, so production conversations use the evaluation.
To build a new one without leaving the agent, click Create a new evaluation at the bottom of the list. The editor opens, and Create and publish publishes the new evaluation and adds it to the agent. See Evaluations in the agent form.
|
Attaching an evaluation makes it run. It doesn’t mean the agent can read the result:
|
Sending Results to Another Agent
To let one agent use the results of conversations with another agent, for example a coaching agent that reviews a roleplay:
-
In Project → Settings, set Featured Agent to the agent that should receive the results (the coaching agent). See Project Settings.
-
In the evaluation, turn on Featured agent.
-
Attach the evaluation only to the agent being evaluated (the roleplay agent), not to the Featured Agent. Otherwise the Featured Agent’s own conversations are scored too.
The Featured Agent gets the results at the start of its next conversation. A project has only one Featured Agent.
Editing, Cloning and Deleting
-
Edit: change anything and click Save Draft. This saves a new Draft version. Agents keep running the published version until you publish the new one. See Versions.
-
Clone: creates a copy named "<name> - clone" with all settings. The copy isn’t attached to any agent.
-
Delete: removes the evaluation with every one of its versions and all its output data elements. This can’t be undone, and Studio doesn’t warn you if agents use the evaluation. Check Agents using first.
Versions
Evaluations keep versions, the way agents do. Every save creates a new version, and agents always run the Published one, so you can change an evaluation without touching the results of live conversations.
|
Saving never changes what is live. Agents keep running the Published version until you publish a newer one. |
The version menu is at the top right of the editor. It shows the state and number of the version you are looking at, for example Published 4 or Draft 5. Open it to see the Active revisions: every version that isn’t archived, newest first, with what you can do with it:
Preview |
On a Draft. The step before Publish. Agents don’t run a Preview version. |
Publish |
On a Preview version. Makes it the version agents run. The version that was published before is archived. |
Revert |
On the live version or a newer Draft. Archives every version above it. |
Clone |
Creates a new evaluation from this version. Use it to reuse an older version. |
To go live with a change:
-
Save it with Save Draft.
-
Open the version menu and click Preview on the Draft. Its row now offers Publish.
-
Click Publish.
Only the newest version can be edited. An older version opens read-only, and the editor says which version it is. Open the newest one from the version menu to make changes.
Evaluations created before versions existed start as Published 0.
Legacy Format
Evaluations created before groups existed can show This evaluation uses the legacy format. They score all their elements together in one call.
Saving the evaluation upgrades it: from then on, each standalone element is scored separately. To keep the elements scored together, put them into one group before you save.
Troubleshooting
- Results don’t appear in the app?
-
Check that Post-Conversation Analysis and User Facing are both on. The Profile tab only shows results from evaluations with User Facing on, and only in projects whose package is Empower. User Facing doesn’t work with Real-time Steering alone. Also check that the evaluation is attached to the agent’s published revision and that the user is signed in to the project.
- Analytics stay empty?
-
Check, in this order: the evaluation is attached to the agent and that revision is published; Post-Conversation Analysis is on; Admin Facing is on (it’s off by default); the conversations happened after you added the element; the Analytics Suite filters (dates, agent versions, user) include them.
- My real-time evaluation shows nothing anywhere?
-
Real-time results stay inside the conversation. To also see them in analytics, the app or a webhook, select Post-Conversation Analysis too and choose a destination.
- Can’t save: an error appears under Analytics & Visualization?
-
With Post-Conversation Analysis on, turn on User Facing, Admin Facing or Webhook integration. Session agent or Featured agent alone isn’t enough.
- The agent doesn’t remember the last conversation’s result?
-
Attaching the evaluation isn’t enough. Turn on Session agent.
- The coaching (featured) agent doesn’t get the results?
-
Set that agent as Featured Agent in Project → Settings, and turn on Featured agent in the evaluation. Don’t attach the evaluation to the Featured Agent itself.
- An overall score ignores the other scores?
-
Standalone elements can’t read each other. Make the overall score the featured output, or put the elements in one group.
- Error "Max value must be greater than 0"?
-
Max value has to be above 0. Set Min value and Max value to the bottom and top of the scale you want.
- Scores changed after I saved an old evaluation?
-
It was in the legacy format and saving upgraded it. See Legacy Format.
- The results describe the agent instead of the person being trained?
-
The prompts don’t say who the subject is. Name it in capitals in the evaluation prompt and in every element’s extraction prompt. See Evaluation Prompt.
- Almost everyone scores 4 or 5?
-
The scale has no anchors, so the model drifts upward. Describe each band in the extraction prompt, say what the top band requires, and add that it should be rare. See Writing Output Data Elements That Work.
- Scores look right but the trainee doesn’t know what to do differently?
-
The evaluation has outcome and score elements but no free-text ones. Add a strongest moment and a biggest gap, each quoting the transcript. See Writing Output Data Elements That Work.
- An element that scores whether the user was factually correct gives inconsistent results?
-
There is nothing to check against. Upload the playbook, rubric or product sheet to the project’s knowledge, then say in the extraction prompt what the answer is judged against.