Telemetry

The app exports OpenTelemetry over OTLP/JSON to the Promethist collector. Two signals are exported — logs and metrics — over one shared pipeline.

Both are produced through a Core API that publishes an EventBus event, so callers never touch the telemetry assembly or any OTLP type and anything in the app can report from anywhere:

Signal API Event

Logs

Log (Core.Logging) — see Logging

LogReported

Metrics

Metric (Core.Metrics)

MetricReported

Producing metrics

Metric.Histogram(Category.Graphics, "fps", fps, FpsBuckets, ("graphics.preset", "High"));

Each call becomes one datapoint, stamped when the telemetry service picks the event up. The name given here is the bare one; the exporter prefixes every metric with promethist.client. on the way out, so the example above lands in the backend as promethist.client.fps. Names throughout this page are written without that prefix. Metrics differ from logs in three ways:

  • They are not gated behind ARC_LOGGING — metrics ship in release builds, everything but Log.Debug does not.

  • They never reach the engine: MetricReported is a plain IEvent, not an IMessagingEvent.

  • They carry their producer’s dimensions plus app.platform and os.type, and nothing else. The project / user / agent / session / turn context that every log line gets is deliberately left off, and so is the device ref — those are unbounded cardinality, and the backend keeps one series per combination of attribute values. Keep producer dimensions bounded for the same reason, and prefer putting them on a counter over a histogram: a counter costs one series per combination where a histogram costs one per bucket.

Kinds

Three: Metric.Gauge, Metric.Count and Metric.Histogram. Only the last two survive being reported by many devices at once — see Gauges.

Every kind is exported with delta temporality: a datapoint carries what happened during one export window rather than a running total. A session that dies mid-flush therefore loses only the window it was in, and the queue path is shared with logs.

Measurements are not exported one per call. TelemetryService folds each window down to one datapoint per stream — one metric name plus one set of attribute values — and one record per name, which is what the OTLP data model asks of a delta exporter on both counts: a name may appear only once in a scope, and a stream’s windows have to be disjoint to chain back into a cumulative series. Two increments of the same counter three seconds apart leave as one datapoint of 2, not two of 1.

Delta temporality means the collector has to convert to cumulative before Prometheus stores it (deltatocumulative), since Prometheus counters and histograms are cumulative. That conversion is the reason the folding matters: a datapoint whose window is empty, or which overlaps another of its own stream, is not something the processor can accumulate.

Counters

A counter records that something happened, and is queried as a rate — loads failing, reconnects giving up. A gauge of "how many so far" loses every event between two reports; a counter does not, because the backend reads the increase between them.

Count the outcome, not the failure. A failure counter on its own is a numerator with no denominator: 40 failed loads means nothing until you know whether there were 50 loads or 50,000. One counter with a result attribute gives both, and the failure rate is a division inside one query:

Metric.Count(Category.Visual, "visual.load.result", ("result", "failed"));

Keep result a small closed set of words. An exception message in there is unbounded cardinality and will cost more than the metric is worth.

A counter is also where a metric’s other dimensions belong. visual.load.result carries avatar and environment and the histogram beside it carries neither, because the counter pays one series per combination against the histogram’s fifteen — and "which avatar fails to load" is the question worth slicing anyway, where "how long did loads of avatar X take" rarely is.

Histograms

A gauge of a latency can only be averaged across a fleet, and the average latency is the one number nobody cares about. A histogram buckets the value client-side, so "the slowest 5% of commits waited over 2 s" survives aggregation across every device.

The producer owns the bucket bounds and passes the same array on every call:

private static readonly double[] LatencyBuckets = {0.05, 0.1, 0.25, /* ... */ 20, 30};

Metric.Histogram(Category.Conversation, "ptt.transcript.latency", latency, LatencyBuckets);

Bounds are upper limits, ascending; bucket i holds bounds[i - 1] < value ⇐ bounds[i], and one implicit bucket above the top bound catches the rest. Pick them around what the value is supposed to be, with enough tail to tell "slow" from "broken" — they cannot be changed later without breaking the series' history.

Every measurement taken in one window lands in a single datapoint: the count, the sum, and the bucket counts across all of them. A histogram is the one kind that gets cheaper the more it is used, and one datapoint per measurement would have thrown that away.

Bucket counts also add, which is what makes a histogram the only shape that survives aggregation across devices. Summing bucket counts from a thousand clients and running histogram_quantile over the result gives a true fleet percentile; averaging a thousand per-device percentiles gives a number with no meaning. This is why a statistic that varies per device is reported as a histogram observation rather than as a gauge of the statistic itself.

Exporting

TelemetryService (Assets/Scripts/Telemetry/) is a MonoBehaviour/IService living on the Telemetry Service prefab. It buffers both signals and shares everything except the payload shape:

  • the OTLP Resource — service name/namespace, device ref, app version, platform and OS attributes;

  • the flush trigger — Update, or a queue reaching 1024 entries, plus a flush of both on OnApplicationPause/OnApplicationFocus/OnApplicationQuit, because that is where a session usually ends and an unflushed queue does not survive it — on the metrics side a whole minute of it;

  • the upload transport — POST to https://otel-collector.{region}.promethist.ai/v1/{signal}, with the region taken from the current project’s engine URL and {signal} being logs or metrics.

Each signal drains its queue into per-Category OTLP scopes, allocated once in Awake and reused across flushes. In the editor the resource uses the preview service namespace; builds use default.

The cadence is not shared. Logs flush every 3 s and metrics every 60 s, because only one of the two has a latency requirement: a log line matters most in the seconds before a crash, while a metric is aggregated over minutes downstream and nobody can tell when it arrived. The longer metric window is what makes the folding pay — measurements closer together than the window collapse into one datapoint, so a conversation’s worth of turns costs Prometheus one sample instead of one per turn.

The two clocks handle an empty tick differently, for the same reason. The log clock only advances on a flush that actually sent something, so its interval is a ceiling between bursts rather than a wait every producer sits through: a lone error after a quiet spell still leaves on the next frame. The metric clock advances either way, so the window stays a fixed minute — carrying a lone measurement over from a quiet spell and uploading it on the next frame would throw away the folding the longer window exists for.

Metrics take one extra step on the way out. MetricAggregator folds the drained measurements into one datapoint per stream and one record per name, and the flush stamps them with the window they cover — from the end of the previous metric flush to this one. Nothing about the window is known when a measurement is taken, which is why Metric reports a value and the aggregator, not the caller, decides what it becomes. A window with no measurements in it is not exported and does not move the boundary on, so a stream that goes quiet leaves a gap rather than a run of empty datapoints.

The two signals do not send the same resource, and the reason is the difference between how Loki and Prometheus store one. The OTLP-to-Prometheus mapping folds resource attributes into Prometheus' two target labels — job becomes {service.namespace}/{service.name}, instance becomes service.instance.id — and moves the rest onto a synthetic target_info series a query has to join against. Loki keeps them all as ordinary labels.

The log resource therefore carries the device ref, OS description and device model. The metric resource carries none of them: on the metrics side every one is a dimension, the series count is the product of all of them, and a per-device instance makes that product grow with every device ever seen — series stay in Prometheus' index for the whole retention period after they go quiet, so the cost tracks total installs rather than concurrent users.

Metrics keep exactly two device dimensions, app.platform and os.type, stamped onto every datapoint rather than the resource so they can be grouped by without a target_info join. Both are small closed sets. Investigating one device’s problem is a job for the logs, which carry the device ref along with the project, user, agent, session and turn — Loki charges a label value for that where Prometheus would charge a series.

Opting out

Share diagnostic data in the Support section of the settings screen (SettingsStore.ShareDiagnostics, backed by Settings.ShareDiagnostics and on unless the user turns it off) is the global off switch for both signals. It sets TelemetryService.Enabled, and nothing leaves the device while that is false.

The switch gates intake, not upload. A line or measurement reported while it is off is dropped at the EventBus handler instead of being queued, and turning it off discards what is already queued. Gating the upload instead would leave both wrong: the opt-out would still send everything recorded in the up-to-a-minute window before it, and a queue nobody drains would grow for the rest of the session.

Turning it back on collects from that moment; nothing from the off period is recovered. The metric window boundary does not advance while it is off, so the first window after re-enabling is simply a wider one — the same as any other quiet spell. Upload debug logs is a separate and narrower dev-only switch, deciding only whether Debug severity lines are kept at all; it does nothing while diagnostics are off.

Platform and OS

The two answer different questions and are resolved differently, because a Web client can be running on any OS:

Attribute Meaning

app.platform

Which client this is — android, ios or web — taken from the build target at compile time. Empty for desktop standalone and in the editor.

os.type

What it is running on — android, ios, windows, darwin or linux.

os.description

SystemInfo.operatingSystem verbatim, so it carries the OS version too.

os.type cannot come from SystemInfo.operatingSystemFamily alone: that enum is only {Other, MacOSX, Windows, Linux}, so both mobile OSes fall into Other. Native builds resolve it from the build target instead. On Web, Unity’s loader regex-parses the user agent into a fixed set of OS names and only Windows/MacOS/Linux map onto the enum, so Android and iOS browsers are read off the front of SystemInfo.operatingSystem ("Android 14", "iPhoneOS 17.5") instead. In the editor both fall through to the family, since the active build target says nothing about the host.

Together they separate the cases that matter: app.platform=ios with os.type=ios is the native app, while app.platform=web with os.type=ios is a browser on an iPhone.

OpenTelemetry’s os.type enum does not actually include android or ios — it lists those as examples for os.name. We emit them anyway, since the alternative loses the distinction this section exists to draw, and OTel enums are open. browser was emitted here previously and was simply wrong: a browser is not an operating system.

The wire model lives in Telemetry.OpenTelemetry, with the two signals mirroring each other file for file — ExportLogsServiceRequest/ResourceLogs/ScopeLogs/LogRecord against ExportMetricsServiceRequest/ResourceMetrics/ScopeMetrics/MetricRecord — over a shared Resource, InstrumentationScope and KeyValue/AnyValue.

Upload failures are reported with Debug.LogWarning, not Log.Warning — a failure logged through Log would re-enter the queue it just failed to drain.

Metrics collected

Graphics

PerformanceMonitor (Assets/Scripts/Visual/Runtime/) samples frame time every frame into a CircularBuffer and reports under Category.Graphics once a minute, clearing the window each time so consecutive reports never share a frame. The window is bounded by time rather than by sample count: a count-bounded window covers a fixed number of frames but a framerate-dependent stretch of wall clock, which on a fast device leaves most of the minute in no window at all — and the stutters that fell in those gaps with it. A window holding fewer than 30 frames is discarded instead of reported, since its percentiles would be decided by one or two frames.

Two kinds of frame never enter the window: anything longer than five seconds, and the first frame after OnApplicationFocus/OnApplicationPause hands control back. That frame spans the entire time the app was away — minutes, on a backgrounded tab — and is not a stutter anybody sat through. The five-second ceiling is the backstop for a suspend path that raises neither callback; it is far above any hitch a user would experience as one, so a two-second freeze still lands in the window and still drags the 0.1% low, which is the entire point of measuring it.

It lives in Promethist.Visual rather than in the telemetry assembly because its dimensions come from VisualManager and QualityManager — telemetry stays a transport that knows nothing about its producers, exactly as it does for logs. It bootstraps itself after scene load (RuntimeInitializeOnLoadMethod), so there is no scene wiring.

Every datapoint carries the dimensions describing what is on screen — avatar and environment, plus graphics.preset on all but resolution.scale. These are bounded by the content catalog and the preset list, so performance can safely be sliced by them, and with no instance label to multiply against, the product stays in the tens of thousands of series. Bundle versions are deliberately not among them: they would multiply that by the number of releases each bundle has ever had, and service.version already narrows a regression to a client build.

Nothing here is a gauge. Each statistic is reported as a distribution across sessions — the histogram bucket a device’s own minute-long window fell into — so a query answers "what fraction of the fleet is below 30 fps" rather than "what was the last number some device happened to push".

Metric Description

fps

Histogram of the framerate a window achieved, with a stat attribute saying which statistic went in: avg, low1 or low0.1 — the average, and the framerate at the 99th / 99.9th percentile frame time (nearest-rank). One metric rather than three, so a dashboard plots all three from one query and they cannot drift apart in name or dimensions. histogram_quantile over it reads as "the worst 10% of sessions ran below X", which is the question a fleet-wide framerate is actually asked.

resolution.scale

Histogram of the dynamic resolution scale from QualityManager, bucketed 0.4–1.0. Its own metric rather than a dimension on the framerate: adaptive quality moves it continuously, which would fragment the framerate series. It is also where load is absorbed when the framerate holds steady. The one metric here that drops graphics.preset: the question it answers is which avatar and environment cost the most to draw, and a preset on every bucket would multiply that comparison for a dimension quality.level already carries.

quality.level

Counter of windows observed at each quality level, named in a level attribute — the name rather than the index, which reads as nothing on a dashboard and changes meaning when levels are reordered. A counter rather than a distribution because the level is a category, not a measurement: rate() over one level is the share of time spent there. graphics.preset says what the user picked, which under Auto says nothing about what is being rendered — this says where auto-scaling settled.

On gauges

Metric.Gauge still exists, but nothing uses it, and new metrics should think twice before it does. A gauge is a level with no window: the last value written to a series wins. That worked while every device had its own instance label and therefore its own series. Without one, two devices reporting the same metric under the same attributes write to the same series at overlapping times, and Prometheus rejects the loser as an out-of-order sample. What survives is neither an average nor a maximum, just whichever device’s flush happened to land first.

Counters and histograms have no such problem, because delta datapoints from different devices add. That is the whole reason the graphics metrics are distributions: a gauge would not merely be imprecise across a fleet, it would be wrong. A gauge is still the right shape for something genuinely singular — but on a client with no device identity in its metrics, that is a narrow set.

Visual

VisualManager times a bundle load under Category.Visual, from the load starting to the avatar and environment being instantiated and linked — the wait the user sits through when entering a conversation.

Metric Description

visual.load.duration

Histogram, in seconds, of a full 3D load. No dimensions at all — avatar and environment ride the counter beside it, which pays one series for each where this would pay fifteen. Buckets run 1–120 s: a few seconds is the good case, tens are routine on a cold cache, and the top buckets exist because loads do run past a minute. Only successful bundle loads are recorded — the orb, an already-loaded no-op and a fallback after a failure all measure something else, and a failed load is a log line rather than a slow one.

visual.load.result

Counter of how a bundle load ended — result of success, failed or cancelled — so the failure rate has a denominator. Every failure exit funnels through FailToOrbAsync, which is the one place failed is counted. Identifiers that never parse or resolve are not counted: they fail before a load starts, and they are a configuration bug rather than a load that went wrong.
Whether the bundles came from the cache is the dimension that would explain the spread best, but RuntimeBundleManager does not report it up to VisualManager today. Add it there first if the distribution turns out bimodal.

Evaluations

EvaluationsController times the wait for the engine, under Category.Engine.

Metric Description

evaluations.wait.duration

Histogram, in seconds, of the wait for a conversation’s evaluation, in poll-sized steps up to the 45 s the poll gives up at. Only a review that actually renders is recorded: a user who leaves first, a conversation with no evaluations (NO_EVALS, or a finished-but-empty result), an engine-side failure and a poll that ran out of attempts are all excluded — none of them is a wait sat through to the end, and the last two would pile onto the top bucket and hide the real distribution.

evaluations.wait.result

Counter of how a wait ended — result of shown, empty, failed, timeout, cancelled or restarted. This is the denominator the duration histogram is missing: on its own the histogram only shows waits that ended well, so a rise in timeout would look like fewer slow waits rather than more. A timeout’s duration is still not recorded, since it is always the poll budget and says nothing.

Conversation

ConversationManager reports under Category.Conversation, from the same place the latency is already logged.

Metric Description

ptt.transcript.latency

Histogram, in seconds, of the wait from a push-to-talk commit to the final transcript answering it — how long the user stands there after letting go of the button. Only the first transcript after a commit is measured, and only in push-to-talk mode. Buckets run 0.05–30 s, dense below a second (the target) and sparse above it, where the interest is only in how bad the stall was.

ptt.transcript.result

Counter of how a commit’s wait ended — result of received, or error / disconnected / restarted for one dropped with the transcript still outstanding. A dropped wait contributes no latency measurement, so without this a transcript that never arrives is indistinguishable from one that was never asked for.

session.reconnect.result

Counter of how an unintended drop ended — result of recovered, gaveup or abandoned. abandoned is the user leaving or starting another conversation mid-reconnect, so it is neither a win nor a loss and has to be told apart from both.

session.reconnect.duration

Histogram, in seconds, of how long a recovered reconnect took — the silence the user sat through. Buckets stop at the 30 s reconnect window, since anything longer has already given up. A gaveup reconnect is not recorded: its duration is always the window, which says nothing.

Adding a metric

Call Metric.Count or Metric.Histogram from wherever the value already lives — the producer owns the sampling and the dimensions, and nothing needs registering with TelemetryService. Count an outcome, measure a duration, and read Gauges before reaching for the third kind. Report no more often than the value is worth: `PerformanceMonitor’s once-a-minute cadence is deliberate, since sampling every second costs sixty times the storage to say the same thing.