Microphone Time Compression

Soniox STT was observed finalizing transcripts in 1.2-1.5x the length of the utterance. The idea under test is that handing it less audio for the same speech buys some of that back.

Treat that premise as unproven. STT cost tracks the frame sequence a model derives from audio duration, not the raw sample count, and the client already streams microphone chunks live rather than uploading a finished utterance — so much of the observed figure is likely a fixed tail (endpointing silence timeout, final decode, network hops) that compression cannot touch. Measure final-transcript latency against utterance length before concluding this helps: a ratio that shrinks on longer utterances is a constant, not a throughput limit.

There is also a reason to expect the opposite of the intended effect in the modes that let the engine decide when a turn ended. An endpointer counts silence in stream time, and compression means each second of stream silence takes speed seconds of wall-clock to accumulate — so a silence threshold is reached later, by exactly the speed factor. Only push-to-talk, where CommitAudioInput ends the turn explicitly, sidesteps that. MicSpeedUpSpeechOnly sidesteps it everywhere, by leaving silence at its real length; see [Compressing speech only].

That mechanism is arithmetic, but whether it applies here is unverified: nothing in PipelineConfiguration configures endpointing and ListeningChanged is derived from transcript.IsInterim, so the decision is entirely engine-side and opaque from the client. A wall-clock endpointer would show none of this. The check is to speak a fixed phrase containing a deliberate mid-phrase pause, then go silent, at several speeds: if the wait for the final transcript scales with the factor the mechanism is real, and if the phrase splits at the pause at 1x but not at 1.5x, compression is also suppressing premature endpointing. Both effects are the same mechanism pulling in opposite directions.

Discarding every other sample was tried first and measurably degraded transcription. Two separate faults compound:

WSOLA avoids both. It resamples nothing, so there is no aliasing, and it preserves pitch and formants, so the spectral envelope the model was trained on survives.

Overlap-add with a similarity search. Output frames advance by a fixed synthesis hop while the read position advances through the input by hop * speed, which is where the compression comes from. Each frame is Hann-windowed and overlap-added onto the previous frame’s tail at 50% overlap; the periodic Hann sums to exactly one there, so the process has unity gain and cannot overshoot into clipping.

The "WS" is the part that matters for quality. Rather than reading each frame at its nominal position, the algorithm searches +/- a search radius for the offset whose waveform best continues what was already emitted, scored by cross-correlation normalized by candidate energy. That keeps pitch pulses aligned across the overlap. Plain overlap-add without the search misaligns them and they partially cancel, which is heard as a warble and read by an STT model as noise.

The first frame has nothing to continue, so it is taken at its nominal position and emitted unwindowed. Windowing it would fade the start of every utterance in over a hop, and since Flush resets the state, "every utterance" means every push-to-talk press and every reopening of the mute gate.

The constants are a latency-versus-quality trade, and the floor is set by voice pitch, not by the code:

Both constants were swept against the log-mel sequence an STT frontend derives from the output, scored against an ideally time-scaled reference. Spectral error is flat across windows of 16-40 ms and radii of 4-16 ms; what grows with the radius is how far the read position wanders from uniform timing. Neither constant is worth tuning further — the only knob that measurably changes quality is the speed factor itself, roughly linearly in it.

Added latency is window + searchRadius, which is 768 samples or 32 ms at 24 kHz — below the granularity at which the microphone already delivers chunks, so it is not the dominant delay in the path. It cannot be reduced much further without shortening the window below a pitch period and losing the alignment the search exists to find.

The input buffer holds only what those two constants require and is compacted on every call. Because input is consumed faster than output is emitted, it does not grow with utterance length.

[[Compressing speech only]] == Compressing speech only

With the MicSpeedUpSpeechOnly dev setting on (off by default, listed under the speed slider), each frame is classified before it is consumed and only speech is shortened. A silence frame skips both the shortening and the similarity search: it is read at its nominal position and advances by the synthesis hop, so consecutive silence frames overlap exactly the samples the Hann pair sums to one over and the region is reproduced sample for sample. Skipping the search is the part that makes it exact — at a hop ratio of one the search still wanders, which would rewrite silence without shortening it.

The gate is SpeechGate in Promethist.Core: frame energy against a noise floor that falls quickly toward quiet frames and rises very slowly, with an 8 dB margin and a three-frame hangover. It decides one frame at a time and leaves framing to the caller, because the compressor’s frames sit at non-uniform read positions while everything else feeds it a steady stream. A fixed threshold cannot work: Microphone is started with processing on, which enables automatic gain control on all three platforms (AVAudioSessionModeVoiceChat and VoiceProcessingIO, AutomaticGainControl, autoGainControl), and AGC continuously renormalises level so the noise floor climbs toward speech whenever the room goes quiet. Measured against ground truth, a fixed threshold leaves almost no silence uncompressed below 30 dB SNR, while the tracked floor keeps 0.84-1.00 of true silence at real length and compresses 0.68-0.79 of true speech from 20 dB SNR down to 0 dB, in stationary noise and babble alike.

Both error directions are harmless, which is why an energy gate is enough here. Misreading silence as speech compresses a stretch of silence and gives back part of the endpointing benefit; misreading speech as silence passes that stretch through at its original length. Neither drops or damages audio, so the gate is tuned to give up rather than guess — in heavy babble at 0 dB it classifies almost nothing as speech and the stream simply comes through uncompressed. Note that platform noise suppression is what makes the gate usable at low SNR at all: it lowers the floor and restores the contrast the gate reads.

The cost is compression ratio. Effective shortening is 1 / (p / speed + (1 - p)) for a speech fraction p, so at 1.3 an utterance that is 90% speech still gets 1.26 while one that is half silence gets 1.13. Push-to-talk audio is mostly speech and keeps most of the benefit; an open microphone is where the ratio collapses, and that is also where compressing silence does the most harm. One further consequence: the stream-to-real-time mapping stops being uniform, so unwarping STT word timestamps needs the gate’s own decisions and is no longer a single divide.

MicrophoneDebugUI is a debug section showing what the microphone is handing over: state, format, chunk size and rate, capture throughput, and bytes in against bytes out with their ratio. That ratio is the only honest measure of the compression achieved, since neither the nominal speed nor the gate’s own numbers account for the uncompressed drain at each utterance end.

It also carries a speech/silence badge and a level graph coloured against the gate threshold, with the noise floor drawn as a marker. The section runs its own SpeechGate over the raw microphone chunks rather than reading the compressor’s, so the indicator works whatever the settings are — which is the point, since the gate is worth watching in a real room before the speed-up is switched on at all. Its frames carry across chunk boundaries: chunk sizes are per-platform and iOS can deliver fewer samples than one frame holds, which would otherwise leave the gate never updating.

Because Microphone gates the stream on mute rather than zero-filling it, nothing arrives while muted and the badge reads NO INPUT rather than showing a stale decision. In push-to-talk that means the gate only moves while the button is held.

The engine’s input handling has two gates that measure the stream rather than the wall clock, and both therefore misbehave on a compressed stream: echo suppression discards a fixed span of input after each agent turn starts, and a first-turn gate drops everything ahead of the first agent turn. A compressed stream packs speed times more speech into any span the engine counts in stream samples, so at 2x the echo suppression eats twice the user’s speech it was tuned to eat.

When the configured speed will actually compress (WsolaTimeStretch.CanCompress, the same arithmetic that decides the bypass), session creation therefore sends two extra parameters, both prescribed by the engine team’s notes on consuming a compressed stream. At 1x neither is sent and the handshake stays byte-identical to the no-compression baseline.

Two more engine behaviours are worth knowing but need no client change. A wall-clock gap of more than 3 s between audio frames tears down the STT socket mid-utterance; the client sends one frame per microphone chunk whatever the compression, which is the safe shape (same cadence, smaller frames). And ignoreShortAudioBursts applies a threshold 4x shorter than the value passed; the client leaves it at its default, off.

PipelineConfiguration.InputSampleRate is set from ConversationManager.MicrophoneSampleRate when the session is created, so the engine is told the rate the microphone actually runs at instead of falling back to its own default. It had been left unassigned, and a default other than 24000 would have produced exactly the pitch-shift pathology described above. Compression does not affect it: WSOLA shortens the stream without resampling, so the declared rate stays 24000 at every speed factor.