Web (WebGL) Workarounds

Unity registers its keyboard listeners with useCapture: true, so stopPropagation from the DOM input is too late — Unity would still swallow the keystrokes. While an input is focused we therefore disable Unity’s capture with WebGLInput.captureAllKeyboardInput = false so characters reach the DOM element (SystemNative.NativeTextInput.SetKeyboardCaptureSuppressed, driven by the single focused field the registry tracks).

Focus is the right scope, and visibility — which an earlier version used — was wrong twice over: DOM focus always precedes the first keydown, so focus is early enough; and keying it to visibility disabled every Unity keyboard shortcut on desktop for as long as a chat input was merely on screen.

Tapping a UITK button (e.g. the chat send button) moves browser focus to the Unity canvas, which blurs the DOM input and dismisses the software keyboard. We keep the keyboard open without a DOM button:

While an input is focused, a capture-phase mousedown/touchstart listener on document calls preventDefault() for taps at or below the focused input’s top edge (the row where send/submit buttons sit). preventDefault() on mousedown/touchstart cancels the browser’s focus-steal, so the input keeps focus and the keyboard stays open — while Unity still receives the event (preventDefault does not stop propagation) and fires the button click.

Regular (non-BetterTextField) UITK text fields still exist elsewhere in the app. On mobile web, focusing one triggers Unity’s TouchScreenKeyboard, which calls the built-in JS_MobileKeyboard_* functions. WebKeyboard.jslib overrides those to spawn our own DOM <input>/<textarea> (positioned via WebKeyboard_SetTargetRect, with blur/hide handling and Enter-to-close), giving those fields the same native typing behaviour. WebKeyboard_IsDesktop lets C# skip this path on desktop, where UITK direct input is acceptable.

WebKit skips its scroll-to-focus when the focused input has opacity: 0 at focus time. So we blank the input for a brief window as it’s focused, then restore it. This lives in Assets/WebGLTemplates/Client/index.html:

The html.ios class is added by a small UA check in index.html (covering iPadOS 13+, which reports a desktop UA). The rule targets any focused input, so it covers both BetterTextField overlays and the mobile-keyboard bridge inputs.

Three details each fix a specific failure mode that earlier attempts hit:

The input is invisible for ~100ms after focus. If that blink is noticeable, shorten the duration (e.g. 0.05s) or soften the tail (0%, 70% { opacity: 0; } 100% { opacity: 1; }) — at the risk of re-introducing the scroll if the window becomes too short on some device. Test on real hardware after any change here.

index.html pins the page so the browser can never scroll or rubber-band it:

The viewport meta uses viewport-fit=cover (render behind the notch/chrome) and interactive-widget=overlays-content (let the keyboard overlay the layout viewport instead of resizing it), set from arc-viewport.js.

arc-viewport.js also locks the canvas CSS height to the large-viewport height at load. Chrome shrinks the large viewport by the mini URL-bar height (~24px) when the keyboard opens; without the lock that change reaches Unity’s ResizeObserver and triggers an expensive buffer resize / layout shift. The lock is refreshed on orientationchange (after a short settle delay).

arc-viewport.js publishes insets on window.__arcViewport (top, bottom, width, height, in CSS px). The bottom obstruction has two disjoint zones inside the 100vh canvas:

The total is their sum. On browsers that honour overlaysContent the zones are disjoint; on Safari (no VK API, shrinks the layout viewport) Zone A collapses to 0 and the shrink term absorbs everything, so the same formula degenerates correctly.

ScreenBridge.jslib exposes __arcViewport to C# in device pixels (multiplying by devicePixelRatio) so the values line up with Screen.width/Screen.height. SafeAreaContainer.cs polls ScreenBridge_GetInsetTop/Bottom each frame and eases its content margins to sit inside the safe area (e.g. lifting above the keyboard). UI screens just drop their content into a <pui:SafeAreaContainer> instead of handling insets themselves.

ScreenHandler.cs handles one Web quirk:

The bottom inset can sample too large on phones at startup, but SafeAreaContainer re-polls every frame, so the value self-corrects within a frame or two.

arc-fullscreen.js can hide the browser navigation bar by entering fullscreen on the first touchend (a spec "activation-triggering" event; captured in the capture phase so Unity’s canvas handlers can’t eat the bubble first). It is gated behind ENABLE_FULLSCREEN_ON_TOUCH, which is currently false.

So the picker is shown even when the project offers a single provider, rather than taking the lone entry and going straight on. Its click is the last thing before the popup, which is what makes the popup open — the multi-provider path already worked for exactly this reason, while the single-provider one was a race against the five seconds and the deep-link start lost it outright.

Anything that reintroduces a shortcut past the picker — an auto-pick, a remembered choice, a retry that skips it — has to open the popup from its own gesture instead, or it brings the block back. The one exception already in place is the retry on the login-failed prompt: that button is itself the gesture, and it re-enters the loop directly at WebView.OpenAuth.

The Web player is built for WebGPU alone — Player Settings ▸ Web ▸ Graphics APIs holds a single entry, WebGPU — so there is no WebGL2 to fall back to when the browser cannot hand out an adapter. Unity’s loader does not report that case: UnityLoader.js checkForWebGPU() records navigator.gpu.requestAdapter() returning null as SystemInfo.hasWebGPU = false and loads the build anyway, and its only refusal messages are for WebGL, WebGL 2 and WebAssembly. The engine then fails to create GfxDeviceWebGPU, which surfaces as the generic "The experience couldn’t be loaded", and the only mention of an adapter is the browser’s own console line. In telemetry that failure is indistinguishable from a bad download.

So index.html asks for an adapter itself, before createUnityInstance, and names the cause:

high-performance is what the build itself asks for (Player Settings ▸ Web ▸ Power Preference), so the pre-flight fails exactly where the engine would. A refusal is retried once without a preference, because a machine with switchable graphics can decline the discrete GPU while still having an integrated adapter to give; that retry logs a distinct warning, which is what separates a hybrid-GPU machine from one where WebGPU is gone altogether. Only a null from both stops the load.

The two failures read differently to the user, so they carry different copy: a missing navigator.gpu is a browser too old for WebGPU, while a present navigator.gpu that yields no adapter is almost always Chrome having turned hardware acceleration off for the profile — after repeated GPU-process crashes, a blocklisted driver, or a session already running on software. That state does not survive a browser restart, which is why the copy asks for one; a page reload alone will not clear it.

The consequence worth internalising: the listener is on window, so any uncaught error or unhandled promise rejection anywhere on the page puts up a full-screen "The experience couldn’t be loaded" — including one thrown by our own .jslib code long after the experience loaded fine.

The Web Audio API is where this bites. AudioContext’s `resume, suspend and close report a context that cannot honour them by rejecting their promise, not by throwing, so wrapping the call in try/catch looks careful and catches nothing. Every such call in a .jslib must settle its promise:

The p && p.then guard is not decoration — older Safari returns undefined from these methods rather than a promise. WebAudio.jslib hangs this as st.settle on its state object so the entry points share one copy.

The same applies to anything else promise-shaped a plugin touches — getUserMedia, navigator.share, clipboard.writeText, requestFullscreen, audioWorklet.addModule — and to a .then() whose callback can throw, since that throw becomes a rejection of the promise .then() returned.

Browsers will not start an AudioContext that was created outside a user gesture; iOS Safari is the strictest. Three rules follow, and all three are load-bearing:

WA_Init is idempotent and owns the whole context lifecycle. It rebuilds only when the requested sample rate differs from the current context’s (an AudioContext sample rate is fixed at construction), and a rebuild re-runs the unlock path.

A rebuild also invalidates any audioWorklet.addModule() still in flight, so the resolve handler checks a generation counter before building its node. Without that check the stale promise builds a second AudioWorkletNode on the current context, doubling both playback and the played-sample counter that lipsync reads.

Starting a capture stream flips the iOS audio session to play-and-record and interrupts existing contexts. In practice that means granting the microphone permission silences playback, and the permission prompt appears at almost exactly the same moment as the first speech (ConversationManager.BeginConversation sends the intro request and starts the mic together).

Recovery is centralised in st.ensureRunning(), called from the gesture listeners, the context’s own onstatechange (iOS also has an interrupted state), on returning to the foreground, and from Microphone.jslib once getUserMedia resolves. It refuses to resume while an explicit WA_Pause or a backgrounded page is in effect, so the recovery paths never fight a deliberate pause.

Native builds pause on background for free: the engine suspends the whole player, which stops both the audio and the frame loop. WebGL gets neither half of that for free. Unity’s loop all but stops when the page is hidden (Rendering in the background has the caveat), but the worklet renders on the browser’s audio thread and carries on — so the agent keeps talking over a frozen app, and comes back with playback and lipsync desynchronised.

st.setHidden() suspends the context on visibilitychange/pagehide and resumes on return.

It has to live in JS. Unity is frozen at exactly the moment the decision has to be made, so C# cannot drive it — WA_Pause would never be called. Nothing else needs to be notified: everything downstream (WatchProgress, FAnimator lipsync) is derived from WA_GetPlayedSamples(), which freezes with the context, so the whole app pauses and resumes coherently on its own.

A page opened in a background tab stays silent until its first user interaction even after being revealed. That is ordinary autoplay policy, not a bug — the user has to tap to start a conversation anyway.

The claim above that the frame loop stops on its own holds only while Unity is driven by requestAnimationFrame. It is not: Application.targetFrameRate is set for every interaction, and a frame rate limit makes Emscripten schedule the loop with setTimeout instead. Browsers throttle that to about a tick a second in a hidden tab rather than stopping it, so the app goes on rendering full frames — GPU work, and on a laptop a fan — for a tab nobody is looking at.

QualityManager polls document.hidden through ScreenBridge_IsPageHidden and parks OnDemandRendering.renderFrameInterval at int.MaxValue while the page is away, putting it back at 1 on return. The poll rides the same throttled ticks, which is exactly enough: the decision only has to land once per visibility change, and the first tick after the tab is revealed restores rendering before a frame would be seen. Update keeps running, so nothing else in the app is paused by this — audio and the microphone are silenced separately, above.

Capture has the same problem in reverse, and it is worse than a cosmetic one: ScriptProcessorNode delivers its buffers on the main thread, which keeps running while the page is hidden. Android Chrome will happily go on streaming the user’s microphone to the engine with the phone in their pocket. (iOS usually revokes capture in the background on its own, but not dependably across versions.)

Module.MIC_SETHIDDEN in Microphone.jslib mirrors the playback side on the same events. It gates capture three ways, because each covers a different failure:

It is re-applied once getUserMedia resolves, since the page can be backgrounded while the permission prompt is still up.

WA_Stop drops the buffered audio but deliberately leaves the context running. Suspending between tracks would mean asking iOS to resume again for every single reply, outside any user gesture — a permission gamble with nothing to gain. An idle worklet renders silence, and it only posts a progress message when the played count actually moves, so an idle context is close to free.

GpuUpdaterStaged<T>.CompleteUpdate reserves staging space from a dirty count taken before the ranges are consolidated:

DataSet<T>.ConsolidateRanges(threshold = 0.9f) collapses several dirty ranges into one spanning range whenever the dirty total covers at least 90% of min..max, and then reports the span as the new total. The span can therefore be up to 1 / 0.9 = 1.111x the number the buffer was just sized against, usedCount runs past capacity, and the emitted copy reads off the end.

The arithmetic is exact. A Vertex is 72 bytes and the vertex staging tiers are 8192 and 65536 elements, so 8192 * 72 = 589824 is the reported srcSize, and a logged srcEnd of 599328 is 8324 * 72 — a reservation of at most 8192 vertices grown into an 8324-vertex span. The index updater has the same flaw with a second growth source, AlignIndexRange, which widens every range outward to an even boundary.

UIRenderDevice’s constructor is the only place the updater is chosen. WebGPU is the sole backend routed to `GpuUpdaterType.StagedGpuOnly; every platform with mapped buffer ranges gets MeshManagerTracked and GpuUpdaterMapped, which has no staging buffer and so no mismatch. WebGL2 falls to StagedCpuGpu, which carries the same defect but as an unchecked MemCpy past the end of a NativeArray — it corrupts silently instead of logging.

The failure needs a frame whose dirty vertex total sits just under a tier boundary with the ranges densely packed, so roughly 7,373–8,192 vertices. UI Toolkit text is four vertices per glyph, which puts the band at about 2,000 glyphs — a screenful, not an extreme.

Not only building. MeshManagerBasic.Update marks an element’s entire vertex allocation dirty whenever the element is re-tessellated or merely moved: a shifted element with unchanged geometry takes RenderEvents.NudgeVerticesToNewSpace, which rewrites every one of its vertices in place. Reflowing a list therefore dirties every item that moved, in the contiguous sequential ranges those items occupy — exactly the dense pattern consolidation collapses.

Assigning identical text is free, however: `TextElement’s setter early-outs on an unchanged string, so rebinding an unchanged item costs nothing.

RenderEvents tests the transform-ID branch before the nudge branch, so an element that owns a transform ID answers a move with a single shader-info write and leaves its vertices alone — and its descendants inherit the ID and are skipped with it. UsageHints.DynamicTransform is what allocates that ID (NeedsTransformID reads RenderHints.BoneTransform).

It is therefore set on every element that shifts as a group while its content stays put:

Nested node cards inside a session row are deliberately left without it. They move only when a fold within a row animates, which shifts far less geometry, and every transform ID costs a slot in the shader-info texture.

Separately, ConversationController rebinds one entry rather than the whole list when an event names its index, so a tool call completing no longer runs a full virtualization refresh over every visible message.

These keep the common frames well clear of the band; they cannot close it. Any frame that genuinely re-tessellates a bandful of geometry can still land on it. The deterministic options are an IL patch of the shipped UnityEngine.UIElementsModule.dll to over-reserve in FindOrAllocateBuffer, or an engine fix — still absent as of 6000.5.9f1, and still absent in 6000.6.0b8.