Devices, quality and mode
Which mode suits your people's devices, what depends on the device, how to check it before measuring and how to read the session rate after.
To decide
Section titled “To decide”| If… | Choose |
|---|---|
| your people use a desktop or recent phones and you want results during the session (for short captures; long ones take minutes to close) | On-device (real time) |
| the video must not leave the device | On-device (real time) |
| you use modest phones, sessions of several minutes, or the rate of your sessions stays low | Deferred (server-side): today you can record, upload and process with Neurologyca's processing service, with nothing to deploy. |
| you do not know which devices your people will use, or their phones vary a lot, and you want the most stable quality | Deferred (server-side) (Preview): the video leaves the device and is uploaded and processed on the server, so you wait for the result after the upload. Your organization has to have it turned on. |
| you want real time by default and a way out when the device cannot keep up | On-device (real time) by default, and Deferred (server-side) (Preview), with the person's permission, when the device check says it cannot measure or cannot transcribe. Your organization has to have it turned on. |
| the video must be processed off the device but inside your infrastructure | Self-hosted (coming soon); until then, deferred with our service. |
| you capture in the browser and then query from your backend (results, polling, catalogue) | SDK + API (the usual) |
| you capture from another language or platform, or you already have the frames | HTTP API |
| you only want to measure, without interpretation | Measurement, on-device (real time) (deferred also works since 0.27.0) |
| you are not sure | The default path: SDK on-device (real time), starting with Measurement |
Where the video goes
Section titled “Where the video goes”On-device, video and audio never leave the device. Deferred (server-side) is Preview and works only if your organization has it turned on: it uploads the video. What each mode sends: Where it is processed.
What depends on the device
Section titled “What depends on the device”Decide by capability, never by the browser or the phone model:
| Level | Verdict | What to do |
|---|---|---|
| Can measure | 'ok' |
Measure |
| Can measure with limits | 'degraded' (live: wasm-simd) |
Measure; the rate tells you after |
| Cannot measure | 'unsupported' |
The remedy of the failing check; if it is capture-route, offer deferred |
- The rate depends on MediaPipe’s face detection on the GPU (WebGL 2). With only the face, CPU is not necessarily slower, but its landmarks differ: compare sessions with the same
delegate. - Without WebGPU, transcription is refused (
blocker: 'no-webgpu') unless you allow the slowertranscribe.allowWasmWorker, or offer deferred instead. Transcribing costs rate. - Hands and body come off: they cost rate and Engine discards them.
- Phones slow down when hot; the first seconds do not count.
- iOS cannot turn off the microphone’s automatic gain; Safari and iOS keep only the face between sessions.
- Private mode: no persistent cache; a recording lives in memory (
recording.storage-memory). - Deferred: quality depends on the bitrate and the light, not on the device’s power; phones record in portrait (
recording.handheld). - kodec’s guidance: WebGL 2 GPU, 8 GB of memory, 720p camera.
Before you start
Section titled “Before you start”It is capability, not quality: the rate is only known when measuring. A prepare() with no model loads nothing: it checks this browser and device, without camera or permission, and gives capture ('ok', 'degraded', 'unsupported' or null) and one DeviceCheck per check. captureCacheStatus() says if speech’s weights are cached; recorder.prepare() adds indexeddb and storage-quota.
Check the device
Section titled “Check the device”// Before measuring: no permission, no call to Engine. Without `keepPrepared`, call it before// preparing: a prepare() with other options releases the current one.
/** @typedef {import('@neurologyca/capture').DeviceCheck} DeviceCheck *//** @typedef {'live' | 'record'} Mode */
// A `fail` blocks the capture. Recording also needs MP4 and IndexedDB.const BLOCKS = { live: new Set(['secure-context', 'get-user-media', 'capture-route']), record: new Set(['secure-context', 'get-user-media', 'capture-route', 'media-recorder-mp4', 'indexeddb']),}// `warn` or `fail` limits it; storage only when recording. Speech checks never limit it.const LIMITS = { live: new Set(['wasm-simd']), record: new Set(['wasm-simd', 'storage-persisted', 'storage-quota']),}
/** * Reads a device diagnosis: the SDK's verdict, what blocks, what limits and whether speech can come. * @param {{ capture: 'ok' | 'degraded' | 'unsupported' | null, device: readonly DeviceCheck[] }} r what prepare() settled with * @param {Mode} mode `'live'` or `'record'` (deferred mode, Preview) * @returns {{ verdict: 'ok' | 'degraded' | 'unsupported' | null, blocking: DeviceCheck[], limiting: DeviceCheck[], speech: 'possible' | 'impossible' | 'unknown' }} */export function readDiagnosis(r, mode) { const blocking = r.device.filter((c) => BLOCKS[mode].has(c.id) && c.status === 'fail') const limiting = r.device.filter((c) => LIMITS[mode].has(c.id) && (c.status === 'warn' || c.status === 'fail')) // The WebGPU adapter: only checked when speech is asked for (with no model, 'unknown'). const gpu = r.device.find((c) => c.id === 'webgpu-adapter')?.status const speech = gpu === 'fail' ? 'impossible' : gpu === 'pass' ? 'possible' : 'unknown' // null: a deciding check did not answer; the capture checks again. return { verdict: r.capture, blocking, limiting, speech }}
/** @typedef {import('@neurologyca/capture').ProtocolClientHandle} ProtocolClientHandle *//** @typedef {import('@neurologyca/capture/record').RecordingClientHandle} RecordingClientHandle *//** @typedef {ReturnType<typeof readDiagnosis>} Diagnosis */
/** * Diagnoses the device with no model. `'live'` (default) with a live client, `'record'` with a recording client. * @overload * @param {ProtocolClientHandle} client * @param {'live'} [mode] * @returns {Promise<Diagnosis>} *//** * @overload * @param {RecordingClientHandle} client * @param {'record'} mode * @returns {Promise<Diagnosis>} *//** * @param {ProtocolClientHandle | RecordingClientHandle} client * @param {Mode} [mode] * @returns {Promise<Diagnosis>} */export async function checkDevice(client, mode = 'live') { const result = mode === 'record' ? await client.prepare() : await client.prepare({ models: { face: false, speech: false } }) return readDiagnosis(result, mode)}If speech cannot come
Section titled “If speech cannot come”Whether speech will come is only known when preparing speech or when transcription starts: a check with no model cannot tell. If speech is refused (transcription: 'unavailable'), offer to record, with permission (deferred, Preview, if your organization has it on): it transcribes on the server. With WebGPU, asking for speech downloads the face (about 13.5 MB) and speech (about 160 MB).
// Speech is only known by asking for it; deferred is Preview, only if your organization has it turned on.import { checkDevice, readDiagnosis } from './check-device.mjs'
/** * @param {object} o * @param {import('@neurologyca/capture').ProtocolClientHandle} o.client your live client, with `transcribe` * @param {import('@neurologyca/capture/record').RecordingClientHandle} o.recorder your recording client * @param {() => Promise<boolean>} o.askToRecord your UI: asks consent to upload the video and process it on the server * @returns {Promise<{ mode: 'live', speech: boolean } | { mode: 'record' } | { mode: 'none', blocking: import('@neurologyca/capture').DeviceCheck[] }>} */export async function chooseMode({ client, recorder, askToRecord }) { const live = await checkDevice(client, 'live') const canLive = live.verdict !== 'unsupported' if (canLive) { // Downloads the face and speech models and waits. The next capture adopts them; with keepPrepared, // later ones too, and asking for another part releases them. const prepared = await client.prepare({ models: { face: true, speech: true } }) if (prepared.transcription !== 'unavailable' && readDiagnosis(prepared, 'live').speech !== 'impossible') return { mode: 'live', speech: true } }
// Only diagnoses: no load, no camera. const rec = await checkDevice(recorder, 'record') const canRecord = rec.verdict === 'ok' || rec.verdict === 'degraded' if (canRecord && (await askToRecord())) { client.release() // the recording takes over return { mode: 'record' } } // Not recording: live without speech, or what blocks. return canLive ? { mode: 'live', speech: false } : { mode: 'none', blocking: live.blocking }}During and after
Section titled “During and after”Read the session rate from onCodecEvent('capture-quality'): about one window per second, a kodec.CaptureQuality. Describe the rate, never grade it: without the first two windows, the hidden ones or those with fps: null, give the median (if even, the mean of the two middle ones), the windows below 8 fps and the lowest; round only to paint.
8 fps is the only floor the SDK publishes. Below it, it does not estimate the pulse (an optional channel, off by default) and it treats a transcription veto as provisional. There is no published threshold for how the rate affects the indicators: compare your own sessions.
// After measuring: the session rate, described, never graded. Feed it the 'capture-quality' windows.
/** @typedef {{ fps: number | null, hidden: boolean }} RateWindow *//** @typedef {'counted' | 'low' | 'skipped' | 'hidden'} WindowState *//** * @typedef {object} SessionRate * @property {number} counted the windows that count (low ones included) * @property {number | null} median the median fps of those windows; null without any * @property {number} lowWindows how many of them fall below the floor * @property {number | null} lowest the lowest fps of them; null without any * @property {'steady' | 'low' | 'none'} verdict `'none'` without a counted window, `'low'` with a low one, `'steady'` otherwise * @property {readonly { fps: number | null, state: WindowState }[]} windows every window in order, at most 600 */
const STARTUP = 2 // the first two windows are the startupconst MAX_WINDOWS = 600 // the strip keeps 600 (ten minutes); the counts cover all
/** * @param {readonly RateWindow[]} windows * @param {number} floor the floor, kodec.PULSE_MIN_FPS * @returns {SessionRate} */export function sessionRate(windows, floor) { /** @type {{ fps: number | null, state: WindowState }[]} */ const strip = [] /** @type {number[]} */ const fps = [] windows.forEach((w, i) => { /** @type {WindowState} */ let state if (i < STARTUP || w.fps === null || !Number.isFinite(w.fps)) state = 'skipped' else if (w.hidden) state = 'hidden' // a hidden tab is throttled: it says nothing of the device else { state = w.fps < floor ? 'low' : 'counted'; fps.push(w.fps) } if (strip.length < MAX_WINDOWS) strip.push({ fps: w.fps, state }) }) const lowWindows = fps.filter((x) => x < floor).length return { counted: fps.length, median: median(fps), lowWindows, lowest: fps.length ? Math.min(...fps) : null, verdict: fps.length === 0 ? 'none' : lowWindows > 0 ? 'low' : 'steady', windows: strip, }}
/** @param {number[]} xs @returns {number | null} */function median(xs) { if (!xs.length) return null const a = [...xs].sort((x, y) => x - y) const mid = a.length >> 1 return a.length % 2 ? a[mid] : (a[mid - 1] + a[mid]) / 2}droppedTotal above zero is a separate notice; delegate, a fact to compare. minFps and maxFps include the startup and the hidden tab. Not the rate either: capture.throttled, relative to the session’s own rate, and quality.warning, from Engine.
For the person being measured
Section titled “For the person being measured”Read this to the person before measuring, or show it on your capture screen:
- Face a light source: light on your face from the front, not a window behind you.
- Sit at about arm’s length from the camera, with your whole face in view.
- Look at the screen and move little; you do not need to be motionless.
- Keep this tab in front and visible until the end: a hidden tab slows the capture down.
- Close the other apps and tabs that use the camera or the graphics card (video calls, games, editors).
And, for each notice the SDK already gives (onWarning), the sentence for the person:
| Notice | What to tell the person |
|---|---|
quality.warning |
The image is not good enough. Face the light, bring your face into view and keep still. |
source.ended |
The video ended, so this part has finished. |
capture.throttled |
The capture slowed down. Keep this tab in front and visible, and close what you do not need. |
recording.rotate-to-portrait |
Turn your phone upright: the recording only starts in portrait. |
The rest of the person’s conditions, and what Engine checks: Capturing for the best accuracy.