Skip to content
DevLab

Devices, quality and mode

Which mode suits your people's devices, what depends on the device, how to check it before measuring and how to read the session rate after.

On this page
  1. To decide
  2. Where the video goes
  3. What depends on the device
  4. Before you start
  5. Check the device
  6. If speech cannot come
  7. During and after
  8. For the person being measured
Which one do I choose?
If…Choose
your people use a desktop or recent phones and you want results during the session (for short captures; long ones take minutes to close)On-device (real time)
the video must not leave the deviceOn-device (real time)
you use modest phones, sessions of several minutes, or the rate of your sessions stays lowDeferred (server-side): today you can record, upload and process with Neurologyca's processing service, with nothing to deploy.
you do not know which devices your people will use, or their phones vary a lot, and you want the most stable qualityDeferred (server-side) (Preview): the video leaves the device and is uploaded and processed on the server, so you wait for the result after the upload. Your organization has to have it turned on.
you want real time by default and a way out when the device cannot keep upOn-device (real time) by default, and Deferred (server-side) (Preview), with the person's permission, when the device check says it cannot measure or cannot transcribe. Your organization has to have it turned on.
the video must be processed off the device but inside your infrastructureSelf-hosted (coming soon); until then, deferred with our service.
you capture in the browser and then query from your backend (results, polling, catalogue)SDK + API (the usual)
you capture from another language or platform, or you already have the framesHTTP API
you only want to measure, without interpretationMeasurement, on-device (real time) (deferred also works since 0.27.0)
you are not sureThe default path: SDK on-device (real time), starting with Measurement

On-device, video and audio never leave the device. Deferred (server-side) is Preview and works only if your organization has it turned on: it uploads the video. What each mode sends: Where it is processed.

Decide by capability, never by the browser or the phone model:

Level Verdict What to do
Can measure 'ok' Measure
Can measure with limits 'degraded' (live: wasm-simd) Measure; the rate tells you after
Cannot measure 'unsupported' The remedy of the failing check; if it is capture-route, offer deferred
  • The rate depends on MediaPipe’s face detection on the GPU (WebGL 2). With only the face, CPU is not necessarily slower, but its landmarks differ: compare sessions with the same delegate.
  • Without WebGPU, transcription is refused (blocker: 'no-webgpu') unless you allow the slower transcribe.allowWasmWorker, or offer deferred instead. Transcribing costs rate.
  • Hands and body come off: they cost rate and Engine discards them.
  • Phones slow down when hot; the first seconds do not count.
  • iOS cannot turn off the microphone’s automatic gain; Safari and iOS keep only the face between sessions.
  • Private mode: no persistent cache; a recording lives in memory (recording.storage-memory).
  • Deferred: quality depends on the bitrate and the light, not on the device’s power; phones record in portrait (recording.handheld).
  • kodec’s guidance: WebGL 2 GPU, 8 GB of memory, 720p camera.

It is capability, not quality: the rate is only known when measuring. A prepare() with no model loads nothing: it checks this browser and device, without camera or permission, and gives capture ('ok', 'degraded', 'unsupported' or null) and one DeviceCheck per check. captureCacheStatus() says if speech’s weights are cached; recorder.prepare() adds indexeddb and storage-quota.

JavaScript
// Before measuring: no permission, no call to Engine. Without `keepPrepared`, call it before
// preparing: a prepare() with other options releases the current one.
/** @typedef {import('@neurologyca/capture').DeviceCheck} DeviceCheck */
/** @typedef {'live' | 'record'} Mode */
// A `fail` blocks the capture. Recording also needs MP4 and IndexedDB.
const BLOCKS = {
live: new Set(['secure-context', 'get-user-media', 'capture-route']),
record: new Set(['secure-context', 'get-user-media', 'capture-route', 'media-recorder-mp4', 'indexeddb']),
}
// `warn` or `fail` limits it; storage only when recording. Speech checks never limit it.
const LIMITS = {
live: new Set(['wasm-simd']),
record: new Set(['wasm-simd', 'storage-persisted', 'storage-quota']),
}
/**
* Reads a device diagnosis: the SDK's verdict, what blocks, what limits and whether speech can come.
* @param {{ capture: 'ok' | 'degraded' | 'unsupported' | null, device: readonly DeviceCheck[] }} r what prepare() settled with
* @param {Mode} mode `'live'` or `'record'` (deferred mode, Preview)
* @returns {{ verdict: 'ok' | 'degraded' | 'unsupported' | null, blocking: DeviceCheck[], limiting: DeviceCheck[], speech: 'possible' | 'impossible' | 'unknown' }}
*/
export function readDiagnosis(r, mode) {
const blocking = r.device.filter((c) => BLOCKS[mode].has(c.id) && c.status === 'fail')
const limiting = r.device.filter((c) => LIMITS[mode].has(c.id) && (c.status === 'warn' || c.status === 'fail'))
// The WebGPU adapter: only checked when speech is asked for (with no model, 'unknown').
const gpu = r.device.find((c) => c.id === 'webgpu-adapter')?.status
const speech = gpu === 'fail' ? 'impossible' : gpu === 'pass' ? 'possible' : 'unknown'
// null: a deciding check did not answer; the capture checks again.
return { verdict: r.capture, blocking, limiting, speech }
}
/** @typedef {import('@neurologyca/capture').ProtocolClientHandle} ProtocolClientHandle */
/** @typedef {import('@neurologyca/capture/record').RecordingClientHandle} RecordingClientHandle */
/** @typedef {ReturnType<typeof readDiagnosis>} Diagnosis */
/**
* Diagnoses the device with no model. `'live'` (default) with a live client, `'record'` with a recording client.
* @overload
* @param {ProtocolClientHandle} client
* @param {'live'} [mode]
* @returns {Promise<Diagnosis>}
*/
/**
* @overload
* @param {RecordingClientHandle} client
* @param {'record'} mode
* @returns {Promise<Diagnosis>}
*/
/**
* @param {ProtocolClientHandle | RecordingClientHandle} client
* @param {Mode} [mode]
* @returns {Promise<Diagnosis>}
*/
export async function checkDevice(client, mode = 'live') {
const result = mode === 'record' ? await client.prepare() : await client.prepare({ models: { face: false, speech: false } })
return readDiagnosis(result, mode)
}

Whether speech will come is only known when preparing speech or when transcription starts: a check with no model cannot tell. If speech is refused (transcription: 'unavailable'), offer to record, with permission (deferred, Preview, if your organization has it on): it transcribes on the server. With WebGPU, asking for speech downloads the face (about 13.5 MB) and speech (about 160 MB).

JavaScript
// Speech is only known by asking for it; deferred is Preview, only if your organization has it turned on.
import { checkDevice, readDiagnosis } from './check-device.mjs'
/**
* @param {object} o
* @param {import('@neurologyca/capture').ProtocolClientHandle} o.client your live client, with `transcribe`
* @param {import('@neurologyca/capture/record').RecordingClientHandle} o.recorder your recording client
* @param {() => Promise<boolean>} o.askToRecord your UI: asks consent to upload the video and process it on the server
* @returns {Promise<{ mode: 'live', speech: boolean } | { mode: 'record' } | { mode: 'none', blocking: import('@neurologyca/capture').DeviceCheck[] }>}
*/
export async function chooseMode({ client, recorder, askToRecord }) {
const live = await checkDevice(client, 'live')
const canLive = live.verdict !== 'unsupported'
if (canLive) {
// Downloads the face and speech models and waits. The next capture adopts them; with keepPrepared,
// later ones too, and asking for another part releases them.
const prepared = await client.prepare({ models: { face: true, speech: true } })
if (prepared.transcription !== 'unavailable' && readDiagnosis(prepared, 'live').speech !== 'impossible') return { mode: 'live', speech: true }
}
// Only diagnoses: no load, no camera.
const rec = await checkDevice(recorder, 'record')
const canRecord = rec.verdict === 'ok' || rec.verdict === 'degraded'
if (canRecord && (await askToRecord())) {
client.release() // the recording takes over
return { mode: 'record' }
}
// Not recording: live without speech, or what blocks.
return canLive ? { mode: 'live', speech: false } : { mode: 'none', blocking: live.blocking }
}

Read the session rate from onCodecEvent('capture-quality'): about one window per second, a kodec.CaptureQuality. Describe the rate, never grade it: without the first two windows, the hidden ones or those with fps: null, give the median (if even, the mean of the two middle ones), the windows below 8 fps and the lowest; round only to paint.

8 fps is the only floor the SDK publishes. Below it, it does not estimate the pulse (an optional channel, off by default) and it treats a transcription veto as provisional. There is no published threshold for how the rate affects the indicators: compare your own sessions.

JavaScript
// After measuring: the session rate, described, never graded. Feed it the 'capture-quality' windows.
/** @typedef {{ fps: number | null, hidden: boolean }} RateWindow */
/** @typedef {'counted' | 'low' | 'skipped' | 'hidden'} WindowState */
/**
* @typedef {object} SessionRate
* @property {number} counted the windows that count (low ones included)
* @property {number | null} median the median fps of those windows; null without any
* @property {number} lowWindows how many of them fall below the floor
* @property {number | null} lowest the lowest fps of them; null without any
* @property {'steady' | 'low' | 'none'} verdict `'none'` without a counted window, `'low'` with a low one, `'steady'` otherwise
* @property {readonly { fps: number | null, state: WindowState }[]} windows every window in order, at most 600
*/
const STARTUP = 2 // the first two windows are the startup
const MAX_WINDOWS = 600 // the strip keeps 600 (ten minutes); the counts cover all
/**
* @param {readonly RateWindow[]} windows
* @param {number} floor the floor, kodec.PULSE_MIN_FPS
* @returns {SessionRate}
*/
export function sessionRate(windows, floor) {
/** @type {{ fps: number | null, state: WindowState }[]} */
const strip = []
/** @type {number[]} */
const fps = []
windows.forEach((w, i) => {
/** @type {WindowState} */
let state
if (i < STARTUP || w.fps === null || !Number.isFinite(w.fps)) state = 'skipped'
else if (w.hidden) state = 'hidden' // a hidden tab is throttled: it says nothing of the device
else { state = w.fps < floor ? 'low' : 'counted'; fps.push(w.fps) }
if (strip.length < MAX_WINDOWS) strip.push({ fps: w.fps, state })
})
const lowWindows = fps.filter((x) => x < floor).length
return {
counted: fps.length,
median: median(fps),
lowWindows,
lowest: fps.length ? Math.min(...fps) : null,
verdict: fps.length === 0 ? 'none' : lowWindows > 0 ? 'low' : 'steady',
windows: strip,
}
}
/** @param {number[]} xs @returns {number | null} */
function median(xs) {
if (!xs.length) return null
const a = [...xs].sort((x, y) => x - y)
const mid = a.length >> 1
return a.length % 2 ? a[mid] : (a[mid - 1] + a[mid]) / 2
}

droppedTotal above zero is a separate notice; delegate, a fact to compare. minFps and maxFps include the startup and the hidden tab. Not the rate either: capture.throttled, relative to the session’s own rate, and quality.warning, from Engine.

Read this to the person before measuring, or show it on your capture screen:

  • Face a light source: light on your face from the front, not a window behind you.
  • Sit at about arm’s length from the camera, with your whole face in view.
  • Look at the screen and move little; you do not need to be motionless.
  • Keep this tab in front and visible until the end: a hidden tab slows the capture down.
  • Close the other apps and tabs that use the camera or the graphics card (video calls, games, editors).

And, for each notice the SDK already gives (onWarning), the sentence for the person:

Notice What to tell the person
quality.warning The image is not good enough. Face the light, bring your face into view and keep still.
source.ended The video ended, so this part has finished.
capture.throttled The capture slowed down. Keep this tab in front and visible, and close what you do not need.
recording.rotate-to-portrait Turn your phone upright: the recording only starts in portrait.

The rest of the person’s conditions, and what Engine checks: Capturing for the best accuracy.