# Devices, quality and mode

## To decide

| If… | Choose |
| --- | --- |
| your people use a desktop or recent phones and you want results during the session (for short captures; long ones take minutes to close) | **On-device (real time)** |
| the video must not leave the device | **On-device (real time)** |
| you use modest phones, sessions of several minutes, or the rate of your sessions stays low | **Deferred (server-side)**: today you can record, upload and process with Neurologyca's processing service, with nothing to deploy. |
| you do not know which devices your people will use, or their phones vary a lot, and you want the most stable quality | **Deferred (server-side)** (Preview): the video leaves the device and is uploaded and processed on the server, so you wait for the result after the upload. Your organization has to have it turned on. |
| you want real time by default and a way out when the device cannot keep up | **On-device (real time)** by default, and **Deferred (server-side)** (Preview), with the person's permission, when the device check says it cannot measure or cannot transcribe. Your organization has to have it turned on. |
| the video must be processed off the device but inside your infrastructure | **Self-hosted (coming soon)**; until then, deferred with our service. |
| you capture in the browser and then query from your backend (results, polling, catalogue) | **SDK + API** (the usual) |
| you capture from another language or platform, or you already have the frames | **HTTP API** |
| you only want to measure, without interpretation | **Measurement**, on-device (real time) (deferred also works since 0.27.0) |
| you are not sure | The default path: **SDK on-device (real time), starting with Measurement** |

### Where the video goes

On-device, video and audio never leave the device. Deferred (server-side) is Preview and works only if your organization has it turned on: it uploads the video. What each mode sends: [Where it is processed](/docs/start/paths/#where-it-is-processed).

## What depends on the device

Decide by capability, never by the browser or the phone model:

| Level | Verdict | What to do |
| --- | --- | --- |
| Can measure | `'ok'` | Measure |
| Can measure with limits | `'degraded'` (live: `wasm-simd`) | Measure; the rate tells you after |
| Cannot measure | `'unsupported'` | The remedy of the failing check; if it is `capture-route`, offer deferred |

- The rate depends on MediaPipe's face detection on the GPU (WebGL 2). With only the face, CPU is not necessarily slower, but its landmarks differ: compare sessions with the same `delegate`.
- Without WebGPU, transcription is refused (`blocker: 'no-webgpu'`) unless you allow the slower `transcribe.allowWasmWorker`, or offer deferred instead. Transcribing costs rate.
- Hands and body come off: they cost rate and Engine discards them.
- Phones slow down when hot; the first seconds do not count.
- iOS cannot turn off the microphone's automatic gain; Safari and iOS keep only the face between sessions.
- Private mode: no persistent cache; a recording lives in memory (`recording.storage-memory`).
- Deferred: quality depends on the bitrate and the light, not on the device's power; phones record in portrait (`recording.handheld`).
- kodec's guidance: WebGL 2 GPU, 8 GB of memory, 720p camera.

## Before you start

It is capability, not quality: the rate is only known when measuring. A `prepare()` with no model loads nothing: it checks this browser and device, without camera or permission, and gives `capture` (`'ok'`, `'degraded'`, `'unsupported'` or `null`) and one `DeviceCheck` per check. `captureCacheStatus()` says if speech's weights are cached; `recorder.prepare()` adds `indexeddb` and `storage-quota`.

### Check the device

```js
// Before measuring: no permission, no call to Engine. Without `keepPrepared`, call it before
// preparing: a prepare() with other options releases the current one.

/** @typedef {import('@neurologyca/capture').DeviceCheck} DeviceCheck */
/** @typedef {'live' | 'record'} Mode */

// A `fail` blocks the capture. Recording also needs MP4 and IndexedDB.
const BLOCKS = {
  live: new Set(['secure-context', 'get-user-media', 'capture-route']),
  record: new Set(['secure-context', 'get-user-media', 'capture-route', 'media-recorder-mp4', 'indexeddb']),
}
// `warn` or `fail` limits it; storage only when recording. Speech checks never limit it.
const LIMITS = {
  live: new Set(['wasm-simd']),
  record: new Set(['wasm-simd', 'storage-persisted', 'storage-quota']),
}

/**
 * Reads a device diagnosis: the SDK's verdict, what blocks, what limits and whether speech can come.
 * @param {{ capture: 'ok' | 'degraded' | 'unsupported' | null, device: readonly DeviceCheck[] }} r what prepare() settled with
 * @param {Mode} mode `'live'` or `'record'` (deferred mode, Preview)
 * @returns {{ verdict: 'ok' | 'degraded' | 'unsupported' | null, blocking: DeviceCheck[], limiting: DeviceCheck[], speech: 'possible' | 'impossible' | 'unknown' }}
 */
export function readDiagnosis(r, mode) {
  const blocking = r.device.filter((c) => BLOCKS[mode].has(c.id) && c.status === 'fail')
  const limiting = r.device.filter((c) => LIMITS[mode].has(c.id) && (c.status === 'warn' || c.status === 'fail'))
  // The WebGPU adapter: only checked when speech is asked for (with no model, 'unknown').
  const gpu = r.device.find((c) => c.id === 'webgpu-adapter')?.status
  const speech = gpu === 'fail' ? 'impossible' : gpu === 'pass' ? 'possible' : 'unknown'
  // null: a deciding check did not answer; the capture checks again.
  return { verdict: r.capture, blocking, limiting, speech }
}

/** @typedef {import('@neurologyca/capture').ProtocolClientHandle} ProtocolClientHandle */
/** @typedef {import('@neurologyca/capture/record').RecordingClientHandle} RecordingClientHandle */
/** @typedef {ReturnType<typeof readDiagnosis>} Diagnosis */

/**
 * Diagnoses the device with no model. `'live'` (default) with a live client, `'record'` with a recording client.
 * @overload
 * @param {ProtocolClientHandle} client
 * @param {'live'} [mode]
 * @returns {Promise<Diagnosis>}
 */
/**
 * @overload
 * @param {RecordingClientHandle} client
 * @param {'record'} mode
 * @returns {Promise<Diagnosis>}
 */
/**
 * @param {ProtocolClientHandle | RecordingClientHandle} client
 * @param {Mode} [mode]
 * @returns {Promise<Diagnosis>}
 */
export async function checkDevice(client, mode = 'live') {
  const result = mode === 'record' ? await client.prepare() : await client.prepare({ models: { face: false, speech: false } })
  return readDiagnosis(result, mode)
}
```

### If speech cannot come

Whether speech will come is only known when preparing speech or when transcription starts: a check with no model cannot tell. If speech is refused (`transcription: 'unavailable'`), offer to record, with permission (deferred, Preview, if your organization has it on): it transcribes on the server. With WebGPU, asking for speech downloads the face (about 13.5 MB) and speech (about 160 MB).

```js
// Speech is only known by asking for it; deferred is Preview, only if your organization has it turned on.
import { checkDevice, readDiagnosis } from './check-device.mjs'

/**
 * @param {object} o
 * @param {import('@neurologyca/capture').ProtocolClientHandle} o.client your live client, with `transcribe`
 * @param {import('@neurologyca/capture/record').RecordingClientHandle} o.recorder your recording client
 * @param {() => Promise<boolean>} o.askToRecord your UI: asks consent to upload the video and process it on the server
 * @returns {Promise<{ mode: 'live', speech: boolean } | { mode: 'record' } | { mode: 'none', blocking: import('@neurologyca/capture').DeviceCheck[] }>}
 */
export async function chooseMode({ client, recorder, askToRecord }) {
  const live = await checkDevice(client, 'live')
  const canLive = live.verdict !== 'unsupported'
  if (canLive) {
    // Downloads the face and speech models and waits. The next capture adopts them; with keepPrepared,
    // later ones too, and asking for another part releases them.
    const prepared = await client.prepare({ models: { face: true, speech: true } })
    if (prepared.transcription !== 'unavailable' && readDiagnosis(prepared, 'live').speech !== 'impossible') return { mode: 'live', speech: true }
  }

  // Only diagnoses: no load, no camera.
  const rec = await checkDevice(recorder, 'record')
  const canRecord = rec.verdict === 'ok' || rec.verdict === 'degraded'
  if (canRecord && (await askToRecord())) {
    client.release() // the recording takes over
    return { mode: 'record' }
  }
  // Not recording: live without speech, or what blocks.
  return canLive ? { mode: 'live', speech: false } : { mode: 'none', blocking: live.blocking }
}
```

## During and after

Read the session rate from `onCodecEvent('capture-quality')`: about one window per second, a `kodec.CaptureQuality`. Describe the rate, never grade it: without the first two windows, the hidden ones or those with `fps: null`, give the median (if even, the mean of the two middle ones), the windows below 8 fps and the lowest; round only to paint.

> 8 fps is the only floor the SDK publishes. Below it, it does not estimate the pulse (an optional channel, off by default) and it treats a transcription veto as provisional. There is no published threshold for how the rate affects the indicators: compare your own sessions.

```js
// After measuring: the session rate, described, never graded. Feed it the 'capture-quality' windows.

/** @typedef {{ fps: number | null, hidden: boolean }} RateWindow */
/** @typedef {'counted' | 'low' | 'skipped' | 'hidden'} WindowState */
/**
 * @typedef {object} SessionRate
 * @property {number} counted the windows that count (low ones included)
 * @property {number | null} median the median fps of those windows; null without any
 * @property {number} lowWindows how many of them fall below the floor
 * @property {number | null} lowest the lowest fps of them; null without any
 * @property {'steady' | 'low' | 'none'} verdict `'none'` without a counted window, `'low'` with a low one, `'steady'` otherwise
 * @property {readonly { fps: number | null, state: WindowState }[]} windows every window in order, at most 600
 */

const STARTUP = 2 // the first two windows are the startup
const MAX_WINDOWS = 600 // the strip keeps 600 (ten minutes); the counts cover all

/**
 * @param {readonly RateWindow[]} windows
 * @param {number} floor the floor, kodec.PULSE_MIN_FPS
 * @returns {SessionRate}
 */
export function sessionRate(windows, floor) {
  /** @type {{ fps: number | null, state: WindowState }[]} */
  const strip = []
  /** @type {number[]} */
  const fps = []
  windows.forEach((w, i) => {
    /** @type {WindowState} */
    let state
    if (i < STARTUP || w.fps === null || !Number.isFinite(w.fps)) state = 'skipped'
    else if (w.hidden) state = 'hidden' // a hidden tab is throttled: it says nothing of the device
    else { state = w.fps < floor ? 'low' : 'counted'; fps.push(w.fps) }
    if (strip.length < MAX_WINDOWS) strip.push({ fps: w.fps, state })
  })
  const lowWindows = fps.filter((x) => x < floor).length
  return {
    counted: fps.length,
    median: median(fps),
    lowWindows,
    lowest: fps.length ? Math.min(...fps) : null,
    verdict: fps.length === 0 ? 'none' : lowWindows > 0 ? 'low' : 'steady',
    windows: strip,
  }
}

/** @param {number[]} xs @returns {number | null} */
function median(xs) {
  if (!xs.length) return null
  const a = [...xs].sort((x, y) => x - y)
  const mid = a.length >> 1
  return a.length % 2 ? a[mid] : (a[mid - 1] + a[mid]) / 2
}
```

`droppedTotal` above zero is a separate notice; `delegate`, a fact to compare. `minFps` and `maxFps` include the startup and the hidden tab. Not the rate either: `capture.throttled`, relative to the session's own rate, and `quality.warning`, from Engine.

## For the person being measured

Read this to the person before measuring, or show it on your capture screen:

- Face a light source: light on your face from the front, not a window behind you.
- Sit at about arm’s length from the camera, with your whole face in view.
- Look at the screen and move little; you do not need to be motionless.
- Keep this tab in front and visible until the end: a hidden tab slows the capture down.
- Close the other apps and tabs that use the camera or the graphics card (video calls, games, editors).

And, for each notice the SDK already gives (`onWarning`), the sentence for the person:

| Notice | What to tell the person |
| --- | --- |
| `quality.warning` | The image is not good enough. Face the light, bring your face into view and keep still. |
| `source.ended` | The video ended, so this part has finished. |
| `capture.throttled` | The capture slowed down. Keep this tab in front and visible, and close what you do not need. |
| `recording.rotate-to-portrait` | Turn your phone upright: the recording only starts in portrait. |

The rest of the person's conditions, and what Engine checks: [Capturing for the best accuracy](/docs/concepts/capture/).
