# On-device (real time)

Processed in the browser as you capture. [Try it in the playground](/playground#measure): the default way there is this one.

## The mode at a glance

| Topic | On-device (real time) · Preview 0.45 |
| --- | --- |
| Today | Available. |
| Where it runs | In the browser, on the person's device, while they capture. |
| When the results arrive | During the session and seconds after the last batch in a short capture. A long capture, with thousands of frames, takes minutes. Past `outcomeDeadlineMs` (240 s by default) `stop()` gives `E_RESULTS_TIMEOUT`, which is not a loss: read the results later. |
| What leaves the device | Landmarks, expression coefficients and head pose, to Engine. Only if you turn them on: the transcript (speech recognition runs in the browser, so the audio stays and the text goes; Engine passes it to an LLM for the interpretation) and the extra signals (voice as numbers, pulse, hands, body). Never image, video or audio. |
| What limits you | The device: the face detection on the GPU (WebGL 2) and, on phones, heat in long sessions, which lower the frame rate. A recent phone or laptop usually stays close to the camera's rate; an entry-level phone captures fewer frames per second, and a phone loses rate as it heats up. |
| Browsers and phones | Chrome and Edge 94+, Safari 16.4+ (iOS 16.4+), on desktop and capable phones. |
| Measurement | Yes. |
| Typical cases | A short check-in on a laptop; a two-minute task on a recent phone; any capture whose video must not leave the device. |
| Moving between them | `createProtocolClient` (from `@neurologyca/capture`) ↔ `createRecordingClient` (from `@neurologyca/capture/record`, without kodec). Some options do not carry over: `transcribe` is on-device only; in deferred, the processing service transcribes. |

## Create the client

One client per protocol context. Your page talks to your backend's proxy: no `getToken`, no key, no token in the page ([Your backend](/docs/sdk/backend/)).

```js
import { createProtocolClient, defineTenantConfig } from '@neurologyca/capture'
import kit from './neuro.config.json' // the configuration of your context, from your kit

const client = createProtocolClient({
  baseUrl: '/engine', // your proxy
  tenant: defineTenantConfig(kit),
  onStatus: (s) => ui.status(s.status),
  onResult: (r) => ui.result(r), // r.raw is the whole GET /results body
  onWarning: (w) => ui.warning(w.code, w.message),
  onError: (e) => ui.error(e.code, e.recoverable),
})
```

> With Measurement, your kit's `neuro.config.json` already says it: `taxonomy: { contextId: 'measurement' }`, and `triggerAnalysis: false` in `captureProtocol`. Since 0.29.0 the SDK skips the taxonomy and the analysis on its own when Engine says `interpreted: false`; keep `triggerAnalysis: false` from your kit, which keeps it working with an Engine that does not say so. The `contextId` is still required ([Measurement](/docs/guides/measurement/)).

## Capture

Collect consent **before** `captureProtocol`, and call it only from a gesture of the person (a click): it opens the camera.

```js
import { describeStep } from '@neurologyca/capture'

const session = await client.captureProtocol({ userProtocolId, protocolMechanicId, source: { type: 'webcam' } })
for (const step of session.plan.steps) {
  const facts = describeStep(step, { lang: 'en' })
  ui.step(step.order, facts.text?.value ?? null)
  await session.step(step.order).run({ durationMs: (facts.maxSeconds ?? 30) * 1000, mediaPid: facts.text?.mediaPid ?? null })
}
const outcome = await session.stop()
```

This loop shows `facts.text`, the step's own media. A step can carry no text of its own (an answer that keeps the previous question, or a reading that borrows the next step's): to choose what to show in each step, use `stepPrompt` from [the same loop, live and recorded](/docs/sdk/steps/#the-same-loop-live-and-recorded).

With a short file, `source.ended` arrives on `onWarning` before the plan is over and a later `run()` fails: handle it by calling `session.abort()`, or end the loop after the warning.

The interpretation is requested by default: `captureProtocol` asks for it unless you pass `triggerAnalysis: false`. Receive it with `onInterpretation` (the narrative, with `structured` and `metadata`) and `onInterpretationDetails` (the same structured details, right after), both options of `createProtocolClient`.

Engine generates the interpretation once and stores it, and `stop()` waits for it (since 0.38.0): it listens for Engine's notice on the event stream up to `analyzeTimeoutMs` (280 s by default; 120 s if the stream never connected) and, if the notice does not arrive, reads the stored one with a single `POST /analyze`. So `stop()` can take minutes: about 13 at worst with the defaults ([Timeouts](/docs/guides/timeouts/)). Show the indicators while it waits. If you want progress by phase, pass `interpretation: 'stream'` to `captureProtocol` and `onProgress` to the client: `onProgress({ phase, route })` receives the phases of `/analyze/stream`, which makes one generation that Engine does not store, so your backend will not read that same text later. `interpretation: 'stream'` is per capture: the client refuses it, and so does a capture with `triggerAnalysis: false` or in recording mode; with a capture token it produces no interpretation.

The pair (`userProtocolId`, `protocolMechanicId`) comes from your backend, which enrolled the person. Who closes each step (a cap, a button, the end of a video) and how the clock works: [Steps and marking](/docs/sdk/steps/).

A `<video>` works as the source too (`{ type: 'video', el }`): give it its `src` before calling `captureProtocol` (the SDK waits for its metadata), keep it `muted` and `playsInline`, and play it in `onStarted`. When the file ends, the SDK warns `source.ended` once: a step driven by `run()` closes at that moment with the frames it got, and with `start()` and `end()` you call `end()` yourself. No later step can be captured, so call `session.abort()`. With several steps on one file, set `loop` on the video while the plan runs. `session.abort()` cancels any capture.

## The startup screen

Between the person's click and the first step there are the Engine reads, the camera permission, the face model (about 13.5 MB the first time) and, with transcription, the speech model (about 160 MB: 136 MB of weights, plus the runtime and the library). On a slow mobile network that is minutes. Since 0.42.0 the SDK caps what does not depend on the person, lets you cancel, and tells you each stage with a message ready to show; it draws nothing.

```js
import { createProtocolClient, defineTenantConfig, CaptureAbortedError, CaptureStartError } from '@neurologyca/capture'
import kit from './neuro.config.json'

const client = createProtocolClient({
  baseUrl: '/engine',
  tenant: defineTenantConfig(kit),
  startup: { timeouts: { speech: { totalMs: 300_000 } } }, // 5 min for speech instead of 10; null removes a cap
  transcribe: { language: 'en', progress: true }, // progress: speech bytes and its stall warning also where kodec does not prefetch
  // A preparation running in the background has its own events (kind 'prepare'): the screen paints the capture's.
  onStartupStage: (e) => { if (e.kind !== 'prepare' && e.userMessage !== null) ui.status(e.userMessage) },
  onWarning: (w) => {
    if (w.code === 'startup.stalled') ui.offer(w.stage === 'speech' ? ['wait', 'cancel', 'without-voice'] : ['wait', 'cancel', 'reload'])
  },
})

let cancel = null // the Cancel button calls cancel?.abort()
async function start(pair, voice = true) {
  cancel = new AbortController()
  try {
    // Without voice: this capture opens no microphone and transcribes nothing, on the same client.
    return await client.captureProtocol({ ...pair, source: { type: 'webcam' }, signal: cancel.signal, ...(voice ? {} : { transcribe: false }) })
  } catch (e) {
    if (e instanceof CaptureAbortedError || e instanceof CaptureStartError) { ui.status(e.userMessage ?? e.message); return null }
    throw e
  }
}
```

- **The stages.** `onStartupStage` is called on every change of a stage (`STARTUP_STAGES`): `engine-setup`, `device`, `camera`, `face-model`, `first-frame`, `speech` and `engine-session`, with a `status` (`pending`, `running`, `stalled`, `done`, `failed`, `skipped`). Paint `userMessage` and you have the screen; it is in `startup.locale` (`'es'` or `'en'`; by default the transcription's language, then the kit's, then the browser's), and `startup.messages` replaces any text by its key (`STARTUP_MESSAGE_KEYS`; the ready-made texts are `STARTUP_MESSAGES`). kodec's sub-stages (`STARTUP_SUB_STAGES`, with `parent: 'face-model'` or `'speech'`) arrive too, with no message and with `bytes`, `total` and `bytesPerSec`: for a finer progress bar or your telemetry. The same events come in `captureBaseline` and in record mode.
- **Speech bytes and its stall.** kodec always counts the bytes of `face-model`, but those of `speech` (and with them its `startup.stalled` warning) only where it prefetches the speech load: not on Safari or iOS, nor with an explicit `model`, an injected `transformers`, WebGPU without JSPI or WASM without `selfHost.wasmPaths`. Elsewhere only `transcribe: { progress: true }` gives them. `speech.running` reads «Downloading speech recognition… (X of Y MB)» only with the bytes and the total, and on that path the total does not come (`total: null`, also with SDK 0.43.0 and kodec 5.0.0-rc.14): the text goes without the figure. Its cost: from an origin without `Content-Length` the library copies the whole buffer on every chunk, and on a phone that raises memory in the phase that already closes tabs. The default weights come with `Content-Length`; if you serve them yourself, send it too ([CSP and transcription](/docs/guides/csp-transcription/)).
- **Texts that fit the source** (since 0.42.1). The `face-model.*` texts speak of getting face analysis ready, not of the camera: the face model loads the same with a webcam, a `stream`, a `<video>` or a preparation with no source. A first-frame timeout uses `first-frame.timeout-media` («No face can be seen in the video…») when the source is your own `stream` or `<video>`, and in the replay of a recording; with the webcam it is still `first-frame.timeout` («We cannot see you…»). The SDK picks it by source, in the stage event and in `CaptureStartError.messageKey` and `userMessage`. A live camera you pass as a `stream` gets the video's wording too: replace `first-frame.timeout-media` in `startup.messages` if you want the webcam's. A text you replaced with `startup.messages` is kept, and a full set of your own texts typed as `Record<StartupMessageKey, string>` has to add each new key (`STARTUP_MESSAGE_KEYS` lists twenty since 0.43.0, with `first-frame.face-reload-failed`).
- **The caps** (defaults; [Timeouts](/docs/guides/timeouts/#the-startup)): the face model fails with `CaptureStartError` (`stage: 'face-model'`, `reason: 'timeout'`) 120 s after the camera is granted; speech gives up after 600 s with the `stt` warning with `reason: 'timeout'`, and the capture goes on without it; 20 s without a new byte raises `startup.stalled` (one per stall), which cuts nothing. `null` removes a cap. The camera permission and the wait for the phone to be turned upright have no cap: `signal` cancels them.
- **Wait, cancel or go on without speech.** On `startup.stalled` of `speech`, the person can wait (the bytes may come back), cancel (`abort()` of your `AbortController`) or go on without speech: abort the start and call `captureProtocol({ …, transcribe: false })` again **on the same client**. In a plan with voice steps, those steps then reach Engine without a transcript. A stalled `face-model` has no «without speech»: wait, cancel or reload.
- **Cancelling.** `signal` (an `AbortSignal`) on `captureProtocol` and `captureBaseline` acts until the call resolves: the SDK releases the camera, the microphone and the pair's slot, cancels the Engine session if it had opened, and rejects with `CaptureAbortedError` (`E_CAPTURE_ABORTED`). After it resolves, end the capture with `session.abort()` or `session.stop()`.
- **After `face-model` runs out, offer a reload.** With `CaptureStartError.subStage` `'vision-runtime'`, a retry on the same page waits again for the hung module and runs out again; with `'face-asset'` or `'vision-create'` a retry is worth it. A reload works in every case, and the `face-model.timeout` message suggests it.
- **The report.** Every startup leaves a `StartupReport` (stages and times, the device checks, `outcome` `ready`, `partial`, `failed`, `aborted` or `unfinished`). Live and in `captureBaseline` it arrives in `SessionReport.startup` by `onReport`, and `SessionReport.transcriptionLoad` says how the speech load ended. To receive it on your server too (the only way in record mode, and the only one that sees a startup abandoned by closing the tab), pass `startup.report: { url, on, context }`: `on: 'failures'` sends only what did not go well, and `context` takes up to 8 pairs of your own, values of up to 128 characters, with no personal data. **Without `report.url` nothing leaves the page.** The report carries the user agent unless you pass `startup.includeUserAgent: false`. Its `device` lists the checks (`DeviceCheck`, ids in `DeviceCheckId`: the secure context, `getUserMedia`, the WebGPU adapter, JSPI, WASM SIMD and others), each `pass`, `warn`, `fail` or `unknown`, decided by capability and never by user agent; and, in a capture that was lent a preparation, `prepared` (`PreparedBlock`) says what it did with the face and with speech (`adopted`, `stale`, `unused` or `not-prepared`) and, in `prepareId`, the `startupId` of the preparation's own report. The receiver: [Your backend](/docs/sdk/backend/#startup-reports-optional).

The ready-made texts of `STARTUP_MESSAGES` (the twenty keys, in the language of `startup.locale`; `{loadedMb}` and `{totalMb}` are whole megabytes, and the span between `[` and `]` disappears if one is missing). This is the English set:

| Key | When | Text |
|---|---|---|
| `startup.preparing` | Any moment without a message of its own | Getting the capture ready… |
| `face-model.running` | The face model loads | Getting face analysis ready… |
| `face-model.cached` | It was already on the device | Getting face analysis ready (already on this device)… |
| `speech.running` | Speech recognition downloads (with the MB when they are known) | `Downloading speech recognition…[ ({loadedMb} of {totalMb} MB)]` |
| `speech.stalled` | The speech download is not moving | Your connection is slow and the speech download is not moving. You can wait or continue without speech recognition. |
| `speech.timeout` | Speech did not arrive in time | Speech recognition did not arrive in time. The capture continues without it. |
| `speech.unavailable` | This device cannot transcribe | This device cannot transcribe speech. The capture continues without it. |
| `face-model.stalled` | The face model load is not moving | Your connection is slow and getting face analysis ready is not moving. You can wait or reload the page. |
| `face-model.timeout` | The face model did not load in time | Face analysis could not be set up in time. Reload the page to try again. |
| `face-model.error` | The face model load failed | Face analysis could not be set up. Reload the page or try another network. |
| `first-frame.timeout` | The webcam shows no face | We cannot see you. Face the camera, with your face well lit, and try again. |
| `first-frame.timeout-media` | Your `stream` or `<video>` (and a replay) shows no face | No face can be seen in the video. Check that a face is visible, facing forward and well lit, and try again. |
| `first-frame.face-reload-failed` | Face analysis stopped as the capture started and loading it again failed (since 0.43.0) | Face analysis stopped working as it started and could not be set up again. Reload the page to try again. |
| `camera.timeout` | The camera did not respond (reserved) | The camera did not respond in time. Reload the page to try again. |
| `capture.aborted` | The capture was cancelled | The capture was cancelled. |
| `startup.ready` | The startup is over | All set. |
| `prepare.speech-timeout` | In a preparation, speech did not finish downloading | Speech recognition did not finish downloading. You can start anyway: it will be tried again when you start. |
| `prepare.speech-error` | In a preparation, speech could not be downloaded | Speech recognition could not be downloaded. You can start anyway: it will be tried again when you start. |
| `prepare.aborted` | The preparation was cancelled | The preparation was cancelled. |
| `prepare.record-unsupported` | In a recording's preparation, the device cannot record | This device can't record the session. Try another browser or device. |

## Prepare before Start

**It is not required**: without it the capture loads what it needs when Start is pressed. `client.prepare()` moves that wait to a moment when the person is doing something else (a form, the instructions), and the **same client's** next `captureProtocol` adopts what was loaded, with no option. It never calls Engine and never opens the camera. The recommended pattern: one client per page, kept for the whole visit; prepare when the page mounts, release when it goes away. Which of the five ways to load the models fits your flow, with a block for each: [Load the models](/docs/guides/load-models/).

```js
import { createProtocolClient, defineTenantConfig } from '@neurologyca/capture'
import kit from './neuro.config.json'

// One client for the page: a capture adopts only what this same client prepared.
const client = createProtocolClient({ baseUrl: '/engine', tenant: defineTenantConfig(kit) })

// When the page mounts. In React: useEffect(() => { void client.prepare(); return () => client.release() }, [client])
const preparing = client.prepare() // the face only; speech too with { models: { face: true, speech: true } }, after consent

// The Start button, from the person's click: it adopts what was prepared and waits, under its own caps, for what is missing.
function onStart(pair) {
  return client.captureProtocol({ ...pair, source: { type: 'webcam' } })
}

// When the person leaves the page, or your view unmounts.
function onLeave() {
  client.release()
}
```

- **What it prepares.** By default the face (about 13.5 MB). Speech (about 160 MB) only with `models: { speech: true }`, which needs the SDK's own transcription (`transcribe` on). Its download asks for no microphone permission, so ask for it after the person has agreed. **On Safari and iOS only the face is prepared**: speech loads when the capture starts (`transcription: 'not-prepared'`, `partialReason: 'speech-not-prepared'`; it is not a failure).
- **`level`.** `'memory'` (the default) loads the models into the page, for this client's next capture. `'cache'` only downloads them into the browser's cache and leaves nothing to adopt: do not mix it with Start on the same page.
- **The result.** `prepare()` resolves a `PrepareResult`: `outcome` (`'ready'`, `'partial'`, `'failed'` or `'aborted'`), `partialReason`, `face`, `transcription`, `estimatedSpeechMs` (a guide to the speech download, which tends to come out high), `cache`, `report`, `messageKey` and `userMessage`. It does not reject because of how the load went, except with `throwOnFailure: true`, and then only on `'failed'` or `'aborted'`: `PrepareError` (`E_PREPARE`), with the whole `PrepareResult` in `result`. `timeouts` changes the client's caps for that call only, and `signal` aborts it.
- **One preparation per client, one capture adopts it.** Only the capture of the client that prepared adopts: a second client on the page loads everything again, and takes double the memory. Without `keepPrepared` ([below](#keep-the-models-between-captures)), speech is released after every session, so each capture needs its own preparation, and without the option a baseline does not adopt: prepare before `captureBaseline` or after it ends. While a capture or a baseline of this client is running, `prepare()` rejects with `PrepareError` and `reason: 'capture-in-progress'`; prepare the next one once the current one ends.
- **Repeating and releasing.** The same options return the same promise; any difference replaces what was prepared (the `prepare.released` warning, `reason: 'replaced'`; its other reasons are `'release'`, `'abort'`, `'pagehide'` and, with `keepPrepared`, `'unusable'`), and a narrower call releases what it no longer asks for: decide once what to prepare. `client.release()` is synchronous and idempotent: it frees what no capture holds and resolves a running preparation `'aborted'` (one at `level: 'cache'` ends only by its `signal`). Call it before dropping a client (a component that unmounts); otherwise the models stay loaded until the page goes away. The client also releases on `pagehide`. The `prepare.stale` warning says a capture could not use a prepared part (it loads it itself); with one client and its own configuration it should not appear.
- **If the prepared face fails.** With or without `keepPrepared` (since 0.43.0), a prepared face that rejects its first frames or loses its GPU context is loaded again once by kodec: the `prepare.face-lost` warning, and the face still `'adopted'` in the report, with `faceReloaded: true`. If that reload fails before the first frame, the start fails at once with `CaptureStartError` (`stage: 'first-frame'`, `reason: 'error'`, key `first-frame.face-reload-failed`). A face the capture loaded itself warns `face.context-lost` in the same case ([Errors](/docs/sdk/errors/#warnings-worth-showing)).

### Keep the models between captures

Since 0.43.0, `keepPrepared` (off by default, live only) keeps what `prepare()`, each capture and each baseline loaded: when a session stops, the face models, and speech whether loaded or still loading, go back to the client, and the next capture adopts them. There is no `prepare()` between captures, and a capture that loaded its models with no `prepare()` keeps them too. With `keepPrepared` a baseline adopts the prepared face (or the one an earlier session kept) and leaves speech for the capture.

```js
import { createProtocolClient, defineTenantConfig } from '@neurologyca/capture'
import kit from './neuro.config.json'

const client = createProtocolClient({ baseUrl: '/engine', tenant: defineTenantConfig(kit), keepPrepared: true })

// Once, before the first baseline or capture (speech after consent). Nothing to prepare after that.
const preparing = client.prepare({ models: { face: true, speech: true } })
// captureBaseline(…) adopts the face; captureProtocol(…) and its stop(), then the next captureProtocol(…): each one
// adopts what the previous one left.

// Between one person and the next on the same device, and before you drop the client (whether it prepared or not).
function onNextPerson() {
  client.release()
}
```

- **What it costs.** The memory stays taken while the person does something else, until `client.release()`, the page going away (`pagehide`, also into the browser's page cache) or kodec leaving a kept part unusable (`prepare.released` with `reason: 'unusable'` and its `part`: only that part is released). That is why, with the option, every client needs `client.release()` before you drop it, whether it prepared or not. Two clients on the page each keep their own, and take double. `client.cacheStatus()` does not see what is kept: it reads the browser's cache, not the memory.
- **Prepare once.** One `client.prepare({ models: { face: true, speech: true } })` before the first capture or baseline; without it the first capture loads both itself, and keeps them. A `prepare()` that asks only for what is kept resolves without loading anything (its report carries `kept: true`); one that asks for a part that is not kept releases what is kept and loads everything again (`prepare.released`, `'replaced'`).
- **Without WebGPU, do not prepare between captures.** Where transcription is refused (no WebGPU and no `allowWasmWorker`) speech is never kept, so a `prepare()` that asks for speech between captures releases the kept face (`'replaced'`) and loads it again every time. There, prepare once before the first capture and do not prepare between captures, or ask for the face only (`prepare({ models: { face: true } })`).
- **On Safari and iOS only the face is kept**: speech loads again with every capture, and a baseline does not take a preparation whose speech is loaded or loading.
- **Between people.** What is kept is models, not what was captured: each session's transcript is released when it closes, and nothing goes to disk. But the face model may carry state from the last frame of one capture into the next. If one device passes from one person to another, call `client.release()` between one person and the next, or leave the option off ([Privacy](/docs/resources/privacy/#what-the-sdk-keeps-in-your-browser)).
- **In the reports.** A capture or a baseline that was lent kept models says so in its `StartupReport`: `prepared.origin` (`'prepare'`, `'previous-session'` or `null` per part; `null`: that part was not among the kept models) and `prepared.prepareId`, which is `null` when no part came from a `prepare()`. Those reports are `schema: 2`, and a receiver validates them with `capture-server` 0.19.0 or later ([Your backend](/docs/sdk/backend/#startup-reports-optional)). In record mode the option is ignored, with the `recording.option-ignored` warning.

### The cache, from another page

- **The cache.** `client.cacheStatus()` says what is already in the browser's cache: only `speechWeights` and `sttRuntime` tell anything (`'cached'`, `'not-cached'` or `'unknown'`); `faceModel`, `sttLibrary` and `visionRuntime` are always `'unknown'`. On an earlier page with no client (a landing page), `prepareCapture({ level: 'cache' })` warms the cache and `captureCacheStatus()` reads it, with **the same configuration** the capture page passes to `createProtocolClient`:

```js
import { captureCacheStatus, prepareCapture } from '@neurologyca/capture'

// One module that both pages import, so the files warmed are the ones the capture asks for.
const captureConfig = { transcribe: { language: 'en' } }

const cached = await captureCacheStatus(captureConfig)
if (cached.speechWeights !== 'cached') await prepareCapture({ ...captureConfig, level: 'cache', prepare: { face: true, speech: true } })
```

What it costs: the person's data is spent even if they never start (about 175 MB cold, the face and speech), and the browser may clear its cache whenever it likes. A recording client also has `prepare()`, which loads no model: it only checks, before any permission, whether the device can record ([Deferred (server-side)](/docs/sdk/deferred/)).

## The outcome

`stop()` resolves with a `CaptureOutcome`: `status` (`'complete'`, `'timeout'`, `'invalidated'` or `'cancelled'`), `hcis` (the means), `hcisByStep`, `frameCount`, `interpretation` (the narrative, or `null`) and, since 0.39.0, `structured` (the interpretation's structured data, or `null`; read a 0 to 100 index from it with `readScoreIndex`, see [A 0 to 100 index](/docs/concepts/results/#a-0-to-100-index)). The fields and the scales: [HCIs and results](/docs/concepts/results/#in-the-sdk).

If you asked for an interpretation, `onInterpretationDetails` fires right after `onInterpretation` (since 0.38.0), so you can save both together. With `interpretation: 'stream'` the details come later, normally after `stop()` has resolved: wait for them with a deadline of your own, and save without them if it passes. A `processing.error` from Engine while `stop()` waits for the interpretation ends the session `'invalidated'`: no interpretation is coming.

Past `outcomeDeadlineMs` (240 s by default), `E_RESULTS_TIMEOUT`: it is not a loss. Engine has the capture; keep the pair and read the results later ([Results in your backend](/docs/sdk/results-backend/)).

## Errors and warnings

Three families: pipeline failures by `onError` (`{ code, recoverable, stepOrder? }`), usage errors thrown to you, and warnings. The rule: a `recoverable` other than `true` during the steps means the capture is lost, so call `abort()`; after `stop()`, do not: codes such as `E_RESULTS_TIMEOUT` arrive with `recoverable: false` but Engine already has the capture.

The warnings worth showing (the codes are strings, not a published union):

- `source.ended`: your file or stream is over. A step driven by `run()` closes; with `start()` and `end()` you close it yourself with `end()`. No later step can be captured.
- `step.duration-below-min` and `step.duration-above-max`: a step closed outside its window.
- `step.media-undeclared`: the step has several media and you did not say which one you showed.
- `capture.throttled` is relative to the session's own rate: it fires when the rate falls to a quarter of that session's own for three windows in a row, with a step open (once per step), typically a hidden tab. A device that is slow from the first window never raises it, so it is not the signal for how fast the device is (read it as in [Try it on your phone](#try-it-on-your-phone)).
- `quality.warning`: Engine says the quality of what reaches it is not good enough.
- `stream.unavailable`: the event stream did not connect; the capture goes on.
- `interpretation.sse-timeout` and `analysis.regenerated` (since 0.38.0): Engine's notice did not arrive in time and the interpretation was read with `POST /analyze`; with `analysis.regenerated`, `/analyze` had to generate a new one, and the session may have been billed twice. Both name the cause; if the stream dropped during the wait, check that your proxy forwards `/stream/{userProtocolId}`.
- `transcription.off-on-voice-step` and `transcribeSteps.excludes-voice-step` (since 0.38.0, before the camera opens): a step where the person is expected to speak (`expectsSpeech: 'yes'` in the plan) will not be transcribed, because `transcribe` is off or `transcribeSteps` leaves it out. `expectsSpeech` is `'yes'`, `'no'`, `'any'` (the person may or may not talk), `null` (not declared) or another value the catalogue declares, and the SDK only warns with `'yes'`. `transcribeSteps.noMatch`: no type in `transcribeSteps` exists in the plan, so no step is transcribed.
- `protocol.not-interpreted` (since 0.29.0): Engine says the protocol is not interpreted (Measurement): no taxonomy is sent and no analysis asked for, even with `triggerAnalysis: true`. `interpretation` will be `null`.

`E_CONFLICT` comes with `reason` and `conflict.<field>`. Every code, by layer: [Errors, warnings and retries](/docs/sdk/errors/).

## Baseline

Read it with `client.getBaselineInfo(userId)` (`null` means none yet) and capture it with `client.captureBaseline({ userId, source, onStarted, onProgress })`; with `keepPrepared` it adopts the prepared face ([Keep the models between captures](#keep-the-models-between-captures)). Engine computes it in the background: read it again until it is there. With a `<video>`, the capture stops when the file ends. The baseline always goes through the live client: the recording client has no `captureBaseline`. See [Baseline](/docs/concepts/baseline/).

```js
const info = await client.getBaselineInfo(userId) // null: no baseline yet
if (!info) await client.captureBaseline({ userId, source: { type: 'webcam' }, onProgress: (p) => ui.progress(p) })
```

## Transcription

Off until something turns it on. Turn it on with `transcribe` in the client, and say which steps with `transcribeSteps` (the speech steps of your protocol). A `transcribe` object in the kit (`tenant`) also turns transcription on, even if your code does not pass `transcribe`; the DevLab kit of every context protocol carries one by default (a language without a region), so, with a camera source, the capture opens the microphone unless your code passes `transcribe: false`. It is off by default in Measurement, which has no speech steps; if you turn it on there, Engine keeps the text and `/results` returns it as the `transcript` of step 1 ([`speech`](/docs/concepts/results/#speech), only in the test environment's Engine).

Speech recognition runs in the browser: the audio does not leave the browser; the text goes to Engine, which passes it to an LLM for the interpretation. The first time, it downloads about 160 MB (136 MB of them the model's weights, from a third-party CDN; [CSP and transcription](/docs/guides/csp-transcription/)); you can download it before Start ([Prepare before Start](#prepare-before-start)). With a `<video>` as the source, the SDK transcribes the file's own audio, without opening the microphone.

```js
const client = createProtocolClient({ baseUrl: '/engine', tenant, transcribe: { language: 'en' }, transcribeSteps: ['free_speech'] })
```

## The session and its event stream

| When | What happens |
| --- | --- |
| `captureProtocol` resolves | The SDK opens `GET /stream/{userProtocolId}` through your proxy. |
| While capturing | Events: `connected`, `hci.warning`, `results.ready` (once per position, and not in every case: the SDK keeps reconciling over HTTP; since 0.29.0 it is a known event, before it gave `stream.unknown-event`), and the interpretation's events when you asked for one. |
| The stream drops | The SDK reopens it, waiting 0.5 to 10 s; after three failed attempts it warns `stream.unavailable` and the capture goes on. |
| `stop()` | With an interpretation on the way, the SDK waits on the stream for Engine's notice (only this mechanic's), then closes it. |
| `abort()` | The stream closes. |
| The person signs out of your app | An open stream survives it (your proxy's `authorize` runs once, when it opens): close the capture when the person leaves, and limit streams per user in your backend. |
| The tab closes | The capture stops; Engine keeps what it received, and the open position follows Engine's session rules ([Sessions](/docs/concepts/sessions/)). |

## Try it on your phone

Serve your page over HTTPS to the phone (a secure context: a tunnel to your development machine or a certificate for your local network, whichever you use), allow the camera, and read the rate at run time with `onCodecEvent('capture-quality')`: kodec reports about one window per second with the frames per second it measured. Skip the first two windows (the start-up) and the ones with the tab hidden (`document.visibilityState`), then read the median of the rest, how many windows fall under 8 fps and the lowest one. 8 fps is the only floor the SDK publishes. Below it, it does not estimate the pulse (an optional channel, off by default) and it treats a transcription veto as provisional. There is no published threshold for how the rate affects the indicators: compare your own sessions. The rule with its code, and the device check before measuring: [Devices, quality and mode](/docs/start/devices-and-modes/). `SessionReport.captureQuality` summarises the same windows but includes the start-up and the hidden ones, so do not judge from it alone; and `capture.throttled` is relative to the session's own rate, so a phone that is slow from the start never raises it. What limits the rate is the face detection on the GPU (WebGL 2) and, on phones, heat in long sessions ([Browsers and frame rate](/docs/concepts/capture/#browsers-and-frame-rate)). The frame rate depends on the device, not on the platform: a recent phone usually stays close to the camera's rate, an entry-level phone captures fewer frames per second, and a phone loses rate as it heats up in a long session. Read it at run time, not from the phone model. The helpers (`keepScreenOn`, `guardUnload`, `classifyMediaError`) are in [Steps and marking](/docs/sdk/steps/#helpers); `guardUnload` depends on the browser: in Chrome for Android, closing the tab does not show the dialog; a closed tab ends the capture, so capture again (when recording in [deferred](/docs/sdk/deferred/), `listPendingRecordings()` finds on the next visit what was not uploaded: upload it with `uploadRecording` or discard it with its `release()`). The cheapest test: open the [playground](/playground) on your phone.

## Options

`batchMs` (5000 by default), `fps`, `stepDurations` (`'warn'` by default; `'enforce'`; `'off'`), `requestTimeoutMs`, `outcomeDeadlineMs` (240 s), `analyzeTimeoutMs` (280 s: the wait for the interpretation, and then the `POST /analyze`; since 0.44.0, after a `202 ANALYSIS_IN_PROGRESS` the SDK waits and asks again within it, counted from the first `202`, and ends with `E_ANALYSIS_IN_PROGRESS` if it runs out), `credentials` (since 0.45.0: `'omit'`, `'same-origin'` or `'include'`, the `credentials` mode of the client's requests to your proxy, the event stream and the streamed interpretation included; absent, the browser's `'same-origin'`; `'include'` for a proxy on another origin that decides by your session cookie, [The proxy on another origin](/docs/api/proxy/#a-proxy-on-another-origin)), `onProgress` (only with `interpretation: 'stream'`), `correlationId` (a prefix for `X-Correlation-Id`), `channels`, `kodec` (see [CSP and transcription](/docs/guides/csp-transcription/)), `startup` (`timeouts`, `locale`, `messages`, `report`, `includeUserAgent`; an unknown key throws `UnknownOptionError`) and `onStartupStage` ([The startup screen](#the-startup-screen)). Since 0.42.0, `createProtocolClient` throws `UnknownOptionError` (`E_UNKNOWN_OPTION`) for any option it does not have; before, it ignored it. The deadlines are in [Timeouts](/docs/guides/timeouts/). Other options in the reference are for advanced tuning; leave them at their defaults unless Neurologyca tells you otherwise.
