Skip to content
DevLab

On-device (real time)

The live client: create it without a token, capture step by step, read the outcome, the warnings, the baseline, transcription and the event stream.

On this page
  1. The mode at a glance
  2. Create the client
  3. Capture
  4. The startup screen
  5. Prepare before Start
  6. Keep the models between captures
  7. The cache, from another page
  8. The outcome
  9. Errors and warnings
  10. Baseline
  11. Transcription
  12. The session and its event stream
  13. Try it on your phone
  14. Options

Processed in the browser as you capture. Try it in the playground: the default way there is this one.

On-device (real time)
TopicOn-device (real time) · Preview 0.45
TodayAvailable.
Where it runsIn the browser, on the person's device, while they capture.
When the results arriveDuring the session and seconds after the last batch in a short capture. A long capture, with thousands of frames, takes minutes. Past outcomeDeadlineMs (240 s by default) stop() gives E_RESULTS_TIMEOUT, which is not a loss: read the results later.
What leaves the deviceLandmarks, expression coefficients and head pose, to Engine. Only if you turn them on: the transcript (speech recognition runs in the browser, so the audio stays and the text goes; Engine passes it to an LLM for the interpretation) and the extra signals (voice as numbers, pulse, hands, body). Never image, video or audio.
What limits youThe device: the face detection on the GPU (WebGL 2) and, on phones, heat in long sessions, which lower the frame rate. A recent phone or laptop usually stays close to the camera's rate; an entry-level phone captures fewer frames per second, and a phone loses rate as it heats up.
Browsers and phonesChrome and Edge 94+, Safari 16.4+ (iOS 16.4+), on desktop and capable phones.
MeasurementYes.
Typical casesA short check-in on a laptop; a two-minute task on a recent phone; any capture whose video must not leave the device.
Moving between themcreateProtocolClient (from @neurologyca/capture) ↔ createRecordingClient (from @neurologyca/capture/record, without kodec). Some options do not carry over: transcribe is on-device only; in deferred, the processing service transcribes.

One client per protocol context. Your page talks to your backend’s proxy: no getToken, no key, no token in the page (Your backend).

JavaScript
import { createProtocolClient, defineTenantConfig } from '@neurologyca/capture'
import kit from './neuro.config.json' // the configuration of your context, from your kit
const client = createProtocolClient({
baseUrl: '/engine', // your proxy
tenant: defineTenantConfig(kit),
onStatus: (s) => ui.status(s.status),
onResult: (r) => ui.result(r), // r.raw is the whole GET /results body
onWarning: (w) => ui.warning(w.code, w.message),
onError: (e) => ui.error(e.code, e.recoverable),
})

Collect consent before captureProtocol, and call it only from a gesture of the person (a click): it opens the camera.

JavaScript
import { describeStep } from '@neurologyca/capture'
const session = await client.captureProtocol({ userProtocolId, protocolMechanicId, source: { type: 'webcam' } })
for (const step of session.plan.steps) {
const facts = describeStep(step, { lang: 'en' })
ui.step(step.order, facts.text?.value ?? null)
await session.step(step.order).run({ durationMs: (facts.maxSeconds ?? 30) * 1000, mediaPid: facts.text?.mediaPid ?? null })
}
const outcome = await session.stop()

This loop shows facts.text, the step’s own media. A step can carry no text of its own (an answer that keeps the previous question, or a reading that borrows the next step’s): to choose what to show in each step, use stepPrompt from the same loop, live and recorded.

With a short file, source.ended arrives on onWarning before the plan is over and a later run() fails: handle it by calling session.abort(), or end the loop after the warning.

The interpretation is requested by default: captureProtocol asks for it unless you pass triggerAnalysis: false. Receive it with onInterpretation (the narrative, with structured and metadata) and onInterpretationDetails (the same structured details, right after), both options of createProtocolClient.

Engine generates the interpretation once and stores it, and stop() waits for it (since 0.38.0): it listens for Engine’s notice on the event stream up to analyzeTimeoutMs (280 s by default; 120 s if the stream never connected) and, if the notice does not arrive, reads the stored one with a single POST /analyze. So stop() can take minutes: about 13 at worst with the defaults (Timeouts). Show the indicators while it waits. If you want progress by phase, pass interpretation: 'stream' to captureProtocol and onProgress to the client: onProgress({ phase, route }) receives the phases of /analyze/stream, which makes one generation that Engine does not store, so your backend will not read that same text later. interpretation: 'stream' is per capture: the client refuses it, and so does a capture with triggerAnalysis: false or in recording mode; with a capture token it produces no interpretation.

The pair (userProtocolId, protocolMechanicId) comes from your backend, which enrolled the person. Who closes each step (a cap, a button, the end of a video) and how the clock works: Steps and marking.

A <video> works as the source too ({ type: 'video', el }): give it its src before calling captureProtocol (the SDK waits for its metadata), keep it muted and playsInline, and play it in onStarted. When the file ends, the SDK warns source.ended once: a step driven by run() closes at that moment with the frames it got, and with start() and end() you call end() yourself. No later step can be captured, so call session.abort(). With several steps on one file, set loop on the video while the plan runs. session.abort() cancels any capture.

Between the person’s click and the first step there are the Engine reads, the camera permission, the face model (about 13.5 MB the first time) and, with transcription, the speech model (about 160 MB: 136 MB of weights, plus the runtime and the library). On a slow mobile network that is minutes. Since 0.42.0 the SDK caps what does not depend on the person, lets you cancel, and tells you each stage with a message ready to show; it draws nothing.

JavaScript
import { createProtocolClient, defineTenantConfig, CaptureAbortedError, CaptureStartError } from '@neurologyca/capture'
import kit from './neuro.config.json'
const client = createProtocolClient({
baseUrl: '/engine',
tenant: defineTenantConfig(kit),
startup: { timeouts: { speech: { totalMs: 300_000 } } }, // 5 min for speech instead of 10; null removes a cap
transcribe: { language: 'en', progress: true }, // progress: speech bytes and its stall warning also where kodec does not prefetch
// A preparation running in the background has its own events (kind 'prepare'): the screen paints the capture's.
onStartupStage: (e) => { if (e.kind !== 'prepare' && e.userMessage !== null) ui.status(e.userMessage) },
onWarning: (w) => {
if (w.code === 'startup.stalled') ui.offer(w.stage === 'speech' ? ['wait', 'cancel', 'without-voice'] : ['wait', 'cancel', 'reload'])
},
})
let cancel = null // the Cancel button calls cancel?.abort()
async function start(pair, voice = true) {
cancel = new AbortController()
try {
// Without voice: this capture opens no microphone and transcribes nothing, on the same client.
return await client.captureProtocol({ ...pair, source: { type: 'webcam' }, signal: cancel.signal, ...(voice ? {} : { transcribe: false }) })
} catch (e) {
if (e instanceof CaptureAbortedError || e instanceof CaptureStartError) { ui.status(e.userMessage ?? e.message); return null }
throw e
}
}
  • The stages. onStartupStage is called on every change of a stage (STARTUP_STAGES): engine-setup, device, camera, face-model, first-frame, speech and engine-session, with a status (pending, running, stalled, done, failed, skipped). Paint userMessage and you have the screen; it is in startup.locale ('es' or 'en'; by default the transcription’s language, then the kit’s, then the browser’s), and startup.messages replaces any text by its key (STARTUP_MESSAGE_KEYS; the ready-made texts are STARTUP_MESSAGES). kodec’s sub-stages (STARTUP_SUB_STAGES, with parent: 'face-model' or 'speech') arrive too, with no message and with bytes, total and bytesPerSec: for a finer progress bar or your telemetry. The same events come in captureBaseline and in record mode.
  • Speech bytes and its stall. kodec always counts the bytes of face-model, but those of speech (and with them its startup.stalled warning) only where it prefetches the speech load: not on Safari or iOS, nor with an explicit model, an injected transformers, WebGPU without JSPI or WASM without selfHost.wasmPaths. Elsewhere only transcribe: { progress: true } gives them. speech.running reads «Downloading speech recognition… (X of Y MB)» only with the bytes and the total, and on that path the total does not come (total: null, also with SDK 0.43.0 and kodec 5.0.0-rc.14): the text goes without the figure. Its cost: from an origin without Content-Length the library copies the whole buffer on every chunk, and on a phone that raises memory in the phase that already closes tabs. The default weights come with Content-Length; if you serve them yourself, send it too (CSP and transcription).
  • Texts that fit the source (since 0.42.1). The face-model.* texts speak of getting face analysis ready, not of the camera: the face model loads the same with a webcam, a stream, a <video> or a preparation with no source. A first-frame timeout uses first-frame.timeout-media («No face can be seen in the video…») when the source is your own stream or <video>, and in the replay of a recording; with the webcam it is still first-frame.timeout («We cannot see you…»). The SDK picks it by source, in the stage event and in CaptureStartError.messageKey and userMessage. A live camera you pass as a stream gets the video’s wording too: replace first-frame.timeout-media in startup.messages if you want the webcam’s. A text you replaced with startup.messages is kept, and a full set of your own texts typed as Record<StartupMessageKey, string> has to add each new key (STARTUP_MESSAGE_KEYS lists twenty since 0.43.0, with first-frame.face-reload-failed).
  • The caps (defaults; Timeouts): the face model fails with CaptureStartError (stage: 'face-model', reason: 'timeout') 120 s after the camera is granted; speech gives up after 600 s with the stt warning with reason: 'timeout', and the capture goes on without it; 20 s without a new byte raises startup.stalled (one per stall), which cuts nothing. null removes a cap. The camera permission and the wait for the phone to be turned upright have no cap: signal cancels them.
  • Wait, cancel or go on without speech. On startup.stalled of speech, the person can wait (the bytes may come back), cancel (abort() of your AbortController) or go on without speech: abort the start and call captureProtocol({ …, transcribe: false }) again on the same client. In a plan with voice steps, those steps then reach Engine without a transcript. A stalled face-model has no «without speech»: wait, cancel or reload.
  • Cancelling. signal (an AbortSignal) on captureProtocol and captureBaseline acts until the call resolves: the SDK releases the camera, the microphone and the pair’s slot, cancels the Engine session if it had opened, and rejects with CaptureAbortedError (E_CAPTURE_ABORTED). After it resolves, end the capture with session.abort() or session.stop().
  • After face-model runs out, offer a reload. With CaptureStartError.subStage 'vision-runtime', a retry on the same page waits again for the hung module and runs out again; with 'face-asset' or 'vision-create' a retry is worth it. A reload works in every case, and the face-model.timeout message suggests it.
  • The report. Every startup leaves a StartupReport (stages and times, the device checks, outcome ready, partial, failed, aborted or unfinished). Live and in captureBaseline it arrives in SessionReport.startup by onReport, and SessionReport.transcriptionLoad says how the speech load ended. To receive it on your server too (the only way in record mode, and the only one that sees a startup abandoned by closing the tab), pass startup.report: { url, on, context }: on: 'failures' sends only what did not go well, and context takes up to 8 pairs of your own, values of up to 128 characters, with no personal data. Without report.url nothing leaves the page. The report carries the user agent unless you pass startup.includeUserAgent: false. Its device lists the checks (DeviceCheck, ids in DeviceCheckId: the secure context, getUserMedia, the WebGPU adapter, JSPI, WASM SIMD and others), each pass, warn, fail or unknown, decided by capability and never by user agent; and, in a capture that was lent a preparation, prepared (PreparedBlock) says what it did with the face and with speech (adopted, stale, unused or not-prepared) and, in prepareId, the startupId of the preparation’s own report. The receiver: Your backend.

The ready-made texts of STARTUP_MESSAGES (the twenty keys, in the language of startup.locale; {loadedMb} and {totalMb} are whole megabytes, and the span between [ and ] disappears if one is missing). This is the English set:

Key When Text
startup.preparing Any moment without a message of its own Getting the capture ready…
face-model.running The face model loads Getting face analysis ready…
face-model.cached It was already on the device Getting face analysis ready (already on this device)…
speech.running Speech recognition downloads (with the MB when they are known) Downloading speech recognition…[ ({loadedMb} of {totalMb} MB)]
speech.stalled The speech download is not moving Your connection is slow and the speech download is not moving. You can wait or continue without speech recognition.
speech.timeout Speech did not arrive in time Speech recognition did not arrive in time. The capture continues without it.
speech.unavailable This device cannot transcribe This device cannot transcribe speech. The capture continues without it.
face-model.stalled The face model load is not moving Your connection is slow and getting face analysis ready is not moving. You can wait or reload the page.
face-model.timeout The face model did not load in time Face analysis could not be set up in time. Reload the page to try again.
face-model.error The face model load failed Face analysis could not be set up. Reload the page or try another network.
first-frame.timeout The webcam shows no face We cannot see you. Face the camera, with your face well lit, and try again.
first-frame.timeout-media Your stream or <video> (and a replay) shows no face No face can be seen in the video. Check that a face is visible, facing forward and well lit, and try again.
first-frame.face-reload-failed Face analysis stopped as the capture started and loading it again failed (since 0.43.0) Face analysis stopped working as it started and could not be set up again. Reload the page to try again.
camera.timeout The camera did not respond (reserved) The camera did not respond in time. Reload the page to try again.
capture.aborted The capture was cancelled The capture was cancelled.
startup.ready The startup is over All set.
prepare.speech-timeout In a preparation, speech did not finish downloading Speech recognition did not finish downloading. You can start anyway: it will be tried again when you start.
prepare.speech-error In a preparation, speech could not be downloaded Speech recognition could not be downloaded. You can start anyway: it will be tried again when you start.
prepare.aborted The preparation was cancelled The preparation was cancelled.
prepare.record-unsupported In a recording’s preparation, the device cannot record This device can’t record the session. Try another browser or device.

It is not required: without it the capture loads what it needs when Start is pressed. client.prepare() moves that wait to a moment when the person is doing something else (a form, the instructions), and the same client’s next captureProtocol adopts what was loaded, with no option. It never calls Engine and never opens the camera. The recommended pattern: one client per page, kept for the whole visit; prepare when the page mounts, release when it goes away. Which of the five ways to load the models fits your flow, with a block for each: Load the models.

JavaScript
import { createProtocolClient, defineTenantConfig } from '@neurologyca/capture'
import kit from './neuro.config.json'
// One client for the page: a capture adopts only what this same client prepared.
const client = createProtocolClient({ baseUrl: '/engine', tenant: defineTenantConfig(kit) })
// When the page mounts. In React: useEffect(() => { void client.prepare(); return () => client.release() }, [client])
const preparing = client.prepare() // the face only; speech too with { models: { face: true, speech: true } }, after consent
// The Start button, from the person's click: it adopts what was prepared and waits, under its own caps, for what is missing.
function onStart(pair) {
return client.captureProtocol({ ...pair, source: { type: 'webcam' } })
}
// When the person leaves the page, or your view unmounts.
function onLeave() {
client.release()
}
  • What it prepares. By default the face (about 13.5 MB). Speech (about 160 MB) only with models: { speech: true }, which needs the SDK’s own transcription (transcribe on). Its download asks for no microphone permission, so ask for it after the person has agreed. On Safari and iOS only the face is prepared: speech loads when the capture starts (transcription: 'not-prepared', partialReason: 'speech-not-prepared'; it is not a failure).
  • level. 'memory' (the default) loads the models into the page, for this client’s next capture. 'cache' only downloads them into the browser’s cache and leaves nothing to adopt: do not mix it with Start on the same page.
  • The result. prepare() resolves a PrepareResult: outcome ('ready', 'partial', 'failed' or 'aborted'), partialReason, face, transcription, estimatedSpeechMs (a guide to the speech download, which tends to come out high), cache, report, messageKey and userMessage. It does not reject because of how the load went, except with throwOnFailure: true, and then only on 'failed' or 'aborted': PrepareError (E_PREPARE), with the whole PrepareResult in result. timeouts changes the client’s caps for that call only, and signal aborts it.
  • One preparation per client, one capture adopts it. Only the capture of the client that prepared adopts: a second client on the page loads everything again, and takes double the memory. Without keepPrepared (below), speech is released after every session, so each capture needs its own preparation, and without the option a baseline does not adopt: prepare before captureBaseline or after it ends. While a capture or a baseline of this client is running, prepare() rejects with PrepareError and reason: 'capture-in-progress'; prepare the next one once the current one ends.
  • Repeating and releasing. The same options return the same promise; any difference replaces what was prepared (the prepare.released warning, reason: 'replaced'; its other reasons are 'release', 'abort', 'pagehide' and, with keepPrepared, 'unusable'), and a narrower call releases what it no longer asks for: decide once what to prepare. client.release() is synchronous and idempotent: it frees what no capture holds and resolves a running preparation 'aborted' (one at level: 'cache' ends only by its signal). Call it before dropping a client (a component that unmounts); otherwise the models stay loaded until the page goes away. The client also releases on pagehide. The prepare.stale warning says a capture could not use a prepared part (it loads it itself); with one client and its own configuration it should not appear.
  • If the prepared face fails. With or without keepPrepared (since 0.43.0), a prepared face that rejects its first frames or loses its GPU context is loaded again once by kodec: the prepare.face-lost warning, and the face still 'adopted' in the report, with faceReloaded: true. If that reload fails before the first frame, the start fails at once with CaptureStartError (stage: 'first-frame', reason: 'error', key first-frame.face-reload-failed). A face the capture loaded itself warns face.context-lost in the same case (Errors).

Since 0.43.0, keepPrepared (off by default, live only) keeps what prepare(), each capture and each baseline loaded: when a session stops, the face models, and speech whether loaded or still loading, go back to the client, and the next capture adopts them. There is no prepare() between captures, and a capture that loaded its models with no prepare() keeps them too. With keepPrepared a baseline adopts the prepared face (or the one an earlier session kept) and leaves speech for the capture.

JavaScript
import { createProtocolClient, defineTenantConfig } from '@neurologyca/capture'
import kit from './neuro.config.json'
const client = createProtocolClient({ baseUrl: '/engine', tenant: defineTenantConfig(kit), keepPrepared: true })
// Once, before the first baseline or capture (speech after consent). Nothing to prepare after that.
const preparing = client.prepare({ models: { face: true, speech: true } })
// captureBaseline(…) adopts the face; captureProtocol(…) and its stop(), then the next captureProtocol(…): each one
// adopts what the previous one left.
// Between one person and the next on the same device, and before you drop the client (whether it prepared or not).
function onNextPerson() {
client.release()
}
  • What it costs. The memory stays taken while the person does something else, until client.release(), the page going away (pagehide, also into the browser’s page cache) or kodec leaving a kept part unusable (prepare.released with reason: 'unusable' and its part: only that part is released). That is why, with the option, every client needs client.release() before you drop it, whether it prepared or not. Two clients on the page each keep their own, and take double. client.cacheStatus() does not see what is kept: it reads the browser’s cache, not the memory.
  • Prepare once. One client.prepare({ models: { face: true, speech: true } }) before the first capture or baseline; without it the first capture loads both itself, and keeps them. A prepare() that asks only for what is kept resolves without loading anything (its report carries kept: true); one that asks for a part that is not kept releases what is kept and loads everything again (prepare.released, 'replaced').
  • Without WebGPU, do not prepare between captures. Where transcription is refused (no WebGPU and no allowWasmWorker) speech is never kept, so a prepare() that asks for speech between captures releases the kept face ('replaced') and loads it again every time. There, prepare once before the first capture and do not prepare between captures, or ask for the face only (prepare({ models: { face: true } })).
  • On Safari and iOS only the face is kept: speech loads again with every capture, and a baseline does not take a preparation whose speech is loaded or loading.
  • Between people. What is kept is models, not what was captured: each session’s transcript is released when it closes, and nothing goes to disk. But the face model may carry state from the last frame of one capture into the next. If one device passes from one person to another, call client.release() between one person and the next, or leave the option off (Privacy).
  • In the reports. A capture or a baseline that was lent kept models says so in its StartupReport: prepared.origin ('prepare', 'previous-session' or null per part; null: that part was not among the kept models) and prepared.prepareId, which is null when no part came from a prepare(). Those reports are schema: 2, and a receiver validates them with capture-server 0.19.0 or later (Your backend). In record mode the option is ignored, with the recording.option-ignored warning.
  • The cache. client.cacheStatus() says what is already in the browser’s cache: only speechWeights and sttRuntime tell anything ('cached', 'not-cached' or 'unknown'); faceModel, sttLibrary and visionRuntime are always 'unknown'. On an earlier page with no client (a landing page), prepareCapture({ level: 'cache' }) warms the cache and captureCacheStatus() reads it, with the same configuration the capture page passes to createProtocolClient:
JavaScript
import { captureCacheStatus, prepareCapture } from '@neurologyca/capture'
// One module that both pages import, so the files warmed are the ones the capture asks for.
const captureConfig = { transcribe: { language: 'en' } }
const cached = await captureCacheStatus(captureConfig)
if (cached.speechWeights !== 'cached') await prepareCapture({ ...captureConfig, level: 'cache', prepare: { face: true, speech: true } })

What it costs: the person’s data is spent even if they never start (about 175 MB cold, the face and speech), and the browser may clear its cache whenever it likes. A recording client also has prepare(), which loads no model: it only checks, before any permission, whether the device can record (Deferred (server-side)).

stop() resolves with a CaptureOutcome: status ('complete', 'timeout', 'invalidated' or 'cancelled'), hcis (the means), hcisByStep, frameCount, interpretation (the narrative, or null) and, since 0.39.0, structured (the interpretation’s structured data, or null; read a 0 to 100 index from it with readScoreIndex, see A 0 to 100 index). The fields and the scales: HCIs and results.

If you asked for an interpretation, onInterpretationDetails fires right after onInterpretation (since 0.38.0), so you can save both together. With interpretation: 'stream' the details come later, normally after stop() has resolved: wait for them with a deadline of your own, and save without them if it passes. A processing.error from Engine while stop() waits for the interpretation ends the session 'invalidated': no interpretation is coming.

Past outcomeDeadlineMs (240 s by default), E_RESULTS_TIMEOUT: it is not a loss. Engine has the capture; keep the pair and read the results later (Results in your backend).

Three families: pipeline failures by onError ({ code, recoverable, stepOrder? }), usage errors thrown to you, and warnings. The rule: a recoverable other than true during the steps means the capture is lost, so call abort(); after stop(), do not: codes such as E_RESULTS_TIMEOUT arrive with recoverable: false but Engine already has the capture.

The warnings worth showing (the codes are strings, not a published union):

  • source.ended: your file or stream is over. A step driven by run() closes; with start() and end() you close it yourself with end(). No later step can be captured.
  • step.duration-below-min and step.duration-above-max: a step closed outside its window.
  • step.media-undeclared: the step has several media and you did not say which one you showed.
  • capture.throttled is relative to the session’s own rate: it fires when the rate falls to a quarter of that session’s own for three windows in a row, with a step open (once per step), typically a hidden tab. A device that is slow from the first window never raises it, so it is not the signal for how fast the device is (read it as in Try it on your phone).
  • quality.warning: Engine says the quality of what reaches it is not good enough.
  • stream.unavailable: the event stream did not connect; the capture goes on.
  • interpretation.sse-timeout and analysis.regenerated (since 0.38.0): Engine’s notice did not arrive in time and the interpretation was read with POST /analyze; with analysis.regenerated, /analyze had to generate a new one, and the session may have been billed twice. Both name the cause; if the stream dropped during the wait, check that your proxy forwards /stream/{userProtocolId}.
  • transcription.off-on-voice-step and transcribeSteps.excludes-voice-step (since 0.38.0, before the camera opens): a step where the person is expected to speak (expectsSpeech: 'yes' in the plan) will not be transcribed, because transcribe is off or transcribeSteps leaves it out. expectsSpeech is 'yes', 'no', 'any' (the person may or may not talk), null (not declared) or another value the catalogue declares, and the SDK only warns with 'yes'. transcribeSteps.noMatch: no type in transcribeSteps exists in the plan, so no step is transcribed.
  • protocol.not-interpreted (since 0.29.0): Engine says the protocol is not interpreted (Measurement): no taxonomy is sent and no analysis asked for, even with triggerAnalysis: true. interpretation will be null.

E_CONFLICT comes with reason and conflict.<field>. Every code, by layer: Errors, warnings and retries.

Read it with client.getBaselineInfo(userId) (null means none yet) and capture it with client.captureBaseline({ userId, source, onStarted, onProgress }); with keepPrepared it adopts the prepared face (Keep the models between captures). Engine computes it in the background: read it again until it is there. With a <video>, the capture stops when the file ends. The baseline always goes through the live client: the recording client has no captureBaseline. See Baseline.

JavaScript
const info = await client.getBaselineInfo(userId) // null: no baseline yet
if (!info) await client.captureBaseline({ userId, source: { type: 'webcam' }, onProgress: (p) => ui.progress(p) })

Off until something turns it on. Turn it on with transcribe in the client, and say which steps with transcribeSteps (the speech steps of your protocol). A transcribe object in the kit (tenant) also turns transcription on, even if your code does not pass transcribe; the DevLab kit of every context protocol carries one by default (a language without a region), so, with a camera source, the capture opens the microphone unless your code passes transcribe: false. It is off by default in Measurement, which has no speech steps; if you turn it on there, Engine keeps the text and /results returns it as the transcript of step 1 (speech, only in the test environment’s Engine).

Speech recognition runs in the browser: the audio does not leave the browser; the text goes to Engine, which passes it to an LLM for the interpretation. The first time, it downloads about 160 MB (136 MB of them the model’s weights, from a third-party CDN; CSP and transcription); you can download it before Start (Prepare before Start). With a <video> as the source, the SDK transcribes the file’s own audio, without opening the microphone.

JavaScript
const client = createProtocolClient({ baseUrl: '/engine', tenant, transcribe: { language: 'en' }, transcribeSteps: ['free_speech'] })
When What happens
captureProtocol resolves The SDK opens GET /stream/{userProtocolId} through your proxy.
While capturing Events: connected, hci.warning, results.ready (once per position, and not in every case: the SDK keeps reconciling over HTTP; since 0.29.0 it is a known event, before it gave stream.unknown-event), and the interpretation’s events when you asked for one.
The stream drops The SDK reopens it, waiting 0.5 to 10 s; after three failed attempts it warns stream.unavailable and the capture goes on.
stop() With an interpretation on the way, the SDK waits on the stream for Engine’s notice (only this mechanic’s), then closes it.
abort() The stream closes.
The person signs out of your app An open stream survives it (your proxy’s authorize runs once, when it opens): close the capture when the person leaves, and limit streams per user in your backend.
The tab closes The capture stops; Engine keeps what it received, and the open position follows Engine’s session rules (Sessions).

Serve your page over HTTPS to the phone (a secure context: a tunnel to your development machine or a certificate for your local network, whichever you use), allow the camera, and read the rate at run time with onCodecEvent('capture-quality'): kodec reports about one window per second with the frames per second it measured. Skip the first two windows (the start-up) and the ones with the tab hidden (document.visibilityState), then read the median of the rest, how many windows fall under 8 fps and the lowest one. 8 fps is the only floor the SDK publishes. Below it, it does not estimate the pulse (an optional channel, off by default) and it treats a transcription veto as provisional. There is no published threshold for how the rate affects the indicators: compare your own sessions. The rule with its code, and the device check before measuring: Devices, quality and mode. SessionReport.captureQuality summarises the same windows but includes the start-up and the hidden ones, so do not judge from it alone; and capture.throttled is relative to the session’s own rate, so a phone that is slow from the start never raises it. What limits the rate is the face detection on the GPU (WebGL 2) and, on phones, heat in long sessions (Browsers and frame rate). The frame rate depends on the device, not on the platform: a recent phone usually stays close to the camera’s rate, an entry-level phone captures fewer frames per second, and a phone loses rate as it heats up in a long session. Read it at run time, not from the phone model. The helpers (keepScreenOn, guardUnload, classifyMediaError) are in Steps and marking; guardUnload depends on the browser: in Chrome for Android, closing the tab does not show the dialog; a closed tab ends the capture, so capture again (when recording in deferred, listPendingRecordings() finds on the next visit what was not uploaded: upload it with uploadRecording or discard it with its release()). The cheapest test: open the playground on your phone.

batchMs (5000 by default), fps, stepDurations ('warn' by default; 'enforce'; 'off'), requestTimeoutMs, outcomeDeadlineMs (240 s), analyzeTimeoutMs (280 s: the wait for the interpretation, and then the POST /analyze; since 0.44.0, after a 202 ANALYSIS_IN_PROGRESS the SDK waits and asks again within it, counted from the first 202, and ends with E_ANALYSIS_IN_PROGRESS if it runs out), credentials (since 0.45.0: 'omit', 'same-origin' or 'include', the credentials mode of the client’s requests to your proxy, the event stream and the streamed interpretation included; absent, the browser’s 'same-origin'; 'include' for a proxy on another origin that decides by your session cookie, The proxy on another origin), onProgress (only with interpretation: 'stream'), correlationId (a prefix for X-Correlation-Id), channels, kodec (see CSP and transcription), startup (timeouts, locale, messages, report, includeUserAgent; an unknown key throws UnknownOptionError) and onStartupStage (The startup screen). Since 0.42.0, createProtocolClient throws UnknownOptionError (E_UNKNOWN_OPTION) for any option it does not have; before, it ignored it. The deadlines are in Timeouts. Other options in the reference are for advanced tuning; leave them at their defaults unless Neurologyca tells you otherwise.