On-device (real time)
The live client: create it without a token, capture step by step, read the outcome, the warnings, the baseline, transcription and the event stream.
Processed in the browser as you capture. Try it in the playground: the default way there is this one.
The mode at a glance
Section titled “The mode at a glance”| Topic | On-device (real time) · Preview 0.45 |
|---|---|
| Today | Available. |
| Where it runs | In the browser, on the person's device, while they capture. |
| When the results arrive | During the session and seconds after the last batch in a short capture. A long capture, with thousands of frames, takes minutes. Past outcomeDeadlineMs (240 s by default) stop() gives E_RESULTS_TIMEOUT, which is not a loss: read the results later. |
| What leaves the device | Landmarks, expression coefficients and head pose, to Engine. Only if you turn them on: the transcript (speech recognition runs in the browser, so the audio stays and the text goes; Engine passes it to an LLM for the interpretation) and the extra signals (voice as numbers, pulse, hands, body). Never image, video or audio. |
| What limits you | The device: the face detection on the GPU (WebGL 2) and, on phones, heat in long sessions, which lower the frame rate. A recent phone or laptop usually stays close to the camera's rate; an entry-level phone captures fewer frames per second, and a phone loses rate as it heats up. |
| Browsers and phones | Chrome and Edge 94+, Safari 16.4+ (iOS 16.4+), on desktop and capable phones. |
| Measurement | Yes. |
| Typical cases | A short check-in on a laptop; a two-minute task on a recent phone; any capture whose video must not leave the device. |
| Moving between them | createProtocolClient (from @neurologyca/capture) ↔ createRecordingClient (from @neurologyca/capture/record, without kodec). Some options do not carry over: transcribe is on-device only; in deferred, the processing service transcribes. |
Create the client
Section titled “Create the client”One client per protocol context. Your page talks to your backend’s proxy: no getToken, no key, no token in the page (Your backend).
import { createProtocolClient, defineTenantConfig } from '@neurologyca/capture'import kit from './neuro.config.json' // the configuration of your context, from your kit
const client = createProtocolClient({ baseUrl: '/engine', // your proxy tenant: defineTenantConfig(kit), onStatus: (s) => ui.status(s.status), onResult: (r) => ui.result(r), // r.raw is the whole GET /results body onWarning: (w) => ui.warning(w.code, w.message), onError: (e) => ui.error(e.code, e.recoverable),})Capture
Section titled “Capture”Collect consent before captureProtocol, and call it only from a gesture of the person (a click): it opens the camera.
import { describeStep } from '@neurologyca/capture'
const session = await client.captureProtocol({ userProtocolId, protocolMechanicId, source: { type: 'webcam' } })for (const step of session.plan.steps) { const facts = describeStep(step, { lang: 'en' }) ui.step(step.order, facts.text?.value ?? null) await session.step(step.order).run({ durationMs: (facts.maxSeconds ?? 30) * 1000, mediaPid: facts.text?.mediaPid ?? null })}const outcome = await session.stop()This loop shows facts.text, the step’s own media. A step can carry no text of its own (an answer that keeps the previous question, or a reading that borrows the next step’s): to choose what to show in each step, use stepPrompt from the same loop, live and recorded.
With a short file, source.ended arrives on onWarning before the plan is over and a later run() fails: handle it by calling session.abort(), or end the loop after the warning.
The interpretation is requested by default: captureProtocol asks for it unless you pass triggerAnalysis: false. Receive it with onInterpretation (the narrative, with structured and metadata) and onInterpretationDetails (the same structured details, right after), both options of createProtocolClient.
Engine generates the interpretation once and stores it, and stop() waits for it (since 0.38.0): it listens for Engine’s notice on the event stream up to analyzeTimeoutMs (280 s by default; 120 s if the stream never connected) and, if the notice does not arrive, reads the stored one with a single POST /analyze. So stop() can take minutes: about 13 at worst with the defaults (Timeouts). Show the indicators while it waits. If you want progress by phase, pass interpretation: 'stream' to captureProtocol and onProgress to the client: onProgress({ phase, route }) receives the phases of /analyze/stream, which makes one generation that Engine does not store, so your backend will not read that same text later. interpretation: 'stream' is per capture: the client refuses it, and so does a capture with triggerAnalysis: false or in recording mode; with a capture token it produces no interpretation.
The pair (userProtocolId, protocolMechanicId) comes from your backend, which enrolled the person. Who closes each step (a cap, a button, the end of a video) and how the clock works: Steps and marking.
A <video> works as the source too ({ type: 'video', el }): give it its src before calling captureProtocol (the SDK waits for its metadata), keep it muted and playsInline, and play it in onStarted. When the file ends, the SDK warns source.ended once: a step driven by run() closes at that moment with the frames it got, and with start() and end() you call end() yourself. No later step can be captured, so call session.abort(). With several steps on one file, set loop on the video while the plan runs. session.abort() cancels any capture.
The startup screen
Section titled “The startup screen”Between the person’s click and the first step there are the Engine reads, the camera permission, the face model (about 13.5 MB the first time) and, with transcription, the speech model (about 160 MB: 136 MB of weights, plus the runtime and the library). On a slow mobile network that is minutes. Since 0.42.0 the SDK caps what does not depend on the person, lets you cancel, and tells you each stage with a message ready to show; it draws nothing.
import { createProtocolClient, defineTenantConfig, CaptureAbortedError, CaptureStartError } from '@neurologyca/capture'import kit from './neuro.config.json'
const client = createProtocolClient({ baseUrl: '/engine', tenant: defineTenantConfig(kit), startup: { timeouts: { speech: { totalMs: 300_000 } } }, // 5 min for speech instead of 10; null removes a cap transcribe: { language: 'en', progress: true }, // progress: speech bytes and its stall warning also where kodec does not prefetch // A preparation running in the background has its own events (kind 'prepare'): the screen paints the capture's. onStartupStage: (e) => { if (e.kind !== 'prepare' && e.userMessage !== null) ui.status(e.userMessage) }, onWarning: (w) => { if (w.code === 'startup.stalled') ui.offer(w.stage === 'speech' ? ['wait', 'cancel', 'without-voice'] : ['wait', 'cancel', 'reload']) },})
let cancel = null // the Cancel button calls cancel?.abort()async function start(pair, voice = true) { cancel = new AbortController() try { // Without voice: this capture opens no microphone and transcribes nothing, on the same client. return await client.captureProtocol({ ...pair, source: { type: 'webcam' }, signal: cancel.signal, ...(voice ? {} : { transcribe: false }) }) } catch (e) { if (e instanceof CaptureAbortedError || e instanceof CaptureStartError) { ui.status(e.userMessage ?? e.message); return null } throw e }}- The stages.
onStartupStageis called on every change of a stage (STARTUP_STAGES):engine-setup,device,camera,face-model,first-frame,speechandengine-session, with astatus(pending,running,stalled,done,failed,skipped). PaintuserMessageand you have the screen; it is instartup.locale('es'or'en'; by default the transcription’s language, then the kit’s, then the browser’s), andstartup.messagesreplaces any text by its key (STARTUP_MESSAGE_KEYS; the ready-made texts areSTARTUP_MESSAGES). kodec’s sub-stages (STARTUP_SUB_STAGES, withparent: 'face-model'or'speech') arrive too, with no message and withbytes,totalandbytesPerSec: for a finer progress bar or your telemetry. The same events come incaptureBaselineand in record mode. - Speech bytes and its stall. kodec always counts the bytes of
face-model, but those ofspeech(and with them itsstartup.stalledwarning) only where it prefetches the speech load: not on Safari or iOS, nor with an explicitmodel, an injectedtransformers, WebGPU without JSPI or WASM withoutselfHost.wasmPaths. Elsewhere onlytranscribe: { progress: true }gives them.speech.runningreads «Downloading speech recognition… (X of Y MB)» only with the bytes and the total, and on that path the total does not come (total: null, also with SDK 0.43.0 and kodec 5.0.0-rc.14): the text goes without the figure. Its cost: from an origin withoutContent-Lengththe library copies the whole buffer on every chunk, and on a phone that raises memory in the phase that already closes tabs. The default weights come withContent-Length; if you serve them yourself, send it too (CSP and transcription). - Texts that fit the source (since 0.42.1). The
face-model.*texts speak of getting face analysis ready, not of the camera: the face model loads the same with a webcam, astream, a<video>or a preparation with no source. A first-frame timeout usesfirst-frame.timeout-media(«No face can be seen in the video…») when the source is your ownstreamor<video>, and in the replay of a recording; with the webcam it is stillfirst-frame.timeout(«We cannot see you…»). The SDK picks it by source, in the stage event and inCaptureStartError.messageKeyanduserMessage. A live camera you pass as astreamgets the video’s wording too: replacefirst-frame.timeout-mediainstartup.messagesif you want the webcam’s. A text you replaced withstartup.messagesis kept, and a full set of your own texts typed asRecord<StartupMessageKey, string>has to add each new key (STARTUP_MESSAGE_KEYSlists twenty since 0.43.0, withfirst-frame.face-reload-failed). - The caps (defaults; Timeouts): the face model fails with
CaptureStartError(stage: 'face-model',reason: 'timeout') 120 s after the camera is granted; speech gives up after 600 s with thesttwarning withreason: 'timeout', and the capture goes on without it; 20 s without a new byte raisesstartup.stalled(one per stall), which cuts nothing.nullremoves a cap. The camera permission and the wait for the phone to be turned upright have no cap:signalcancels them. - Wait, cancel or go on without speech. On
startup.stalledofspeech, the person can wait (the bytes may come back), cancel (abort()of yourAbortController) or go on without speech: abort the start and callcaptureProtocol({ …, transcribe: false })again on the same client. In a plan with voice steps, those steps then reach Engine without a transcript. A stalledface-modelhas no «without speech»: wait, cancel or reload. - Cancelling.
signal(anAbortSignal) oncaptureProtocolandcaptureBaselineacts until the call resolves: the SDK releases the camera, the microphone and the pair’s slot, cancels the Engine session if it had opened, and rejects withCaptureAbortedError(E_CAPTURE_ABORTED). After it resolves, end the capture withsession.abort()orsession.stop(). - After
face-modelruns out, offer a reload. WithCaptureStartError.subStage'vision-runtime', a retry on the same page waits again for the hung module and runs out again; with'face-asset'or'vision-create'a retry is worth it. A reload works in every case, and theface-model.timeoutmessage suggests it. - The report. Every startup leaves a
StartupReport(stages and times, the device checks,outcomeready,partial,failed,abortedorunfinished). Live and incaptureBaselineit arrives inSessionReport.startupbyonReport, andSessionReport.transcriptionLoadsays how the speech load ended. To receive it on your server too (the only way in record mode, and the only one that sees a startup abandoned by closing the tab), passstartup.report: { url, on, context }:on: 'failures'sends only what did not go well, andcontexttakes up to 8 pairs of your own, values of up to 128 characters, with no personal data. Withoutreport.urlnothing leaves the page. The report carries the user agent unless you passstartup.includeUserAgent: false. Itsdevicelists the checks (DeviceCheck, ids inDeviceCheckId: the secure context,getUserMedia, the WebGPU adapter, JSPI, WASM SIMD and others), eachpass,warn,failorunknown, decided by capability and never by user agent; and, in a capture that was lent a preparation,prepared(PreparedBlock) says what it did with the face and with speech (adopted,stale,unusedornot-prepared) and, inprepareId, thestartupIdof the preparation’s own report. The receiver: Your backend.
The ready-made texts of STARTUP_MESSAGES (the twenty keys, in the language of startup.locale; {loadedMb} and {totalMb} are whole megabytes, and the span between [ and ] disappears if one is missing). This is the English set:
| Key | When | Text |
|---|---|---|
startup.preparing |
Any moment without a message of its own | Getting the capture ready… |
face-model.running |
The face model loads | Getting face analysis ready… |
face-model.cached |
It was already on the device | Getting face analysis ready (already on this device)… |
speech.running |
Speech recognition downloads (with the MB when they are known) | Downloading speech recognition…[ ({loadedMb} of {totalMb} MB)] |
speech.stalled |
The speech download is not moving | Your connection is slow and the speech download is not moving. You can wait or continue without speech recognition. |
speech.timeout |
Speech did not arrive in time | Speech recognition did not arrive in time. The capture continues without it. |
speech.unavailable |
This device cannot transcribe | This device cannot transcribe speech. The capture continues without it. |
face-model.stalled |
The face model load is not moving | Your connection is slow and getting face analysis ready is not moving. You can wait or reload the page. |
face-model.timeout |
The face model did not load in time | Face analysis could not be set up in time. Reload the page to try again. |
face-model.error |
The face model load failed | Face analysis could not be set up. Reload the page or try another network. |
first-frame.timeout |
The webcam shows no face | We cannot see you. Face the camera, with your face well lit, and try again. |
first-frame.timeout-media |
Your stream or <video> (and a replay) shows no face |
No face can be seen in the video. Check that a face is visible, facing forward and well lit, and try again. |
first-frame.face-reload-failed |
Face analysis stopped as the capture started and loading it again failed (since 0.43.0) | Face analysis stopped working as it started and could not be set up again. Reload the page to try again. |
camera.timeout |
The camera did not respond (reserved) | The camera did not respond in time. Reload the page to try again. |
capture.aborted |
The capture was cancelled | The capture was cancelled. |
startup.ready |
The startup is over | All set. |
prepare.speech-timeout |
In a preparation, speech did not finish downloading | Speech recognition did not finish downloading. You can start anyway: it will be tried again when you start. |
prepare.speech-error |
In a preparation, speech could not be downloaded | Speech recognition could not be downloaded. You can start anyway: it will be tried again when you start. |
prepare.aborted |
The preparation was cancelled | The preparation was cancelled. |
prepare.record-unsupported |
In a recording’s preparation, the device cannot record | This device can’t record the session. Try another browser or device. |
Prepare before Start
Section titled “Prepare before Start”It is not required: without it the capture loads what it needs when Start is pressed. client.prepare() moves that wait to a moment when the person is doing something else (a form, the instructions), and the same client’s next captureProtocol adopts what was loaded, with no option. It never calls Engine and never opens the camera. The recommended pattern: one client per page, kept for the whole visit; prepare when the page mounts, release when it goes away. Which of the five ways to load the models fits your flow, with a block for each: Load the models.
import { createProtocolClient, defineTenantConfig } from '@neurologyca/capture'import kit from './neuro.config.json'
// One client for the page: a capture adopts only what this same client prepared.const client = createProtocolClient({ baseUrl: '/engine', tenant: defineTenantConfig(kit) })
// When the page mounts. In React: useEffect(() => { void client.prepare(); return () => client.release() }, [client])const preparing = client.prepare() // the face only; speech too with { models: { face: true, speech: true } }, after consent
// The Start button, from the person's click: it adopts what was prepared and waits, under its own caps, for what is missing.function onStart(pair) { return client.captureProtocol({ ...pair, source: { type: 'webcam' } })}
// When the person leaves the page, or your view unmounts.function onLeave() { client.release()}- What it prepares. By default the face (about 13.5 MB). Speech (about 160 MB) only with
models: { speech: true }, which needs the SDK’s own transcription (transcribeon). Its download asks for no microphone permission, so ask for it after the person has agreed. On Safari and iOS only the face is prepared: speech loads when the capture starts (transcription: 'not-prepared',partialReason: 'speech-not-prepared'; it is not a failure). level.'memory'(the default) loads the models into the page, for this client’s next capture.'cache'only downloads them into the browser’s cache and leaves nothing to adopt: do not mix it with Start on the same page.- The result.
prepare()resolves aPrepareResult:outcome('ready','partial','failed'or'aborted'),partialReason,face,transcription,estimatedSpeechMs(a guide to the speech download, which tends to come out high),cache,report,messageKeyanduserMessage. It does not reject because of how the load went, except withthrowOnFailure: true, and then only on'failed'or'aborted':PrepareError(E_PREPARE), with the wholePrepareResultinresult.timeoutschanges the client’s caps for that call only, andsignalaborts it. - One preparation per client, one capture adopts it. Only the capture of the client that prepared adopts: a second client on the page loads everything again, and takes double the memory. Without
keepPrepared(below), speech is released after every session, so each capture needs its own preparation, and without the option a baseline does not adopt: prepare beforecaptureBaselineor after it ends. While a capture or a baseline of this client is running,prepare()rejects withPrepareErrorandreason: 'capture-in-progress'; prepare the next one once the current one ends. - Repeating and releasing. The same options return the same promise; any difference replaces what was prepared (the
prepare.releasedwarning,reason: 'replaced'; its other reasons are'release','abort','pagehide'and, withkeepPrepared,'unusable'), and a narrower call releases what it no longer asks for: decide once what to prepare.client.release()is synchronous and idempotent: it frees what no capture holds and resolves a running preparation'aborted'(one atlevel: 'cache'ends only by itssignal). Call it before dropping a client (a component that unmounts); otherwise the models stay loaded until the page goes away. The client also releases onpagehide. Theprepare.stalewarning says a capture could not use a prepared part (it loads it itself); with one client and its own configuration it should not appear. - If the prepared face fails. With or without
keepPrepared(since 0.43.0), a prepared face that rejects its first frames or loses its GPU context is loaded again once by kodec: theprepare.face-lostwarning, and the face still'adopted'in the report, withfaceReloaded: true. If that reload fails before the first frame, the start fails at once withCaptureStartError(stage: 'first-frame',reason: 'error', keyfirst-frame.face-reload-failed). A face the capture loaded itself warnsface.context-lostin the same case (Errors).
Keep the models between captures
Section titled “Keep the models between captures”Since 0.43.0, keepPrepared (off by default, live only) keeps what prepare(), each capture and each baseline loaded: when a session stops, the face models, and speech whether loaded or still loading, go back to the client, and the next capture adopts them. There is no prepare() between captures, and a capture that loaded its models with no prepare() keeps them too. With keepPrepared a baseline adopts the prepared face (or the one an earlier session kept) and leaves speech for the capture.
import { createProtocolClient, defineTenantConfig } from '@neurologyca/capture'import kit from './neuro.config.json'
const client = createProtocolClient({ baseUrl: '/engine', tenant: defineTenantConfig(kit), keepPrepared: true })
// Once, before the first baseline or capture (speech after consent). Nothing to prepare after that.const preparing = client.prepare({ models: { face: true, speech: true } })// captureBaseline(…) adopts the face; captureProtocol(…) and its stop(), then the next captureProtocol(…): each one// adopts what the previous one left.
// Between one person and the next on the same device, and before you drop the client (whether it prepared or not).function onNextPerson() { client.release()}- What it costs. The memory stays taken while the person does something else, until
client.release(), the page going away (pagehide, also into the browser’s page cache) or kodec leaving a kept part unusable (prepare.releasedwithreason: 'unusable'and itspart: only that part is released). That is why, with the option, every client needsclient.release()before you drop it, whether it prepared or not. Two clients on the page each keep their own, and take double.client.cacheStatus()does not see what is kept: it reads the browser’s cache, not the memory. - Prepare once. One
client.prepare({ models: { face: true, speech: true } })before the first capture or baseline; without it the first capture loads both itself, and keeps them. Aprepare()that asks only for what is kept resolves without loading anything (its report carrieskept: true); one that asks for a part that is not kept releases what is kept and loads everything again (prepare.released,'replaced'). - Without WebGPU, do not prepare between captures. Where transcription is refused (no WebGPU and no
allowWasmWorker) speech is never kept, so aprepare()that asks for speech between captures releases the kept face ('replaced') and loads it again every time. There, prepare once before the first capture and do not prepare between captures, or ask for the face only (prepare({ models: { face: true } })). - On Safari and iOS only the face is kept: speech loads again with every capture, and a baseline does not take a preparation whose speech is loaded or loading.
- Between people. What is kept is models, not what was captured: each session’s transcript is released when it closes, and nothing goes to disk. But the face model may carry state from the last frame of one capture into the next. If one device passes from one person to another, call
client.release()between one person and the next, or leave the option off (Privacy). - In the reports. A capture or a baseline that was lent kept models says so in its
StartupReport:prepared.origin('prepare','previous-session'ornullper part;null: that part was not among the kept models) andprepared.prepareId, which isnullwhen no part came from aprepare(). Those reports areschema: 2, and a receiver validates them withcapture-server0.19.0 or later (Your backend). In record mode the option is ignored, with therecording.option-ignoredwarning.
The cache, from another page
Section titled “The cache, from another page”- The cache.
client.cacheStatus()says what is already in the browser’s cache: onlyspeechWeightsandsttRuntimetell anything ('cached','not-cached'or'unknown');faceModel,sttLibraryandvisionRuntimeare always'unknown'. On an earlier page with no client (a landing page),prepareCapture({ level: 'cache' })warms the cache andcaptureCacheStatus()reads it, with the same configuration the capture page passes tocreateProtocolClient:
import { captureCacheStatus, prepareCapture } from '@neurologyca/capture'
// One module that both pages import, so the files warmed are the ones the capture asks for.const captureConfig = { transcribe: { language: 'en' } }
const cached = await captureCacheStatus(captureConfig)if (cached.speechWeights !== 'cached') await prepareCapture({ ...captureConfig, level: 'cache', prepare: { face: true, speech: true } })What it costs: the person’s data is spent even if they never start (about 175 MB cold, the face and speech), and the browser may clear its cache whenever it likes. A recording client also has prepare(), which loads no model: it only checks, before any permission, whether the device can record (Deferred (server-side)).
The outcome
Section titled “The outcome”stop() resolves with a CaptureOutcome: status ('complete', 'timeout', 'invalidated' or 'cancelled'), hcis (the means), hcisByStep, frameCount, interpretation (the narrative, or null) and, since 0.39.0, structured (the interpretation’s structured data, or null; read a 0 to 100 index from it with readScoreIndex, see A 0 to 100 index). The fields and the scales: HCIs and results.
If you asked for an interpretation, onInterpretationDetails fires right after onInterpretation (since 0.38.0), so you can save both together. With interpretation: 'stream' the details come later, normally after stop() has resolved: wait for them with a deadline of your own, and save without them if it passes. A processing.error from Engine while stop() waits for the interpretation ends the session 'invalidated': no interpretation is coming.
Past outcomeDeadlineMs (240 s by default), E_RESULTS_TIMEOUT: it is not a loss. Engine has the capture; keep the pair and read the results later (Results in your backend).
Errors and warnings
Section titled “Errors and warnings”Three families: pipeline failures by onError ({ code, recoverable, stepOrder? }), usage errors thrown to you, and warnings. The rule: a recoverable other than true during the steps means the capture is lost, so call abort(); after stop(), do not: codes such as E_RESULTS_TIMEOUT arrive with recoverable: false but Engine already has the capture.
The warnings worth showing (the codes are strings, not a published union):
source.ended: your file or stream is over. A step driven byrun()closes; withstart()andend()you close it yourself withend(). No later step can be captured.step.duration-below-minandstep.duration-above-max: a step closed outside its window.step.media-undeclared: the step has several media and you did not say which one you showed.capture.throttledis relative to the session’s own rate: it fires when the rate falls to a quarter of that session’s own for three windows in a row, with a step open (once per step), typically a hidden tab. A device that is slow from the first window never raises it, so it is not the signal for how fast the device is (read it as in Try it on your phone).quality.warning: Engine says the quality of what reaches it is not good enough.stream.unavailable: the event stream did not connect; the capture goes on.interpretation.sse-timeoutandanalysis.regenerated(since 0.38.0): Engine’s notice did not arrive in time and the interpretation was read withPOST /analyze; withanalysis.regenerated,/analyzehad to generate a new one, and the session may have been billed twice. Both name the cause; if the stream dropped during the wait, check that your proxy forwards/stream/{userProtocolId}.transcription.off-on-voice-stepandtranscribeSteps.excludes-voice-step(since 0.38.0, before the camera opens): a step where the person is expected to speak (expectsSpeech: 'yes'in the plan) will not be transcribed, becausetranscribeis off ortranscribeStepsleaves it out.expectsSpeechis'yes','no','any'(the person may or may not talk),null(not declared) or another value the catalogue declares, and the SDK only warns with'yes'.transcribeSteps.noMatch: no type intranscribeStepsexists in the plan, so no step is transcribed.protocol.not-interpreted(since 0.29.0): Engine says the protocol is not interpreted (Measurement): no taxonomy is sent and no analysis asked for, even withtriggerAnalysis: true.interpretationwill benull.
E_CONFLICT comes with reason and conflict.<field>. Every code, by layer: Errors, warnings and retries.
Baseline
Section titled “Baseline”Read it with client.getBaselineInfo(userId) (null means none yet) and capture it with client.captureBaseline({ userId, source, onStarted, onProgress }); with keepPrepared it adopts the prepared face (Keep the models between captures). Engine computes it in the background: read it again until it is there. With a <video>, the capture stops when the file ends. The baseline always goes through the live client: the recording client has no captureBaseline. See Baseline.
const info = await client.getBaselineInfo(userId) // null: no baseline yetif (!info) await client.captureBaseline({ userId, source: { type: 'webcam' }, onProgress: (p) => ui.progress(p) })Transcription
Section titled “Transcription”Off until something turns it on. Turn it on with transcribe in the client, and say which steps with transcribeSteps (the speech steps of your protocol). A transcribe object in the kit (tenant) also turns transcription on, even if your code does not pass transcribe; the DevLab kit of every context protocol carries one by default (a language without a region), so, with a camera source, the capture opens the microphone unless your code passes transcribe: false. It is off by default in Measurement, which has no speech steps; if you turn it on there, Engine keeps the text and /results returns it as the transcript of step 1 (speech, only in the test environment’s Engine).
Speech recognition runs in the browser: the audio does not leave the browser; the text goes to Engine, which passes it to an LLM for the interpretation. The first time, it downloads about 160 MB (136 MB of them the model’s weights, from a third-party CDN; CSP and transcription); you can download it before Start (Prepare before Start). With a <video> as the source, the SDK transcribes the file’s own audio, without opening the microphone.
const client = createProtocolClient({ baseUrl: '/engine', tenant, transcribe: { language: 'en' }, transcribeSteps: ['free_speech'] })The session and its event stream
Section titled “The session and its event stream”| When | What happens |
|---|---|
captureProtocol resolves |
The SDK opens GET /stream/{userProtocolId} through your proxy. |
| While capturing | Events: connected, hci.warning, results.ready (once per position, and not in every case: the SDK keeps reconciling over HTTP; since 0.29.0 it is a known event, before it gave stream.unknown-event), and the interpretation’s events when you asked for one. |
| The stream drops | The SDK reopens it, waiting 0.5 to 10 s; after three failed attempts it warns stream.unavailable and the capture goes on. |
stop() |
With an interpretation on the way, the SDK waits on the stream for Engine’s notice (only this mechanic’s), then closes it. |
abort() |
The stream closes. |
| The person signs out of your app | An open stream survives it (your proxy’s authorize runs once, when it opens): close the capture when the person leaves, and limit streams per user in your backend. |
| The tab closes | The capture stops; Engine keeps what it received, and the open position follows Engine’s session rules (Sessions). |
Try it on your phone
Section titled “Try it on your phone”Serve your page over HTTPS to the phone (a secure context: a tunnel to your development machine or a certificate for your local network, whichever you use), allow the camera, and read the rate at run time with onCodecEvent('capture-quality'): kodec reports about one window per second with the frames per second it measured. Skip the first two windows (the start-up) and the ones with the tab hidden (document.visibilityState), then read the median of the rest, how many windows fall under 8 fps and the lowest one. 8 fps is the only floor the SDK publishes. Below it, it does not estimate the pulse (an optional channel, off by default) and it treats a transcription veto as provisional. There is no published threshold for how the rate affects the indicators: compare your own sessions. The rule with its code, and the device check before measuring: Devices, quality and mode. SessionReport.captureQuality summarises the same windows but includes the start-up and the hidden ones, so do not judge from it alone; and capture.throttled is relative to the session’s own rate, so a phone that is slow from the start never raises it. What limits the rate is the face detection on the GPU (WebGL 2) and, on phones, heat in long sessions (Browsers and frame rate). The frame rate depends on the device, not on the platform: a recent phone usually stays close to the camera’s rate, an entry-level phone captures fewer frames per second, and a phone loses rate as it heats up in a long session. Read it at run time, not from the phone model. The helpers (keepScreenOn, guardUnload, classifyMediaError) are in Steps and marking; guardUnload depends on the browser: in Chrome for Android, closing the tab does not show the dialog; a closed tab ends the capture, so capture again (when recording in deferred, listPendingRecordings() finds on the next visit what was not uploaded: upload it with uploadRecording or discard it with its release()). The cheapest test: open the playground on your phone.
Options
Section titled “Options”batchMs (5000 by default), fps, stepDurations ('warn' by default; 'enforce'; 'off'), requestTimeoutMs, outcomeDeadlineMs (240 s), analyzeTimeoutMs (280 s: the wait for the interpretation, and then the POST /analyze; since 0.44.0, after a 202 ANALYSIS_IN_PROGRESS the SDK waits and asks again within it, counted from the first 202, and ends with E_ANALYSIS_IN_PROGRESS if it runs out), credentials (since 0.45.0: 'omit', 'same-origin' or 'include', the credentials mode of the client’s requests to your proxy, the event stream and the streamed interpretation included; absent, the browser’s 'same-origin'; 'include' for a proxy on another origin that decides by your session cookie, The proxy on another origin), onProgress (only with interpretation: 'stream'), correlationId (a prefix for X-Correlation-Id), channels, kodec (see CSP and transcription), startup (timeouts, locale, messages, report, includeUserAgent; an unknown key throws UnknownOptionError) and onStartupStage (The startup screen). Since 0.42.0, createProtocolClient throws UnknownOptionError (E_UNKNOWN_OPTION) for any option it does not have; before, it ignored it. The deadlines are in Timeouts. Other options in the reference are for advanced tuning; leave them at their defaults unless Neurologyca tells you otherwise.