Steps and marking
How to drive each step without a component: its data, who closes it, the clock, the duration checks, the media you showed and the manifest of a recording.
There is no capture component: your page draws each step and decides when it ends. The SDK gives you the data and checks the durations.
The data of a step
Section titled “The data of a step”describeStep(step, { lang }) reads a plan step without rules per type.
| Field | What it is |
|---|---|
StepFacts.minSeconds |
The minimum duration of the step, or null if the catalogue does not say |
StepFacts.maxSeconds |
The maximum duration of the step, or null |
StepFacts.responseClock |
The catalogue value, which the SDK does not interpret ('paused', 'running', another value or null; it is null in many step types). Decide what it means per protocol |
StepFacts.expectsSpeech |
Whether the step expects the person to speak: 'yes', 'no', 'any' or null; 'yes' marks an answer step (below) |
StepFacts.text |
The text to show, { value, lang, mediaPid }, or null |
StepFacts.texts |
Every text of the step, with its mediaPid and one body per language |
StepFacts.phases |
The phases of the step, when it has them |
StepFacts.rounds |
The number of rounds, or null |
StepFacts.bands |
The time bands of the step, when it has them |
StepFacts.unknown |
What the catalogue did not say: 'duration', 'text', 'responseClock', 'bands' |
The language: the one you ask for, then the plan’s default.
Who closes a step
Section titled “Who closes a step”run({ durationMs }) closes it after that time (a cap). start() and end() let you close it yourself: start the step, then call end() on whichever happens first, your button, maxSeconds or the end of your video (the source.ended warning). The order is strict: one step at a time, in the plan’s order (E_STEP_SEQUENCE). start() returns a promise: await it. Live it settles at once (resolves, or rejects when called); recording, it can also reject later. Only a wrong order (StepSequenceError) is thrown synchronously; anything else arrives as a rejected promise, which a try/catch without await does not catch.
The clock
Section titled “The clock”session.stepTiming() returns null outside a step, or a StepTiming:
| Field | What it is |
|---|---|
StepTiming.elapsedMs |
Milliseconds since the step started |
StepTiming.minMs |
The minimum, in milliseconds, or null |
StepTiming.maxMs |
The maximum, in milliseconds, or null |
StepTiming.minReached |
Whether the minimum has passed; null when there is no minimum |
StepTiming.maxReached |
Whether the maximum has passed; null when there is no maximum |
StepTiming.remainingMs |
Milliseconds left until the maximum, or null |
Enable your button when minReached is not false, and draw remainingMs as a countdown.
Checking the durations
Section titled “Checking the durations”stepDurations in the client: 'warn' (the default) warns when a step ends outside its window (step.duration-below-min, step.duration-above-max); 'enforce' refuses with E_STEP_DURATION; 'off' checks nothing. With 'enforce':
- a
run({ durationMs })out of range is rejected before the step opens; - an
end()before the minimum is rejected and the step stays open: callend()again whenstepTiming().minReachedistrue; - past the maximum,
end()only warns: closing the step is yours; - with a
<video>or stream source that has already ended,'enforce'drops to'warn', because there is nothing left to wait for.
Warnings have 250 ms of tolerance. With no minimum and no maximum nothing is checked.
The media you showed
Section titled “The media you showed”Declare the text you showed with mediaPid, once per step: start({ mediaPid }) or run({ durationMs, mediaPid }). When the step has one text, facts.text.mediaPid is it. When it has several (alternative questions), choose one from describeStep(step).texts, which lists every text media as { mediaPid, body }: facts.text is only the first that matches the language, so taking it blindly is a guess. A step with several media and no mediaPid warns step.media-undeclared, and the interpretation then lacks the question (metadata.missing_inputs includes question_text; see the pitfall). The warning only fires in steps without phases: in a step with phases the text media are the cues of each phase, not alternative questions, and there is nothing to choose. A mediaPid that is not the step’s throws E_MEDIA_UNKNOWN.
Steps without a clock or without text
Section titled “Steps without a clock or without text”A step without a clock or without text of its own is not an answer step by default. Never decide its role by the name of the step type or by responseClock: the catalogue says it. What each step shows:
- Its own media: the text you declare with
mediaPid(above). - No media of its own:
delivery.media_fromsays where its text comes from.'previous_step': keep the previous step’s question on screen.'next_step': a silent reading step shows the question chosen for the next step; it lends the text, never the media, so declare nomediaPidthere. - An answer step (
category: 'answer'orexpectsSpeech: 'yes') keeps the previous question on screen and shows its own media as the line under it (unless the previous step is a silent reading of this step’s own question: then its own media, once). Any other step is not an answer step: do not show it as “answer now”. - Show only media bodies, never
delivery.instructionsordelivery.prompt.
StepFacts carries expectsSpeech, but not category nor media_from today: read them from the plan step in session.plan.steps (PlanStep.category, and media_from inside PlanStep.delivery, which StepDelivery does not type). Asked of the SDK. The pitfall has the symptom.
When the catalogue gives no duration (unknown lists 'duration'), how long the step lasts is your decision: close it with your button, and write the decision down per protocol.
Marking in deferred: the manifest
Section titled “Marking in deferred: the manifest”When recording, the SDK writes a manifest with the video. Each ManifestStep says where the step starts and ends in the video and who ended it:
| Field | What it is |
|---|---|
ManifestStep.startMediaMs |
Where the step starts in the video, in milliseconds |
ManifestStep.endMediaMs |
Where the step ends in the video, in milliseconds |
ManifestStep.mediaPid |
The media you declared for the step |
ManifestStep.endedBy |
How it was closed: 'timer' (run()), 'gesture' (end()) or, since 0.28.0, 'source' (the end of the source closed it: a file or a stream that finished, whether run() or end() was waiting). The type also allows 'media' and 'cap', which the Guide does not document. The processor does not read it |
And the rest of the manifest:
| Field | What it is |
|---|---|
RecordingManifest.prerollMs |
The pre-roll recorded before the first step |
RecordingManifest.stopMediaMs |
Where the recording stops in the video |
RecordingManifest.hidden |
When the tab was hidden during the recording |
RecordingManifest.orientationChanges |
The orientation changes of the device |
RecordingManifest.audioGaps |
The gaps in the audio |
RecordingManifest.videoGaps |
The gaps in the video |
RecordingManifest.stimulus |
The timing of the video stimulus, if you tracked one |
RecordingManifest.interpreted |
Since 0.29.0: false when Engine said the protocol is not interpreted (Measurement), and absent otherwise. It only goes with triggerAnalysis: false, and the processing service’s replay then sends no taxonomy and asks for no analysis either |
The pre-roll and post-roll are recording options (prerollMs, postrollMs), not manifest fields. If you show a video stimulus, session.trackStimulus(order, el) records its timing. The manifest is not frozen yet: read only these fields.
Helpers
Section titled “Helpers”From @neurologyca/capture/helpers: keepScreenOn (the screen stays on), guardUnload (asks before closing the tab while a capture or upload runs), classifyMediaError (turns a camera error into a reason you can show), waitForPortraitOrientation and startOrAsk (on a POSITION_RESERVED it asks the person, and repeats with discardPending: true only on their yes; since 0.29.0, if the holder is already receiving frames, holder.receivingPackets: true, it does not ask: Engine would refuse the discard, so it throws the POSITION_RESERVED again, and you try later). keepScreenOn and classifyMediaError follow the browsers’ public documentation. guardUnload depends on the browser: in Chrome for Android, closing the tab does not show the dialog, so do not count on it to keep a capture open on a phone. When recording, the local copy is what protects you there: on the next visit, listPendingRecordings() returns the recordings that were not uploaded (as RecordingResult); upload each one with uploadRecording or discard it with its release(). Live, a closed tab ends the capture: capture again.
What the SDK does not do with a plan
Section titled “What the SDK does not do with a plan”flowMode: 'free' mechanics (microinterventions) are not supported (E_FLOW_UNSUPPORTED), phases are not driven, and there are no interruptions: your page handles them.
The same loop, live and recorded
Section titled “The same loop, live and recorded”What each step shows comes from stepPrompt, the rule of the section above and of the playground: the question (text), the line under it on an answer step (cue), kind ('question', 'answer' or 'silent') and the mediaPid to declare. Show «answer now» only when kind is 'answer'. It reads category and media_from from the plan step, because StepFacts does not carry them:
// What each step shows, from the plan: its own question; without one, the previous step's (media_from 'previous_step')// or the next step's text (media_from 'next_step', a silent reading step: it lends the text, never the media).// An answer step (category 'answer' or expectsSpeech 'yes') keeps the previous question on screen, its own media under it.// Only media bodies, never delivery.instructions or delivery.prompt.import { describeStep } from '@neurologyca/capture/record' // the light entry: it does not download kodec
/** * @typedef {{ kind: 'question' | 'answer' | 'silent', text: string | null, cue: string | null, mediaPid: string | null }} StepPrompt * @param {import('@neurologyca/capture/record').PlanStep[]} steps session.plan.steps * @param {number} order the step to show * @param {Record<string, string>} choice the mediaPid you chose per step order; absent, the step's first text * @param {string} lang the spoken language * @returns {StepPrompt | null} mediaPid: what to declare in start({ mediaPid }), null when the step has no media of its own */export function stepPrompt(steps, order, choice, lang) { const sorted = [...steps].sort((a, b) => a.order - b.order) const at = sorted.findIndex((s) => s.order === order) if (at < 0) return null const step = sorted[at], prev = sorted[at - 1], next = sorted[at + 1] /** @param {import('@neurologyca/capture/record').PlanStep} s */ const pidOf = (s) => choice[String(s.order)] ?? describeStep(s).texts[0]?.mediaPid ?? null /** @param {import('@neurologyca/capture/record').PlanStep} s */ const own = (s) => { const body = describeStep(s).texts.find((t) => t.mediaPid === pidOf(s))?.body return body ? (body[lang] ?? body.en ?? Object.values(body)[0] ?? null) : null } /** @param {import('@neurologyca/capture/record').PlanStep} s */ const isAnswer = (s) => s.category === 'answer' || s.expectsSpeech === 'yes' const from = step.delivery?.media_from // not in StepFacts today: read it from the plan step /** @type {StepPrompt['kind']} */ const kind = isAnswer(step) ? 'answer' : from === 'next_step' && step.expectsSpeech === 'no' ? 'silent' : 'question' const mine = own(step) if (mine === null) { const lent = from === 'previous_step' && prev ? stepPrompt(steps, prev.order, choice, lang)?.text ?? null : from === 'next_step' && next ? own(next) : null return lent === null ? null : { kind, text: lent, cue: null, mediaPid: null } } // Not after a next_step reading step: that one already showed this step's own question. if (isAnswer(step) && prev && !isAnswer(prev) && prev.delivery?.media_from !== 'next_step') { const question = stepPrompt(steps, prev.order, choice, lang)?.text ?? null if (question !== null) return { kind, text: question, cue: mine, mediaPid: pidOf(step) } } return { kind, text: mine, cue: null, mediaPid: pidOf(step) }}Live, a button closes each step once its minimum is reached, and its maximum closes it on its own:
// The step loop, live: a button closes each step once its minimum is reached, and its maximum closes it on its own.// stepPrompt (step-prompt.js) says what each step shows: the question, and on an answer step the question kept on screen.import { stepPrompt } from './step-prompt.js'
/** * @param {import('@neurologyca/capture').ProtocolClientHandle} client * @param {{ userProtocolId: string, protocolMechanicId: string }} pair from your backend * @param {HTMLButtonElement} next your "Next" button * @param {(p: import('./step-prompt.js').StepPrompt | null, remainingMs: number | null) => void} show your screen * @param {{ choice?: Record<string, string>, lang?: string }} [o] the question chosen per step, the spoken language */export async function captureSteps(client, pair, next, show, { choice = {}, lang = 'en' } = {}) { const session = await client.captureProtocol({ ...pair, source: { type: 'webcam' } }) const steps = session.plan.steps for (const step of steps) { const prompt = stepPrompt(steps, step.order, choice, lang) await session.step(step.order).start({ mediaPid: prompt?.mediaPid ?? null }) // settles at once when live await new Promise((/** @type {(v?: unknown) => void} */ done) => { const tick = setInterval(() => { const t = session.stepTiming() next.disabled = t?.minReached === false // enabled once the minimum is reached show(prompt, t?.remainingMs ?? null) // "answer now" only when prompt.kind is 'answer' if (t?.maxReached) finish() }, 250) const finish = () => { clearInterval(tick); next.onclick = null; done() } next.onclick = finish }) await session.step(step.order).end() } return session.stop() // CaptureOutcome}Recorded, the steps are marked in the video, and the manifest says where each one starts and ends:
// The same loop, recorded: the steps are marked in the video, and the manifest says where each one starts and ends.import { createRecordingClient, uploadRecording } from '@neurologyca/capture/record'import { stepPrompt } from './step-prompt.js'
/** * @param {{ userProtocolId: string, protocolMechanicId: string }} pair from your backend * @param {HTMLButtonElement} next your "Next" button * @param {(p: import('./step-prompt.js').StepPrompt | null) => void} show your screen */export async function recordSteps(pair, next, show) { const recorder = createRecordingClient({ baseUrl: '/engine', taxonomy: { contextId: 'measurement' }, stepDurations: 'warn' }) const session = await recorder.captureProtocol({ ...pair, source: { type: 'webcam' }, triggerAnalysis: false }) const steps = session.plan.steps for (const step of steps) { const prompt = stepPrompt(steps, step.order, {}, 'en') show(prompt) await session.step(step.order).start({ mediaPid: prompt?.mediaPid ?? null }) // can reject when recording await new Promise((done) => { next.onclick = () => { next.onclick = null; done(undefined) } }) await session.step(step.order).end() } const recording = await session.stop() // the video and its manifest, kept in this browser until uploaded console.log(recording.manifest.steps.map((s) => [s.order, s.startMediaMs, s.endMediaMs, s.endedBy])) return uploadRecording(recording, { url: '/uploads' }) // your receiver}