Skip to content
DevLab

Steps and marking

How to drive each step without a component: its data, who closes it, the clock, the duration checks, the media you showed and the manifest of a recording.

On this page
  1. The data of a step
  2. Who closes a step
  3. The clock
  4. Checking the durations
  5. The media you showed
  6. Steps without a clock or without text
  7. Marking in deferred: the manifest
  8. Helpers
  9. What the SDK does not do with a plan
  10. The same loop, live and recorded

There is no capture component: your page draws each step and decides when it ends. The SDK gives you the data and checks the durations.

describeStep(step, { lang }) reads a plan step without rules per type.

Field What it is
StepFacts.minSeconds The minimum duration of the step, or null if the catalogue does not say
StepFacts.maxSeconds The maximum duration of the step, or null
StepFacts.responseClock The catalogue value, which the SDK does not interpret ('paused', 'running', another value or null; it is null in many step types). Decide what it means per protocol
StepFacts.expectsSpeech Whether the step expects the person to speak: 'yes', 'no', 'any' or null; 'yes' marks an answer step (below)
StepFacts.text The text to show, { value, lang, mediaPid }, or null
StepFacts.texts Every text of the step, with its mediaPid and one body per language
StepFacts.phases The phases of the step, when it has them
StepFacts.rounds The number of rounds, or null
StepFacts.bands The time bands of the step, when it has them
StepFacts.unknown What the catalogue did not say: 'duration', 'text', 'responseClock', 'bands'

The language: the one you ask for, then the plan’s default.

run({ durationMs }) closes it after that time (a cap). start() and end() let you close it yourself: start the step, then call end() on whichever happens first, your button, maxSeconds or the end of your video (the source.ended warning). The order is strict: one step at a time, in the plan’s order (E_STEP_SEQUENCE). start() returns a promise: await it. Live it settles at once (resolves, or rejects when called); recording, it can also reject later. Only a wrong order (StepSequenceError) is thrown synchronously; anything else arrives as a rejected promise, which a try/catch without await does not catch.

session.stepTiming() returns null outside a step, or a StepTiming:

Field What it is
StepTiming.elapsedMs Milliseconds since the step started
StepTiming.minMs The minimum, in milliseconds, or null
StepTiming.maxMs The maximum, in milliseconds, or null
StepTiming.minReached Whether the minimum has passed; null when there is no minimum
StepTiming.maxReached Whether the maximum has passed; null when there is no maximum
StepTiming.remainingMs Milliseconds left until the maximum, or null

Enable your button when minReached is not false, and draw remainingMs as a countdown.

stepDurations in the client: 'warn' (the default) warns when a step ends outside its window (step.duration-below-min, step.duration-above-max); 'enforce' refuses with E_STEP_DURATION; 'off' checks nothing. With 'enforce':

  • a run({ durationMs }) out of range is rejected before the step opens;
  • an end() before the minimum is rejected and the step stays open: call end() again when stepTiming().minReached is true;
  • past the maximum, end() only warns: closing the step is yours;
  • with a <video> or stream source that has already ended, 'enforce' drops to 'warn', because there is nothing left to wait for.

Warnings have 250 ms of tolerance. With no minimum and no maximum nothing is checked.

Declare the text you showed with mediaPid, once per step: start({ mediaPid }) or run({ durationMs, mediaPid }). When the step has one text, facts.text.mediaPid is it. When it has several (alternative questions), choose one from describeStep(step).texts, which lists every text media as { mediaPid, body }: facts.text is only the first that matches the language, so taking it blindly is a guess. A step with several media and no mediaPid warns step.media-undeclared, and the interpretation then lacks the question (metadata.missing_inputs includes question_text; see the pitfall). The warning only fires in steps without phases: in a step with phases the text media are the cues of each phase, not alternative questions, and there is nothing to choose. A mediaPid that is not the step’s throws E_MEDIA_UNKNOWN.

A step without a clock or without text of its own is not an answer step by default. Never decide its role by the name of the step type or by responseClock: the catalogue says it. What each step shows:

  • Its own media: the text you declare with mediaPid (above).
  • No media of its own: delivery.media_from says where its text comes from. 'previous_step': keep the previous step’s question on screen. 'next_step': a silent reading step shows the question chosen for the next step; it lends the text, never the media, so declare no mediaPid there.
  • An answer step (category: 'answer' or expectsSpeech: 'yes') keeps the previous question on screen and shows its own media as the line under it (unless the previous step is a silent reading of this step’s own question: then its own media, once). Any other step is not an answer step: do not show it as “answer now”.
  • Show only media bodies, never delivery.instructions or delivery.prompt.

StepFacts carries expectsSpeech, but not category nor media_from today: read them from the plan step in session.plan.steps (PlanStep.category, and media_from inside PlanStep.delivery, which StepDelivery does not type). Asked of the SDK. The pitfall has the symptom.

When the catalogue gives no duration (unknown lists 'duration'), how long the step lasts is your decision: close it with your button, and write the decision down per protocol.

When recording, the SDK writes a manifest with the video. Each ManifestStep says where the step starts and ends in the video and who ended it:

Field What it is
ManifestStep.startMediaMs Where the step starts in the video, in milliseconds
ManifestStep.endMediaMs Where the step ends in the video, in milliseconds
ManifestStep.mediaPid The media you declared for the step
ManifestStep.endedBy How it was closed: 'timer' (run()), 'gesture' (end()) or, since 0.28.0, 'source' (the end of the source closed it: a file or a stream that finished, whether run() or end() was waiting). The type also allows 'media' and 'cap', which the Guide does not document. The processor does not read it

And the rest of the manifest:

Field What it is
RecordingManifest.prerollMs The pre-roll recorded before the first step
RecordingManifest.stopMediaMs Where the recording stops in the video
RecordingManifest.hidden When the tab was hidden during the recording
RecordingManifest.orientationChanges The orientation changes of the device
RecordingManifest.audioGaps The gaps in the audio
RecordingManifest.videoGaps The gaps in the video
RecordingManifest.stimulus The timing of the video stimulus, if you tracked one
RecordingManifest.interpreted Since 0.29.0: false when Engine said the protocol is not interpreted (Measurement), and absent otherwise. It only goes with triggerAnalysis: false, and the processing service’s replay then sends no taxonomy and asks for no analysis either

The pre-roll and post-roll are recording options (prerollMs, postrollMs), not manifest fields. If you show a video stimulus, session.trackStimulus(order, el) records its timing. The manifest is not frozen yet: read only these fields.

From @neurologyca/capture/helpers: keepScreenOn (the screen stays on), guardUnload (asks before closing the tab while a capture or upload runs), classifyMediaError (turns a camera error into a reason you can show), waitForPortraitOrientation and startOrAsk (on a POSITION_RESERVED it asks the person, and repeats with discardPending: true only on their yes; since 0.29.0, if the holder is already receiving frames, holder.receivingPackets: true, it does not ask: Engine would refuse the discard, so it throws the POSITION_RESERVED again, and you try later). keepScreenOn and classifyMediaError follow the browsers’ public documentation. guardUnload depends on the browser: in Chrome for Android, closing the tab does not show the dialog, so do not count on it to keep a capture open on a phone. When recording, the local copy is what protects you there: on the next visit, listPendingRecordings() returns the recordings that were not uploaded (as RecordingResult); upload each one with uploadRecording or discard it with its release(). Live, a closed tab ends the capture: capture again.

flowMode: 'free' mechanics (microinterventions) are not supported (E_FLOW_UNSUPPORTED), phases are not driven, and there are no interruptions: your page handles them.

What each step shows comes from stepPrompt, the rule of the section above and of the playground: the question (text), the line under it on an answer step (cue), kind ('question', 'answer' or 'silent') and the mediaPid to declare. Show «answer now» only when kind is 'answer'. It reads category and media_from from the plan step, because StepFacts does not carry them:

JavaScript
// What each step shows, from the plan: its own question; without one, the previous step's (media_from 'previous_step')
// or the next step's text (media_from 'next_step', a silent reading step: it lends the text, never the media).
// An answer step (category 'answer' or expectsSpeech 'yes') keeps the previous question on screen, its own media under it.
// Only media bodies, never delivery.instructions or delivery.prompt.
import { describeStep } from '@neurologyca/capture/record' // the light entry: it does not download kodec
/**
* @typedef {{ kind: 'question' | 'answer' | 'silent', text: string | null, cue: string | null, mediaPid: string | null }} StepPrompt
* @param {import('@neurologyca/capture/record').PlanStep[]} steps session.plan.steps
* @param {number} order the step to show
* @param {Record<string, string>} choice the mediaPid you chose per step order; absent, the step's first text
* @param {string} lang the spoken language
* @returns {StepPrompt | null} mediaPid: what to declare in start({ mediaPid }), null when the step has no media of its own
*/
export function stepPrompt(steps, order, choice, lang) {
const sorted = [...steps].sort((a, b) => a.order - b.order)
const at = sorted.findIndex((s) => s.order === order)
if (at < 0) return null
const step = sorted[at], prev = sorted[at - 1], next = sorted[at + 1]
/** @param {import('@neurologyca/capture/record').PlanStep} s */
const pidOf = (s) => choice[String(s.order)] ?? describeStep(s).texts[0]?.mediaPid ?? null
/** @param {import('@neurologyca/capture/record').PlanStep} s */
const own = (s) => {
const body = describeStep(s).texts.find((t) => t.mediaPid === pidOf(s))?.body
return body ? (body[lang] ?? body.en ?? Object.values(body)[0] ?? null) : null
}
/** @param {import('@neurologyca/capture/record').PlanStep} s */
const isAnswer = (s) => s.category === 'answer' || s.expectsSpeech === 'yes'
const from = step.delivery?.media_from // not in StepFacts today: read it from the plan step
/** @type {StepPrompt['kind']} */
const kind = isAnswer(step) ? 'answer' : from === 'next_step' && step.expectsSpeech === 'no' ? 'silent' : 'question'
const mine = own(step)
if (mine === null) {
const lent = from === 'previous_step' && prev ? stepPrompt(steps, prev.order, choice, lang)?.text ?? null
: from === 'next_step' && next ? own(next) : null
return lent === null ? null : { kind, text: lent, cue: null, mediaPid: null }
}
// Not after a next_step reading step: that one already showed this step's own question.
if (isAnswer(step) && prev && !isAnswer(prev) && prev.delivery?.media_from !== 'next_step') {
const question = stepPrompt(steps, prev.order, choice, lang)?.text ?? null
if (question !== null) return { kind, text: question, cue: mine, mediaPid: pidOf(step) }
}
return { kind, text: mine, cue: null, mediaPid: pidOf(step) }
}

Live, a button closes each step once its minimum is reached, and its maximum closes it on its own:

JavaScript
// The step loop, live: a button closes each step once its minimum is reached, and its maximum closes it on its own.
// stepPrompt (step-prompt.js) says what each step shows: the question, and on an answer step the question kept on screen.
import { stepPrompt } from './step-prompt.js'
/**
* @param {import('@neurologyca/capture').ProtocolClientHandle} client
* @param {{ userProtocolId: string, protocolMechanicId: string }} pair from your backend
* @param {HTMLButtonElement} next your "Next" button
* @param {(p: import('./step-prompt.js').StepPrompt | null, remainingMs: number | null) => void} show your screen
* @param {{ choice?: Record<string, string>, lang?: string }} [o] the question chosen per step, the spoken language
*/
export async function captureSteps(client, pair, next, show, { choice = {}, lang = 'en' } = {}) {
const session = await client.captureProtocol({ ...pair, source: { type: 'webcam' } })
const steps = session.plan.steps
for (const step of steps) {
const prompt = stepPrompt(steps, step.order, choice, lang)
await session.step(step.order).start({ mediaPid: prompt?.mediaPid ?? null }) // settles at once when live
await new Promise((/** @type {(v?: unknown) => void} */ done) => {
const tick = setInterval(() => {
const t = session.stepTiming()
next.disabled = t?.minReached === false // enabled once the minimum is reached
show(prompt, t?.remainingMs ?? null) // "answer now" only when prompt.kind is 'answer'
if (t?.maxReached) finish()
}, 250)
const finish = () => { clearInterval(tick); next.onclick = null; done() }
next.onclick = finish
})
await session.step(step.order).end()
}
return session.stop() // CaptureOutcome
}

Recorded, the steps are marked in the video, and the manifest says where each one starts and ends:

JavaScript
// The same loop, recorded: the steps are marked in the video, and the manifest says where each one starts and ends.
import { createRecordingClient, uploadRecording } from '@neurologyca/capture/record'
import { stepPrompt } from './step-prompt.js'
/**
* @param {{ userProtocolId: string, protocolMechanicId: string }} pair from your backend
* @param {HTMLButtonElement} next your "Next" button
* @param {(p: import('./step-prompt.js').StepPrompt | null) => void} show your screen
*/
export async function recordSteps(pair, next, show) {
const recorder = createRecordingClient({ baseUrl: '/engine', taxonomy: { contextId: 'measurement' }, stepDurations: 'warn' })
const session = await recorder.captureProtocol({ ...pair, source: { type: 'webcam' }, triggerAnalysis: false })
const steps = session.plan.steps
for (const step of steps) {
const prompt = stepPrompt(steps, step.order, {}, 'en')
show(prompt)
await session.step(step.order).start({ mediaPid: prompt?.mediaPid ?? null }) // can reject when recording
await new Promise((done) => { next.onclick = () => { next.onclick = null; done(undefined) } })
await session.step(step.order).end()
}
const recording = await session.stop() // the video and its manifest, kept in this browser until uploaded
console.log(recording.manifest.steps.map((s) => [s.order, s.startMediaMs, s.endMediaMs, s.endedBy]))
return uploadRecording(recording, { url: '/uploads' }) // your receiver
}