Skip to content
DevLab

CSP and transcription

The external origins Capture loads, a Content Security Policy that works with them, how to host them yourself, and how to set up transcription.

On this page
  1. What Capture loads, and from where
  2. Content Security Policy
  3. Hosting the files yourself
  4. Transcription
  5. Language
  6. What leaves the device
  7. When the device cannot transcribe
  8. Bundlers

Your page loads the analysis models from public CDNs; if your site has a Content-Security-Policy, it must allow them, or you host the files yourself.

Capture analyses the camera (and, with transcription, the microphone) in the page. By default it loads its models and runtimes from public CDNs:

When Origin What Host it yourself with
Always cdn.jsdelivr.net The vision runtime (MediaPipe Tasks Vision, a pinned version) and its WebAssembly kodec.visionUrl, kodec.wasmUrl
Always storage.googleapis.com The face model kodec.faceModel
With hands or body models storage.googleapis.com The hand and pose models kodec.handModel, kodec.poseModel
With transcription cdn.jsdelivr.net The speech runtime (Transformers.js) transcribe.selfHost.transformersUrl
With transcription huggingface.co and *.hf.co The speech model (Whisper), about 136 MB transcribe.selfHost.modelHost
With transcription the speech runtime’s own path The ONNX Runtime WebAssembly transcribe.selfHost.wasmPaths

Those URLs load code: set them in your build, never from a query string or a remote configuration.

This policy lets your page capture with the SDK, transcribe and talk to your proxy, with everything loaded from the CDNs above. We tested it with the live client, transcription on, through the SDK’s proxy (Your backend).

HTTP
Content-Security-Policy: default-src 'self'; script-src 'self' 'wasm-unsafe-eval' blob: https://cdn.jsdelivr.net; worker-src 'self' blob:; connect-src 'self' https://cdn.jsdelivr.net https://storage.googleapis.com https://huggingface.co https://*.hf.co; style-src 'self'; img-src 'self' data: blob:; media-src 'self' blob:
  • 'wasm-unsafe-eval' is enough: 'unsafe-eval' is not needed.
  • style-src 'self' is enough: Capture does not need inline styles ('unsafe-inline').
  • blob: in script-src is required: kodec’s audio processing (its AudioWorklet) loads from a Blob. Without it, the whole transcription is lost, and it shows: codec.error with audio-start-failed, the stt warning, and the report’s transcription is null.
  • connect-src 'self' covers your proxy when it lives in your origin, and the startup report URL (startup.report.url) when it is yours; on another origin, add it.
  • If you host everything yourself, the CDN origins can go: script-src 'self' 'wasm-unsafe-eval' blob: and connect-src 'self'.

When the policy blocks Capture, the camera never gets ready (a download is blocked), or the transcription is lost as above. workerBlockedByCsp in the report only says whether the speech worker was blocked: workerBlockedByCsp does not cover the AudioWorklet.

Pass your own URLs to the client:

JavaScript
import { createProtocolClient } from '@neurologyca/capture'
const client = createProtocolClient({
baseUrl: '/engine',
tenant,
kodec: { visionUrl: '/vendor/tasks-vision/', wasmUrl: '/vendor/tasks-vision/wasm', faceModel: '/vendor/face_landmarker.task' },
transcribe: { language: 'es', selfHost: { modelHost: '/vendor/models/', transformersUrl: '/vendor/transformers.js', wasmPaths: '/vendor/ort/' } },
})

wasmPaths since 0.42.0 (breaking). As a string it is a prefix that must end in /: kodec puts it in front of the file name and asks for the ONNX Runtime pair of the executor it runs on: ort-wasm-simd-threaded.jspi.mjs and ort-wasm-simd-threaded.jspi.wasm on WebGPU with JSPI; ort-wasm-simd-threaded.mjs and ort-wasm-simd-threaded.wasm on WASM and on WebKit (Safari, and every browser on iOS). On WebGPU without JSPI the prefix goes to ONNX Runtime as before. The four files come from onnxruntime-web/dist/: serve both pairs if your people arrive with both executors. Before loading, kodec checks both files of the pair with a HEAD; if one answers 404 or 410, speech does not start and the capture goes on without it, with the stt warning carrying subStage: 'stt-runtime', reason: 'network' and, in cut, code: 'stt-runtime-missing' with the missing url (in prepare(), partial with speech-error). A host that serves only the Asyncify pair (ort-wasm-simd-threaded.asyncify.*), which used to work on WebGPU with JSPI, now fails that way. As an object, wasmPaths: { wasm, mjs } points at the exact files and reaches kodec as it is; on WebKit, point it at the plain build.

Transcription turns what the person says into text on the device. It is off until something turns it on: transcribe: { … } in the client, or a transcribe object in the kit (tenant), which turns transcription on even if your code does not pass transcribe. Only transcribe: false in the client keeps it off whatever the kit says; transcribe: false in the kit keeps it off unless the client passes an object. With the camera it opens the microphone; with a <video> source it transcribes the file’s own audio and does not open the microphone. In Safari, with the <video> muted, the SDK adds no audio track and transcription fails with an explicit error.

The first time, the browser downloads about 160 MB for speech (136 MB of them the Whisper weights, plus the runtime and the library) and keeps it for that origin; on a slow mobile connection that is minutes. client.prepare({ models: { speech: true } }) can download it before Start, except on Safari and iOS (Prepare before Start).

The language is declared, not detected: it is the language the person will speak, not your page’s. Set it as a code without a region:

JSON
{ "transcribe": { "language": "es" } }
  • If you pass a code with a region, such as es-ES, Capture uses its language part (es) and notes it in the console.
  • Without language, Capture uses the language of the page (<html lang>, then the browser’s).
  • language: null detects the language from the audio. It is not reliable with short answers: it can return the text translated into English, with no error. Capture warns stt.autodetect.
  • A code the model does not know falls back to detection, with a warning.
  • The language can also come from the kit (transcribe.language); the client’s fields win over the kit’s.
  • With a source that is not a microphone (a <video> captured with captureStream(), like DevLab’s sample video), the track does not report autoGainControl: kodec warns once that energy and shimmer are untrusted (the message names WebKit/iOS, but any such track triggers it). It is expected; those two features are not comparable with a microphone’s.

In the client:

JavaScript
const client = createProtocolClient({ baseUrl: '/engine', tenant, transcribe: { language: 'es' } })

kodec forces the model to the declared language. If the person speaks another one, one of two things happens, depending on the pair of languages: the model translates into the declared language, fluently and with good confidence, so the text reaches Engine translated and with no warning; or it repeats itself or scores low, and kodec discards those utterances on purpose, so the voice steps reach Engine with no text or very little (only the stt.silent warning, when no utterance is left). In one test, with English declared over Spanish speech the first happened; with Spanish declared over English speech, the second. Pinning the wrong language is worse than not pinning it. Deferred, the job’s language works the same way: the transcript arrives labelled with the language asked for, and its text is not reliable (Deferred).

Ways to get it right:

  • Keep it with the person or the interview in your backend (the language the candidate answers in, for instance) and pass it to every capture. Do not infer it from the interface language if the two can differ.
  • If you do not know it, ask before the capture, on the consent screen.
  • One language per capture. Whoever switches language mid-capture is transcribed in the declared one: split it into captures.
  • Catch it afterwards with onReport: in SessionReport.transcription, vad.utterances above 0 with few or no emitted and high discardedLowConfidence or discardedRepetitionLoop points to the wrong language (with none emitted, Capture warns stt.silent). Offer to repeat the capture in the right one. A translation passes every counter: only declaring the right language prevents it.
  • Autodetection is not the way out. In one test with language: null, the longer answers in Spanish also came back translated into English. When you know the person’s language, declare it.
  • The interpretation’s language is separate: narrative.language on the person.

Only text. The audio is processed in the browser and never sent. The transcript travels once, in the Final of the last step: the whole text (speech) and the text of each step with its times (sttByStep). Engine adds it to the context of the interpretation.

Some devices cannot run the speech model. Capture tells you before it starts, with a warning whose code is stt and a blocker, or during the capture, with stt.cut. The blockers are in STT_BLOCKERS, and sttBlockerOf(error) reads one from an error:

Blocker Meaning
no-webgpu No usable WebGPU adapter in this browser (it may have WebGPU and still offer none)
ladder-exhausted Every speech configuration closed the tab while loading on this device, so it is not tried again
video-cost-veto Transcribing made the camera analysis too slow on this device
no-memory-to-learn The browser stores nothing (private mode or blocked storage) and would run speech without WebGPU, so it is not risked
dtype-required The model you chose needs an explicit dtype on this device

The veto is kept in that browser’s storage and outlives the page. When it was measured while the video ran below 8 fps, typically with the tab hidden, it is stored as provisional. Since SDK 0.41.0 the first capture that shows video above 8 fps deletes it and transcribes: only that capture waits, up to about 3 s (about 1 s on a healthy device), and SessionReport.transcriptionSetup.provisionalVetoLifted reports the deleted veto, its only trace. A capture that stays at 8 fps or less, such as a hidden tab or a capture fps of 8 or less, keeps it. Up to 0.40.0 it never lapsed inside a capture (transcription started before the rate was measured), and 0.40.1 transcribed but kept it, waiting in every capture. Show every warning, not only the last one, because stt is the only sign that a capture is not transcribing.

video-cost-veto is the only one the SDK lets you lift, with forgetTranscriptionVeto behind a press of the person, never automatically. transcribe.allowWasmWorker: true lets devices without a usable WebGPU adapter try the speech model in WebAssembly, more slowly. Since 0.43.0 that path needs its transcription Worker: if the Worker does not start (for instance a Content-Security-Policy that blocks workers), there is no voice (a codec.error with wasm-needs-worker, and transcriptionLoad.reason: 'needs-worker' in the report) instead of falling back to the main thread, which froze the video on every utterance. With WebGPU nothing changes. The capture of the indicators goes on either way.

Pass both functions the kodec the SDK itself exports (kodec, next to readTranscriptionVeto and forgetTranscriptionVeto, in @neurologyca/capture): you do not install kodec, the SDK brings it pinned. readTranscriptionVeto(kodec) only reads, and returns { veto, learned }: call it first to know whether there is a veto to lift. forgetTranscriptionVeto(kodec) lifts that veto and nothing else; it never erases the rest of what kodec learned on the device, such as the memory limits (do not use kodec’s forgetMemoryLimit for this). The veto belongs to the browser, not to the person: lifting it only changes the next capture in this browser, and if transcribing slows the video down again, the SDK vetoes it again.

onReport in the client receives the SessionReport when the capture ends, also when it ends with endedBy: 'abort' or 'error'; CaptureOutcome does not carry it. For transcription, read transcriptionSetup (model, language, languageSource, backend and provisionalVetoLifted) and, per step, transcription.stepsTranscription[] (order, transcribed, emitted, spokenMs): a step with transcribed: false or emitted: 0 heard no speech, which explains an empty transcript in /results. DevLab’s playground keeps only those fields, with provisionalVetoLifted as a yes or no (never its content), and shows them in «Technical data of the response». When a session is not transcribed, the playground says so under the stage and in the result card with the sentence of its blocker from the table above, and, only with video-cost-veto and once the session has ended, offers «Retry transcription», which does exactly this: reads the veto and, if there is one, lifts it.

  • The SDK brings kodec 5.0.0-rc.14, pinned: you do not install kodec yourself. Minification does not break transcription (fixed in kodec 4.0.2). With webpack (Next.js), kodec 4.2.2 or later, which the SDK brings since 0.34.0; Turbopack is not tested.
  • Do not install the speech runtime or the vision runtime “just in case”: Capture loads them when needed. Passing the speech runtime you import as transcribe.transformers only saves its download: the model still comes from modelHost and ONNX Runtime from wasmPaths, and without transformersUrl there is no speech worker (speech runs on the main thread).