Integration paths
How you integrate (SDK and HTTP API) and where it is processed (on-device or deferred), with the measured numbers and a short guide to choose.
We recommend capturing with the SDK, on-device (real time), for most integrations, and using the HTTP API for one-off calls and your backend: reading results later, polling or searching the catalogue. Capture deferred (server-side) when the device cannot keep up with on-device capture (modest phones, long sessions) and you can upload video: today you can record, upload and process with Neurologyca’s processing service, with nothing to deploy. If you capture outside the browser (another language or platform, or you already have the frames), the HTTP API also captures.
The SDK for capturing; the API for one-off calls and your backend. Most integrations combine both.
The default path
Section titled “The default path”If you are not sure: the SDK on-device (real time), with Measurement as your first protocol (in the test environment: the production Engine does not offer it yet (Your environments)); move to deferred (server-side) if your tests on your phones call for it. Start with the Quickstart and Measurement.
Without an account: the recorded examples, the code and the docs; measuring needs an invitation (request access).
Two questions
Section titled “Two questions”You decide two things, and they do not depend on each other: how you integrate (the SDK, the HTTP API, or both) and where it is processed (on the device or on a server).
On-device (real time)
- Your page · SDK, live client
- Your backend · proxy (adds the token)
- Engine
Deferred (server-side)
- Your page · SDK, recording client
- Your backend · upload receiver
- Processing service Preview
- Engine
HTTP API
- Any language: the HTTP API, straight from your backend
- Engine
How you integrate
Section titled “How you integrate”| Topic | SDK · Recommended for capturing | HTTP API · Complementary | SDK + API · The usual |
|---|---|---|---|
| What for | For capturing: protocol, steps, batches, baseline and the session's results. | For one-off calls and your backend: results later, polling, catalogue, participants, contexts, any language. | Capture in the browser with the SDK, then read from your backend with the API: results later, polling, the catalogue. See Results in your backend. |
| What it does for you | Routes, the token (your proxy adds it), batching, retries, conflicts and the position reservation. | Nothing: you build the calls, the batches, the retries and the event stream. | The SDK runs the capture; your backend makes plain reads. |
| Your backend | @neurologyca/capture-server: the proxy (createEngineProxy) and createEngineAdmin, which is the same API from Node. | Any language: POST /token with your key, then the calls of the HTTP guide. | createEngineAdmin creates the person and the enrolment; later, getResults or GET /results. |
| Time to the first result | Seconds after the last step: the CaptureOutcome arrives shortly after the session ends, so a short protocol gives its first result in under a minute; creating the person and the enrolment before Start takes a moment more. | Seconds after the last batch: the cURL quickstart goes from POST /token to the results in well under a minute. | The same as the SDK; your backend reads whenever it needs to. |
| Effort | One backend file (proxy, person and enrolment) and one page: what the quickstart builds. | The HTTP guide by hand: the calls, the batches, the retries and the event stream. | The SDK in the page, plus a few reads in your backend. |
SDK and API work in both directions:
- Backend first:
createEngineAdmincreates the person and the enrolment, and the page captures with that pair of ids. - Browser first: the page captures with the SDK, and your backend reads the results later (
GET /results, orcreateEngineAdmin().getResults, the same API from Node). See Results in your backend.
Where it is processed
Section titled “Where it is processed”| Topic | On-device (real time) · Preview 0.45 | Deferred (server-side) · Preview | Self-hosted (coming soon) |
|---|---|---|---|
| Today | Available. | You can record, upload and process with Neurologyca's processing service: nothing to deploy. | Coming later. |
| Where it runs | In the browser, on the person's device, while they capture. | On our processing service, shortly after the upload. It is not hours later: it just does not run on the device. | On your own servers. |
| When the results arrive | During the session and seconds after the last batch in a short capture. A long capture, with thousands of frames, takes minutes. Past outcomeDeadlineMs (240 s by default) stop() gives E_RESULTS_TIMEOUT, which is not a loss: read the results later. | Minutes after the upload, adding three stretches: the upload (about 7 min for 3 min of video at 1.75 Mbps, a calculation), the queue (one job at a time) and the processing, somewhat longer than the video itself. | Coming later. |
| What leaves the device | Landmarks, expression coefficients and head pose, to Engine. Only if you turn them on: the transcript (speech recognition runs in the browser, so the audio stays and the text goes; Engine passes it to an LLM for the interpretation) and the extra signals (voice as numbers, pulse, hands, body). Never image, video or audio. | Video and audio (with the camera, the microphone is always on) to your receiver, and from there to the processing service, which, with transcription on, transcribes the whole audio of the recording and passes to Engine only the text of the steps the job asks for (transcribeSteps). The raw audio identifies a person more than numbers do, and the manifest that travels with the video holds the full userAgent of the browser. | Video and audio, to your own servers. |
| What limits you | The device: the face detection on the GPU (WebGL 2) and, on phones, heat in long sessions, which lower the frame rate. A recent phone or laptop usually stays close to the camera's rate; an entry-level phone captures fewer frames per second, and a phone loses rate as it heats up. | Upload bandwidth. At 4 Mbps, 3 min of video are about 93 MB (about 310 MB per 10 min). The upload resumes after a cut. The quality depends on the recording's bitrate and the light, not on the device's power: the processing runs on the server. | Coming later. |
| Browsers and phones | Chrome and Edge 94+, Safari 16.4+ (iOS 16.4+), on desktop and capable phones. | Records MP4/H.264: desktop Chrome and Edge, Chrome on Android and Safari on iOS. An iPhone records MP4 too: its file can end about 1.1 s before the stop, which is why the SDK keeps recording 2 s more. On a phone or a tablet it records in portrait: it waits until the device is upright. | Coming later. |
| Measurement | Yes. | Yes, since 0.27.0: triggerAnalysis: false travels in the manifest and the processor does not call /analyze. | Coming later. |
| Typical cases | A short check-in on a laptop; a two-minute task on a recent phone; any capture whose video must not leave the device. | A ten-minute interview on a modest phone; a field session on a weak device, uploaded later. | Video that must be processed inside your own infrastructure. |
| Moving between them | createProtocolClient (from @neurologyca/capture) ↔ createRecordingClient (from @neurologyca/capture/record, without kodec). Some options do not carry over: transcribe is on-device only; in deferred, the processing service transcribes. | Add the receiver to your backend (createUploadReceiver); record from the camera, a <video> or a stream. | Coming later. |
Firefox is not a target: kodec has not been tested there (on-device), and it does not record MP4/H.264 (deferred).
Which one do I choose?
Section titled “Which one do I choose?”| If… | Choose |
|---|---|
| your people use a desktop or recent phones and you want results during the session (for short captures; long ones take minutes to close) | On-device (real time) |
| the video must not leave the device | On-device (real time) |
| you use modest phones, sessions of several minutes, or the rate of your sessions stays low | Deferred (server-side): today you can record, upload and process with Neurologyca's processing service, with nothing to deploy. |
| you do not know which devices your people will use, or their phones vary a lot, and you want the most stable quality | Deferred (server-side) (Preview): the video leaves the device and is uploaded and processed on the server, so you wait for the result after the upload. Your organization has to have it turned on. |
| you want real time by default and a way out when the device cannot keep up | On-device (real time) by default, and Deferred (server-side) (Preview), with the person's permission, when the device check says it cannot measure or cannot transcribe. Your organization has to have it turned on. |
| the video must be processed off the device but inside your infrastructure | Self-hosted (coming soon); until then, deferred with our service. |
| you capture in the browser and then query from your backend (results, polling, catalogue) | SDK + API (the usual) |
| you capture from another language or platform, or you already have the frames | HTTP API |
| you only want to measure, without interpretation | Measurement, on-device (real time) (deferred also works since 0.27.0) |
| you are not sure | The default path: SDK on-device (real time), starting with Measurement |
To decide by your people’s devices, and to check a device before measuring: Devices, quality and mode.
What you decide and what the SDK does
Section titled “What you decide and what the SDK does”| Topic | You | The SDK and capture-server |
|---|---|---|
Your screen: what everything looks like, its texts and language; consent before captureProtocol, and only from a gesture of the person |
✔ | Open the camera and microphone in that call |
| Per protocol: how long a step lasts when Engine does not say, who closes it (a cap, a button, the end of the video) and what a step without text says | ✔, with the data of describeStep |
Check each step against Engine’s minimum and maximum (stepDurations) |
Per protocol: whether the person can redo a capture, and what happens with a 409 |
✔, with ConflictDetail |
Turn every 409 into one shape |
| Your backend: signing in your users before the proxy and the receiver, and which ids each one can touch | ✔ (authorize with ownsAll, checkManifest) |
Refuse to build without them |
Your backend: where recordings are stored, how long confirmed ones are kept, and calling sweep() |
✔ | storage is required: localDiskStorage or your own; define what is abandoned |
| What the SDK does: Engine routes, headers, correlation; minting, caching and adding the token | — | ✔ |
| What the SDK does: batching, retries, resuming an upload, the position reservation and its release | — | ✔ |