Automated inference pipeline

Ingestion jobs

An ingestion job tells the control plane which studies to pull and which model to run on them. Jobs are created by owners and admins in the viewer or through POST /v1/inference/ingestion/job/create with dicomModality, containerId, modelId, modelName, modelVersion, modalities (the Orthanc peers to query), intervalInMinutes, scheduleStartTimestamp and scheduleEndTimestamp. Related endpoints: GET /jobs, PUT /job/{id}/update, POST /job/{id}/start, POST /job/{id}/stop, DELETE /job/{id}/remove and POST /jobs/import (CSV) under /v1/inference/ingestion. Every transition is logged with tenant_id, ingestion_job_id, candidate_id, study_instance_uid and request_id; retrieval logs add the Orthanc job IDs and callback logs the study-service job IDs.

Candidate lifecycle

The candidate status column itself holds only the discovery and retrieval values (DISCOVERED, GROWING, STABLE, RETRIEVAL_QUEUED, RETRIEVED, DISAPPEARED, FAILED). Processing is a sibling rollup on the same row, processing_status with processing_status_at (queued, running, completed, partial, failed), so RETRIEVING, the PROCESSING_* states and PARTIAL above are the plan-level view of those two columns together, and the cleanup states are not built yet.

• The retrieval worker waits for all C-MOVE jobs to finish before dispatch, and studies already in Orthanc are dispatched without a new C-MOVE. The ingestion plan records retrieval attempts in their own table (queued, running, succeeded, failed, timed_out, cancelled; triggered by worker, manual_retry or reconciliation) with the candidate keeping orthanc_job_ids, last_retrieval_state, last_retrieval_error and last_retrieval_checked_at; a manual retry creates a new attempt rather than overwriting prior error context.
• Before dispatch, api-pacs inspects the registered inference container, starts it if stopped, and polls its model-info endpoint every 500 ms for up to 30 seconds; a container that never becomes ready records a failed execution with a dispatch attention flag.
• Each dispatch creates an idempotent processing run with one execution per expected model, and only one non-terminal run per study can exist at a time (a partial unique index; a second request returns HTTP 409). Run N+1 preserves prior history, each run freezes its expected-model plan, and executions carry optimistic version numbers that consumers use to discard older events. study-service callbacks are validated and applied atomically and in order; out-of-order, replayed or regressive state is ignored and logged.
• The reconciliation worker runs once at startup and then every 5 minutes, selects a bounded batch of active runs that are stale (pending after 2 minutes, queued after 10, running after 65 by default) or already need attention, looks the execution up in study-service by stored study_service_job_id, then by processing_run_id, then by candidate, and passes recovered state through the same transition handler as live callbacks. Resolved warnings are cleared, manual ones persist, and three consecutive failures raise RECONCILIATION_FAILED. Per-cycle counts (checked, repaired, failed, unresolved) are exported.

Worklist vocabulary

Live worklist

The worklist separates three axes: lifecycle (phase), outcome, and attention flags. A model that did not run carries an explicit skip code or failure reason rather than a silent gap. Tenant users can view the worklist; reprocessing stays admin-only. Manual dispatches carry a dispatch_intent and a processing_execution_id: a replay returns the same job with already_present=true, a reused execution ID with different correlation returns 409, a foreign already_present response records DISPATCH_FAILED and refunds the quota unit, and an accepted dispatch with wrong correlation records STATE_CONFLICT and keeps the reservation charged.

Result output modes

AI-generated DICOM series can be written back to Orthanc with POST /v1/orthanc/modality/{modalityID}/study/{studyInstanceUID}/series/store (multipart: file, seriesInstanceUIDs as a JSON-encoded array, patientID, patientName, modelName, modelVersion; the file part must carry Content-Type: application/pdf or application/dicom; audited as stored_custom_series). Readers rate any result with a thumbs up or down (feedbackType APPROVE or REJECT); each opens the model fixed questionnaire, approveFeedbackQuestionnaires or rejectFeedbackQuestionnaires (GET /v1/inference/model/{modelId}/feedback, PUT /v1/inference/model/feedback/update, DELETE /v1/inference/model/{modelId}/feedback/remove).

Planned retention cleanup (ingestion plan phase 4, not yet shipped)

The design records cleanup jobs (queued, running, completed, failed, skipped, held) driven by tenant configuration, per-job overrides, study age, processing terminal state, manual holds and legal holds, with three safety rules: never delete while processing, never delete on manual hold, and treat deletion of already-missing Orthanc resources as idempotent. Until it ships, candidate rows persist and Orthanc cache expiry is the only DICOM deletion.