Operations

Health checks

Logs and SIEM forwarding

All services log to stdout: docker compose logs -f <service> or docker compose logs --since 1h --tail 200 api-pacs. Forward to a SIEM with one of three patterns in docker-compose.override.yml: the Docker syslog driver for low volume, a Filebeat sidecar when the customer already runs ELK, or Vector for everything else.

# A) Docker syslog driver (per service)
services:
  api-pacs:
    logging:
      driver: syslog
      options:
        syslog-address: "udp://siem.customer.local:514"
        tag: "pacs-ai/{{.Name}}"

# B) Filebeat sidecar (filebeat/filebeat.yml: filebeat.autodiscover docker provider with hints, output.logstash hosts)
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:8.13.4
    user: root
    volumes:
      - /var/lib/docker/containers:/var/lib/docker/containers:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro
    command: ["-strict.perms=false"]
    networks: [pacs-net]

# C) Vector (vector/vector.toml: sources.docker type docker_logs, sinks.siem type socket tcp json to siem.customer.local:514)
services:
  vector:
    image: timberio/vector:0.39.0-debian
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./vector/vector.toml:/etc/vector/vector.toml:ro
    networks: [pacs-net]

Metrics and alerts

• Prometheus instrumentation is wired but not scraped by default: study-service GET /metrics (bearer operator token; HTTP, pipeline transition and duration, polling-mode and model-execution series) and api-pacs GET /debug/vars (Go expvar JSON at the server root, behind the OpenAPI docs basic-auth gate; dispatch attempts and latency, callback status and event-delivery lag, run phase, outcome and attention counts, skip and attention codes, reconciliation counters, SSE connections and publish or write failures). No Prometheus or Grafana ships; point your platform at those endpoints. Metric labels are enums only and never carry run IDs, tenant IDs, study UIDs, model error text, DICOM metadata or results.
• Ingestion metrics named in the plan: ingestion_candidates_discovered_total, ingestion_candidate_transitions_total{from,to}, ingestion_retrieval_attempts_total{outcome}, ingestion_retrieval_latency_seconds, ingestion_processing_callbacks_total{status,outcome}, ingestion_processing_rollups_total{status}, ingestion_cleanup_jobs_total{outcome}.
• Recommended alerts: candidate stuck in RETRIEVAL_QUEUED beyond N minutes, Celery queue depth above N, Orthanc disk above 80%, any service unhealthy for 5 minutes, failed C-MOVE rate above N% over 15 minutes, sustained callback lag or callback errors, reconciliation failures, notification publish failures, subscriber drops, and unexpected polling fallback.

Upgrade and rollback

# pre-upgrade: read release notes, verify backups, drain the Celery queue, snapshot both Postgres volumes
docker compose stop api-pacs
until [ "$(docker compose exec -T redis redis-cli -n 2 LLEN celery)" = "0" ]; do sleep 5; done
docker compose exec study-service celery -A workers.celery_app inspect active

# upgrade
git fetch && git checkout <new-tag>
docker compose pull
docker compose up -d
docker compose exec api-pacs make -C /app migrate-up      # or: cd api-pacs && make migrate-up
docker compose exec study-service alembic upgrade head

# rollback: previous image tags + docker compose up -d
STEPS=1 make migrate-down                                  # api-pacs, when a down migration exists
docker compose exec study-service alembic downgrade -1     # cardio-agent
# if an irreversible migration was applied, restore from the pre-upgrade snapshot

Real-time worklist rollout runbook

The rollout of processing runs on an existing install follows api-pacs/docs/realtime-worklist-rollout-runbook.md. Take consistent backups of both PostgreSQL databases and prove them by restoring into isolated instances (a backup that has not been restored successfully is not a rollout gate). Deploy study-service first, then Go with INFERENCE_REQUIRE_PROCESSING_RUN_ID=false, confirm health, callback auth, operator job lookup, Redis, reconciliation, the REST snapshot and the SSE heartbeat, switch study-service atomically to ENABLE_ORTHANC_POLLING=false and ENABLE_GO_CALLBACKS=true, then run the backfill. Keep the nullable columns and additive migrations; never run down migrations while either version may still be active.

go run ./cmd/legacy-processing-run-backfill --dry-run      # plus docs/legacy-processing-run-backfill-audit.sql
go run ./cmd/legacy-processing-run-backfill --apply --confirm=LEGACY_IMPORT \
  --expected-studies=<fresh-eligible-studies> --expected-executions=<fresh-eligible-executions>
go run ./cmd/legacy-processing-run-backfill --verify \
  --expected-studies=<approved-studies> --expected-executions=<approved-executions>   # must report "passed": true
go run ./cmd/legacy-processing-run-backfill --rollback --confirm=ROLLBACK_LEGACY_IMPORT

• Stop the preflight if any study is skipped, if the SQL and Go eligible counts differ, if tenant mismatches, duplicate plans or existing-run conflicts are non-zero, if ingestion or callbacks still create run-less work, or if the fresh counts differ from the approved change record. The 2026-08-06 baseline of 669 studies and 1,179 executions is historical evidence, not a constant.
• Apply exactly once, with ingestion dispatch paused and fresh counts; if the command stops, run a new dry run and obtain approval for the new remaining counts. Each completed study is atomic.
• Verification checks a two-model terminal aggregate, a completed-plus-failed mix producing PARTIAL_SUCCESS, a no-usable-DICOM structured skip, a manual run N+1, reconciliation repairing a withheld callback, and tenant B being blocked from tenant A run detail, history, snapshot and SSE.
• Rollback trap: the execution foreign key is ON DELETE CASCADE, so deleting a LEGACY_IMPORT run removes its executions; and a pre-contract cardio-agent image whose Alembic graph ends at 0009 cannot start against revision 20260814_0012. Only then set INFERENCE_REQUIRE_PROCESSING_RUN_ID=true; the nullable compatibility paths are removed once no run-less work remains.

Runbooks

-- candidates queued for retrieval too long
SELECT id, study_instance_uid, status, processing_status, updated_at, last_retrieval_error
FROM ingestion_candidates
WHERE status = 'RETRIEVAL_QUEUED' AND updated_at < now() - interval '15 min'
ORDER BY updated_at;

-- breakdown of stuck states
SELECT status, count(*), min(updated_at), max(updated_at)
FROM ingestion_candidates
WHERE updated_at < now() - interval '30 min'
GROUP BY status;

-- requeue one dead retrieval (only after confirming the C-MOVE is not in flight)
UPDATE ingestion_candidates
SET status = 'STABLE', updated_at = now(), last_retrieval_error = NULL
WHERE id = '<candidate-id>' AND status = 'RETRIEVAL_QUEUED';

Data retention

# sample Elasticsearch ILM policy: hot rollover at 30d / 50gb, delete after <retention-days>
curl -X PUT "http://localhost:9200/_ilm/policy/pacs-ai-logs" -H 'Content-Type: application/json' -d '{
  "policy": { "phases": {
    "hot":    { "actions": { "rollover": { "max_age": "30d", "max_primary_shard_size": "50gb" } } },
    "delete": { "min_age": "<retention-days>d", "actions": { "delete": {} } } } } }'
curl -X PUT "http://localhost:9200/_index_template/pacs-ai-logs-template" -H 'Content-Type: application/json' -d '{
  "index_patterns": ["pacs-ai-*", "logs-*"],
  "template": { "settings": { "index.lifecycle.name": "pacs-ai-logs", "index.lifecycle.rollover_alias": "pacs-ai-logs" } } }'
curl http://localhost:9200/_cat/indices          # adjust index_patterns to what api-pacs actually writes
curl http://localhost:9200/_ilm/policy/pacs-ai-logs

Backups and recovery objectives

• What is lost on host failure today: all pipeline_results and all ingestion job and candidate history. They can be regenerated by re-running ingestion against the same source-PACS window, but only if the studies are still present at the source.
• Recommended additions, in priority order: a pg_dump cron with an offsite copy for PostgreSQL 5434 (the only data that is costly to regenerate), then PostgreSQL 5433 (audit history), Firestore through Google Cloud project export; skip Orthanc (refills within the cache window) and Elasticsearch (rebuildable logs).
• Secrets (.env, the Firebase service-account JSON, TLS certificates) live in the customer secrets vault and are backed up there.