DatabasesEverywhere live monitoring

Use short-lived, scoped WebSocket JWTs for browser dashboards. Combine them with authenticated health, system, metrics, and resource endpoints for node operations and scheduling.

Mint a WebSocket token

The panel uses its node credential to mint a user-specific token:

POST /api/ws-token
{
  "subject": "user-42",
  "scopes": ["monitor:read", "logs:read"],
  "instances": ["cust-42-db"],
  "ttl_seconds": 900
}

This endpoint requires ws-tokens:write. The response contains a Bearer JWT and expires_at_unix. TTL defaults to 900 seconds and cannot exceed 3600 seconds.

instances is an allow-list. An empty list grants no instance access. Node-wide access must explicitly set "all_instances": true, which cannot be combined with an allow-list.

Each token ID is accepted for one WebSocket upgrade only. Mint a fresh token for every reconnect. Frames and messages are capped at 16 KiB with bounded write buffering.

Connect from a browser

Browsers cannot set Authorization during a WebSocket upgrade, so send the JWT with the dbe.jwt subprotocol:

const socket = new WebSocket(
  "wss://node.example.com/ws/instances/cust-42-db/logs",
  ["dbe.jwt", token]
);

Server-side clients may use the same subprotocol or Authorization: Bearer <jwt>.

Monitoring stream

/ws/monitoring requires monitor:read and sends a full snapshot every 500 ms:

{
  "type": "stats",
  "instances": [
    {
      "instance_id": "cust-42-db",
      "protocol": "postgres",
      "status": "running",
      "runtime": "docker",
      "cpu_cores": 1.0,
      "cpu_usage_percent": 12.5,
      "memory_mib": 2048,
      "memory_usage_bytes": 104857600,
      "disk_mib": 10240,
      "disk_used_bytes": 52428800,
      "disk_enforced": true,
      "network_rx_bytes": 1234,
      "network_tx_bytes": 5678,
      "resource_error": null
    }
  ],
  "install_progress": []
}

Monitoring samples are shared, then filtered against each JWT before serialization. Disk usage uses quota accounting when available; bounded background directory sampling is only a fallback.

Creation and image operations appear in install_progress. Actions include create, image_update, and major_upgrade. Common stages include queued, prepare, pull_image, create_container, start, healthcheck, backend, completed, and failed.

Major upgrades also emit export, snapshot, prepare_replacement, import, and validate. The retained healthcheck name represents a bounded startup-readiness check rather than a permanent database probe.

Instance logs

/ws/instances/{instance_id}/logs requires logs:read and token access to the instance. It sends a snapshot every three seconds:

{
  "type": "logs",
  "instance_id": "cust-42-db",
  "sequence": 7,
  "stdout": "database output",
  "stderr": "",
  "error": null
}

Connection URLs are redacted before logs leave the daemon. When log retrieval fails, output fields are null and error contains a short reason.

Import and export events

Connect to /ws/instances/{instance_id}/import-export?job_id=... with import-export:read. The optional job ID narrows the stream.

The daemon sends an initial import_export_snapshot, then import_export_job updates. If the consumer falls behind it sends import_export_lagged followed by a fresh snapshot.

When an export succeeds, its event includes a single-use temporary download valid for about 120 seconds. Show the download action immediately rather than persisting the URL.

System endpoints

• GET /api/system returns node identity, daemon and API versions, readiness, rate-limit contract, runtime, disk mode, network isolation mode, import support, enabled protocols, and gateway readiness.
• GET /api/heartbeat returns {"status":"ok"} as a cheap authenticated management API liveness check.
• PATCH /api/system/config merges an allowed configuration object and returns restart_required: true.
• GET /metrics returns Prometheus text for instance counts, job counts, and disk enforcement state.
• GET /api/admin/resources/summary returns allocation and host-pressure data for placement decisions.

Management API readiness and database gateway readiness are separate. Heartbeat stays healthy once the management API accepts authenticated requests even if an instance or gateway is unavailable.

Existing containers auto-start in a bounded background phase after critical metadata, recovery, runtime, socket, and disk checks. A slow database does not keep the management API offline.

Quarantine and fail-closed recovery

Legacy bridge-network or docker_tcp instances are stopped and marked quarantined; they are never converted in place. Export or preserve required data offline, delete the legacy instance explicitly, recreate it, and import the artifact.

Duplicate route identities retain the deterministic first claimant and quarantine the others before gateways open.

If the daemon exits while an import or export job is durably running, the affected instance is quarantined at next startup so an orphaned restore cannot race new database traffic. Queued jobs that never started are failed without quarantining the instance.

If create cleanup was interrupted, a normal retry refuses orphaned resources. After preserving required data, explicitly add "purge_stale_resources": true to the create request to remove stale paths and containers before retrying.

Import and export admission is bounded to 64 jobs per node and two running or queued jobs per instance. Durable records retain the newest 10,000 completed jobs; active records are never pruned.

Benchmark a running node

Run the benchmark client as a second process on the same Linux node:

sudo dbev \
  --config /etc/databases-everywhere/config.yml \
  --bench

The default test is read-only. It measures authenticated heartbeat latency and throughput, real monitoring WebSocket upgrades with fresh JWTs, and daemon/client CPU and resident memory.

For a sustained, rate-limit-aware test with a random selection of running instances:

sudo dbev \
  --config /etc/databases-everywhere/config.yml \
  --bench \
  --time 5 \
  --max_instances 4

Timed mode uses at most 80 percent of the configured rate limit and reserves the rest for control calls and normal panel traffic. --bench-unthrottled is intended only for an isolated stress node and can deliberately produce many 429 responses.

Import/export throughput testing is destructive and requires explicit authorization:

sudo dbev \
  --config /etc/databases-everywhere/config.yml \
  --bench \
  --bench-instance perf-postgres \
  --bench-import-export

Use only a disposable instance. The benchmark exports its data and imports that artifact back into the same database; logical imports can leave partial changes after a native client failure, and Redis or Qdrant stop during physical import.

Each run writes private report.json, report.md, request and resource CSV samples, and a redacted diagnostics log in a unique dbev-benchmarks directory.

Panel integration checklist

• Generate node UUID, token ID, a random API token, and a different random JWT signing key.
• Render the node configuration and have an administrator run setup.
• Call GET /api/system to verify the contract and enabled capabilities.
• Store the mapping between customer records and daemon instance_id values.
• Create and manage instances through /api/instances.
• Poll /api/heartbeat for management liveness and /api/admin/resources/summary for placement.
• Mint per-user JWTs for monitoring and logs instead of exposing the node token.
• Queue exports, watch the instance import/export stream, and surface temporary downloads at click time.
• Scrape /metrics when Prometheus is available.