DatabasesEverywhere data operations

Exports, imports, and backups solve different problems. Keep them separate in the panel UI and apply the correct permissions, retention, and recovery behavior to each one.

Choose the right operation

• Exports are portable database-native dumps. SQL and document stores use logical formats; Redis and Qdrant use physical archives.
• Imports load an instance-owned artifact or acquire data directly from a typed remote database source.
• Backups are physical archives of the complete instance volume for disaster recovery on the same daemon.

The API never accepts arbitrary host paths, helper images, shell commands, or connection URLs from a client.

Import and export jobs

Exports and imports are asynchronous. Queue the operation, then poll the instance-scoped job route or subscribe to its WebSocket.

{
  "job_id": "job-uuid",
  "instance_id": "cust-42-db",
  "action": "export",
  "status": "queued",
  "artifact_id": "9c39d836.postgres.sql",
  "error": null,
  "artifact_size_bytes": null
}

Status moves from queued to running, then to succeeded or failed. Successful artifact size appears after the file is published.

• POST /api/instances/{id}/export queues an export with import-export:write.
• POST /api/instances/{id}/import queues an import with import-export:write.
• GET /api/instances/{id}/import-export/jobs lists that instance's jobs with import-export:read.
• GET /api/instances/{id}/import-export/jobs/{job_id} returns one owned job.

Accepted operations return 202 and a Location header for the job status route.

Export options

An empty request body creates a full plain dump. To request compression or a selective logical export:

{
  "archive_format": "gzip",
  "selection": {
    "mode": "selective",
    "include": ["table_a"],
    "exclude": [],
    "fields": {}
  }
}

archive_format can be plain, gzip, or bzip2. Omit it for Redis and Qdrant.

• PostgreSQL exports .postgres.sql logical dumps.
• MariaDB exports .mariadb.sql logical dumps.
• MySQL exports .mysql.sql logical dumps.
• MongoDB exports .mongodb.archive.gz native archives.
• ClickHouse exports .clickhouse.sql logical dumps.
• Redis exports complete .redis.tar.gz physical archives.
• Qdrant exports complete .qdrant.tar.gz physical archives.

Redis and Qdrant exports are not selective.

Import an artifact

Import an artifact owned by the target instance:

{
  "source": {
    "type": "artifact",
    "artifact_id": "9c39d836.postgres.sql.gz",
    "archive_format": "gzip"
  }
}

An operator can stage a trusted file under paths.imports/<instance_id>/ and use its filename as the artifact ID. API clients cannot provide an arbitrary filesystem path.

Logical protocols accept plain dumps or supported gzip, bzip2, tar, and zip wrappers. Redis and Qdrant artifact imports replace the complete physical database.

Import from a remote database

The target instance determines the protocol. Submit typed connection fields rather than a free-form URL:

{
  "source": {
    "type": "remote",
    "host": "source-db.example.com",
    "port": 5432,
    "tls": true,
    "database": "app",
    "username": "migration_user",
    "password": "source-only-secret"
  },
  "mode": "merge"
}

merge replaces source-named objects while preserving target-only data. wipe clears the target first. Physical Redis and Qdrant artifacts always replace the entire database regardless of this field.

Remote SQL, MongoDB, and ClickHouse acquisition uses one-shot helpers with mode-0600 job-private credential files. Redis and Qdrant keep credentials in process memory. Credentials are not persisted in job records, so a failed credential import must be submitted again.

MySQL and MariaDB imports structurally rebase source-qualified names to the managed target and rewrite object definers to the tenant account. ClickHouse rebases identifiable qualified references. Ambiguous forms are rejected before target data changes.

Qdrant imports migrate collection snapshots and aliases together. They require matching major and minor versions, and the target patch version cannot be older than the source.

Remote-import security

TLS verification is on by default. Plaintext requires both "tls": false in the request and security.remote_import.allow_plaintext: true in node configuration.

Private RFC1918, ULA, or CGNAT destinations require an exact security.remote_import.allowed_private_hosts entry. Loopback, link-local, metadata, multicast, reserved, and mixed public/private DNS answers are rejected.

Redis Cluster and distributed Qdrant sources are rejected because a node-local scan could silently produce a partial migration. Use the database's cluster-aware tooling instead.

Remote acquisition does not lock the source. Quiesce source writes for point-in-time consistency, especially with MongoDB, ClickHouse, Redis, and multi-collection Qdrant. Also quiesce writes to target objects being replaced.

Backups API

• GET /api/instances/{id}/backups lists owned backups with backups:read.
• POST /api/instances/{id}/backups creates a backup immediately with backups:write.
• GET /api/instances/{id}/backups/{backup_id}/contents browses its bounded catalog.
• POST /api/instances/{id}/backups/{backup_id}/restore restores after explicit confirmation with recovery:admin.
• DELETE /api/instances/{id}/backups/{backup_id} deletes one owned backup.
• GET /api/admin/backups/status reports schedule and retention configuration.
• POST /api/admin/backups/run backs up all eligible instances.

List items expose only {id, instance_id, size_bytes, modified_at, sha256}. Host paths and remote object keys never leave the daemon. Ownership is enforced on listing, browsing, downloading, restoring, and deleting.

Local, S3, and Kopia storage

The local driver atomically publishes archives, catalogs, and metadata under paths.backups/<instance_id>/. It also discovers compatible legacy local physical archives.

The s3 driver uses SigV4 requests and bounded-memory multipart upload for archives of 64 MiB and larger. Metadata is written last so incomplete uploads never appear as valid backups. Restores verify recorded size and SHA-256.

backups:
  storage:
    driver: s3
    s3:
      bucket: customer-node-backups
      region: eu-central-1
      endpoint: ""
      prefix: dbev
      access_key_id: ""
      secret_access_key: ""
      session_token: ""
      path_style: false
      allow_http: false
      request_timeout_seconds: 900
      max_retries: 3

Empty AWS credential fields use AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and optional AWS_SESSION_TOKEN. S3-compatible endpoints are supported; plaintext HTTP requires explicit allow_http.

The kopia driver stores one pinned private bundle per backup and identifies it with instance, backup, protocol, size, hash, and creation tags:

backups:
  storage:
    driver: kopia
    kopia:
      executable: /usr/local/bin/kopia
      config_file: /var/lib/dbev/backups/.kopia/repository.config
      repository_password: ""
      operation_timeout_seconds: 3600

The Kopia executable and connected repository configuration must be daemon- or root-owned real files that are not writable by group or others. The repository password can come from configuration or KOPIA_PASSWORD.

Changing drivers does not migrate or merge backup inventories. Remote backups still need enough local staging space for one complete backup or restore.

Backup catalogs

When browsing is enabled, each new backup carries a bounded catalog captured immediately before the physical archive.

SQL, MongoDB, and ClickHouse catalogs include object or schema information and optional truncated row previews. Redis and Qdrant return descriptive schema-less entries without record previews.

This endpoint does not boot an untrusted clone or parse live database files. Existing backups created without a catalog return catalog_available: false.

Treat previews as production database content. Set preview_rows_per_object: 0 to store schema information without rows.

Temporary downloads

The panel asks for a short-lived capability URL immediately before a user clicks download:

POST /api/instances/{id}/artifacts/{artifact_id}/download
POST /api/instances/{id}/backups/{backup_id}/download
{ "expires_in_seconds": 120, "single_use": true }

The daemon returns an origin-relative URL:

{
  "url": "/api/instances/cust-42-db/artifacts/export.postgres.sql/download?token=JWT",
  "expires_at_unix": 1751900000,
  "single_use": true
}

Resolve it against the panel's trusted daemon origin and hand it to the browser. Do not store it. The embedded JWT is the complete credential, expires quickly, and can burn after its first request.

Downloads use bounded buffers with node-wide and per-peer admission limits. A rejected stream returns 429 without consuming a single-use ticket.

Artifact retention and recovery

• GET /api/instances/{id}/artifacts lists owned export artifacts.
• DELETE /api/instances/{id}/artifacts/{artifact_id} deletes one artifact.
• POST /api/instances/{id}/artifacts/retention applies retention to one instance.
• GET /api/admin/recovery/failed-jobs lists failed import and export jobs.
• POST /api/instances/{id}/recovery/jobs/{job_id}/retry safely requeues a replayable export or artifact import.
• POST /api/instances/{id}/recovery/restore force-imports an owned artifact.

Credential imports cannot be replayed because their secrets are not stored. Jobs created before replay metadata existed must also be resubmitted.

Destructive recovery and backup restore require explicit confirmation plus an audit reason:

{
  "artifact_id": "9c39d836.postgres.sql",
  "confirm": true,
  "reason": "customer ticket #123"
}