ExStorageService core

This umbrella app owns object metadata, local content-addressed storage, cluster placement, durability, repair, scrub, drain, and safe cleanup. It uses Concord/VSR for metadata and does not use Ecto or an external database.

Embedding

Add the versioned core package without the S3 or admin applications:

def deps do
  [
    {:ex_storage_service, "~> 0.6"}
  ]
end

Disable the default local instance when the host owns its supervision tree:

config :ex_storage_service,
  data_root: "/var/lib/singularity/ess",
  blob_root: "/var/lib/singularity/ess/cas",
  tmp_root: "/var/lib/singularity/ess/cas/tmp",
  ra_root: "/var/lib/singularity/ess/ra",
  metadata_root: "/var/lib/singularity/ess/concord",
  instance_config: [auto_start: false, web_enabled: false]

config :concord,
  data_dir: "/var/lib/singularity/ess/concord"

Then supervise an instance with stable roots. Staging and ready roots must be on the same filesystem so publication can use an atomic rename.

instance_options = [
  instance: :singularity,
  auto_start: false,
  web_enabled: false,
  workers: %{
    multipart_gc: false,
    content_gc: false,
    cas_gc: false,
    packer: false,
    lifecycle: false,
    cross_cluster_replication: false,
    repair: false,
    scrub: false
  }
]

children = [{ExStorageService, instance_options}]

For direct content-addressed storage, derive options from the validated instance context and use ExStorageService.BlobStore.LocalCAS:

{:ok, context} = ExStorageService.context(instance_options)
blob_options = ExStorageService.Context.blob_store_options(context)

{:ok, staged} = ExStorageService.BlobStore.LocalCAS.stage(data, blob_options)
# staged.hash is the lowercase SHA-256 digest; staged.size is the byte count.

{:ok, ready} = ExStorageService.BlobStore.LocalCAS.commit(staged, blob_options)
# Commit is put-if-absent: identical content reuses the verified ready blob.

{:ok, %{size: size}} =
  ExStorageService.BlobStore.LocalCAS.stat(ready.hash, blob_options)

{:ok, {:file, path, offset, length}} =
  ExStorageService.BlobStore.LocalCAS.open(ready.hash, {10, 20}, blob_options)

:ok = ExStorageService.BlobStore.LocalCAS.verify(ready.hash, blob_options)

Call LocalCAS.discard/2 to abort a staged write. LocalCAS.commit/2 is the finalization/pinning boundary: it syncs the staged file and atomically publishes the immutable SHA-256 path. A successful commit may be retried safely and never overwrites different content. LocalCAS.open/3 returns a bounded file source; consumers should stream it rather than load object-sized data into memory.

Ready blobs live below objects/sha256/<first-two-hex>/<remaining-hex> under the configured blob root. Staging paths are private implementation details. Preserve the entire data, blob, temporary, and Concord metadata roots together during backup and upgrades. Package updates preserve ready-blob paths; any future incompatible on-disk or metadata migration will require an explicit release-note and migration boundary rather than an automatic destructive startup migration. Concord/VSR remains shared infrastructure, so only one metadata instance is supported per BEAM.

Operator tasks

Run source-tree tasks from the repository root:

PAGER=cat MIX_ENV=prod mix ess.metadata.schema_status --replication-factor 2
PAGER=cat MIX_ENV=prod mix ess.metadata.migrate_v2 --replication-factor 2
PAGER=cat MIX_ENV=prod mix ess.cluster.bootstrap
PAGER=cat MIX_ENV=prod mix ess.cluster.status
PAGER=cat MIX_ENV=prod mix ess.blob.locate SHA256
PAGER=cat MIX_ENV=prod mix ess.blob.audit --limit 100
PAGER=cat MIX_ENV=prod mix ess.repair.plan --shard 00 --limit 100
PAGER=cat MIX_ENV=prod mix ess.repair.run --shard 00 --limit 100
PAGER=cat MIX_ENV=prod mix ess.node.drain NODE_ID
PAGER=cat MIX_ENV=prod mix ess.node.status NODE_ID --limit 100

Inventory and plan commands are read-only. Migration must run with object writers quiesced after a verified metadata and blob backup. Repair and audit commands operate on bounded pages; process every reported cursor and owned shard before treating the result as a cluster-wide preflight.

Release images do not contain Mix or source files. Use the release RPC equivalents and exact backup, migration, activation, and rollback sequence in the cluster upgrade runbook.