Wsdataselect

Implementation of the FDSN dataselect webservice for the EPOS-France seismological datacenter. Serves seismological miniSEED data by querying the sigma inventory database and shelling out to the dataselect binary to extract and trim data files.

Stack: Elixir 1.19 / Erlang 28 / Phoenix 1.8 / PostgreSQL / Bandit

Architecture

                         Client (HTTP)
                              │
┌─────────────────────────────▼─────────────────────────────────────────────┐
│                          Phoenix Endpoint                                 │
│                                                                           │
│  ┌────────┐    ┌──────────┐    ┌────────┐    ┌──────────┐               │
│  │ Health │───►│ Traffic  │───►│ Version│───►│  Router  │               │
│  │        │    │  Drain   │    │        │    │          │               │
│  └────────┘    └──────────┘    └────────┘    └────┬─────┘               │
│                                                    │                     │
│        (post-response)                             │                     │
│  ┌──────────────────┐   ┌──────────────────────────▼──────────────┐      │
│  │  DeliveryMetrics │   │  /query, /queryauth     /openapi        │      │
│  └────────┬─────────┘   │  pipeline: :fdsn       pipeline:openapi│      │
│           │             └──────────────────────────┬──────────────┘      │
│  ┌────────▼─────────┐                              │                     │
│  │    Cleanup       │   ┌──────────────────────────▼──────────────┐      │
│  └──────────────────┘   │          FdsnDataselectPlugs            │      │
│                         │    (parse & validate FDSN parameters)   │      │
│                         └──────────────────────────┬──────────────┘      │
│                                                    │                     │
│                         ┌──────────────────────────▼──────────────┐      │
│                         │           QueryController               │      │
│                         │  ┌────────┐ ┌───────────┐ ┌────────┐   │      │
│                         │  │Authent.│ │Authoriz.  │ │Filters │   │      │
│                         │  └───┬────┘ └─────┬─────┘ └───┬────┘   │      │
│                         │      │            │           │        │      │
│                         │  ┌───▼────────────▼───────────▼────┐   │      │
│                         │  │        evaluate_size            │   │      │
│                         │  └─────────────────────────────────┘   │      │
│                         └──────────────┬────────────────────────┘      │
│                                        │                               │
└────────────────────────────────────────┼───────────────────────────────┘
                                         │
    ┌──────────┐              ┌──────────▼──────────┐    ┌───────────────┐
    │  Auth DB │              │    Dataselect        │    │   Inventory   │
    │ (Digest) │              │      binary          │    │      DB       │
    └──────────┘              └──────────┬───────────┘    └───────┬───────┘
                                         │                       │
                                  ┌──────▼───────┐        ┌──────▼───────┐
                                  │  temp workdir│        │  Data files  │
                                  │   (trimmed)  │        │  (archive)   │
                                  └──────┬───────┘        └──────────────┘
                                         │
                                  ┌──────▼───────┐
                                  │   chunked    │
                                  │   response   │
                                  │  (miniSEED)  │
                                  └──────────────┘

Quick start

Prerequisites

Elixir 1.19+ and Erlang 28+ (use mise or asdf)
PostgreSQL 13+
The dataselect binary (for data extraction)

Setup

git clone https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect.git
cd wsdataselect
mix setup          # deps.get + ecto.create + migrate + seed
MIX_ENV=dev mix phx.server

The server starts at http://localhost:4000. The Swagger UI is served at /.

Run tests

Tests require a PostgreSQL instance on port 5432 with trust auth.

# Option 1: auto-start a Postgres container via podman
mise run test

# Option 2: manually
podman run --rm -d -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust docker.io/postgres:13.22-trixie
MIX_ENV=test mix test

A pre-built dataselect binary for tests is at test/bin/dataselect.

Request flow

 Client                Endpoint              Controller            DB / Binary
   │                      │                      │                     │
   │  HTTP GET /query     │                      │                     │
   ├─────────────────────►│                      │                     │
   │                      │  FdsnDataselectPlugs │                     │
   │                      ├─────────────────────►│                     │
   │                      │  parse FDSN params   │                     │
   │                      │◄─────────────────────┤                     │
   │                      │                      │                     │
   │                      │  Authentication      │                     │
   │                      ├─────────────────────►│  query Auth DB      │
   │                      │                      ├────────────────────►│
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │  Authorization       │                     │
   │                      ├─────────────────────►│  get closed nets    │
   │                      │                      ├────────────────────►│
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │  Filters             │                     │
   │                      ├─────────────────────►│                     │
   │                      │                      │  query inventory    │
   │                      │                      ├────────────────────►│
   │                      │                      │  (data files list)  │
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │                      │  evaluate_size      │
   │                      │                      │  (too large? → 413) │
   │                      │                      │                     │
   │                      │                      │  dataselect binary  │
   │                      │                      ├────────────────────►│
   │                      │                      │  (read & trim MSEED)│
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │  chunked response    │                     │
   │◄─────────────────────┤◄─────────────────────┤                     │
   │                      │                      │                     │
   │                      │  DeliveryMetrics     │                     │
   │                      ├─────────────────────►│  write metrics DB   │
   │                      │                      ├────────────────────►│
   │                      │                      │                     │
   │                      │  Cleanup             │                     │
   │                      ├─────────────────────►│  rm temp workdir    │
   │                      │                      │                     │

FdsnDataselectPlugs (external fdsn_plugs library) parses FDSN query parameters into a structured fdsn_parameters map.
Authentication — /queryauth uses HTTP Digest (RFC 2617); /query sets the user as anonymous.
Authorization — closed networks are checked against the user. Access can be granted via the WSDATASELECT_AUTHORIZATIONS env var.
Filters — source identifiers are converted to Filter structs for the database query.
Inventory query — Repo.get_contents/1 fetches matching data files from the inventory database.
Size evaluation — if the total response exceeds WSDATASELECT_MAX_RESPONSE_SIZE, a 413 is returned.
Dataselect binary — reads and trims miniSEED files in parallel (up to WSDATASELECT_MAX_CONCURRENCY), writing output to a temporary workdir.
Chunked response — the trimmed miniSEED data is streamed back to the client.
DeliveryMetrics — computes usage statistics per source identifier and writes them to the metrics database.
Cleanup — removes the temporary workdir.

Authentication

HTTP Digest (`/queryauth`)

Implements RFC 2617. The username (or anonymous if unknown) is added to the data request structure.

The default realm is "FDSN", hardcoded in the credential hashes at RESIF. Override at compile time:

AUTH_REALM="MyRealm" mix compile

JWT

Planned but not yet implemented. The JOSE library is available in deps for future JWT-based authentication.

Configuration

Configuration is done with environment variables, at runtime.

Application

Variable	Default	Description
`PHX_SERVER`	(unset)	Set to any value to enable the Phoenix endpoint server
`PHX_HOST`	`"example.com"`	Hostname for URL generation
`PORT`	`"4000"`	HTTP listen port
`SECRET_KEY_BASE`	required	Phoenix secret for signing/encryption
`WSDATASELECT_URL_PREFIX`	`"/fdsnws/dataselect/1/"`	URL prefix where the service is accessible
`WSDATASELECT_WORKDIR`	`"/tmp/dataselect"`	Temporary directory where dataselect writes output
`DNS_CLUSTER_QUERY`	(none)	DNS cluster query for Phoenix DNSCluster

Dataselect binary

Variable	Default	Description
`WSDATASELECT_DATASELECT_PATH`	`"/usr/local/bin/dataselect"`	Path to the dataselect binary
`WSDATASELECT_DATASELECT_TIMEOUT`	`"5000"`	Timeout for reading data with the dataselect binary (ms)
`WSDATASELECT_MAX_CONCURRENCY`	`"8"`	Number of dataselect processes to start simultaneously
`WSDATASELECT_MAX_RESPONSE_SIZE`	`"2000000000"`	Maximum response size in bytes (~2 GB)
`WSDATASELECT_REPOSITORIES_ROOT`	`"/data"`	Root mountpoint of the data repositories

Database

Variable	Default	Description
`DATABASE_URL`	required (prod/dev/staging)	Main inventory Postgres URL
`AUTH_DATABASE_URL`	required (prod/dev/staging)	Authentication Postgres URL
`METRICS_DATABASE_URL`	required (if metrics enabled)	Delivery metrics Postgres URL
`WSDATASELECT_POOL_SIZE`	`"10"`	Ecto repo pool size
`WSDATASELECT_POOL_COUNT`	`"1"`	Ecto repo pool count
`WSDATASELECT_REPO_TIMEOUT`	`"15000"`	Ecto repo query timeout (ms)
`ECTO_IPV6`	(unset)	Set to `"true"` or `"1"` to enable IPv6 for DB connections

Delivery metrics

Variable	Default	Description
`WSDATASELECT_DELIVERY_METRICS`	`"true"`	Set `"false"` to disable delivery metrics
`WSDATASELECT_CLEANUP_INTERVAL`	`"360000"`	Vacuum cleaner interval (ms)
`WSDATASELECT_AUTHORIZATIONS`	`"{}"`	JSON-encoded authorization overrides
`WSDATASELECT_AUTHORIZATIONS_CACHE_TTL`	`"3600"`	Authorization cache TTL (seconds)
`MAXMIND_LICENSE_KEY`	`nil`	MaxMind GeoIP license key

S3 / Presigned URLs

Variable	Default	Description
`AWS_ACCESS_KEY_ID`	(none)	AWS/S3 access key
`AWS_SECRET_ACCESS_KEY`	(none)	AWS/S3 secret key
`WSDATASELECT_S3_PUBLIC_HOST`	`"s3.seisdata.epos-france.fr"`	S3 public host for data access
`WSDATASELECT_S3_BUCKET`	`"sandbox"`	S3 bucket name

Observability (Sentry)

Variable	Default	Description
`SENTRY_DSN`	(none)	Sentry DSN for error tracking (read by the Sentry library)
`SENTRY_ENVIRONMENT`	(none)	Sentry environment name (read by the Sentry library)
`SENTRY_RELEASE`	(set at build)	Sentry release identifier
`SENTRY_TRACES_SAMPLE_RATE`	`"0.001"`	Sentry traces sample rate (compile-time)

Authentication & Networking

Variable	Default	Description
`AUTH_REALM`	`"FDSN"`	HTTP Digest auth realm (compile-time: `AUTH_REALM="MyRealm" mix compile`)
`WSDATASELECT_PROXIES`	`""`	Space-separated list of trusted proxy IPs (compile-time)

Test

Variable	Default	Description
`MIX_TEST_DBHOST`	`"localhost"`	Test DB hostname
`MIX_TEST_PARTITION`	`""`	Test DB partition suffix

Deployment

Prerequisites

PostgreSQL databases: inventory (sigma), authentication, and optionally delivery metrics
Data archives mounted at WSDATASELECT_REPOSITORIES_ROOT, coherent with the repositories table in the inventory DB
dataselect binary compiled and present at WSDATASELECT_DATASELECT_PATH

Containers

Pre-built containers are available in the Gricad GitLab forge.

The Dockerfile builds an Elixir release and copies the dataselect binary from a dedicated container image.

CI/CD

GitLab CI pipeline with stages: test → build → publish → deploy.

Staging deploys on every commit to the default branch
Production deploys on version tags (v*)
Deployments target a Kubernetes cluster via kubectl rollout restart

Test

Tests use ex_machina for factories, excoveralls for coverage, and junit_formatter for CI reports.

# Run all tests
MIX_ENV=test mix test

# Run a single test file
mix test test/path/to/file_test.exs

# Run previously failed tests
mix test --failed

# Full precommit check (compile + lint + format + test)
mix precommit

CI runs mix coveralls.multiple --type html --type cobertura with the dataselect binary at test/bin/dataselect.

Wsdataselect

Architecture

Quick start

Prerequisites

Setup

Run tests

Request flow

Authentication

HTTP Digest (/queryauth)

JWT

Configuration

Application

Dataselect binary

Database

Delivery metrics

S3 / Presigned URLs

Observability (Sentry)

Authentication & Networking

Test

Deployment

Prerequisites

Containers

CI/CD

Test

HTTP Digest (`/queryauth`)