[![coverage report](https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/badges/sigma/coverage.svg)](https://osug.gricad-pages.univ-grenoble-alpes.fr/RESIF/wsdataselect/coverage/)

# Wsdataselect

Implementation of the [FDSN dataselect webservice](https://www.fdsn.org/webservices/) for the [EPOS-France seismological datacenter](https://seismology.resif.fr). Serves seismological miniSEED data by querying the [sigma](https://github.com/OSUG/RESIF/sigma) inventory database and shelling out to the [`dataselect`](https://github.com/EarthScope/dataselect) binary to extract and trim data files.

**Stack:** Elixir 1.20 / Erlang 29 / Phoenix 1.8 / PostgreSQL / Bandit

## Architecture

```
                         Client (HTTP)
                              │
┌─────────────────────────────▼──────────────────────────────────────────┐
│                          Phoenix Endpoint                              │
│                                                                        │
│  ┌────────┐    ┌──────────┐    ┌────────┐    ┌──────────┐              │
│  │ Health │───>│ Traffic  │───>│ Version│───>│  Router  │              │
│  │        │    │  Drain   │    │        │    │          │              │
│  └────────┘    └──────────┘    └────────┘    └────┬─────┘              │
│                                                   │                    │
│        (post-response)                            v                    │
│  ┌──────────────────┐   ┌────────────────────────────────────────┐     │
│  │  DeliveryMetrics │   │  /query, /queryauth     /openapi       │     │
│  └────────┬─────────┘   │  pipeline: :fdsn       pipeline:openapi│     │
│           │             └──────────────────────────┬─────────────┘     │
│           │                                        │                   │
│  ┌────────▼─────────┐                              │                   │
│  │    Cleanup       │   ┌──────────────────────────▼─────────────┐     │
│  └──────────────────┘   │          FdsnDataselectPlugs           │     │
│                         │    (parse & validate FDSN parameters)  │     │
│                         └──────────────────────────┬─────────────┘     │
│                                                    │                   │
│                         ┌──────────────────────────▼────────────┐      │
│                         │           QueryController             │      │
│                         │  ┌────────┐ ┌───────────┐ ┌────────┐  │      │
│                         │  │Authent.│ │Authoriz.  │ │Filters │  │      │
│                         │  └───┬────┘ └─────┬─────┘ └───┬────┘  │      │
│                         │      │            │           │       │      │
│                         │      v            v           v       │      │
│                         │  ┌─────────────────────────────────┐  │      │
│                         │  │        evaluate_size            │  │      │
│                         │  └─────────────────────────────────┘  │      │
│                         └──────────────┬────────────────────────┘      │
│                                        │                               │
└────────────────────────────────────────┼───────────────────────────────┘
                                         │
    ┌──────────┐              ┌──────────▼──────────┐    ┌───────────────┐
    │  Auth DB │              │    Dataselect       │    │   Inventory   │
    │ (Digest) │              │      binary         │    │      DB       │
    └──────────┘              └──────────┬──────────┘    └───────┬───────┘
                                         │                       │
                                  ┌──────▼───────┐        ┌──────▼───────┐
                                  │  temp workdir│        │  Data files  │
                                  │   (trimmed)  │        │  (archive)   │
                                  └──────┬───────┘        └──────────────┘
                                         │
                                  ┌──────▼───────┐
                                  │   chunked    │
                                  │   response   │
                                  │  (miniSEED)  │
                                  └──────────────┘
```

## Quick start

### Prerequisites

- Elixir 1.19+ and Erlang 28+ (use [mise](https://mise.jdx.dev/) or [asdf](https://asdf-vm.com/))
- PostgreSQL 13+
- The [`dataselect`](https://github.com/EarthScope/dataselect) binary (for data extraction)

### Setup

```bash
git clone https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect.git
cd wsdataselect
mix setup          # deps.get + ecto.create + migrate + seed
MIX_ENV=dev mix phx.server
```

The server starts at `http://localhost:4000`. The Swagger UI is served at `/`.

### Run tests

Tests require a PostgreSQL instance on port 5432 with trust auth.

```bash
# Option 1: auto-start a Postgres container via podman
mise run test

# Option 2: manually
podman run --rm -d -p 5432:5432 -e POSTGRES_HOST_AUTH_METHOD=trust docker.io/postgres:13.22-trixie
MIX_ENV=test mix test
```

A pre-built `dataselect` binary for tests is at `test/bin/dataselect`.

## Request flow

```
 Client                Endpoint              Controller            DB / Binary
   │                      │                      │                     │
   │  HTTP GET /query     │                      │                     │
   ├─────────────────────►│                      │                     │
   │                      │  FdsnDataselectPlugs │                     │
   │                      ├─────────────────────►│                     │
   │                      │  parse FDSN params   │                     │
   │                      │◄─────────────────────┤                     │
   │                      │                      │                     │
   │                      │  Authentication      │                     │
   │                      ├─────────────────────►│  query Auth DB      │
   │                      │                      ├────────────────────►│
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │  Authorization       │                     │
   │                      ├─────────────────────►│  get closed nets    │
   │                      │                      ├────────────────────►│
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │  Filters             │                     │
   │                      ├─────────────────────►│                     │
   │                      │                      │  query inventory    │
   │                      │                      ├────────────────────►│
   │                      │                      │  (data files list)  │
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │                      │  evaluate_size      │
   │                      │                      │  (too large? → 413) │
   │                      │                      │                     │
   │                      │                      │  dataselect binary  │
   │                      │                      ├────────────────────►│
   │                      │                      │  (read & trim MSEED)│
   │                      │                      │◄────────────────────┤
   │                      │                      │                     │
   │                      │  chunked response    │                     │
   │◄─────────────────────┤◄─────────────────────┤                     │
   │                      │                      │                     │
   │                      │  DeliveryMetrics     │                     │
   │                      ├─────────────────────►│  write metrics DB   │
   │                      │                      ├────────────────────►│
   │                      │                      │                     │
   │                      │  Cleanup             │                     │
   │                      ├─────────────────────►│  rm temp workdir    │
   │                      │                      │                     │
```

1. **FdsnDataselectPlugs** (external [`fdsn_plugs`](https://hex.pm/packages/fdsn_plugs) library) parses FDSN query parameters into a structured `fdsn_parameters` map.
2. **Authentication** — `/queryauth` uses HTTP Digest ([RFC 2617](https://datatracker.ietf.org/doc/html/rfc2617)); `/query` sets the user as `anonymous`.
3. **Authorization** — closed networks are checked against the user. Access can be granted via the `WSDATASELECT_AUTHORIZATIONS` env var.
4. **Filters** — source identifiers are converted to `Filter` structs for the database query.
5. **Inventory query** — `Repo.get_contents/1` fetches matching data files from the inventory database.
6. **Size evaluation** — if the total response exceeds `WSDATASELECT_MAX_RESPONSE_SIZE`, a 413 is returned.
7. **Dataselect binary** — reads and trims miniSEED files in parallel (up to `WSDATASELECT_MAX_CONCURRENCY`), writing output to a temporary workdir.
8. **Chunked response** — the trimmed miniSEED data is streamed back to the client.
9. **DeliveryMetrics** — computes usage statistics per source identifier and writes them to the metrics database.
10. **Cleanup** — removes the temporary workdir.

## Authentication

### HTTP Digest (`/queryauth`)

Implements [RFC 2617](https://datatracker.ietf.org/doc/html/rfc2617). The username (or `anonymous` if unknown) is added to the data request structure.

The default realm is `"FDSN"`, hardcoded in the credential hashes at RESIF. Override at compile time:

```bash
AUTH_REALM="MyRealm" mix compile
```

### JWT

JWT is supported through the `fdsn_plug` library.

You only need to declare the JWKS endpoints as an environment variable in a json format:

    JWKS_AUTHORITIES='[{ "name": "eida", "jwks_url": "https://geofon.gfz.de/eas2/jwk" }]'

If the token is valid, we look for the `email` attribute in the privileges map to grant access to restricted networks.

## Configuration

Configuration is done with environment variables, at runtime.

### Application

| Variable                  | Default                   | Description                                            |
|---------------------------|---------------------------|--------------------------------------------------------|
| `PHX_SERVER`              | _(unset)_                 | Set to any value to enable the Phoenix endpoint server |
| `PHX_HOST`                | `"example.com"`           | Hostname for URL generation                            |
| `PORT`                    | `"4000"`                  | HTTP listen port                                       |
| `SECRET_KEY_BASE`         | **required**              | Phoenix secret for signing/encryption                  |
| `WSDATASELECT_URL_PREFIX` | `"/fdsnws/dataselect/1/"` | URL prefix where the service is accessible             |
| `WSDATASELECT_WORKDIR`    | `"/tmp/dataselect"`       | Temporary directory where dataselect writes output     |
| `DNS_CLUSTER_QUERY`       | _(none)_                  | DNS cluster query for Phoenix DNSCluster               |

### Dataselect binary

| Variable | Default | Description |
|---|---|---|
| `WSDATASELECT_DATASELECT_PATH` | `"/usr/local/bin/dataselect"` | Path to the dataselect binary |
| `WSDATASELECT_DATASELECT_TIMEOUT` | `"5000"` | Timeout for reading data with the dataselect binary (ms) |
| `WSDATASELECT_MAX_CONCURRENCY` | `"8"` | Number of dataselect processes to start simultaneously |
| `WSDATASELECT_MAX_RESPONSE_SIZE` | `"2000000000"` | Maximum response size in bytes (~2 GB) |
| `WSDATASELECT_MAX_SOURCE_IDENTIFIERS` | `"300"` | Maximum number of selected source identifiers per request (GET and POST) |
| `WSDATASELECT_REPOSITORIES_ROOT` | `"/data"` | Root mountpoint of the data repositories |

### Database

| Variable | Default | Description |
|---|---|---|
| `DATABASE_URL` | **required** (prod/dev/staging) | Main inventory Postgres URL |
| `AUTH_DATABASE_URL` | **required** (prod/dev/staging) | Authentication Postgres URL |
| `METRICS_DATABASE_URL` | **required** (if metrics enabled) | Delivery metrics Postgres URL |
| `WSDATASELECT_POOL_SIZE` | `"10"` | Ecto repo pool size |
| `WSDATASELECT_POOL_COUNT` | `"1"` | Ecto repo pool count |
| `WSDATASELECT_REPO_TIMEOUT` | `"15000"` | Ecto repo query timeout (ms) |
| `ECTO_IPV6` | _(unset)_ | Set to `"true"` or `"1"` to enable IPv6 for DB connections |

### Delivery metrics

| Variable | Default | Description |
|---|---|---|
| `WSDATASELECT_DELIVERY_METRICS` | `"true"` | Set `"false"` to disable delivery metrics |
| `WSDATASELECT_GEOIP_CACHE_DIR` | `<workdir>/geoip` | Directory for persistent GeoIP database cache (City + ASN) |
| `MAXMIND_LICENSE_KEY` | `nil` | MaxMind license key for GeoIP database downloads |
| `WSDATASELECT_CLEANUP_INTERVAL` | `"360000"` | Vacuum cleaner interval (ms) |
| `WSDATASELECT_AUTHORIZATIONS` | `"{}"` | JSON-encoded authorization overrides |
| `WSDATASELECT_AUTHORIZATIONS_CACHE_TTL` | `"3600"` | Authorization cache TTL (seconds) |

### S3 / Presigned URLs

| Variable | Default | Description |
|---|---|---|
| `AWS_ACCESS_KEY_ID` | _(none)_ | AWS/S3 access key |
| `AWS_SECRET_ACCESS_KEY` | _(none)_ | AWS/S3 secret key |
| `WSDATASELECT_S3_PUBLIC_HOST` | `"s3.seisdata.epos-france.fr"` | S3 public host for data access |
| `WSDATASELECT_S3_BUCKET` | `"sandbox"` | S3 bucket name |

### Observability (Sentry)

| Variable | Default | Description |
|---|---|---|
| `SENTRY_DSN` | _(none)_ | Sentry DSN for error tracking (read by the Sentry library) |
| `SENTRY_ENVIRONMENT` | _(none)_ | Sentry environment name (read by the Sentry library) |
| `SENTRY_RELEASE` | _(set at build)_ | Sentry release identifier |
| `SENTRY_TRACES_SAMPLE_RATE` | `"0.001"` | Sentry traces sample rate (compile-time) |

### Authentication & Networking

| Variable | Default | Description |
|---|---|---|
| `AUTH_REALM` | `"FDSN"` | HTTP Digest auth realm (compile-time: `AUTH_REALM="MyRealm" mix compile`) |
| `WSDATASELECT_PROXIES` | `""` | Space-separated list of trusted proxy IPs (compile-time) |

### Test

| Variable | Default | Description |
|---|---|---|
| `MIX_TEST_DBHOST` | `"localhost"` | Test DB hostname |
| `MIX_TEST_PARTITION` | `""` | Test DB partition suffix |

## Deployment

### Prerequisites

- PostgreSQL databases: inventory (sigma), authentication, and optionally delivery metrics
- Data archives mounted at `WSDATASELECT_REPOSITORIES_ROOT`, coherent with the `repositories` table in the inventory DB
- [`dataselect`](https://github.com/EarthScope/dataselect) binary compiled and present at `WSDATASELECT_DATASELECT_PATH`

### Containers

Pre-built containers are available in the [Gricad GitLab forge](https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/RESIF/wsdataselect/container_registry/931).

The Dockerfile builds an Elixir release and copies the `dataselect` binary from a dedicated container image.

### CI/CD

GitLab CI pipeline with stages: `test` → `build` → `publish` → `deploy`.

- **Staging** deploys on every commit to the default branch
- **Production** deploys on version tags (`v*`)
- Deployments target a Kubernetes cluster via `kubectl rollout restart`

## Test

Tests use `ex_machina` for factories, `excoveralls` for coverage, and `junit_formatter` for CI reports.

```bash
# Run all tests
MIX_ENV=test mix test

# Run a single test file
mix test test/path/to/file_test.exs

# Run previously failed tests
mix test --failed

# Full precommit check (compile + lint + format + test)
mix precommit
```

CI runs `mix coveralls.multiple --type html --type cobertura` with the `dataselect` binary at `test/bin/dataselect`.