Precompiled.Fetch (precompiled v0.2.1)

Copy Markdown View Source

Bytes in, verified files in the project's _build out. Manifests are checked against the hash the batch names; files against the hash the manifest names, before anything reaches the build directory. Install is atomic per dependency: staged under _build/<t>/lib/.precompiled/<dep>.<os pid>-<n> (one dir per install), renamed into place with Mix's "already compiled" marker.

Summary

Functions

GET url's body, capped at :max_bytes. A file:// URL is read from the local disk instead: an absolute, canonical path with no symlink on it (else {:error, "unsafe file URL"}), and a missing file is {:error, "not found: <path>"}.

GET a metadata document (the batch), sending etag as If-None-Match when there is one: the body and its ETag, or :not_modified. A failure to reach the source (transport error, timeout, the run deadline, a 5xx, offline) is {:error, {:unreachable, reason}}; anything the source actually answered (a 4xx, an oversized or refused redirect) is {:error, reason}.

Verifies and installs one dependency's build into build_lib. Everything (files, Mix's marker, provenance, the include/src links) is put together in staging first; replacing the old dep dir by a rename is the last step, so any failure leaves _build as it was.

Whether dep's build dir holds exactly entry's files, judged from the disk alone, never from the provenance file: the entry passes the same checks as an install; ebin (and priv, when the entry ships one) are plain directories holding only regular files, exactly the entry's paths; each has the entry's mode and size, and its bytes hash to the entry's sha256; and the .app owns exactly the beams. A priv the entry does not ship must be absent, or Mix's symlink to deps/<dep>/priv, which is not followed: Mix makes and remakes that link itself from the sources deps.get verified, so it holds nothing an install placed. A real priv dir, even an empty one, is refused. Other top-level entries (.mix, a consolidated dir, Mix's include link) are not looked at: Mix writes them, and none of them is code the build's .app loads.

installed_as?/3 for a directory named anything (<app>-<vsn> in a store, say): dir is the entry's own directory and dep the app whose .app it must hold. A priv the entry does not ship must be absent: even an empty one is refused, since :code.priv_dir/1 points there. Only with mix_priv_link: true (a Mix project's _build) may it be a symlink to .../<dep>/priv, Mix's link to the dep's sources.

The manifest the batch names by expected_sha, under path: "manifests" for a format-1 document, "v2/manifests" for a format-2 one. Either comes back as format 2 (see Precompiled.Manifest).

Mix's "already compiled" marker (Mix.Dep.ElixirSCM) in the format the running Mix reads: Elixir 1.20 added the lock (format 2); 1.17 to 1.19 read format 1 and would treat format 2 as stale and recompile the dep.

The provenance file under a dependency's build dir, written by install/6 when it is given :provenance.

Fetches entry's files into staging and verifies them: the entry's paths and modes, each file's size and sha256, opts[:verify_app] (a check on the staged .app, fn path -> :ok | {:fail, why} end), and the .app owning exactly the staged beams. Nothing else: no Mix marker, no provenance, no links, no rename, so it serves an install outside a Mix project too. On a failure staging is removed.

Removes the temp files (*.tmp-*) that runs killed mid-write left in the cache's files and manifests dirs, when last modified before cutoff (Unix seconds). Another run sharing the home may be writing its own right now, so the cutoff is well before this run started. Only regular files directly in a plain cache dir are removed, and a failure is ignored: it costs disk, never a build.

Rewrites an installed dep's provenance: a kept build that a newer manifest still lists. Only into a .mix that is a plain directory.

Functions

get_body(url, opts \\ [])

@spec get_body(String.t(), keyword()) :: {:ok, binary()} | {:error, String.t()}

GET url's body, capped at :max_bytes. A file:// URL is read from the local disk instead: an absolute, canonical path with no symlink on it (else {:error, "unsafe file URL"}), and a missing file is {:error, "not found: <path>"}.

get_conditional(config, url, etag)

@spec get_conditional(map(), String.t(), String.t() | nil) ::
  {:ok, binary(), String.t() | nil}
  | :not_modified
  | {:error, String.t() | {:unreachable, String.t()}}

GET a metadata document (the batch), sending etag as If-None-Match when there is one: the body and its ETag, or :not_modified. A failure to reach the source (transport error, timeout, the run deadline, a 5xx, offline) is {:error, {:unreachable, reason}}; anything the source actually answered (a 4xx, an oversized or refused redirect) is {:error, reason}.

A file:// URL (a bundle on local storage) is read from the disk, never over HTTP, and never answers :not_modified: see get_body/2.

install(config, dep, entry, lock_entry, build_lib, opts \\ [])

@spec install(map(), atom(), map(), tuple(), Path.t(), keyword()) ::
  {:ok,
   %{
     files: non_neg_integer(),
     bytes: non_neg_integer(),
     cached: non_neg_integer()
   }}
  | {:fail, String.t()}

Verifies and installs one dependency's build into build_lib. Everything (files, Mix's marker, provenance, the include/src links) is put together in staging first; replacing the old dep dir by a rename is the last step, so any failure leaves _build as it was.

Options: :deps_path (required), :provenance, :verify_app (a check on the staged .app), and :link_sources, a test seam that replaces the linker ((dep, entry, staging, dest, deps_path -> :ok)).

installed_as?(build_lib, dep, entry)

@spec installed_as?(Path.t(), atom(), map()) :: boolean()

Whether dep's build dir holds exactly entry's files, judged from the disk alone, never from the provenance file: the entry passes the same checks as an install; ebin (and priv, when the entry ships one) are plain directories holding only regular files, exactly the entry's paths; each has the entry's mode and size, and its bytes hash to the entry's sha256; and the .app owns exactly the beams. A priv the entry does not ship must be absent, or Mix's symlink to deps/<dep>/priv, which is not followed: Mix makes and remakes that link itself from the sources deps.get verified, so it holds nothing an install placed. A real priv dir, even an empty one, is refused. Other top-level entries (.mix, a consolidated dir, Mix's include link) are not looked at: Mix writes them, and none of them is code the build's .app loads.

installed_at?(dir, dep, entry, opts \\ [])

@spec installed_at?(Path.t(), atom(), map(), keyword()) :: boolean()

installed_as?/3 for a directory named anything (<app>-<vsn> in a store, say): dir is the entry's own directory and dep the app whose .app it must hold. A priv the entry does not ship must be absent: even an empty one is refused, since :code.priv_dir/1 points there. Only with mix_priv_link: true (a Mix project's _build) may it be a symlink to .../<dep>/priv, Mix's link to the dep's sources.

manifest(config, name, expected_sha, path \\ "manifests")

@spec manifest(map(), String.t(), String.t(), String.t()) ::
  {:ok, map()}
  | {:miss, String.t()}
  | {:fail, String.t()}
  | {:unreachable, String.t()}

The manifest the batch names by expected_sha, under path: "manifests" for a format-1 document, "v2/manifests" for a format-2 one. Either comes back as format 2 (see Precompiled.Manifest).

Looked up, in order: the cache (<home>/manifests/<sha>.json, re-hashed on every read); where by-sha paths exist, <origin>/<path>/by-sha/<sha>.json for each content origin and then the source; and last the source's <path>/<name>.json, whose answer decides the result. Whatever bytes come back are used only if they hash to expected_sha.

{:unreachable, why}: the manifest is not cached and the network could not deliver it (offline, a transport error, a 5xx, the run deadline), as opposed to a source that answered and refused.

marker_term(vsn, scm, lock_entry)

@spec marker_term({String.t(), charlist()}, module(), tuple()) :: tuple()

Mix's "already compiled" marker (Mix.Dep.ElixirSCM) in the format the running Mix reads: Elixir 1.20 added the lock (format 2); 1.17 to 1.19 read format 1 and would treat format 2 as stale and recompile the dep.

marker_term(elixir_version, vsn, scm, lock_entry)

@spec marker_term(String.t(), {String.t(), charlist()}, module(), tuple()) :: tuple()

provenance_path(build_lib, dep)

@spec provenance_path(Path.t(), atom() | String.t()) :: Path.t()

The provenance file under a dependency's build dir, written by install/6 when it is given :provenance.

safe_mode?(mode)

@spec safe_mode?(term()) :: boolean()

safe_path?(path)

@spec safe_path?(term()) :: boolean()

stage_verified(config, dep, entry, staging, opts)

@spec stage_verified(map(), atom(), map(), Path.t(), keyword()) ::
  {:ok,
   %{
     files: non_neg_integer(),
     bytes: non_neg_integer(),
     cached: non_neg_integer()
   }}
  | {:fail, String.t()}

Fetches entry's files into staging and verifies them: the entry's paths and modes, each file's size and sha256, opts[:verify_app] (a check on the staged .app, fn path -> :ok | {:fail, why} end), and the .app owning exactly the staged beams. Nothing else: no Mix marker, no provenance, no links, no rename, so it serves an install outside a Mix project too. On a failure staging is removed.

sweep_tmp(config, cutoff)

@spec sweep_tmp(map(), integer()) :: :ok

Removes the temp files (*.tmp-*) that runs killed mid-write left in the cache's files and manifests dirs, when last modified before cutoff (Unix seconds). Another run sharing the home may be writing its own right now, so the cutoff is well before this run started. Only regular files directly in a plain cache dir are removed, and a failure is ignored: it costs disk, never a build.

write_provenance_file(build_lib, dep, provenance)

@spec write_provenance_file(Path.t(), atom() | String.t(), map()) ::
  :ok | {:error, term()}

Rewrites an installed dep's provenance: a kept build that a newer manifest still lists. Only into a .mix that is a plain directory.