Limen.Signal.Asn.Loader (limen v0.2.1)

Copy Markdown View Source

Loads IP-to-ASN data for Limen.Signal.Asn and keeps it current, off the request path.

Loading

At startup the loader reads the :file of the :asn configuration, if it exists. Each table (see Limen.Signal.Asn.Table) is built in a short-lived process, so the memory building takes goes with it, and is complete before it replaces the published one: lookups see the old table or the new one, never a partial load. load/2 and load_rows/2 load other data by hand.

Refreshing

With a :url, the loader downloads the data into :file and checks it for changes on the schedule of the :refresh options (see Limen.Signal.Asn.Schedule and Limen.Config):

  • a check downloads the data only if it changed since the file was last confirmed current, which restarting does not reset;
  • a scheduled check is postponed while the BEAM's schedulers are busier than :max_utilization (sampled for a second) or its memory is above :max_memory: building a table takes a few seconds of one scheduler and, for the full iptoasn.com data, about 100 MB for that time;
  • new data with less than half the ranges of the loaded table is rejected as a broken download, and so is data with no range at all;
  • a failed check keeps the current data and is retried with backoff.

refresh/1 checks at once, whatever the schedule and the load, and status/1 reports what the loader did and plans to do. Every load and check emits telemetry, see Limen.Telemetry.

Summary

Types

result()

The outcome of a check for new data.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

load(instance, path)

Loads an iptoasn.com TSV file (optionally gzipped) into instance.

load_rows(instance, rows)

Loads ranges given as {first, last, asn, country, name} tuples into instance. Meant for tests and small custom tables.

refresh(instance)

Checks the :url for new data now, whatever the schedule and the load, and reschedules the next check from there.

status(instance, timeout \\ :infinity)

What instance has loaded and when it checks next.

utilization(ms)

The share of time the BEAM's schedulers were busy over the next ms milliseconds, between 0.0 and 1.0.

Types

result()

@type result() :: :updated | :unchanged | {:postponed, term()} | {:error, term()}

The outcome of a check for new data.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

load(instance, path)

@spec load(atom(), Path.t()) :: {:ok, non_neg_integer()} | {:error, term()}

Loads an iptoasn.com TSV file (optionally gzipped) into instance.

Returns the number of ranges loaded.

load_rows(instance, rows)

@spec load_rows(atom(), [Limen.Signal.Asn.Table.row()]) ::
  {:ok, non_neg_integer()} | {:error, term()}

Loads ranges given as {first, last, asn, country, name} tuples into instance. Meant for tests and small custom tables.

refresh(instance)

@spec refresh(atom()) :: result()

Checks the :url for new data now, whatever the schedule and the load, and reschedules the next check from there.

status(instance, timeout \\ :infinity)

@spec status(atom(), timeout()) :: map()

What instance has loaded and when it checks next.

  • :ranges, :bytes, :loaded_at and :source describe the loaded table, or are nil when none is;
  • :next_check is when the next check is due, in milliseconds since the epoch, or nil without a :url;
  • :last_check is %{at: milliseconds, result: result}, or nil;
  • :failures counts consecutive failed checks.

The loader answers once it finishes what it is doing, which can be a download; pass a timeout in milliseconds to wait less.

utilization(ms)

@spec utilization(pos_integer()) :: float()

The share of time the BEAM's schedulers were busy over the next ms milliseconds, between 0.0 and 1.0.