internetdata (internetdata v1.0.0)
View SourceThe official Erlang client for the InternetData database API.
Build a client once with new/1 and pass the term around. It holds no process and no socket of its own, so there is nothing to release. Every call answers {ok, Term} or {error, Error}; nothing here raises for a failure the API can report.
Results are maps keyed by ATOMS for the fields the API documents, and by BINARIES for anything the server names: the format keys under schema, sample and size, and the dataset column names inside a sample row. See internetdata_result for why.
The catalog is not the same for everyone. A dataset commissioned for a single customer is simply ABSENT from database_list/1 for anybody else, rather than listed with an unlicensed standing. Read the listing this client returns; do not cache one and reuse it under a different key, and do not assemble a catalog from anywhere else.
Summary
Functions
Every digest published for one dataset file.
Download one dataset file to Path, and answer how many bytes landed.
Download one dataset file and hand back its bytes.
The time-limited URL for one dataset file.
Your organization's recent download attempts, newest first.
Every dataset FAMILY your organization may see, and where each one stands.
What is inside one dataset: column schema and sample rows per format, the row count, the byte size of each artifact, and the day it was built.
Build a client against production with no key.
Build a client.
Types
-opaque client()
-type downloads_options() :: #{limit => pos_integer()}.
-type format() :: csvgz | mmdb.
-type options() :: #{api_key => binary() | string(), base_url => binary() | string(), retries => non_neg_integer(), timeout_ms => pos_integer(), http => internetdata_http:http_fun()}.
Functions
-spec database_checksums(client(), binary() | string(), format()) -> {ok, map()} | {error, internetdata_error:error()}.
Every digest published for one dataset file.
The whole set, not one algorithm: which digests a dataset publishes is the API's choice rather than ours, and they arrive nested under checksums.
-spec database_download(client(), binary() | string(), format(), binary() | string()) -> {ok, non_neg_integer()} | {error, internetdata_error:error()}.
Download one dataset file to Path, and answer how many bytes landed.
The transfer is streamed, so nothing beyond a single chunk is ever held in memory whatever the dataset weighs. The bytes go to a neighboring .part file that is renamed only once the whole body has arrived: a transfer that dies halfway leaves neither a truncated file that reads as a complete dataset nor a .part for the next attempt to append to.
The client's timeout_ms bounds the wait between chunks here rather than the whole transfer, because a deadline that suits a listing is the wrong one for a gigabyte while a stalled transfer is stalled at any size.
-spec database_download_bytes(client(), binary() | string(), format()) -> {ok, binary()} | {error, internetdata_error:error()}.
Download one dataset file and hand back its bytes.
This holds the ENTIRE file in memory, and the published catalog runs from a few hundred bytes to several gigabytes, so reach for it at the small end and use database_download/4 for anything you have not checked database_metadata/2 for first. It transfers over exactly the same streamed path, so the bytes are the ones database_download/4 would have written.
-spec database_download_url(client(), binary() | string(), format()) -> {ok, binary()} | {error, internetdata_error:error()}.
The time-limited URL for one dataset file.
The API answers a 302 straight to object storage and this reads the Location without following it, so what comes back is a link that carries NO credential of yours and can be handed to whatever does the transfer. It authorizes the START of a transfer, so one already running is not interrupted when it lapses.
-spec database_downloads(client()) -> {ok, [map()]} | {error, internetdata_error:error()}.
-spec database_downloads(client(), downloads_options()) -> {ok, [map()]} | {error, internetdata_error:error()}.
Your organization's recent download attempts, newest first.
Refusals are listed too: a denial is what answers "it stopped working", and its absence answers nothing. limit defaults to 50 and the API clamps it to 200.
-spec database_list(client()) -> {ok, [map()]} | {error, internetdata_error:error()}.
Every dataset FAMILY your organization may see, and where each one stands.
The whole published catalog, not only what you license: standing says whether a family is yours today (<<"licensed">>), was (<<"expired">>), or has never been bought (<<"unlicensed">>). Families built for one customer are not listed to anybody else at all, so what comes back depends on the key that asked.
A license covers a family (base), while a download names one of its versions, so the ids the other calls take come from a family's versions rather than from the family itself.
-spec database_metadata(client(), binary() | string()) -> {ok, map()} | {error, internetdata_error:error()}.
What is inside one dataset: column schema and sample rows per format, the row count, the byte size of each artifact, and the day it was built.
Answered from the top level of the response rather than from an envelope, and cheap enough to poll: updated and entries say whether today's build is worth fetching without moving any of it. size is what to check a transfer against before starting one - the catalog spans five orders of magnitude.
-spec new() -> client().
Build a client against production with no key.
Build a client.
api_key is optional. Every endpoint published today is authenticated, so a client built without one answers 401 - but that is what the API serves rather than a property of its shape, and a client that could not be BUILT without a key would have to break its own signature the day a dataset is served free. Without one no authorization header is sent at all.
Create a key in the console with the db.download scope. Keys are default-deny, so an existing key does not gain database access until that scope is added to it.