internetdata (internetdata v1.6.1)
View SourceThe official Erlang client for the InternetData database API.
Build a client once with new/1 and pass the term around. It holds no process and no socket of its own, so there is nothing to release. Every call answers {ok, Term} or {error, Error}; nothing here raises for a failure the API can report.
Results are maps keyed by ATOMS for the fields the API documents, and by BINARIES for anything the server names: the format keys under schema, sample and size, and the dataset column names inside a sample row. See internetdata_result for why.
Summary
Types
The same set at runtime, because format() checks nothing once compiled.
Functions
Every digest published for one dataset file.
Download one dataset file to Path, and answer how many bytes landed.
Download one dataset file and hand back its bytes.
The time-limited URL for one dataset file.
Your organization's recent download attempts, newest first.
Every format a database file is published in: the values format() takes, for checking one that came from a flag or a config file.
Every dataset FAMILY your organization may see, and where each one stands.
What is inside one dataset: column schema and sample rows per format, the row count, the byte size of each artifact, and the day it was built.
Every license_type the pinned spec publishes when it is not null.
Build a client against production with no key.
Build a client.
Start a device sign-in: show the person user_code and verification_uri, then call oauth_poll_device_token/3.
Exchange a device code for tokens, once. Until the person approves, this answers {error, #{error_code := <<"authorization_pending">>}}; oauth_poll_device_token/3 is the loop that waits for them.
Exchange a refresh token for a new pair. The token presented is spent, so keep the refresh_token this answers. Never retried; the answer may name the key in apikey_id but never carries apikey.
The authorization server's discovery document.
Wait for the person to approve a device sign-in, and answer its tokens.
Revoke an access or a refresh token. A refresh token ends the whole grant and every token it issued, which is how a machine signs out.
Every standing the pinned spec publishes for a family in database_list/1. A value added later still reads through as a binary.
Types
-opaque client()
-type device_authorization_options() :: #{scope => binary() | string(), resource => binary() | string(), timeout_ms => pos_integer() | infinity}.
-type downloads_options() :: #{limit => pos_integer(), timeout_ms => pos_integer() | infinity}.
-type format() :: csvgz | mmdb.
The same set at runtime, because format() checks nothing once compiled.
-type oauth_options() :: #{timeout_ms => pos_integer() | infinity}.
-type options() :: #{api_key => binary() | string(), base_url => binary() | string(), retries => non_neg_integer(), timeout_ms => pos_integer() | infinity, http => internetdata_http:http_fun()}.
Functions
-spec database_checksums(client(), binary() | string(), format()) -> {ok, map()} | {error, internetdata_error:error()}.
Every digest published for one dataset file.
The whole set, not one algorithm: which digests a dataset publishes is the API's choice rather than ours, and they arrive nested under checksums.
-spec database_download(client(), binary() | string(), format(), binary() | string()) -> {ok, non_neg_integer()} | {error, internetdata_error:error()}.
Download one dataset file to Path, and answer how many bytes landed.
The transfer is streamed, so nothing beyond a single chunk is ever held in memory whatever the dataset weighs. The bytes go to a neighboring .part file that is renamed only once the whole body has arrived: a transfer that dies halfway leaves neither a truncated file that reads as a complete dataset nor a .part for the next attempt to append to.
The client's timeout_ms bounds the wait between chunks here rather than the whole transfer, because a deadline that suits a listing is the wrong one for a gigabyte while a stalled transfer is stalled at any size.
-spec database_download_bytes(client(), binary() | string(), format()) -> {ok, binary()} | {error, internetdata_error:error()}.
Download one dataset file and hand back its bytes.
This holds the ENTIRE file in memory, and the published catalog runs from a few hundred bytes to several gigabytes, so reach for it at the small end and use database_download/4 for anything you have not checked database_metadata/2 for first. It transfers over exactly the same streamed path, so the bytes are the ones database_download/4 would have written.
-spec database_download_url(client(), binary() | string(), format()) -> {ok, binary()} | {error, internetdata_error:error()}.
The time-limited URL for one dataset file.
The API answers a 302 straight to object storage and this reads the Location without following it, so what comes back is a link that carries NO credential of yours and can be handed to whatever does the transfer. It authorizes the START of a transfer, so one already running is not interrupted when it lapses.
-spec database_downloads(client()) -> {ok, [map()]} | {error, internetdata_error:error()}.
-spec database_downloads(client(), downloads_options()) -> {ok, [map()]} | {error, internetdata_error:error()}.
Your organization's recent download attempts, newest first.
Refusals are listed too: a denial is what answers "it stopped working", and its absence answers nothing. limit defaults to 50 and the API clamps it to 200. timeout_ms bounds each attempt of this call alone, so a retried call can take longer in total.
-spec database_formats() -> [format()].
Every format a database file is published in: the values format() takes, for checking one that came from a flag or a config file.
-spec database_list(client()) -> {ok, [map()]} | {error, internetdata_error:error()}.
Every dataset FAMILY your organization may see, and where each one stands.
The whole published catalog, not only what you license: standing says whether a family is yours today (<<"licensed">>), was (<<"expired">>), or has never been bought (<<"unlicensed">>).
A license covers a family (base), while a download names one of its versions, so the ids the other calls take come from a family's versions rather than from the family itself.
-spec database_metadata(client(), binary() | string()) -> {ok, map()} | {error, internetdata_error:error()}.
What is inside one dataset: column schema and sample rows per format, the row count, the byte size of each artifact, and the day it was built.
Answered from the top level of the response rather than from an envelope, and cheap enough to poll: updated and entries say whether today's build is worth fetching without moving any of it. size is what to check a transfer against before starting one - the catalog spans five orders of magnitude.
-spec license_types() -> [binary()].
Every license_type the pinned spec publishes when it is not null.
-spec new() -> client().
Build a client against production with no key.
Build a client.
api_key is optional. Every database endpoint published today is authenticated, so a client built without one answers 401 there - but that is what the API serves rather than a property of its shape, and a client that could not be BUILT without a key would have to break its own signature the day a dataset is served free. Without one no authorization header is sent at all, and the oauth_* functions never send one.
Create a key in the console with the db.download scope. Keys are default-deny, so an existing key does not gain database access until that scope is added to it.
timeout_ms is milliseconds from 1 to 4294967295, or infinity for no bound, and anything else raises badarg here rather than failing every call.
-spec oauth_device_authorization(client(), binary() | string()) -> {ok, internetdata_oauth:device_authorization()} | {error, internetdata_error:error()}.
-spec oauth_device_authorization(client(), binary() | string(), device_authorization_options()) -> {ok, internetdata_oauth:device_authorization()} | {error, internetdata_error:error()}.
Start a device sign-in: show the person user_code and verification_uri, then call oauth_poll_device_token/3.
Client IDs are issued on request from support@internetdata.io. scope is one space-delimited string, sent as given; the server grants what the client may ask for and silently drops the rest.
-spec oauth_exchange_device_code(client(), binary() | string(), binary() | string()) -> {ok, internetdata_oauth:token_response()} | {error, internetdata_error:error()}.
-spec oauth_exchange_device_code(client(), binary() | string(), binary() | string(), oauth_options()) -> {ok, internetdata_oauth:token_response()} | {error, internetdata_error:error()}.
Exchange a device code for tokens, once. Until the person approves, this answers {error, #{error_code := <<"authorization_pending">>}}; oauth_poll_device_token/3 is the loop that waits for them.
Never retried: the server spends the code when it answers, so a retry after a lost success could only fail and lose the tokens. The answer carries apikey_id and apikey when the person picked a key.
-spec oauth_exchange_refresh_token(client(), binary() | string(), binary() | string()) -> {ok, internetdata_oauth:token_response()} | {error, internetdata_error:error()}.
-spec oauth_exchange_refresh_token(client(), binary() | string(), binary() | string(), oauth_options()) -> {ok, internetdata_oauth:token_response()} | {error, internetdata_error:error()}.
Exchange a refresh token for a new pair. The token presented is spent, so keep the refresh_token this answers. Never retried; the answer may name the key in apikey_id but never carries apikey.
-spec oauth_metadata(client()) -> {ok, internetdata_oauth:metadata()} | {error, internetdata_error:error()}.
-spec oauth_metadata(client(), oauth_options()) -> {ok, internetdata_oauth:metadata()} | {error, internetdata_error:error()}.
The authorization server's discovery document.
Every oauth_* call sends NO credential, whatever the client was built with, and works the same on a client built without a key. timeout_ms in Options bounds each attempt of this call alone.
-spec oauth_poll_device_token(client(), binary() | string(), internetdata_oauth:device_authorization()) -> {ok, internetdata_oauth:token_response()} | {error, internetdata_error:error()}.
-spec oauth_poll_device_token(client(), binary() | string(), internetdata_oauth:device_authorization(), oauth_options()) -> {ok, internetdata_oauth:token_response()} | {error, internetdata_error:error()}.
Wait for the person to approve a device sign-in, and answer its tokens.
Waits the device's interval seconds before EVERY exchange, the first included, and five seconds longer for good each time the server answers slow_down. A refusal answers error_code<<"access_denied">>; an expired code <<"expired_token">>, with no status when the device's expires_in ran out here first. Any other failure, a timeout or an outage included, ends the wait unchanged; calling again with the same device is safe until it expires.
It blocks the calling process until one of those outcomes. There is no cancellation handle, so run it in a process you can kill. timeout_ms bounds each exchange, never the whole wait.
-spec oauth_revoke(client(), binary() | string(), binary() | string(), oauth_options()) -> ok | {error, internetdata_error:error()}.
Revoke an access or a refresh token. A refresh token ends the whole grant and every token it issued, which is how a machine signs out.
-spec standings() -> [binary()].
Every standing the pinned spec publishes for a family in database_list/1. A value added later still reads through as a binary.