cets_discovery behaviour (cets v0.3.1)

View Source

Node discovery logic.

Joins table together when a new node appears.

Things that make discovery logic harder:

- A table list is dynamic (but eventually we add all the tables into it).

- Creating Erlang distribution connection is async, but it net_kernel:ping/1 is blocking.

- net_kernel:ping/1 could block for unknown number of seconds (but net_kernel default timeout is 7 seconds).

- Resolving nodename could take a lot of time (5 seconds in tests). It is unpredictable blocking.

- join tables should be one by one to avoid OOM.

- Backend:get_nodes/1 could take a long time.

- cets_discovery:get_tables/1, cets_discovery:add_table/2 should be fast.

- The most important net_kernel flags for us to consider are:

* dist_auto_connect=never

* connect_all

* prevent_overlapping_partitions

These flags change the way the discovery logic behaves. Also the module would not try to connect to the hidden nodes.

Retry logic considerations:

- Backend:get_nodes/1 could return an error during startup, so we have to retry fast.

- There are two periods of operation for this module:

* startup phase, usually first 5 minutes.

* regular operation phase, after the startup phase.

- We don't need to check for the updated get_nodes too often in the regular operation phase.

Summary

Types

Result of get_nodes/2 call.

Backend could define its own options.

Discovery server process.

Discovery status.

Functions

Adds a table to be tracked and joined.

Deletes a table from being tracked or joined.

Gets a list of the tracked tables.

Gets information for each tracked table.

Starts a discovery process.

Starts a discovery process with a link.

Gets discovery process status.

Waits for the current get_nodes call to return.

Blocks until the initial discovery is done.

Types

get_nodes_result/0

-type get_nodes_result() :: {ok, [node()]} | {error, term()}.

Result of get_nodes/2 call.

opts/0

-type opts() :: #{name := atom(), _ := _}.

Backend could define its own options.

server/0

-type server() :: pid() | atom().

Discovery server process.

start_result/0

-type start_result() :: {ok, pid()} | {error, term()}.

Result of start_link/1.

system_info/0

-type system_info() :: map().

Discovery status.

Callbacks

get_nodes/1

-callback get_nodes(backend_state()) -> {get_nodes_result(), backend_state()}.

init/1

-callback init(map()) -> backend_state().

Functions

add_table(Server, Table)

-spec add_table(server(), cets:table_name()) -> ok.

Adds a table to be tracked and joined.

delete_table(Server, Table)

-spec delete_table(server(), cets:table_name()) -> ok.

Deletes a table from being tracked or joined.

get_tables(Server)

-spec get_tables(server()) -> {ok, [cets:table_name()]}.

Gets a list of the tracked tables.

info(Server)

-spec info(server()) -> [cets:info()].

Gets information for each tracked table.

start(Opts)

-spec start(opts()) -> start_result().

Starts a discovery process.

start_link(Opts)

-spec start_link(opts()) -> start_result().

Starts a discovery process with a link.

system_info(Server)

-spec system_info(server()) -> system_info().

Gets discovery process status.

wait_for_get_nodes(Server, Timeout)

-spec wait_for_get_nodes(server(), timeout()) -> ok.

Waits for the current get_nodes call to return.

Just returns if there is no gen_nodes call running. Waits for another get_nodes, if should_retry_get_nodes flag is set. It is different from wait_for_ready, because it does not wait for unavailable nodes to return pang.

wait_for_ready(Server, Timeout)

-spec wait_for_ready(server(), timeout()) -> ok.

Blocks until the initial discovery is done.

This call would also wait till the data is loaded from the remote nodes.