Overview

Module eysql_cluster

• Description
• Data Types
• Function Index
• Function Details

Behaviours: gen_server.

Description

One process per database cluster: which servers exist, how many connections this node holds to each, and which hosts have failed.

It refreshes on a timer, early after any connection failure, and at start unless load_balance is false. A pick also starts a refresh once the refresh interval has passed, without waiting for it; with an interval of 0 every pick does, and there is no timer. A pick that finds no host starts none: as in the JDBC smart driver, only the interval, a connection failure or a failed discovery makes a refresh due. A refresh discovers servers with yb_servers(), and probes the hosts that have failed (see below). On PostgreSQL, where that function does not exist, and with load_balance false, it stays with the configured hosts, and a refresh only probes. connect/1 asks it for a host and opens the connection in the calling process, so a slow connect never blocks the cluster process.

A discovery asks the configured hosts, the seeds, first, each once and in the order given, and then the discovered servers that are not down, as the JDBC driver's checkAndRefresh dials the hosts in its URL before the servers its getAllAvailableHosts lists. It never dials a server that is down.

A discovery that fails leaves the refresh due, as the JDBC driver's failed refresh leaves its clock where it was: the next pick starts another, though not within a second of the last refresh, and the timer comes back after the full interval, not sooner. So while discovery fails, it is tried again at most once a second while connections are being opened, and once per interval while none are, and each of those refreshes probes the failed hosts that are due. The driver would retry on every connection, and un-mark its down hosts only in a refresh that succeeded; this keeps a busy pool from hammering a dead cluster, and still brings connections back within seconds of a full outage. A discovery that succeeds probes at once the failed hosts that are due and have no probe running, so that they come back with the cluster rather than at the next refresh. A run of failed discoveries is logged once, as a warning, and its end once, at info, by the rules for a host's failures below.

connect/1 waits for the first discovery to finish, either way, as the JDBC driver runs its first refresh before it picks a server. It then tries each host a pick offers, at most once each, until one opens: every eligible server, least loaded first, a tie going to one at random, and then the seeds, as the driver's getConnection tries every eligible server and its caller then makes a plain connection to the hosts in its URL.

The seeds are taken in the order given, as that plain connection takes the hosts in the URL (pgjdbc's loadBalanceHosts is false by default): a connection goes to the first seed that is working, and to the next only when that one fails it. With load_balance false the seeds are all there is, so every connection goes to the first host that works; with target_session_attrs => read_write, to the first that accepts writes.

Each server has two addresses, its host and its public IP, and which one connections go to is decided as the JDBC driver's refresh decides it (eysql_topology:address_column/3). The discovery process resolves the name it dialled and each server's two names, and compares addresses: the first server whose host is the address that answered, or whose public IP is, or whose two names are one address, decides for the life of the cluster process. A client outside the cluster's network reaches it at a public IP, and so connects to each server's. Until a discovery decides, each connects to the public IPs when every server has one and every one resolves, and to the hosts otherwise.

While no discovered server is working, a pick takes the first seed that is; and while no seed is working either, the seeds regardless, as a smart driver falls back to a plain connection to the hosts in its URL. It does neither in the modes where the JDBC driver refuses instead of falling back: only_primary, only_rr, and fallback_to_topology_keys_only with topology keys under true or any. There, once servers are discovered, a pick that finds no eligible server fails with {no_node_available, primary | read_replica | cluster}, as the driver's getLeastLoadedServer throws "No node available", whether or not the connect has tried servers already. Before a discovery has succeeded, and on PostgreSQL, the seeds are all there is, and every mode takes them, whatever their node type: the driver makes a plain connection to its URL hosts when its refresh fails, or finds no yb_servers().

Only a failure to connect marks a host down: the failures pgjdbc reports as SQLSTATE 08001, such as a refused or timed-out connect or a name that does not resolve (eysql_error:connect_failure/1). A host that answers but fails the connection otherwise, with a failed login, a missing database, too many connections, a server starting up or shutting down, or a failed TLS handshake, is not down; it is kept apart as rejected. Either way, and for a standby (below), the host is left out of picks until a refresh's probe, or a connection to it that opens, finds it back. The smart drivers do that for a host that is down: they mark it down, and un-mark it only at a refresh once its delay has passed. A rejected host they skip only for the rest of that one connection, and their next connection tries it again. Here it stays out, so that a server that turns connections away, as one does while it restarts, takes no new connection until a probe finds it taking them, and takes them again from that refresh on. Only the seeds' last resort above takes a failed host, and then only a seed, once per connection. Nothing that happens on a connection once it is open marks its host: not a query error, not a cancelled or slow statement, not one that socket_timeout cuts off, and not the connection's death. Only open_failed and the probes below mark hosts, and both follow a connect.

At each refresh the cluster probes every failed host, down, standby or rejected, whose delay has passed: it connects in a separate process, with the same target_session_attrs check as a pick's connect, and closes the connection. A probe that succeeds ends the failure, as does any connection to the host that opens. One that fails, or takes longer than twice connect_timeout, starts another delay, as a failure of the kind its reason makes it, and starts no refresh: a server that answers 57P03 while it shuts down, and then refuses connections, turns from rejected to down. So a host that stays failed is tried once per refresh, not once per delay. There is at most one probe per host, it is not counted as a connection, and it dies with the cluster. The cluster traps exits for that: probes are linked to it, and it kills them when it stops.

A standby that target_session_attrs => read_write turns away is reachable, but has the wrong role. It is left out and probed like a host that is down, so a promotion is found at the next refresh, or at once through the seeds when the primary fails. snapshot/1 reports it apart from the hosts that are down.

A host that fails is logged once, as a warning with the reason when the caller gives one, and not again for the rest of that run of failures. A standby is logged the same way, at info. The run ends once a connection to the host, a probe's included, has opened since its last failure and a minute has passed since that failure; that is logged at info. A failure of another kind starts a new run: down, standby and rejected are each logged in their own words, so that a failed login never reads as a host that is down.

A discovery that succeeds forgets the failures, runs and probes of hosts that are neither servers nor seeds any more.

Data Types

cluster()

cluster() = pid()

connect_spec()

connect_spec() = #{driver := module(), settings := map(), target_session_attrs := any | read_write}

pick_error()

pick_error() = no_server_available | {no_node_available, primary | read_replica | cluster}

reservation()

reservation() = {reference(), eysql_topology:key()}

Function Index

Function Details

cancel/2

Give Reservation back with no outcome, for example because the connect raised. Its host is not left out.

connect/1

Open a connection to the host this cluster picks. Runs in the caller; the connection is linked to the caller, as with epgsql.

Waits first for the cluster's first discovery to finish, either way, as the JDBC driver runs its first refresh before it picks; the discovery's own timeout bounds the wait. Then tries hosts until one opens, each at most once: every server a pick allows, least loaded first, then the seeds in the order given, where the mode allows them. That is the JDBC driver's getConnection loop, which tries every eligible server, followed by its plain connection to the hosts in its URL.

format_status/1

handle_call/3

handle_cast/2

handle_info/2

init/1

open/1

As connect/1, also returning the host the connection went to.

open/2

As open/1, preferring the hosts a pick would choose among that are not in Avoid (see pick/3). The pool passes the hosts its after_connect hook has just failed on. It marks nothing: other callers still connect to those hosts.

open_failed/2

The connection for Reservation could not be made; picks leave its host out until a refresh finds it back, as after a reason that eysql_error:connect_failure/1 calls unreachable. As open_failed/3, with no reason to log.

open_failed/3

As open_failed/2, failed with Reason, which decides what the host counts as. An unreachable reason (eysql_error:connect_failure/1) marks it down, read_only_server, a standby turned away by target_session_attrs => read_write, marks it a standby, and any other reason, such as a failed login, marks it rejected. Whichever it is, picks leave the host out until a refresh's probe, or a connection to it that opens, finds it back. The first failure of a run is logged with the reason, a standby at info and the others as warnings.

opened/3

The connection for Reservation is open; count it until its process exits.

pick/1

Reserve the best host for a new connection. The caller must follow with opened/3, open_failed/2 or cancel/2; connect/1 does. Until then the host counts as pending, unless the caller exits first. As pick/2 with nothing tried yet.

pick/2

As pick/1, leaving out the hosts in Tried, those the caller has tried already for this connection. {error, no_server_available} when no host is left that the mode allows; in a mode that refuses rather than fall back to the seeds, once servers are discovered, {error, {no_node_available, Type}} instead, Type being primary under only_primary, read_replica under only_rr, and cluster under fallback_to_topology_keys_only.

pick/3

As pick/2, choosing among the hosts it would choose among less those in Avoid, or among all of them when that leaves none. Avoid never changes which hosts those are: the tier, the topology level, the node type and, for the seeds and with load_balance false, the order stay what pick/2 uses. With load_balance false that is the one host that comes first, so Avoid changes nothing there. Nor is it a failure: the hosts are not marked, logged or probed.

refresh/1

Refresh now: read yb_servers() again, unless on PostgreSQL, and probe the failed hosts whose delay has passed.

snapshot/1

The cluster's current view.

• placement: the hosts a new connection may go to now, as a pick chooses them: failed hosts left out, of any kind, node type and topology preference applied; the first working seed while no discovered server is working; and, while no seed is working either, the first seed regardless. With load_balance false, the first host that is working. In the modes that refuse the seeds once servers are discovered, it is empty while no eligible server is working;
• allowed: the hosts a pool may keep connections on, so it can drain the others. A failure alone never takes a host out: a failed host counts as up again once a connection to it has opened, a probe's included. A host is allowed while it is in the level new connections would use on that reckoning, or in the level they would use if no host had failed (see eysql_topology:allowed/5). The seeds, taken in order, are allowed while no discovered server is up on that reckoning, unless the mode refuses them: the first seed, and the first seed that is up, or every seed while none is up. A host discovery no longer lists is in neither;
• failed: the hosts that are down: they could not be connected to, and have not been since;
• read_only: the standbys that target_session_attrs => read_write turned away and that have not been connected to since. They are left out as the hosts in failed are;
• rejected: the hosts that answered and failed a connection otherwise, such as with a failed login, and that no connection has opened to since. They are left out and probed as the hosts in failed are;
• counts: open connections per host, and pending picks per host not yet reported. A probe's connection is in neither.

start_link/1

Start a cluster process from a normalized config (eysql_config).

stop/1

terminate/2

when_ready/1

ready if the first discovery has finished (or there is none to do); otherwise pending, and the caller gets {eysql_cluster_ready, Cluster} when it finishes. open/1 waits for this, so that a pick follows the mode's rules for discovered servers, and a pool's first connections go to the servers rather than all to the seeds.

Overview

Generated by EDoc