View Source Elsa.ElsaSupervisor (Elsa.fi v3.2.0)
Top-level supervisor that orchestrates all other components of the Elsa library. Allows for a single point of integration into your application supervision tree and configuration by way of a series of nested keyword lists
Components not needed by a running application (if your application only consumes messages from Kafka and never producers back to it) can be safely omitted from the configuration.
Link to this section Summary
Functions
Returns a specification to start this module under a supervisor.
Callback implementation for Supervisor.init/1
.
Defines a connection for locating the Elsa Registry process.
Starts the top-level Elsa supervisor and links it to the current process. Starts a brod client and a custom process registry by default and then conditionally starts and takes supervision of any brod group-based consumers or producer processes defined.
Starts producer processes under Elsa's DynamicSupervisor
for the specified connection.
Link to this section Functions
Returns a specification to start this module under a supervisor.
See Supervisor
.
Callback implementation for Supervisor.init/1
.
Defines a connection for locating the Elsa Registry process.
@spec start_link(keyword()) :: GenServer.on_start()
Starts the top-level Elsa supervisor and links it to the current process. Starts a brod client and a custom process registry by default and then conditionally starts and takes supervision of any brod group-based consumers or producer processes defined.
options
Options
:endpoints
- Required. Keyword list of kafka brokers. ex.[localhost: 9092]
:connection
- Required. Atom used to track kafka connection.:config
- Optional. Client configuration options passed to brod.:producer
- Optional. Can be a single producer configuration of multiples in a list.:group_consumer
- Optional. Group consumer configuration.:consumer
- Optional. Simple topic consumer configuration.
producer-config
Producer Config
:topic
- Required. Producer will be started for configured topic.:poll
- Optional. If set to a number in milliseconds, will poll for new partitions and startup producers on the fly.:config
- Optional. Producer configuration options passed tobrod_producer
.:metadata_request_config
- Optional. See Metadata Request Config
group-consumer-config
Group Consumer Config
:group
- Required. Name of consumer group.:topics
- Required. List of topics to subscribe to.:handler
- Required. Module that implements Elsa.Consumer.MessageHandler behaviour.:handler_init_args
- Optional. Any args to be passed to init function in handler module.:assignment_received_handler
- Optional. Will be called with any partition assignments. SeeElsa.Group.Manager.assignment_received_handler/0
for expected function arity and types.This function will be called once per assignment with arguments (group, topic, partition, generation_id).
Return
:ok
to allow subscription to the current assignment. Return{:error, reason}
to stop subscription.:assignments_complete_handler
- Optional. Will be called after each batch of partition assignments. SeeElsa.Group.Manager.assignments_complete_handler/0
for expected function arity and types.If subscription is successful, this function will be called after all workers are started with (group, generation_id, :ok).
If subscription is unsuccessful, this function will be called with (group, generation_id, {:error, reason}).
:assignments_revoked_handler
- Optional. Will be called when assignments are revoked. SeeElsa.Group.Manager.assignments_revoked_handler/0
for expected function arity and types.All workers will be shut down before this function is called.
:worker_supervisor_max_restarts
- Optional. max_restarts option passed to the WorkerSupervisor. Default 30.:worker_supervisor_max_seconds
- Optional. max_seconds option passed to the WorkerSupervisor. Default 5 seconds.:config
- Optional. Consumer configuration options passed tobrod_consumer
.:metadata_request_config
- Optional. See Metadata Request Config
consumer-config
Consumer Config
:topic
- Required. Topic to subscribe to.:begin_offset
- Required. Where to begin consuming from. Must be either:earliest
,:latest
, or a valid offset integer.:handler
- Required. Module that implementsElsa.Consumer.MessageHandler
behaviour.:partition
- Optional. Topic partition to subscribe to. Ifnil
, will default to all partitions.:handler_init_args
- Optional. Any args to be passed to init function in handler module.:poll
- Optional. If set to number of milliseconds, will poll for new partitions and startup consumers on the fly.:metadata_request_config
- Optional. See Metadata Request Config
metadata-request-config
Metadata Request Config
:metadata_request_tries
- Optional, default 5. The number of tries allowed when querying topic metadata. Typically these retries are necessary when creating topics on the fly and immediately attempting to connect a producer or consumer to them. That's because topic creation in kafka is asynchronous -- even though a call to Elsa.create_topic may return success, that doesn't mean that the metadata for that new topic has propagated to all the brokers.The defaults for
metadata_request_tries
andmetadata_request_dwell_ms
work well for testing with kafka in a local docker container. These values may need to be increased if creating topics on the fly with remote kafka brokers.:metadata_request_dwell_ms
- Optional, default 100. The amount of time to wait between tries when querying topic metadata.
example
Example
Elsa.ElsaSupervisor.start_link([
endpoints: [localhost: 9092],
connection: :conn,
producer: [topic: "topic1"],
consumer: [
topic: "topic2",
partition: 0,
begin_offset: :earliest,
handler: ExampleHandler
],
group_consumer: [
group: "example-group",
topics: ["topic1"],
handler: ExampleHandler,
config: [
begin_offset: :earliest,
offset_reset_policy: :reset_to_earliest
]
]
])
@spec start_producer( String.t() | atom(), keyword() ) :: [DynamicSupervisor.on_start_child()]
Starts producer processes under Elsa's DynamicSupervisor
for the specified connection.
Polling cannot be configured for producers at runtime. Configuration at Elsa.ElsaSupervisor
start
is how polling will behave for all producers on that connection. Other than polling, producer
configuration is the same as Elsa.ElsaSupervisor.start_link/1
.
producer-config
Producer Config
:topic
- Required. Producer will be started for configured topic.:config
- Optional. Producer configuration options passed tobrod_producer
.