View Source ExWal (ex_wal v0.1.1)
ExWal is a project that aims to provide a solution for managing write-ahead log (WAL) in Elixir.
ExWal is a GenServer-based module that provides functionality for working with a Write-Ahead Log (WAL).
The module includes utilities for managing WAL files, segments, blocks, and entries. It also supports options for customizing the WAL path, synchronization behavior, segment size, segment cache size, and the underlying store module.
Installation
The package can be installed by adding ex_wal
to your list of dependencies in mix.exs
:
def deps do
[
{:ex_wal, "~> 0.1.1"}
]
end
Design
This project has heavily borrowed the code experience from the project at tidwall/wal. In terms of design, WAL packages log entries into individual segments for maintenance. I take the last segment being written as the "hot" part, and while writing hot data, corresponding files will be appended. When the volume of "hot" data exceeds the configured value, we will transfer the hot data to the "cold" part and reopen a new "hot" segment.
In the process of reading, in order to avoid frequent opening of cold data, I added an LRU cache to accelerate the reading speed of local data.
In the operations of maintaining Segment and corresponding log entry, there are a considerable number of operations that find field based on index. However, the performance of List provided by Elixir is not ideal in this situation, so we choose to use the array module of erlang to store segment and block.
Usage
Supervisor.start_link(
[
{ExWal,
[
name: :wal_test,
path: "/tmp/wal_test",
nosync: false,
# 4k per segment
segment_size: 4 * 1024,
# cache max 5 segments
segment_cache_size: 5
]}
],
strategy: :one_for_one
)
latest = ExWal.last_index(:wal_test)
Logger.info("latest: #{latest}")
# write 10k entries
entries =
Enum.map((latest + 1)..(latest + 10_000), fn i -> Entry.new(i, "Hello Elixir #{i}") end)
:ok = ExWal.write(:wal_test, entries)
latest = ExWal.last_index(:wal_test)
Logger.info("latest: #{latest}") # should be latest + 10_000
# read
{:ok, ret} = ExWal.read(:wal_test, latest - 10)
Logger.info("idx: #{latest - 10}, content: #{ret}")
# truncate before
:ok = ExWal.truncate_before(:wal_test, latest - 100)
first = ExWal.first_index(:wal_test)
Logger.info("first: #{first}") # should be latest - 100
# truncate after
:ok = ExWal.truncate_after(:wal_test, latest - 5)
latest = ExWal.last_index(:wal_test)
Logger.info("latest: #{latest}") # should be latest - 5
Benchmark
See benchmarks/report.md for more details.
Documentation can be generated with ExDoc and published on HexDocs. Once published, the docs can be found at https://hexdocs.pm/ex_wal.
Summary
Functions
Returns a specification to start this module under a supervisor.
Clears the write-ahead log (WAL) by removing all entries.
Get the first index of WAL
Get the last index of WAL
Read entry content from WAL by index. if index is not found, return {:error, :index_not_found}
Get the segment count of WAL
Start a WAL process
Stop a WAL process
Truncates the write-ahead log (WAL) after a specific index.
Truncates the write-ahead log (WAL) before a specific index.
Write entries to WAL, the entries must be strictly consecutive and incremental, and the index of the first entry must be WAL's last_index + 1.
Types
@type index() :: non_neg_integer()
@type t() :: %ExWal{ cold: :array.array(), data_path: String.t(), first_index: non_neg_integer(), hot: ExWal.Models.Segment.t(), last_index: non_neg_integer(), lru_cache: atom(), opts: [ nosync: boolean(), segment_size: non_neg_integer(), dir_permission: non_neg_integer(), file_permission: non_neg_integer() ], store: ExWal.Store.t(), tail_store_handler: ExWal.Store.handler() }
Functions
Returns a specification to start this module under a supervisor.
See Supervisor
.
Clears the write-ahead log (WAL) by removing all entries.
Examples
iex> MyApp.WAL.clear(:my_wal)
:ok
Parameters
name_or_pid
- The name or process identifier of the WAL GenServer.
Returns
:ok
- If the clearing is successful.
Get the first index of WAL
Get the last index of WAL
Read entry content from WAL by index. if index is not found, return {:error, :index_not_found}
Examples
iex> {:ok, data} = ExWal.read(:wal_name, 1)
@spec segment_count(atom() | pid()) :: non_neg_integer()
Get the segment count of WAL
@spec start_link(wal_option_schema_t()) :: GenServer.on_start()
Start a WAL process
Options
:path
(String.t/0
) - The path of the WAL. Default is./wal
. The default value is"./wal"
.:nosync
(boolean/0
) - If true, WAL will not sync to disk. Default is false. The default value isfalse
.:segment_size
(integer/0
) - The size of each segment file. Default is 16MB. The default value is16777216
.:segment_cache_size
(integer/0
) - The size of the segment cache. Default is 100. The default value is2
.:store
(atom/0
) - The store module which implementExWal.Store
. Default isExWal.Store.File
. The default value isExWal.Store.File
.:store_opts
(keyword/0
) - The options for the store module. The default value is[]
.:dir_permission
(integer/0
) - The permission of the directory. Default is 0o750. The default value is488
.:file_permission
(integer/0
) - The permission of the file. Default is 0o640. The default value is416
.:name
(atom/0
) - The name of the WAL. Default iswal
. The default value isExWal
.
Stop a WAL process
@spec truncate_after(atom() | pid(), index(), non_neg_integer()) :: :ok | {:error, :index_out_of_range}
Truncates the write-ahead log (WAL) after a specific index.
Examples
iex> MyApp.WAL.truncate_after(:my_wal, 10)
:ok
Parameters
name_or_pid
- The name or process identifier of the WAL GenServer.index
- The index after which to truncate the WAL.timeout
(optional) - The timeout value in milliseconds (default: 5000).
Returns
:ok
- If the truncation is successful.{:error, :index_out_of_range}
- If the provided index is out of range.
@spec truncate_before(atom() | pid(), index(), non_neg_integer()) :: :ok | {:error, :index_out_of_range}
Truncates the write-ahead log (WAL) before a specific index.
Examples
iex> MyApp.WAL.truncate_before(:my_wal, 10)
:ok
Parameters
name_or_pid
- The name or process identifier of the WAL GenServer.index
- The index before which to truncate the WAL.timeout
(optional) - The timeout value in milliseconds (default: 5000).
Returns
:ok
- If the truncation is successful.{:error, :index_out_of_range}
- If the provided index is out of range.
@spec write(atom() | pid(), [ExWal.Models.Entry.t()], non_neg_integer()) :: :ok
Write entries to WAL, the entries must be strictly consecutive and incremental, and the index of the first entry must be WAL's last_index + 1.
Examples
iex> last_index = ExWal.last_index(:wal_name)
iex> enties = last_index..(last_index+10) |> Enum.map(fn i -> Entry.new(i, "some data") end)
iex> :ok = ExWal.write(:wal_name, entries)