Electric.ShapeCache.PureFileStorage (electric v1.1.0)
View SourceMain architecture & feature overview:
2 file formats: for snapshots and for main log, because snapshots have a requirement to be streamed as they're written
Snapshot format: comma-separated JSON lines (for future copy-to-socket possibilities), one file per chunk, ends with a
0x04
byte (end-of-transmission) to distinguish EOF because reader is up to date with writer from EOF because writer is finishedMain log format:
Log file: binary file, formatted as:
<<tx_offset::64, op_offset::64, key_size::32, key::binary(key_size), op_type::8, flag::8, json_size::64 json::binary(json_size)>>
Chunk file: binary file, formatted as:
<<min_tx_offset::64, min_op_offset::64, start_pos::64, key_start_pos::64, max_tx_offset::64, max_op_offset::64, end_pos::64, key_end_pos::64>>
where start_pos & end_pos can be used for full chunk read into memory if needed, and min/max offsets are inclusive. Last chunk might not have the max/end part of the binary (i.e. it's half width)
Writes are buffered at 64kb or 1s boundary, and the main pointer is the "last persisted full txn offset" - it's updated atomically, and last, and the readers are expected to respect that pointer as an upper bound for reading - any entires in the log file beyond that are considered volatile and might be trimmed in case the writer hard-crashes without flushing. Any reads beyond that boundary should rely on system being live (see 2 next points).
For read consistency on live tail of the log, buffered writes are also made available to readers through an ETS. Anything not flushed to disk yet is addressable inside an ETS, and deleted from there as soon as flushed
Buffering is not transaction-aligned. Flush might include multiple transactions, or be done mid-transaction. To allow for consistent reads, we maintain an in-memory pointer to "last written offset" (always gte than "last persisted full txn offset") which acts as a definitely-synced boundary for ongoing reads which read part of the transaction from ETS and disk when a transaction is partially written.
This 2-layer setup is there to allow for read-consistent buffered writes without reader processes going to the writer.
In case the writer is offline and in-memory ETS/buffer is not present, reads still succeed using on-disk information (i.e. last persisted full txn offset).
Summary
Functions
Callback implementation for c:Electric.ShapeCache.Storage.append_to_log!/2
.
Callback implementation for c:Electric.ShapeCache.Storage.cleanup!/1
.
Callback implementation for c:Electric.ShapeCache.Storage.compact/2
.
Callback implementation for c:Electric.ShapeCache.Storage.for_shape/2
.
Callback implementation for c:Electric.ShapeCache.Storage.get_all_stored_shapes/1
.
Callback implementation for c:Electric.ShapeCache.Storage.get_chunk_end_log_offset/2
.
Callback implementation for c:Electric.ShapeCache.Storage.get_current_position/1
.
Callback implementation for c:Electric.ShapeCache.Storage.get_log_stream/3
.
Callback implementation for c:Electric.ShapeCache.Storage.get_total_disk_usage/1
.
Callback implementation for c:Electric.ShapeCache.Storage.init_writer!/2
.
Callback implementation for c:Electric.ShapeCache.Storage.make_new_snapshot!/2
.
Callback implementation for c:Electric.ShapeCache.Storage.mark_snapshot_as_started/1
.
Callback implementation for c:Electric.ShapeCache.Storage.set_pg_snapshot/2
.
Callback implementation for c:Electric.ShapeCache.Storage.shared_opts/1
.
Callback implementation for c:Electric.ShapeCache.Storage.snapshot_started?/1
.
Callback implementation for c:Electric.ShapeCache.Storage.stack_start_link/1
.
Callback implementation for c:Electric.ShapeCache.Storage.start_link/1
.
Callback implementation for c:Electric.ShapeCache.Storage.terminate/1
.
Functions
Callback implementation for c:Electric.ShapeCache.Storage.append_to_log!/2
.
Callback implementation for c:Electric.ShapeCache.Storage.cleanup!/1
.
Callback implementation for c:Electric.ShapeCache.Storage.compact/2
.
Callback implementation for c:Electric.ShapeCache.Storage.for_shape/2
.
Callback implementation for c:Electric.ShapeCache.Storage.get_all_stored_shapes/1
.
Callback implementation for c:Electric.ShapeCache.Storage.get_chunk_end_log_offset/2
.
Callback implementation for c:Electric.ShapeCache.Storage.get_current_position/1
.
Callback implementation for c:Electric.ShapeCache.Storage.get_log_stream/3
.
Callback implementation for c:Electric.ShapeCache.Storage.get_total_disk_usage/1
.
Callback implementation for c:Electric.ShapeCache.Storage.init_writer!/2
.
Callback implementation for c:Electric.ShapeCache.Storage.make_new_snapshot!/2
.
Callback implementation for c:Electric.ShapeCache.Storage.mark_snapshot_as_started/1
.
Callback implementation for c:Electric.ShapeCache.Storage.set_pg_snapshot/2
.
Callback implementation for c:Electric.ShapeCache.Storage.snapshot_started?/1
.
Callback implementation for c:Electric.ShapeCache.Storage.stack_start_link/1
.
Callback implementation for c:Electric.ShapeCache.Storage.start_link/1
.
Callback implementation for c:Electric.ShapeCache.Storage.terminate/1
.