Bedrock.ObjectStorage.LocalFilesystem (bedrock v0.7.0)

View Source

Local filesystem implementation of the ObjectStorage behaviour.

This backend stores objects as files on the local filesystem, useful for development and testing. The directory structure mirrors the object keys.

Write atomicity

Readers are written against S3's contract: an object is complete or it is absent, and put_if_not_exists/4 claims a key only by publishing a whole object.

That matters most for put_if_not_exists/4, the writer for chunks, snapshots and the bootstrap record. Its callers in the data plane (Demux.ShardServer, ChunkWriter, Snapshot) read :already_exists as success, so a key claimed by a short object would report success to everyone forever and could never be rewritten. The bootstrap callers (ClusterBootstrap.Discovery, recovery's PersistencePhase) instead treat it as a lost race, which is only sound if the winner's object is whole.

Every write therefore goes to a scratch file in the target's own directory, is fsynced, and is published in one step — rename for put/4, link for put_if_not_exists/4 (rename would clobber an existing object and so cannot express create-only). A failure at any point removes the scratch file and leaves the key untouched, so a retry can still take it.

Scratch files left by a killed writer

In-process failures clean up after themselves. A process killed between the scratch write and the publish cannot, so its scratch file survives — full size, hidden from list/3, and reclaimed by nothing. That is the deliberate trade: before, the same crash left wreckage visible under the real key, where it was permanent and poisonous; now it is inert but invisible. Reclaiming it needs a sweep that can tell a dead scratch file from a live writer's, which is a separate piece of work (bedrock-ck3).

Durability boundary

Object content is fsynced before publication, but the parent DIRECTORY is not: Erlang's :file.open/2 refuses a directory with :eisdir, so the directory entry cannot be synced from the BEAM without a NIF. On power loss a just-published object may therefore be absent rather than short. Absent is a state the writers already handle; short under a permanently-claimed key is the one that could not be recovered from.

Configuration

  • :root - Required. The root directory for storing objects.

Example

backend = ObjectStorage.backend(ObjectStorage.LocalFilesystem, root: "/tmp/objects")
:ok = ObjectStorage.put(backend, "test/key", "data")
{:ok, "data"} = ObjectStorage.get(backend, "test/key")