Using Resources
Resources are C ABI objects which you declare should be managed by the BEAM reference-counted garbage collector. If you are passing data between function calls, it is best practice to pass them as a resource. Note that Zigler does not currently support passing references between multiple Zigler modules.
Important Note
The examples in this guide build upon each other and are tested in an integration
test; so ~Z
blocks present build upon code that is declared in previous ~Z
blocks.
Basics: Definition
A resource is declared by using the /// resource: <name> definition
directive.
The name
should match a zig type declared in the next line.
Examples
In the following example, the resource i64_res
is bound to the builtin
i64
type. This would allow you to store a mutable 64-bit integer and pass
its pointer between function calls in the BEAM. NOTE: although it's mutable it
is NOT threadsafe.
~Z"""
/// resource: i64_res definition
const i64_res = i64;
"""
In the next example, resource struct_res
is bound to a custom struct type.
This is a more typical situation, where structured data, possibly requiring
internal memory allocation, needs to be passed between function calls.
~Z"""
/// resource: struct_res definition
const struct_res = struct {
first_name: []u8,
last_name: []u8,
};
"""
Creation
In order to create a resource, a __resource__
struct is autogenerated which
makes manipulating BEAM resources much simpler. When returned from the nif, the
resource will appear as a reference/0
term. As this term is passed around,
potentially between processes, it will only be garbage collected when all
references have been to the term have been lost (more on that in
cleanup).
The create
function takes three parameters. First, the type that is being
encapsulated, second, the BEAM environment of the calling function, and third,
the initial value for those parameters.
Important notes
- You zig code will only have access to the
__resource__
struct in the context of NIF code associated with your module in which it resides, within theZig.sigil_z/2
blocks; you cannot use the__resource__
struct in other modules or zig code which has been@include
d. - Erlang/Elixir code will have no direct visibility of the content of these resources: you MUST use nif functions to access or modify those contents.
- Resource references must remain local. Unlike other erlang terms, the content will not be automatically serialized, transferred to another node, and deserialized on the other side.
- Resource creation is failable; you must
catch
the resource error and handle it before returning the resource term. In a future release it will be possible to pass it upwards with thetry
function with a full error return trace. - For datastructures (structs, slices) you have the choice of storing it directly as the resource, or storing the pointer to the datastructure as the resource.
Example: Resource Creation
~Z"""
/// nif: create_i64/0
fn create_i64(env: beam.env) beam.term {
return __resource__.create(i64_res, env, 47)
catch return beam.raise_resource_error(env);
}
/// nif: create_struct/2
fn create_struct(env: beam.env, first_name: []u8, last_name: []u8) beam.term {
var resource_fn = beam.allocator.alloc(u8, first_name.len)
catch return beam.raise_enomem(env);
errdefer beam.allocator.free(resource_fn);
var resource_ln = beam.allocator.alloc(u8, last_name.len)
catch return beam.raise_enomem(env);
errdefer beam.allocator.free(resource_ln);
std.mem.copy(u8, resource_fn, first_name);
std.mem.copy(u8, resource_ln, last_name);
return __resource__.create(struct_res, env, .{
.first_name = resource_fn,
.last_name = resource_ln,
}) catch return beam.raise_resource_error(env);
}
"""
test "resource creation generates references" do
i64_res_example = create_i64()
assert is_reference(i64_res_example)
struct_res_example = create_struct("john", "doe")
assert is_reference(struct_res_example)
end
Example: Resource Pointer Creation
~Z"""
/// resource: struct_ptr_res definition
const struct_ptr_res = *struct_res;
/// nif: create_struct_ptr/2
fn create_struct_ptr(env: beam.env, first_name: []u8, last_name: []u8) beam.term {
var resource = beam.allocator.create(struct_res)
catch return beam.raise_enomem(env);
errdefer beam.allocator.destroy(resource);
resource.first_name = beam.allocator.alloc(u8, first_name.len)
catch return beam.raise_enomem(env);
errdefer beam.allocator.free(resource_fn);
resource.last_name = beam.allocator.alloc(u8, last_name.len)
catch return beam.raise_enomem(env);
errdefer beam.allocator.free(resource_ln);
std.mem.copy(u8, resource.first_name, first_name);
std.mem.copy(u8, resource.last_name, last_name);
// note the signature vv here vv
return __resource__.create(struct_ptr_res, env, resource)
catch return beam.raise_resource_error(env);
}
"""
test "resource pointer generation creates a reference" do
struct_ptr_example = create_struct_ptr("john", "doe")
assert is_reference(struct_ptr_example)
end
Fetching and Updating
The __resource__
struct has fetch
and update
functions which allow you to
retrieve or modify the contents in the struct, respectively. Note that although
you can potentially access these data from different processes, it is not
threadsafe by default without implementing mutexes.
The fetch
function takes three parameters, the internal type of the resource,
the calling function environment, and the resource beam.term
value.
The update
function takes four parameters, the internal type of the resource,
the calling function environment, the resource beam.term
value, and the replacement
value.
Important considerations
- Resource fetching is failable, and you should handle it by returning a beam raise term. This may not necessarily be the same type as the contents of the resource itself! So you might have to marshall it into a term manually. Note: this code will likely become simpler in the future.
- The data encapsulated by the Resource term are mutable! If you update the term, that change will persist within the same call scope, across function call scopes, etc! Also importantly, if you pass the data to another process you must take care to guard changes around mutexes and be aware that change or could be subject to unexpected race conditions without additional coordination.
- For structs, you must replace the entire contents of the struct; depending on your needs, you may want to consider using a pointer, which will let you change the contents of the struct WITHOUT performing an update. Note that extra care must be taken in this case to avoid race conditions resulting in use-after-free segfaults.
Example: Fetching
~Z"""
/// nif: fetch_i64/1
fn fetch_i64(env: beam.env, res: beam.term) beam.term {
var result = __resource__.fetch(i64_res, env, res)
catch return beam.raise_resource_error(env);
return beam.make_i64(env, result);
}
/// nif: fetch_struct_fn/1
fn fetch_struct_fn(env: beam.env, res: beam.term) beam.term {
var result = __resource__.fetch(struct_res, env, res)
catch return beam.raise_resource_error(env);
return beam.make_slice(env, result.first_name);
}
/// nif: fetch_struct_ln/1
fn fetch_struct_ln(env: beam.env, res: beam.term) beam.term {
var result = __resource__.fetch(struct_res, env, res)
catch return beam.raise_resource_error(env);
return beam.make_slice(env, result.last_name);
}
// note that the function bodies for the struct pointer form look NEARLY
// identical to that of the direct struct resource form.
/// nif: fetch_struct_ptr_fn/1
fn fetch_struct_ptr_fn(env: beam.env, res: beam.term) beam.term {
var result = __resource__.fetch(struct_ptr_res, env, res)
catch return beam.raise_resource_error(env);
return beam.make_slice(env, result.first_name);
}
/// nif: fetch_struct_ptr_ln/1
fn fetch_struct_ptr_ln(env: beam.env, res: beam.term) beam.term {
var result = __resource__.fetch(struct_ptr_res, env, res)
catch return beam.raise_resource_error(env);
return beam.make_slice(env, result.last_name);
}
"""
test "resources can be fetched" do
i64_res_example = create_i64()
assert 47 == fetch_i64(i64_res_example)
struct_res_example = create_struct("john", "doe")
assert "john" == fetch_struct_fn(struct_res_example)
assert "doe" == fetch_struct_ln(struct_res_example)
struct_ptr_res_example = create_struct_ptr("john", "doe")
assert "john" == fetch_struct_ptr_fn(struct_ptr_res_example)
assert "doe" == fetch_struct_ptr_ln(struct_ptr_res_example)
end
Example: Updating
~Z"""
/// nif: update_i64/2
fn update_i64(env: beam.env, res: beam.term, update_value: i64) beam.term {
__resource__.update(i64_res, env, res, update_value)
catch return beam.raise_resource_error(env);
return beam.make_ok(env);
}
/// nif: update_struct_fn/2
fn update_struct_fn(env: beam.env, res: beam.term, update_value: []u8) beam.term {
var old_value = __resource__.fetch(struct_res, env, res)
catch return beam.raise_resource_error(env);
var new_first_name = beam.allocator.alloc(u8, update_value.len)
catch return beam.raise_enomem(env);
errdefer beam.allocator.free(new_first_name);
std.mem.copy(u8, new_first_name, update_value);
defer beam.allocator.free(old_value.first_name);
var new_struct = .{
.first_name = new_first_name,
.last_name = old_value.last_name,
};
__resource__.update(struct_res, env, res, new_struct)
catch return beam.raise_resource_error(env);
return beam.make_ok(env);
}
/// nif: update_struct_ln/2
fn update_struct_ln(env: beam.env, res: beam.term, update_value: []u8) beam.term {
var old_value = __resource__.fetch(struct_res, env, res)
catch return beam.raise_resource_error(env);
var new_last_name = beam.allocator.alloc(u8, update_value.len)
catch return beam.raise_enomem(env);
errdefer beam.allocator.free(new_last_name);
std.mem.copy(u8, new_last_name, update_value);
defer beam.allocator.free(old_value.last_name);
var new_struct = .{
.first_name = old_value.first_name,
.last_name = new_last_name,
};
__resource__.update(struct_res, env, res, new_struct)
catch return beam.raise_resource_error(env);
return beam.make_ok(env);
}
// note that updating POINTER structs does not require calling update.
/// nif: update_struct_ptr_fn/2
fn update_struct_ptr_fn(env: beam.env, res: beam.term, update_value: []u8) beam.term {
var struct_ptr = __resource__.fetch(struct_ptr_res, env, res)
catch return beam.raise_resource_error(env);
beam.allocator.free(struct_ptr.first_name);
struct_ptr.first_name = beam.allocator.alloc(u8, update_value.len)
catch return beam.raise_enomem(env);
errdefer beam.allocator.free(struct_ptr.first_name);
std.mem.copy(u8, struct_ptr.first_name, update_value);
return beam.make_ok(env);
}
/// nif: update_struct_ptr_ln/2
fn update_struct_ptr_ln(env: beam.env, res: beam.term, update_value: []u8) beam.term {
var struct_ptr = __resource__.fetch(struct_ptr_res, env, res)
catch return beam.raise_resource_error(env);
beam.allocator.free(struct_ptr.last_name);
struct_ptr.last_name = beam.allocator.alloc(u8, update_value.len)
catch return beam.raise_enomem(env);
errdefer beam.allocator.free(struct_ptr.last_name);
std.mem.copy(u8, struct_ptr.last_name, update_value);
return beam.make_ok(env);
}
"""
test "resources can be updated" do
i64_res_example = create_i64()
assert :ok == update_i64(i64_res_example, 50)
assert 50 == fetch_i64(i64_res_example)
struct_res_example = create_struct("john", "doe")
assert :ok == update_struct_fn(struct_res_example, "jane")
assert "jane" == fetch_struct_fn(struct_res_example)
assert "doe" == fetch_struct_ln(struct_res_example)
assert :ok == update_struct_ln(struct_res_example, "smith")
assert "jane" == fetch_struct_fn(struct_res_example)
assert "smith" == fetch_struct_ln(struct_res_example)
struct_res_example = create_struct_ptr("john", "doe")
assert :ok == update_struct_ptr_fn(struct_res_example, "jane")
assert "jane" == fetch_struct_ptr_fn(struct_res_example)
assert "doe" == fetch_struct_ptr_ln(struct_res_example)
assert :ok == update_struct_ptr_ln(struct_res_example, "smith")
assert "jane" == fetch_struct_ptr_fn(struct_res_example)
assert "smith" == fetch_struct_ptr_ln(struct_res_example)
end
Keep and Release
If you are using resources in threaded or yielding long-running nifs, you should
perform keep
and release
operations on these resources. This will prevent the
garbage collector from triggering cleanup on your resource in the event that
the calling process has been terminated. If you don't perform keep/release
on the resource, it will at some point result in a segfault due to use-after-free
and it may be difficult to detect because triggering segfault may depend on
race conditions, VM computational load, and memory page layout.
keep
is failable and takes out a gc reference to your resource; it should always
be matched by a release
call (typically with a defer
statement). Release
is not failable.
Both functions take three parameters, the type of the internal data structure, the calling function environment, and the resource term.
Example: Keep and Release
~Z"""
/// nif: long_running_i64/1 threaded
fn long_running_i64(env: beam.env, res: beam.term) beam.term {
__resource__.keep(i64_res, env, res)
catch return beam.raise_resource_error(env);
defer __resource__.release(i64_res, env, res);
var int_value = __resource__.fetch(i64_res, env, res)
catch return beam.raise_resource_error(env);
return beam.make_i64(env, int_value);
}
"""
test "keep and release" do
i64_res_example = create_i64()
assert 47 == long_running_i64(i64_res_example)
end
Cleanup
Most resources should define a cleanup function. This function will be
invoked when the last reference to that resource has been garbage collected
by the virtual machine. The cleanup function is declared by using the
/// resource: <name> cleanup
, this should be followed by a void
function taking a pointer to the resource. If there are any internal data
that need freeing, you should do it here.
Important Note
- The pointer for the resource itself will be cleaned up by Zigler, so don't free that.
- If your resource is a struct pointer, DO clean that up.
Example: Cleanup
~Z"""
/// resource: struct_res cleanup
fn struct_res_cleanup(env: beam.env, resource: *struct_res) void {
beam.allocator.free(resource.first_name);
beam.allocator.free(resource.last_name);
// don't clean up the resource pointer itself.
}
/// resource: struct_res_ptr cleanup
fn struct_res_ptr_cleanup(env: beam.env, resource: *struct_res_ptr) void {
beam.allocator.free(resource.first_name);
beam.allocator.free(resource.last_name);
// DO clean up the pointer in the resource.
beam.allocator.destroy(resource.*);
// don't clean up the resource pointer (pointer).
}
"""