Mix.install(
[
{:diffo, "~> 0.3.0"}
],
config: [
diffo: [ash_domains: [Diffo.Provider]]
],
consolidate_protocols: false
)Overview
Diffo is a Telecommunications Management Forum (TMF) Service and Resource Manager, built for autonomous networks.
It is implemented using the Ash Framework leveraging core and community extensions including some created and maintained by diffo-dev. As such it is highly customizable using Spark DSL and as necessary Elixir. If you are not already familiar with Ash then please explore Ash Get Started
First ensure you've explored the Diffo Livebook for an introduction to Diffo:
In this livebook you will learn about:
- TMF Services and Resources
- Building your own Domain
- Declaring Instance resources with the unified
provider doDSL - Using the Assigner for partial resource allocation and assignment
- Composing a Resource from partially assigned Resources
- Declaring Party kinds with
provider do - Declaring Place kinds with
provider do
Installing Neo4j and Configuring Bolty
Diffo uses the Ash Neo4j DataLayer, which requires Neo4j to be installed.
While Neo4j community edition is open source and you can build from source it is likely that you'll use an installation.
AshNeo4j uses neo4j which must be installed and running. You can install latest major Neo4j versions from the community tab at Neo4j Deployment Center, or use the 5.26.8 direct link
When you install neo4j you'll typically have a default username and password. Take note of this and any other non-standard config.
Update the configuration below as necessary and evaluate.
config = [
uri: "bolt://localhost:7687",
auth: [username: "neo4j", password: "password"],
user_agent: "diffoLivebook/1",
pool_size: 15,
max_overflow: 3,
prefix: :default,
name: Bolt,
log: false,
log_hex: false
]Bolty needs a process in your supervision tree, this will start one with the config if not already running:
AshNeo4j.BoltyHelper.start(config)Now you should be able to verify that Neo4j is running:
AshNeo4j.BoltyHelper.is_connected()You can get all nodes related to other nodes the following query:
AshNeo4j.Cypher.run("MATCH (n1)-[r]->(n2) RETURN r, n1, n2 LIMIT 50")It is helpful to have a Neo4j browser open locally, typically:
http://localhost:7474/browser/
Once you connect and issue a query like the one above you'll be able to explore the results interactively.
OPTIONAL If you want to clear your database you can evaluate:
AshNeo4j.Neo4jHelper.delete_all()TMF Services and Resources
TMF Services are network services with industry standard structure and API that are operated for you by a Provider Entity. Ideally TMF Services are as abstract as possible, such that the Consumer specifies their intent (often by selecting a service from a catalog and providing minimal configuration of features and/or characteristics) allowing the provider to deliver the service as best it sees fit. This is powerful as it allows the service to perform advanced uses cases, like move, technology change, and allow the provider to optimise and even dynamically recompose the service. TMF Resources are generally a network resource that needs to be assigned to provide a service. They are generally too low level to have value on their own and where possible are entirely hidden from the product layer.
TMF Services are generally composed of services and/or resources. TMF Resources can also be composed of resources (but not services).
TMF Services and Resources are similar in that they each have a Specification, and are defined by Features and Characteristics. They also can have outgoing relationships with other services and resources, indeed this is fundamental to composition and in particular resource assignment.
Resources are generally created/managed/owned by a Provider, and assigned to a Consumer. Often the assignment is effectively a lease during which period the consumer has exclusive use of the resource under the provider's conditions, effectively 'owning' the resource.
When a Provider creates a pool of resources this is known as 'allocation'. For instance a VLAN pool may contain VLAN ID's 0..4095, and perhaps a new pool is inherently allocated with either a new interface, or the creation of a logical L2 VLAN domain.
When a Consumer is leased a resource this is assignment.
Assignment is effectively a request for a relationship from a Provider Resource 'back up' to a Consumer Service or Resource. There are different variants on this:
- Specific Resource assignment - the specific resource requested by the Consumer is assigned
- 'To specification' Resource assignment - an entire resource is assigned by the Provider, allocation may be 'just in time'
- Partial Resource assignment - a partial resource is assigned by the Provider, the consumer is aware of the 'pool resource'.
- Specific partial resource assignment - a partial resource requested by the Consumer is assigned
In all cases the assignment is only successful if the Provider allows the requested relationship to occur from it back to the Consumer.
Partial resource assignment uses a relationship characteristic to indicate which part of the resource is optionally requested and ultimately assigned.
Provider Extension
Diffo.Provider.BaseInstance is an Ash Resource Fragment for domain-specific Instance kinds
(services and resources). It provides a rich set of base attributes — id, href, name,
type, state and more — plus the unified Diffo.Provider.Extension DSL.
The extension provides a single provider do section containing everything needed to
describe and wire an Instance kind. Declarations are baked into the module at compile time
and introspectable at runtime via generated functions (specification/0, characteristics/0,
features/0, parties/0, places/0) and Diffo.Provider.Extension.Info.
The provider do section contains:
specification do — the TMF Specification (id, name, type, version, description, category).
The id is a stable UUID4, the same in every environment for this Instance kind.
characteristics do — typed value slots backed by Diffo.Provider.BaseCharacteristic-derived resources.
features do — optional capabilities with their own typed characteristic payload.
pools do — assignable pools for partial resource allocation. Each pool :name, :thing declaration creates an AssignableCharacteristic node during build and generates pools/0 / pool/1 on the module. Pool bounds (first, last, algorithm, assignable_type) are set in a :define action via Pool.update_pools/3. Assignment actions use Assigner.assign/3 — the thing name is looked up from the pool declaration.
parties do — party roles: party (singular), parties (plural), party_ref (reference, no direct edge).
places do — place roles: place (singular), places (plural), place_ref (reference).
behaviour do — declares which Ash create actions to wire for build lifecycle management.
Declaring create :name injects :specified_by, :features, and :characteristics
arguments automatically onto that action.
Each characteristic is a dedicated Ash resource using the Diffo.Provider.BaseCharacteristic fragment. It carries direct typed attributes and a :value calculation that builds a companion <Module>.Value TypedStruct for ordered JSON encoding. The TypedStruct uses AshJason.TypedStruct to control field order in the JSON output.
For partial resource allocation and assignment we've created Diffo.Provider.Assigner. The host resource declares a pools do section listing each assignable pool by name and the kind of thing being assigned. Pool bounds (first/last value, algorithm) are set via a :define action. Each assignment is stored as a Diffo.Provider.AssignedToRelationship node (Neo4j label :AssignmentRelationship) carrying pool, thing, and the assigned value. These are distinct from regular TMF Diffo.Provider.Relationship nodes and are accessible on an instance via instance.assignments.
Let's imagine a Compute domain which operates GPU and NPU resources. We want to expose a Cluster composite resource which can be dynamically composed of a number of GPU and NPU cores.
Each instance of Cluster could be created on Consumer demand as a 'container' for the GPU and NPU core partial resources.
Each of the GPU and NPU Resource instances is created and managed by the Provider and is effectively a resource pool for individually assignable cores.
We'll define all the resources first, then declare the Diffo.Compute domain once they are all compiled — Ash validates code_interface at domain compile time so all referenced resources must exist first.
Declaring a Composite Resource
We will start by declaring the Cluster Resource. It is going to be a composite resource, where it can be assigned individual GPU and NPU cores via resource relationships. It is an Ash.Resource incorporating the Diffo.Provider.BaseInstance fragment.
First we define the ClusterCharacteristic typed resource and its companion Value TypedStruct:
defmodule Diffo.Compute.ClusterCharacteristic do
@moduledoc "Typed characteristic carrying cluster composition fields."
use Ash.Resource,
fragments: [Diffo.Provider.BaseCharacteristic],
domain: Diffo.Compute
resource do
plural_name :cluster_characteristics
end
attributes do
attribute :gpu_cores, :integer, public?: true, default: 0, constraints: [min: 0]
attribute :npu_cores, :integer, public?: true, default: 0, constraints: [min: 0]
end
calculations do
calculate :value, Diffo.Type.CharacteristicValue,
Diffo.Provider.Calculations.CharacteristicValue do
public? true
end
end
actions do
create :create do
accept [:name, :gpu_cores, :npu_cores]
argument :instance_id, :uuid
argument :feature_id, :uuid
change manage_relationship(:instance_id, :instance, type: :append)
change manage_relationship(:feature_id, :feature, type: :append)
end
update :update do
accept [:gpu_cores, :npu_cores]
end
end
preparations do
prepare build(load: [:value])
end
jason do
pick [:name, :value]
compact true
end
end
defmodule Diffo.Compute.ClusterCharacteristic.Value do
@moduledoc "Value struct for ClusterCharacteristic — controls JSON field order."
use Ash.TypedStruct, extensions: [AshJason.TypedStruct]
typed_struct do
field :gpu_cores, :integer
field :npu_cores, :integer
end
jason do
pick [:gpu_cores, :npu_cores]
compact true
end
endNow the Cluster resource itself. It declares ClusterCharacteristic as the :cluster characteristic — updates to it are made directly on the characteristic resource, so no update :define is needed here:
defmodule Diffo.Compute.Cluster do
@moduledoc """
Cluster Resource Instance
"""
alias Diffo.Provider.BaseInstance
alias Diffo.Provider.Instance.Relationship
alias Diffo.Compute
alias Diffo.Compute.ClusterCharacteristic
alias Diffo.Compute.Tenant
alias Diffo.Compute.Engineer
use Ash.Resource,
fragments: [BaseInstance],
domain: Compute
resource do
description "An Ash Resource representing a Cluster"
plural_name :Clusters
end
provider do
specification do
id "4bcfc4c9-e776-4878-a658-e8d81857bed7"
name "cluster"
type :resourceSpecification
description "A Cluster Resource Instance"
category "Network Resource"
end
characteristics do
characteristic :cluster, ClusterCharacteristic
end
parties do
party :operator, Tenant
party :manager, Engineer
end
places do
place :data_centre, Diffo.Compute.DataCentre
end
behaviour do
actions do
create :build
end
end
end
actions do
create :build do
description "creates a new Cluster resource instance for build"
accept [:id, :name, :type, :which]
argument :relationships, {:array, :struct}
argument :places, {:array, :struct}
argument :parties, {:array, :struct}
change set_attribute(:type, :resource)
change load [:href]
upsert? false
end
update :relate do
description "relates the cluster with other instances"
argument :relationships, {:array, :struct}
change after_action(fn changeset, result, _context ->
with {:ok, _cluster} <- Relationship.relate_instance(result, changeset),
{:ok, cluster} <- Compute.get_cluster_by_id(result.id),
do: {:ok, cluster}
end)
end
end
endUsing the Assigner
We'll now define a GPU Resource which uses the Diffo.Provider.Assigner functionality.
First define the GpuCharacteristic typed resource and its Value TypedStruct:
defmodule Diffo.Compute.GpuCharacteristic do
@moduledoc "Typed characteristic carrying GPU identity fields."
use Ash.Resource,
fragments: [Diffo.Provider.BaseCharacteristic],
domain: Diffo.Compute
resource do
plural_name :gpu_characteristics
end
attributes do
attribute :family, :atom, public?: true, description: "the GPU family name"
attribute :model, :string, public?: true, description: "the GPU model name"
attribute :technology, :atom, public?: true, description: "the GPU technology"
end
calculations do
calculate :value, Diffo.Type.CharacteristicValue,
Diffo.Provider.Calculations.CharacteristicValue do
public? true
end
end
actions do
create :create do
accept [:name, :family, :model, :technology]
argument :instance_id, :uuid
argument :feature_id, :uuid
change manage_relationship(:instance_id, :instance, type: :append)
change manage_relationship(:feature_id, :feature, type: :append)
end
update :update do
accept [:family, :model, :technology]
end
end
preparations do
prepare build(load: [:value])
end
jason do
pick [:name, :value]
compact true
end
end
defmodule Diffo.Compute.GpuCharacteristic.Value do
@moduledoc "Value struct for GpuCharacteristic — controls JSON field order."
use Ash.TypedStruct, extensions: [AshJason.TypedStruct]
typed_struct do
field :family, :atom
field :model, :string
field :technology, :atom
end
jason do
pick [:family, :model, :technology]
compact true
end
endThe GPU resource declares GpuCharacteristic for the typed :gpu slot and uses pools do to declare the :cores assignable pool. The update :define action updates both the typed characteristic and the pool bounds. The update :assign_core action uses Assigner.assign/3 — the thing name (:core) is looked up from the pool declaration automatically:
defmodule Diffo.Compute.GPU do
@moduledoc """
GPU Resource Instance
"""
alias Diffo.Provider.BaseInstance
alias Diffo.Provider.Instance.Relationship
alias Diffo.Provider.Extension.Characteristic
alias Diffo.Provider.Extension.Pool
alias Diffo.Provider.Assigner
alias Diffo.Provider.Assignment
alias Diffo.Compute
alias Diffo.Compute.GpuCharacteristic
use Ash.Resource,
fragments: [BaseInstance],
domain: Compute
resource do
description "An Ash Resource representing a GPU"
plural_name :gpus
end
provider do
specification do
id "ad50073f-17e0-45cb-b9b1-aa4296876156"
name "gpu"
type :resourceSpecification
description "A GPU Resource Instance"
category "Network Resource"
end
characteristics do
characteristic :gpu, GpuCharacteristic
end
pools do
pool :cores, :core
end
behaviour do
actions do
create :build
end
end
end
actions do
create :build do
description "creates a new GPU resource instance for build"
accept [:id, :name, :type, :which]
argument :relationships, {:array, :struct}
argument :places, {:array, :struct}
argument :parties, {:array, :struct}
change set_attribute(:type, :resource)
change load [:href]
upsert? false
end
update :define do
description "sets GPU identity and allocates the cores pool"
argument :characteristic_value_updates, {:array, :term}
change after_action(fn changeset, result, _context ->
with {:ok, result} <-
Characteristic.update_all(result, changeset, characteristics()),
{:ok, result} <- Pool.update_pools(result, changeset, pools()),
{:ok, result} <- Compute.get_gpu_by_id(result.id),
do: {:ok, result}
end)
end
update :relate do
description "relates the GPU with other instances"
argument :relationships, {:array, :struct}
change after_action(fn changeset, result, _context ->
with {:ok, result} <- Relationship.relate_instance(result, changeset),
{:ok, result} <- Compute.get_gpu_by_id(result.id),
do: {:ok, result}
end)
end
update :assign_core do
description "assigns a core from this GPU to another instance"
argument :assignment, :struct, constraints: [instance_of: Assignment]
change after_action(fn changeset, result, _context ->
with {:ok, result} <- Assigner.assign(result, changeset, :cores),
{:ok, result} <- Compute.get_gpu_by_id(result.id),
do: {:ok, result}
end)
end
end
endParty Extension
Diffo.Provider.BaseParty is an Ash Resource Fragment for domain-specific Party kinds, mirroring BaseInstance. It provides common Party attributes — id, href, name, type, referred_type — and the unified Diffo.Provider.Extension DSL. Within provider do, a Party kind uses instances do, parties do, and places do sections to declare the roles it plays.
type defaults to :PartyRef and can be set to :Individual, :Organization, or :Entity. Domain party kinds typically set type in their build action. The id defaults to a generated uuid but can be set to any meaningful string (such as an ABN or a data centre identifier).
The Diffo.Provider.Extension DSL cheat sheet is at DSL-Diffo.Provider.Extension.
Defining Party kinds
We'll add two Party kinds to our Compute domain — Tenant for the operating company, and Engineer for the individuals who manage resources.
defmodule Diffo.Compute.Tenant do
@moduledoc """
Tenant in the Compute domain
"""
alias Diffo.Provider.BaseParty
alias Diffo.Compute
use Ash.Resource,
fragments: [BaseParty],
domain: Compute
resource do
description "A Compute Tenant"
plural_name :tenants
end
actions do
create :build do
accept [:id, :name]
change set_attribute(:type, :Organization)
end
end
provider do
instances do
role :operator, Diffo.Compute.Cluster
role :operator, Diffo.Compute.GPU
end
end
enddefmodule Diffo.Compute.Engineer do
@moduledoc """
Engineer in the Compute domain
"""
alias Diffo.Provider.BaseParty
alias Diffo.Compute
use Ash.Resource,
fragments: [BaseParty],
domain: Compute
resource do
description "A Compute Engineer"
plural_name :engineers
end
actions do
create :build do
accept [:id, :name]
change set_attribute(:type, :Individual)
end
end
provider do
instances do
role :manager, Diffo.Compute.Cluster
end
parties do
role :employer, Diffo.Compute.Tenant
end
end
endPlace Extension
Diffo.Provider.BasePlace is an Ash Resource Fragment for domain-specific Place kinds, mirroring BaseInstance and BaseParty. It provides common Place attributes — id, href, name, type, referred_type — and the unified Diffo.Provider.Extension DSL. Within provider do, a Place kind uses instances do, parties do, and places do sections to declare the roles it plays.
type defaults to :PlaceRef and is typically set in the build action to the concrete place type (:GeographicSite, :GeographicLocation, or :GeographicAddress). When referred_type is present, type must be :PlaceRef — meaning this Place is a reference rather than a physical location.
Defining Place kinds
We'll add a DataCentre Place kind to our Compute domain. Clusters are hosted at a data centre; the instances do block records that relationship from the DataCentre's perspective.
defmodule Diffo.Compute.DataCentre do
@moduledoc """
DataCentre in the Compute domain
"""
alias Diffo.Provider.BasePlace
alias Diffo.Compute
use Ash.Resource,
fragments: [BasePlace],
domain: Compute
resource do
description "A Compute Data Centre"
plural_name :data_centres
end
jason do
pick [:id, :href, :name, :type]
compact true
rename type: "@type"
end
outstanding do
expect [:id, :name, :type]
end
actions do
create :build do
accept [:id, :href, :name]
change set_attribute(:type, :GeographicSite)
end
end
provider do
instances do
role :data_centre, Diffo.Compute.Cluster
role :data_centre, Diffo.Compute.GPU
end
end
endCompute Domain
With all resources defined we can now declare the Diffo.Compute domain, which exposes a typed API for each resource:
defmodule Diffo.Compute do
@moduledoc """
Compute - example domain
"""
use Ash.Domain,
otp_app: :diffo,
validate_config_inclusion?: false
alias Diffo.Compute.GPU
alias Diffo.Compute.GpuCharacteristic
#alias Diffo.Compute.NPU
alias Diffo.Compute.Cluster
alias Diffo.Compute.ClusterCharacteristic
alias Diffo.Compute.Tenant
alias Diffo.Compute.Engineer
alias Diffo.Compute.DataCentre
resources do
resource GPU do
define :get_gpu_by_id, action: :read, get_by: :id
define :build_gpu, action: :build
define :define_gpu, action: :define
define :relate_gpu, action: :relate
define :assign_gpu_core, action: :assign_core
end
resource GpuCharacteristic do
define :update_gpu_characteristic, action: :update
end
#resource NPU do
#define :get_npu_by_id, action: :read, get_by: :id
#define :build_npu, action: :build
#define :define_npu, action: :define
#define :relate_npu, action: :relate
#define :assign_npu_core, action: :assign_core
#end
resource Cluster do
define :get_cluster_by_id, action: :read, get_by: :id
define :build_cluster, action: :build
define :relate_cluster, action: :relate
end
resource ClusterCharacteristic do
define :update_cluster_characteristic, action: :update
end
resource Tenant do
define :create_tenant, action: :build
define :get_tenant_by_id, action: :read, get_by: :id
define :list_tenants, action: :read
end
resource Engineer do
define :create_engineer, action: :build
define :get_engineer_by_id, action: :read, get_by: :id
define :list_engineers, action: :read
end
resource DataCentre do
define :create_data_centre, action: :build
define :get_data_centre_by_id, action: :read, get_by: :id
end
end
endCreating Party instances
Clear any data from previous runs before starting (safe to re-evaluate):
AshNeo4j.Neo4jHelper.delete_all()Now the domain is defined we'll create our Tenant and Engineer first — we'll need them when building Cluster instances. The id for the Tenant is set to a meaningful string — the company's ABN.
alias Diffo.Compute
alias Diffo.Provider.Instance.Party
{:ok, tenant} = Compute.create_tenant(%{
id: "51824753556",
name: "Acme Compute Pty Ltd"
})
{:ok, engineer} = Compute.create_engineer(%{
name: "Alice Zhang"
})Creating a Cluster
First we create the data centre — our DataCentre resource uses BasePlace, so it is managed via the Compute domain API like any other domain resource:
alias Diffo.Provider.Instance.Place
{:ok, dc} = Compute.create_data_centre(%{id: "NXTM2", name: "NextDC M2"})Now build the cluster, passing the data centre as a place and our party members by id and role:
places = [%Place{id: dc.id, role: :data_centre}]
parties = [
%Party{id: tenant.id, role: :operator},
%Party{id: engineer.id, role: :manager}
]
cluster_1 = Diffo.Compute.build_cluster!(%{name: "cluster_1", places: places, parties: parties})Jason.encode!(cluster_1, pretty: true) |> IO.putsUsing the Assigner
Now we'll create a couple of GPU instances:
gpu_1 = Compute.build_gpu!(%{name: "GPU 1"})
gpu_2 = Compute.build_gpu!(%{name: "GPU 2"})We define each GPU: setting its typed :gpu characteristic fields and allocating the :cores pool bounds. Both are passed via characteristic_value_updates to the :define action — Characteristic.update_all handles the typed :gpu update and Pool.update_pools handles the :cores pool bounds:
gpu_attrs = [
gpu: [family: :nvidia, model: "GeForce RTX5090", technology: :blackwell],
cores: [first: 1, last: 680, assignable_type: "tensor"]
]
gpu_1 = Compute.define_gpu!(gpu_1, %{characteristic_value_updates: gpu_attrs})
gpu_2 = Compute.define_gpu!(gpu_2, %{characteristic_value_updates: gpu_attrs})The :cores pool is backed by an AssignableCharacteristic node that records the range bounds and algorithm. Free cores are computed at assignment time from the count of existing AssignmentRelationship records — there is no stored free counter. We can render one as json:
Jason.encode!(gpu_1, pretty: true) |> IO.putsComposing a Resource from partially assigned Resources
Now we can auto-assign GPU cores from each GPU to our cluster_1. We'll assign 3 cores from gpu_1, and one from gpu_2.
alias Diffo.Provider.Assignment
assignment = %{assignment: %Assignment{assignee_id: cluster_1.id, operation: :auto_assign}}
gpu_1 = Compute.assign_gpu_core!(gpu_1, assignment)
gpu_1 = Compute.assign_gpu_core!(gpu_1, assignment)
gpu_1 = Compute.assign_gpu_core!(gpu_1, assignment)
gpu_2 = Compute.assign_gpu_core!(gpu_2, assignment)Now our cluster should have a core from each GPU. Check in the Neo4j browser for :AssignmentRelationship nodes from gpu_1 and gpu_2 to cluster_1. There should be four — each carries pool: :cores, thing: :core, and the assigned integer value (e.g. 1, 2, 3 from gpu_1 and 1 from gpu_2).
The GPU's assignments hold each assignment, showing the assigned core number in the JSON encoding as a resourceRelationshipCharacteristic:
Jason.encode!(gpu_1, pretty: true) |> IO.putsMake sure you have a look at it in the neo4j browser. There should be :AssignmentRelationship nodes from each GPU resource instance to the cluster_1 resource instance, each carrying the assigned core number.
There is no central assignment table — the AssignedToRelationship nodes ARE the assignments. They are separate from the regular :Relationship nodes used for TMF service/resource relationships, and are accessible in Elixir via instance.assignments.
As an exercise, clone the GPU resource to create an NPU resource and assign some NPU cores from it to your cluster. Check that the assigned NPU cores are unique.
What happens when there are none left to assign? What happens when I request a specific assignment from an instance to which the partial resource is already assigned?
What Next?
In this tutorial you've used Diffo's unified provider do extension to define a Compute domain with:
- A composite Cluster resource that receives GPU cores via
Diffo.Provider.Assigner - A GPU resource using
pools doto declare the:coresassignable pool —pool :cores, :corereplaces the oldcharacteristic :cores, AssignableValuepattern - Assignments stored on
Diffo.Provider.AssignedToRelationshipnodes (Neo4j label:AssignmentRelationship, distinct from TMF:Relationshipnodes); accessible viainstance.assignments TenantandEngineerParty kinds declared withprovider dothat express which instances they operate and manage- A
DataCentrePlace kind that declares the instances located at it
BaseParty and BasePlace follow the same provider do pattern as BaseInstance — domain-specific resources use them as fragments, write their own actions for domain-specific attributes, and declare their roles via the unified DSL sections.
The full DSL reference is at DSL-Diffo.Provider.Extension.
If you find Diffo useful please visit and star on github. Feel free to join discussions and raise issues to discuss PR's.