Ash Neo4j DataLayer

Mix.install(
  [
    {:ash_neo4j, "~> 0.5.0"}
  ],
  consolidate_protocols: false
)

Overview

If already familiar with the Ash Framework in this livebook tutorial you will learn:

• Why have an Ash Neo4j DataLayer?
• Brief Introduction to Neo4j
• Installing Neo4j and configuring Bolty
• How to persist your Ash Resources with Neo4j
• How your Ash Resources and their relationships are stored in Neo4j

Why have an Ash Neo4j DataLayer?

I've been using developing systems for about 30 years, often working with relational databases. I've been developing with graph databases for the last five years. I've found graph aligns naturally with the way I think about things and how they are related. Graphistas see graphs everywhere, because they are everywhere. Graph is similar to Elixir, in that once you start you don't want to go back. In fact life is too short, I won't go back...

Similarly I've found Ash to be an exceptional framework for declarative application development. It has a learning curve, however the Ash way is to use and build abstractions for exceptional maintainability.

The Ash Framework contains some basic datalayers and the ecosystem already offers a choice of datalayer extensions, including the popular and feature rich AshPostgres above the open source Postgres.

I hitched my wagon to Ash Framework aware of the lack of graph datalayers, knowing that they could be built using an Ash Extension. Indeed Ash does a good job of abstracting SQL databases away and making refactorings easier, so arguably there is less demand for such a thing. However as you refactor your resources you still need to migrate database schemas - so the complexity doesn't go away, it is just better hidden.

Neo4j is defacto in data science and well entrenched in security, fraud, logistics, network management and recommendation applications. Increasingly our applications need to work with both facts and knowledge and these are best represented using graphs.

So here we both are with Ash and graph. Enjoy the combination.

Brief Introduction to Neo4j

Neo4j is a leading Graph Based Database. It is schemaless, and your data is modelled naturally using nodes and relationships. Nodes can have properties and be involved in any number of relationships. Relationships are directed and relate exactly two nodes. Relationships are also named and may themselves have properties.

Cypher (also known as Cypher Query Language CQL) is the native language of Neo4j, and was originally modelled after SQL.

We can create an Actor node CREATE (s:Actor {name: 'Bill Nighy'}) and a Movie node CREATE (d:Movie {title: 'Love Actually'})

We can relate them with the cypher MATCH (s:Actor {name: 'Bill Nighy'}), (d:Movie {title: 'Love Actually'}) MERGE (s) -[r:ACTED_IN]-> (d)

Cypher is not meant as a graphical language, but has an ASCII art influence and is fairly expressive in that (s) -[r:ACTED_IN]-> (d), which mirrors how these nodes could look related in a browser. Cypher influenced international standard GQL and is being aligned, however to maintain back compatibility with Neo4j 4.x we use Cypher.

We connect to the Neo4j database using the Bolt protocol. Ash Neo4j uses bolty which is a fork of the inactive bolt-sips.

Installing Neo4j and Configuring Bolty

While Neo4j community edition is open source and you can build from source it is likely that you'll use an installation.

AshNeo4j uses neo4j which must be installed and running. You can install latest major Neo4j versions from the community tab at Neo4j Deployment Center, or use the 5.26.10 direct link

When you install neo4j you'll typically have a default username and password. Take note of this and any other non-standard config.

Update the configuration below as necessary and evaluate.

config = [
  uri: "bolt://localhost:7687",
  auth: [username: "neo4j", password: "password"],
  user_agent: "ashNeo4jLivebook/1",
  pool_size: 15,
  max_overflow: 3,
  prefix: :default,
  name: Bolt,
  log: false,
  log_hex: false
]

Bolty needs a process in your supervision tree, this will start one with the config if not already running:

AshNeo4j.BoltyHelper.start(config)

Now you should be able to verify that Ash Neo4j is connected to Neo4j using Bolty:

AshNeo4j.BoltyHelper.is_connected()

You can get all nodes related to other nodes the following query:

AshNeo4j.Cypher.run("MATCH (n1)-[r]->(n2) RETURN r, n1, n2 LIMIT 25")

It is helpful to have a Neo4j browser open locally, typically:

http://localhost:7474/browser/

Once you connect and issue a query like the one above you'll be able to explore the results interactively.

OPTIONAL If you want to clear your database you can evaluate:

AshNeo4j.Neo4jHelper.delete_all()

How to persist your Ash Resources with Neo4j

First we define a minimal Domain to hold our Ash Resources:

defmodule Demo do
  @moduledoc false
  use Ash.Domain

  resources do
    allow_unregistered? true
  end
end

We'll start by defining an Ash Resource for a Specification. We intend to have a Specification instance which SPECIFIES either Service or Resource instances:

For the Specification resource we simply declare the data_layer when using Ash.Resource. Usually we'll need to do some minor configuration with a 'neo4j do' declaration block:

defmodule Specification do
  use Ash.Resource,
    domain: Demo,
    data_layer: AshNeo4j.DataLayer

  actions do
    defaults [:read, :destroy, create: :*, update: :*]
  end

  attributes do
    uuid_primary_key :id
    attribute :href, :ci_string, constraints: [casing: :lower], public?: true
    attribute :name, :string, public?: true
    attribute :type, :atom, constraints: [one_of: [:service, :resource]], public?: true
    attribute :major_version, :integer, default: 1, public?: true
    attribute :minor_version, :integer, default: 0, public?: true
  end
end

Now we create some service and resource specification instances:

broadband_v1 = Specification |> Ash.create!(%{name: "broadband"})
nbn_ethernet_v1 = Specification |> Ash.create!(%{name: "nbnEthernet", type: :resource})
mobile_backup_v2 = Specification |> Ash.create!(%{name: "mobileBackup", major_version: 2, minor_version: 3})
esim_v1 = Specification |> Ash.create!(%{name: "esim", type: :resource})
specifications = [broadband_v1, mobile_backup_v2, nbn_ethernet_v1, esim_v1]

Next we define an Ash Resource for Resource, using the AshNeo4j datalayer. A Resource is specified by a Specification, can use other Resources, and also belong to a Service. We can use the default relates for Resource belongs to Service.

defmodule Resource do
  @moduledoc false
  use Ash.Resource,
    domain: Demo,
    data_layer: AshNeo4j.DataLayer

  neo4j do
    relate [
      {:specification, :SPECIFIES, :incoming, :Specification},
      {:resources, :ASSIGNED_TO, :incoming, :Resource},
      {:resource, :ASSIGNED_TO, :outgoing, :Resource}
    ]
  end

  actions do
    defaults [:destroy]

    read :read do
      primary? true
    end

    create :create do
      primary? true
      accept [:name]
      argument :specified_by, :uuid

      change manage_relationship(:specified_by, :specification, type: :append_and_remove)
    end

    update :update do
      primary? true
      require_atomic? false
      accept [:state, :status]
      argument :use_resources, {:array, :uuid}

      change manage_relationship(:use_resources, :resources, type: :append_and_remove)
    end
  end

  attributes do
    uuid_primary_key :id
    attribute :name, :string, public?: true
    attribute :state, :atom, public?: true
    attribute :status, :atom, public?: true
  end

  relationships do
    belongs_to :specification, Specification, public?: true
    belongs_to :service, Service, public?: true
    belongs_to :resource, Resource, public?: true
    has_many :resources, Resource
  end
end

We also need to declare a Service, also defined by a Specification and can manage child Services and be assigned Resources. Similarly we can use the default relates for Service belongs to Service.

defmodule Service do
  @moduledoc false
  use Ash.Resource,
    domain: Demo,
    data_layer: AshNeo4j.DataLayer

  neo4j do
    label :InternalService

    relate [
      {:specification, :SPECIFIES, :incoming, :Specification},
      {:services, :BELONGS_TO, :incoming, :Service},
      {:resources, :IS_ASSIGNED, :incoming, :Resource},
      {:resource, :IS_ASSIGNED, :outgoing, :Resource}
    ]

    guard [
      {:SPECIFIES, :incoming, :Specification}
    ]
  end

  actions do
    defaults [:destroy]

    read :read do
      primary? true
    end

    create :create do
      primary? true
      accept [:name]
      argument :specified_by, :uuid

      change manage_relationship(:specified_by, :specification, type: :append_and_remove)
    end

    update :update do
      primary? true
      require_atomic? false
      argument :manage_services, {:array, :uuid}
      argument :use_resources, {:array, :uuid}

      change manage_relationship(:manage_services, :services, type: :append_and_remove)
      change manage_relationship(:use_resources, :resources, type: :append_and_remove)
    end
  end

  attributes do
    uuid_primary_key :id
    attribute :name, :string, public?: true
    attribute :state, :atom, public?: true
    attribute :status, :atom, public?: true
  end

  relationships do
    belongs_to :specification, Specification, public?: true
    belongs_to :service, Service, public?: true
    has_many :services, Service, public?: true
    has_many :resources, Resource, public?: true
    belongs_to :resource, Resource, public?: true
  end
end

Now we'll create some resource instances, these will be related to the specifications we created earlier:

esim_0001 = Resource |> Ash.create!(%{name: "esim_0001", specified_by: esim_v1.id})
nbn_ethernet_0001 = Resource |> Ash.create!(%{name: "nbnEthernet_0001", specified_by: nbn_ethernet_v1.id})
resources = [esim_0001, nbn_ethernet_0001]

Now we create some services instances, also specified_by specifications.

broadband_0001 = Service |> Ash.create!(%{name: "broadband_0001", specified_by: broadband_v1.id})
mobile_backup_0001 = Service |> Ash.create!(%{name: "mobileBackup_0001", specified_by: mobile_backup_v2.id})
services = [broadband_0001, mobile_backup_0001]

Now we relate the services and resources with each other, such that the broadband service uses the nbnEthernet resource, and manages the mobileBackup service which has the esim resource:

mobile_backup_0001 |> Ash.update!(%{use_resources: [esim_0001.id]})
broadband_0001 
|> Ash.update!(%{manage_services: [mobile_backup_0001.id]})
|> Ash.update!(%{use_resources: [nbn_ethernet_0001.id]})

How your Ash Resources and their relationships are stored in Neo4j

Now check your Neo4j browser and you should see the full broadband service and resource hierarchy.

Explore the nodes and relationships. We strictly enforce Neo4j naming conventions at resource compile time such that:

• node labels are PascalCase
• node properties are camelCase
• relationships are UPPER_CASE

Labels

By default AshNeo4j labels each node with the rightmost Module.split of the Ash Resource module name. The label can be overridden using the DSL label to provide an atom, such as label :InternalService. All labels are verified to be PascalCase at compile time.

Properties

By default AshNeo4j translates snake_case attribute names to camelCase Neo4j property names. Neo4j reserves id as an internal node property, so AshNeo4j translates the :id attribute to the camelCased short name of the type — for example an :id attribute of :uuid type is translated to the uuid node property.

AshNeo4j also supports the source: field in the attribute DSL — if present it is used directly as the Neo4j property name.

Relationships

We use relate to direct relationships where we don't want AshNeo4j defaults. For the Service above:

relate [
  {:specification, :SPECIFIES, :incoming, :Specification},
  {:services, :BELONGS_TO, :incoming, :Service},
  {:resources, :IS_ASSIGNED, :incoming, :Resource},
  {:resource, :IS_ASSIGNED, :outgoing, :Resource}
]

This directs the Service's belongs_to :specification relationship to the semantic Cypher (:Specification)-[:SPECIFIES]->(:InternalService).

Default relate clauses are always :outgoing from the source resource, and the edge label is derived from the Ash relationship type.

Guard

We use guard to prevent destruction of a node while relationships exist. In the Service above:

guard [
  {:SPECIFIES, :incoming, :Specification}
]

This prevents a Specification from being destroyed while it SPECIFIES a Service. Guard is useful where a resource has no explicit relationships but other resources depend on it existing.

Skip

We use skip to avoid persisting certain attributes as node properties. This is useful for transient attributes or attributes you want to default using the resource but not store explicitly:

neo4j do
  skip [:transient_field]
end

Types and Storage

AshNeo4j handles the mapping between Ash/Elixir types and Neo4j property storage. Where possible it uses native Neo4j types — string, integer, float, boolean, and temporal types. For complex types it uses JSON-encoded strings. For raw binary types it uses Base64-encoded strings.

Key points:

• Atoms are stored as strings — :groodle is stored as "groodle"
• UUID types (Ash.Type.UUID, Ash.Type.UUIDv7) are stored as their printable string form — "0274972c-161c-4dc9-882f-6851704c2af9"
• Complex types (maps, structs, tuples, keywords, unions, embedded resources, Ash.Type.NewType, Ash.TypedStruct) are JSON-encoded — {"name":"Henry","age":8,"breed":"groodle"}
• Binary types (Ash.Type.Binary, Ash.Type.UrlEncodedBinary) are Base64-encoded — "AQID"
• Temporal types (date, time, datetime, naive_datetime, duration) use native Neo4j types via Bolty PackStream
• CiString (Ash.Type.CiString) is stored as a plain string using the casing applied during cast_input. Queries against CiString values use toLower() on both sides in Cypher since Neo4j has no native case-insensitive string type
• Arrays use Neo4j native arrays — each element is individually encoded using the rules above
• nil values are not persisted as properties — they are simply absent from the node, and removed on update to nil

Interestingly many Ash types have identical JSON representations (Map, Struct, TypedStruct, Tuple, Keyword). This means schema evolution between these types requires no data migration — you can change the Elixir type of an attribute without touching Neo4j.

JSON encoding uses AshNeo4j.Util.json_encode, which converts structs to plain maps and sorts keys for consistency. It deliberately avoids calling Jason.Encoder on structs, so you are free to implement Jason.Encoder (possibly via ash_jason) for presentation or API concerns without affecting storage.

For a complete type reference see the AshNeo4j README.

What Next?

In this tutorial you've used the AshNeo4j DataLayer to persist some Ash Resources.

If you are already using Ash you may enjoy the simplicity of a schemaless DataLayer and the natural beauty of graphs.

If you find Ash Neo4j useful please visit and star on github. Feel free to join discussions and raise issues to discuss PRs.