HuggingfaceClient.Serialization (huggingface_client v0.1.0)

HuggingFace Serialization — save and load ML model weights.

Provides helpers for the DDUF (Diffusion model Distributed Unified Format) file format and general model serialization utilities.

DDUF is inspired by GGUF and allows saving an entire diffusion pipeline (VAE, text encoder, UNet/transformer, scheduler configs) in a single file.

See: https://huggingface.co/docs/huggingface_hub/package_reference/serialization

Example — Export a DDUF file

# Export a local model folder as DDUF
:ok = HuggingfaceClient.export_folder_as_dduf(
  "FLUX.1-dev.dduf",
  folder_path: "/path/to/FLUX.1-dev"
)

# Export specific entries (more flexible)
:ok = HuggingfaceClient.export_entries_as_dduf(
  "stable-diffusion.dduf",
  entries: [
    {"model_index.json", "/path/to/model_index.json"},
    {"vae/config.json", "/path/to/vae/config.json"},
    {"vae/diffusion_pytorch_model.safetensors", "/path/to/vae/model.safetensors"},
  ]
)

# Read a DDUF file
{:ok, entries} = HuggingfaceClient.read_dduf_file("FLUX.1-dev.dduf")
{:ok, index}   = HuggingfaceClient.Serialization.DDUF.read_entry(entries, "model_index.json")
IO.inspect(Jason.decode!(index))

Summary

Functions

export_entries_as_dduf(dduf_path, opts)

Exports specific entries as a DDUF file.

export_folder_as_dduf(dduf_path, opts)

Exports a local model folder as a DDUF (Diffusion Unified Format) file.

load_dduf_from_hub(repo_id, opts \\ [])

Downloads a DDUF file from the Hub and opens it.

read_dduf_file(dduf_path)

Reads a DDUF file and returns entry metadata and access handles.

Functions

export_entries_as_dduf(dduf_path, opts)

@spec export_entries_as_dduf(
  String.t(),
  keyword()
) :: :ok | {:error, Exception.t()}

Exports specific entries as a DDUF file.

More flexible than export_folder_as_dduf/2 — lets you pick exactly which files to include and optionally provide content as binaries.

Options

:entries — list of {filename_in_dduf, source} tuples where source is a file path string or binary content (required)

Example

:ok = HuggingfaceClient.export_entries_as_dduf(
  "stable-diffusion-fp16.dduf",
  entries: [
    {"model_index.json", "/path/to/model_index.json"},
    {"vae/config.json", "/path/to/vae/config.json"},
    {"vae/diffusion_pytorch_model.safetensors",
     "/path/to/vae/diffusion_pytorch_model.fp16.safetensors"},
    # Inline binary content
    {"text_encoder/config.json", Jason.encode!(%{model_type: "clip_text_model"})},
  ]
)

export_folder_as_dduf(dduf_path, opts)

@spec export_folder_as_dduf(
  String.t(),
  keyword()
) :: :ok | {:error, Exception.t()}

Exports a local model folder as a DDUF (Diffusion Unified Format) file.

The folder must contain a model_index.json file at its root.

Options

:folder_path — path to the model folder (required)
:allow_patterns — list of glob patterns to include (default: all files)
:ignore_patterns — list of glob patterns to exclude

Example

:ok = HuggingfaceClient.export_folder_as_dduf(
  "FLUX.1-dev.dduf",
  folder_path: "/models/FLUX.1-dev"
)

# Export only safetensors weights (skip optimizer states)
:ok = HuggingfaceClient.export_folder_as_dduf(
  "model-fp16.dduf",
  folder_path: "/models/stable-diffusion",
  allow_patterns: ["*.json", "*.safetensors"],
  ignore_patterns: ["optimizer*", "training_args*"]
)

load_dduf_from_hub(repo_id, opts \\ [])

@spec load_dduf_from_hub(
  String.t(),
  keyword()
) :: {:ok, HuggingfaceClient.Serialization.DDUF.t()} | {:error, Exception.t()}

Downloads a DDUF file from the Hub and opens it.

Example

{:ok, dduf} = HuggingfaceClient.load_dduf_from_hub(
  "black-forest-labs/FLUX.1-dev",
  filename: "FLUX.1-dev.dduf",
  access_token: token
)

read_dduf_file(dduf_path)

@spec read_dduf_file(String.t()) ::
  {:ok, HuggingfaceClient.Serialization.DDUF.t()} | {:error, Exception.t()}

Reads a DDUF file and returns entry metadata and access handles.

Example

{:ok, dduf} = HuggingfaceClient.read_dduf_file("FLUX.1-dev.dduf")
IO.puts("Entries: #{map_size(dduf.entries)}")

# Read model index
{:ok, json_bytes} = HuggingfaceClient.Serialization.DDUF.read_entry(
  dduf, "model_index.json"
)
config = Jason.decode!(json_bytes)
IO.puts("Pipeline class: #{config["_class_name"]}")

# Read VAE weights (returns raw safetensors bytes)
{:ok, weights} = HuggingfaceClient.Serialization.DDUF.read_entry(
  dduf, "vae/diffusion_pytorch_model.safetensors"
)
IO.puts("VAE weights: #{byte_size(weights)} bytes")