ExLLM.Providers.Gemini.Chunk (ex_llm v0.8.1)

View Source

Chunk management for Gemini's Semantic Retrieval API.

A Chunk is a subpart of a Document that is treated as an independent unit for the purposes of vector representation and storage. A Corpus can have a maximum of 1 million Chunks.

Authentication

Chunk operations require authentication. Both API key and OAuth2 are supported:

  • API key: Most operations work with API keys
  • OAuth2: Required for some operations, especially those involving user-specific data

Examples

# Create a chunk
{:ok, chunk} = Chunk.create_chunk(
  "corpora/my-corpus/documents/my-doc",
  %{data: %{string_value: "This is chunk content."}},
  api_key: "your-api-key"
)

# List chunks with pagination
{:ok, result} = Chunk.list_chunks(
  "corpora/my-corpus/documents/my-doc",
  page_size: 20,
  api_key: "your-api-key"
)

# Update chunk content
{:ok, updated} = Chunk.update_chunk(
  "corpora/my-corpus/documents/my-doc/chunks/my-chunk",
  %{data: %{string_value: "Updated content"}},
  "data",
  api_key: "your-api-key"
)

# Batch create chunks
{:ok, result} = Chunk.batch_create_chunks(
  "corpora/my-corpus/documents/my-doc",
  [
    %{chunk: %{data: %{string_value: "First chunk"}}},
    %{chunk: %{data: %{string_value: "Second chunk"}}}
  ],
  api_key: "your-api-key"
)

# Delete chunk
:ok = Chunk.delete_chunk(
  "corpora/my-corpus/documents/my-doc/chunks/my-chunk",
  api_key: "your-api-key"
)

Summary

Functions

Batch create Chunks in a Document.

Batch delete Chunks from a Document.

Creates a Chunk in the specified Document.

Gets information about a specific Chunk.

Lists all Chunks in a Document with pagination support.

Types

t()

@type t() :: %ExLLM.Providers.Gemini.Chunk{
  create_time: String.t() | nil,
  custom_metadata: [CustomMetadata.t()] | nil,
  data: ChunkData.t() | nil,
  name: String.t() | nil,
  state: atom() | nil,
  update_time: String.t() | nil
}

Functions

batch_create_chunks(parent, chunk_requests, opts \\ [])

@spec batch_create_chunks(String.t(), [map()], Keyword.t()) ::
  {:ok, ExLLM.Providers.Gemini.Chunk.BatchResult.t()} | {:error, map()}

Batch create Chunks in a Document.

Parameters

  • parent - The document name in format "corpora/{corpus_id}/documents/{document_id}"
  • chunk_requests - List of chunk creation requests (max 100)
  • opts - Keyword list of options:
    • :api_key - API key for authentication
    • :oauth_token - OAuth2 token for authentication

Chunk Request Format

Each request should have:

  • :parent - The document name (must match parent parameter)
  • :chunk - The chunk data (same format as create_chunk params)

Examples

{:ok, result} = Chunk.batch_create_chunks(
  "corpora/my-corpus/documents/my-doc",
  [
    %{
      parent: "corpora/my-corpus/documents/my-doc",
      chunk: %{data: %{string_value: "First chunk content"}}
    },
    %{
      parent: "corpora/my-corpus/documents/my-doc",
      chunk: %{
        name: "corpora/my-corpus/documents/my-doc/chunks/special-chunk",
        data: %{string_value: "Second chunk with custom name"},
        custom_metadata: [%{key: "type", string_value: "important"}]
      }
    }
  ]
)

batch_delete_chunks(parent, delete_requests, opts \\ [])

@spec batch_delete_chunks(String.t(), [map()], Keyword.t()) :: :ok | {:error, map()}

Batch delete Chunks from a Document.

Parameters

  • parent - The document name in format "corpora/{corpus_id}/documents/{document_id}"
  • delete_requests - List of chunk names to delete
  • opts - Keyword list of options:
    • :api_key - API key for authentication
    • :oauth_token - OAuth2 token for authentication

Delete Request Format

Each request should have:

  • :name - The full chunk name to delete

Examples

:ok = Chunk.batch_delete_chunks(
  "corpora/my-corpus/documents/my-doc",
  [
    %{name: "corpora/my-corpus/documents/my-doc/chunks/chunk-1"},
    %{name: "corpora/my-corpus/documents/my-doc/chunks/chunk-2"}
  ]
)

batch_update_chunks(parent, update_requests, opts \\ [])

@spec batch_update_chunks(String.t(), [map()], Keyword.t()) ::
  {:ok, ExLLM.Providers.Gemini.Chunk.BatchResult.t()} | {:error, map()}

Batch update Chunks in a Document.

Parameters

  • parent - The document name in format "corpora/{corpus_id}/documents/{document_id}"
  • update_requests - List of chunk update requests (max 100)
  • opts - Keyword list of options:
    • :api_key - API key for authentication
    • :oauth_token - OAuth2 token for authentication

Update Request Format

Each request should have:

  • :chunk - The chunk data with name and fields to update
  • :update_mask - Fields to update ("data", "customMetadata")

Examples

{:ok, result} = Chunk.batch_update_chunks(
  "corpora/my-corpus/documents/my-doc",
  [
    %{
      chunk: %{
        name: "corpora/my-corpus/documents/my-doc/chunks/chunk-1",
        data: %{string_value: "Updated content"}
      },
      update_mask: "data"
    },
    %{
      chunk: %{
        name: "corpora/my-corpus/documents/my-doc/chunks/chunk-2",
        custom_metadata: [%{key: "status", string_value: "reviewed"}]
      },
      update_mask: "customMetadata"
    }
  ]
)

create_chunk(parent, params, opts \\ [])

@spec create_chunk(String.t(), map(), Keyword.t()) :: {:ok, t()} | {:error, map()}

Creates a Chunk in the specified Document.

Parameters

  • parent - The document name in format "corpora/{corpus_id}/documents/{document_id}"
  • params - Chunk creation parameters
  • opts - Keyword list of options:
    • :api_key - API key for authentication
    • :oauth_token - OAuth2 token for authentication

Params

  • :name - Optional custom chunk name
  • :data - Required. The chunk content (max 2043 tokens)
  • :custom_metadata - List of key-value metadata (max 20 items)

Examples

# Create with auto-generated name
{:ok, chunk} = Chunk.create_chunk(
  "corpora/my-corpus/documents/my-doc",
  %{data: %{string_value: "This is the content."}}
)

# Create with custom name and metadata
{:ok, chunk} = Chunk.create_chunk(
  "corpora/my-corpus/documents/my-doc",
  %{
    name: "corpora/my-corpus/documents/my-doc/chunks/chapter-1",
    data: %{string_value: "Chapter 1 content."},
    custom_metadata: [
      %{key: "chapter", numeric_value: 1},
      %{key: "keywords", string_list_value: %{values: ["intro", "overview"]}}
    ]
  }
)

delete_chunk(name, opts \\ [])

@spec delete_chunk(String.t(), Keyword.t()) :: :ok | {:error, map()}

Deletes a Chunk.

Parameters

  • name - The chunk name in format "corpora/{corpus_id}/documents/{document_id}/chunks/{chunk_id}"
  • opts - Keyword list of options:
    • :api_key - API key for authentication
    • :oauth_token - OAuth2 token for authentication

Examples

:ok = Chunk.delete_chunk("corpora/my-corpus/documents/my-doc/chunks/my-chunk")

get_chunk(name, opts \\ [])

@spec get_chunk(String.t(), Keyword.t()) :: {:ok, t()} | {:error, map()}

Gets information about a specific Chunk.

Parameters

  • name - The chunk name in format "corpora/{corpus_id}/documents/{document_id}/chunks/{chunk_id}"
  • opts - Keyword list of options:
    • :api_key - API key for authentication
    • :oauth_token - OAuth2 token for authentication

Examples

{:ok, chunk} = Chunk.get_chunk("corpora/my-corpus/documents/my-doc/chunks/my-chunk")

list_chunks(parent, opts \\ [])

@spec list_chunks(String.t(), Keyword.t()) ::
  {:ok, ExLLM.Providers.Gemini.Chunk.ListResult.t()} | {:error, map()}

Lists all Chunks in a Document with pagination support.

Parameters

  • parent - The document name in format "corpora/{corpus_id}/documents/{document_id}"
  • opts - Keyword list of options:
    • :page_size - Maximum chunks per page (1-100, default 10)
    • :page_token - Token for pagination
    • :api_key - API key for authentication
    • :oauth_token - OAuth2 token for authentication

Examples

# List all chunks
{:ok, result} = Chunk.list_chunks("corpora/my-corpus/documents/my-doc")

# List with pagination
{:ok, result} = Chunk.list_chunks(
  "corpora/my-corpus/documents/my-doc",
  %{page_size: 50, page_token: "next-page-token"}
)

update_chunk(name, updates, update_mask, opts \\ [])

@spec update_chunk(String.t(), map(), String.t() | [String.t()], Keyword.t()) ::
  {:ok, t()} | {:error, map()}

Updates a Chunk.

Parameters

  • name - The chunk name in format "corpora/{corpus_id}/documents/{document_id}/chunks/{chunk_id}"
  • updates - Fields to update
  • update_mask - Required. Fields to update ("data", "customMetadata")
  • opts - Authentication options

Updates

  • :data - New chunk content
  • :custom_metadata - New custom metadata

Examples

# Update content
{:ok, chunk} = Chunk.update_chunk(
  "corpora/my-corpus/documents/my-doc/chunks/my-chunk",
  %{data: %{string_value: "New content"}},
  "data"
)

# Update metadata
{:ok, chunk} = Chunk.update_chunk(
  "corpora/my-corpus/documents/my-doc/chunks/my-chunk",
  %{custom_metadata: [%{key: "status", string_value: "reviewed"}]},
  "customMetadata"
)