ExLLM.Providers.Gemini.Chunk (ex_llm v0.8.1)
View SourceChunk management for Gemini's Semantic Retrieval API.
A Chunk is a subpart of a Document that is treated as an independent unit for the purposes of vector representation and storage. A Corpus can have a maximum of 1 million Chunks.
Authentication
Chunk operations require authentication. Both API key and OAuth2 are supported:
- API key: Most operations work with API keys
- OAuth2: Required for some operations, especially those involving user-specific data
Examples
# Create a chunk
{:ok, chunk} = Chunk.create_chunk(
"corpora/my-corpus/documents/my-doc",
%{data: %{string_value: "This is chunk content."}},
api_key: "your-api-key"
)
# List chunks with pagination
{:ok, result} = Chunk.list_chunks(
"corpora/my-corpus/documents/my-doc",
page_size: 20,
api_key: "your-api-key"
)
# Update chunk content
{:ok, updated} = Chunk.update_chunk(
"corpora/my-corpus/documents/my-doc/chunks/my-chunk",
%{data: %{string_value: "Updated content"}},
"data",
api_key: "your-api-key"
)
# Batch create chunks
{:ok, result} = Chunk.batch_create_chunks(
"corpora/my-corpus/documents/my-doc",
[
%{chunk: %{data: %{string_value: "First chunk"}}},
%{chunk: %{data: %{string_value: "Second chunk"}}}
],
api_key: "your-api-key"
)
# Delete chunk
:ok = Chunk.delete_chunk(
"corpora/my-corpus/documents/my-doc/chunks/my-chunk",
api_key: "your-api-key"
)
Summary
Functions
Batch create Chunks in a Document.
Batch delete Chunks from a Document.
Batch update Chunks in a Document.
Creates a Chunk in the specified Document.
Deletes a Chunk.
Gets information about a specific Chunk.
Lists all Chunks in a Document with pagination support.
Updates a Chunk.
Types
Functions
@spec batch_create_chunks(String.t(), [map()], Keyword.t()) :: {:ok, ExLLM.Providers.Gemini.Chunk.BatchResult.t()} | {:error, map()}
Batch create Chunks in a Document.
Parameters
parent
- The document name in format "corpora/{corpus_id}/documents/{document_id}"chunk_requests
- List of chunk creation requests (max 100)opts
- Keyword list of options::api_key
- API key for authentication:oauth_token
- OAuth2 token for authentication
Chunk Request Format
Each request should have:
:parent
- The document name (must match parent parameter):chunk
- The chunk data (same format as create_chunk params)
Examples
{:ok, result} = Chunk.batch_create_chunks(
"corpora/my-corpus/documents/my-doc",
[
%{
parent: "corpora/my-corpus/documents/my-doc",
chunk: %{data: %{string_value: "First chunk content"}}
},
%{
parent: "corpora/my-corpus/documents/my-doc",
chunk: %{
name: "corpora/my-corpus/documents/my-doc/chunks/special-chunk",
data: %{string_value: "Second chunk with custom name"},
custom_metadata: [%{key: "type", string_value: "important"}]
}
}
]
)
Batch delete Chunks from a Document.
Parameters
parent
- The document name in format "corpora/{corpus_id}/documents/{document_id}"delete_requests
- List of chunk names to deleteopts
- Keyword list of options::api_key
- API key for authentication:oauth_token
- OAuth2 token for authentication
Delete Request Format
Each request should have:
:name
- The full chunk name to delete
Examples
:ok = Chunk.batch_delete_chunks(
"corpora/my-corpus/documents/my-doc",
[
%{name: "corpora/my-corpus/documents/my-doc/chunks/chunk-1"},
%{name: "corpora/my-corpus/documents/my-doc/chunks/chunk-2"}
]
)
@spec batch_update_chunks(String.t(), [map()], Keyword.t()) :: {:ok, ExLLM.Providers.Gemini.Chunk.BatchResult.t()} | {:error, map()}
Batch update Chunks in a Document.
Parameters
parent
- The document name in format "corpora/{corpus_id}/documents/{document_id}"update_requests
- List of chunk update requests (max 100)opts
- Keyword list of options::api_key
- API key for authentication:oauth_token
- OAuth2 token for authentication
Update Request Format
Each request should have:
:chunk
- The chunk data with name and fields to update:update_mask
- Fields to update ("data", "customMetadata")
Examples
{:ok, result} = Chunk.batch_update_chunks(
"corpora/my-corpus/documents/my-doc",
[
%{
chunk: %{
name: "corpora/my-corpus/documents/my-doc/chunks/chunk-1",
data: %{string_value: "Updated content"}
},
update_mask: "data"
},
%{
chunk: %{
name: "corpora/my-corpus/documents/my-doc/chunks/chunk-2",
custom_metadata: [%{key: "status", string_value: "reviewed"}]
},
update_mask: "customMetadata"
}
]
)
Creates a Chunk in the specified Document.
Parameters
parent
- The document name in format "corpora/{corpus_id}/documents/{document_id}"params
- Chunk creation parametersopts
- Keyword list of options::api_key
- API key for authentication:oauth_token
- OAuth2 token for authentication
Params
:name
- Optional custom chunk name:data
- Required. The chunk content (max 2043 tokens):custom_metadata
- List of key-value metadata (max 20 items)
Examples
# Create with auto-generated name
{:ok, chunk} = Chunk.create_chunk(
"corpora/my-corpus/documents/my-doc",
%{data: %{string_value: "This is the content."}}
)
# Create with custom name and metadata
{:ok, chunk} = Chunk.create_chunk(
"corpora/my-corpus/documents/my-doc",
%{
name: "corpora/my-corpus/documents/my-doc/chunks/chapter-1",
data: %{string_value: "Chapter 1 content."},
custom_metadata: [
%{key: "chapter", numeric_value: 1},
%{key: "keywords", string_list_value: %{values: ["intro", "overview"]}}
]
}
)
Deletes a Chunk.
Parameters
name
- The chunk name in format "corpora/{corpus_id}/documents/{document_id}/chunks/{chunk_id}"opts
- Keyword list of options::api_key
- API key for authentication:oauth_token
- OAuth2 token for authentication
Examples
:ok = Chunk.delete_chunk("corpora/my-corpus/documents/my-doc/chunks/my-chunk")
Gets information about a specific Chunk.
Parameters
name
- The chunk name in format "corpora/{corpus_id}/documents/{document_id}/chunks/{chunk_id}"opts
- Keyword list of options::api_key
- API key for authentication:oauth_token
- OAuth2 token for authentication
Examples
{:ok, chunk} = Chunk.get_chunk("corpora/my-corpus/documents/my-doc/chunks/my-chunk")
@spec list_chunks(String.t(), Keyword.t()) :: {:ok, ExLLM.Providers.Gemini.Chunk.ListResult.t()} | {:error, map()}
Lists all Chunks in a Document with pagination support.
Parameters
parent
- The document name in format "corpora/{corpus_id}/documents/{document_id}"opts
- Keyword list of options::page_size
- Maximum chunks per page (1-100, default 10):page_token
- Token for pagination:api_key
- API key for authentication:oauth_token
- OAuth2 token for authentication
Examples
# List all chunks
{:ok, result} = Chunk.list_chunks("corpora/my-corpus/documents/my-doc")
# List with pagination
{:ok, result} = Chunk.list_chunks(
"corpora/my-corpus/documents/my-doc",
%{page_size: 50, page_token: "next-page-token"}
)
@spec update_chunk(String.t(), map(), String.t() | [String.t()], Keyword.t()) :: {:ok, t()} | {:error, map()}
Updates a Chunk.
Parameters
name
- The chunk name in format "corpora/{corpus_id}/documents/{document_id}/chunks/{chunk_id}"updates
- Fields to updateupdate_mask
- Required. Fields to update ("data", "customMetadata")opts
- Authentication options
Updates
:data
- New chunk content:custom_metadata
- New custom metadata
Examples
# Update content
{:ok, chunk} = Chunk.update_chunk(
"corpora/my-corpus/documents/my-doc/chunks/my-chunk",
%{data: %{string_value: "New content"}},
"data"
)
# Update metadata
{:ok, chunk} = Chunk.update_chunk(
"corpora/my-corpus/documents/my-doc/chunks/my-chunk",
%{custom_metadata: [%{key: "status", string_value: "reviewed"}]},
"customMetadata"
)