Beamscope.Chunking.Support (Beamscope v0.1.0)

Copy Markdown View Source

Line-extraction, sliding-window, and oversized-chunk-splitting helpers shared by the per-language chunkers. Mirrors chunker.py's _slice_lines, chunk_by_lines, and _split_oversized_chunks so chunk boundaries stay comparable to the original Python pipeline.

Summary

Functions

extract_text(source_lines, start_line, end_line)
sliding_windows(lines, window, overlap, max_chars)

Walks lines in a window-line sliding window with overlap lines of overlap between consecutive windows, shrinking any window whose joined text exceeds max_chars. Returns {start_idx0, end_idx_exclusive, text} triples (0-based, matching Python slice semantics), skipping whitespace-only windows.

split_if_oversized(chunk, opts \\ [])

Splits a chunk into a sliding window (default 60 lines/10-line overlap, shrinking further if any window exceeds the char budget) when it exceeds :max_chunk_lines or :max_chunk_chars, preserving the original symbol on every sub-chunk. Returns [chunk] unchanged otherwise.

Functions

extract_text(source_lines, start_line, end_line)

@spec extract_text([String.t()], pos_integer(), pos_integer()) :: String.t()

sliding_windows(lines, window, overlap, max_chars)

@spec sliding_windows([String.t()], pos_integer(), non_neg_integer(), pos_integer()) ::
  [
    {non_neg_integer(), non_neg_integer(), String.t()}
  ]

Walks lines in a window-line sliding window with overlap lines of overlap between consecutive windows, shrinking any window whose joined text exceeds max_chars. Returns {start_idx0, end_idx_exclusive, text} triples (0-based, matching Python slice semantics), skipping whitespace-only windows.

split_if_oversized(chunk, opts \\ [])

@spec split_if_oversized(
  map(),
  keyword()
) :: [map()]

Splits a chunk into a sliding window (default 60 lines/10-line overlap, shrinking further if any window exceeds the char budget) when it exceeds :max_chunk_lines or :max_chunk_chars, preserving the original symbol on every sub-chunk. Returns [chunk] unchanged otherwise.

All thresholds are overridable via opts (:max_chunk_lines, :max_chunk_chars, :window_lines, :overlap_lines) — defaults match chunker.py's constants for parity with the original pipeline.