Voile.Migration.Common (Voile v0.1.26)

Copy Markdown View Source

Common utilities and helpers for data migration.

Summary

Functions

batch_insert(repo, schema, records, batch_size, opts \\ [])

batch_insert_optimized(repo, schema, records, opts \\ [])

Optimized batch insert that minimizes connection idle time. Uses transactions and proper timeouts for better performance.

csv_base_path()

extract_unit_id_from_filename(filename)

filter_duplicates(records, seen_set, field_getter, line_getter \\ nil)

Filter out duplicate records based on a specified field. Returns {filtered_records, seen_set, duplicate_count}

get_csv_files(data_type)

get_specific_files(data_type, pattern)

measure_time(operation, func)

parse_date(val)

parse_datetime(val)

parse_int(val)

safe_string_trim(val)

utc_now_truncated()