Spark's catalog: databases, tables, views, caching.
Session-first and eager — every call answers from the server. Listings come back exactly as
Latu.collect/2 returns rows: maps with atom keys spelled the way the server spells its
columns (:tableType, :isTemporary).
Latu.Catalog.list_tables(session, pattern: "latu_*")
#=> {:ok, [%{name: "latu_tbl_1", tableType: "MANAGED", isTemporary: false, ...}]}This is the useful subset of pyspark.sql.Catalog, not all of it: catalogs, databases,
tables, views and the cache. Creating or describing a single object, and the function
catalog, are left out — each is one Latu.sql/2 away.
Summary
Functions
Cache the table at Spark's default storage level (docs/deviations.md).
Like cache_table/2, raising on failure.
Drop every cached table.
Like clear_cache/1, raising on failure.
The current catalog's name.
Like current_catalog/1, raising on failure.
The current database's name.
Like current_database/1, raising on failure.
Whether the database exists.
Like database_exists/2, raising on failure.
Drop a global temp view; answers like drop_temp_view/2.
Like drop_global_temp_view/2, raising on failure.
Drop a table. if_exists: true tolerates a missing one; purge: true skips the trash.
Like drop_table/3, raising on failure.
Drop a temp view. {:ok, true} when it existed — Spark answers rather than raising.
Like drop_temp_view/2, raising on failure.
Drop a (non-temporary) view. if_exists: true tolerates a missing one.
Like drop_view/3, raising on failure.
Whether the table is cached.
Like is_cached/2, raising on failure.
Every catalog, optionally filtered: pattern: "spark*" (SQL LIKE, * and |).
Like list_catalogs/2, raising on failure.
The table's columns: name, dataType, nullable, partition and bucket flags.
Like list_columns/3, raising on failure.
Every database, optionally filtered by :pattern.
Like list_databases/2, raising on failure.
Tables and views, temporary ones included.
Like list_tables/2, raising on failure.
Refresh Spark's metadata and cache for a table whose files changed underneath it.
Like refresh_table/2, raising on failure.
Switch catalogs.
Like set_current_catalog/2, raising on failure.
Switch databases.
Like set_current_database/2, raising on failure.
Whether the table or view exists. db_name: as in list_tables/2.
Like table_exists/3, raising on failure.
Drop the table from the cache.
Like uncache_table/2, raising on failure.
Functions
@spec cache_table(Latu.Session.t(), String.t() | atom()) :: void()
Cache the table at Spark's default storage level (docs/deviations.md).
@spec cache_table!(Latu.Session.t(), String.t() | atom()) :: :ok
Like cache_table/2, raising on failure.
@spec clear_cache(Latu.Session.t()) :: void()
Drop every cached table.
@spec clear_cache!(Latu.Session.t()) :: :ok
Like clear_cache/1, raising on failure.
@spec current_catalog(Latu.Session.t()) :: result(String.t())
The current catalog's name.
@spec current_catalog!(Latu.Session.t()) :: String.t()
Like current_catalog/1, raising on failure.
@spec current_database(Latu.Session.t()) :: result(String.t())
The current database's name.
@spec current_database!(Latu.Session.t()) :: String.t()
Like current_database/1, raising on failure.
@spec database_exists(Latu.Session.t(), String.t() | atom()) :: result(boolean())
Whether the database exists.
@spec database_exists!(Latu.Session.t(), String.t() | atom()) :: boolean()
Like database_exists/2, raising on failure.
@spec drop_global_temp_view(Latu.Session.t(), String.t() | atom()) :: result(boolean())
Drop a global temp view; answers like drop_temp_view/2.
@spec drop_global_temp_view!(Latu.Session.t(), String.t() | atom()) :: boolean()
Like drop_global_temp_view/2, raising on failure.
@spec drop_table(Latu.Session.t(), String.t() | atom(), keyword()) :: void()
Drop a table. if_exists: true tolerates a missing one; purge: true skips the trash.
The inverse of Latu.save_as_table/3 — what the write tests clean up with.
@spec drop_table!(Latu.Session.t(), String.t() | atom(), keyword()) :: :ok
Like drop_table/3, raising on failure.
@spec drop_temp_view(Latu.Session.t(), String.t() | atom()) :: result(boolean())
Drop a temp view. {:ok, true} when it existed — Spark answers rather than raising.
@spec drop_temp_view!(Latu.Session.t(), String.t() | atom()) :: boolean()
Like drop_temp_view/2, raising on failure.
@spec drop_view(Latu.Session.t(), String.t() | atom(), keyword()) :: void()
Drop a (non-temporary) view. if_exists: true tolerates a missing one.
@spec drop_view!(Latu.Session.t(), String.t() | atom(), keyword()) :: :ok
Like drop_view/3, raising on failure.
@spec is_cached(Latu.Session.t(), String.t() | atom()) :: result(boolean())
Whether the table is cached.
@spec is_cached!(Latu.Session.t(), String.t() | atom()) :: boolean()
Like is_cached/2, raising on failure.
@spec list_catalogs( Latu.Session.t(), keyword() ) :: result([map()])
Every catalog, optionally filtered: pattern: "spark*" (SQL LIKE, * and |).
@spec list_catalogs!( Latu.Session.t(), keyword() ) :: [map()]
Like list_catalogs/2, raising on failure.
@spec list_columns(Latu.Session.t(), String.t() | atom(), keyword()) :: result([map()])
The table's columns: name, dataType, nullable, partition and bucket flags.
@spec list_columns!(Latu.Session.t(), String.t() | atom(), keyword()) :: [map()]
Like list_columns/3, raising on failure.
@spec list_databases( Latu.Session.t(), keyword() ) :: result([map()])
Every database, optionally filtered by :pattern.
@spec list_databases!( Latu.Session.t(), keyword() ) :: [map()]
Like list_databases/2, raising on failure.
@spec list_tables( Latu.Session.t(), keyword() ) :: result([map()])
Tables and views, temporary ones included.
db_name: "other" looks elsewhere than the current database; pattern: "latu_*" filters.
@spec list_tables!( Latu.Session.t(), keyword() ) :: [map()]
Like list_tables/2, raising on failure.
@spec refresh_table(Latu.Session.t(), String.t() | atom()) :: void()
Refresh Spark's metadata and cache for a table whose files changed underneath it.
@spec refresh_table!(Latu.Session.t(), String.t() | atom()) :: :ok
Like refresh_table/2, raising on failure.
@spec set_current_catalog(Latu.Session.t(), String.t() | atom()) :: void()
Switch catalogs.
@spec set_current_catalog!(Latu.Session.t(), String.t() | atom()) :: :ok
Like set_current_catalog/2, raising on failure.
@spec set_current_database(Latu.Session.t(), String.t() | atom()) :: void()
Switch databases.
@spec set_current_database!(Latu.Session.t(), String.t() | atom()) :: :ok
Like set_current_database/2, raising on failure.
@spec table_exists(Latu.Session.t(), String.t() | atom(), keyword()) :: result(boolean())
Whether the table or view exists. db_name: as in list_tables/2.
@spec table_exists!(Latu.Session.t(), String.t() | atom(), keyword()) :: boolean()
Like table_exists/3, raising on failure.
@spec uncache_table(Latu.Session.t(), String.t() | atom()) :: void()
Drop the table from the cache.
@spec uncache_table!(Latu.Session.t(), String.t() | atom()) :: :ok
Like uncache_table/2, raising on failure.