Latu.ML.FPM (latu_ml v0.2.0)

Copy Markdown View Source

Frequent pattern mining.

Spark's own abbreviation, kept as one: fpm in PySpark, FPM here. Two operators, and both answer with frames rather than predictions — the frequent itemsets and the association rules a transaction log implies.

Every constructor here is generated from PySpark 4.2.0's own param table, and every accessor module from the server's own attribute allowlist. Nothing in this file is hand-written but the words you are reading — see Latu.ML.operators/1 for the table it all comes from.

Summary

Functions

fp_growth(opts \\ [])

A parallel FP-growth algorithm to mine frequent itemsets.

prefix_span(opts \\ [])

A parallel PrefixSpan algorithm to mine frequent sequential patterns. The PrefixSpan algorithm is described in J. Pei, et al., PrefixSpan: Mining Sequential Patterns Efficiently by Prefix-Projected Pattern Growth (see here). This class is not yet an Estimator/Transformer, use findFrequentSequentialPatterns method to run the PrefixSpan algorithm.

Functions

fp_growth(opts \\ [])

@spec fp_growth(keyword()) :: Latu.ML.Estimator.t()

A parallel FP-growth algorithm to mine frequent itemsets.

Latu.ML.fit/2 fits it, and hands back a Latu.ML.Model — a reference into the session's ML cache, not a value. Latu.ML.with_model/3 releases it for you; Latu.ML.delete/1 is the explicit form. Its attributes are on Latu.ML.FPM.FPGrowthModel.

Status :probed — dev/probe_ml.exs fitted it against a live Spark 4.2.0 server, and every allowlisted attribute it could ask answered.

Params

  • :items_col — items column name. Default "items".
  • :min_confidence — Minimal confidence for generating Association Rule. [0.0, 1.0]. minConfidence will not affect the mining for frequent itemsets, but will affect the association rules generation. Default 0.8.
  • :min_support — Minimal support level of the frequent pattern. [0.0, 1.0]. Any pattern that appears more than (minSupport * size-of-the-dataset) times will be output in the frequent itemsets. Default 0.3.
  • :num_partitions — Number of partitions (at least 1) used by parallel FP-growth. By default the param is not set, and partition number of the input dataset is used.
  • :prediction_col — prediction column name. Default "prediction".

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.

prefix_span(opts \\ [])

@spec prefix_span(keyword()) :: Latu.ML.Helper.t()

A parallel PrefixSpan algorithm to mine frequent sequential patterns. The PrefixSpan algorithm is described in J. Pei, et al., PrefixSpan: Mining Sequential Patterns Efficiently by Prefix-Projected Pattern Growth (see here). This class is not yet an Estimator/Transformer, use findFrequentSequentialPatterns method to run the PrefixSpan algorithm.

Latu.ML.find_frequent_sequential_patterns/2 is the one thing it does, and it is a lazy builder: it hands back a Latu.DataFrame and reaches no server until you collect one. There is no fit and nothing cached — its params ride that call as positional arguments, so every one is sent whether you set it or not, and the ones you leave alone are sent as the defaults below.

Status :built — generated from PySpark 4.2.0's own param table, and not yet exercised against a live server by dev/probe_ml.exs.

Params

  • :max_local_proj_db_size — The maximum number of items (including delimiters used in the internal storage format) allowed in a projected database before local processing. If a projected database exceeds this size, another iteration of distributed prefix growth is run. Must be > 0. Default 32000000.
  • :max_pattern_length — The maximal length of the sequential pattern. Must be > 0. Default 10.
  • :min_support — The minimal support level of the sequential pattern. Sequential pattern that appears more than (minSupport * size-of-the-dataset) times will be output. Must be >= 0. Default 0.1.
  • :sequence_col — The name of the sequence column in dataset, rows with nulls in this column are ignored. Default "sequence".

Defaults are documented, never sent: a param the caller did not set and a param sent with its default value are different requests, and only the first is right. A value's kind is refused here; its range is Spark's own ParamValidators to refuse, with a better message than this package could write.