lastfm_archive v0.7.1 LastfmArchive.Load View Source

This module provides functions for loading Lastfm data into databases and search engines.

Link to this section Summary

Functions

Check a Solr core/collection to ensure it has the required Lastfm data fields

Load a TSV file data from the archive into Solr for a Lastfm user

Ping a Solr core/collection endpoint to check if it is running

Read and parse a TSV file from the archive for a Lastfm user

Link to this section Functions

Link to this function check_solr_schema(url) View Source
check_solr_schema(binary() | atom()) :: {:ok, map()} | {:error, Hui.Error.t()}

Check a Solr core/collection to ensure it has the required Lastfm data fields.

The check currently inspects Solr schema for a list of Lastfm fields and returns error if one or more of the fields are missing. See LastfmArchive.Transform.transform/3 for the list of fields.

Example

  LastfmArchive.Load.check_solr_schema("http://solr_url...")
  LastfmArchive.Load.check_solr_schema(:lastfm_archive) # ping a configured endpoint

See ping_solr/1 for more details on URL configuration.

Link to this function load_solr(url, user, filename) View Source
load_solr(Hui.URL.t(), binary(), binary()) ::
  {:ok, HTTPoison.Response.t()} | {:error, :enoent}

Load a TSV file data from the archive into Solr for a Lastfm user.

The function reads and converts scrobbles in a TSV file from the file archive into a list of maps. The maps are sent to Solr for ingestion. Use Hui.URL.t/0 struct to specify the Solr endpoint.

Example

  # define a Solr endpoint with %Hui.URL{} struct
  headers = [{"Content-type", "application/json"}]
  url = %Hui.URL{url: "http://localhost:8983/solr/lastfm_archive", handler: "update", headers: headers}

  # ingest data scrobbled in 2018
  LastfmArchive.Load.load_solr(url, "a_lastfm_user", "tsv/2018.tsv.gz")

TSV files must be pre-created by transforming raw JSON Lastfm data - see LastfmArchive.transform_archive/2.

Link to this function ping_solr(url) View Source
ping_solr(binary() | atom()) :: {:ok, map()} | {:error, Hui.Error.t()}

Ping a Solr core/collection endpoint to check if it is running.

The endpoint can either be a URL string or an atom referring to an endpoint in configuration. The library uses Hui to interact with Solr, an endpoint can be specified as below:

Example

  LastfmArchive.Load.ping_solr("http://solr_url...")
  LastfmArchive.Load.ping_solr(:lastfm_archive) # check a configured endpoint

:lastfm_archive refers to the following Solr update endpoint in configuration:

  config :hui, :lastfm_archive,
    url: "http://solr_url..",
    handler: "update",
    headers: [{"Content-type", "application/json"}]

See Hui.URL module for more details.

Link to this function read(user, filename) View Source
read(binary(), binary()) :: {:ok, [binary()]} | {:error, :file.posix()}

Read and parse a TSV file from the archive for a Lastfm user.

TSV files generated by transforming raw JSON Lastfm data - see LastfmArchive.transform_archive/2. The file is parsed into a list of scrobbles.

Example

  LastfmArchive.Load.read "a_lastfm_user", "tsv/2007.tsv.gz"