lastfm_archive v0.7.2 LastfmArchive.Load View Source
This module provides functions for loading Lastfm data into databases and search engines.
Link to this section Summary
Functions
Check a Solr core/collection to ensure it has the required Lastfm data fields
Load a TSV file data from the archive into Solr for a Lastfm user
Ping a Solr core/collection endpoint to check if it is running
Read and parse a TSV file from the archive for a Lastfm user
Link to this section Functions
check_solr_schema(binary() | atom()) :: {:ok, map()} | {:error, Hui.Error.t()}
Check a Solr core/collection to ensure it has the required Lastfm data fields.
The check currently inspects Solr schema for a list of Lastfm fields
and returns error if one or more of the fields are missing.
See LastfmArchive.Transform.transform/3
for the list of fields.
Example
LastfmArchive.Load.check_solr_schema("http://solr_url...")
LastfmArchive.Load.check_solr_schema(:lastfm_archive) # ping a configured endpoint
See ping_solr/1
for more details on URL configuration.
load_solr(Hui.URL.t(), binary(), binary()) :: {:ok, HTTPoison.Response.t()} | {:error, :enoent}
Load a TSV file data from the archive into Solr for a Lastfm user.
The function reads and converts scrobbles in a TSV file from the file
archive into a list of maps. The maps are sent to Solr for ingestion.
Use Hui.URL.t/0
struct to specify the Solr endpoint.
Example
# define a Solr endpoint with %Hui.URL{} struct
headers = [{"Content-type", "application/json"}]
url = %Hui.URL{url: "http://localhost:8983/solr/lastfm_archive", handler: "update", headers: headers}
# ingest data scrobbled in 2018
LastfmArchive.Load.load_solr(url, "a_lastfm_user", "tsv/2018.tsv.gz")
TSV files must be pre-created by transforming raw JSON Lastfm data - see
LastfmArchive.transform_archive/2
.
ping_solr(binary() | atom()) :: {:ok, map()} | {:error, Hui.Error.t()}
Ping a Solr core/collection endpoint to check if it is running.
The endpoint can either be a URL string or an atom referring to an endpoint in configuration.
The library uses Hui
to interact with Solr, an endpoint can be specified as below:
Example
LastfmArchive.Load.ping_solr("http://solr_url...")
LastfmArchive.Load.ping_solr(:lastfm_archive) # check a configured endpoint
:lastfm_archive
refers to the following Solr update endpoint in configuration:
config :hui, :lastfm_archive,
url: "http://solr_url..",
handler: "update",
headers: [{"Content-type", "application/json"}]
See Hui.URL
module for more details.
read(binary(), binary()) :: {:ok, [binary()]} | {:error, :file.posix()}
Read and parse a TSV file from the archive for a Lastfm user.
TSV files generated by transforming raw JSON Lastfm data - see
LastfmArchive.transform_archive/2
. The file is parsed
into a list of scrobbles.
Example
LastfmArchive.Load.read "a_lastfm_user", "tsv/2007.tsv.gz"