lastfm_archive v0.2.0 LastfmArchive View Source

lastfm_archive is a tool for creating local Last.fm scrobble data archive and analytics.

The software is currently experimental and in preliminary development. It should eventually provide capability to perform ETL and analytic tasks on Lastfm scrobble data.

Current usage:

Link to this section Summary

Functions

Download all scrobbled tracks and create an archive on local filesystem for the default user

Download all scrobbled tracks and create an archive on local filesystem for a Lastfm user

Issues a request to Lastfm to extract scrobbled tracks for a user

Write binary data or Lastfm response to a configured directory on local filesystem for a Lastfm user

Link to this section Types

Link to this type lastfm_response() View Source
lastfm_response() :: {:ok, map()} | {:error, binary(), HTTPoison.Error.t()}

Link to this section Functions

Link to this function archive() View Source
archive() :: :ok | {:error, :file.posix()}

Download all scrobbled tracks and create an archive on local filesystem for the default user.

Example

  LastfmArchive.archive

The archive belongs to a default user specified in configuration, for example user_a (in config/config.exs):

  config :lastfm_archive,
    user: "user_a",
    ...

See archive/2 for further details on archive format and file location.

Link to this function archive(user, interval \\ Application.get_env(:lastfm_archive, :req_interval) || 500) View Source
archive(binary(), integer()) :: :ok | {:error, :file.posix()}

Download all scrobbled tracks and create an archive on local filesystem for a Lastfm user.

Example

  LastfmArchive.archive("a_lastfm_user")

The data is currently in raw Lastfm recenttracks JSON format, chunked into 200-track compressed (gzip) pages and stored within directories corresponding to the years when tracks were scrobbled.

interval is the duration (in milliseconds) between successive requests sent to Lastfm API. It provides a control of the max rate of requests. The default (500ms) ensures a safe rate that is within Lastfm’s term of service - no more than 5 requests per second.

The data is written to a main directory, e.g. ./lastfm_data/a_lastfm_user/ as configured in config/config.exs:

  config :lastfm_archive,
    ...
    data_dir: "./lastfm_data/"

Note: Lastfm API calls could timed out occasionally. When this happen the function will continue archiving and move on to the next data chunk (page). It will log the missing page in an error directory. Re-run the function to download any missing data chunks. The function will skip all existing archived pages.

To create a fresh or refresh part of the archive: delete all or some files in the archive and re-run the function.

Link to this function extract(user, page \\ 1, limit \\ 1, from \\ 0, to \\ 0) View Source
extract(binary(), integer(), integer(), integer(), integer()) ::
  lastfm_response()

Issues a request to Lastfm to extract scrobbled tracks for a user.

See Lastfm API documentation for details on the use of parameters.

Link to this function write(user, data, filename \\ "1") View Source
write(binary(), binary() | lastfm_response(), binary()) ::
  :ok | {:error, :file.posix()}

Write binary data or Lastfm response to a configured directory on local filesystem for a Lastfm user.

The data is compressed, encoded and stored in a file of given filename within the data directory, e.g. ./lastfm_data/user/ as configured below:

config :lastfm_archive,
  ...
  data_dir: "./lastfm_data/"