lastfm_archive v0.1.0 LastfmArchive View Source
lastfm_archive
is a tool for creating local Last.fm scrobble data archive and analytics.
The software is currently experimental and in preliminary development. It should eventually provide capability to perform ETL and analytic tasks on Lastfm scrobble data.
Current usage:
archive/1
: download raw Lastfm scrobble data to local filesystem.
Link to this section Summary
Functions
Download all scrobbled tracks and create an archive on local filesystem for a Lastfm user specified in configuration
Issues a request to Lastfm to extract scrobbled tracks for a user
Write binary data or Lastfm response to a configured directory on local filesystem
Link to this section Types
lastfm_response() :: {:ok, map()} | {:error, binary(), HTTPoison.Error.t()}
Link to this section Functions
archive(integer()) :: :ok | {:error, :file.posix()}
Download all scrobbled tracks and create an archive on local filesystem for a Lastfm user specified in configuration.
The data is currently in raw Lastfm recenttracks
JSON format, chunked into
200-track compressed (gzip
) pages and stored within directories corresponding
to the years when tracks were scrobbled.
interval
is the duration (in milliseconds) between successive requests
sent to Lastfm API.
It provides a control for the max rate of requests.
The default (500ms) ensures a safe rate that is
within Lastfm’s term of service - no more than 5 requests per second.
The data is written to a main directory,
e.g. ./lastfm_data/a_user/
as configured below - see
config/config.exs
:
config :lastfm_archive,
user: "a_user",
data_dir: "./lastfm_data/"
Example
LastfmArchive.archive
Note: Lastfm API calls can timed out occasionally. When this happen
the function will continue archiving and move on to the next data chunk (page).
It will log the missing page in an error
directory. Re-run the function
to download any missing data chunks. The function will skip all existing
archived pages.
To create a fresh or refresh part of the archive: delete all or some files in the archive and re-run the function.
Issues a request to Lastfm to extract scrobbled tracks for a user.
See Lastfm API documentation for details on the use of parameters.
write(binary() | lastfm_response(), binary()) :: :ok | {:error, :file.posix()}
Write binary data or Lastfm response to a configured directory on local filesystem.
The data is compressed, encoded and stored in a file of given filename
within the user data directory, e.g. ./lastfm_data/a_user/
as configured
below:
config :lastfm_archive,
user: "a_user",
data_dir: "./lastfm_data/"