last.fm scrobbling implementation progress

[ungive]

This issue tracks the progress for the upcoming scrobbling feature. By funding this project you allow me to work on it full-time. Feel free to comment and make suggestions or join the discussion in the #scrobbling channel on the Discord server! Related issues: #53 #271 #62

Overview

• Minimal working version checklist
• Replacement rule examples with regular expressions
• Rollout plan for the first version
• Notes and ideas for extended scrobbling features

Minimal working version checklist

Summary

This checklist contains a detailed description of everything that is needed for a minimal, but fully functional last.fm scrobbling implementation. This includes metadata replacement and filtering of media using regular expressions with useful default replacements built-in, to allow you to achieve a perfect scrobble with any media player on any operating system.

Preparation / Core logic

A lot of time (more than 50%) goes into these preparation steps because they are related to the core scrobbling component. Once this is implement, the steps in "Base implementation" and thereafter are merely for integrating the already working scrobbling component into the Music Presence app, which is arguably much less work. Most of the work on designing the scrobbler takes place here.

• Create a separate project for a scrobbling library
• Research how other scrobblers operate, what scrobbling services are available beyond last.fm, what data these APIs need, how to best split artists, what options to provide to influence the scrobblers functions, etc. and come up with an architecture that allows both automatic metadata correction using online music databases (may not always work) and deterministic metadata correction via user configuration using concepts like corrections using regular expression, automatic artist separation using a list of separators, etc. - a few concepts are already introduced in the sections after this one and will be updated once this step is complete
• Implement reliable artist splitting, so you can scrobble the primary artist or all artists as a list
• Implement the scrobbling library with all the needed scrobbling logic for Music Presence
  • Implement and test logic for determining when to scrobble a playing track with real-time stats
  • Test manually with real music playback from the local machine and make a small demo
  • In progress: Implement the database and the interaction with the OS's keystore to store account information and credentials, scrobbles, API request logs, scrobble submissions and any failed scrobble submission attempts. The database is the backbone of the scrobbling implementation and allows you to offline scrobble, see your history of scrobbles, all scrobble submissions to the respective service and their success state and any other logs for API requests that were made to the services you logged in with
  • Implement the needed orchestration for an arbitrary amount of playing tracks and forwarding of their scrobbles to all configured scrobbling targets (last.fm, ListenBrainz, etc.).
  • Implement the actual network requests for each individual scrobbling target and the needed authentication.
  • Put everything together and test with real music playback from the local machine

Base implementation

• Allow the user to log in with last.fm by entering basic API credentials
• Implement basic scrobbling of the currently playing track using the default scrobbling rules
• Implement sending the "Now Playing" request whenever the current song changes (rate limit this)
• Add a toggle to enable or disable scrobbling globally (enabled by default)
• Add a toggle to enable the "Now Playing" request globally (enabled by default)
• Add toggles to enable scrobbling per player (all players are disabled by default)
• Add a slider: Minimum playback percentage to scrobble, starting at 30% (default: 50%)
• Add a slider: Minimum playback duration to scrobble, starting at 30 seconds (default: 4 minutes)
• Add a checkbox: Scrobble tracks shorter than 30 seconds (default: no)

Tools to modify metadata and filter media before scrobbling

• Add a settings category to add "rules" for modifying and filtering media by matching its metadata
• A rule can match the following metadata: Title, artist, album and album artist. A rule uses one regular expression to match each of these metadata fields respectively. Allow the use of the grammar variation icase to match case-insensitively. When no regular expression is entered or it is empty (""), then any value for the metadata field (present or not) matches. A rule needs at least one regular expression to match. A rule can be toggled either for all players (the default) or a specific media player. A rule can be applied either globally for all use cases (the default) or for a specific use case only (e.g. only scrobbling or only the Discord status)
• Allow the user to add, remove, re-order and toggle each of these rules individually
• Add a "replacement" rule type: When a replacement rule matches, you can use matched text in a matching group to modify either the matched metadata field or other metadata fields. For instance, consider media with the title "SIA - Titanium" and no other metadata fields, where the artist name ("SIA") is part of the title and you want to either swap the two or move the artist name to the "artist" metadata field. For illustration purposes, you could use the very simple regular expression (SIA) - (Titanium) for the "title" field and then do either of the following:
  • Use $2 - $1 in the "title" replacement field to swap the two words so that the title "SIA - Titanium" becomes "Titanium - SIA"
  • Use $title:2 for the "title" replacement field and $title:1 for the "artist" replacement field, so that the media has the title "Titanium" and the artist is "SIA". This is a convoluted example, but this can be very useful when media players report metadata in the wrong fields or when they combine metadata in a similar way
  • Full replacement syntax:
    • $N (where N is a number) as the replacement for metadata field X (e.g. the "title" metadata field) uses the match of the N-th capture group in the regular expression for the metadata field X. You can only use this replacement syntax for a metadata field, when that metadata field has a regular expression
    • $FIELD:N (where "FIELD" is either title, artist, album, albumArtist, album_artist or albumartist) as the replacement for any metadata field uses the match of the N-th capture group in the regular expression for the metadata field identified by "FIELD". Example: $title:2 uses the match of the second capture group of the title regex
    • $FIELD:NAME (where "NAME" is the name of any capture group in the regular expression identified by "FIELD") is identical to the $FIELD:N syntax, but is used to reference capture groups by name (e.g. (?<title>.+)) instead of by index
    • Use $$ to use a literal $ in the replacement
    • A backslash \ can also be used instead of the $ symbol
  • You can find more examples for useful replacement rules further down
• Add a "filter" rule type: When a filter rule matches, the media is not scrobbled. A filter rule can match either when all entered regular expressions match or when one of them match, i.e. either by "AND" or by "OR" logic. The user can switch the used logic per rule. Each filter rule's regular expression can be negated ("NOT" logic) to only match when the regular expression doesn't match
  • Combined AND/OR logic will not be supported. Multiple rules should be created instead, to achieve the same result. E.g. a AND (b OR c) can be rewritten as (a AND b) OR (a AND c) which can be split into two separate rules, a AND b and a AND c. A rule like (a OR b) AND (c OR d) can be split into four separate rules or can be simplified with DeMorgan's Law into two rules NOT a AND NOT b and NOT c AND NOT d
• Add default rules for specific media players, e.g. for Apple Music: Removing the user's name from a station that is playing (anonymization), removing "- EP" from album names (clean names) and splitting the album and artist name on Windows, which is often reported together and separated by a dash. Default rules should not be modifiable by the user so that they can be automatically updated without overwriting the user's changes. These should be separated from custom replacements and filters. Also the user should have the option to edit a replacement by creating a copy and moving it to the user's custom replacements, enabling that and disabling the default

These tools are meant to be available globally, so that media can be filtered and modified for any use case, i.e. not only last.fm scrobbling, but also for the Discord status and all use cases that will be added in the future. This would then fully implement #62.

Basic quality of life features

• Allow toggling scrobbling globally in the tray menu
• See the current scrobble in real-time when opening the tray menu
• See the current scrobble in real-time when hovering the tray icon
• Modify the tray icon to show when the app is currently scrobbling

Dev notes

• For regular expressions the RE2 C++ library looks promising: https://github.com/google/re2. The RE2 syntax is a subset of the powerful PCRE syntax, without features that could cause catastrophic backtracking. The features that are left out (e.g. lookbehind, backreferences, conditional patterns, recursion and conditional branching) are overkill for this and their ommission guarantees faster execution times and no risk for certain inputs completely blocking the application or causing high CPU usage. Replacement syntax is basic, but must be implement manually anyway because of the option to reference matches from multiple regular expressions ($title:1). Named capture groups (e.g. (?P<artist>...)) should be referenced like this instead: $title:artist
• Try to move over as many filters that are already built-in to Music Presence to the new regular expression filters as possible, e.g. for TIDAL, where on app start the app reports no artist and a song with the title "TIDAL", "Home - TIDAL", "Home – TIDAL" or "desktop.tidal.com".

Replacement rule examples with regular expressions

These are examples for default replacement rules that should be enabled by default. The order here matters: Rules that are listed first are executed first, before any rules that follow. Rules after that build upon the replacements that have been done beforehand.

• Apple Music reports the user's station name which leaks personal information. This replacement rule completely removes the station name from any field where it might be present.

  • Name: "Remove the Apple Music station name"
  • Artist regex (case-insensitive): ^(.+—.+?)\\s*—\\s*(.+s\ Sender|.+’s\ Station|.+さんのステーション|.+的电台|.+的電台|.+의\ 스테이션|Emisora\ de\ .+|Estación\ de\ .+|Estação\ de\ .+|Station\ de\ :\ .+|Station\ för\ .+|Station\ van\ .+|Stazione\ di\ .+|Моя\ станция\ \(.+\)).*$
  • Album regex (case-insensitive): ^(.+?)\s*—\\s*(.+s\ Sender|.+’s\ Station|.+さんのステーション|.+的电台|.+的電台|.+의\ 스테이션|Emisora\ de\ .+|Estación\ de\ .+|Estação\ de\ .+|Station\ de\ :\ .+|Station\ för\ .+|Station\ van\ .+|Stazione\ di\ .+|Моя\ станция\ \(.+\)).*$
  • Artist replacement: $1
  • Album replacement: $1
  • Example: Title - Station for Jonas becomes Title
  • Station regex source: https://github.com/ungive/apple-music-station-translations/blob/master/out/regex.txt
  • Applied globally, to all media players, for scrobbling and the Discord status

• Apple Music on Windows reports the artist and the album name together, separated by a hyphen (—), in both the "artist" field and the "album artist" field. This replacement rule splits the album and artist. This should be enabled globally, for all media players and all use cases.

  • Name: "Split Apple Music artist and album"
  • Artist regex: ^([^—]+)\s+—\s+(.+)$
  • Album regex: ^$ – Optional, this ensures the rule only matches when the album is missing
  • Artist replacement: $artist:1
  • Album replacement: $artist:2
  • Example: Artist — Album becomes Artist in the "artist" field Album in the "album" field
  • Applied globally, to all media players, for scrobbling and the Discord status

• For scrobbling specifically: If the artist field is empty (for whatever reason) use the album artist.

  • Name "Use the album artist, when the artist is missing"
  • Artist regex: ^$
  • Album artist regex: ^(.+)$
  • Artist replacement: $album_artist:1
  • Applied to scrobbling only

• For scrobbling specifically: Always remove the album artist, use the artist field only. This is useful in cases where a streaming platform might report an incorrect/unexpected album artist or a value like "Various Artists".

  • Name "Remove the album artist"
  • Album artist regex: ^(.+)$
  • Album artist replacement: ​ (empty)
  • Applied to scrobbling only

• For scrobbling specifically: Remove any artist that comes after the first listed artist

  • Name: "Only scrobble the primary artist"
  • Artist regex: ^([^;,]+)\s*[;,].*$
  • Artist replacement: $1
  • Example: Wes Montgomery, Tommy Flanagan, Percy Heath, Albert Heath becomes Wes Montgomery
  • Applied to scrobbling only

• Some media players report multiple artists separated by a semicolon. A comma looks nicer.

  • Name: "Replace semicolons in the artist field with commas"
  • Artist regex: ;(\s)?\s*
  • Artist replacement: ,$1
  • Example: Shakey Graves; Monica Martin becomes Shakey Graves, Monica Martin
  • Applied globally, to all media players, for scrobbling and the Discord status

TODO Add " - EP", " - Single", etc. removal of album names with specific players/streaming services.

Rollout plan for the first version

The first working version is planned to be available in September 2025 in accordance to the planned project timeline on my website. The first people to have access to it will be financial supporters and access will be granted with an activation code that can be entered in the settings of the app. I will make test builds available before the deadline, when exactly that will be is not clear yet.

Once scrobbling is in a state where it can be publicly released, it will be available as a paid feature. I will figure out the details for that later, once scrobbling is fully implemented and working. Anyone who is already supporting me financially at the time of the public release will get the option to contact me to receive a free activation code.

Ideas for extended scrobbling features

• See past and current scrobbles in real-time in the settings window. Show which scrobbles failed and how long it takes until the currently playing song will be scrobbled
• Allow hiding any Discord controls in the tray menu when the Discord presence is disabled. This is useful for people who do not use Discord or never intend to use Music Presence for showing a Discord status
• Add a reminder pop-up to disable scrobbling in a player, whenever the user enables scrobbling for that player in the settings, to aid with preventing double scrobbles. The user can opt to "never show this again"
• Allow getting additional metadata from APIs, mainly missing song durations, album artists and album track numbers, since these are not reported always. The album track number should only be set when the album name is known. The album name should never be "guessed" using an API. When implementing this, move the global "Services" settings tab under "Discord" and make a separate services tab for last.fm
• Allow the user to give each media filter and replacement rule a name and collapse or expand it to make rule management easier. If no name is given, derive a default name from one of the regular expressions instead of displaying nothing
• Allow the user to choose the default regex syntax for new rules
• Allow logging in with user name and password by opening the log in page in the default browser and redirecting
• Add an offline mode, to store scrobbles that couldn't be transmitted due to lack of internet connectivity
• Allow showing your current scrobble in your Discord status, so you can scrobble from your phone and show that in the Discord status
• Allow choosing which media types to scrobble (e.g. do not scrobble podcasts)
• ...

Notes

last.fm API notes

• Use an identifiable User-Agent header
• Adhere to API limits to not be blocked, especially with the default settings, see https://www.last.fm/api/intro
• last.fm API requests and responses should be logged to a log file, e.g. scrobbling.log. See https://www.last.fm/api/scrobbling#error-handling

What users want for scrobbling

• Clean up metadata with regular expressions (1)
• Scrobble songs under 30 seconds (1)
• Listening to tracks on repeat and scrobbling accordingly (1)
• Skip scrobbling certain songs/blacklist them (1)
• Do not scrobble all artists, only the primary one (1) (2) (3)
• ...

TODO Collect more from the #scrobbling channel on the Discord server.

What users want in regard to Discord

• Show the current last.fm scrobble in the Discord status (1)
• A "View on last.fm" button to redirect to the song page or your own profile (1)

Links

• Help fund this project, so I can release scrobbling as planned in September of 2025: https://musicpresence.app
• Join the Discord server, ask questions and participate in the discussion: https://discord.gg/musicpresence