Lastgenre: New config option keep_existing

[JOJ0]

Description (moved)

Initially this PR included fixes which moved to a separate PR #5582

Description

• Fix the behavior of theforce option. Previously disabling the option had "incomplete" behaviour:
  • If content was found, a whitelist check was issued and if valid the plugin exited early and logged ("keep").
  • This whitelist check was not aware of multiple genres (separated typically by a string like , ), thus it failed erased all existing genres and overwrote with new ones.

This didn't feel like a typical behaviour of a force option, which this PR tries to improve as follows...

• String-separated multi-genres are now compiled into a list and depending on the whitelist option are kept and enriched with freshly fetched last.fm genres.

• If force is off, pre-populated tags are not touched.

• A lot of refactoring was done, some absolutely required, some as a preparation for future work on the plugin.

• The main processing function _get_genre was massively overhauled and got a new pytest.mark.parametrize test which includes much more test cases.

Details & Docs

Back in 2023-09 we decided on an additional option named keep_allowed, details on what we came up with: #4982 (comment)):

My final conclusion is to change that option name to keep_existing, which feels slightly more self-explanatory. I also decided on Setup 3 (see below) as the default because:

• force always was the plugin's default.
• with keep-existing also enabled by default it feels like a pretty common use-case.

Setup 1

Overwrite all. Only fresh last.fm genres remain.

force: yes
keep_existing: no

Setup 2

Add new last.fm genres when empty. Present tags stay untouched.

force: no
keep_existing: no

Setup 3 (default)

Add new last.fm genres. Combine genres in present tags with new ones
(depending on the whitelist setting, allowed or any).

force: yes
keep_existing: yes

To Do

• Documentation
• Changelog.
• Fix existing tests.
• Refactor _get_genre tests using pytest.mark.parametrize and add new test-cases.
• Implement Case 1
• Implement Case 2
• Implement Case 3
• Implement Case 4

[JOJ0]

I'd request a review from you @sampsyo since I think you initially created it. Also @rain0r would be good since 5 years ago they added the -A option. Hi @rain0r , you wanna take a look? :-)

In short: I think I fixed the plugin to now really reflect what's documented. Any nitpicking in my code or functionality-wise is appreciated.

One question already. Here we do not state that a -a/--album option exists: https://beets.readthedocs.io/en/latest/plugins/lastgenre.html#running-manually

When I started out with using this plugin I was confused a verry long time about this option. As far as I understand it now: It doesn't do anything since it is default. So why keep it? Or is having a -a option that is the default anyway a common thing in beets? I know we have a lot of -a commands which streamlines usablity, and that is a very good thing! Usuall they change behaviour to not do something with items but with albums. I'm just not sure about this one....do we have such a pattern anywhere else? So, just leave it? Should I add some words to the docs?

I think the both of you decided these options should look like that around here: #3220 (comment)

[sampsyo]

Thanks for the extra context, @JOJ0!

About the existence of -a (the default mode) specifically: it's not too uncommon… for example, the beet import command has several flags that are opposites of each other, one of which is the default. Of course, it's important in that case because the default mode can be set in the config, so the user needs a way to override the default in either direction. That's not the case here, so maybe it at least makes sense to add "(default)" to the -a option's help string, or to remove it altogether?

[sampsyo]

Thanks for the ping!! Here are a couple of straightforward comments.

[JOJ0]

I'd like to pull out this conversation #4982 (comment) into a new thread, to make it more obvious for others as well. I think it could be a broader discussion of where this plugin should go. Basically we were talking about the current force: no behaviour being weird as well as the new behaviour I am initially proposing with this PR. I gave all this some thought and came up with this idea. Let's discuss it:

So from my point of view, the main problem with the current behaviour when force is disabled, is that it's not really what a user would typically expect. So what could we do to make force: no more predictable?

The following idea would require a new config setting as well as a whole new branch of behaviour (Case 3):

Case 1

force: yes
overwrite all, only fresh last.fm genres remain

Case 2

force: no

keep any string in present genre tag, only write last.fm genres when empty

Case 3

force: yes
keep_allowed: yes

keep present genres when whitelisted and add new last.fm genres (this is a new branch of behaviour and needs to be coded, I think there is open feature requests for it. Update: Something was feature-requested, but it might not be exactly as I'm proposing here: #4750)

Case 4

force: no
keep_allowed: yes

cleanup only - keep present genres when whitelisted but don't add new last.fm genres; Only when genre is empty, add last.fm genres.

That last combination is weird though....but it's what I proposed for force:no before!

Which of these would now make sense to be the new default? The new force: no (Case 2) would be the least invasive IMO...

@sampsyo brainstorming request 🧐

[JOJ0]

Some more context / cross-linking:

The initial reason why I got my hands dirty with this plugin was when I realised that comma separated multi-genres where not recognized: #4751 (comment)

Here @arsaboo requests a feature that goes in direction of Case 3 above: #4750

[arsaboo]

So, we have two config options - force and keep_allowed, i.e., 4 options in all. Given that, keep_allowed is no in cases 1 and 2. Thus, here's a slightly modified behavior in the 4 cases above:

Case 1: overwrite all, only fresh last.fm genres remain

force: yes
keep_allowed: no

Case 2: Since keep_allowed is no, we only write last.fm genres when empty. There may be incorrect genres in pre-existing tags even after this, as this option is not touching pre-existing tags

force: no
keep_allowed: no

Case 3: keep present genres when whitelisted and add new last.fm genres

force: yes
keep_allowed: yes

Case 4: keep any string in the present genre tag; only write last.fm genres when empty. This will not touch pre-existing genre tags.

force: no
keep_allowed: yes

Thus, Case 4 seems like the best default choice. It does not affect existing genre tags and updates the empty ones. Case 3, on the other hand, is the most useful one (at least for me).

[sampsyo]

This brainstorming honestly sounds great, y'all. It is indeed really weird that the force: no mode can still update old genres; keeping all nonempty genres seems like it should at least be an option. I feel less specific about what the default should be, but I like your idea about decoupling the two aspects of the behavior (when to override existing, nonempty data and what to do to old data) into two different options.

[JOJ0]

Ähem I might be slow or too tired already. Which of those 4 cases are now different from my proposal @arsaboo ? Sorry I must have missed it! Help! :-)

[arsaboo]

Not different....just a little more explicit about the force and keep_allowed config options. I think we have an agreement about the options.

[github-actions]

Thank you for the PR! The changelog has not been updated, so here is a friendly reminder to check if you need to add an entry.

[JOJ0]

Hi @arsaboo! I finally managed to find time to almost finish this PR. The general behaviour and docs of the new config options combinations are finished. If you want to, an "early" review would be super helpful. Since it probably also for you is a long time ago it might be interesting what you think if you read through the docs. Is it 100% clear what force/keep_allowed options do? Certainly but only if you have the time, some playing around and checking if it also really works that way would be great. Thanks a ton!

[arsaboo]

@JOJ0 this is AWESOME 🎉🎉

The docs look reasonably clear. I will play with this. The debug logs are great to see what is going on.

[JOJ0]

Main points:

1. The configuration is very confusing, I think the three options could be reduced to a single one called mode.

To be honest, yes it is a little confusing. I have considered a single option, many times. I thought it through often and always came to the same conclusion: I like when tools follow certain usability patterns. For example in beets we have -a for album-level things, -A for track-level, --pretend for dry-runs, and we have --force options. It kind of makes tools more self-explanatory and give a good overall usability feeling. If someone would submit a new plugin and call an option --dry-run I would suggest to call it --pretend for that reason.

Finally force: no does the right thing (even though now does a little polishing - we can discuss that). If we remove a force option, it's kind of a breaking change, we need to inform all beets users via the changelog, and still some won't read it, I don't have the best feeling about this.

I mean we can discuss again what would be the best default for keep_existing. I set it to yes, because I felt might be common to find that useful, which means, given that force: yes stays (as it always was), the change now is that additionally to new genres old ones will get thrown in the mix and handled. I just thought it might be a common and useful setting (since in my opinion lastgenre can not always be trusted to get something useful with rare music or artist with the same name for example).

Also, to make docs less confusing what do you think about formatting/structuring them similar to how I propose the new option in this PR's description headline "Details & Docs"

2. Types are missing in the new/adjusted functions: see the comments where I found a bug which would have been caught by the type checker

I simply ignored typing. Thanks for the hints. I think I catched all of the new ones. While I'm at it, I could even add them for all the existing ones. Or leave that for another PR, not sure.

Hmm but I do see a lot of mypy warnings in the github diff, so I guess I should handle all those?

3. Some simplifications in the code

Thanks so much for an amazing review, there was a lot of room for improvement. Your suggestions encouraged to me to touch things that I did not initially take the time to even understand entirely and I think a lot is improved now and thanks to a not bad test coverage in the plugin I'm quite sure I didn't introduce new bugs (but we'll see, there always are some ;-)

[JOJ0]

Am I hitting things you are addressing in #5588 @snejus ?