Add "Meta-Archives" collection of IDs

[Difegue]

Unlike categories, which loosely regroup IDs, Meta-Archives should hide the IDs they contain and expose themselves instead.

Adding this new element raises a ton of questions relative to the current architecture:

• How do tags work?
  • I think Meta-Archives should only have a dedicated title(and maybe artist), and aggregate metadata of all their contained IDs otherwise.
• Can Meta-Archives be put in Categories?
  • yes, and their contained IDs should be hidden.
• Can Meta-Archives be edited through Batch Tagging?
  • no, edit their contained IDs instead.
• Where do Meta-Archives land in the database?
  • Likely in the Redis root with their own prefix, like Categories currently do.
• How to integrate them to the Search Engine?
  • Search should be modified to use a precalculated table containing "all archives", which would be a mix of real and meta-IDs. (Instead of the current behavior where it grabs all IDs off the Redis root)
  • Creating this table would also allow us to use its count for the total, solving Incorrect total entries count #510 .
• How to integrate them to the Reader?
  • Opening a Meta-Archive should extract/open the first ID, and add a combobox to show all the entries contained in the Meta-Archive.
  • Reaching the end of the first ID opens the second, and so on.
• How to handle reading progression?
  • This might be a bit naive, but just showing the sum of all contained ID's reading progression and pagecount seems to be the easiest way.
  • This might require precalculating pagecounts for all archives instead of the current find-it-out-as-we-go approach.
• Should there be new API endpoints for Meta-Archives?
  • Yes for creating/managing them, no for everything else. (The current archive APIs should be enhanced to accept Meta-Archive IDs)

[CirnoT]

(and maybe artist)

I'd hate if it would mean that they can't inherit artist from their children, as it would make them that much harder for grouping magazine one-shots.

Search should be modified to use a precalculated table containing "all archives", which would be a mix of real and meta-IDs. (Instead of the current behavior where it grabs all IDs off the Redis root)

I assume this means that it would show meta archives only and not their children, is that correct?

[AbyssalMonkey]

Unlike categories, which loosely regroup IDs, Meta-Archives should hide the IDs they contain and expose themselves instead.

I'm fine with this as long as there is a way to expose the child IDs and have the child IDs be able to be a part of multiple meta-archives. Certain magazines release weekly (looking at you kairakuten) and it would be a pain in the ass to have duplicate IDs for each of them. So one ID should be in both the weekly and monthly aggregate. This also goes for artists who rerelease their works in multiple magazines.

[AbyssalMonkey]

How to integrate them to the Search Engine?

For filtering purposes, could it be possible to search for only meta-archives, and by extension, IDs not included in them (negative search)?

[Difegue]

This is officially in progress thanks to #818. 🎊
Meta-Archives are now Tankoubons, and the creation/management APIs are now available in nightlies.

Remaining work to consider this issue closed:

• Hide IDs that are in a Tankoubon, and only show the Tank in the Search Index
  • I'm not really planning to allow for IDs to be both in tanks and in regular searches, but you can add one ID to multiple tanks no problem.
• Have the tank come up when searching for tags that are in any of the included IDs (basically, include the tank ID in every tag index its included IDs contain)
• Having the reader be able to handle groups by opening the first ID, then the subsequent ones when done reading
• Cleaner tank management UI
• Handling reading progress in a tankoubon
• Modify the Category APIs to handle Tankoubon IDs (TANK_xxxxxxx) alongside regular archive IDs

[EfronC]

Thinking a little about the first 2 points: Are they really necessary? I mean, I think they will probably be very problematic to implement because:

Tanks, while being made mainly for Tankoubons, can be used for multiple types of aggroupations, which include non-related works, for example, imagine that someone have a few tanks called: "Pending to read", "Favorites", "Recommended works", etc. In this case you could have 1 same file in every one of them, and now, let's take the problem to the edge and think that, for some reason I wouldn't know why but it's possible with the current back, someone have 1 same file in 100 Tanks: What would happen when you search for the file in the search index? Will you show the 100 Tanks(Which would mess the view), or will you choose one of them and show it(Which would make it not work as expected, because for example, if you have a "Favorites" tank and a related works tank, you would expect the second to be the one to be shown, but if the choosen happens to be the first one, it will miss its goal).

What I believe could be done here, is not to merge the search API with tanks, and instead just have them as separate functionalities. In tanks there is already an EP to get the tanks that a file is on, so maybe that could be used to add a button in the context menu of the search index that can be used to call it, and in a second view(Or a modal) just show a list of them(Whether with one thumb or a list of names), and then you can click it and open it. Something like this:

Or in a different approach, an intermediate view between clicking the thumb in the index view and the reader, like the one in Ichaival, could be implemented, and in that view you could put at the bottom a section(Maybe even a carrousel) called "Works featured in:" and then provide the list of Tanks that returns the EP.

I'm not sure, I just think that merging the search API with tank API has the potential to break something very easily, so I'm leaving those 2 ideas just in case.

[Difegue]

My thoughts were that for use cases like pending to read/favorites, you would use categories instead. (which should be expanded to contain tank IDs as mentioned)

If you have the same id in 100 tanks and search for it you would indeed get 100 entries, but that'd be working as intended for me.
We could block IDs from being in more than one tank, but I don't think this is a common enough scenario to warrant bothering with.
(also for people who'd want to sort stuff like weekly/monthly kairakutens as mentioned above, ID duplication is something they might actually want)

[Difegue]

Tanks don't really have much reason to exist currently if all they do is "categories but ordered" in my opinion, the goal is more to declutter large archive indexes by allowing users to view stuff like manga series with multiple chapters as one item.

Mixing them in search results also helps third-party clients (and the webreader) with their reading implementation as there's way less API to implement - I believe they'd only have to implement the call to get the IDs of a tank to get a working reader. (or i could implement a shim to the extract api to handle tanks 100% transparently, we'll see ig)

The current tank management endpoints will remain as they currently are if they want to add support for those later down the line, ofc.

[CirnoT]

(also for people who'd want to sort stuff like weekly/monthly kairakutens as mentioned above, ID duplication is something they might actually want)

^ Exactly this

the goal is more to declutter large archive indexes by allowing users to view stuff like manga series with multiple chapters as one item.

I'd consider adding option to disable tank grouping in search, in case someone wants to quickly search for very specific entry. It would also be very useful when searching by source namespace which includes exact link for E-H.

For example use case, my little script that checks whether E-H archive is added to LRR would work better if it would be able to request search without tank grouping, as then it would be able to link directly to specific entry in LRR instead of navigating to first page of tank.

[AbyssalMonkey]

(also for people who'd want to sort stuff like weekly/monthly kairakutens as mentioned above, ID duplication is something they might actually want)

^ Exactly this

the goal is more to declutter large archive indexes by allowing users to view stuff like manga series with multiple chapters as one item.

I'd consider adding option to disable tank grouping in search, in case someone wants to quickly search for very specific entry. It would also be very useful when searching by source namespace which includes exact link for E-H.

For example use case, my little script that checks whether E-H archive is added to LRR would work better if it would be able to request search without tank grouping, as then it would be able to link directly to specific entry in LRR instead of navigating to first page of tank.

This all, exactly. I want it to organize all the little archives I have into books, and then I can shove those books into categories. Ordered categories might be useful at that point because I could then order those books in the category by release for things like "kairakuten", "dascomi", "beast", etc. But hey, I could probably make meta-meta archives for this too if the system is robust enough.

If it doesn't have ID duplication, the feature is a bust because then I need to duplicate my archives anyway; once for the serial release (anthology), and once for the artist compilation. I want it in two places because it goes in two places. This isn't a library, its a database, and we should be treating it as such.

[EfronC]

I'd consider adding option to disable tank grouping in search, in case someone wants to quickly search for very specific entry. It would also be very useful when searching by source namespace which includes exact link for E-H.

For example use case, my little script that checks whether E-H archive is added to LRR would work better if it would be able to request search without tank grouping, as then it would be able to link directly to specific entry in LRR instead of navigating to first page of tank.

Yes please, actually, I think this would be the best intermediate, in case you're someone that likes to create groups for everything, an option to just have the search API not to replace for the tank, but to show the individual files.

Having to search for an specific archive, but then not being able to find it easily because it comes inside another was another problem I was thinking, just didn't mentioned it, but if the option to group the archives in Tanks was a toggleable in the search Index, I think it would be the best option to meet both sides.