Request for Additional Information Regarding Thumbnail Configuration

[armouredking]

Currently doing a file move so my CP instance is in the midst of a several hour scan operation and stressing the CPU so don't have anything I can test with atm.

With regards to the functionality discussed here, is ext-th similar to th-covers where you can specify multiple mime-types in a comma separated list, or does ext-th need to be an array / multi-line entry?

i.e. would it be something like:

ext-th: exe=/path/to/exe.icon md=/path/to/md.icon ...

OR

ext-th: exe=/path/to/exe.icon,md=/path/to/md.icon,...

OR

ext-th: exe=/path/to/exe.icon
ext-th: md=/path/to/md.icon
...

Additional question: what is an acceptable thumbnail file? Because a lot of downloadable thumbnail packs (like the stuff you can get for OS - particularly Linux - themes) use svgs, would that be allowed?

[9001]

thanks for the heads-up, the readme should be better now 👍

I'm suspecting that .icon is the same as .ico, and in that case I'd recommend renaming them to .ico since that's the more common extension (greater chance of mapping to the correct mimetype).

I've closed the issue but go ahead and ask if there's anything else :>

[armouredking]

So I can't make this work. I don't see any errors in the logs (but every time I restart it's scanning more files, even though it finished... not sure why, I'm guessing it didn't actually do the full drive even though it said it did, is there a timeout on scan?)

Relevant Config:

[global]
  e2dsa  # enable file indexing and filesystem scanning
  e2ts   # enable multimedia indexing
  ansi   # enable colors in log messages
  re-maxage: 86400    # rescan every day
  hist: /cfg/hists/  # pull all copyparty thumbs into a single location
  dbpath: /cfg/dbs/  # pull all copyparty sqlite dbs into a single location
  xff-hdr: x-forwarded-for
  xff-src: 172.0.0.0/8
  ver              # show copyparty version in the controlpanel
  # Create our own little MIME setup
  ext-th: 7z=/.icons/ark.svg
  ext-th: rar=/.icons/ark.svg
  ext-th: zip=/.icons/ark.svg
  ext-th: tar=/.icons/ark.svg
  ext-th: bz=/.icons/ark.svg
  ext-th: gz=/.icons/ark.svg
  ext-th: doc=/.icons/word.svg
  ext-th: docx=/.icons/word.svg
  ext-th: epub=/.icons/epub.svg
  ext-th: exe=/.icons/exe.svg
  ext-th: html=/.icons/html.svg
  ext-th: java=/.icons/java.svg
  ext-th: json=/.icons/json.svg
  ext-th: md=/.icons/md.svg
  ext-th: ods=/.icons/excel.svg
  ext-th: pdf=/.icons/pdf.svg
  ext-th: php=/.icons/php.svg
  ext-th: py=/.icons/python.svg
  ext-th: sh=/.icons/shell.svg
  ext-th: sql=/.icons/sql.svg
  ext-th: ttf=/.icons/ttf.svg
  ext-th: txt=/.icons/txt.svg
  ext-th: xls=/.icons/excel.svg
  ext-th: xlsx=/.icons/excel.svg
  ext-th: xml=/.icons/xml.svg
  ext-th: yaml=/.icons/yaml.svg
  ext-th: zim=/.icons/zim.svg

[/]
  /tank
  accs:
    rwmd.: *

Stripped the logins out, but otherwise, this is all there is. The folder ".icons" is located under [/] (/tank) on the docker volume, and has the correct permissions (I can see the files by allowing view of dots, or just typing in the address)

[armouredking]

Okay, so interesting bit, it won't do it to all of them, only the scanned files. If I go hunting, I can find folders that have partials. Here's an example:

The tail of the log shows that it's 'finished' with it's scan after startup:

00:38:28.918 up2k                  commit 0 new files; 4 updates
00:38:28.966 up2k                  uploads are now possible
00:38:28.966 up2k                  online (reading tags) [/tank]
00:38:28.966 up2k                  online (reading tags) [/wiki]
00:38:28.968 up2k                  2 volumes in 483.47 sec
00:38:28.969 up2k                  mtp finished in 0.00 sec (0:00)
------------------------

So, a few new questions:

1. How to force it to go and not stop until I say so, or it's done done?
2. Why does using the SVGs result in the ugliness here? Specifically with file names? As SVGs, they should resize to fill the 'preview' box no? Not sure what's happening here.

Shouldn't we have the full box? Because otherwise you get a mess.

---

Edit: Random other note, again because I'm looking at logs too much:

01:21:30.420 192.168.139.54 50338  GET  /?ls&rtt=27 @*
OPYTHAT
01:22:20.741 hsrv                  ok bye
01:22:20.741 tcpsrv                ok bye
01:22:20.741 up2k                  writing snapshot
01:22:20.793 root                  nailed it

Looks like the goodbye COPYTHAT is missing its leading C. I thought maybe it was getting cutoff in my logs but I've seen it a few times.

[9001]

gotcha -- I think I have explanations for all of these,

every time I restart it's scanning more files, even though it finished

on startup, it does a quick scanthrough of all the folders, looking for new files (or deleted files). If necessary, it does this by exactly comparing the database to the contents on the disk, but there is also an optimization (default-enabled) which instead compares a hash of the contents of each directory, instead of individual files. This optimization saves quite a bit of time -- but, as a safety measure, if copyparty is terminated mid-startup, it will disengage that optimization for the next startup only. So it should start faster the next time if you let it finish scanning before terminating it.

as for the thumbnails not applying to all files, this is primarily due to copyparty trying to identify file extensions such as "tar.gz". Specifically, it thinks that "export.json" is the file extension, and not just "json". Currently the limit is two words, each of 7 characters or less, applies both serverside and clientside, so the file extension of foo.bar.qwertyu.asdfghj would be assumed to be qwertyu.asdfghj.

question 2, the text getting messed up and not filling the whole box, is due to two reasons -- the svg has a native size in its metadata, so the browser renders it at that size, instead of filling as much space as possible (which the raster images generally do, having a resolution larger than its shrunk-for-presentation size). Copyparty doesn't currently have any logic for detecting and mitigating this. This leaves a lot of extra space, making the next issue more obvious:

The text getting truncated to 3 lines is working as intended, and that's to prevent excessively long filenames making the entire row taller. The max number of lines to allow is the "chop:" option in the toolbar. It's not smart enough to realize that there's enough space to include some more lines, and maybe that's possible to do something about -- I'm not entirely sure yet. And the thing about seeing a tiny bit of the 4th line at the bottom, that's probably some jank in my line height calculation, I'll look into that.

I can think of two workarounds; one is to convert all the svg files into png's at a higher resolution (512 or so), or alternatively you could force all images loaded from the "/.icons" folder to fill the available space with the following custom css:

#ggrid img[src^="/.icons/"] {
  height: calc(var(--grid-sz)/1.25);
}

finally and most importantly...

Looks like the goodbye COPYTHAT is missing its leading C

hehe, so the reason that's there is because the terminal itself prints the ^C when you hit ctrl-c, shifting all the text on the next line by two characters... so that's why copyparty prints "OPYTHAT\n" to put the ^C to good use 😁 so the ^C part doesn't make it into the logs since it's not something that copyparty is emitting, but I think we can accept that :>

[armouredking]

on startup, it does a quick scanthrough of all the folders, looking for new files (or deleted files). If necessary, it does this by exactly comparing the database to the contents on the disk, but there is also an optimization (default-enabled) which instead compares a hash of the contents of each directory, instead of individual files. This optimization saves quite a bit of time -- but, as a safety measure, if copyparty is terminated mid-startup, it will disengage that optimization for the next startup only. So it should start faster the next time if you let it finish scanning before terminating it.

That's not what's happening though. I'm always letting it finish (getting to the code below), it's just that it takes different amounts of time every reboot. I don't remember this behaviour before, but I also wasn't rebooting it a lot to tinker with before, either. The initial, fresh install startup for this instance took ~11.5k seconds to complete. I remember because, like I said in the first post, it was going for a few hours.

I always let it finish before tinkering. Every reboot after this one required 400~500 seconds to startup. So, it didn't finish scanning with the 11.5k one -- and it should have, or there ought to be an option that we can do to force a preload (kind of like how the NextCloud preview function works) that doesn't stop until it's done done. I'm not sure if the re-maxage setting is the solution here or not.

Just to play with this more (and because whenever I do transfers like this I go through and prune a lot of duplicate files and the like that hopefully CP will help alleviate in the future... I didn't do that with the podman instance before, because I was being lazy and the one thing I do miss about using Synology is the duplicate file finder), I did my pruning and restarted it again overnight. This time, it ran for:

06:18:14.381 up2k                  forgetting 6802 deleted dirs, 108962 files
06:18:23.587 up2k                  forgetting 3111 deleted covers
06:18:23.595 up2k                  commit 1010 new files; 1093 updates
06:18:23.657 up2k                  uploads are now possible
06:18:23.657 up2k                  online (reading tags) [/tank]
06:18:35.647 up2k                  discarded media tags for 41318 deleted files
06:18:39.970 up2k                  using 8x ffprobe
06:18:40.045 up2k                  online (reading tags) [/wiki]
06:18:42.149 up2k                  2 volumes in 3421.58 sec
06:18:42.150 up2k                  mtp finished in 0.00 sec (0:00)
------------------------

To be clear, I never interrupt it until it gets to this point, which should be the "I'm done" part, no? And yet, on reboot, it needs to scan more files. It didn't do this before, or at least, I don't remember this behaviour before.

I know it's a lot of files to scan -- hence my looking for a "please do run all night long and finish everything, thanks" option. Especially since with these big runs, (per picture above) you can see that it hammers the CPU with all the FFMPEG going on. I'm fine with this (as I'm the only one using it), but forcing it to happen at night would be preferred. I couldn't turn off the scanning either, since I needed it to find the files to test with.

I suppose that's the simplest solution, just hold off on the file-scan option in the header until the boot at night, but that assumes it finishes the scan, and it doesn't look like it's finishing the scan even when it goes for a long while.

as for the thumbnails not applying to all files, this is primarily due to copyparty trying to identify file extensions such as "tar.gz". Specifically, it thinks that "export.json" is the file extension, and not just "json". Currently the limit is two words, each of 7 characters or less, applies both serverside and clientside, so the file extension of foo.bar.qwertyu.asdfghj would be assumed to be qwertyu.asdfghj.

Does the ext-th option allow for wildcarding? Because that would be really helpful. Even a simple ?json or tar? (I don't think it's necessary to go crazy, but hi2u unnecessary JPEG extensions) would be good, since right now with it looking at the last two dots that means .tar and .tar.gz are two different file types (and sure, they are, but for the purposes of slapping a thumb on them, they're not). The reason this option is desirable to me is because seeing text above text is not what I'm used to doing when scanning for a specific thing in a folder. Without needing to use the (admittedly very nice but still needs a separate action to use) search feature all the time. I want a quick-scan file-find and my brain is trained to look for an icon+text, not for text+text.

finally and most importantly...

Looks like the goodbye COPYTHAT is missing its leading C

hehe, so the reason that's there is because the terminal itself prints the ^C when you hit ctrl-c, shifting all the text on the next line by two characters... so that's why copyparty prints "OPYTHAT\n" to put the ^C to good use 😁 so the ^C part doesn't make it into the logs since it's not something that copyparty is emitting, but I think we can accept that :>

Ah, so Docker does ^C as part of the end? I'm not doing ^C, just to be clear.

[9001]

yep, assuming you're running with e2ds or e2dsa, which you have confirmed that you are, then it will always do a recursive directory listing on startup to ensure nothing was added/removed in the meantime. If you know you haven't added any new files out-of-band since the last restart, then switching to e2d to skip the on-boot scan is entirely fine. So,

The initial, fresh install startup for this instance took 11.5k seconds to complete.
Every reboot after this one required 400-500 seconds to startup

given enough folders, that's not entirely unreasonable -- the startup time should be comparable to find /mnt/tank -ls | sha1sum , or perhaps twice as long due to the overhead of python. If the whole directory tree is still cached in RAM (which it should be, for a long time, after a scan) then the next time will be much faster, which could explain the variance you're seeing too.

what it does is list everything recursively, stat()ing all the files, and creating a checksum of the contents of each folder, to see if any new files have appeared -- so that's not entirely free.

but this is still assuming that my theory holds true, that it isn't somehow discovering any "new" files, or forgetting any "deleted" files (which it really shouldn't, and I don't think it is). Essentially, it should not be hashing file contents, or spawning ffprobes for tag-scanning. I'd expect seeing a single cpu core getting a workout from copyparty, in addition to the load produced by zfs itself.

so to confirm, yes, every time it says "uploads are now possible" then it is done indexing all of the files, and "mtp finished" means it is also done indexing metadata. And yes, this does mean that it is done done; if it suddenly finds more new files after a restart then there might be something funky with the scanning logic.

however, it is safe to kill the process at any time, even mid-scan; it will pick up where it left off -- but it might take a little bit to get there, since it needs to disengage the previously mentioned optimization to figure out where that was exactly. It does keep track of all the files which have been tag-scanned so far, and which ones still need to be ffprobed after a restart.

To be clear, I never interrupt it until it gets to this point, which should be the "I'm done" part, no? And yet, on reboot, it needs to scan more files. It didn't do this before, or at least, I don't remember this behaviour before.

yep; at that point it should have entirely finished indexing everything. What does the logs look like afterwards? I'm expecting:

• commit 0 new files; N updates -- where N is not necessarily zero, if that is even printed at all
• uploads are now possible
• online (reading tags) [...]
• mtp finished in 0.00 sec (0:00)

...which confirms it was just the expected cursory check for new/gone files. However if it starts mentioning forgetting N deleted dirs/files and/or discarding media tags for N deleted files, then that's more of a reason for concern.

I suppose that's the simplest solution, just hold off on the file-scan option in the header until the boot at night, but that assumes it finishes the scan, and it doesn't look like it's finishing the scan even when it goes for a long while.

the idea of making it possible to defer the scanning to "some other time" is interesting though; that way you could run it with e2d to skip the onboot scan entirely, and maybe there could be an api-call you could issue to perform a full rescan when the time is right -- perhaps on a cronjob or something. I'll give that a shot.

Does the ext-th option allow for wildcarding? Because that would be really helpful.

I think making it realize that the thumbnail you've defined for "gz" should also apply to "tar.gz" is a fair assumption, so I'll get that added.

And spinning off that for a bit, since you're adding a lot of filetypes at this point, maybe a new option to automatically map all icons found inside a given folder would be good? For example ext-th-d: /.icons would find exe.svg inside that path and automatically ext-th: exe=/.icons/exe.svg ?

Ah, so Docker does ^C as part of the end?

yes, no, kind of -- it's the same signal, so the program can't really tell the difference :>

[armouredking]

but this is still assuming that my theory holds true, that it isn't somehow discovering any "new" files, or forgetting any "deleted" files (which it really shouldn't, and I don't think it is). Essentially, it should not be hashing file contents, or spawning ffprobes for tag-scanning. I'd expect seeing a single cpu core getting a workout from copyparty, in addition to the load produced by zfs itself.

so to confirm, yes, every time it says "uploads are now possible" then it is done indexing all of the files, and "mtp finished" means it is also done indexing metadata. And yes, this does mean that it is done done; if it suddenly finds more new files after a restart then there might be something funky with the scanning logic.

however, it is safe to kill the process at any time, even mid-scan; it will pick up where it left off -- but it might take a little bit to get there, since it needs to disengage the previously mentioned optimization to figure out where that was exactly. It does keep track of all the files which have been tag-scanned so far, and which ones still need to be ffprobed after a restart.

To be clear, I never interrupt it until it gets to this point, which should be the "I'm done" part, no? And yet, on reboot, it needs to scan more files. It didn't do this before, or at least, I don't remember this behaviour before.

yep; at that point it should have entirely finished indexing everything. What does the logs look like afterwards? I'm expecting:

* `commit 0 new files; N updates` -- where N is not necessarily zero, if that is even printed at all

* `uploads are now possible`

* `online (reading tags) [...]`

* `mtp finished in 0.00 sec (0:00)`

...which confirms it was just the expected cursory check for new/gone files. However if it starts mentioning forgetting N deleted dirs/files and/or discarding media tags for N deleted files, then that's more of a reason for concern.

It does look like that from the instances I've seen, but it's never 0 for new files or updates.

I think I now know what's happening here, though. It's never 0... and it wouldn't be, because of .stfolder. I've been moving away from using SMB and utilizing just the CP interface itself, so the only other thing that is touching anything is SyncThing (My instance is local at home, but for backups I have a CoLo server and it is doing things in the background).

root@HKH4U:~# zfs list
NAME                              USED  AVAIL  REFER  MOUNTPOINT
nexus                            9.31G  1.79T   104K  /mnt/nexus
nexus/Containers                 9.31G  1.79T  3.57G  /mnt/nexus/Containers
nexus/Containers/copypartydbs    73.4M  1.79T  73.4M  /mnt/user/Containers/appdata/cpapp/dbs
nexus/Containers/copypartyhists  22.7M  1.79T  22.7M  /mnt/user/Containers/appdata/cpapp/hists
nexus/Containers/mariadbs        1.32G  1.79T  1.32G  /mnt/user/Containers/appdata/mariadb/databases
nexus/Containers/marialog        4.32G  1.79T  4.32G  /mnt/user/Containers/appdata/mariadb/log
vaults                           1.41T  3.87T   186K  /mnt/vaults
vaults/Books                      140K  3.87T   140K  /mnt/vaults/Books
vaults/Cabinet                    140K  3.87T   140K  /mnt/vaults/Cabinet
vaults/Calibre                   47.3G  3.87T  47.3G  /mnt/vaults/Calibre
vaults/Library                   1.34T  3.87T  1.34T  /mnt/vaults/Library
vaults/Manga                     21.1G  3.87T  21.1G  /mnt/vaults/Manga
vaults/Wiki                      99.4M  3.87T  99.4M  /mnt/vaults/Wiki

This is the actual arrangement. For the Docker container:

   volumes:
      - /mnt/user/Containers/appdata/cpapp:/cfg
      - /mnt/user/Library:/tank
      - /mnt/user/Wiki:/wiki

Unraid (and uCore) are ephemeral, so they can't do the typical ZFS /tank thing. They have to mount in /mnt or /var instead, so for the sake of keeping everyone equal (between uCore which I still use for my HDD NAS and Unraid which I'm using for the Flash NAS) I keep the paths in the containers the same. The reason for differentiating between Library and Wiki is so that I can set the daw option on the latter folders. Just realized they got cut out of my C&P earlier, but it's the same thing with the extra flag to allow editing the tiddly wikis direct in browser.

So, probably the most correct thing to do would be the exclude-patterns bit from the readme for the .stfolder instances? I'm guessing that to do so globally, I'd need to set something like this under [global]?

  noidx: .stfolder

For a folder, does it need the escape before the dot like with the ISO example? Would it need the search exclude given that the indexing wouldn't be capturing those files anymore?

the idea of making it possible to defer the scanning to "some other time" is interesting though; that way you could run it with e2d to skip the onboot scan entirely, and maybe there could be an api-call you could issue to perform a full rescan when the time is right -- perhaps on a cronjob or something. I'll give that a shot.

A cron, systemd, or even just right in. How does the call for re-maxage work? Seems simplest to just modify that so it has a delay. Granted, if you wanted it to happen by clock, then cron or systemd is likely best.

Does the ext-th option allow for wildcarding? Because that would be really helpful.

I think making it realize that the thumbnail you've defined for "gz" should also apply to "tar.gz" is a fair assumption, so I'll get that added.

And spinning off that for a bit, since you're adding a lot of filetypes at this point, maybe a new option to automatically map all icons found inside a given folder would be good? For example ext-th-d: /.icons would find exe.svg inside that path and automatically ext-th: exe=/.icons/exe.svg ?

That would probably work and be a lot less trouble overall... but is it going to be bothered about the dupes? Like, for instance, I went ahead and swapped them over to PNGs to fix the size, but you'll note I'm using the same image for multiple archival mimes:

  ext-th: 7z=/.icons/ark.png
  ext-th: rar=/.icons/ark.png
  ext-th: zip=/.icons/ark.png
  ext-th: tar=/.icons/ark.png
  ext-th: tar.bz=/.icons/ark.png
  ext-th: tar.gz=/.icons/ark.png

Using a folder, they'd all need to be named like 7z.png, rar.png, zip.png, tar.png, tar.bz.png
For that last guy, it's got two dot names... is that also going to make things unhappy?

And back to the earlier thing with the doubles, sometimes you don't have doubles at all but version numbers:

It seems to know it's a zip file, at least, it doesn't print all the numbers in the above text-box (like it did with export.json). But it also doesn't show the zip icon. Then there's the files I've told it what to do, and it ignores me:

  ext-th: jutoh=/.icons/jutoh.png
  ext-th: jutohbak=/.icons/jutohbak.png
  ext-th: md=/.icons/md.png

At least it's consistently inconsistent. The *.jutoh files with no numbers do get the picture, but the *.jutohbak files just show up as unk. There's not a single *.jutohbak (the writing software I use creates backups when you open a file, and can also create snapshots -- though the snapshots don't use the *.jutohbak filetype, they do use timestamps in the filename which creates the dot problem again) that has the picture that it has been assigned.

[9001]

Nice! I'll respond to the rest later, but:

So, probably the most correct thing to do would be the exclude-patterns bit from the readme for the .stfolder instances? I'm guessing that to do so globally, I'd need to set something like this under [global]?

assuming you want to ignore the folder .stfolder anywhere and everywhere it may appear in the filesystem, be it at the toplevel or in a subfolder, as well as recursively forget everything that is inside each .stfolder, then the following will work:

[global]
  no-idx: /\.stfolder$

For a folder, does it need the escape before the dot like with the ISO example?

it's a regex, so yep

Would it need the search exclude given that the indexing wouldn't be capturing those files anymore?

nope! they'll get purged from the database on next rescan, so they won't show up in search either regardless :>

[armouredking]

Nice! I'll respond to the rest later, but:

So, probably the most correct thing to do would be the exclude-patterns bit from the readme for the .stfolder instances? I'm guessing that to do so globally, I'd need to set something like this under [global]?

assuming you want to ignore the folder .stfolder anywhere and everywhere it may appear in the filesystem, be it at the toplevel or in a subfolder, as well as recursively forget everything that is inside each .stfolder, then the following will work:

[global]
no-idx: /.stfolder$

For a folder, does it need the escape before the dot like with the ISO example?

it's a regex, so yep

Would it need the search exclude given that the indexing wouldn't be capturing those files anymore?

nope! they'll get purged from the database on next rescan, so they won't show up in search either regardless :>

Yeah, it'll need to be everywhere since I don't copy the full kit to the CoLo server, just the really important stuff. I don't want to have to remember to change it -- like how above I renamed copyparty to cpapp to keep from doubling up on copyparty/copyparty in the path and forgot to change the mount points. At least I didn't run afoul of that problem since it's only been a day, but that could have been another fun TS&R session.