database / backend storage fragility safeguards

[mrpops2ko]

is your feature request related to a problem? Please describe.
i have my storage (nas) and copyparty docker container on separate computers. the indexing db, if it encounters that the storage is down (say a host critical failure / kernel panic or something similar)

will then dump the entire database, if this happened during a normal up scan (i.e. if copyparty was just starting)

this in turn them causes you to have to rescan everything from scratch, the process of which i'd been doing for the past week until i ran into some kind of kernel failure / storage issue

at which point it dropped the entire db again, i don't have the docker log anymore of it doing so but if it happens again i'll capture it

Describe the idea / solution you'd like
some kind of chunking / incremental 'saves' and a flag which doesn't drop the db or just has some kind of awareness baked into it for this kind of storage fragility. maybe periodically save DB snapshots or deltas, making recovery from the last known-good state faster?

something like the DB to support "partial re-index," restoring only directories that have changed since the last snapshot, based on timestamp checks and/or hashes?

or instead it could behave something like upon detecting storage is gone (or a significant amount of files all disappear at once, which could be used as a proxy to signify more than just regular file deletion), it waits until it sees the storage again with those files and then starts to scrutinise them more (in terms of doing increased hash check focus)

or it could instead of marking them as deleted, silently mark them as missing until the next [n] reruns of copyparty, at which point it can partial / in depth rescan them or flag them as fully gone?

Describe any alternatives you've considered
disabling the db entirely, but i don't think thats a good idea - the db makes directory traversal snappy and search performant, i would like to keep the db given a choice

maybe before initiating a rescan or at Copyparty startup or mid scan, perform a storage health check? try a simple test read from the root of the storage and if its all gone, then that is a good signifier that storage failure has occured rather than the admin deleting everything

If the test fails, postpone the scan and flag an error rather than acting on incomplete storage?

[9001]

quick summary for anyone else reading this:

the filesystem index db is usually (and by default) stored inside the volume, in a subfolder named .hist next to all your files. This helps ensure that it stays in sync. But since the files are on another machine, mrpops2ko configured copyparty to put the database on a local disk instead. This is a good choice because it improves performance greatly, at the slight risk of something like this happening.

---

back to the issue at hand --

there is currently a safeguard for this exact situation, and I'm curious why it didn't trigger as intended:

if Copyparty finds that a volume is entirely empty during startup, then it will skip the startup-scan for that volume with the following message:

volume /nas at [/mnt/nas] is empty; will not be indexed as this could be due to an offline filesystem

this check assumes that any remote storage is its own volume, and not mounted as a subfolder inside a bigger volume. So let's say your nas is mounted at /mnt/nas, then something like:

[global]
  e2dsa  # enable filesystem indexing on startup

[/]
  /mnt
  accs:
    r: *

[/nas]
  /mnt/nas
  accs:
    r: *
  flags:
    hist: /var/copyparty/hist-nas  # a local path for the database

...would be okay, since then /mnt/nas is a separate volume. And creating separate volumes for any number of folders inside the nas mountpoint would also be OK:

[global]
  e2dsa
  hist: /var/copyparty/hists  # keep all databases inside this local path

[/]
  /mnt
  accs:
    r: *

[/nas]
  /mnt/nas
  # no accs section, so all access is forbidden -- this just prevents copyparty from indexing the files inside /nas, if the usecase is that we don't want to share all of it

[/music]
  /mnt/nas/music
  accs:
    r: *

But, if you only have the [/] volume, so that Copyparty sees local files next to the remote storage, then this safeguard would fail, and wipe the database like you're experiencing. Could this be it?

you have some good ideas! but want to verify this before we continue :>

specifically, I'm not opposed to adding some sort of automatic backup of the databases on a configurable schedule (nightly/weekly), keeping the last N copies for manual intervention if something goes wrong -- but I'm wary of making the recovery happen automatically, since heuristics are usually a recipe for unexpected disasters.

and one more idea for the road -- perhaps an option to specify one or more filenames to look for during startup, and the database for that volume would get entirely disengaged if one of those files are not present? for example some files/folders which you know for certain will always be present, or just something like .fs-ok

and hope you didn't lose any data btw!

[mrpops2ko]

hmm it might not have triggered through my layers of abstraction, or because of the abstraction

so i run proxmox, which runs an LXC, which runs docker, which runs copyparty

to share my nas storage to my other machine (small form factor), i use NFS and the NFS is mounted on the SFF proxmox machine at the host level

I then passthrough that NFS path on the host, to the LXC container using

mp0: /mnt/dkr,mp=/mnt/dkr,mountoptions=noatime
mp1: /mnt/user/data,mp=/mnt/user/data,mountoptions=noatime

and then i pass that through to the copyparty docker

what can happen when the NFS storage is disconnected, is that you get some stale file handle messages too - i'm not sure how copyparty handles those

ai mentions

Why Does This Happen After Storage Disconnection?
When you disconnect the underlying storage (such as a physical disk, a SAN/LUN, or an NFS-exported volume), here’s what happens:

NFS operates using file handles: The NFS protocol uses file handles as persistent references to files and directories. Each file handle corresponds to a specific inode on the filesystem exported by the NFS server.

Handles become invalid: If the storage is disconnected, the underlying filesystem can no longer provide access to the inode. If the file or directory is deleted, unexported, or the volume is unmounted and remounted (with different inodes), the previously issued file handles stored on the NFS clients no longer point to valid objects.

Client attempts to access old handles: When the client tries to access something using an invalid (now "stale") file handle, the server cannot resolve the handle to an object, resulting in a stale file handle error.

Typical Scenarios Leading to Stale File Handles
Unmounting and remounting storage on the server while clients are still connected.
Removing or moving files/directories on the server that are still open or referenced by clients.
Server reboot or failover after a storage change.
Underlying storage (e.g., EBS, SAN LUN, iSCSI disk) disconnected or made unavailable.

How to Resolve the Error
Remount the NFS share on the client.
Ensure all operations on the NFS server (like unmounting storage or deleting files) are coordinated with connected clients.
Avoid unexporting/unmounting NFS volumes in use.

but its a little harder for me to remount, and its such a rare issue that i just restart the whole machine

in terms of my config, i do all the folder management at the docker level as bindmounts (which is also abstracted from the LXC because its a mount point on the host)
my share is

# create volumes:
[/]         # create a volume at "/" (the webroot), which will
  .         # share the contents of "." (the current directory)

but the db location is a real local disk

    volumes:
      - /mnt/dkr/appdata/cpp/cpp/:/cfg
      - /mnt/dkr/appdata/cpp/db/:/db  # the copyparty config folder

regarding the storage issue, i suspect its not fully to do with storage at all but due to the change of nics - i recently bought 2x connectx 5 cards and wanted to play around with all the fun switchdev ASAP2 DOCA offloading

but this doesn't seem very stable outside of the very specific and nvidia vetted use cases - so i'm trying to get it to a working state without kernel panics... its not going so well 😭

[9001]

ah yep, that sounds plausible -- think I'm leaning heavily towards adding that option now, just trying to think of a good name for it... eh, landmark is fine :p

what can happen when the NFS storage is disconnected, is that you get some stale file handle messages too

right, it would be good to make it read a bit of each of the landmark files to ensure that the filesystem is actually healthy; that should correctly detect such situations. Maybe it should even be possible to tell copyparty what contents it should expect when reading each file, since some FUSEs might return some bogus data if something's funky...

been a while since I used NFS, but I think that when a process tries to read from a struggling NFS-share, then the entire process could possibly deadlock for a looong time, until the NFS-share itself actually times out -- but that's arguably an advantage, since it makes it very obvious that something is wrong.

to make that situation more obvious and less confusing, there will be a log-message right before it pokes each of the files, so you can tell why it's dead.

dayjob's pretty lively this week but I should have my head back in working order come weekend, so maybe I'll have something ready by sunday :>

recently bought 2x connectx 5 cards and wanted to play around with all the fun switchdev ASAP2 DOCA offloading

oh man, good luck 😁

[9001]

I think the landmarks feature is good to go -- hoping for a new version tonight or tomorrow. I'll keep this issue open to add automatic database backups at some point later; don't think now is the time :>

for now the safest approach is probably to shutdown copyparty, then create a .tar.gz of up2k.db* for archival somewhere safe -- that way you'll get a nice coherent snapshot in case the landmarks somehow fail to detect a bad fs