Build nix packages from source #253

toast003 · 2025-07-29T01:40:47Z

Removed the u2c and partyfuse packages since the main copyparty package has them now.

This PR complies with the DCO; https://developercertificate.org/

toast003 · 2025-07-29T01:42:22Z

contrib/package/nix/copyparty/default.nix

@@ -73,25 +79,13 @@ let
    ++ lib.optional withBasicAudioMetadata mutagen
    ++ lib.optional withHashedPasswords argon2-cffi
    ++ lib.optional withZeroMQ pyzmq
-    ++ (extraPythonPackages ps)
-  );
+    ++ (extraPythonPackages python.packages);


I'm not sure if I understand how this worked, so if someone could test it out I'll appreciate it

That seems correct. We don't strictly need it anymore, since it was only used in the partyfuse package, but it's nice to have an escape hatch like this. Maybe @dtomvan can confirm.

Actually that's still useful for using hooks with additional dependencies, so we keep it.

python.packages doesn't exist
I believe it's called python.pkgs?

True, not sure where I got python.packages from, sorry 😅

Well, I was planning to use it for hooks which might require other arbitrary dependencies, like plyer for notifications, but yes ps was originally just the python package set when you do python3.withPackages, so python.pkgs is equivalent here.

Usage is like this: copyparty.override { extraPythonPackages = (ps: with ps; [ plyer matplotlib ]); }

chinponya · 2025-07-29T05:49:13Z

Thanks for working on this. It has been bothering me since I realized that the release source isn't just a copy of the repo and does actually contain the built frontend dependencies (which was the final argument for using sfx.py for the package). We never got around to changing it due to it being in the process of merging into nixpkgs, putting into question the future of our version of the package (I still don't know what's the right move here, but I digress).

Only a few things stand out to me:

partyfuse package had an additional dependency that's not present in the default copyparty package now (pyfuse), so it's broken. Perhaps it should be included by default.
The update script no longer supports using the local release file (and make assumptions about the future release url), but looking at the commit order during a release, that feature seems unused (right, @9001 ?). We might be better off removing it completely instead of trying to update it to work with release source archives.

toast003 · 2025-07-29T09:04:51Z

The update script no longer supports using the local release file (and make assumptions about the future release url), but looking at the commit order during a release, that feature seems unused (right, @9001 ?). We might be better off removing it completely instead of trying to update it to work with release source archives.

Yeah, I was thinking about that too, I just got sidetracked and forgot to comment on it here. If we want to keep that feature I think we could pull the version from somewhere in the tarball (copyparty/__version__.py maybe?)

partyfuse package had an additional dependency that's not present in the default copyparty package now (pyfuse), so it's broken. Perhaps it should be included by default.

Didn't notice, thanks for pointing that out!
I'll add it now

chinponya · 2025-07-29T09:29:03Z

The update script no longer supports using the local release file (and make assumptions about the future release url), but looking at the commit order during a release, that feature seems unused (right, @9001 ?). We might be better off removing it completely instead of trying to update it to work with release source archives.

Yeah, I was thinking about that too, I just got sidetracked and forgot to comment on it here. If we want to keep that feature I think we could pull the version from somewhere in the tarball (copyparty/__version__.py maybe?)

That would work. If you want to add it, that's great, but feel free to ignore it, since it's an unused feature anyway. Personally I'm leaning towards just removing it, to simplify the update script.

partyfuse package had an additional dependency that's not present in the default copyparty package now (pyfuse), so it's broken. Perhaps it should be included by default.

Didn't notice, thanks for pointing that out! I'll add it now

Sorry, I lied, it's fusepy not pyfuse

toast003 · 2025-07-29T10:12:49Z

Ok, did both.

chinponya · 2025-07-29T11:10:03Z

Okay, I've tried building and running it and we have some dependencies missing. They were a part of the sfx.py bundle hence why they were omitted in buildInputs.

I found a couple looking at pyproject.toml, setup.py and make-sfx.sh script. It might not be an exhaustive list:

pyftpdlib
partftpy
magic

toast003 · 2025-07-29T11:18:03Z

Ok, I'll add them in a bit.

Btw, how did you find out that we're missing dependencies? Haven't done extensive testing yet since I'm not too familiar with copyparty yet, but it does seem to run just fine as is?

chinponya · 2025-07-29T11:18:47Z

These features are disabled by default but will hit import errors when enabled in the config.

toast003 · 2025-07-29T11:28:35Z

Ok, since they are optional should we lock them behind parameters like the other optional dependencies?

Also partytftp isn't packaged in nixpkgs, so that'll have to be packaged separately

chinponya · 2025-07-29T11:44:32Z

Not surprising. That's @9001's fork of tftpy
https://github.com/9001/partftpy

Feature flags would be nice, but maybe keep pyftpdlib enabled by default to not break existing setups. magic is a recent addition and partftpy is for a very obscure feature, so keeping them off by default should be fine.

I've also noticed copyparty itself will provide details about optional dependencies when ran with --deps:

dependencies            found: sqlite (sessions and file/media indexing)
dependencies            found: pillow (image thumbnails (plenty fast))
dependencies            found: vips (image thumbnails (faster, eats more ram))
dependencies            found: pillow-webp (create thumbnails as webp files)
dependencies            found: ffmpeg (transcode audio, create spectrograms, video thumbnails, good-but-slow image thumbnails)
dependencies            found: ffprobe (transcode audio, create spectrograms, video thumbnails, read audio/media tags)
dependencies            found: mutagen (read audio tags (ffprobe is better but slower))
dependencies            found: argon2 (secure password hashing (advanced users only))
dependencies            found: pyzmq (send zeromq messages from event-hooks)
dependencies          missing: pillow-heif (read .heif images with pillow (rarely useful))
dependencies            found: pillow-avif (read .avif images with pillow (rarely useful))

Somehow we have pillow-avif despite never explicitly listing it as a dependency (it may have been an external dependency that got merged into pillow at some point?). pillow-heif might be worth including whenever withThumbnails is enabled.

I'd expect a lot more items on this list. @9001 Is this check just outdated/incomplete?

toast003 · 2025-07-29T12:06:18Z

From my (little) time looking at partyftpy I think we can get away with overriding the tftpy package to point to partyftpy's source code, so we don't need to package it right now.

Feature flags would be nice, but maybe keep pyftpdlib enabled by default to not break existing setups. magic is a recent addition and partftpy is for a very obscure feature, so keeping them off by default should be fine.

I'd say keep pyftpdlib on a default on flag instead, and the other two on default off flags

toast003 · 2025-07-29T12:07:38Z

For --deps being outdated, while I do think it's something we should address I would prefer if that happened on its own issue/pr

dtomvan · 2025-07-29T13:15:02Z

since I realized that the release source isn't just a copy of the repo and does actually contain the built frontend dependencies

Well, that makes it a whole lot easier, doesn't it? I initially was scared of migrating the package to a source build, because it looked like I needed to replicate the entire docker build process in nix. Luckily that's not the case. Thanks for this PR!

From my (little) time looking at partyftpy I think we can get away with overriding the tftpy package to point to partyftpy's source code, so we don't need to package it right now.

Well, I think they removed the setuptools build process here so it might not be possible. Chances are high if you specify hatch for the build system and set pyproject = true; it will work. I think it's best to do that without relying on the structure of nixpkgs' tftpy package though.

Also, great news, this actually improves startup performance due to the python files being stored uncompressed! Bench: (the 2nd copyparty is from the nix package at v1.18.4, it's outdated, I know, but it doesn't really matter here)

Benchmark 1: ./result/bin/copyparty --exit cfg
  Time (mean ± σ):     251.2 ms ±   7.5 ms    [User: 204.8 ms, System: 37.2 ms]
  Range (min … max):   233.5 ms … 260.8 ms    11 runs

Benchmark 2: copyparty --exit cfg
  Time (mean ± σ):     518.6 ms ±  13.4 ms    [User: 425.5 ms, System: 75.4 ms]
  Range (min … max):   496.2 ms … 546.6 ms    10 runs

Summary
  ./result/bin/copyparty --exit cfg ran
    2.06 ± 0.08 times faster than copyparty --exit cfg

dtomvan

LGTM after partytftp gets added

dtomvan · 2025-07-29T13:07:32Z

contrib/package/nix/copyparty/default.nix

    mainProgram = "copyparty";
-    sourceProvenance = [ lib.sourceTypes.binaryBytecode ];
+    sourceProvenance = [ lib.sourceTypes.fromSource ];


By the way this is implicitly the default when no sourceProvenance was specified.

toast003 · 2025-07-29T13:49:01Z

Well, I think they removed the setuptools build process here so it might not be possible. Chances are high if you specify hatch for the build system and set pyproject = true; it will work. I think it's best to do that without relying on the structure of nixpkgs' tftpy package though.

I didn't check github when I was adding partftpy so I ended up doing what I said, and it seems to work just fine.
Edit: pyproject.toml still has set setuptools as the build backend

That said, I'm not sure what you mean with not relying on the structure of nixpkgs?

chinponya · 2025-07-29T14:26:34Z

That said, I'm not sure what you mean with not relying on the structure of nixpkgs?

As in, we don't want any potential changes to the tftpy package in nixpkgs to break our partftpy package, so it's best to include a complete package definition for it instead of overriding tftpy.

chinponya · 2025-07-29T16:47:46Z

Ok, I ran the current version and see no issues. I even deployed it on my big instance and nothing broke, with no changes necessary to my config. It's not a big sample size but it's better than nothing.
It looks ready to merge (unless you still want to change something).
I hope all this back and forth discussion wasn't too discouraging for you.

toast003 · 2025-07-29T16:54:24Z

It looks ready to merge (unless you still want to change something).

Nah I don't want to change anything for now, I was thinking of trying to build the frontend deps but I don't want to deal with that rn. Maybe some other time.

I hope all this back and forth discussion wasn't too discouraging for you.

Don't worry about it, y'all are pretty nice ^^

chinponya · 2025-07-30T16:23:30Z

@9001 👀

chinponya · 2025-07-31T11:35:43Z

contrib/package/nix/copyparty/pin.json

    "version": "1.18.7",
-    "hash": "sha256-GRoiNnYRWNcq5ITywPv7bK4pdgTey6Or1cZqyE2XDf4="
+    "hash": "sha256-Zzj2I5BSduhmS9h5HxSX0MYfiBbGYI52xNkJehKEkXA="


New copyparty version released since then (again), so this will need to be bumped before (or after) merging.
You could wait until @9001 finds the time to review and merge it, to not play a catch-up game with every bugfix.

This particular bugfix doesn't matter for the PR since we don't use the sfx anymore, but yeah, I'll wait for @9001 to review it

Yeah the bugfix is for "older versions of python3", and well, nixpkgs always keeps their default python3 fairly recent, and get rid of EOL versions, so this is a non-issue to us.

Also, to make review easier, I think it's completely fine to squash some stuff upfront. Like 3 would be fine I think: (1) remove u2c and partyfuse, (2) build from source, (3) fixup dependencies. I think that makes it more straightforward for ed.

Ok, squashed.

Btw sorry for messing up the commit history, had a bit of a brain fart 😅

Lol that's fine. Honestly I like the transparency it sometimes provides while one is responding to review suggestions. You can see the thought process. But yeah in most cases succinctness is more important IMO.

But enough bikeshedding 😅

nix: remove u2c and partyfuse packages The main copyparty package has u2c and partyfuse, so these packages are redundant now nix: add fusepy dependency fix: nix: use replace pyfuse with fusepy

toast003 · 2025-07-31T23:19:20Z

Turns out the code that 48dfd73 removed is used by a script :)
Don't have the brainpower to deal with this rn, I'll mark the pr as a draft, and fix it tomorrow

dtomvan · 2025-08-01T00:37:25Z

contrib/package/nix/copyparty/default.nix

+  src = fetchurl {
+    inherit (pinData) url hash;
+  };


Just saying this is something that can be done, it's quite common for python packages in nixpkgs and I figured why not...

Suggested change

src = fetchurl {

inherit (pinData) url hash;

};

src = fetchPypi {

inherit (pinData) pname version hash;

};

I think this also allows us to move away from the custom update script, because nix-update has special support for PyPi packages. That's for another PR I think though.

Then all the info would be contained in the derivation, like this:

# no pin.json anymore buildPythonApplication rec { pname = "copyparty"; version = "1.18.8"; src = fetchPyPi { inherit pname version; hash = "sha256-AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA=";

And the update script would then be a measly nix-update copyparty --flake

While I do think it would be a good idea, if we were to do this @9001 would need to have nix installed to be able to update the nix packages (assuming that nix-update depends on nix), so we would need to know if he's ok with that.
If he is I'll be glad to do this in a separate PR tho

Alternatively (maybe digging the hole even further in the process lol) you could have a Github action with on: workflow_dispatch that does the update with nix-update I guess...

assuming that nix-update depends on nix
Yeah, it needs nix to evaluate stuff. That's about the only advantage of the home-grown py+json solution right now: it doesn't involve any nix files... but at the same time that's a decoupling that usually isn't really desirable

TBF the updater for openjdk in nixpkgs also does this...

Revert "nix: remove unused code from update.py" This reverts commit 48dfd73. nix: update local release pin function in update.py

justinas · 2025-08-01T23:16:37Z

Very new to this project and mostly just curious. Given this now builds from source and not build artifacts, what is the point of the pin, i.e. why isn't the source of the package just set roughly to ../../..?

Currently this approach results in counter-intuitive behavior: if you do nix run github:9001/copyparty, i.e. try to run the latest commit (implicit), it runs the pinned stable release. What's more, if you attempt to "run" a specific version, i.e. nix run github:9001/copyparty/v1.18.9, it will build and run v1.18.8 instead (as that was the pin as of v1.18.9 release). Not to mention that nix run in a local checkout will not use local sources, and thus is unsuitable for development.

toast003 · 2025-08-02T00:01:52Z

You can set the source to ../../.. and copyparty itself will build, however that build would not include the web dependencies, so part of the UI would be missing/broken. The build procedure for the web deps requires docker/podman, and as far as I know that can't run inside of a nix build, so we would need to figure out how to build them without docker, run docker inside of a VM (which nix does surprisingly support), or do what we do and take the source tarballs, which do include the web deps. Aug 2, 2025, 1:17 AM by ***@***.***:

…

justinas> left a comment > (9001/copyparty#253) <#253 (comment)> Very new to this project and mostly just curious. Given this now builds from source and not build artifacts, what is the point of the pin, i.e. why isn't the source of the package just set roughly to > ../../..> ? Currently this approach results in counter-intuitive behavior: if you do > nix run github:9001/copyparty> , i.e. try to run the latest commit (implicit), it runs the pinned stable release. What's more, if you attempt to "run" a specific version, i.e. > nix run github:9001/copyparty/v1.18.9> , it will build and run v1.18.8 instead (as that was the pin as of v1.18.9 release). Not to mention that > nix run> in a local checkout will not use local sources, and thus is unsuitable for development. — Reply to this email directly, > view it on GitHub <#253 (comment)>> , or > unsubscribe <https://github.com/notifications/unsubscribe-auth/AJJUMAS55FK3R3OFNBCG75T3LPYOXAVCNFSM6AAAAACCSO6W5KVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTCNBVHE4TCNJTGQ>> . You are receiving this because you authored the thread.> Message ID: > <9001/copyparty/pull/253/c3145991534> @> github> .> com>

justinas · 2025-08-02T22:45:56Z

Thanks, I see. I missed the fact that this still builds from somewhat "pre-processed" source code, not the literal contents of the repo. I understand why that is, as the build process for the frontend assets seems complex and quite ad-hoc.

nix: get source tarball with update.py

9f6c983

toast003 commented Jul 29, 2025

View reviewed changes

chinponya mentioned this pull request Jul 29, 2025

various improvements to the nix files #228

Merged

toast003 marked this pull request as ready for review July 29, 2025 10:12

dtomvan reviewed Jul 29, 2025

View reviewed changes

chinponya mentioned this pull request Jul 30, 2025

nix: make usage in non-flake setups easier #296

Open

chinponya approved these changes Jul 31, 2025

View reviewed changes

chinponya reviewed Jul 31, 2025

View reviewed changes

dtomvan mentioned this pull request Jul 31, 2025

[Question]: Why on EARTH is half of copyparty-sfx.py just obfuscated? #345

Open

toast003 force-pushed the nix-overhaul branch 2 times, most recently from c4a18ac to 145d413 Compare July 31, 2025 15:47

toast003 added 2 commits July 31, 2025 17:50

nix: build from source

2bada04

nix: remove u2c and partyfuse packages The main copyparty package has u2c and partyfuse, so these packages are redundant now nix: add fusepy dependency fix: nix: use replace pyfuse with fusepy

nix: remove unused code from update.py

48dfd73

toast003 added 7 commits July 31, 2025 17:50

nix: fix extra python packages

ddecb19

nix: add optional dependencies

fe529da

nix: add partftpy package

9858b71

nix: add tftp parameter to package

6bd0172

nix: enable pyproject for partftpy package

023f019

nix: replace partftpy overlay with real package

f356ef9

nix: add updater for partftpy

910e2e3

toast003 force-pushed the nix-overhaul branch from 145d413 to 910e2e3 Compare July 31, 2025 15:50

toast003 marked this pull request as draft July 31, 2025 23:46

dtomvan reviewed Aug 1, 2025

View reviewed changes

nix: bring back local release pin to update.py

0f57518

Revert "nix: remove unused code from update.py" This reverts commit 48dfd73. nix: update local release pin function in update.py

toast003 marked this pull request as ready for review August 1, 2025 12:12

Build nix packages from source #253

Are you sure you want to change the base?

Build nix packages from source #253

Conversation

toast003 commented Jul 29, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

toast003 Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

chinponya commented Jul 29, 2025

Uh oh!

toast003 commented Jul 29, 2025

Uh oh!

chinponya commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toast003 commented Jul 29, 2025

Uh oh!

chinponya commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toast003 commented Jul 29, 2025

Uh oh!

chinponya commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toast003 commented Jul 29, 2025

Uh oh!

chinponya commented Jul 29, 2025

Uh oh!

toast003 commented Jul 29, 2025

Uh oh!

toast003 commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dtomvan commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dtomvan left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

toast003 commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chinponya commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chinponya commented Jul 29, 2025

Uh oh!

toast003 commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chinponya commented Jul 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

toast003 commented Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

toast003 Jul 29, 2025 •

edited

Loading

chinponya commented Jul 29, 2025 •

edited

Loading

chinponya commented Jul 29, 2025 •

edited

Loading

chinponya commented Jul 29, 2025 •

edited

Loading

toast003 commented Jul 29, 2025 •

edited

Loading

dtomvan commented Jul 29, 2025 •

edited

Loading

toast003 commented Jul 29, 2025 •

edited

Loading

chinponya commented Jul 29, 2025 •

edited

Loading

toast003 commented Jul 29, 2025 •

edited

Loading

toast003 commented Jul 31, 2025 •

edited

Loading

dtomvan Aug 2, 2025 •

edited

Loading