Support dependencies using tracked paths #5

omus · 2025-05-27T13:28:47Z

I ran into a scenario where I wanted to use a tracked path dependency with pkg-precompile.jl and found out by using JULIA_DEBUG=pkg-precompile,loading that the distinct project environment interferes with tracked path dependencies:

#16 1.732 ┌ Debug: Rejecting cache file /usr/local/share/julia-depot/compiled/v1.10/Demo/tYFCI_15tAM.ji because it is for file /project-971996bcc9d7e4bbf8c09193266b2e88daaf26959d5e1c64097840b097b07d3a/Demo.jl/src/Demo.jl not file /project/Demo.jl/src/Demo.jl
#16 1.732 └ @ Base loading.jl:3196
#16 1.732 ┌ Debug: Rejecting cache file /usr/local/share/julia-depot/compiled/v1.10/Demo/tYFCI_zDkSN.ji because it is for file /project-3c4da0e3947323d78673b631b29a25d48c8937dc91ccf1c16882019ef0c94c96/Demo.jl/src/Demo.jl not file /project/Demo.jl/src/Demo.jl
#16 1.732 └ @ Base loading.jl:3196
#16 1.776 ┌ Debug: Rejecting cache file /usr/local/share/julia-depot/compiled/v1.10/DemoServiceExt/vg8vZ_15tAM.ji because it is for file /project-971996bcc9d7e4bbf8c09193266b2e88daaf26959d5e1c64097840b097b07d3a/Demo.jl/ext/DemoServiceExt.jl not file /project/Demo.jl/ext/DemoServiceExt.jl
#16 1.776 └ @ Base loading.jl:3196
#16 1.776 ┌ Debug: Rejecting cache file /usr/local/share/julia-depot/compiled/v1.10/DemoServiceExt/vg8vZ_zDkSN.ji because it is for file /project-3c4da0e3947323d78673b631b29a25d48c8937dc91ccf1c16882019ef0c94c96/Demo.jl/ext/DemoServiceExt.jl not file /project/Demo.jl/ext/DemoServiceExt.jl
#16 1.776 └ @ Base loading.jl:3196

I considered some alternatives but if tracked path dependencies require the path to be unchanged and we would store these in our shared cache mount we could run into issues with erroneous cache hits across image builds. It seems best to exclude these entries from the cache entirely and precompile these everytime for now.

We can improve upon this further but it would require some changes to base Julia which considers the contents of the source code and not the path of the source code.

To see the specific tasks where the Asana app for GitHub is being used, see below:
- https://app.asana.com/0/0/1210389681818144
- https://app.asana.com/0/0/1210453217724957

omus · 2025-05-27T15:44:24Z

README.md

-# Copy files necessary to load package and perform the first initialization.
-COPY src ${JULIA_PROJECT}/src
-RUN julia -e 'using Pkg; name = Pkg.Types.EnvCache().project.name; Pkg.precompile(name; timing=true); Base.require(Main, Symbol(name))'


I realized that we can just use the pkg-precompile.jl precompilation and initialization if we copy the src beforehand. This does mean that we invalidate the precompilation layer when the package changes but that doesn't matter much as we copy the .ji files from the cache mount and will typically only precompile the changed source files.

The only downsides of this is that the cache mount is needed for this not to impact performance. Say if we built a Docker image in CI (a different host each time) without using the mount cache then the loss of Docker layer caching could be an issue.

Additionally, the curl call could fail if there are network issues but that should also be a rare issue and could be mitigated by always having this script installed.

This does mean that we invalidate the precompilation layer when the package changes but that doesn't matter much as we copy the .ji files from the cache mount and will typically only precompile the changed source files.

The only downsides of this is that the cache mount is needed for this not to impact performance. Say if we built a Docker image in CI (a different host each time) without using the mount cache then the loss of Docker layer caching could be an issue.

Yeah this is my main concern with this approach. Seems like a step backwards to give up good layer caching...

Moving COPY src isn't a requirement for this change but without this we need to either make a empty package so that precompilation can pass or skip precompilation for that package. I'll take another look into those options.

At worse you can review those options and can make an informed decision

The alternative approaches where we create empty placeholder packages for tracked dependencies or skip precompilation are overall pretty similar approaches. Taking these approaches would allow us to precompile dependencies which are not tracked and do not depend on a tracked package in pkg-precompile.jl. We would need to perform another RUN statement which performs Pkg.precompile after the tracked dependencies have been added into the image. In the worst case scenario where a tracked package is used by all other dependencies the pkg-precompile.jl script would do nothing and we'd end up fully dependent on Docker layer caching.

I do think the simplicity of the Dockerfile provided by the current solution is worth it. The current solution is only an issue when cache mounting isn't used which isn't a problem for our CI setup anymore.

I'm considering some other options such as:

Modifying the path stored in the .ji file to avoid invalidation

Avoiding using set_distinct_active_project for tracked dependencies. Probably not feasible due to issue with .ji reuse across multiple containers.

Modifying the path stored in the .ji file to avoid invalidation

I've gone with this approach. It's transparent for users and doesn't break Docker layer caching and allows us to keep using set_distinct_active_project for avoiding slug collisions. It does mean having to read the .ji binary file but luckily they do have a format version included so we should be able to abort if we encounter a new version we don't yet understand.

omus · 2025-05-27T15:53:02Z

pkg-precompile.jl

+# # Listing all cached precompile files can be useful in debugging unexpected failures but
+# # it can be extremely verbose.
+# @debug let paths = String[]
+#     for (root, dirs, files) in walkdir(joinpath(DEPOT_PATH[1], "compiled", "v$(VERSION.major).$(VERSION.minor)"))
+#         for file in files
+#             if endswith(file, ".ji")
+#                 push!(paths, joinpath(root, file))
+#             end
+#         end
+#     end
+#     "All precompile files within the cache mount:\n$(join(paths, '\n'))"
+# end


This was useful in determining the source of the problem but is far too verbose for typical debug logs

omus · 2025-05-27T15:54:03Z

test/tracked-path/Manifest.toml

+project_hash = "3b30178eead5751cd2e80845fb6da37f5f121d23"
+
+[[deps.Demo]]
+path = "Demo.jl"


We require a Manifest.toml so we can add a path here but the downside is we'll get warnings on other versions of Julia which want us to re-resolve

kleinschmidt

Am I correct that we're trying to avoid precompiling path-tracking dependencies twice (because of the absolute path dependency on 1.10)?

I kinda feel like it may not be worth it if it means giving up layer caching for dependencies/precompilation...

run into issues with erroneous cache hits across image builds

seems like this can be accomplished by (like you said) just omitting path-tracking manifest entries from precompilation which wouldn't require installing the src/ext from the entrypoint project right? or am I missing something?

kleinschmidt · 2025-06-02T15:24:08Z

README.md

-# Copy files necessary to load package and perform the first initialization.
-COPY src ${JULIA_PROJECT}/src
-RUN julia -e 'using Pkg; name = Pkg.Types.EnvCache().project.name; Pkg.precompile(name; timing=true); Base.require(Main, Symbol(name))'


This does mean that we invalidate the precompilation layer when the package changes but that doesn't matter much as we copy the .ji files from the cache mount and will typically only precompile the changed source files.

The only downsides of this is that the cache mount is needed for this not to impact performance. Say if we built a Docker image in CI (a different host each time) without using the mount cache then the loss of Docker layer caching could be an issue.

Yeah this is my main concern with this approach. Seems like a step backwards to give up good layer caching...

omus · 2025-06-03T19:43:07Z

I'm planning on getting back to this but I suspect it may be deferred until next week

omus · 2025-07-07T14:45:05Z

MWE example of the problem we're trying to solve:

rm -rf /tmp/depot /tmp/project /tmp/project2
mkdir /tmp/depot /tmp/project
JULIA_DEPOT_PATH=/tmp/depot julia -e 'using Pkg; Pkg.Registry.add("General"); Pkg.precompile()'

JULIA_DEPOT_PATH=/tmp/depot julia --project=/tmp/project -e 'using Pkg; cd(dirname(Base.active_project())); Pkg.generate("Demo"); Pkg.develop(path="./Demo"); using Demo'
cat /tmp/project/Manifest.toml
find /tmp/depot/compiled/v1.11/Demo -name "*.ji"

mv /tmp/project /tmp/project2
JULIA_DEBUG=loading JULIA_DEPOT_PATH=/tmp/depot julia --project=/tmp/project2 -e 'using Demo'

❯ rm -rf /tmp/depot /tmp/project /tmp/project2

❯ mkdir /tmp/depot /tmp/project

❯ JULIA_DEPOT_PATH=/tmp/depot julia -e 'using Pkg; Pkg.Registry.add("General"); Pkg.precompile()'
       Added `General` registry to /tmp/depot/registries
  No Changes to `/private/tmp/depot/environments/v1.11/Project.toml`
  No Changes to `/private/tmp/depot/environments/v1.11/Manifest.toml`

❯ JULIA_DEPOT_PATH=/tmp/depot julia --project=/tmp/project -e 'using Pkg; cd(dirname(Base.active_project())); Pkg.generate("Demo"); Pkg.develop(path="./Demo"); using Demo'
  Generating  project Demo:
    Demo/Project.toml
    Demo/src/Demo.jl
   Resolving package versions...
    Updating `/private/tmp/project/Project.toml`
  [25faeaa9] + Demo v0.1.0 `Demo`
    Updating `/private/tmp/project/Manifest.toml`
  [25faeaa9] + Demo v0.1.0 `Demo`
Precompiling Demo...
  1 dependency successfully precompiled in 1 seconds

❯ cat /tmp/project/Manifest.toml
# This file is machine-generated - editing it directly is not advised

julia_version = "1.11.5"
manifest_format = "2.0"
project_hash = "6ab95279a0e4da063010f90285aa5e8243990fad"

[[deps.Demo]]
path = "Demo"
uuid = "25faeaa9-fcc1-4249-8e23-c1744a6c1ff7"
version = "0.1.0"

❯ find /tmp/depot/compiled/v1.11/Demo -name "*.ji"
/tmp/depot/compiled/v1.11/Demo/lo7dN_3d9pM.ji

❯ mv /tmp/project /tmp/project2

❯ JULIA_DEBUG=loading JULIA_DEPOT_PATH=/tmp/depot julia --project=/tmp/project2 -e 'using Demo'
┌ Debug: Rejecting cache file /tmp/depot/compiled/v1.11/Demo/lo7dN_3d9pM.ji because it is for file /tmp/project/Demo/src/Demo.jl not file /tmp/project2/Demo/src/Demo.jl
└ @ Base loading.jl:3873
┌ Debug: Rejecting cache file /tmp/depot/compiled/v1.11/Demo/lo7dN_3d9pM.ji because it is for file /tmp/project/Demo/src/Demo.jl not file /tmp/project2/Demo/src/Demo.jl
└ @ Base loading.jl:3873
Precompiling Demo...
  1 dependency successfully precompiled in 1 seconds
┌ Debug: Loading object cache file /tmp/depot/compiled/v1.11/Demo/lo7dN_khGfN.dylib for Demo [25faeaa9-fcc1-4249-8e23-c1744a6c1ff7]
└ @ Base loading.jl:1244

Using some custom code to re-write the .ji file contents with the updated path I could get the project directory change to work:

❯ rm -rf /tmp/depot/compiled/v1.11/Demo
mv /tmp/project2 /tmp/project
JULIA_DEBUG=loading JULIA_DEPOT_PATH=/tmp/depot julia --project=/tmp/project -e 'using Demo'
find /tmp/depot/compiled/v1.11/Demo -name "*.ji"
Precompiling Demo...
  1 dependency successfully precompiled in 1 seconds
┌ Debug: Loading object cache file /tmp/depot/compiled/v1.11/Demo/lo7dN_3d9pM.dylib for Demo [25faeaa9-fcc1-4249-8e23-c1744a6c1ff7]
└ @ Base loading.jl:1244
/tmp/depot/compiled/v1.11/Demo/lo7dN_3d9pM.ji

❯ mv /tmp/project /tmp/project2
julia -e '
    include("ji_rewrite.jl")
    for ji_file in filter(endswith(".ji"), readdir("/tmp/depot/compiled/v1.11/Demo"; join=true))
        @show ji_file
        rewrite(ji_file, "/tmp/project/" => "/tmp/project2/")
    end'
JULIA_DEBUG=loading JULIA_DEPOT_PATH=/tmp/depot julia --project=/tmp/project2 -e 'using Demo'
ji_file = "/tmp/depot/compiled/v1.11/Demo/lo7dN_3d9pM.ji"
┌ Debug: Loading object cache file /tmp/depot/compiled/v1.11/Demo/lo7dN_3d9pM.dylib for Demo [25faeaa9-fcc1-4249-8e23-c1744a6c1ff7]
└ @ Base loading.jl:1244

Note we didn't have to update the slug to match which means we should be able to still use set_distinct_active_project for making unique .ji filenames.

omus · 2025-07-08T19:21:06Z

The Julia 1.10.8 issue discovered here wasn't from these changes. I've separated that into #6

kleinschmidt

this is an impressive piece of work. I still hate it lol, it really feels like a bad idea to be messing around so much in undocumented and possibly unstable internals just to optimize our docker builds a little bit.

correct me if I'm wrong here, but IIUC the precompile cache that's invalidated is just for code that is tracked by path, and in our use cases that's generally just a single package.

unless and until that becomes really onerous, I'm inclined to leave this as a "well we tried to find a way to do this but everything we could come up with was so cursed that it doesn't really seem worthwhile".

omus · 2025-07-09T18:57:41Z

this is an impressive piece of work. I still hate it lol, it really feels like a bad idea to be messing around so much in undocumented and possibly unstable internals just to optimize our docker builds a little bit.

I've been trying to use this script as a test bed for what we need to change in .ji files so that all/most of this nonsense is eliminated. I do agree that modifying into the .ji internals isn't ideal but I do think it gets the job done. Maybe a better path forward is to add a flag to opt-into this behaviour?

correct me if I'm wrong here, but IIUC the precompile cache that's invalidated is just for code that is tracked by path, and in our use cases that's generally just a single package.

You are correct here that it doesn't add much time to precompile the path tracked packages twice.

unless and until that becomes really onerous, I'm inclined to leave this as a "well we tried to find a way to do this but everything we could come up with was so cursed that it doesn't really seem worthwhile".

We do need to do something however as otherwise we run into:

#18 2.815 Precompiling Demo...
#18 3.140     316.6 ms  ✓ Demo
#18 3.140   1 dependency successfully precompiled in 0 seconds
#18 3.263 ┌ Debug: Precompile files to transfer (new additions 0/0):
#18 3.263 └ @ Main /usr/local/bin/pkg-precompile.jl:494
#18 3.268 [ Info: Copy precompilation files into image...
#18 3.289 [ Info: Initialize dependencies...
#18 3.779 ERROR: LoadError: Precompilation incomplete for Demo
#18 4.472 Stacktrace:
#18 4.472  [1] error(s::String)
#18 4.487    @ Base ./error.jl:35
#18 4.487  [2] top-level scope
#18 4.487    @ /usr/local/bin/pkg-precompile.jl:564
#18 4.487 in expression starting at /usr/local/bin/pkg-precompile.jl:555
#18 ERROR: process "/bin/sh -c JULIA_DEBUG=pkg-precompile pkg-precompile.jl \"${JULIA_DEPOT_CACHE_TARGET}\" &&     find \"${JULIA_DEPOT_CACHE_TARGET}\" -name \"*.ji\" -type f | sort" did not complete successfully: exit code: 1

I've reverted the code back to the old logic which just postpones the precompilation of path tracked packages to occur outside of the cache mount.

omus added 14 commits May 26, 2025 13:51

Debug message for initialization

230c28d

Debug

e8e3158

Debug

a40857f

Debug

86517b3

Debug

e86effc

Debug

67c8e32

Debug

d4d8e33

Show cache depot contents

b5cd9c8

Debug

2374392

Work around relative packages

f826a87

Attempt to skip path tracked

e030f12

Delay precompilation of path tracked dependencies

042d897

Add tests

be43321

Comment about noisy output

e0fb82e

omus commented May 27, 2025

View reviewed changes

Update verbose debug logging code

8fd7e14

omus commented May 27, 2025

View reviewed changes

Merge branch 'main' into cv/tracked-path

2c664d0

omus marked this pull request as ready for review May 27, 2025 16:40

omus requested a review from kleinschmidt May 27, 2025 16:40

kleinschmidt reviewed Jun 2, 2025

View reviewed changes

omus added 6 commits July 8, 2025 09:09

Rewrite approach

adb62bb

Improve rewrite reliability

58cc066

Julia 1.10 compat

682b4a5

fixup! Julia 1.10 compat

fdd030f

fixup! Improve rewrite reliability

ca0f072

Run against all versions

81695ce

omus added 3 commits July 8, 2025 13:18

Test additionally against 1.10.8

5552e5d

Handle format changes for relocatable PR

d7168c0

Add dependency to tracked package

c24d6e6

omus mentioned this pull request Jul 8, 2025

Restrict support to Julia 1.10.9+ #6

Merged

Avoid testing against 1.10.8

9c7f330

omus requested a review from kleinschmidt July 9, 2025 15:11

kleinschmidt reviewed Jul 9, 2025

View reviewed changes

Revert to postponing path tracked packages

d5876f8

omus requested a review from kleinschmidt July 9, 2025 19:00

fixup! Revert to postponing path tracked packages

e77bf9d

kleinschmidt approved these changes Jul 11, 2025

View reviewed changes

omus merged commit f640030 into main Jul 11, 2025
6 checks passed

omus deleted the cv/tracked-path branch July 11, 2025 17:39

omus mentioned this pull request Jul 23, 2025

Rewrite .ji contents to support tracked packages #7

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support dependencies using tracked paths #5

Support dependencies using tracked paths #5

Uh oh!

omus commented May 27, 2025 •

edited

Loading

Uh oh!

omus May 27, 2025

Uh oh!

kleinschmidt Jun 2, 2025

Uh oh!

omus Jun 2, 2025 •

edited

Loading

Uh oh!

omus Jul 4, 2025

Uh oh!

omus Jul 8, 2025

Uh oh!

omus May 27, 2025

Uh oh!

omus May 27, 2025

Uh oh!

kleinschmidt left a comment

Uh oh!

kleinschmidt Jun 2, 2025

Uh oh!

omus commented Jun 3, 2025

Uh oh!

omus commented Jul 7, 2025

Uh oh!

omus commented Jul 8, 2025

Uh oh!

kleinschmidt left a comment

Uh oh!

omus commented Jul 9, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Support dependencies using tracked paths #5

Support dependencies using tracked paths #5

Uh oh!

Conversation

omus commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

omus May 27, 2025

Choose a reason for hiding this comment

Uh oh!

kleinschmidt Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

omus Jun 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

omus Jul 4, 2025

Choose a reason for hiding this comment

Uh oh!

omus Jul 8, 2025

Choose a reason for hiding this comment

Uh oh!

omus May 27, 2025

Choose a reason for hiding this comment

Uh oh!

omus May 27, 2025

Choose a reason for hiding this comment

Uh oh!

kleinschmidt left a comment

Choose a reason for hiding this comment

Uh oh!

kleinschmidt Jun 2, 2025

Choose a reason for hiding this comment

Uh oh!

omus commented Jun 3, 2025

Uh oh!

omus commented Jul 7, 2025

Uh oh!

omus commented Jul 8, 2025

Uh oh!

kleinschmidt left a comment

Choose a reason for hiding this comment

Uh oh!

omus commented Jul 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

omus commented May 27, 2025 •

edited

Loading

omus Jun 2, 2025 •

edited

Loading

omus commented Jul 9, 2025 •

edited

Loading