Doc: Provide example of building image

[m3dwards]

I'm trying to build and I'm not getting reproducible images. This is the buildx command I'm using:

docker buildx build -o type=oci,dest=image.img,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .

Could there be an example buildx invocation in the README to help someone use this strategy?

[AkihiroSuda]

I'm not getting reproducible images

What's non-reproducible?

[m3dwards]

Here is my console output:

➜  repro-sources-list.sh git:(master) ✗ docker buildx build -o type=oci,dest=image.img,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .

[+] Building 97.3s (8/8) FINISHED                                             docker-container:multiarch2
 => [internal] load build definition from Dockerfile.debian-12                                       0.0s
 => => transferring dockerfile: 416B                                                                 0.0s
 => [internal] load metadata for docker.io/library/debian:bookworm-20230904-slim                     1.3s
 => [auth] library/debian:pull token for registry-1.docker.io                                        0.0s
 => [internal] load .dockerignore                                                                    0.0s
 => => transferring context: 2B                                                                      0.0s
 => [internal] load build context                                                                    0.0s
 => => transferring context: 43B                                                                     0.0s
 => CACHED [stage-0 1/2] FROM docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d9  0.0s
 => => resolve docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d928b21de66096126  0.0s
 => [stage-0 2/2] RUN   --mount=type=cache,target=/var/cache/apt,sharing=locked   --mount=type=cac  81.8s
 => exporting to oci image format                                                                   14.1s
 => => exporting layers                                                                              6.5s
 => => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC)            6.9s
 => => exporting manifest sha256:d6997b9712dcae63aa13aba30a08fab9a18d7a65981d921fcb91c296eec8359e    0.0s
 => => exporting config sha256:3fae53ed7a611e1ad0f533e234cc89a7238276f8bc766bac496111a1d65da272      0.0s
 => => sending tarball                                                                               0.8s
➜  repro-sources-list.sh git:(master) ✗ docker buildx build -o type=oci,dest=image2.img,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .

[+] Building 89.6s (7/7) FINISHED                                             docker-container:multiarch2
 => [internal] load build definition from Dockerfile.debian-12                                       0.0s
 => => transferring dockerfile: 416B                                                                 0.0s
 => [internal] load metadata for docker.io/library/debian:bookworm-20230904-slim                     0.5s
 => [internal] load .dockerignore                                                                    0.0s
 => => transferring context: 2B                                                                      0.0s
 => [internal] load build context                                                                    0.0s
 => => transferring context: 43B                                                                     0.0s
 => CACHED [stage-0 1/2] FROM docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d9  0.0s
 => => resolve docker.io/library/debian:bookworm-20230904-slim@sha256:050f00e86cc4d928b21de66096126  0.0s
 => [stage-0 2/2] RUN   --mount=type=cache,target=/var/cache/apt,sharing=locked   --mount=type=cac  74.9s
 => exporting to oci image format                                                                   14.2s
 => => exporting layers                                                                              6.5s
 => => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC)            6.8s
 => => exporting manifest sha256:f3f7ebaa5cd2498431d4c1cc275e3ecc7490a0f77743e7b02d7bf787d45faf6c    0.0s
 => => exporting config sha256:2b0a504fbecbdce0009173ef63d3db74df58800aaab8aaf44c3fd50608c0b2f1      0.0s
 => => sending tarball                                                                               0.8s
➜  repro-sources-list.sh git:(master) ✗ sha256sum image.img
2001dd290e510347e01bc04a03e2b73b842cda6256418fb44fc802284a562843  image.img
➜  repro-sources-list.sh git:(master) ✗ sha256sum image2.img
\2fa5d75d6665dad76924ca277ccf3543be5c94f071e12cc78e9ecdc27f7f1316  image2.img

Manfest, config and actual image files have different hashes with two back to back runs on the same machine.

Do you have an example buildx command where they come out the same?

[AkihiroSuda]

The differences are caused by:

• var/cache/ldconfig/aux-cache
• var/log/alternatives.log
• var/log/apt/history.log
• var/log/apt/term.log

You may just remove them with RUN rm -f.

Full example:

• Support reproducible builds for Debian docker-library/httpd#248

[AkihiroSuda]

See also

• Reproducible builds docker-library/official-images#16044

[m3dwards]

@AkihiroSuda I'm not sure if this is the place to talk about this but you seem like the most knowledgeable person on this process.

With the changes made in #16 I now have a reproducible build for linux/amd64 and the same dockerfile for linux/arm64 but when I tell buildx to make both into a multi-arch image for some reason the output is different even though the two separate arches are reproducible when built alone. Where is the right place to raise this?

This was my command:

docker buildx build -o type=oci,dest=image.img,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .

Interestingly, I'm building on two different machines and comparing the output. One is an amd64, and one is arm64. They both produce the same output (when producing a single arch image) even though obviously each time for one of them it's a non-native build.

[AkihiroSuda]

You may use https://github.com/reproducible-containers/diffoci to check the diff.

It should be noted that the scope of repro build is limited to manifests, not indexes, as an index may contain references to SPDX provenances, etc.

[m3dwards]

Thanks for the tool @AkihiroSuda!

Output of diffoci:

DebainBuild ➜  repro-sources-list.sh git:(master) ✗ ./diffoci-v0.1.4.linux-amd64 diff docker://maxedw/image1 docker://maxedw/image2
INFO[0000] Target platforms: [linux/amd64]
INFO[0000] Loading image "docker.io/maxedw/image1:latest" from "docker"
docker.io/maxedw/image1:latest          	saved
Importing	elapsed: 0.8 s	total:   0.0 B	(0.0 B/s)
INFO[0001] Loading image "docker.io/maxedw/image2:latest" from "docker"
docker.io/maxedw/image2:latest          	saved
Importing	elapsed: 0.8 s	total:   0.0 B	(0.0 B/s)
TYPE     NAME                                                                             INPUT-0                                                                                                         INPUT-1
Desc     application/vnd.oci.image.manifest.v1+json                                       fd5379d8878302e938698586db3869b25bbb0bca1030eb48835bd84a47187b6d                                                1ea38231b7eea6d91ac8bd96602b30aafd6e6c88f25b5e514f420be676a421d0
Desc     application/vnd.oci.image.manifest.v1+json                                       fd5379d8878302e938698586db3869b25bbb0bca1030eb48835bd84a47187b6d                                                1ea38231b7eea6d91ac8bd96602b30aafd6e6c88f25b5e514f420be676a421d0
Mani     ctx:/manifest                                                                    ?                                                                                                               ?
Desc     application/vnd.oci.image.config.v1+json                                         0611c3c137f98c0f4990d26f57a8edaa156fc4be613cfd34811356dee03c3dfe                                                1ba11898aa9b6d3e579bb6a5b51c82b1e2dade40fc1c834164a4ae09fda879ef
Cfg      ctx:/config/config                                                               ?                                                                                                               ?
Desc     application/vnd.oci.image.layer.v1.tar                                           cf3420c34116c7777c0660d1e99d477bb48d347fa660da216867ccbdee7fd02c                                                f02cc727caf22e74251fcdec0164e1e6b78e87b2edbedccac4b2ca975a73af65
File     blobs/sha256/155eab17d86c47443adc8cebe7fc62c847c03db8cfb1ca53aa6276564fff23ef    Index 3                                                                                                         Index 4
File     blobs/sha256/d2c7ba5b367ab427569aa8e0f6551a224ee6b1b31cc585b3625fb676355666af    Index 13                                                                                                        Index 12
File     blobs/sha256/d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511    Index 14                                                                                                        Index 13
File     blobs/sha256/ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec    Index 15                                                                                                        Index 14
File     blobs/sha256/1f2a5c5f21abf788427963c515a0b60359ebc0b594789f425428a6b8cf97eda9    Index 4                                                                                                         Index 5
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/2ee6937ec697a293be718f27df65fc665d355f2edede02fa0a071b43f8089302" only appears in input 0
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/47f94f0e4dda0e127756f9aa48541b15e140b923ceddcc9c8e2597e40779fd80" only appears in input 0
File     blobs/sha256/0b9622d7cd405af087a352f882207fa2e54e85988bb395956840c3a13afcf21d    Index 2                                                                                                         Index 3
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/a44de156a7f46cac4ae15feb3e8e2d3127b9b6349761089b2f0f9bcf43e62bd5" only appears in input 0
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/8abb5ebae0c47c283756a9de0ca2b2287a7563e935b2162a57ea7c6651107774" only appears in input 0
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/94f26548633ca6e1b1af4486913c750c5628c79179b2a3cc67443a9bef6b625b" only appears in input 0
File     index.json                                                                       c9549cee48733bf4aa5ad635a19113f925a3b0508061af409adf017ac50d387c                                                d3b66896f36f0645af0086ac1beeab1a648b3c79e823e7eeaafb607c5d4aa537
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/33bf26bf869c7497c4ffdde1842fc76d1fc8db78cc2103395d37d59c85c61392" only appears in input 0
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/5abc1189feeab99aad971fdae2a199a70e559b440a119ba3b9a928e7db746b55" only appears in input 0
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/ae656186cf62c3ffff1f3ffaff1e569e92c8f79302e333fa9a4ff14e6570fd43" only appears in input 1
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/d22f304ed9b9719867ebdab1f6a4fce6a8b4d14e66a241dff3a1d4b4e7d297fe" only appears in input 1
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/0b8418c757238f8c0bebdf63724fa00c36d3ecb1b7eb866780364a56f982f7de" only appears in input 1
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/27153e7f7ca35cfe83c9e6ed9b12b32dbcac84c613776d3ea34ef9d4fcc0ef86" only appears in input 1
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/c3eebf09ec733b8f01c4d3db506b42fd66331fe157335faa34a94e3c95c436ee" only appears in input 1
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/acf1548d261354d48be809a35ae2e8730a178866adbdf0b32d98d7010e3148b0" only appears in input 1
Layer    ctx:/layers-0/layer                                                              name "blobs/sha256/ee87862996bc639ac76fd54dfd0778c97a44bbd2efb2184a1c3957bf198bc6be" only appears in input 1

Don't fully follow what the doffoci is telling me here but shall I just assume that it's currently not possible to make reproducible multi-arch images?

[AkihiroSuda]

Don't fully follow what the doffoci is telling me here but shall I just assume that it's currently not possible to make reproducible multi-arch images?

Possible, but you may have to specify registry reference strings rather than docker:// strings.

[AkihiroSuda]

This is the result I got (repro-sources-list.sh v0.1.3, buildx v0.14.0, BuildKit v0.13.1):

$ docker buildx build -o type=image,name=docker.io/akihirosuda/tmp20240424:0,push=true,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .                                                                                                
[...]
 => exporting to image                                                                                                                                                                             61.9s 
 => => exporting layers                                                                                                                                                                            12.9s 
 => => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC)                                                                                                          10.8s 
 => => exporting manifest sha256:efbda9bc79ec583cd48edd015ace1f05a5807e27477f868add652e6d64dc6f20                                                                                                   0.0s 
 => => exporting config sha256:d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511                                                                                                     0.0s 
 => => exporting attestation manifest sha256:4e97c4b92482f5798563eb0294eddbd3d991573726261e4dfc3230c21db58fee                                                                                       0.0s 
 => => exporting manifest sha256:1f2a5c5f21abf788427963c515a0b60359ebc0b594789f425428a6b8cf97eda9                                                                                                   0.0s
 => => exporting config sha256:ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec                                                                                                     0.0s
 => => exporting attestation manifest sha256:c94dde3b8ce891dfca5ad673b253d04ebe48cc9246e3415e9a89d45fd3cfb267                                                                                       0.0s
 => => exporting manifest list sha256:787688757b1caad6539976fd5a64dc50416b24505d40f01e6d034b70341f4d98                                                                                              0.0s
 => => pushing layers                                                                                                                                                                              19.9s
 => => pushing manifest for docker.io/akihirosuda/tmp20240424:0@sha256:787688757b1caad6539976fd5a64dc50416b24505d40f01e6d034b70341f4d98                                                             4.5s
 => [auth] akihirosuda/tmp20240424:pull,push token for registry-1.docker.io                                                                                                                         0.0s

$ docker buildx build -o type=image,name=docker.io/akihirosuda/tmp20240424:1,push=true,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 .
[...]
 => exporting to image                                                                                                                                                                             51.4s 
 => => exporting layers                                                                                                                                                                            12.7s 
 => => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC)                                                                                                          13.7s 
 => => exporting manifest sha256:efbda9bc79ec583cd48edd015ace1f05a5807e27477f868add652e6d64dc6f20                                                                                                   0.0s 
 => => exporting config sha256:d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511                                                                                                     0.0s 
 => => exporting attestation manifest sha256:64aa10db60f094ade29be8c949c83e012a35f140e7f8da6ea9f3beef869e73d6                                                                                       0.0s 
 => => exporting manifest sha256:1f2a5c5f21abf788427963c515a0b60359ebc0b594789f425428a6b8cf97eda9                                                                                                   0.0s
 => => exporting config sha256:ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec                                                                                                     0.0s
 => => exporting attestation manifest sha256:6c0dc5e87e4e5a0434c000d524d6f06b7ad3d5e5ae4196a99574beea6da34731                                                                                       0.0s
 => => exporting manifest list sha256:71b28118565e07eece365aabaa6b40d3acdc2eb210c57c22a62addc9c4586189                                                                                              0.0s
 => => pushing layers                                                                                                                                                                               5.3s
 => => pushing manifest for docker.io/akihirosuda/tmp20240424:1@sha256:71b28118565e07eece365aabaa6b40d3acdc2eb210c57c22a62addc9c4586189                                                             4.9s
 => [auth] akihirosuda/tmp20240424:pull,push token for registry-1.docker.io

While the digest of the manifest list (aka OCI index) is not reproducible as it contains the attestation manifest (SLSA provenance), the image manifests themselves are 100% reproducible:

• manifest sha256:efbda9bc79ec583cd48edd015ace1f05a5807e27477f868add652e6d64dc6f2 (amd64)
• manifest sha256:1f2a5c5f21abf788427963c515a0b60359ebc0b594789f425428a6b8cf97eda9 (arm64)

See https://explore.ggcr.dev/?image=docker.io%2Fakihirosuda%2Ftmp20240424%3A0 and https://explore.ggcr.dev/?image=docker.io%2Fakihirosuda%2Ftmp20240424%3A1

[m3dwards]

@AkihiroSuda I truly appreciate you taking the time to run those examples for me!

[AkihiroSuda]

The manifest list (index) can be made reproducible too by specifying --provenance=false

$ docker buildx build -o type=image,name=docker.io/akihirosuda/tmp20240424:no-provenance-0,push=true,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 --provenance=false .
[...]
 => exporting to image                                                                                                                                                                             46.8s 
 => => exporting layers                                                                                                                                                                            14.1s 
 => => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC)                                                                                                          11.7s 
 => => exporting manifest sha256:9959b500c85383c0c534d67d4411b98e4031db54d629f28cfef0a945d7dfdd09                                                                                                   0.0s 
 => => exporting config sha256:d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511                                                                                                     0.0s 
 => => exporting manifest sha256:c463b35ef852663064a67fe055b59df86991403275de7da113f4eaa5cdc50e26                                                                                                   0.0s 
 => => exporting config sha256:ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec                                                                                                     0.0s
 => => exporting manifest list sha256:13954d9f9d01070d648d2e00b15d152fcb58ecf4a93fac1cde89672c61fb0e7e                                                                                              0.0s
 => => pushing layers                                                                                                                                                                               2.4s
 => => pushing manifest for docker.io/akihirosuda/tmp20240424:no-provenance-0@sha256:13954d9f9d01070d648d2e00b15d152fcb58ecf4a93fac1cde89672c61fb0e7e                                               3.9s
 => [auth] akihirosuda/tmp20240424:pull,push token for registry-1.docker.io                                                                                                                         0.0s

$ docker buildx build -o type=image,name=docker.io/akihirosuda/tmp20240424:no-provenance-1,push=true,rewrite-timestamp=true -f Dockerfile.debian-12 --platform linux/amd64,linux/arm64 --no-cache --build-arg SOURCE_DATE_EPOCH=1675298208 --provenance=false .
[...]
 => exporting to image                                                                                                                                                                             45.0s 
 => => exporting layers                                                                                                                                                                            13.5s 
 => => rewriting layers with source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC)                                                                                                          11.2s 
 => => exporting manifest sha256:9959b500c85383c0c534d67d4411b98e4031db54d629f28cfef0a945d7dfdd09                                                                                                   0.0s 
 => => exporting config sha256:d5a1697de149c4584e824aed15ffe82412466e700f7a1c15451b255cd8a79511                                                                                                     0.0s 
 => => exporting manifest sha256:c463b35ef852663064a67fe055b59df86991403275de7da113f4eaa5cdc50e26                                                                                                   0.0s 
 => => exporting config sha256:ee6efcfa9510fa714668f0a2f00b256cfcaa3538c1108a42e36e32ea6faf12ec                                                                                                     0.0s
 => => exporting manifest list sha256:13954d9f9d01070d648d2e00b15d152fcb58ecf4a93fac1cde89672c61fb0e7e                                                                                              0.0s
 => => pushing layers                                                                                                                                                                               2.5s
 => => pushing manifest for docker.io/akihirosuda/tmp20240424:no-provenance-1@sha256:13954d9f9d01070d648d2e00b15d152fcb58ecf4a93fac1cde89672c61fb0e7e                                               1.6s
 => [auth] akihirosuda/tmp20240424:pull,push token for registry-1.docker.io

[AkihiroSuda]

BTW source-date-epoch 1675298208 (2023-02-02 00:36:48 +0000 UTC) isn't legit for a Dockerfile with FROM debian:bookworm-20230904-slim, and the value was not consumed due to lack of ARG SOURCE_DATE_EPOCH in the Dockerfile