Add more detail on how to add use custom certificate using Rootless

[Arcelone]

Description

Context

GitLab pipeline using the docker executor with moby/buildkit:rootless.
Purpose of the job: build a Docker image and push it to an internal registry.
This registry uses a TLS certificate signed by an internal root CA, and the runner is behind a corporate proxy.

Config

In my before_script section:

• I echo my certificate in CRT format (-----BEGIN CERTIFICATE----- ...).
• Then I add:

[registry."project.url.domain"]
  ca=["path/to/my/file.crt"]

to ~/.config/buildkit/buildkitd.toml.

• I also add auth credentials and proxy settings in ~/.docker/config.json.

I tried adding:

--registry-auth-tlscontext host=project.url.domain,insecure=false,ca=path/to/my/file.crt

I made sure to set no_proxy in the container environment and noProxy in Docker’s config.json so that connections to our local domain are direct.

Yet I always get:

failed to push to project.url ... TLS failed to verify x509: unknown authority

The root cert is installed and working on the host machine where the GitLab runner is running.

Attempts and workarounds

• If I add registry.insecure=true to:

buildctl-daemonless.sh build \
  ... \
  --output type=image,...,push=true,registry.insecure=true

it works.

• If I add the following to buildkit.toml:

insecure = true

it also works.

There seem to be many (~15) issues in this repo related to this situation, but with different setups: k8s, GitLab Docker executor, GitLab k8s executor, standalone setups, and of course rootless vs non-rootless.

If anyone has figured out what I’m missing here, help would be much appreciated. 🤗

[OohSorry]

Had same issue. I Just created rootless image with cat mycert.pem >> /etc/ssl/certs/ca-certificates.crt and after that removed ca=["path/to/my/file.crt"] from config and set only mirrors. Also override entrypoint with [""]. Everything works fine

[Arcelone]

So you created a custom rootless buldkit image with your cert inside /etc/.

I don't want to do this since it will imply to create a custom image for every new release.

[tonistiigi]

I don't want to do this since it will imply to create a custom image for every new release.

I don't understand what other solution you are expecting. If you want to use custom ca file for buildkit inside a container, that buildkit binary needs to have some access to this file.

In buildx we have some special code to help with these transfers on docker buildx create if this is what you are looking for https://github.com/docker/buildx/blob/master/util/confutil/container.go#L27

[Arcelone]

I don't understand the difference between these two options:

[registry."project.url.domain"]
  ca=["path/to/my/file.crt"]

and

--registry-auth-tlscontext host=project.url.domain,insecure=false,ca=path/to/my/file.crt

There seems to be no difference whether I use them or not — it just doesn't work. I always get : failed to push to project.url ... TLS failed to verify x509: unknown authority

My custom root CA certificate is already present in the user’s home directory, so it should be accessible by BuildKit (if I’m correct).

What I expect is that when BuildKit pushes my images to my internal registry, it uses my internal root CA certificate to verify the TLS session.
However, the two options above don’t seem to have any effect, as BuildKit still appears to rely on the system certificates inside the container.

[Arcelone]

@tonistiigi is there a way (in debug logs for example) to verify which certificate buildkitD is using ?

[mus65]

I got this to work by writing our custom root CA to ~/certs/ca.crt and setting the env var SSL_CERT_DIR to /home/user/certs:/etc/ssl/certs, see also golang/go#35325

[Arcelone]

I managed to push to a private repository using $CI_SERVER_TLS_CA_FILE in my gitlab-ci.yml.
However, since I don’t manage the GitLab instance and $CI_SERVER_TLS_CA_FILE is a non-printable variable, I can’t determine its exact format.

Here’s the snippet I used:

- |
  echo "
  [registry.\"${CI_REGISTRY}\"]
    ca = [\"${CI_SERVER_TLS_CA_FILE}\"]
  "

[Arcelone]

I don't understand the difference between these two options:

[registry."project.url.domain"]
ca=["path/to/my/file.crt"]
and

--registry-auth-tlscontext host=project.url.domain,insecure=false,ca=path/to/my/file.crt

@tonistiigi any thoughts about this ?