What can cause docker to just hang for multiple minutes right in the middle of building?

[Voldemat]

I`m kinda going insane with current docker performance, I have ci server with gitlab docker runner, and each time I have more that few jobs in pipeline start at once docker just hangs in random places, I`m convinced it is some kind of race condition or deadlock, because I dont see such stuff when only one job is running. Here is an example of logs from last job. Question is why docker spend 546.1 seconds loading build context, when it is only 518 kb?

#0 building with "default" instance using docker driver
#1 [internal] load build definition from api-tests
#1 transferring dockerfile:
#1 transferring dockerfile: 1.37kB done
#1 DONE 8.7s
#2 [internal] load metadata for docker.io/oven/bun:1.2.19
#2 DONE 1.6s
#3 [internal] load build context
#3 DONE 0.0s
#4 [ 1/10] FROM docker.io/oven/bun:1.2.19@sha256:31f25ad4c661322a3dc9d9d98fbf34989502b1ea588a2ca629da98c8e5a2d116
#4 DONE 0.0s
#3 [internal] load build context
#3 transferring context:
#3 transferring context: 518.12kB 0.0s done
#3 DONE 546.1s
#5 [ 3/10] RUN apt-get -y update &&     apt-get install -y --no-install-recommends wget=1.21.3-1+deb12u1 ca-certificates=20230311+deb12u1 &&     apt-get clean &&     rm -rf /var/lib/apt/lists/* &&     wget --progress=dot:giga "https://github.com/sqlc-dev/sqlc/releases/download/v1.28.0/sqlc_1.28.0_linux_$(dpkg --print-architecture).tar.gz" -O /tmp/sqlc.tar.gz &&     tar -xvf /tmp/sqlc.tar.gz -C /usr/local/bin/ &&     mkdir -p /usr/local/app/backend/tests/api &&     chown tests:tests /usr/local/app/backend/tests/api && 	wget --progress=dot:giga https://downloads.sqlc.dev/plugin/sqlc-gen-typescript_0.1.3.wasm -O /usr/local/app/backend/tests/api/sqlc-gen-typescript.wasm
#5 CACHED
#6 [ 4/10] WORKDIR /usr/local/app/backend/tests/api
#6 CACHED
#7 [ 5/10] COPY --chown=tests:tests shared/graphql/schema.json /usr/local/app/shared/graphql/schema.json
#7 CACHED
#8 [ 6/10] COPY --chown=tests:tests shared/atlas/schema.sql /usr/local/app/shared/atlas/schema.sql
#8 CACHED
#9 [ 7/10] COPY --chown=tests:tests backend/tests/api/package.json backend/tests/api/bun.lock ./
#9 CACHED
#10 [ 2/10] RUN useradd -ms /bin/sh tests && ln -s "$(which bun)" /usr/bin/node
#10 CACHED
#11 [ 8/10] RUN bun install --frozen-lockfile
#11 CACHED

[crazy-max]

Question is why docker spend 546.1 seconds loading build context, when it is only 518 kb?

Might depend on the underlying filesystem on your host. Can you give the output of docker info and docker buildx ls?

Also can you show your Dockerfile? And what's the size of your context (du -sh . assuming context is your working dir)

[Voldemat]

here is screenshot of htop output when docker hangs on "exporting layers..."

[thaJeztah]

Curious if it would be faster without the .dockerignore

When building with BuildKit, only content that's used by the dockerfile will be copied; I see your Dockerfile explicitly copies only those files (so not doing a COPY . . or similar). Curious if the .dockerignore would have a negative impact, as it may be traversing path to calculate what needs to be in/excluded (but maybe @crazy-max has more thoughts on that w.r.t. the implementation in BuildKit)

[Voldemat]

I would assume if problem was dockerignore then docker would hang at start where it copies context

[Voldemat]

Output of strace on buildkit process when it does exporting layers:

strace: Process 1221870 attached
futex(0x44ae9a0, FUTEX_WAIT_PRIVATE, 0, NULL

[Voldemat]

Output of strace on docker daemon:

strace: Process 772 attached
futex(0x582a77bd7f40, FUTEX_WAIT_PRIVATE, 0, NULL

I clicked through all docker related processes and they all stuck on futex.