Problems with qemu 5.2

[alejandro-perez]

Hi, we use multiarch/alpine to build some images for armv7 target on gitlab, and since the last update to qemu 5.2 we.re getting this error:

 qemu-arm-static: Unable to reserve 0xffff0000 bytes of virtual address space at 0x1000 (Success) for use as guest address space (check yourvirtual memory ulimit setting, min_mmap_addr or reserve less using -R option)

Any idea of why it could be or how to workaround it?

[lafin]

@alejandro-perez Thanks for the report. Could you provide some minimal example to reproduce?

Unable to find image 'multiarch/alpine:armhf-v3.12' locally
armhf-v3.12: Pulling from multiarch/alpine
e7d6450f2346: Pull complete
Digest: sha256:41b45849f66ff08e5a479a8e00228b51a2de4fa7b9b9b4db85c7a2b9d720e5bc
Status: Downloaded newer image for multiarch/alpine:armhf-v3.12
/ # apk add gcc
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/main/armhf/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.12/community/armhf/APKINDEX.tar.gz
(1/11) Installing libgcc (9.3.0-r2)
(2/11) Installing libstdc++ (9.3.0-r2)
(3/11) Installing binutils (2.34-r1)
(4/11) Installing gmp (6.2.0-r0)
(5/11) Installing isl (0.18-r0)
(6/11) Installing libgomp (9.3.0-r2)
(7/11) Installing libatomic (9.3.0-r2)
(8/11) Installing libgphobos (9.3.0-r2)
(9/11) Installing mpfr4 (4.0.2-r4)
(10/11) Installing mpc1 (1.1.0-r1)
(11/11) Installing gcc (9.3.0-r2)
Executing busybox-1.31.1-r19.trigger
OK: 95 MiB in 30 packages
/ # apk add libc-dev
(1/2) Installing musl-dev (1.1.24-r10)
(2/2) Installing libc-dev (0.7.2-r3)
OK: 104 MiB in 32 packages
/ # vi hello.c
/ # gcc -Wall hello.c -o hello
/ # ./hello
Hello, world!

[alejandro-perez]

HI, I cannot reproduce it locally either, only on our remote Gitlab runners. Guess it has to be with the configuration our our Gitlab runners, that do not get along with this new version of qemu. I'll try fiddling with them, but truth is that it still works with previous versions, so it must be something that qemu 5.2 has introduced (perhaps is asking for more memory than it used to?).

I'll see if we can raise the virtual memory ulimit on the runner and will be back with the outcome.

[alejandro-perez]

Hi,

on the gitlab runner, ulimit says unlimited virtual memory, and min_mmap_addr is set to 4096, but with those two exact values it works on my local machine.

I've been testing around and playing with QEMU_RESERVED_VA (equivalemnt to the -R parameter) to make it request less than 4GB, but it always fails to request it. Example, that's requesting just 20Mbytes.

qemu-arm-static: Unable to reserve 0x1400000 bytes of virtual address space at 0x1000 (Success) for use as guest address space (check yourvirtual memory ulimit setting, min_mmap_addr or reserve less using -R option)

I guess the way in which qemu is requesting virtual memory is different for 5.2 and our gitlab runner does not like it.

[klutchell]

@alejandro-perez are you still facing this issue or have you found a workaround?

[alejandro-perez]

Hi @klutchell. I'm still facing it. My workaround was using 3.11 and updating the repositories to 3.12

[klutchell]

@alejandro-perez so your workaround was to use a newer multiarch/alpine container and repositories, and didn't involve any changes to the host OS for your gitlab runners? Any guesses as to which package change specifically?

[alejandro-perez]

Sorry I replied too fast and probably without much clarity.
In order to run an Alpine v3.12 directly, which didn't work because of Qemu 5.2, I'm using a custom image which bases from 3.11, replaces the version from 3.11 to 3.12 in /etc/apk/repositories, then upgrades the system:

FROM multiarch/alpine:armhf-v3.11
RUN sed -i "s/11/12/g" /etc/apk/repositories \
         && apk update && apk upgrade

docker build -t my_alpine_312_armhf .

[klutchell]

@alejandro-perez Thanks for the details! Unfortunately it doesn't help with my use case where qemu-arm-static 5.2 isn't able to allocate guest memory when running in a container on EC2. For now we rolled back to 4.0 until we can figure this out.

[alejandro-perez]

Oh right, sorry I got confused since I'm fnding a different problem with 3.13 and that was the solution I was using.

The actual solution for the problem was to stick to an earlier version of the image that I happened to have in my cache which ships with qemu-static 5.1.
How have you stuck to 4.0? Are you overwriting the qemu-arm-static binary?

[alejandro-perez]

BTW, my gitlab instance is running on Azure, also finding the same problem. I guess they share similar configurations.

[klutchell]

I'm not actually using multiarch/alpine at all but manually installing qemu-arm-static binaries from here in our host instance before building containers with emulation. I only ended up here because you have the exact same error message and it seems like the root cause would be similar.

Good to know that 5.1 works for you though, I'll try that and see if I can narrow down the change between versions.

[alejandro-perez]

Ok good to know. I read it was possible to use qemu-arm-static fromthe host rather than rellying on it being available in the container, but I never tried. Will take a look at that too :)

[klutchell]

@alejandro-perez FYI, the fix for us was actually discovered by the debian package maintainers. We saw this commit and applied the same --disable-pie to our internal qemu build scripts.

https://salsa.debian.org/qemu-team/qemu/-/commit/4adae9b8654d58b3d26d9d35568347f8da5b03bb

Perhaps something similar is required by the alpine package maintainers?

[alejandro-perez]

That's very interesting and seems like a great fix to me. @lafin what do you think?