301 redo migration to clang 18

[m-hemmings]

PURPOSE

Update Clang from version 16 to 18 to enable compatibility with newer toolchains and prepare for upcoming features.
STATUS

Working:

• Dockerfile.dev builds successfully with Clang 18
• All tests pass in the development image

Not working:

• Dockerfile.e2e is not yet compatible with Clang 18

[m-hemmings]

The major changes to Dockerfile.e2e are:

1. Changing LLVM and CLANG versions
2. The installation of libtinfo5

The other changes are in make_glibc_and_sysroot.sh

1. Update the CFLAGS
2. Add manual compilation of elision-lock and eilision-unlock

When I do this in Dockerfile.dev, it works and all the tests passed, but the same is not true for Dockerfile.e2e, as you can see in this build log.

It looks to me like the addition of the compile steps for elision-lock and elision-unlock below

# Compile elision-lock.c
$CC $CFLAGS $WARNINGS $EXTRA_FLAGS \
    $INCLUDE_PATHS $SYS_INCLUDE $DEFINES $EXTRA_DEFINES \
    -o $GLIBC/build/nptl/elision-lock.o \
    -c $GLIBC/sysdeps/unix/sysv/linux/x86/elision-lock.c \
    -MD -MP -MF $GLIBC/build/nptl/elision-lock.o.dt \
    -MT $GLIBC/build/nptl/elision-lock.o

# Compile elision-unlock.c
$CC $CFLAGS $WARNINGS $EXTRA_FLAGS \
    $INCLUDE_PATHS $SYS_INCLUDE $DEFINES $EXTRA_DEFINES \
    -o $GLIBC/build/nptl/elision-unlock.o \
    -c $GLIBC/sysdeps/unix/sysv/linux/x86/elision-unlock.c \
    -MD -MP -MF $GLIBC/build/nptl/elision-unlock.o.dt \
    -MT $GLIBC/build/nptl/elision-unlock.o

either aren't compiling correctly, or the result isn't being passed further down the chain, because the test failures all seem to reference it like this:

"/tests/unit-tests/file_tests/non-deterministic/dup.c": {
#20 24.04                 "status": "Failure",
#20 24.04                 "error_type": "Lind_wasm_compiling",
#20 24.04                 "output": "\nwasm-ld: warning: function signature mismatch: __clone\n>>> defined as (i32, i32, i32, i32, 
i32) -> i32 in //src/glibc/sysroot/lib/wasm32-wasi/libc.a(clone-internal.o)\n>>> defined as () -> i32 in //src/glibc/sysroot/lib/
wasm32-wasi/libc.a(clone.o)\n\nwasm-ld: error: //src/glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_mutex_lock.o): undefined 
symbol: __lll_lock_elision\nwasm-ld: error: //src/glibc/sysroot/lib/wasm32-wasi/libc.a(pthread_mutex_unlock.o): undefined 
symbol: __lll_unlock_elision\nclang: error: linker command failed with exit code 1 (use -v to see invocation)\n"
#20 24.04             },

@lukpueh I would appreciate any insight into this that you might offer

[lukpueh]

When I do this in Dockerfile.dev, it works and all the tests passed, but the same is not true for Dockerfile.e2e, as you can see in this build log.

Did you compare the logs from Dockerfile.dev and Dockerfile.e2e? The build log you attached has fatal error: 'stddef.h' file not found under RUN make sysroot. Is that expected? Maybe there's a missing dependency? (you seem to install different things in .dev and .e2e)

[m-hemmings]

@J7-7-7 I added 3 files here to see if you can see something. I'm trying to more closely mimic the successful building in .dev with .staged, and if you pull it and run like this:

docker buildx build   --file ./scripts/Dockerfile.staged   --target build-glibc   --tag staged-glibc   --platform=linux/amd64   --load   --build-arg BRANCH_NAME=301-redo-migration-to-clang .

It fails, but maybe you can see why with me

[m-hemmings]

@J7-7-7 Could you try pulling this and building like this?

docker buildx build   --file ./scripts/Dockerfile.e2e   --tag clang-18-prod-test:2.0   --cache-from type=local,src=~/.cache/docker-buildx   --cache-to type=local,dest=~/.cache/docker-buildx   --platform=linux/amd64   --load

I got a pass on the build and all the tests, attached below:
builder-prod.log

[J7-7-7]

It builds, takes time but it works. Great work @m-hemmings

[rennergade]

It builds, takes time but it works. Great work @m-hemmings

does it take significantly more time?

[m-hemmings]

It builds, takes time but it works. Great work @m-hemmings

does it take significantly more time?

I have to do some code cleanup on this anyway, I'll time them today and post results

[m-hemmings]

@rennergade I timed them as follows on a system with 16 vCPUs, 64 GB Memory:

Timings (h:mm:ss):
  branch: 22:50.36
  main: 15:49.69

[rennergade]

@rennergade I timed them as follows on a system with 16 vCPUs, 64 GB Memory:

Timings (h:mm:ss):
  branch: 22:50.36
  main: 15:49.69

do we have an idea why its 7min longer?

[m-hemmings]

@rennergade I timed them as follows on a system with 16 vCPUs, 64 GB Memory:

Timings (h:mm:ss):
  branch: 22:50.36
  main: 15:49.69

do we have an idea why its 7min longer?

No, honestly, that seems extreme considering all that got added was two manual compiles, a library install, and some padding around a curl that shouldn't even lengthen it, just protect it from a flaky network connection.
I did run them back to back on the same machine, so it's also possible that there was some caching that helped the second one be faster because of similarities in base layers