Rust support #27

tonistiigi · 2021-08-30T04:48:58Z

Allow building rust projects. Support installing rust via rustup (alpine/debian) and alpine packages (rust/rust-stdlib). Haven't tested official image. Debian package-based installs will probably not be supported as packages don't seem to be properly separated.

Currently have tested with these Dockerfile stages:

ARG RUST_ALG=alpine-rustup

FROM --platform=$BUILDPLATFORM alpine AS rust-alpine-rustup
COPY --from=xx / /
RUN apk add clang lld musl-dev gcc git curl
WORKDIR /work
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git
WORKDIR BLAKE3/b3sum
RUN cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
ENV XX_VENDOR=unknown
RUN xx-clang --setup-target-triple
RUN rustup target add $(xx-info)
RUN export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_LINKER=$(xx-info)-clang && \
    export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_RUSTFLAGS="" && \
    export CC_$(xx-info | tr - _)=$(xx-info)-clang && \
    env && cargo build --release --target-dir /tmp/blake --target $(xx-info) && \
    mkdir -p /out && cp /tmp/blake/$(xx-info)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM --platform=$BUILDPLATFORM alpine:edge AS rust-apk
COPY --from=xx / /
RUN apk add clang lld musl-dev gcc git rust cargo
WORKDIR /work
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git
WORKDIR BLAKE3/b3sum
RUN cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc rust-stdlib
RUN xx-clang --setup-target-triple
RUN export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_LINKER=$(xx-info)-clang && \
    export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_RUSTFLAGS="-C target-feature=+crt-static" && \
    export CC_$(xx-info | tr - _)=$(xx-info)-clang && \
    env && cargo build --release --target-dir /tmp/blake --target $(xx-info) && \
    mkdir -p /out && cp /tmp/blake/$(xx-info)/release/b3sum /out/ && \
    xx-verify /out/b3sum && rm -rf /tmp/blake
# alpine rust disables static linking? https://github.com/alpinelinux/aports/commit/69851bdae1177246337f51a35734e93f1fd7e3d3#diff-92eb9787919d42fb458bdae3211d4ee3d18385106441d644ec49b10ec3dfaa8c

FROM --platform=$BUILDPLATFORM debian:sid AS rust-debian-rustup
COPY --from=xx / /
RUN apt update && apt-get install -y clang lld libc6-dev binutils libc6-dev git curl
WORKDIR /work
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git
WORKDIR BLAKE3/b3sum
RUN cargo fetch
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev libgcc-10-dev
ENV XX_VENDOR=unknown
RUN xx-clang --setup-target-triple
RUN rustup target add $(xx-info)
RUN export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_LINKER=$(xx-info)-clang && \
    export CARGO_TARGET_$(xx-info | tr '[:lower:]' '[:upper:]' | tr - _)_RUSTFLAGS="-C target-feature=+crt-static" && \
    export CC_$(xx-info | tr - _)=$(xx-info)-clang && \
    env && cargo build --release --target-dir /tmp/blake --target $(xx-info) && \
    mkdir -p /out && cp /tmp/blake/$(xx-info)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM rust-${RUST_ALG} as rust-build

FROM scratch AS b3sum
COPY --from=rust-build /out/b3sum /

The main complexity is that the binary releases use triple with unknown vendor instead of alpine and as C compiler always gets called with cc --target=wrong-triple it doesn't really matter that the CC is wrapped. I've added a way for cc-clang to work with any vendor by creating symlinks to the official ones. An alternative would be to try to rename the --target flag in the wrapper.

I think it makes sense to add xx-cargo script that would set the 3 env automatically. xx-cargo --setup-target could make the correct calls to rustup target add or xx-apk add rust-stdlib (and set up xx-clang for correct triple).

Risc-V doesn't currently work because the gnu triple is riscv64gc-* not riscv64. Might need some special override for that. In alpine:edge there is no riscv package for rust-stdlib yet, and no musl build in rustup either.

add xx-cargo
support riscv64gc-gnu triple
CI testing

@crazy-max

tonistiigi · 2021-09-01T06:53:43Z

Added xx-cargo and riscv support. Works with both Alpine and Debian with either rustup or package based installs. RiscV only works on debian, musl support is blocked in rust-lang/libc#1994

Test stages with xx-cargo

ARG RUST_ALG=alpine-rustup

FROM --platform=$BUILDPLATFORM alpine AS blake
RUN apk add git
WORKDIR /src
RUN git clone git://github.com/BLAKE3-team/BLAKE3.git

FROM --platform=$BUILDPLATFORM alpine AS rust-alpine-rustup
RUN apk add clang lld musl-dev gcc git curl
COPY --from=xx / /
WORKDIR /work
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
RUN --mount=from=blake,src=/src/BLAKE3,rw cd b3sum && \
    xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM --platform=$BUILDPLATFORM alpine:edge AS rust-apk
RUN apk add clang lld musl-dev gcc git rust cargo
COPY --from=xx / /
WORKDIR /work
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apk add musl-dev gcc
RUN xx-cargo --setup-target-triple # optional
RUN --mount=from=blake,src=/src/BLAKE3,rw \
    cd b3sum && xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify /out/b3sum && rm -rf /tmp/blake
# alpine rust disables static linking? https://github.com/alpinelinux/aports/commit/69851bdae1177246337f51a35734e93f1fd7e3d3#diff-92eb9787919d42fb458bdae3211d4ee3d18385106441d644ec49b10ec3dfaa8c

FROM --platform=$BUILDPLATFORM debian:sid AS rust-debian-rustup
RUN apt update && apt-get install -y clang lld libc6-dev binutils libc6-dev git curl
WORKDIR /work
COPY --from=xx / /
RUN curl -O https://static.rust-lang.org/rustup/archive/1.24.3/$(XX_VENDOR=unknown xx-info)/rustup-init && \
    chmod +x rustup-init && \
    ./rustup-init -y --no-modify-path --profile=minimal
ENV PATH=/root/.cargo/bin:$PATH
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev libgcc-10-dev
RUN xx-cargo --setup-target-triple # optional
RUN --mount=from=blake,src=/src/BLAKE3,rw \
    export XX_RUSTFLAGS="-C target-feature=+crt-static" && \
    cd b3sum && xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && ls -lR /tmp/blake && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify --static /out/b3sum && rm -rf /tmp/blake

FROM --platform=$BUILDPLATFORM debian:sid AS rust-debian-apt
RUN apt update && apt-get install -y clang lld libc6-dev binutils libc6-dev git curl cargo
WORKDIR /work
COPY --from=xx / /
RUN --mount=from=blake,src=/src/BLAKE3,rw cargo fetch
ARG TARGETPLATFORM
RUN xx-apt install -y libc6-dev libgcc-10-dev
RUN --mount=from=blake,src=/src/BLAKE3,rw git checkout 0.3.7 && \
    cd b3sum && xx-cargo build --release --target-dir /tmp/blake && \
    mkdir -p /out && cp /tmp/blake/$(xx-cargo --print-target)/release/b3sum /out/ && \
    xx-verify /out/b3sum && rm -rf /tmp/blake

FROM rust-${RUST_ALG} as rust-build

FROM scratch AS b3sum
COPY --from=rust-build /out/b3sum /

Signed-off-by: Tonis Tiigi <[email protected]>

crazy-max · 2022-09-21T20:35:46Z

musl support is blocked in rust-lang/libc#1994

@tonistiigi Looks like it has been merged: rust-lang/libc#2537

Signed-off-by: Tonis Tiigi <[email protected]>

Using xx-cargo is optional. Alternative is to install stdlib with xx-apt/xx-apk or with rustup with XX_VENDOR=unknown xx-info. Then CC, and _LINKER needs to be defined via xx-info. Signed-off-by: Tonis Tiigi <[email protected]>

tonistiigi · 2022-09-23T18:29:05Z

@crazy-max So I think the last commit needs to be reverted to make CI work. Or do you have additional analysis?

crazy-max · 2022-09-23T18:54:32Z

@tonistiigi I looked across GitHub for XX_VENDOR or xx-info env but didn't find any use out there except here when dealing with xx-go, xx-apt, xx-apk and xx-verify.

I'm a bit worried about this case:

xx/base/xx-apt

Line 38 in 5596639

case "${XX_VENDOR}" in

I think we could just use case "$(xx-info vendor)" in.

tonistiigi · 2022-09-23T18:56:33Z

Do you understand what is causing the CI failure. Maybe there is a solution that would allow keeping that line. Having XX_VENDOR and xx-info vendor return different values doesn't make a lot of sense.

crazy-max · 2022-09-23T20:12:15Z

Do you understand what is causing the CI failure.

Yes it fails with xx-go because triple is wrong when XX_VENDOR is being exported in:

xx/base/xx-go

Line 6 in 5596639

export "${l?}"

tonistiigi · 2022-09-23T21:23:45Z

That file does not use XX_VENDOR afaics. So the issue needs to be already in xx-info not it xx-go

Signed-off-by: CrazyMax <[email protected]>

tonistiigi mentioned this pull request Aug 30, 2021

Going further denzp/cargo-wharf#34

Open

tonistiigi force-pushed the rust-support branch from 7564e6c to b7528bc Compare August 30, 2021 06:48

tonistiigi added this to the 1.1 milestone Oct 10, 2021

tonistiigi modified the milestones: 1.1, 1.2 Nov 29, 2021

Allow custom vendor to support unknown triple in rust

ada701d

Signed-off-by: Tonis Tiigi <[email protected]>

crazy-max mentioned this pull request Sep 22, 2022

Rust support rebased #76

Closed

tonistiigi added 3 commits September 22, 2022 13:12

apk: add special detection for rust library installs

50e54fa

Signed-off-by: Tonis Tiigi <[email protected]>

allow setting riscv64 arch to riscv64gc for rust support

ab1effd

Signed-off-by: Tonis Tiigi <[email protected]>

add xx-cargo to simplify rust building

3e9dd7f

Using xx-cargo is optional. Alternative is to install stdlib with xx-apt/xx-apk or with rustup with XX_VENDOR=unknown xx-info. Then CC, and _LINKER needs to be defined via xx-info. Signed-off-by: Tonis Tiigi <[email protected]>

rumpl mentioned this pull request Sep 22, 2022

Add a Dockerfile for cross-compilation second-state/runwasi#1

Merged

crazy-max force-pushed the rust-support branch from 132a063 to 2543538 Compare September 22, 2022 17:17

crazy-max marked this pull request as ready for review September 22, 2022 17:17

crazy-max self-requested a review September 22, 2022 17:17

crazy-max force-pushed the rust-support branch from 8d23e49 to 9e46daa Compare September 23, 2022 19:37

crazy-max force-pushed the rust-support branch from 9e46daa to ea9cf16 Compare September 27, 2022 12:19

xx-info: fix vendor

b8dde93

Signed-off-by: CrazyMax <[email protected]>

crazy-max force-pushed the rust-support branch from ea9cf16 to b8dde93 Compare September 27, 2022 12:22

crazy-max approved these changes Sep 30, 2022

View reviewed changes

tonistiigi merged commit dad71a2 into master Sep 30, 2022

crazy-max deleted the rust-support branch September 30, 2022 23:59

crazy-max mentioned this pull request Oct 14, 2022

Add tests and documentation for rust support #79

Closed

crazy-max mentioned this pull request Oct 30, 2022

xx-cargo: add tests and docs #83

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Rust support #27

Rust support #27

Uh oh!

tonistiigi commented Aug 30, 2021 •

edited

Loading

Uh oh!

tonistiigi commented Sep 1, 2021

Uh oh!

crazy-max commented Sep 21, 2022

Uh oh!

tonistiigi commented Sep 23, 2022

Uh oh!

crazy-max commented Sep 23, 2022

Uh oh!

tonistiigi commented Sep 23, 2022

Uh oh!

crazy-max commented Sep 23, 2022 •

edited

Loading

Uh oh!

tonistiigi commented Sep 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Rust support #27

Rust support #27

Uh oh!

Conversation

tonistiigi commented Aug 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tonistiigi commented Sep 1, 2021

Uh oh!

crazy-max commented Sep 21, 2022

Uh oh!

tonistiigi commented Sep 23, 2022

Uh oh!

crazy-max commented Sep 23, 2022

Uh oh!

tonistiigi commented Sep 23, 2022

Uh oh!

crazy-max commented Sep 23, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tonistiigi commented Sep 23, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

tonistiigi commented Aug 30, 2021 •

edited

Loading

crazy-max commented Sep 23, 2022 •

edited

Loading