Publish OCaml/Unikraft packages v1.0.0

[shym]

OCaml/Unikraft: an OCaml cross compiler to Unikraft backends

OCaml/Unikraft is structured into many packages to allow users to build only what they need.

Packages use one of two version numbers:

• 0.18.0, the upstream release number of Unikraft: used for unikraft and backend packages,
• 1.0.0, for packages that are more on the OCaml side.

The 15 packages are as follows:

• unikraft: the upstream sources.
• ocaml-unikraft-backend-<backend>-<arch>: all the object files and headers necessary to build a unikernel for the specific <backend> and <arch> combination.
• ocaml-unikraft-option-debug: if that package is installed, the backends will be built with debugging messages.
• ocaml-unikraft-toolchain-<arch>: a toolchain using the <arch>-unikraft-ocaml- prefix; the toolchain is a set of wrapper shell scripts to add the necessary options to drive the underlying tools.
• ocaml-unikraft-<arch>: the actual OCaml cross compilers for the given <arch>.
• ocaml-unikraft-default-<arch>: simple packages that only install a ocamlfind toolchain named unikraft (so that switching between architecture do not require to rebuild the full OCaml compiler).
• ocaml-unikraft-backend-<backend>: virtual packages to ensure that the given <backend> is installed for the default architecture, as set by which ocaml-unikraft-default-<arch> package is installed.
• ocaml-unikraft: the virtual package to ensure one of the default compiler package is installed.

[avsm]

This is very exciting, great work @shym! Quick question: you are maintaining a lot of extra-patches here; instead, why not move your musl fork over to mirage/ and publish that branch directly as the opam package with the patches applied? This is makes pinning and dev workflows significantly easier than having opam-level patch application.

[shym]

why not move your musl fork over to mirage/ and publish that branch directly as the opam package with the patches applied

Good idea, I’ll do that.

This is makes pinning and dev workflows significantly easier than having opam-level patch application.

Indeed, I admit this is quite involved at the moment 😅

[shym]

I’ve updated the PR. I have:

• updated the unikraft package to pull from (already patched) mirage/unikraft;
• created a unikraft-musl package to pull from (already patched) mirage/unikraft-lib-musl;
• simplified the build system and the backend packages accordingly;
• extracted the depexts into conf-<arch>-linux-gnu-gcc packages, with:
  • the fedora package which was missing,
  • a tentative suse package (it’s less clear to me which package to install there)
  • I couldn’t find a package in alpine nor in ARM arch;
• switched to C11+GNU as the version of GCC on suse doesn’t support C17;
• explicitly requested a 64-bit architecture for unikraft, as it doesn’t build on 32-bit platforms;
• explicitly stated that the OCaml cross compiler packages are incompatible with a bytecode-only non-cross compiler;
• added the dev-repo fields.

The packages without sources are there to express dependencies so I think those errors can be ignored.

[shym]

I think I’ve exhausted my bag of tricks to have the CI handle gracefully the errors when trying to install the conf-* packages on platforms where the corresponding packages are not available.

I added a no-known-package depext to install on all platforms or distributions where I couldn’t find a package containing <arch>-linux-gnu-gcc, as it makes the installation fail gracefully in many cases but not all. In particular:

• on Windows, the installation of the no-known-package cygwin package fails but opam goes on running the build instructions nevertheless,
• on FreeBSD and macOS, the installation stops after seeing that no-known-package is still missing but it doesn’t convert this to a graceful SKIP. I tried to add a x-ci-accept-failures for those cases, but it didn’t change the final result, so I dropped that annotation.

Side note: the extensive OS/distribution coverage is a great help to get this into shape, kudos to the opam-ci people!

[shym]

I’ve reviewed all the CI results. All the outcomes (failures and successes) are what I expect. So I see no further improvement to make.

[shonfeder]

Thank you for this exciting contribution! I just have a suggestion on how the installation failures are handed.

[shonfeder]

I think the cleanest, and a more helpful way of addressing this would be to use a post-message.

Here is an example:

opam-repository/packages/conf-llvm/conf-llvm.3.4/opam

Lines 15 to 17 in d4d1aa6

	post-messages:
	"If you use homebrew, please build and install llvm-3.4 manually, since brew formula is not available for this version"
	{failure & (os = "macos" | os = "macos")}

Do you think it would make sense to use something like that in this case?

[shym]

The more uniform these cases are handled, the better!
I suppose I’d better add a post-message just on {failure} to cover all cases.
Do you mean I should drop the "no-known-package" hack also? (It’s helpful for CI but might be annoying for end users).

[shonfeder]

The more uniform these cases are handled, the better!
I suppose I’d better add a post-message just on {failure} to cover all cases.

Yep, this makes sense to me! Just a general note on failure to advise people why/that automated system install is not available for them would help.

Do you mean I should drop the "no-known-package" hack also? (It’s helpful for CI but might be annoying for end users).

Yea, IIUC, it should not be needed for our CI or for end users. But if you need it for your dev CI, I don't want to make things more complicated! Leaving is not a problem, IMO. But I would defer to @mseri or @raphael-proust on this point.

[shym]

As I was thinking about opam-CI, I updated as discussed, namely adding a post-messages field and removing the "no-known-package" hack.
(Many CI tests which were converted into SKIP thanks to the hack are back to failures.)

[shonfeder]

Many CI tests which were converted into SKIP thanks to the hack are back to failures

We can address these via the appropriate x-ci-accept-failures in the future. But I don't think this is necessary currently!

[shym]

I would be curious to see how to set this up, as I tried to use those directives but they made no difference in the cases that failed. I had concluded that the test that the check for this field doesn’t trigger as expected at least in some configurations.

[shonfeder]

the check for this field doesn’t trigger as expected at least in some configurations.

Yes, I think that's true. The logic around this is not very robust currently.

[shonfeder]

Sorry that this was stuck in limbo! Sounds like all the expected CI issues are expected, and we can work on improvements with coverage and/or noise reduction in the future.

Thanks very much!