Ko-built Windows images aren't pullable/runnable #361
Description
Reported in Tekton's #windows Slack channel: https://tektoncd.slack.com/archives/C020SPF0B7Y/p1620277036006500 -- copied here for posterity and to aid future discussion.
tl;dr: Images that Ko builds for Windows don't appear to be pullable, possibly due to KO_DATA_PATH and the /var/run/ko directory in the tar file.
It sounds like we could benefit from some Windows CI at least, to ensure that Windows images built with ko are runnable. We could have a GitHub Actions Windows runner pull and run a ko-built image, or after we cut a release with #339 we could build and run ko within CI to detect breakages before merging pull requests.
The specific issue might require us to have Windows-specific logic in gobuild.go, to change where kodata ends up in the image.
I have been doing some testing with a Windows node and have encountered a problem which prevents the entrypoint image from running (and as a result the TaskRun's pod won't initialise). Describing the pod reveals the following error under Events:
Failed to pull image "10.115.11.41:30002/tekton/entrypoint-bff0a22da108bc2f16c818c97641a296@sha256:1f900110f0103c208200d0be7e5614836d036845ac934d093192826dd2b94d84": rpc error: code = Unknown desc = failed to register layer: re-exec error: exit status 1: output: mkdir \\?\C:\ProgramData\docker\tmp\hcs523070999\var\run\ko: expected relative path
After inspecting the container image, I can see that the path /var/run/ko is injected into the image as an environment variable called KO_DATA_PATH. According to the Ko documentation this path is used to bundle static assets from a kodata sub-directory into the image's KO_DATA_PATH directory. I'm not sure why this extra path is being appended to the beginning - \\?\C:\ProgramData\docker\tmp\hcs523070999.
As this problem occurs when processing an image layer it can also be reproduced by running docker pull <image>, after building the Windows image with ko publish ./cmd/entrypoint --platform=windows/amd64. Note: for testing purposes, I'm using a Ko base image override of github.com/tektoncd/pipeline/cmd/entrypoint: mcr.microsoft.com/windows/servercore:ltsc2019
I haven't used Ko previously so I'm not all that familiar with it. I'm also not sure if this is a Docker issue or a Ko issue. My Ko version is 0.8.2 and Windows Docker runtime version 20.10.4 (Mirantis). I can't inspect the image using Docker because I cannot pull it. It seems either Ko is producing an incorrect image layer, or Docker is interpreting/converting the path incorrectly. Is there any way I can troubleshoot Ko further to see what's going on?
Your suggestion does sound plausible to me. Perhaps a relative path could be the solution (as the error seems to suggest). Maybe just removing the root path separator would be enough? (i.e. var/run/ko)
If we need a more specific location, I've found some directories in nanoserver (smallest image) which might be suitable:
C:\ProgramData\ >>> for application data that is not user-specific
C:\Windows\Temp\ >>> for temporary files
C:\Users\ContainerUser\ >>> user's home directory
C:\Users\ContainerUser\AppData\Local\ >>> user-specific application data
C:\Users\ContainerUser\AppData\Local\Temp\ >>> user's temp directory
Looking at the implementation of tarAddDirectories(), it looks like the intended file permissions for the kodata directory are world readable and executable, but not writable? If static data files don't need to be writable then C:\ProgramData\ko might be the ideal candidate, since the primary benefit of opting for a temporary or user-specific directory would be that the resulting files can be modified. (Avoiding user-specific directories also sidesteps any issues around determining whether a given Windows container image uses the ContainerUser or ContainerAdministrator user by default.)