bug: fluentd unable to start with error="fork/exec /usr/local/bundle/bin/fluentd: no such file or directory"

[joshuabaird]

Describe the issue

It seems a recent image was pushed to the kubesphere/fluentd:1.15.3 tag (docker.io/kubesphere/fluentd@sha256:bc06e880c224e76e659bf59250e5302ad159ee6b5474a2c5ee45f3a0969644c5) which breaks fluentd:

level=error msg="start Fluentd error" error="fork/exec /usr/local/bundle/bin/fluentd: no such file or directory"

Pinning to a previous SHA fixes the issue -- kubesphere/fluentd:v1.15.3@sha256:794311919658aee8eb9829836cd6c3437dffd9c7112556d5dc2f01ca3fcb826b.

To Reproduce

Repull the kubesphere/fluentd:1.15.3 latest SHA.

Expected behavior

Fluentd should start.

Your Environment

N/A

How did you install fluent operator?

No response

Additional context

No response

[joshuabaird]

@benjaminhuo @wenchajun Can someone please review this?

[m-gavrilyuk]

same error on fluent/fluent-operator/fluentd:v2.8.0

[rurus9]

I think that in general each new image should have a new tag (does not apply to floating tags, like "latest").

[joshuabaird]

I would agree. Folks rely on versioned tags for stability and they should be immutable. If these images are going to be rebuilt for whatever reason, perhaps an internal "patch" version should be added (eg, v1.15.3.x).

[benjaminhuo]

I would agree. Folks rely on versioned tags for stability and they should be immutable. If these images are going to be rebuilt for whatever reason, perhaps an internal "patch" version should be added (eg, v1.15.3.x).

This might be related to the CI changes we made recently, cc @sarathchandra24

#1183
#1079

I also remember there is an PR for a similar issue from @sarathchandra24
#1093

Would you help to take a look? @sarathchandra24

Thanks

[benjaminhuo]

I've built fluentd v1.17.0 image

[joshuabaird]

@benjaminhuo It looks like the 1.17.0 image has the same bug for x86_64 images. Is this expected?

[sarathchandra24]

Sorry for the late response everyone, I realized the problem after running it locally.

Root cause is defaultBinPath on main.go#L22 for amd64 it is"/usr/bin/fluentd" and for arm64 it is "/usr/local/bundle/bin/fluentd".

Creating a PR for logic to choose path based on arch.

[benjaminhuo]

Sorry for the late response everyone, I realized the problem after running it locally.

Root cause is defaultBinPath on main.go#L22 for amd64 it is"/usr/bin/fluentd" and for arm64 it is "/usr/local/bundle/bin/fluentd".

Creating a PR for logic to choose path based on arch.

Thank you very much @sarathchandra24