Potential race condition!

[k8sdevlk]

Hello Team,

Not sure if this has addressed or not, but I have a situation in need of a solution.

Condition: when a node gets online, the images for pods (daemonset or deployment pods) on the same node would get re-written to pull from proxy via localhost:7439/whateverImageAndTagHere

If proxy pod is not up yet on the same node, then you would get ErrImagePull. I understand that the proxy pods are using priorityClassName: system-node-critical but other system pods might be using the same class name. Note, eventually pods would be able to pull image after a few minutes of delay ...

Questions:

1. Instead of using daemonset for proxy - can we use deployment and call it via service name?

2. Serve from remote registry rather than local pod when pod is not ready?

Thank you,

K

[plaffitt]

Hello,

You make a good point, maybe the solution would be to ignore pods with priorityClassName: system-node-critical like we already ignore pods in kuik-system and kube-system, but I understand than not caching critical images is a bit counter-intuitive..

Answers:

1. Not currently, and it would require substantial changes. Indeed the endpoint of the registry need to be reachable by kubelet and thus from the host, which is not possible using only a service, it would also require an ingress. Configuring an ingress means exposing the registry to the outside, which would require to add an authentication layer.
2. I can give you almost the same answer than for the previous question, also it would require to update the chart a bit to make the registry optional.

We already thought about those changes before and it seemed to us that they are not a priority and that we may not want to add additional complexity. Also we currently have other plans in our roadmap.

Feel free to share what you think about it, it is always nice to have multiple points of view and we might reconsider this in the future.

[euven]

This is a shame, as it makes your great tool a bit of a deal breaker for using in scenarios where your require fast node scaling...

[james-gonzalez]

Thanks @plaffitt - Just to add to this, we've also implemented this, works great in test environments, but as soon as you try to implement it in a large scale, truly scaling cluster, you run into this race condition often, sometimes it can be a dozen times a day, in our case, our cluster scales up and down quite a bit because we have several hundred projects using our cluster for gitlab ci runners, which need to be rock solid, when this works, its an amazing speed boost and even a cost saver, but as @euven says, unless we can resolve this race condition somehow, it's not suitable for us either.

Is there any sort of development/support that you would need to make this change happen? Can you think of any potential solutions, even if for particular workloads?

[plaffitt]

Ok, I understand better the use case and why this is important to you.

For now I don't have a solution in mind, and I'm personally busy on other open-source projects for the next weeks/months so I don't plan to fix it soon. However, if you can imagine a solution, I would be glad to discuss it with you and maybe you could even come up with a PR that I would be happy to review.