fix: Clear related caches on clusters update

[Callisto13]

This fixes an issue where the clustersNamespaces and the userNamespaces
caches were not updated when Clusters were deleted.

This was most noticeable for clustersNamespaces since the GetAll there
would return everything in the cache, dated or not.

For the userNamespaces cache this was hidden, since the GetAll in this
case would receive an up to date list of the cached clusters and then
use that to look up individual keys, returning them in a collated list.
This meant it looked like the cache was updated, but it was only
showing us what we wanted to see; the cache was still quietly building
up.

The solution here is to generate a Hash based on an ordered list of the
names of current clusters. When updating the clustersNamespaces cache, we
compare the saved with the current hash, and clear the clustersNamespaces if
they differ.
We also clear the userNamespaces at this point; since the userNamespaces
cache is dependent on both the clusters and the clustersNamespaces
cache, it is simpler to clear everything together. (I can change this if wanted
nbd.)

The userNsList fun has been pulled apart for testing purposes, but is
otherwise the same. The testing that the userNamespaces cache has been
cleared was harder, since, as I said above the GetAll does not really
get all, but "gets all based on this cluster list". I can expose new
methods and change this into a genuine List if that is preferred, but
have not done that for now.

Closes #2083

[luizbafilho]

This looks good, but it's not enough, since we need to clean up the cf.usersNamespaces too, but different from the clustersNamespaces we cannot do every 30 seconds, other wise will have to ask user's permissions to the cluster too frequently, slowing things down.

So I propose that we create a hash for the list of clusters, and only reset everything up if that hash changes, so basically you'd do:

• List the cluster names ordered and save a concatenated string in the factory, we could add a GetHash to the Clusters cache for instance.
• When updating the namespaces, check if current hash is different from the saved one, if so, clear clustersNamespace and the usersNamespaces, if not leave as is.

With that, we should expect things to be eventually consistent, after 30 seconds or so.

[foot]

I get some very strange behaviour..

1. https://localhost:8000/v1/helmreleases? returns a cannot find cluster= (empty) kind of thing (sorry I'll try and get the corrent error message again)
2. https://localhost:8000/v1/helmreleases?clusterName=Default complains it cannot find a secret that only exists in the leaf (ewq2)
3. https://localhost:8000/v1/helmreleases?clusterName=ewq2 complains it cannot find a secret in mgmt (Default)

[Callisto13]

we need to clean up the cf.usersNamespaces too

I had deliberately left that to handle separately, but you are right that they have to be considered together 👍

[luizbafilho]

I get some very strange behaviour..

1. https://localhost:8000/v1/helmreleases? returns a cannot find cluster= (empty) kind of thing (sorry I'll try and get the corrent error message again)
2. https://localhost:8000/v1/helmreleases?clusterName=Default complains it cannot find a secret that only exists in the leaf (ewq2)
3. https://localhost:8000/v1/helmreleases?clusterName=ewq2 complains it cannot find a secret in mgmt (Default)

This is a separate issue, related to that handler specifically, looking at the code it tries to do some secret querying indeed.

I also noticed you are trying to query by ClusterName I don't think all list handlers support that, feel free to open another issue to add that too.

[Callisto13]

Updated commit and description

[luizbafilho]

this looks great beside couple small nits.

@foot would be possible for you to test it from this branch? or does it need to be merged?

[foot]

I can test from this branch 👌, will take it for a spin tomo thanks team!

…

On Thu 5. May 2022 at 18:47, Luiz Filho ***@***.***> wrote: ***@***.**** approved this pull request. this looks great beside couple small nits. @foot <https://github.com/foot> would be possible for you to test it from this branch? or does it need to be merged? — Reply to this email directly, view it on GitHub <#2085 (review)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAFL6DSCTOGQWI6RSKKHFDVIP3SJANCNFSM5VBNNO5A> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[foot]

Still not quite working unfortunately.

/v1/flux_runtime_objects returns:

{
  "code": 2,
  "message": "could not list deployments in namespace flux-system: cluster=ewq2 not found",
  "details": []
}

We could add some debug logging? 🤔 , I can put together a cluster to play with if it helps too.

[Callisto13]

@foot we could jump on a call together, might be faster?

I only have a couple of hours today before I go on holiday, so someone may need to take this off me if it drags on.

[foot]

After some testing and being more careful about observing the 30s window etc it seems to be working great! 💯

LGTM ⭐