[SPIRE Agent] Add size based backoff strategy for fetchSVIDs

[prasadborole1]

• Implemented size based backoff that reduces the num of csrs by half if encountered failure and doubles on success.
• Currently synchronization backoff task has maxInterval of 2 mins (for 5 sec of sync interval). Adding Option to set maxInterval that will override default.

Size based backoff will recover in 8 steps (40 secs for 5 sec of sync interval) for current limit of 500.

Failure	num of CSRs
1	250
2	125
3	62
4	31
5	15
6	7
7	3
8	1

fixes #4264

[MarcosDY]

some minor comments,
I think you will need to update documentation, at least the one related with sync interval,
because if some user set it to a big number this backoff can result confusing

[MarcosDY]

may we add a defer, and validate if no error returned, and based on that play with backoff?

[prasadborole1]

moved it closer to server call

[prasadborole1]

some minor comments, I think you will need to update documentation, at least the one related with sync interval, because if some user set it to a big number this backoff can result confusing

@MarcosDY Can you please point to docs which needs update or where to add new backoff details?

[amoore877]

My comments are addressed. Looks like just some a couple open things from @MarcosDY needing changes still.

Thanks for this!

[MarcosDY]

looks like agent sync interval was never documented https://github.com/spiffe/spire/blob/main/doc/spire_agent.md#agent-configuration-file
what do you think about adding that inside experimental table? with the default value?
I'm not sure how much information add about this new backoff strategy but a single explanation about it will works with a backoff can be very useful for people that is debugging it

[rturner3]

This looks good to me, just a couple small comments