Feature request: Awareness of rolling restart requirement for version upgrade #136
Description
An Elasticsearch version upgrade is a situation where the number of spare instances needs to exceed the number of index replicas in order to allow both primaries and replicas to be allocated on one of the new nodes. This is different from a normal rolling restart where one extra instance is enough.
To accommodate this, we need either need to make the es-operator aware of a version upgrade, and make treat this specially, or allow the users to define the spare instances in the EDS (e.g. spec.maxSurge) to control the es-operator behavior during the rolling restart. Or, we don't change anything, and users will need to control the version upgrade by temporarily increasing minReplicas.