Add BestFit algorithm to TAS (kubernetes-sigs#4228)

* Add info about BestFit algorithm to KEP

* Implement BestFit algorithm

* Use built-in sort.Search function, add optimizations at every level

* Improve bestFit function, add test scenario

* Change BestFit into LargestFit, fix a test scenario

* Fix checker, correct featureGate name

* Adjust new feature gate

* Fix tests with changed feature gate

* Cleaned up logic

* Adjusted test results for the BestFit default algorithm

* Change naming LargestFit -> LeastAllocated, BestFit -> MostAllocated

* Pick the first MostAllocated domain that can accommodate all pods

* Fix linter, improve naming

* Simplify logic

* Simplify logic, adjust comments

* Mark some test scenarios to drop after removing the feature gate

* Add TODOs comments, mention dropping the feature gate in GA in KEP

* Improve logic, add test

* Improve logic

* Update keps/2724-topology-aware-scheduling/README.md

Co-authored-by: Michał Woźniak <[email protected]>

* Address nits

* Address nits

* Rename feature gate

---------

Co-authored-by: Michał Woźniak <[email protected]>

Author: 2 people

keps/2724-topology-aware-scheduling/README.md

@@ -648,6 +648,12 @@ For a given PodSet Kueue:
   level. Kueue starts the search from the specified level, but if the PodSet
   does not fit, then it tries higher levels in the hierarchy.
 
+Kueue packs pods on domains starting from the domains with the most free capacity. However, Kueue can operate in two modes when it comes to choosing the last domain if there is more than one capable of accommodating the remaining pods:
+- `MostAllocated` - Kueue chooses the domain with the least available resource that is capable of accommodating all the pods to mitigate resource fragmentation
+- `LeastAllocated` - Kueue chooses the domain that has the most available resources, providing better nodes utilization
+
+By default Kueue uses the `MostAllocated` algorithm. To use `LeastAllocated` algorithm, a user needs to set the feature gate `TASLeastAllocated` to `true`
+
 ### Enforcing the assignment
 
 When the workload has the PodSet assignments and is about to start we modify the
@@ -753,6 +759,7 @@ Consider the following improvements and implement if feasible:
  is running
 - perform full scheduling simulation rather than just capacity counting
  (including pod affinities and anti-affinities)
+- drop `TASLeastAllocated` feature gate
 
 ## Implementation History
 

keps/2724-topology-aware-scheduling/kep.yaml

@@ -31,6 +31,7 @@ milestone:
 # List the feature gate name and the components for which it must be enabled
 feature-gates:
   - name: TopologyAwareScheduling
+  - name: TASLeastAllocated
 disable-supported: true
 
 # The following PRR answers are required at beta release