Marathon should set a non-default filter when declining offers (on master)

[ConnorDoyle]

Use case: As a cluster operator I would like to run many frameworks (possibly multiple Marathon instances) on the same Mesos cluster. I'd like to mitigate offer starvation; that is, schedulers with work to do should receive offers from the Mesos master.

From the SchedulerDriver documentation, Marathon can use both the overloaded definition of declineOffer to set filters:

  /**
   * Declines an offer in its entirety and applies the specified
   * filters on the resources (see mesos.proto for a description of
   * Filters). Note that this can be done at any time, it is not
   * necessary to do this within the {@link Scheduler#resourceOffers}
   * callback.
   *
   * @param offerId The ID of the offer to be declined.
   * @param filters The filters to set for any remaining resources.
   *
   * @return        The state of the driver after the call.
   *
   * @see OfferID
   * @see Filters
   * @see Status
   */
  Status declineOffer(OfferID offerId, Filters filters);

and clear them later on.

  /**
   * Removes all filters, previously set by the framework (via {@link
   * #launchTasks}). This enables the framework to receive offers
   * from those filtered slaves.
   *
   * @return    The state of the driver after the call.
   *
   * @see Status
   */
  Status reviveOffers();

Proposal: When the task queue becomes empty, decline subsequent offers with a long timeout (say, max double value). When the task queue becomes non-empty, invoke reviveOffers. (per offline conversation with @kolloch, @aquamatthias, and @gkleiman).

[adam-mesos]

Couldn't you also just abort() the SchedulerDriver and then create a new one (sends ReregisterFrameworkMessage) when you have tasks again? This abort() results in a DeactivateFrameworkMessage, which will temporarily deactivate the framework in the Mesos master allocator, preventing any new offers, while still leaving tasks running. Since abort() doesn't actually terminate the SchedulerDriver, you might still get status updates, etc. (I'd have to double-check). Reregistering will start the offers again.
Also note that other users have found that an excess of filters (one per each of 100 frameworks per each of 1000 nodes) can slow down the master's allocation/offer cycle. So I'd be wary of abusing the refuse filters too much. See https://issues.apache.org/jira/browse/MESOS-3052 and https://issues.apache.org/jira/browse/MESOS-3075 and https://issues.apache.org/jira/browse/MESOS-3157

[ConnorDoyle]

@adam-mesos we could do that, however I'm concerned about coordinating a reconciliation cycle after each re-registration since handling StatusUpdate messages can become a bottleneck for Marathon. Thanks for pointing out the potential issue with setting too many filters though, it's good to know.

[kolloch]

Hey @adam-mesos, just to save me the time to understand all this: I thought that declineOffers without a filters argument would actually use a default filter?

[adam-mesos]

@kolloch That's correct. Decline sets a default 5sec filter timeout for offers from that slave.
But why bother setting a filter on each slave as you receive each offer when you could just deactivate your framework's offers entirely until you're ready to get more offers? What you really want is the QuiesceOffers/SUPPRESS call that MESOS-3075/MESOS-3037 propose. You still get all your status updates, can even send framework messages, but you're just not getting any offers until you REVIVE.
This mechanism already exists using DeactivateFramework/ReregisterFramework, and would be easy to implement using pure bindings. I'm not sure if the native libmesos' SchedulerDriver's abort() method does exactly what we want here.

[kolloch]

I agree with @ConnorDoyle here, at least in my current state (= tired), I will not able to cover all code paths.

It does sound kind of hacky to me but that is subjective.