Locust distributed mode does not work with service-mesh/istio

[prateek-vats]

Prerequisites

• I am using the latest version of Locust
• I am suggesting a new feature, not asking a question

Description

Hello, i'm trying to setup locust in a distributed manner.

Intially this setup worked because the workers communicated with the leader through plaintext tcp. I used stateful sets at that time, and this direct communication was possible.

However, when now, with using istio/service mesh, i am using stateless pods. I was able to verify connection from the worker to the leader on the master-bind-port. But for some reason workers are never able to connect to master.

Looking at this further it seems like the issue is with mTLS and ZMQ protocol being incompatible.

I tried setting up my own zmq server on the leader, and tried connecting with the leader through a zmq client and i saw the same issue.

Can we please have support for mTLS compatible communication protocol?

[cyberw]

Hi! Switching communication protocol would be a pretty big change and is pretty low on our list of priorities.

If we were to replace zmq with anything then it would probably be websockets/socketio. If someone were to contribute such a switch I'd be happy to review it.

[prateek-vats]

Hi! Switching communication protocol would be a pretty big change and is pretty low on our list of priorities.

If we were to replace zmq with anything then it would probably be websockets/socketio. If someone were to contribute such a switch I'd be happy to review it.

How about using gRPC? That seems like a really good alternative and its well supported

[cyberw]

Yea, technically gRPC could be even better. But gRPC still has gevent support as "experimental". As far as I know it works really well, but because it is experimental they could remove it at any point.

[prateek-vats]

Yea, technically gRPC could be even better. But gRPC still has gevent support as "experimental". As far as I know it works really well, but because it is experimental they could remove it at any point.

Gotcha. Do you think we could potentially make the master-worker communication protocol configurable? Default can be ZMQ. That would ensure that in case gRPC removes gevent support, this still works. But if that doesn't work, Websockets can work too

[cyberw]

Yea that would be ok, as long as the abstraction layer doesnt become too weird/complex.

For example ZMQ doesnt have delivery guarantees or connection management, so it probably would be a little limiting to have a 1:1 translation to a protocol that does offer this, like gRPC or SocketIO.