Bazel 4.2.1 doesn't respect --remote_max_connections

[Vertexwahn]

Description of the problem:

There is a problem with Bazel 4.2.1. For some reason, it does not respect the --remote_max_connections flag. Builds that run with this version of Bazel open up thousands of sockets when they should in fact be restricted to 100.

We do not see this behavior with Bazel 4.0.0.

When using socketstat (we do here always a build with a remote cache - the system has around 240 sockets open before Bazel starts) we see a large increase of max sockets:

RUN_ID	Bazel version	Max Sockets
1	4.0.0	341
2	4.2.1	1241
3	4.0.0	332
4	4.2.1	1804
5	4.0.0	360
6	4.2.1	1813
7	4.2.1	2643

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

TODO (we work on it)

What operating system are you running Bazel on?

Ubuntu 20.04 LTS

What's the output of bazel info release?

release 4.0.0 or release 4.2.1

Have you found anything relevant by searching the web?

Nope.

Further notes

We are using Buildbarn

[katre]

Is this present in Bazel at head? We've just cut the 5.0.0 release, and are preparing to test and verify it, so this could be addressed in that.

[coeuvre]

Not sure why but --remote_max_connections is only used for http cache. You can verify with 4.0.0 by setting it to a lower number e.g. 10.

The difference between 4.0.0 and 4.2.1 for grpc cache is caused by the dynamic connection pool which we introduced after 4.0.0. #11801 (comment)

That said, I do think we should provide a way to set the max connections for grpc remote cache/execution. Reusing --remote_max_connections is an incompatible change since its default value is 100 while people expect the number of grpc connections is unlimited by default.

[brentleyjones]

@Wyverald Can this get the release blocker milestone instead of #14202, and can it be reopened until the cherry-pick is in?

[Wyverald]

Cherrypicked \o/