Allow more than one http rest & grpc listeners #3749
Conversation
dkalinowski
changed the title
dkalinowski
changed the title
docs/parameters.md
Outdated
| | `rest_port` | `integer` | Number of the port used by HTTP server (if not provided or set to 0, HTTP server will not be launched). | | ||
| | `grpc_bind_address` | `string` | Network interface address or a hostname, to which gRPC server will bind to. Default: all interfaces: 0.0.0.0 | | ||
| | `rest_bind_address` | `string` | Network interface address or a hostname, to which REST server will bind to. Default: all interfaces: 0.0.0.0 | | ||
| | `grpc_bind_address` | `string` | Network interface address or a hostname, to which gRPC server will bind to. Default: all interfaces (both ipv4 and ipv6): 0.0.0.0 | |
There was a problem hiding this comment.
todo: document in security considerations how to limit to 127.0.0.1,::1
There was a problem hiding this comment.
change localhost to 127.0.0.1 in demos
There was a problem hiding this comment.
changed, but only in places where requests package is used: 4ef60de
| EXPECT_EQ(ovms::Config::check_hostname_or_ip("::ffff:0:192.0.2.128"), true); | ||
|
|
||
| // Multiple selections | ||
| EXPECT_EQ(ovms::Config::check_hostname_or_ip("0.0.0.0"), true); |
There was a problem hiding this comment.
127.0.0.1,::1 as actual e2e unit test?
There was a problem hiding this comment.
not all build environments support ipv6, therefore this is bad idea to make such e2e test
| --- | ||
| OpenVINO Model Server currently does not provide access restrictions and traffic encryption on gRPC and REST API endpoints. The endpoints can be secured using network settings like docker network settings or network firewall on the host. The recommended configuration is to place OpenVINO Model Server behind any reverse proxy component or load balancer, which provides traffic encryption and user authorization. | ||
|
|
||
| When deploying in environments where only local access is required, administrators can configure the server to bind exclusively to localhost addresses. This can be achieved by setting the bind address to `127.0.0.1` for IPv4 or `::1` for IPv6, which restricts incoming connections to the local machine only. This configuration prevents external network access to the server endpoints, providing an additional layer of security for local development or testing environments. |
There was a problem hiding this comment.
- performance guideline doc about potential 2s issue on windows
|
|
||
| ## Network Configuration for Optimal Performance | ||
|
|
||
| When clients connect to the server using hostname resolution (particularly "localhost"), the system may attempt IPv6 resolution first before falling back to IPv4. If IPv6 is disabled, misconfigured, or unavailable, this can cause connection timeouts and delays before the IPv4 fallback occurs, which is especially noticeable when minimizing time to first token in generative AI applications. |
There was a problem hiding this comment.
By default, OVMS endpoints are bound to all ipv4 addresses. On same systems, which route localhost name to ipv6 address, it might cause extra time on the client side to switch to ipv4. It can effectively results with extra 1-2s latency.
It can be overcome by switching the API URL to http://127.0.0.1 instead.
Alternatively ipv6 can be enabled in the model server using --grpc_bind_address and --rest_bind_address.
For example:
--grpc_bind_address 127.0.0.1,::1 --rest_bind_address 127.0.0.1,::1
or
--grpc_bind_address 0.0.0.0,:: --rest_bind_address 0.0.0.0,::
🛠 Summary
CVS-170537