[Serve] Check multiple FastAPI ingress deployments in a single application #53647

Ziy1-Tan · 2025-06-08T09:55:29Z

Why are these changes needed?

Currently, Serve can not catch multiple FastAPI deployments in a single application if user sets the docs path to None in their FastAPI app.
We can check multiple ASGIAppReplicaWrapper in a single application to avoid this issue.

Related issue number

Closes #53024

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

Copilot

Pull Request Overview

This PR enhances Ray Serve’s application state logic to detect multiple FastAPI ingress deployments via @serve.ingress rather than relying on docs_path.

Import ASGIAppReplicaWrapper and use it to count ingress deployments.
Update _check_routes to raise if more than one ingress deployment is found.
Revise exception messaging and docstring to reflect the new check.

Comments suppressed due to low confidence (1)

python/ray/serve/_private/application_state.py:710

The docstring’s Returns section still references docs_path. Consider updating it to clarify that the function returns (route_prefix, docs_path) and that docs_path is retained for backward compatibility.

    """Check route prefixes and @serve.ingress of deployments in app.

Copilot · 2025-06-08T09:56:09Z

python/ray/serve/_private/application_state.py

These adjacent string literals concatenate without a space, producing @serve.ingressin. Add a trailing space in the first string or a leading space in the second to fix the message formatting.

Suggested change

"Please only include one deployment with @serve.ingress"

"Please only include one deployment with @serve.ingress "

Ziy1-Tan · 2025-06-08T13:49:40Z

Hey, @abrarsheikh . Do you know why this test case fails? It seems not to be related to my code changes.

Serve test is blocked by test_delete_multi_app:

(myenv) simple@wsl2-ubuntu:~/c/p/r/python[multi_app]> pytest -vs ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_delete_multi_app
"NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2025-06-08 21:46:22,213 - INFO - NumExpr defaulting to 8 threads.
2025-06-08 21:46:23,578 - INFO - Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
2025-06-08 21:46:23,578 - INFO - NumExpr defaulting to 8 threads.
*** Starting Iteration 1/2 ***

Sending PUT request for config.
PUT request sent successfully.


――――――――――――――――――――――――――――――――――――――――― test_delete_multi_app ―――――――――――――――――――――――――――――――――――――――――

ray_start_stop = None

    @pytest.mark.skipif(
        sys.platform == "darwin" and not TEST_ON_DARWIN, reason="Flaky on OSX."
    )
    def test_delete_multi_app(ray_start_stop):
        py_module = (
            "https://github.com/ray-project/test_module/archive/"
            "aa6f366f7daa78c98408c27d917a983caa9f888b.zip"
        )
        config = {
            "applications": [
                {
                    "name": "app1",
                    "route_prefix": "/app1",
                    "import_path": "dir.subdir.a.add_and_sub.serve_dag",
                    "runtime_env": {
                        "working_dir": (
                            "https://github.com/ray-project/test_dag/archive/"
                            "78b4a5da38796123d9f9ffff59bab2792a043e95.zip"
                        )
                    },
                    "deployments": [
                        {
                            "name": "Subtract",
                            "ray_actor_options": {
                                "runtime_env": {"py_modules": [py_module]}
                            },
                        }
                    ],
                },
                {
                    "name": "app2",
                    "route_prefix": "/app2",
                    "import_path": "ray.serve.tests.test_config_files.world.DagNode",
                },
            ],
        }

        # Ensure the REST API is idempotent
        num_iterations = 2
        for iteration in range(1, num_iterations + 1):
            print(f"*** Starting Iteration {iteration}/{num_iterations} ***\n")

            print("Sending PUT request for config.")
            deploy_config_multi_app(config, SERVE_HEAD_URL)
>           wait_for_condition(
                lambda: requests.post("http://localhost:8000/app1", json=["ADD", 1]).text
                == "2",
                timeout=15,
            )

ray/dashboard/modules/serve/tests/test_serve_dashboard.py:240:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

condition_predictor = <function test_delete_multi_app.<locals>.<lambda> at 0x7fe56c8e1700>, timeout = 15
retry_interval_ms = 100, raise_exceptions = False, kwargs = {}, start = 1749390391.7724278
last_ex = None, message = "The condition wasn't met before the timeout expired."

    def wait_for_condition(
        condition_predictor,
        timeout=10,
        retry_interval_ms=100,
        raise_exceptions=False,
        **kwargs: Any,
    ):
        """Wait until a condition is met or time out with an exception.

        Args:
            condition_predictor: A function that predicts the condition.
            timeout: Maximum timeout in seconds.
            retry_interval_ms: Retry interval in milliseconds.
            raise_exceptions: If true, exceptions that occur while executing
                condition_predictor won't be caught and instead will be raised.
            **kwargs: Arguments to pass to the condition_predictor.

        Raises:
            RuntimeError: If the condition is not met before the timeout expires.
        """
        start = time.time()
        last_ex = None
        while time.time() - start <= timeout:
            try:
                if condition_predictor(**kwargs):
                    return
            except Exception:
                if raise_exceptions:
                    raise
                last_ex = ray._private.utils.format_error_message(traceback.format_exc())
            time.sleep(retry_interval_ms / 1000.0)
        message = "The condition wasn't met before the timeout expired."
        if last_ex is not None:
            message += f" Last exception: {last_ex}"
>       raise RuntimeError(message)
E       RuntimeError: The condition wasn't met before the timeout expired.

ray/_private/test_utils.py:615: RuntimeError

abrarsheikh · 2025-06-08T21:07:59Z

Most likely, the master is broken. Give it a few days foe someone to fix it, then you can pull master into your branch to fix the broken test

abrarsheikh · 2025-06-08T21:06:12Z

python/ray/serve/_private/application_state.py

I know that the old implementation abused this function, but let's extract out the logic for checking duplicate ASGI apps outside of this function.

Ziy1-Tan · 2025-06-15T09:14:24Z

Most likely, the master is broken. Give it a few days foe someone to fix it, then you can pull master into your branch to fix the broken test

@abrarsheikh, It is wired that running a single test ace pytest -vs ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_put_get_multi_app works. Running all tests fails: pytest -vs ray/dashboard/modules/serve/tests/test_serve_dashboard.py.

test_serve_dashboard.py::test_put_get_multi_app and test_serve_dashboard.py::test_delete_multi_app seem to affect each other, making them unstable. (Observed by skipping one of them and then running all test cases)

Ziy1-Tan · 2025-06-22T02:33:52Z

Most likely, the master is broken. Give it a few days foe someone to fix it, then you can pull master into your branch to fix the broken test

@abrarsheikh, It is wired that running a single test ace pytest -vs ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_put_get_multi_app works. Running all tests fails: pytest -vs ray/dashboard/modules/serve/tests/test_serve_dashboard.py.

test_serve_dashboard.py::test_put_get_multi_app and test_serve_dashboard.py::test_delete_multi_app seem to affect each other, making them unstable. (Observed by skipping one of them and then running all test cases)

cc @abrarsheikh @zcin . Do you have any idea about it?

abrarsheikh · 2025-06-23T16:14:56Z

master seems good

``` ❯ pytest -vs python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py Test session starts (platform: linux, Python 3.9.21, pytest 7.4.4, pytest-sugar 0.9.5) cachedir: .pytest_cache rootdir: /home/ubuntu/ray configfile: pytest.ini plugins: docker-tools-3.1.3, timeout-2.1.0, asyncio-0.17.2, forked-1.4.0, virtualenv-1.7.0, sugar-0.9.5, shutil-1.7.0, aiohttp-1.1.0, httpserver-1.0.6, nbval-0.11.0, anyio-3.7.1, sphinx-0.5.1.dev0, rerunfailures-11.1.2, lazy-fixture-0.6.3 timeout: 180.0s timeout method: signal timeout func_only: False asyncio: mode=auto collecting ... 2025-06-23 16:11:48,908 - INFO - NumExpr defaulting to 8 threads. 2025-06-23 16:11:50,464 - INFO - NumExpr defaulting to 8 threads. *** Starting Iteration 1/2 ***

Sending PUT request for config1.
PUT request sent successfully.
Deployments are live and reachable over HTTP.

Sending PUT request for config2.
PUT request sent successfully.
Adder deployment updated correctly.

Sending PUT request for config3.
PUT request sent successfully.
Deployments are live and reachable over HTTP.

*** Starting Iteration 2/2 ***

Sending PUT request for config1.
PUT request sent successfully.
Deployments are live and reachable over HTTP.

Sending PUT request for config2.
PUT request sent successfully.
Adder deployment updated correctly.

Sending PUT request for config3.
PUT request sent successfully.
Deployments are live and reachable over HTTP.

2025-06-23 16:12:11,299 - INFO - NumExpr defaulting to 8 threads.
python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_put_get_multi_app ✓ 11% █▎ 2025-06-23 16:12:13,032 - INFO - NumExpr defaulting to 8 threads.
2025-06-23 16:12:14,490 - INFO - NumExpr defaulting to 8 threads.

2025-06-23 16:12:19,255 - INFO - NumExpr defaulting to 8 threads.
python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_put_bad_schema ✓ 22% ██▎ 2025-06-23 16:12:20,960 - INFO - NumExpr defaulting to 8 threads.
2025-06-23 16:12:22,424 - INFO - NumExpr defaulting to 8 threads.

2025-06-23 16:12:27,374 - INFO - NumExpr defaulting to 8 threads.
python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_put_duplicate_apps ✓ 33% ███▍ 2025-06-23 16:12:29,042 - INFO - NumExpr defaulting to 8 threads.
2025-06-23 16:12:30,515 - INFO - NumExpr defaulting to 8 threads.

2025-06-23 16:12:35,480 - INFO - NumExpr defaulting to 8 threads.
python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_put_duplicate_routes ✓ 44% ████▌ 2025-06-23 16:12:37,140 - INFO - NumExpr defaulting to 8 threads.
2025-06-23 16:12:38,616 - INFO - NumExpr defaulting to 8 threads.
*** Starting Iteration 1/2 ***

Sending PUT request for config.
PUT request sent successfully.
Deployments are live and reachable over HTTP.

Sending DELETE request for config.
DELETE request sent successfully.
Deployments have been deleted and are not reachable.

*** Starting Iteration 2/2 ***

Sending PUT request for config.
PUT request sent successfully.
Deployments are live and reachable over HTTP.

Sending DELETE request for config.
DELETE request sent successfully.
Deployments have been deleted and are not reachable.

2025-06-23 16:13:00,779 - INFO - NumExpr defaulting to 8 threads.
python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_delete_multi_app ✓ 56% █████▋ 2025-06-23 16:13:02,525 - INFO - NumExpr defaulting to 8 threads.
2025-06-23 16:13:04,052 - INFO - NumExpr defaulting to 8 threads.

2025-06-23 16:13:08,938 - INFO - NumExpr defaulting to 8 threads.
python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_get_serve_instance_details_not_started ✓ 67% ██████▋ 2025-06-23 16:13:10,681 - INFO - NumExpr defaulting to 8 threads.
2025-06-23 16:13:12,150 - INFO - NumExpr defaulting to 8 threads.
PUT request sent successfully.
All applications are in a RUNNING state.
Confirmed fetched proxy location, HTTP host, HTTP port, gRPC port, and grpc_servicer_functions metadata correct.
Checked HTTP Proxy details.
Finished checking application details.

2025-06-23 16:13:21,414 - INFO - NumExpr defaulting to 8 threads.
python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_get_serve_instance_details[f_deployment_options0] ✓ 78% ███████▊ 2025-06-23 16:13:23,093 - INFO - NumExpr defaulting to 8 threads.
2025-06-23 16:13:24,590 - INFO - NumExpr defaulting to 8 threads.
PUT request sent successfully.
All applications are in a RUNNING state.
Confirmed fetched proxy location, HTTP host, HTTP port, gRPC port, and grpc_servicer_functions metadata correct.
Checked HTTP Proxy details.
Finished checking application details.

2025-06-23 16:13:34,133 - INFO - NumExpr defaulting to 8 threads.
python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_get_serve_instance_details[f_deployment_options1] ✓ 89% ████████▉ 2025-06-23 16:13:35,889 - INFO - NumExpr defaulting to 8 threads.
2025-06-23 16:13:37,346 - INFO - NumExpr defaulting to 8 threads.
Deployed app1
Deployed app2

All applications are in a RUNNING state.
Finished checking application details.

2025-06-23 16:13:48,140 - INFO - NumExpr defaulting to 8 threads.
python/ray/dashboard/modules/serve/tests/test_serve_dashboard.py::test_get_serve_instance_details_for_imperative_apps ✓ 100% ██████████

Results (123.63s):
9 passed

</details>

abrarsheikh · 2025-06-26T20:29:42Z

nice work handling the imperative and declarative code paths. I think its worth adding tests for both flows.

…ation Signed-off-by: Ziy1-Tan <[email protected]>

Signed-off-by: Ziy1-Tan <[email protected]>

…ation (ray-project#53647) ## Why are these changes needed? - Currently, Serve can not catch multiple FastAPI deployments in a single application if user sets the docs path to None in their FastAPI app. - We can check multiple ASGIAppReplicaWrapper in a single application to avoid this issue. ## Related issue number Closes ray-project#53024 ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [x] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Ziy1-Tan <[email protected]> Signed-off-by: ChanChan Mao <[email protected]>

…ation (ray-project#53647) ## Why are these changes needed? - Currently, Serve can not catch multiple FastAPI deployments in a single application if user sets the docs path to None in their FastAPI app. - We can check multiple ASGIAppReplicaWrapper in a single application to avoid this issue. ## Related issue number Closes ray-project#53024 ## Checks - [x] I've signed off every commit(by using the -s flag, i.e., `git commit -s`) in this PR. - [x] I've run `scripts/format.sh` to lint the changes in this PR. - [x] I've included any doc changes needed for https://docs.ray.io/en/master/. - [ ] I've added any new APIs to the API Reference. For example, if I added a method in Tune, I've added it in `doc/source/tune/api/` under the corresponding `.rst` file. - [x] I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/ - Testing Strategy - [ ] Unit tests - [ ] Release tests - [ ] This PR is not tested :( --------- Signed-off-by: Ziy1-Tan <[email protected]> Signed-off-by: jugalshah291 <[email protected]>

Copilot AI review requested due to automatic review settings June 8, 2025 09:55

Copilot AI reviewed Jun 8, 2025

View reviewed changes

Ziy1-Tan mentioned this pull request Jun 8, 2025

[Serve] Set the docs path after app is initialized on the replica #53463

Merged

8 tasks

abrarsheikh reviewed Jun 8, 2025

View reviewed changes

Ziy1-Tan force-pushed the multi_app branch 2 times, most recently from 4222805 to f4d7566 Compare June 15, 2025 08:42

Ziy1-Tan requested a review from a team as a code owner June 15, 2025 08:42

Ziy1-Tan force-pushed the multi_app branch from f4d7566 to 4153e7b Compare June 25, 2025 15:55

Ziy1-Tan requested a review from abrarsheikh June 25, 2025 23:43

Ziy1-Tan force-pushed the multi_app branch from 4153e7b to 0b9c676 Compare June 26, 2025 14:02

abrarsheikh added the go add ONLY when ready to merge, run all tests label Jun 26, 2025

akshay-anyscale added the serve Ray Serve Related Issue label Jun 27, 2025

Ziy1-Tan added 2 commits July 9, 2025 20:55

[Serve] Check multiple FastAPI ingress deployments in a single applic…

1ae21d8

…ation Signed-off-by: Ziy1-Tan <[email protected]>

Add tests for declarative code path

0729493

Signed-off-by: Ziy1-Tan <[email protected]>

Ziy1-Tan force-pushed the multi_app branch from 1886f2b to 0729493 Compare July 9, 2025 12:58

abrarsheikh approved these changes Jul 9, 2025

View reviewed changes

zcin merged commit 6e30704 into ray-project:master Jul 9, 2025
5 checks passed

darthhexx mentioned this pull request Aug 22, 2025

[Serve] job submit with custom modules fails to deserialize serialized_deployment_def #55836

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Serve] Check multiple FastAPI ingress deployments in a single application #53647

[Serve] Check multiple FastAPI ingress deployments in a single application #53647

Uh oh!

Ziy1-Tan commented Jun 8, 2025 •

edited

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Copilot AI Jun 8, 2025

Uh oh!

Ziy1-Tan commented Jun 8, 2025 •

edited

Loading

Uh oh!

abrarsheikh commented Jun 8, 2025

Uh oh!

abrarsheikh Jun 8, 2025

Uh oh!

Ziy1-Tan commented Jun 15, 2025 •

edited

Loading

Uh oh!

Ziy1-Tan commented Jun 22, 2025

Uh oh!

abrarsheikh commented Jun 23, 2025

Uh oh!

abrarsheikh commented Jun 26, 2025

Uh oh!

Uh oh!

Uh oh!

	"Please only include one deployment with @serve.ingress"
	"Please only include one deployment with @serve.ingress "

[Serve] Check multiple FastAPI ingress deployments in a single application #53647

[Serve] Check multiple FastAPI ingress deployments in a single application #53647

Uh oh!

Conversation

Ziy1-Tan commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why are these changes needed?

Related issue number

Checks

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Uh oh!

Copilot AI Jun 8, 2025

Choose a reason for hiding this comment

Uh oh!

Ziy1-Tan commented Jun 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abrarsheikh commented Jun 8, 2025

Uh oh!

abrarsheikh Jun 8, 2025

Choose a reason for hiding this comment

Uh oh!

Ziy1-Tan commented Jun 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Ziy1-Tan commented Jun 22, 2025

Uh oh!

abrarsheikh commented Jun 23, 2025

Uh oh!

abrarsheikh commented Jun 26, 2025

Uh oh!

Uh oh!

Uh oh!

Ziy1-Tan commented Jun 8, 2025 •

edited

Loading

Ziy1-Tan commented Jun 8, 2025 •

edited

Loading

Ziy1-Tan commented Jun 15, 2025 •

edited

Loading