[RLlib] Enhance ConnectorV2 ObservationPreprocessor APIs (add multi-agent support; add episode arg).

[sven1977]

Enhance ConnectorV2 ObservationPreprocessor APIs:

• add multi-agent support
• add episode arg (so that for example rewards/actions can be added to an observation during preprocessing).

Why are these changes needed?

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[Copilot]

Pull Request Overview

This PR enhances the ConnectorV2 ObservationPreprocessor APIs to add multi-agent support and introduces an extra episode argument to the preprocess function. Key changes include:

• Renaming the existing ObservationPreprocessor to SingleAgentObservationPreprocessor and adding a new MultiAgentObservationPreprocessor.
• Updating the preprocess method signature and call implementations to pass the corresponding episode object.
• Adjusting several connector components (e.g., numpy_to_tensor, add_time_dim_to_batch_and_zero_pad, add_states_from_episodes_to_batch) to safely handle cases when rl_module is None and updating documentation accordingly.

Reviewed Changes

Copilot reviewed 12 out of 21 changed files in this pull request and generated no comments.

Show a summary per file

File	Description
rllib/connectors/env_to_module/observation_preprocessor.py	Renamed and extended preprocessor APIs for single-agent and multi-agent setups.
rllib/connectors/common/numpy_to_tensor.py	Removed the as_learner_connector parameter and added a conditional check on rl_module.
rllib/connectors/common/add_time_dim_to_batch_and_zero_pad.py	Updated rl_module check to account for None values.
rllib/connectors/common/add_states_from_episodes_to_batch.py	Updated rl_module check to account for None values.
doc/source/rllib/*.rst	Updated documentation to include connector-v2 and related new APIs.

Comments suppressed due to low confidence (2)

rllib/connectors/env_to_module/observation_preprocessor.py:162

• Consider implementing a proper 'set_observations' API for multi-agent episodes instead of relying on the single-agent workaround.

            # TODO (sven): Implement set_observations API for multi-agent episodes.

rllib/connectors/common/numpy_to_tensor.py:60

• The removal of the 'as_learner_connector' parameter may affect users expecting that configuration. Ensure that the associated documentation and any dependent code reflect this change.

        *,

[simonsays1980]

LGTM.