[BUG] AttributeError: vllm_model #1003
Description
Describe the bug
The following error occurred when using the vllm module. It did not occur when using the openai_llm module. This error may have occurred when the vllm_model was deleted. I am unsure of the exact cause, so I would appreciate it if you could investigate.
configuration
vectordb:
...
node_lines:
- node_line_name: retrieve_node_line
...
- node_line_name: post_retrieve_node_line
nodes:
- node_type: prompt_maker
...
- node_type: generator
strategy:
metrics: # bert_score 및 g_eval 사용 역시 추천합니다. 빠른 실행을 위해 여기서는 제외하고 하겠습니다.
- metric_name: rouge
- metric_name: sem_score
embedding_model: openai
# - metric_name: bert_score
# lang: ko
modules:
- module_type: openai_llm
llm: gpt-4o-mini
temperature: [ 0.1, 1.0 ]
batch: 16
- module_type: vllm
llm: meta-llama/Llama-3.2-1B-instruct
temperature: [ 0.1, 1.0 ]
# tensor_parallel_size: 4 # If the gpu is two.
max_tokens: 128
max_model_len: 200
gpu_memory_utilization: 0.5Full Error log
Ingesting VectorDB... ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100% 1/1 0:00:00
Evaluating... ━━━━━━━━━━━━━━━━━━━━━━━━━━╸━━━━━━━━━━━━━ 67% 2/3 0:03:08
[11/25/24 18:06:53] ERROR [__init__.py:60] >> Unexpected exception __init__.py:60
╭──────────────────────────────────────────── Traceback (most recent call last) ────────────────────────────────────────────╮
│ /root/.pyenv/versions/3.10.13/lib/python3.10/runpy.py:196 in _run_module_as_main │
│ │
│ 193 │ main_globals = sys.modules["__main__"].__dict__ │
│ 194 │ if alter_argv: │
│ 195 │ │ sys.argv[0] = mod_spec.origin │
│ ❱ 196 │ return _run_code(code, main_globals, None, │
│ 197 │ │ │ │ │ "__main__", mod_spec) │
│ 198 │
│ 199 def run_module(mod_name, init_globals=None, │
│ │
│ /root/.pyenv/versions/3.10.13/lib/python3.10/runpy.py:86 in _run_code │
│ │
│ 83 │ │ │ │ │ __loader__ = loader, │
│ 84 │ │ │ │ │ __package__ = pkg_name, │
│ 85 │ │ │ │ │ __spec__ = mod_spec) │
│ ❱ 86 │ exec(code, run_globals) │
│ 87 │ return run_globals │
│ 88 │
│ 89 def _run_module_code(code, init_globals=None, │
│ │
│ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher │
│ /../../debugpy/__main__.py:71 in <module> │
│ │
│ 68 │ │
│ 69 │ from debugpy.server import cli │
│ 70 │ │
│ ❱ 71 │ cli.main() │
│ 72 │
│ │
│ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher │
│ /../../debugpy/../debugpy/server/cli.py:501 in main │
│ │
│ 498 │ │ │ │ "code": run_code, │
│ 499 │ │ │ │ "pid": attach_to_pid, │
│ 500 │ │ │ }[options.target_kind] │
│ ❱ 501 │ │ │ run() │
│ 502 │ except SystemExit as exc: │
│ 503 │ │ log.reraise_exception( │
│ 504 │ │ │ "Debuggee exited via SystemExit: {0!r}", exc.code, level="debug" │
│ │
│ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/adapter/../../debugpy/launcher │
│ /../../debugpy/../debugpy/server/cli.py:351 in run_file │
│ │
│ 348 │ log.describe_environment("Pre-launch environment:") │
│ 349 │ │
│ 350 │ log.info("Running file {0!r}", target) │
│ ❱ 351 │ runpy.run_path(target, run_name="__main__") │
│ 352 │
│ 353 │
│ 354 def run_module(): │
│ │
│ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundl │
│ e/pydevd_runpy.py:310 in run_path │
│ │
│ 307 │ │ # Not a valid sys.path entry, so run the code directly │
│ 308 │ │ # execfile() doesn't help as we want to allow compiled files │
│ 309 │ │ code, fname = _get_code_from_file(run_name, path_name) │
│ ❱ 310 │ │ return _run_module_code(code, init_globals, run_name, pkg_name=pkg_name, │
│ script_name=fname) │
│ 311 │ else: │
│ 312 │ │ # Finder is defined for path, so add it to │
│ 313 │ │ # the start of sys.path │
│ │
│ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundl │
│ e/pydevd_runpy.py:127 in _run_module_code │
│ │
│ 124 │ fname = script_name if mod_spec is None else mod_spec.origin │
│ 125 │ with _TempModule(mod_name) as temp_module, _ModifiedArgv0(fname): │
│ 126 │ │ mod_globals = temp_module.module.__dict__ │
│ ❱ 127 │ │ _run_code(code, mod_globals, init_globals, mod_name, mod_spec, pkg_name, │
│ script_name) │
│ 128 │ # Copy the globals of the temporary module, as they │
│ 129 │ # may be cleared when the temporary module goes away │
│ 130 │ return mod_globals.copy() │
│ │
│ /root/.vscode-server/extensions/ms-python.debugpy-2024.12.0-linux-x64/bundled/libs/debugpy/_vendored/pydevd/_pydevd_bundl │
│ e/pydevd_runpy.py:118 in _run_code │
│ │
│ 115 │ run_globals.update( │
│ 116 │ │ __name__=mod_name, __file__=fname, __cached__=cached, __doc__=None, │
│ __loader__=loader, __package__=pkg_name, __spec__=mod_spec │
│ 117 │ ) │
│ ❱ 118 │ exec(code, run_globals) │
│ 119 │ return run_globals │
│ 120 │
│ 121 │
│ │
│ /app/evaluate/main.py:138 in <module> │
│ │
│ 135 │ # parse() │
│ 136 │ # chunk() │
│ 137 │ # make_qa() │
│ ❱ 138 │ evaluate() │
│ 139 │
│ │
│ /app/evaluate/main.py:127 in evaluate │
│ │
│ 124 │ │ os.makedirs(project_dir) │
│ 125 │ │
│ 126 │ evaluator = Evaluator(qa_data_path, corpus_data_path, project_dir=project_dir) │
│ ❱ 127 │ evaluator.start_trial(config) │
│ 128 │
│ 129 │
│ 130 if __name__ == "__main__": │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/evaluator.py:138 in │
│ start_trial │
│ │
│ 135 │ │ │ validator = Validator( │
│ 136 │ │ │ │ qa_data_path=self.qa_data_path, corpus_data_path=self.corpus_data_path │
│ 137 │ │ │ ) │
│ ❱ 138 │ │ │ validator.validate(yaml_path) │
│ 139 │ │ │
│ 140 │ │ os.environ["PROJECT_DIR"] = self.project_dir │
│ 141 │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/validator.py:92 in │
│ validate │
│ │
│ 89 │ │ │ │ corpus_data_path=corpus_path.name, │
│ 90 │ │ │ │ project_dir=temp_project_dir, │
│ 91 │ │ │ ) │
│ ❱ 92 │ │ │ evaluator.start_trial(yaml_path, skip_validation=True) │
│ 93 │ │ │ qa_path.close() │
│ 94 │ │ │ corpus_path.close() │
│ 95 │ │ │ os.unlink(qa_path.name) │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/evaluator.py:206 in │
│ start_trial │
│ │
│ 203 │ │ │ │ if i == 0: │
│ 204 │ │ │ │ │ previous_result = self.qa_data │
│ 205 │ │ │ │ logger.info(f"Running node line {node_line_name}...") │
│ ❱ 206 │ │ │ │ previous_result = run_node_line( │
│ 207 │ │ │ │ │ node_line, node_line_dir, previous_result, progress, task_eval │
│ 208 │ │ │ │ ) │
│ 209 │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/node_line.py:52 in │
│ run_node_line │
│ │
│ 49 │ │
│ 50 │ summary_lst = [] │
│ 51 │ for node in nodes: │
│ ❱ 52 │ │ previous_result = node.run(previous_result, node_line_dir) │
│ 53 │ │ node_summary_df = load_summary_file( │
│ 54 │ │ │ os.path.join(node_line_dir, node.node_type, "summary.csv") │
│ 55 │ │ ) │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/schema/node.py:57 │
│ in run │
│ │
│ 54 │ def run(self, previous_result: pd.DataFrame, node_line_dir: str) -> pd.DataFrame: │
│ 55 │ │ logger.info(f"Running node {self.node_type}...") │
│ 56 │ │ input_modules, input_params = self.get_param_combinations() │
│ ❱ 57 │ │ return self.run_node( │
│ 58 │ │ │ modules=input_modules, │
│ 59 │ │ │ module_params=input_params, │
│ 60 │ │ │ previous_result=previous_result, │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/nodes/generator/run │
│ .py:47 in run_generator_node │
│ │
│ 44 │ if "generation_gt" not in qa_data.columns: │
│ 45 │ │ raise ValueError("You must have 'generation_gt' column in qa.parquet.") │
│ 46 │ │
│ ❱ 47 │ results, execution_times = zip( │
│ 48 │ │ *map( │
│ 49 │ │ │ lambda x: measure_speed( │
│ 50 │ │ │ │ x[0].run_evaluator, │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/nodes/generator/run │
│ .py:49 in <lambda> │
│ │
│ 46 │ │
│ 47 │ results, execution_times = zip( │
│ 48 │ │ *map( │
│ ❱ 49 │ │ │ lambda x: measure_speed( │
│ 50 │ │ │ │ x[0].run_evaluator, │
│ 51 │ │ │ │ project_dir=project_dir, │
│ 52 │ │ │ │ previous_result=previous_result, │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/strategy.py:14 in │
│ measure_speed │
│ │
│ 11 │ Method for measuring execution speed of the function. │
│ 12 │ """ │
│ 13 │ start_time = time.time() │
│ ❱ 14 │ result = func(*args, **kwargs) │
│ 15 │ end_time = time.time() │
│ 16 │ return result, end_time - start_time │
│ 17 │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/schema/base.py:25 │
│ in run_evaluator │
│ │
│ 22 │ │ *args, │
│ 23 │ │ **kwargs, │
│ 24 │ ): │
│ ❱ 25 │ │ instance = cls(project_dir, *args, **kwargs) │
│ 26 │ │ result = instance.pure(previous_result, *args, **kwargs) │
│ 27 │ │ del instance │
│ 28 │ │ return result │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/nodes/generator/vll │
│ m.py:29 in __init__ │
│ │
│ 26 │ │ sampling_params_init_params = pop_params( │
│ 27 │ │ │ SamplingParams.from_optional, input_kwargs │
│ 28 │ │ ) │
│ ❱ 29 │ │ self.vllm_model = LLM(model, **input_kwargs) │
│ 30 │ │ │
│ 31 │ │ # delete not sampling param keys in the kwargs │
│ 32 │ │ kwargs_keys = list(kwargs.keys()) │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/utils.py:1028 in inner │
│ │
│ 1025 │ │ │ │ │ │ stacklevel=3, # The inner function takes up one level │
│ 1026 │ │ │ │ │ ) │
│ 1027 │ │ │ │
│ ❱ 1028 │ │ │ return fn(*args, **kwargs) │
│ 1029 │ │ │
│ 1030 │ │ return inner # type: ignore │
│ 1031 │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/entrypoints/llm.py:210 │
│ in __init__ │
│ │
│ 207 │ │ self.engine_class = self.get_engine_class() │
│ 208 │ │ │
│ 209 │ │ # TODO(rob): enable mp by default (issue with fork vs spawn) │
│ ❱ 210 │ │ self.llm_engine = self.engine_class.from_engine_args( │
│ 211 │ │ │ engine_args, usage_context=UsageContext.LLM_CLASS) │
│ 212 │ │ │
│ 213 │ │ self.request_counter = Counter() │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py:5 │
│ 85 in from_engine_args │
│ │
│ 582 │ │ engine_config = engine_args.create_engine_config() │
│ 583 │ │ executor_class = cls._get_executor_cls(engine_config) │
│ 584 │ │ # Create the LLM engine. │
│ ❱ 585 │ │ engine = cls( │
│ 586 │ │ │ vllm_config=engine_config, │
│ 587 │ │ │ executor_class=executor_class, │
│ 588 │ │ │ log_stats=not engine_args.disable_log_stats, │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/engine/llm_engine.py:3 │
│ 47 in __init__ │
│ │
│ 344 │ │ self.input_processor = input_registry.create_input_processor( │
│ 345 │ │ │ model_config) │
│ 346 │ │ │
│ ❱ 347 │ │ self.model_executor = executor_class(vllm_config=vllm_config, ) │
│ 348 │ │ │
│ 349 │ │ if self.model_config.task != "embedding": │
│ 350 │ │ │ self._initialize_kv_caches() │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/executor/executor_base │
│ .py:36 in __init__ │
│ │
│ 33 │ │ self.speculative_config = vllm_config.speculative_config │
│ 34 │ │ self.prompt_adapter_config = vllm_config.prompt_adapter_config │
│ 35 │ │ self.observability_config = vllm_config.observability_config │
│ ❱ 36 │ │ self._init_executor() │
│ 37 │ │
│ 38 │ @abstractmethod │
│ 39 │ def _init_executor(self) -> None: │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/executor/gpu_executor. │
│ py:39 in _init_executor │
│ │
│ 36 │ │ │ "GPUExecutor only supports single GPU.") │
│ 37 │ │ │
│ 38 │ │ self.driver_worker = self._create_worker() │
│ ❱ 39 │ │ self.driver_worker.init_device() │
│ 40 │ │ self.driver_worker.load_model() │
│ 41 │ │
│ 42 │ def _get_worker_kwargs( │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/worker/worker.py:145 │
│ in init_device │
│ │
│ 142 │ │ │ raise RuntimeError( │
│ 143 │ │ │ │ f"Not support device type: {self.device_config.device}") │
│ 144 │ │ # Initialize the distributed environment. │
│ ❱ 145 │ │ init_worker_distributed_environment(self.parallel_config, self.rank, │
│ 146 │ │ │ │ │ │ │ │ │ │ │ self.distributed_init_method, │
│ 147 │ │ │ │ │ │ │ │ │ │ │ self.local_rank) │
│ 148 │ │ # Set random seed. │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/worker/worker.py:462 │
│ in init_worker_distributed_environment │
│ │
│ 459 │ init_distributed_environment(parallel_config.world_size, rank, │
│ 460 │ │ │ │ │ │ │ │ distributed_init_method, local_rank) │
│ 461 │ │
│ ❱ 462 │ ensure_model_parallel_initialized(parallel_config.tensor_parallel_size, │
│ 463 │ │ │ │ │ │ │ │ │ parallel_config.pipeline_parallel_size) │
│ 464 │
│ 465 │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/vllm/distributed/parallel_s │
│ tate.py:1102 in ensure_model_parallel_initialized │
│ │
│ 1099 │ or ensure tensor-parallel and pipeline-parallel sizes are equal to expected │
│ 1100 │ values if the model parallel groups are initialized. │
│ 1101 │ """ │
│ ❱ 1102 │ backend = backend or torch.distributed.get_backend( │
│ 1103 │ │ get_world_group().device_group) │
│ 1104 │ if not model_parallel_is_initialized(): │
│ 1105 │ │ initialize_model_parallel(tensor_model_parallel_size, │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/torch/distributed/distribut │
│ ed_c10d.py:1215 in get_backend │
│ │
│ 1212 │ if _rank_not_in_group(pg): │
│ 1213 │ │ raise ValueError("Invalid process group specified") │
│ 1214 │ pg_store = _world.pg_map[pg] if pg in _world.pg_map else None │
│ ❱ 1215 │ return Backend(not_none(pg_store)[0]) │
│ 1216 │
│ 1217 │
│ 1218 def _get_process_group_uid(pg: ProcessGroup) -> int: │
│ │
│ /root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/torch/utils/_typing_utils.p │
│ y:13 in not_none │
│ │
│ 10 │
│ 11 def not_none(obj: Optional[T]) -> T: │
│ 12 │ if obj is None: │
│ ❱ 13 │ │ raise TypeError("Invariant encountered: value was None when it should not be") │
│ 14 │ return obj │
│ 15 │
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: Invariant encountered: value was None when it should not be
Exception ignored in: <function Vllm.__del__ at 0x7f35f96b6dd0>
Traceback (most recent call last):
File "/root/.cache/pypoetry/virtualenvs/app-rag-sample-9TtSrW0h-py3.10/lib/python3.10/site-packages/autorag/nodes/generator/vllm.py", line 53, in __del__
AttributeError: vllm_model
[rank0]:[W1125 18:06:59.980650822 ProcessGroupNCCL.cpp:1250] Warning: WARNING: process group has NOT been destroyed before we destruct ProcessGroupNCCL. On normal program exit, the application should call destroy_process_group to ensure that any pending NCCL operations have finished in this process. In rare cases this process can exit before this point and block the progress of another member of the process group. This constraint has always been present, but this warning has only been added since PyTorch 2.4 (function operator())
Desktop (please complete the following information):
- OS: Ubuntu 20.04
- Python version: 3.10.13