AttributeError: 'DistributedDataParallel' object has no attribute 'generate'

[boblee22]

🐛 Describe the bug

When I ran accelerate launch examples/ppo_sentiments.py, the error below happened. Am I supposed to unwrap the ddp model?

AttributeError: 'DistributedDataParallel' object has no attribute 'generate'
╭──────────────────────────── Traceback (most recent call last) ────────────────────────────╮
│                                                                                           │
│ /home/user/bob_workspace/code/trlx/examples/ppo_sentiments.py:38 in <module
│   35 │   orch: PPOOrchestrator = get_orchestrator(cfg.train.orchestrator)(                │
│   36 │   │   model, pipeline, reward_fn=reward_fn, chunk_size=cfg.method.chunk_size       │
│   37 │   )                                                                                │
│ ❱ 38 │   orch.make_experience(cfg.method.num_rollouts)                                    │
│   39 │   model.learn()                                                                    │
│   40 │                                                                                    │
│   41 │   print("DONE!")                                                                   │
│ /home/user/bob_workspace/code/trlx/trlx/orchestrator/ppo_orchestrator.py:64 in        │
│    63 │   │   │                                                                   [82/2259]
│ ❱  64 │   │   │   query_tensors, response_tensors, response_text = self.rl_model.act(batc │
│    65 │   │   │   texts = [q + r for q, r in zip(batch.text, response_text)]              │
│    66 │   │   │   scores = self.score(texts)                                              │
│    67                                                                                     │
│                                                                                           │
│ /home/user/bob_workspace/code/trlx/trlx/model/accelerate_base_model.py:121 in act     │
│                                                                                           │
│   118 │   │   │   │   self.dummy_input.to(self.accelerator.device)                        │
│   119 │   │   │   )  # Dummy pass to make things play nice with accelerate                │
│   120 │   │   │   # Removed synced gpus                                                   │
│ ❱ 121 │   │   │   response = self.model.generate(                                         │
│   122 │   │   │   │   query_tensors,                                                      │
│   123 │   │   │   │   pad_token_id=self.tokenizer.eos_token_id,                           │
│   124 │   │   │   │   **self.config.method.gen_kwargs                                     │
│                                                                                           │
│ /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1185 in __getattr__     │
│                                                                                           │
│   1182 │   │   │   modules = self.__dict__['_modules']                                    │
│   1183 │   │   │   if name in modules:                                                    │
│   1184 │   │   │   │   return modules[name]                                               │
│ ❱ 1185 │   │   raise AttributeError("'{}' object has no attribute '{}'".format(           │
│   1186 │   │   │   type(self).__name__, name))                                            │
│   1187 │                                                                                  │
│   1188 │   def __setattr__(self, name: str, value: Union[Tensor, 'Module']) -> None:

My accelerate config

- `Accelerate` version: 0.13.2
- Platform: Linux-5.4.0-107-generic-x86_64-with-glibc2.31
- Python version: 3.9.5
- Numpy version: 1.23.4
- PyTorch version (GPU?): 1.11.0 (True)
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: MULTI_GPU
        - mixed_precision: no
        - use_cpu: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: all
        - main_process_ip: None
        - main_process_port: None
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {}
        - fsdp_config: {}
        - downcast_bf16: no

Which trlX version are you using?

trlx==1.0.0

Additional system and package information

No response

[LouisCastricato]

@reciprocated

[LouisCastricato]

Looks like that example was not updated with a recent api change. We'll be fixing this in a merge @reciprocated is doing later today or tomorrow.

[maxreciprocate]

It's fixed in #24, this should be open until that is merged. However there is also #51, so maybe we should clarify that we only test deepspeed integration, because fsdp also won't work.

[Dahoas]

Yes we should clarify we only support deepspeed. ppo_sentiments.py and the entire generation procedure assumes this.

@reciprocated can you mention this in the readme in your pr?

[boblee22]

Thanks! I just changed me config to use deepspeed.

- `Accelerate` version: 0.13.2
- Platform: Linux-5.4.0-107-generic-x86_64-with-glibc2.31
- Python version: 3.9.5
- Numpy version: 1.23.4
- PyTorch version (GPU?): 1.11.0 (True)
- `Accelerate` default config:
        - compute_environment: LOCAL_MACHINE
        - distributed_type: DEEPSPEED
        - mixed_precision: no
        - use_cpu: False
        - num_processes: 8
        - machine_rank: 0
        - num_machines: 1
        - gpu_ids: None
        - main_process_ip: None
        - main_process_port: None
        - rdzv_backend: static
        - same_network: True
        - main_training_function: main
        - deepspeed_config: {'gradient_accumulation_steps': 1, 'gradient_clipping': 1.0, 'zero3_init_flag': False, 'zero_stage': 1}
        - fsdp_config: {}
        - downcast_bf16: no

However, I encountered another error

Token indices sequence length is longer than the specified maximum sequence length for this model (626 > 512). Running this sequence through the model will result in indexing errors
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│                                                                                                  │
│ /home/user/bob_workspace/code/trlx/examples/ppo_sentiments.py:39 in <module>                 │
│                                                                                                  │
│   36 │   │   model, pipeline, reward_fn=reward_fn, chunk_size=cfg.method.chunk_size              │
│   37 │   )                                                                                       │
│   38 │   orch.make_experience(cfg.method.num_rollouts)                                           │
│ ❱ 39 │   model.learn()                                                                           │
│   40 │                                                                                           │
│   41 │   print("DONE!")                                                                          │
│   42                                                                                             │
│ /home/user/bob_workspace/code/trlx/trlx/model/accelerate_ppo_model.py:217 in learn           │
│                                                                                                  │
│   214 │   │   │   │   self.post_backward_callback()                                              │
│   215 │   │   │   │   self.accelerator.wait_for_everyone()                                       │
│   216 │   │   │                                                                                  │
│ ❱ 217 │   │   │   self.post_epoch_callback()                                                     │
│   218 │   │   │   self.accelerator.wait_for_everyone()                                           │
│   219                                                                                            │
│                                                                                                  │
│ /home/user/bob_workspace/code/trlx/trlx/model/accelerate_ppo_model.py:131 in                 │
│ post_epoch_callback                                                                              │
│                                                                                                  │
│   128 │   │   # TODO(dahoas): are experiences being made for dataloaders on each process or sa   │
│   129 │   │   self.epoch += 1                                                                    │
│   130 │   │   self.store.clear_history()                                                         │
│ ❱ 131 │   │   self.orch.make_experience(                                                         │
│   132 │   │   │   self.config.method.num_rollouts, self.iter_count                               │
│   133 │   │   )  # Collect more rollouts for training                                            │
│   134                                                                                            │
│                                                                                                  │
│ /home/user/bob_workspace/code/trlx/trlx/orchestrator/ppo_orchestrator.py:66 in               │
│ make_experience                                                                                  │
│                                                                                                  │
│    63 │   │   │                                                                                  │
│    64 │   │   │   query_tensors, response_tensors, response_text = self.rl_model.act(batch)      │
│    65 │   │   │   texts = [q + r for q, r in zip(batch.text, response_text)]                     │
│ ❱  66 │   │   │   scores = self.score(texts)                                                     │
│    67 │   │   │                                                                                  │
│    68 │   │   │   # Precompute logprobs, values                                                  │
│    69 │   │   │   all_tokens = torch.cat((query_tensors, response_tensors), dim=1)               │
│                                                                                                  │
│ /home/user/bob_workspace/code/trlx/trlx/orchestrator/ppo_orchestrator.py:49 in score         │
│                                                                                                  │
│    46 │   │   """                                                                                │
│    47 │   │   Batched scoring function taking text and generating scalar                         │
│    48 │   │   """                                                                                │
│ ❱  49 │   │   return self.rl_model.reward_fn(samples)                                            │
│    50 │                                                                                          │
│    51 │   def make_experience(self, num_rollouts: int = 1024, iter_count: int = 0):              │
│    52                                                                                            │
│                                                                                                  │
│ /home/user/bob_workspace/code/trlx/examples/ppo_sentiments.py:26 in reward_fn                │
│                                                                                                  │
│   23 │   │   │   "function_to_apply": None,                                                      │
│   24 │   │   │   "batch_size": cfg.method.chunk_size,                                            │
│   25 │   │   }                                                                                   │
│ ❱ 26 │   │   pipe_outputs = sentiment_pipe(samples, **sent_kwargs)                               │
│   27 │   │   scores = torch.tensor([output[1]["score"] for output in pipe_outputs])              │
│   28 │   │   return scores                                                                       │
│   29                                                                                             │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/pipelines/text_classification.py:140 in      │
│ __call__                                                                                         │
│                                                                                                  │
│   137 │   │   │                                                                                  │
│   138 │   │   │   If `top_k` is used, one such dictionary is returned per label.                 │
│   139 │   │   """                                                                                │
│ ❱ 140 │   │   result = super().__call__(*args, **kwargs)                                         │
│   141 │   │   # TODO try and retrieve it in a nicer way from _sanitize_parameters.               │
│   142 │   │   _legacy = "top_k" not in kwargs                                                    │
│   143 │   │   if isinstance(args[0], str) and _legacy:                                           │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/pipelines/base.py:1063 in __call__           │
│                                                                                                  │
│   1060 │   │   │   │   final_iterator = self.get_iterator(                                       │
│   1061 │   │   │   │   │   inputs, num_workers, batch_size, preprocess_params, forward_params,   │
│   1062 │   │   │   │   )                                                                         │
│ ❱ 1063 │   │   │   │   outputs = [output for output in final_iterator]                           │
│   1064 │   │   │   │   return outputs                                                            │
│   1065 │   │   │   else:                                                                         │
│   1066 │   │   │   │   return self.run_multi(inputs, preprocess_params, forward_params, postpro  │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/pipelines/base.py:1063 in <listcomp>         │
│                                                                                                  │
│   1060 │   │   │   │   final_iterator = self.get_iterator(                                       │
│   1061 │   │   │   │   │   inputs, num_workers, batch_size, preprocess_params, forward_params,   │
│   1062 │   │   │   │   )                                                                         │
│ ❱ 1063 │   │   │   │   outputs = [output for output in final_iterator]                           │
│   1064 │   │   │   │   return outputs                                                            │
│   1065 │   │   │   else:                                                                         │
│   1066 │   │   │   │   return self.run_multi(inputs, preprocess_params, forward_params, postpro  │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/pipelines/pt_utils.py:111 in __next__        │
│                                                                                                  │
│   108 │   │   │   return self.loader_batch_item()                                                │
│   109 │   │                                                                                      │
│   110 │   │   # We're out of items within a batch                                                │
│ ❱ 111 │   │   item = next(self.iterator)                                                         │
│   112 │   │   processed = self.infer(item, **self.params)                                        │
│   113 │   │   # We now have a batch of "inferred things".                                        │
│   114 │   │   if self.loader_batch_size is not None:                                             │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/pipelines/pt_utils.py:112 in __next__        │
│                                                                                                  │
│   109 │   │                                                                                      │
│   110 │   │   # We're out of items within a batch                                                │
│   111 │   │   item = next(self.iterator)                                                         │
│ ❱ 112 │   │   processed = self.infer(item, **self.params)                                        │
│   113 │   │   # We now have a batch of "inferred things".                                        │
│   114 │   │   if self.loader_batch_size is not None:                                             │
│   115 │   │   │   # Try to infer the size of the batch                                           │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/pipelines/base.py:990 in forward             │
│                                                                                                  │
│    987 │   │   │   │   inference_context = self.get_inference_context()                          │
│    988 │   │   │   │   with inference_context():                                                 │
│    989 │   │   │   │   │   model_inputs = self._ensure_tensor_on_device(model_inputs, device=se  │
│ ❱  990 │   │   │   │   │   model_outputs = self._forward(model_inputs, **forward_params)         │
│    991 │   │   │   │   │   model_outputs = self._ensure_tensor_on_device(model_outputs, device=  │
│    992 │   │   │   else:                                                                         │
│    993 │   │   │   │   raise ValueError(f"Framework {self.framework} is not supported")          │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/pipelines/text_classification.py:167 in      │
│ _forward                                                                                         │
│                                                                                                  │
│   164 │   │   return self.tokenizer(inputs, return_tensors=return_tensors, **tokenizer_kwargs)   │
│   165 │                                                                                          │
│   166 │   def _forward(self, model_inputs):                                                      │
│ ❱ 167 │   │   return self.model(**model_inputs)                                                  │
│   168 │                                                                                          │
│   169 │   def postprocess(self, model_outputs, function_to_apply=None, top_k=1, _legacy=True):   │
│   170 │   │   # `_legacy` is used to determine if we're running the naked pipeline and in back   │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1110 in _call_impl             │
│                                                                                                  │
│   1107 │   │   # this function, and just call forward.                                           │
│   1108 │   │   if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o  │
│   1109 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1110 │   │   │   return forward_call(*input, **kwargs)                                         │
│   1111 │   │   # Do not call functions when jit is used                                          │
│   1112 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1113 │   │   if self._backward_hooks or _global_backward_hooks:                                │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py:747 │
│ in forward                                                                                       │
│                                                                                                  │
│    744 │   │   """                                                                               │
│    745 │   │   return_dict = return_dict if return_dict is not None else self.config.use_return  │
│    746 │   │                                                                                     │
│ ❱  747 │   │   distilbert_output = self.distilbert(                                              │
│    748 │   │   │   input_ids=input_ids,                                                          │
│    749 │   │   │   attention_mask=attention_mask,                                                │
│    750 │   │   │   head_mask=head_mask,                                                          │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1110 in _call_impl             │
│                                                                                                  │
│   1107 │   │   # this function, and just call forward.                                           │
│   1108 │   │   if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o  │
│   1109 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1110 │   │   │   return forward_call(*input, **kwargs)                                         │
│   1111 │   │   # Do not call functions when jit is used                                          │
│   1112 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1113 │   │   if self._backward_hooks or _global_backward_hooks:                                │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py:566 │
│ in forward                                                                                       │
│                                                                                                  │
│    563 │   │   head_mask = self.get_head_mask(head_mask, self.config.num_hidden_layers)          │
│    564 │   │                                                                                     │
│    565 │   │   if inputs_embeds is None:                                                         │
│ ❱  566 │   │   │   inputs_embeds = self.embeddings(input_ids)  # (bs, seq_length, dim)           │
│    567 │   │   return self.transformer(                                                          │
│    568 │   │   │   x=inputs_embeds,                                                              │
│    569 │   │   │   attn_mask=attention_mask,                                                     │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/torch/nn/modules/module.py:1110 in _call_impl             │
│                                                                                                  │
│   1107 │   │   # this function, and just call forward.                                           │
│   1108 │   │   if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks o  │
│   1109 │   │   │   │   or _global_forward_hooks or _global_forward_pre_hooks):                   │
│ ❱ 1110 │   │   │   return forward_call(*input, **kwargs)                                         │
│   1111 │   │   # Do not call functions when jit is used                                          │
│   1112 │   │   full_backward_hooks, non_full_backward_hooks = [], []                             │
│   1113 │   │   if self._backward_hooks or _global_backward_hooks:                                │
│                                                                                                  │
│ /opt/conda/lib/python3.9/site-packages/transformers/models/distilbert/modeling_distilbert.py:130 │
│ in forward                                                                                       │
│                                                                                                  │
│    127 │   │   word_embeddings = self.word_embeddings(input_ids)  # (bs, max_seq_length, dim)    │
│    128 │   │   position_embeddings = self.position_embeddings(position_ids)  # (bs, max_seq_len  │
│    129 │   │                                                                                     │
│ ❱  130 │   │   embeddings = word_embeddings + position_embeddings  # (bs, max_seq_length, dim)   │
│    131 │   │   embeddings = self.LayerNorm(embeddings)  # (bs, max_seq_length, dim)              │
│    132 │   │   embeddings = self.dropout(embeddings)  # (bs, max_seq_length, dim)                │
│    133 │   │   return embeddings                                                                 │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RuntimeError: The size of tensor a (626) must match the size of tensor b (512) at non-singleton dimension 1

[LouisCastricato]

We're about to merge something that fixes this, give us a few hours. Thanks

[Dahoas]

It looks like you're generating sequences that exceed the sentiment pipeline's context size. Did you change the target generation length of the model to something larger than 48?

[boblee22]

I just ran accelerate launch examples/ppo_sentiments.py without changing anything. Just saw the codebase was updated just now. Let me pull the latest code and try again.

[LouisCastricato]

Is this resolved?

[boblee22]

Yes!