modeling_phi3 errors with AttributeError: 'DynamicCache' object has no attribute 'get_max_length'

[doctorpangloss]

System Info

• transformers version: 4.49.0.dev0 (315a9f4~1)
• Platform: Windows-10-10.0.20348-SP0
• Python version: 3.11.7
• Huggingface_hub version: 0.28.1
• Safetensors version: 0.5.2
• Accelerate version: 1.3.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.6.0+cu124 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): 0.10.2 (cpu)
• Jax version: 0.5.0
• JaxLib version: 0.5.0
• Using distributed or parallel set-up in script?: no
• Using GPU in script?: no
• GPU type: NVIDIA RTX A5000

Who can help?

@ArthurZucker

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Use Phi3 with any cache configuration, including default (DynamicCache)

I think get_max_length is probably declared on a mixin that isn't on the cache classes yet?

comfy_extras\nodes\nodes_language.py:361: in execute
    return model.generate(tokens, max_new_tokens, repetition_penalty, seed, sampler),
comfy\language\transformers_model_management.py:228: in generate
    output_ids = transformers_model.generate(
..\..\.venv\Lib\site-packages\torch\utils\_contextlib.py:116: in decorate_context
    return func(*args, **kwargs)
..\..\.venv\Lib\site-packages\transformers\generation\utils.py:2224: in generate
    result = self._sample(
..\..\.venv\Lib\site-packages\transformers\generation\utils.py:3198: in _sample
    model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
C:\Users\bberman\.cache\huggingface\modules\transformers_modules\c1358f8a35e6d2af81890deffbbfa575b978c62f\modeling_phi3.py:1292: in prepare_inputs_for_generation
    max_cache_length = past_key_values.get_max_length()

Expected behavior

related to #35168?

I'm not sure why this is only coming up with phi-3 so far

[zucchini-nlp]

Hey @doctorpangloss !

The get_max_length() has been deprecated for several releases and is removed in the last v4.48. It should be cache.get_max_cache_shape().

Since the source of error is the remote code stored on hub, you need to open an issue in model repo and ask authors to fix that. Also you can downgrade transformers version a a workaround

[jillianmclements]

I need to upgrade to v4.49 to use GRPO, so downgrading the transformers version isn't an option. Since the issue is with the HF hub's modeling_phi3.py, is there an easy way to get it to use the transformers' modeling_phi3.py instead (where it looks like the issue is fixed)? My current workaround is to download the transformers version and modify it to use absolute imports instead of relative imports.

[zucchini-nlp]

@doctorpangloss you can set trust_remote_code=False if you are using one of the supported checkpoints. Check out #32365 (comment)

[kylesayrs]

Unfortunately the same workaround doesn't seem to exist for phi3_v. @zucchini-nlp Are you aware of any limitations that would prevent a HF-maintained version of this model as well? I'd normally open a PR on HF, but the project hasn't been updated in 7 months.

[zucchini-nlp]

@kylesayrs yeah, we didn't ship Phi3-Vision to transformers and it is supported only by hub code. You can still run it by cloning the repo and changing the problematic line. Not sure if authors are actively updating the model code

[alothomas]

Hey I have the same error, with Phi2 backbone, did you manage to fix it ?

[ArthurZucker]

@kylesayrs limitation is just time 🤣 happy to have it in transformers

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

[Rocketknight1]

We currently have a user PR to add Phi-3.5-vision to the main library, which should resolve this: #36036

[jranaraki]

System Info

• transformers version: 4.49.0.dev0 (315a9f4~1)
• Platform: Windows-10-10.0.20348-SP0
• Python version: 3.11.7
• Huggingface_hub version: 0.28.1
• Safetensors version: 0.5.2
• Accelerate version: 1.3.0
• Accelerate config: not found
• PyTorch version (GPU?): 2.6.0+cu124 (True)
• Tensorflow version (GPU?): not installed (NA)
• Flax version (CPU?/GPU?/TPU?): 0.10.2 (cpu)
• Jax version: 0.5.0
• JaxLib version: 0.5.0
• Using distributed or parallel set-up in script?: no
• Using GPU in script?: no
• GPU type: NVIDIA RTX A5000

Who can help?

@ArthurZucker

Information

• The official example scripts[ ] My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)[x] My own task or dataset (give details below)

Reproduction

Use Phi3 with any cache configuration, including default (DynamicCache)

I think get_max_length is probably declared on a mixin that isn't on the cache classes yet?

comfy_extras\nodes\nodes_language.py:361: in execute
    return model.generate(tokens, max_new_tokens, repetition_penalty, seed, sampler),
comfy\language\transformers_model_management.py:228: in generate
    output_ids = transformers_model.generate(
..\..\.venv\Lib\site-packages\torch\utils\_contextlib.py:116: in decorate_context
    return func(*args, **kwargs)
..\..\.venv\Lib\site-packages\transformers\generation\utils.py:2224: in generate
    result = self._sample(
..\..\.venv\Lib\site-packages\transformers\generation\utils.py:3198: in _sample
    model_inputs = self.prepare_inputs_for_generation(input_ids, **model_kwargs)
C:\Users\bberman\.cache\huggingface\modules\transformers_modules\c1358f8a35e6d2af81890deffbbfa575b978c62f\modeling_phi3.py:1292: in prepare_inputs_for_generation
    max_cache_length = past_key_values.get_max_length()

Expected behavior

related to #35168?

I'm not sure why this is only coming up with phi-3 so far

Updating the line 1292 of modeling_phi3.py from max_cache_length = past_key_values.get_max_length() to max_cache_length = past_key_values.get_max_cache_shape() seems to be fixing the issue.

[kevalmorabia97]

Submitted a fix to Phi-3.5-Vision-Instruct huggingface repo: https://huggingface.co/microsoft/Phi-3.5-vision-instruct/discussions/38