Skip to content

Conversation

Etherll
Copy link
Contributor

@Etherll Etherll commented Apr 16, 2025

This PR introduces a new boolean flag unsloth_force_compile (defaulting to False) to FastModel.from_pretrained.

When set to True, this flag forces Unsloth's compilation and patching steps (unsloth_compile_transformers) to run even if trust_remote_code=True. Previously, trust_remote_code=True would always skip these steps due to potential tracing issues and return this error:

--> 666     model_types, supports_sdpa = unsloth_compile_transformers(
    667         dtype                   = dtype,
    668         model_name              = model_name,
    669         model_types             = model_types,
    670         token                   = token,
    671         sdpa_dynamic_mask       = True,
    672         sdpa_bool_masks         = True,
    673         sdpa_gqa_replace        = True,
    674         sdpa_dynamic_compile    = True,
    675         compile_attention       = True,
    676         disable_causal_masks    = True,
    677         compile_torch_modules   = True,
    678         compile_custom_modules  = True,
    679         compile_function_calls  = True,
    680         fuse_lm_head            = True,
    681         gradient_checkpointing  = True,
    682         manual_replacements     = True,
    683         fast_lora_forwards      = True,
    684         fast_residual_stream    = False,
    685         accurate_accumulation   = True,
    686         epilogue_fusion         = True,
    687         max_autotune            = False,
    688         shape_padding           = True,
    689         cudagraphs              = False,
    690         debug                   = False,
    691         fullgraph               = fullgraph,
    692         import_from_cache       = False,
    693         disable                 = False,
    694         return_logits           = return_logits,
    695         trust_remote_code       = trust_remote_code,
    696     )
    697 pass
    699 # Check if this is local model since the tokenizer gets overwritten

TypeError: cannot unpack non-iterable NoneType object

Additionally, this PR improves the propagation and usage of the trust_remote_code flag, particularly within the vision model patching logic (unsloth/models/vision.py), ensuring it's consistently applied.

@danielhanchen danielhanchen merged commit 6e5534d into unslothai:main May 16, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants