Add a fix for custom code tokenizers in pipelines#32300

Rocketknight1 merged 10 commits intomainhuggingface/transformers:mainfrom

fix_custom_tokenizer_in_pipelinehuggingface/transformers:fix_custom_tokenizer_in_pipelineCopy head branch name to clipboard