Refactor `Dataset.map` to reuse cache files mapped with different `num_proc`#7434

Merged

lhoestq merged 18 commits intohuggingface:mainfrom

ringohoffman:reuse-cache-on-different-num_proc

May 12, 2025

Commits on Mar 4, 2025

Refactor Dataset.map to reuse cache files mapped with different num_proc
Matthew Hoffman
committed
Only give reprocessing message doing a partial remap
Matthew Hoffman
committed
Update logging message to account for if a cache file will be written at all and written by the main process or not
Matthew Hoffman
committed
Refactor string_to_dict to return None if there is no match instead of raising ValueError
Matthew Hoffman
committed
Merge branch 'return-none-if-string_to_dict-no-match' into reuse-cache-on-different-num_proc
Matthew Hoffman
committed
Simplify existing existing_cache_file_map with string_to_dict
Matthew Hoffman
committed
Set initial value if there are already existing cache files
Matthew Hoffman
committed

Commits on Mar 5, 2025

Merge branch 'main' into return-none-if-string_to_dict-no-match
Matthew Hoffman
committed
Merge branch 'return-none-if-string_to_dict-no-match' into reuse-cache-on-different-num_proc
Matthew Hoffman
committed

Commits on Mar 7, 2025

Allow for source_url_fields to be None
Matthew Hoffman
committed
Merge branch 'main' into return-none-if-string_to_dict-no-match
Matthew Hoffman
authored

Commits on Mar 9, 2025

Merge branch 'return-none-if-string_to_dict-no-match' into reuse-cache-on-different-num_proc
Matthew Hoffman
committed

Commits on Mar 12, 2025

Merge branch 'main' into reuse-cache-on-different-num_proc
Matthew Hoffman
committed
Add unicode escape to handle parsing string_to_dict in Windows paths
Matthew Hoffman
committed

Commits on Mar 14, 2025

Merge branch 'main' into reuse-cache-on-different-num_proc
lhoestq
authored
Remove glob_pattern_to_regex
Matthew Hoffman
committed

Commits on May 12, 2025