[FEATURE REQUEST] Multiprocessing with for dataset.map, dataset.filter

[timothyjlaurent]

It would be nice to be able to speed up dataset.map or dataset.filter. Perhaps this is as easy as sharding the dataset sending each shard to a process/thread/dask pool and using the new nlp.concatenate_dataset() function to join them all together?

[lhoestq]

Yes that's definitely something we plan to add ^^

[thomwolf]

Yes, that would be nice. We could take a look at what tensorflow tf.data does under the hood for instance.

[timothyjlaurent]

So tf.data.Dataset.map() returns a ParallelMapDataset if num_parallel_calls is not None link.

There, num_parallel_calls is turned into a tensor and and fed to gen_dataset_ops.parallel_map_dataset where it looks like tensorflow takes over.

We could start with something simple like a thread or process pool that imaps over some shards.

[lhoestq]

Multiprocessing was added in #552 . You can set the number of processes with .map(..., num_proc=...). It also works for filter

Closing this one, but feel free to reo-open if you have other questions

[Chen-Hailin]

@lhoestq Great feature implemented! Do you have plans to add it to official tutorials Processing data in a Dataset? It took me sometime to find this parallel processing api.

[lhoestq]

Thanks for the heads up !

I just added a paragraph about multiprocessing:
https://huggingface.co/docs/datasets/master/processing.html#multiprocessing