Add custom fingerprint support to from_generator

[simonreise]

This PR adds dataset_id_suffix parameter to 'Dataset.from_generator' function.

Dataset.from_generator function passes all of its arguments to BuilderConfig.create_config_id, including generator function itself. BuilderConfig.create_config_id function tries to hash all the args, which can take a large amount of time or even cause MemoryError if the dataset processed in a generator function is large enough.

This PR allows user to pass a custom fingerprint (dataset_id_suffix) to be used as a suffix in a dataset name instead of the one generated by hashing the args.

This PR is a possible solution of #7513

[lhoestq]

This is great !

What do you think of passing config_id= directly to the builder instead of just the suffix ? This would be a power user argument though, or for internal use. And in from_generator the new argument can be fingerprint= as in Dataset.__init__()

The config_id can be defined using something like config_id = "default-fingerprint=" + fingerprint

I feel ike this could make the Dataset API more coherent if we avoid introducing a new argument while we can juste use fingerprint=

[lhoestq]

added more comments

[lhoestq]

no need to modify create_config_id() imo, you can just use the user-defined config_id in the __init__

and in src/datasets/io/generator.py you can pass Generator(..., config_id=fingerpring)

[simonreise]

Thanks! I renamed fingerprint to config_id in Generator and DatasetBuilder. I removed the changes from create_config_id, but now I set config_id not in __init__, but in _create_builder_config, because we anyway have to stop this function from generating config_id from hash if we have custom config_id, and we also can use builder_config.name as default name in config_id = "default-fingerprint=" + fingerprint. I can move it to init, if needed.

[simonreise]

@lhoestq could you please re-review the changes I made?

[simonreise]

@lhoestq ping
I also added a simple test for the fingerprint parameter

[simonreise]

@lhoestq could you please review the PR? I implemented the requested changes and added a test for the added fingerprint arg

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[lhoestq]

lgtm :) I just did some minor edits