Remove scripts altogether

[lhoestq]

TODO:

• remplace fixtures based on script with no-script fixtures
• windaube

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[jp1924]

Hi @lhoestq,
I wanted to ask
are you planning to stop supporting dataset builds using GeneratorBasedBuilder?

If so, could you share the reason why?

[lhoestq]

We stopped supporting dataset scripts altogether, whether they are based on GeneratorBasedBuilder or any other builder. This means you can't load_dataset() a dataset script anymore. We did this mostly for security reasons which is blocking for many users and also impossible to build upon (e.g. the for the Dataset Viewer on HF)

[jp1924]

Ah, so only the trust_remote_code feature of load_dataset is deprecated, and

from datasets import load_dataset_builder
  
builder = load_dataset_builder('cornell-movie-review-data/rotten_tomatoes')  
builder.download_and_prepare()  

we can still load data using load_dataset_builder and download_and_prepare, right?
That's a relief. I thought the removal of trust_remote_code in load_dataset meant GeneratorBasedBuilder was being deprecated too, haha.
Got it, thanks for the clarification!

[autosquid]

Can you give an example on how to upgrade from using trust_remote_code? I used to load_dataset from a script generating my training data in a streaming way.

[autosquid]

For guys who dislike this change +1