Using custom DownloadConfig results in an error

[ynouri]

Version / Environment

Ubuntu 18.04
Python 3.6.8
nlp 0.4.0

Description

Loading imdb dataset works fine when when I don't specify any download_config argument. When I create a custom DownloadConfig object and pass it to the nlp.load_dataset function, this results in an error.

How to reproduce

Example without DownloadConfig --> works

import os

os.environ["HF_HOME"] = "/data/hf-test-without-dl-config-01/"

import logging
import nlp

logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":
    imdb = nlp.load_dataset(path="imdb")

Example with DownloadConfig --> doesn't work

import os

os.environ["HF_HOME"] = "/data/hf-test-with-dl-config-01/"

import logging
import nlp
from nlp.utils import DownloadConfig

logging.basicConfig(level=logging.INFO)

if __name__ == "__main__":
    download_config = DownloadConfig()
    imdb = nlp.load_dataset(path="imdb", download_config=download_config)

Error traceback:

Traceback (most recent call last):
  File "/.../example_with_dl_config.py", line 13, in <module>
    imdb = nlp.load_dataset(path="imdb", download_config=download_config)
  File "/.../python3.6/python3.6/site-packages/nlp/load.py", line 549, in load_dataset
    download_config=download_config, download_mode=download_mode, ignore_verifications=ignore_verifications,
  File "/.../python3.6/python3.6/site-packages/nlp/builder.py", line 463, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/.../python3.6/python3.6/site-packages/nlp/builder.py", line 518, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/.../python3.6/python3.6/site-packages/nlp/datasets/imdb/76cdbd7249ea3548c928bbf304258dab44d09cd3638d9da8d42480d1d1be3743/imdb.py", line 86, in _split_generators
    arch_path = dl_manager.download_and_extract(_DOWNLOAD_URL)
  File "/.../python3.6/python3.6/site-packages/nlp/utils/download_manager.py", line 220, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/.../python3.6/python3.6/site-packages/nlp/utils/download_manager.py", line 158, in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
  File "/.../python3.6/python3.6/site-packages/nlp/utils/download_manager.py", line 108, in _record_sizes_checksums
    self._recorded_sizes_checksums[url] = get_size_checksum_dict(path)
  File "/.../python3.6/python3.6/site-packages/nlp/utils/info_utils.py", line 79, in get_size_checksum_dict
    with open(path, "rb") as f:
IsADirectoryError: [Errno 21] Is a directory: '/data/hf-test-with-dl-config-01/datasets/extracted/b6802c5b61824b2c1f7dbf7cda6696b5f2e22214e18d171ce1ed3be90c931ce5'

[ynouri]

From my limited understanding, part of the issue seems related to the prepare_module and download_and_prepare functions each handling the case where no config is passed. For example, prepare_module does mutate the object passed and forces the flags extract_compressed_file and force_extract to True.

See:

• https://github.com/huggingface/nlp/blob/5fb61e1012bda724a9b6b847307d90a1380abfa5/src/nlp/load.py#L227
• https://github.com/huggingface/nlp/blob/5fb61e1012bda724a9b6b847307d90a1380abfa5/src/nlp/builder.py#L388

Maybe a cleaner solution would be to always instantiate a default DownloadConfig object at the top-level, have it as non-optional for the lower-level functions and treat it as immutable.

[thomwolf]

Thanks for the report, I'll take a look.

What is your specific use-case for providing a DownloadConfig object?

[ynouri]

Thanks. Our use case involves running a training job behind a corporate firewall with no access to any external resources (S3, GCP or other web resources).

I was thinking about a 2-steps process:

1. Download the resources / artifacts using some secure corporate channel, ie run nlp.load_dataset() without a specific DownloadConfig. After that, collect the files from the $HF_HOME folder
2. Copy the $HF_HOME folder in the firewalled environment. Run nlp.load_dataset() with a custom config DownloadConfig(local_files_only=True)

However this ends up a bit clunky in practice, even when solving the DownloadConfig issue above. For example, the filename hash computed in get_from_cache() differs in the local_files_only=False vs local_files_only=True case (local case defaults etag to None, which results in a different hash). So effectively step 2) above doesn't work because the hash computed differs from the hash in the cache folder. Some hacks / workaround are possible but this solution becomes very convoluted.
https://github.com/huggingface/nlp/blob/c214aa5a4430c1df1bcd0619fd94d6abdf9d2da7/src/nlp/utils/file_utils.py#L417

Would you recommend a different path?

[thomwolf]

I see.

Probably the easiest way for you would be that we add simple serialization/deserialization methods to the Dataset and DatasetDict objects once the data files have been downloaded and all the dataset is processed.

What do you think @lhoestq ?

[thomwolf]

This use-case will be solved with #571

[ynouri]

Thank you very much @thomwolf and @lhoestq we will give it a try