Skip to content

Commit 59738d7

Browse files
authored
No need for dataset_info (#7234)
* no need for dataset_info * actually do a single call in dataset_module_factory * update tests * fix tests * again * again * better check for gated error response code
1 parent d4422cc commit 59738d7

File tree

5 files changed

+200
-114
lines changed

5 files changed

+200
-114
lines changed

src/datasets/download/download_manager.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,11 @@ def _download_batched(
189189
download_func = partial(self._download_single, download_config=download_config)
190190

191191
fs: fsspec.AbstractFileSystem
192-
fs, path = url_to_fs(url_or_filenames[0], **download_config.storage_options)
192+
path = str(url_or_filenames[0])
193+
if is_relative_path(path):
194+
# append the relative path to the base_path
195+
path = url_or_path_join(self._base_path, path)
196+
fs, path = url_to_fs(path, **download_config.storage_options)
193197
size = 0
194198
try:
195199
size = fs.info(path).get("size", 0)

0 commit comments

Comments
 (0)