Error while downloading datasets from Sfaira

[AlmogAngel]

Dear developers,

After using setup_sfaira, I tried to download all human blood-related datasets by running the following command:

ds_blood <- dataset_sfaira_multiple(sfaira_setup = setup_list,
                                    organisms = "Homo sapiens",
                                    tissues = "blood",
                                    name="human_blood")

However, I ran into this error:

[1] "Starting to download dataset from Sfaria with organism: Homo sapiens , tissue: blood and assay: "
[1] "Removing datasets without cell-type annotation..."
[1] "Downloading datasets..."
Downloading: NCOMMS-19-7936188_scRNAseq_raw_UMIs.mtx
Downloading: NCOMMS-19-7936188_scRNAseq_genes.tsv
Downloading: NCOMMS-19-7936188_scRNAseq_barcodes.tsv
Downloading: NCOMMS-19-7936188_metadata.txt
Could not download all datasets for specified filters.
[1] "urllib.error.HTTPError: HTTP Error 404: Not Found\n"
Error in t.default(sfaira_data$X) : argument is not a matrix

Could you please provide me with some insights on how should I approach this?

Thanks!
Almog

[alex-d13]

Hi Almog,

Thanks for reaching out to us with this issue.
There seems to be an issue with one dataset in your selected subset: homosapiens_blood_2020_10x3v3_schulteschreppingjonas_001_10.1016/j.cell.2020.08.001, where the metadata download link is not existing.
What you could try for now is the following:

• use sfaira_overview to get a table with all existing datasets in sfaira
• subset this table by human and blood
• remove this one sample manually from the table
• store all remaining ids (id column in table) as list
• use download_sfaira with this list of ids

If you encounter similar issues with other datasets, maybe this approach can also help.

I will add a option to filter out datasets which would fail to download, so that this issue can be avoided in future.

Please let me know if this helped you!

Best,
Alex

[AlmogAngel]

Hi Almog,

Thanks for reaching out to us with this issue. There seems to be an issue with one dataset in your selected subset: homosapiens_blood_2020_10x3v3_schulteschreppingjonas_001_10.1016/j.cell.2020.08.001, where the metadata download link is not existing. What you could try for now is the following:

• use sfaira_overview to get a table with all existing datasets in sfaira
• subset this table by human and blood
• remove this one sample manually from the table
• store all remaining ids (id column in table) as list
• use download_sfaira with this list of ids

If you encounter similar issues with other datasets, maybe this approach can also help.

I will add a option to filter out datasets which would fail to download, so that this issue can be avoided in future.

Please let me know if this helped you!

Best, Alex

Thank you for the response!

I tried the following commands:

all_datasets <- SimBu::sfaira_overview(setup_list = setup_list)

ds_ids <- all_datasets %>% 
  filter(organism == "Homo sapiens" & organ == "blood" & annotated == TRUE) %>% 
  pull(id) %>% 
  unique()

ds_ids <- ds_ids[!ds_ids == "homosapiens_blood_2020_10x3v3_schulteschreppingjonas_001_10.1016/j.cell.2020.08.001"]

ds_blood <- SimBu::download_sfaira(sfaira_setup = setup_list, id = ds_ids)

This leads me to another error:

Error: 'download_sfaira' is not an exported object from 'namespace:SimBu'

This one is strange since the ?download_sfaira command for the documentation works fine.

[alex-d13]

Sorry, I think I got the dataset_sfaira and download_sfaira methods mixed up.
Could you try the dataset_sfaria() function? The other one is supposed to be internally and is therefore not exported.

[AlmogAngel]

Sorry, I think I got the dataset_sfaira and download_sfaira methods mixed up. Could you try the dataset_sfaria() function? The other one is supposed to be internally and is therefore not exported.

Unfortunately, seems like all blood-related datasets have some errors:

Could not download dataset for id: homosapiens_blood_2020_10x3v2_canogamez_001_10.1038/s41467-020-15543-yCould not download dataset for id: homosapiens_blood_2020_10x3v3_schulteschreppingjonas_002_10.1016/j.cell.2020.08.001Could not download dataset for id: homosapiens_blood_2020_singlecelllibraryconstruction_schulteschreppingjonas_003_10.1016/j.cell.2020.08.001Could not download dataset for id: homosapiens_blood_2020_singlecelllibraryconstruction_schulteschreppingjonas_004_10.1016/j.cell.2020.08.001Could not download dataset for id: homosapiens_blood_2020_singlecelllibraryconstruction_schulteschreppingjonas_005_10.1016/j.cell.2020.08.001Could not download dataset for id: homosapiens_blood_2020_singlecelllibraryconstruction_schulteschreppingjonas_006_10.1016/j.cell.2020.08.001Could not download dataset for id: homosapiens_blood_2019_10x3transcriptionprofiling_szabo_013_10.1038/s41467-019-12464-3Could not download dataset for id: homosapiens_blood_2019_10x3transcriptionprofiling_szabo_014_10.1038/s41467-019-12464-3Could not download dataset for id: homosapiens_blood_2019_10x3transcriptionprofiling_szabo_015_10.1038/s41467-019-12464-3Could not download dataset for id: homosapiens_blood_2019_10x3transcriptionprofiling_szabo_016_10.1038/s41467-019-12464-3Could not download dataset for id: homosapiens_blood_2021_citeseq(cellsurfaceproteinprofiling)_nathanaparna_001_10.1038/s41590-021-00933-1Could not download dataset for id: homosapiens_blood_2020_citeseq(cellsurfaceproteinprofiling)_haoyuhan_001_10.1016/j.cell.2021.04.048
[1] "urllib.error.HTTPError: HTTP Error 404: Not Found\n"
Error in t.default(sfaira_data$X) : argument is not a matrix

[alex-d13]

Hi @AlmogAngel

I recently tried to fix the issue that you were facing. Could you update SimBu and check if the error above has been resolved?

Best,
Alex

[AlmogAngel]

Hi @AlmogAngel

I recently tried to fix the issue that you were facing. Could you update SimBu and check if the error above has been resolved?

Best, Alex

Thanks!

It took me a long time to download all blood-related datasets and this time I have a different error:

> ds_blood <- dataset_sfaira_multiple(sfaira_setup = setup_list,
+                                        organisms = "Homo sapiens",
+                                        tissues = "blood",
+                                        name = "human_blood")
[1] "Starting to download dataset from Sfaria with organism: Homo sapiens , tissue: blood and assay: "
* Owlready2 * Warning: optimized Cython parser module 'owlready2_optimized' is not available, defaulting to slower Python implementation
2022-06-08 12:13:52.580997: I tensorflow/core/util/util.cc:169] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
[1] "Removing datasets without cell-type annotation..."
Ontology <class 'sfaira.versions.metadata.base.OntologyMondo'> is not a DAG, treat child-parent reasoning with care.
Ontology <class 'sfaira.versions.metadata.base.OntologyUberon'> is not a DAG, treat child-parent reasoning with care.
Ontology <class 'sfaira.versions.metadata.base.OntologyUberonLifecyclestage'> is not a DAG, treat child-parent reasoning with care.
Checking file accessibility for id:  homosapiens_blood_2020_10x3v2_canogamez_001_10.1038/s41467-020-15543-y
Checking file accessibility for id:  homosapiens_blood_2020_10x3v3_schulteschreppingjonas_001_10.1016/j.cell.2020.08.001
Checking file accessibility for id:  homosapiens_blood_2020_10x3v3_schulteschreppingjonas_002_10.1016/j.cell.2020.08.001
Checking file accessibility for id:  homosapiens_blood_2020_singlecelllibraryconstruction_schulteschreppingjonas_003_10.1016/j.cell.2020.08.001
Checking file accessibility for id:  homosapiens_blood_2020_singlecelllibraryconstruction_schulteschreppingjonas_004_10.1016/j.cell.2020.08.001
Checking file accessibility for id:  homosapiens_blood_2020_singlecelllibraryconstruction_schulteschreppingjonas_005_10.1016/j.cell.2020.08.001
Checking file accessibility for id:  homosapiens_blood_2020_singlecelllibraryconstruction_schulteschreppingjonas_006_10.1016/j.cell.2020.08.001
Checking file accessibility for id:  homosapiens_blood_2019_10x3transcriptionprofiling_szabo_013_10.1038/s41467-019-12464-3
Checking file accessibility for id:  homosapiens_blood_2019_10x3transcriptionprofiling_szabo_014_10.1038/s41467-019-12464-3
Checking file accessibility for id:  homosapiens_blood_2019_10x3transcriptionprofiling_szabo_015_10.1038/s41467-019-12464-3
Checking file accessibility for id:  homosapiens_blood_2019_10x3transcriptionprofiling_szabo_016_10.1038/s41467-019-12464-3
Checking file accessibility for id:  homosapiens_blood_2021_citeseq(cellsurfaceproteinprofiling)_nathanaparna_001_10.1038/s41590-021-00933-1
Checking file accessibility for id:  homosapiens_blood_2020_citeseq(cellsurfaceproteinprofiling)_haoyuhan_001_10.1016/j.cell.2021.04.048
[1] "Checking if IDs contain datasets which are currently known to be broken."
[1] "Downloading datasets..."
Setting options('download.file.method.GEOquery'='auto')
Setting options('GEOquery.inmemory.gpl'=FALSE)
... storing 'TB_status' as categorical
... storing 'cluster_name' as categorical
... storing 'cluster_ids' as categorical
... storing 'donor' as categorical
... storing 'disease' as categorical
... storing 'orig.ident' as categorical
... storing 'lane' as categorical
... storing 'donor' as categorical
... storing 'celltype.l1' as categorical
... storing 'celltype.l2' as categorical
... storing 'celltype.l3' as categorical
... storing 'Phase' as categorical
... storing 'Batch' as categorical
[1] "Streamlining features & meta-data..."
/home/almogangel/.cache/R/basilisk/1.6.0/SimBu/0.99.7/SimBu_env_0_99_4/lib/python3.9/site-packages/scipy/sparse/_index.py:116: SparseEfficiencyWarning: Changing the sparsity structure of a csc_matrix is expensive. lil_matrix is more efficient.
  self._set_arrayXarray_sparse(i, j, x)
Variable names are not unique. To make them unique, call `.var_names_make_unique`.
/home/almogangel/.cache/R/basilisk/1.6.0/SimBu/0.99.7/SimBu_env_0_99_4/lib/python3.9/site-packages/anndata/_core/anndata.py:120: ImplicitModificationWarning: Transforming to str index.
  warnings.warn("Transforming to str index.", ImplicitModificationWarning)
WARNING:root:name and ontology keys are both missing
Downloading: GSE158769_exprs_raw.tsv.gz
Downloading: GSE158769_meta_data.txt.gz
Downloading: GSE164378_RAW.tar
Downloading: GSE164378_sc.meta.data_3P.csv.gz
loading homosapiens_blood_2021_citeseq(cellsurfaceproteinprofiling)_nathanaparna_001_10.1038/s41590-021-00933-1
loading homosapiens_blood_2020_citeseq(cellsurfaceproteinprofiling)_haoyuhan_001_10.1016/j.cell.2021.04.048
Could not download all datasets for specified filters.
[1] "ValueError: 'nan' is not a valid entry for cell_type in data set ['10.1038/s41590-021-00933-1'].\n"
Error in t.default(sfaira_data$X) : argument is not a matrix
In addition: There were 23 warnings (use warnings() to see them)

[AlmogAngel]

Also, I was trying to download a single dataset with dataset_sfaira:

> ds_blood <- dataset_sfaira(sfaira_setup = setup_list,
+                            sfaira_id = "homosapiens_blood_2020_citeseq(cellsurfaceproteinprofiling)_haoyuhan_001_10.1016/j.cell.2021.04.048")
[1] "Starting to download dataset from Sfaria with id: homosapiens_blood_2020_citeseq(cellsurfaceproteinprofiling)_haoyuhan_001_10.1016/j.cell.2021.04.048"
Checking file accessibility for id:  homosapiens_blood_2020_citeseq(cellsurfaceproteinprofiling)_haoyuhan_001_10.1016/j.cell.2021.04.048
[1] "Checking if IDs contain datasets which are currently known to be broken."
[1] "Downloading datasets..."
[1] "Streamlining features & meta-data..."
Using rownames for cell-IDs.
Error in generate_summarized_experiment(annotation = annotation, count_matrix = count_matrix,  : 
  argument "name" is missing, with no default
In addition: Warning messages:
1: In any(meta_url_private) : coercing argument of type 'list' to logical
2: In any(data_url_private) : coercing argument of type 'list' to logical