Dataverse data access module

[abearab]

Aim

Including a draft dataverse data collector as part of gget tool boxes.

e.g.

gget dataverse -t dataverse.tsv -o path-to-dir

import gget
# df table with `id`, `name`, and `type` columns
gget.dataverse(df = df,  path = 'path-to-dir')

Tasks

• make a python module and CLI for data collection from dataverse
• collect data for a list of datasets defined as a JSON
• build a JSON file for a list of datasets deposited in a given dataverse DOI at certain version
• collect data by providing a dataverse DOI and version
• collect data by providing a list of dataverse datafiles (i.e. id, name, and type) in dataframe or csv/tsv file
• collect data by providing a list of dataverse datafiles (i.e. id, name, and type) as arguments

#170 (comment)

• Add documentation pages in docs/src/en
• Check/improve error handling
• Match the logging of other gget modules instead of using the print_sys function
• The table arg should be the positional argument for use from command line – gget dataverse table.tsv
• Make out a non-required argument with the current working directory as default

History

As part of GilbertLabUCSF/CanDI#34, related to #121, I draft a module to collect data from dataverse. Then I thought data collection from dataverse can be a more general functionality here as part of gget.

I mostly borrowed codes from gears/utils.py or TDC/load.py. However, there are other efforts such as pyDataverse and easyDataverse.

References

• https://guides.dataverse.org/en/latest/api/dataaccess.html
• https://github.com/mims-harvard/TDC/blob/main/tdc/utils/load.py

[abearab]

@lauraluebbert – I simplified this feature as discussed here #124 (comment). It should be more straightforward now and it works smoothly on my end, let me know what you think!

[lauraluebbert]

This looks great already!

I have two comments and a few minor requests listed below:

1. Please add documentation pages for your tool. Make sure to list yourself as the developer of this tool so you can get due credit (example: https://pachterlab.github.io/gget/en/bgee.html)
2. Error handling: I haven't played around with this yet, but what does your error handling look like (e.g. what if the user inputs a dataframe that does not have 'name', 'id', 'type' columns or there is a server error with Dataverse)? Please make sure the error messages are clear and tell the user what to do.

• Add documentation pages in docs/src/en (feel free to skip the Spanish versions for now or draft something with Google translate and I'll check it later)
• Check/improve error handling
• Match the logging of other gget modules instead of using the print_sys function
• The table arg should be the positional argument for use from command line (see other modules for examples). Like that, the user does not need to specify gget dataverse --table table.tsv, but can just run gget dataverse table.tsv
• Make out a non-required argument with the current working directory as default