Skip to content

Fix 'dir' handling for cloud file systems #122

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
pinin4fjords merged 6 commits into from
Sep 23, 2025
Merged

Fix 'dir' handling for cloud file systems #122

pinin4fjords merged 6 commits into from
Sep 23, 2025

Conversation

@pinin4fjords
Copy link
Member

@pinin4fjords pinin4fjords commented Sep 22, 2025 •
edited
Loading

The current code has a big problem, in that here it puts a file object from UNTAR through a map, which flattens it into a string. A Path object representing an S3 bucket path like s3://foo/bar becomes just /foo/bar.

That causes problems when you get to here, because the flattened path string (see above) looks like, but isn't, a local file path, so the FASTQ files are not found.

This PR addresses the issue by letting the Path object from UNTAR remain a path, and by making a path object (using file) from any actual directory paths in the sample sheet). Then we don't need the file() call in the Groovy function, and we can derive fastq files by querying the object.

Local paths DO flatten to something valid, which is why it worked in those contexts.

Also snuck a tower.yml in.

PR checklist

  • This comment contains a description of changes (with reason).
  • If you've fixed a bug or added code that should be tested, add tests!
  • If you've added a new tool - have you followed the pipeline conventions in the contribution docs
  • If necessary, also make a PR on the nf-core/spatialvi branch on the nf-core/test-datasets repository.
  • Make sure your code lints (nf-core pipelines lint).
  • Ensure the test suite passes (nextflow run . -profile test,docker --outdir <OUTDIR>).
  • Check for unexpected warnings in debug mode (nextflow run . -profile debug,test,docker --outdir <OUTDIR>).
  • Usage Documentation in docs/usage.md is updated.
  • Output Documentation in docs/output.md is updated.
  • CHANGELOG.md is updated.
  • README.md is updated (including new tool citations and authors/contributors).

@github-actions
Copy link

github-actions bot commented Sep 22, 2025 •
edited
Loading

nf-core pipelines lint overall result: Passed ✅ ⚠️

Posted for pipeline commit 693c0f2

+| ✅ 218 tests passed       |+
#| ❔   8 tests were ignored |#
!| ❗   6 tests had warnings |!

❗ Test warnings:

  • readme - README contains the placeholder zenodo.XXXXXXX. This should be replaced with the zenodo doi (after the first release).
  • pipeline_todos - TODO string in README.md: Add citation for pipeline after first release. Uncomment lines below and update Zenodo doi and badge at the top of this file.
  • pipeline_todos - TODO string in README.md: Add bibliography of tools and data used in your pipeline
  • pipeline_todos - TODO string in nextflow.config: Optionally, you can add a pipeline-specific nf-core config at https://github.com/nf-core/configs
  • pipeline_todos - TODO string in awsfulltest.yml: You can customise AWS full pipeline tests as required
  • pipeline_todos - TODO string in nextflow.config: Specify any additional parameters here

❔ Tests ignored:

  • files_exist - File is ignored: conf/igenomes.config
  • files_exist - File is ignored: conf/igenomes_ignored.config
  • files_unchanged - File ignored due to lint config: .gitattributes
  • files_unchanged - File ignored due to lint config: .github/PULL_REQUEST_TEMPLATE.md
  • files_unchanged - File ignored due to lint config: assets/nf-core-spatialvi_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-spatialvi_logo_light.png
  • files_unchanged - File ignored due to lint config: docs/images/nf-core-spatialvi_logo_dark.png
  • files_unchanged - File ignored due to lint config: .gitignore or .prettierignore

✅ Tests passed:

Run details

  • nf-core/tools version 3.3.2
  • Run at 2025-09-22 16:54:12

@pinin4fjords pinin4fjords linked an issue Sep 22, 2025 that may be closed by this pull request
No fastq_dir specified or no samples found in folder #116
Closed
FloWuenne
FloWuenne approved these changes Sep 22, 2025
View reviewed changes
Copy link
Contributor

@FloWuenne FloWuenne left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unblocks running test profiles and the pipeline on cloud infrastructure so LGMT 🚀 !

fasterius
fasterius approved these changes Sep 23, 2025
View reviewed changes
Copy link
Collaborator

@fasterius fasterius left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for the fix!

@pinin4fjords pinin4fjords merged commit 729c2c1 into dev Sep 23, 2025
14 checks passed
@pinin4fjords pinin4fjords deleted the debug_fastq_issue branch September 23, 2025 07:51
@nathanweeks nathanweeks mentioned this pull request Nov 4, 2025
Merged
11 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@FloWuenne FloWuenne FloWuenne approved these changes

@fasterius fasterius fasterius approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

No fastq_dir specified or no samples found in folder

4 participants

@pinin4fjords @FloWuenne @fasterius