Skip to content

Commit f73ffd0

Browse files
BioWilkormcolq
BioWilko
and
rmcolq
authored
Version 1.5.1 Release (#137)
* 1.4.0 pre-release version (#135) * Deprecate old tools and black format * Support fragmented data * use new trim primer arg in minion * reverse the (revolting) primer position logic * Fix find primer logic * Improve the cursed report writer logic * explicitly named function args, what a novel idea * add temp debugging prints to find_primer * Add logic to handle segments outside of primer scheme * fix typo * General bugfixes, test fix for overlapping variant issue * add min mapq argument * properly provide min_mapq to handle_segment * Fix unmatched read group issue * Fix 8 bit int error * First attempt to integrate clair3 as optional variant caller * Slightly change env yaml format * Specify env name in mamba gh action * Checkout repo before installing env * python 3.6.15 * Run on ubuntu 20.04 * unpin python * Unpin python entirely * Separate out tests * correct unit test path * remove retvar hints for compatibility * Fix silly relative imports * Switch to cyvcf and fix unit tests * Fix pipeline parse unit test * Fix fstring * Large changes to make clair3 work, various unit test updates * Fix fuzzy primer site match logic * Fix minion_validator path * Fix vcf merge bug with empty VCF files * Increase fuzzy match default threshold to 20 * Fix CHROM bug in vcf merge properly * Report failed workflow when variants are missed * Fix vcf merge again * Separate out tests * No counter anymore * Add C241T to SP1 variants * Add limit on ref pileup * Single copy of CVR1 12733 var * re-enable other consensus tests * First attempt to make minion validator less fragile * Unify testvariants, improve test logic * medaka has to be different smh * Index ref fasta when using the clair3 workflow * Add mean amplicon depth TSV output * Update CLI and scheme fetcher * Update align trim unit test * Use proper version of V3 bedfile * Fix pipeline unit test * Fix align trim amp depth bug * Call variants on non primertrimmed bam * Don't test MT vs clair * properly disable test * Add docker build push action on release * Grab wget in dockerfile * Update docs and readme * docker push on release only * Remove debugging print statement * Make fuzzy primer match logic clearer * Install procps for nxf compatibility * 1.4.0 dev m1 (#136) * mods to Dockerfile * wip * update * 1.4.1 version bump * Properly name fasta header * Fix correctly paired logic * Version bump * Fix double gzipping of vcf issue * Version bump * Ignore deletions in depth masking * Remove medaka and longshot, pre 1.4.5 push * Fix args.clair3 reference * Fix artic.utils import * Remove defaults from conda env yaml * Update docs / changelog * Add min-depth and min-variant-quality arguments * Fix colored file arg * 1.4.6 version bump * Fix small check_filter bug * Version bump * rename minion model-path to model-dir * version bump * Fix model_dir arg in minion * Add stale issues gh action * 1.5 version bump * properly put low quality variants in mask * 1.5.1 version bump * Update vcf filtering and clair3 command --------- Co-authored-by: Rachel Colquhoun <[email protected]>
1 parent 314f52b commit f73ffd0

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

58 files changed

+3720
-2345
lines changed

‎.github/workflows/docker-build-push.yml

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,39 @@
1+
name: quay_build_push
2+
3+
on:
4+
release:
5+
types: [published]
6+
7+
jobs:
8+
docker:
9+
runs-on: ubuntu-latest
10+
steps:
11+
- name: Docker meta
12+
id: meta
13+
uses: docker/metadata-action@v5
14+
with:
15+
images: quay.io/artic-network/fieldbioinformatics
16+
flavor: |
17+
latest=true
18+
tags: |
19+
type=semver,pattern={{version}}
20+
- name: Checkout
21+
uses: actions/checkout@v3
22+
- name: Set up QEMU
23+
uses: docker/setup-qemu-action@v3
24+
- name: Set up Docker Buildx
25+
uses: docker/setup-buildx-action@v3
26+
- name: Login to Docker Hub
27+
uses: docker/login-action@v3
28+
with:
29+
registry: quay.io
30+
username: ${{ secrets.QUAY_USERNAME }}
31+
password: ${{ secrets.QUAY_PASSWORD }}
32+
- name: Build and push
33+
uses: docker/build-push-action@v5
34+
with:
35+
context: .
36+
platforms: linux/amd64
37+
push: true
38+
tags: ${{ steps.meta.outputs.tags }}
39+
labels: ${{ steps.meta.outputs.labels }}

‎.github/workflows/stale-issues.yml

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
name: Close inactive issues
2+
on:
3+
schedule:
4+
- cron: "30 1 * * *"
5+
6+
jobs:
7+
close-issues:
8+
runs-on: ubuntu-latest
9+
permissions:
10+
issues: write
11+
pull-requests: write
12+
steps:
13+
- uses: actions/stale@v5
14+
with:
15+
days-before-issue-stale: 30
16+
days-before-issue-close: 14
17+
stale-issue-label: "stale"
18+
stale-issue-message: "This issue is stale because it has been open for 30 days with no activity."
19+
close-issue-message: "This issue was closed because it has been inactive for 14 days since being marked as stale."
20+
days-before-pr-stale: -1
21+
days-before-pr-close: -1
22+
repo-token: ${{ secrets.GITHUB_TOKEN }}

‎.github/workflows/unittests.yml

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
name: fieldbioinformatics_unit_tests
2+
3+
on: [push, pull_request]
4+
5+
jobs:
6+
build:
7+
runs-on: ubuntu-latest
8+
strategy:
9+
fail-fast: false
10+
steps:
11+
- uses: actions/checkout@v3
12+
- uses: mamba-org/setup-micromamba@v1
13+
name: Setup dependencies
14+
with:
15+
environment-file: environment.yml
16+
init-shell: >-
17+
bash
18+
cache-environment: true
19+
post-cleanup: "all"
20+
21+
- name: Pip install artic
22+
run: |
23+
python3 -m pip install -e .
24+
shell: micromamba-shell {0}
25+
26+
- name: Run unit tests
27+
run: |
28+
pytest -s tests/*_unit_test.py
29+
shell: micromamba-shell {0}
30+
31+
- name: Run clair3 test
32+
run: |
33+
./test-runner.sh clair3
34+
shell: micromamba-shell {0}
35+
36+
- name: Run minion_validator tests
37+
run: |
38+
pytest -s tests/minion_validator.py
39+
shell: micromamba-shell {0}
40+

‎.gitignore

Lines changed: 11 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,14 @@ dist/
2626

2727
# test data
2828
*.index*
29-
CVR1/
30-
NRW01/
31-
SP1/
29+
CVR1.*
30+
NRW01.*
31+
SP1.*
32+
MT007544*
33+
tmp/
34+
test-data/CVR1/
35+
test-data/NRW01/
36+
test-data/SP1/
37+
test-data/MT007544/
38+
rg_*
39+
primer-schemes/

‎.travis.yml

Lines changed: 0 additions & 38 deletions
This file was deleted.

‎CHANGELOG

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,3 +1,16 @@
1+
v1.4.5:
2+
* Fieldbioinformatics now supports rapid barcoded (fragmented) primer trimming and normalisation
3+
* Nanopolish has been removed completely due to several compatibility issues
4+
* Medaka has also been removed completely due to kicking out long indels in a way that cannot be changed.
5+
* Clair3 is now the default variant caller, by default only the r9.4.1 models are available but a artic_get_models command has been added which will fetch the ONT created r10.4.1 models listed in the rerio repository.
6+
* The pipeline will also attempt to pick an appropriate model based on the basecall_model_version_id field that is added to read headers by default by ONT sequencers.
7+
* Removed longshot entirely, it also kicks out long variants and is now unnecessary due to clair3 being a much better variant caller.
8+
* Primer scheme fetcher has been updated to pull from the quick-lab primal hub schemes repository. For schemes not available in this repository you may provide them directly with the arguments --bed and --ref.
9+
* Automated docker builds pushing to quay.io for use in nextflow pipelines etc.
10+
* Remove some old functionality which is no longer relevant (basecalling, gather, etc)
11+
* Re-implement CI as a gh action.
12+
* Fix the overlapping variants issue by normalising variants against the pre-consensus using bcftools norm.
13+
114
v1.1.0-rc1:
215
* Support for read groups:
316
* Support ‘pool’ read groups taken from BED file, e.g.:

‎Dockerfile

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
FROM mambaorg/micromamba:1.5.8
2+
3+
COPY . ./fieldbioinformatics/
4+
5+
USER root
6+
7+
RUN apt-get update && apt-get install -y --no-install-recommends build-essential wget procps
8+
9+
USER $MAMBA_USER
10+
11+
COPY --chown=$MAMBA_USER:$MAMBA_USER environment.yml /tmp/env.yml
12+
13+
RUN sed -i 's/name: artic/name: base/' /tmp/env.yml
14+
15+
RUN micromamba install --yes --file /tmp/env.yml && \
16+
micromamba clean --all --yes
17+
18+
ARG MAMBA_DOCKERFILE_ACTIVATE=1
19+
20+
USER root
21+
22+
RUN python3 -m pip install ./fieldbioinformatics
23+
24+
RUN artic_get_models
25+
26+
USER $MAMBA_USER
27+
28+
ENTRYPOINT ["/usr/local/bin/_entrypoint.sh"]
29+
30+
CMD ["/bin/bash"]

‎README.md

Lines changed: 9 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -25,36 +25,34 @@ Features include:
2525
- variant calling
2626
- consensus building
2727

28-
There are **2 workflows** baked into this pipeline, one which uses signal data (via [nanopolish](https://github.com/jts/nanopolish)) and one that does not (via [medaka](https://github.com/nanoporetech/medaka)).
29-
30-
## Installation
28+
<!-- ## Installation
3129
3230
### Via conda
3331
3432
```sh
3533
conda install -c bioconda -c conda-forge artic
3634
```
3735
38-
If conda reports that nothing provides particular packages when running the above command ensure your `channel_priority` is set to `flexible` using the following command:
39-
40-
```sh
41-
conda config --set channel_priority false
42-
```
36+
Please make sure you are using either mamba or conda version >= 23.10.0 where libmamba solver was made the default conda solver. -->
4337

4438
### Via source
4539

4640
#### 1. downloading the source:
4741

48-
Download a [release](https://github.com/artic-network/fieldbioinformatics/releases) or use the latest master (which tracks the current release):
42+
<!-- Download a [release](https://github.com/artic-network/fieldbioinformatics/releases) or use the latest master (which tracks the current release): -->
43+
44+
Clone the repository then checkout the 1.4.0-dev branch to test the 1.4.0 pre-release.
4945

5046
```sh
5147
git clone https://github.com/artic-network/fieldbioinformatics
5248
cd fieldbioinformatics
49+
git checkout 1.4.0-dev
5350
```
5451

5552
#### 2. installing dependencies:
5653

57-
The `artic pipeline` has several [software dependencies](https://github.com/artic-network/fieldbioinformatics/blob/master/environment.yml). You can solve these dependencies using the minimal conda environment we have provided:
54+
The `artic pipeline` has several [software dependencies](https://github.com/artic-network/fieldbioinformatics/blob/master/environment.yml). You can solve these dependencies using the minimal conda environment we have provided, Please make sure you are using either mamba or conda version >= 23.10.0 where libmamba solver was made the default conda solver.
55+
:
5856

5957
```sh
6058
conda env create -f environment.yml
@@ -64,7 +62,7 @@ conda activate artic
6462
#### 3. installing the pipeline:
6563

6664
```sh
67-
python setup.py install
65+
pip install .
6866
```
6967

7068
#### 4. testing the pipeline:

‎artic/.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
# pixi environments
2+
.pixi
3+
*.egg-info

0 commit comments

Comments
 (0)