pathogensurveillance
nf-core/pathogensurveillance
Surveillance of pathogens using high-throughput sequencing
Key tools
Run it
nfclaw run pathogensurveillance --input samplesheet.csv --outdir results -profile dockernextflow run pipelines/pathogensurveillance/upstream -profile docker --input samplesheet.csv --outdir resultsnfclaw run pathogensurveillance --demo --outdir results # adds the upstream test profile (-profile test,docker)Inputs
| column | type | required | allowed values | constraints |
|---|---|---|---|---|
sample_id | string | no | ||
name | string | no | ||
description | string | no | ||
path | string | no | matches ^([\S\s]*\/)?[^\s\/]+\.f(ast)?q\.gz$ | |
path_2 | string | no | matches ^([\S\s]*\/)?[^\s\/]+\.f(ast)?q\.gz$ | |
ncbi_accession | string | no | matches ^[A-Z]{3,6}[0-9]+$ | |
ncbi_query | string | no | ||
ncbi_query_max | integer | no | ≥ 1 | |
sequence_type | string | no | matches (?i)illumina|nanopore|pacbio|bgiseq | |
report_group_ids | string | no | matches ^[a-zA-Z0-9_ -]+(?:;[a-zA-Z0-9_ -]+)*$ | |
color_by | string | no | ||
ploidy | integer | no | ≥ 1 | |
enabled | boolean | no | ||
ref_group_ids | string | no |
--input must match ^\S+\.[ct]sv$.
The samplesheet is a CSV with this header; fill each value per the table above and reference.md (no example value is invented here):
sample_id,name,description,path,path_2,ncbi_accession,ncbi_query,ncbi_query_max,sequence_type,report_group_ids,color_by,ploidy,enabled,ref_group_idsFrom data files. --input may also be the data files — a directory (searched recursively), a quoted glob, or files (--input once per file) — from which nfclaw writes this samplesheet: sample_id from file names; path/path_2 from FastQ files (mates paired by _R1/_R2, or by _1/_2 when their read names show they are mates). Optional columns left out take their schema defaults in the pipeline: ncbi_query_max = 10, report_group_ids = _no_group_defined_, ploidy = 1, enabled = true — set them with --sheet-set where your design differs. Set other columns with --sheet-set COLUMN=VALUE. Preview it with --check; a run records it in <outdir>/provenance/ for replay.
Secrets
This release reads Nextflow secrets: NCBI_API_KEY (in conf/modules.config, modules/local/initial_classification/main.nf and 1 more files). Set each one a run needs with nextflow secrets set NAME VALUE before launching; a secret that is not set reaches its tasks as an empty value (Nextflow warns that the variable evaluates to an empty value), and a task that needs it then fails. Secrets are not written to nfclaw's provenance.
Resources
Resource requests follow the pipeline's configs and selected profiles. nf-core's resource labels often assume a server — a single step can request far more memory than a workstation has, and retry rules can increase requests. If a run fails with Process requirement exceeds available memory (or CPUs), cap every request, and every retry, at what this machine actually has:
nfclaw run pathogensurveillance --input samplesheet.csv --outdir results -profile docker \ --limit-cpus 4 --limit-memory 15.GBnfclaw turns those into Nextflow's process.resourceLimits and passes them as a -c config — the mechanism nf-core prescribes for exactly this (docs). Set them to the machine's real capacity. --limit-time DURATION also caps how long any one task may run: a task that needs longer is stopped, and so is every retry, so set it only as a deliberate ceiling, never as a guess at how long the analysis takes. The generated config is kept in <outdir>/provenance/, so commands.sh replays the run under the same ceiling.
Nextflow engine
This release declares nextflowVersion = '!>=25.04.6'.
To pin this release's declared minimum engine — useful when a newer Nextflow has incompatible config parsing:
nfclaw run pathogensurveillance ... --nxf-ver 25.04.6--nxf-ver is recorded in <outdir>/provenance/, so the replay uses the same engine. See known-issues.
Demo
nfclaw run pathogensurveillance --demo --outdir results # adds the upstream test profile (-profile test,docker)Samplesheet
A CSV with this header (the columns the schema requires). Fill each value per the constraints below.
sample_id,name,description,path,path_2,ncbi_accession,ncbi_query,ncbi_query_max,sequence_type,report_group_ids,color_by,ploidy,enabled,ref_group_ids| Column | Type | Required | Allowed / constraints |
|---|---|---|---|
sample_id | string | optional | — |
name | string | optional | — |
description | string | optional | — |
path | string | optional | matches ^([\S\s]*\/)?[^\s\/]+\.f(ast)?q\.gz$ |
path_2 | string | optional | matches ^([\S\s]*\/)?[^\s\/]+\.f(ast)?q\.gz$ |
ncbi_accession | string | optional | matches ^[A-Z]{3,6}[0-9]+$ |
ncbi_query | string | optional | — |
ncbi_query_max | integer | optional | ≥ 1 |
sequence_type | string | optional | matches (?i)illumina|nanopore|pacbio|bgiseq |
report_group_ids | string | optional | matches ^[a-zA-Z0-9_ -]+(?:;[a-zA-Z0-9_ -]+)*$ |
color_by | string | optional | — |
ploidy | integer | optional | ≥ 1 |
enabled | boolean | optional | — |
ref_group_ids | string | optional | — |
Required parameters
--inputstring (file path)Path to comma/tab-separated file containing information about samples.
Constraint · matches ^\S+\.[ct]sv$
--outdirstring (directory path)The output directory where the results will be saved. You have to use absolute paths to storage if running on Cloud infrastructure.
Parameter explorer
53 parameters · 4 groupsEvery parameter from the pinned nextflow_schema.json. nfclaw checks explicit values before launch; complex rules use the pinned pipeline's runtime validation. Search, filter by group, and expand any parameter for its default, allowed values and constraints.
34 parameters · 19 hidden
Analysis Parameters19
When selecting references automatically, allow assemblies that NCBI considers atypical.
When selecting references automatically, allow assemblies not in RefSeq.
When selecting references automatically, allow assemblies that do not have annotations. The pipeline will attempt to annotate these references as needed.
A scaling factor to adjust the number of CPUs used by multithreaded processes. The number of CPUs for each process will be multiplied by this value and rounded to the nearest integer (minimum 1).
Maximum depth of reads to be used for all analyses. Samples with more reads are subsampled to this depth.
Maximum number of samples to analyze. The first n samples will be used.
The maximum number of variants to use for the SNP tree and minimum spanning nextwork produced by the variant analysis. The first N variants are used when more than N are available.
The minimum number of bases after quality filtering needed to attempt an genome assembly. Samples with few than this amount of bases will be excluded from further analysis.
The number of references most similar to each sample based on estimated ANI to include in phyogenetic anlyses.
Same as the 'n_ref_closest' option except that it only applies to referneces with what apppear to be standard latin binomaial names (i.e. two words with no numbers or symbols). This is intended to ensure that a refernece with an informative name is present even if it is not the most similar.
The number of references representing the entire range of ANI relative to each sample. These are meant to provide context for more similar references. For a group of samples, the fewest total references will be selected that satisify this count for each sample.
The maximum number/percentage of references representing unique genera to download from RefSeq for each family predicited to be in each sample. Samples with similar initial indentifications will usually use the same references, so the total number of references downloaded for a goup of samples will depend on the taxonomic diversity of the samples.
The maximum number/percentage of references representing unique species to download from RefSeq for each genus predicited to be in each sample. Samples with similar initial indentifications will usually use the same references, so the total number of references downloaded for a goup of samples will depend on the taxonomic diversity of the samples.
The maximum number/percentage of references representing unique subspecies to download from RefSeq for each species predicited to be in each sample. Samples with similar initial indentifications will usually use the same references, so the total number of references downloaded for a goup of samples will depend on the taxonomic diversity of the samples.
When selecting references automatically, only consider references with names that appear to be standard latin bionomials (i.e. no numbers or symbols in the first two words).
The maximum number of genes used to conduct a core gene phylogeny.
The minimum number of genes needed to conduct a core gene phylogeny. Samples and references will be removed (as allowed by the min_core_samps and min_core_refs options) until this minimum is met.
The minimum ANI between a sample and potential reference for that reference to be used for mapping reads from that sample. To force all the samples in a report group to use the same reference, set this value very low.
Skip the core genome phylogeny analysis. This will not affect downstream processes like MultiQC and the main report.
Generic options4
Display the help message.
Display the full detailed help message.
Custom MultiQC yaml file containing HTML including a methods description.
Display hidden parameters in the help message (only works when --help or --help_full are provided).
Input Output options11
The path to the Bakta database folder. This or --download_bakta_db must be included.
Which type of the Bakta database to download. Must be 'light' (~2Gb) or 'full' (~40Gb).
The location to save downloaded files for later use. This is seperate from the cached data (usually stored in the 'work' directory), so that the cache can be cleared without having to repeat many large downloads.
Download the database required for running Bakta. This or --bakta_db must be included. For more information, see: github.com/oschwengers/bakta?tab=readme-ov-file#database-download
Email address for completion summary.
Path to comma/tab-separated file containing information about samples.
Maximum number of parallel downloads allowed for certain processes. This can be set to control the number of concurrent downloads to avoid overloading servers or hitting API rate limits.
MultiQC report title. Printed as page header, used for filename if not otherwise specified.
The output directory where the results will be saved. You have to use absolute paths to storage if running on Cloud infrastructure.
Path to comma-separated file containing information about references.
The location to save temporary files for processes. This is only used for some processes that produce large temporary files such as PICARD_SORTSAM.
Outputs
Results land in --outdir; the files and directory layout depend on the selected workflow, parameters and publication settings. Run metadata is normally placed in <outdir>/pipeline_info/ (execution report, software versions), when those outputs are enabled. The pinned tree includes MultiQC support; a report is produced only when its workflow step runs. nfclaw run also writes <outdir>/provenance/ with the exact params file and the run log, <outdir>/provenance/logs/run.log — the whole launch, whose last line states the outcome (Nextflow's own log is <outdir>/.nextflow.log); unless --no-provenance it adds a run manifest (pinned version, commit and exact command), input/output SHA-256 checksums, and a replayable commands.sh.
The exact output files and directory layout for this release are documented upstream: github.com/nf-core/pathogensurveillance/blob/1.1.0/docs/output.md
Check a run — in the foreground or the background — with nfclaw status <outdir>: success, still running, how it ended (with the error), or stopped without an outcome; exit 0 success, 3 running, 1 otherwise.