mag
nf-core/mag
Assembly, binning and annotation of metagenomes
Key tools
Run it
nfclaw run mag --input samplesheet.csv --outdir results -profile dockernextflow run pipelines/mag/upstream -profile docker --input samplesheet.csv --outdir resultsnfclaw run mag --demo --outdir results # adds the upstream test profile (-profile test,docker)Inputs
| column | type | required | allowed values | constraints |
|---|---|---|---|---|
sample | string | yes | matches ^\S+$ | |
run | string or integer | no | matches ^\S+$ | |
group | string or integer | yes | matches ^\S+$ | |
short_reads_1 | string (file path) | no | matches ^\S+\.f(ast)?q\.gz$ | |
short_reads_2 | string (file path) | no | matches ^\S+\.f(ast)?q\.gz$ | |
short_reads_platform | string | no | ILLUMINA, BGISEQ, LS454, ION_TORRENT, DNBSEQ, ELEMENT, ULTIMA, VELA_DIAGNOSTICS, GENAPSYS, GENEMIND, TAPESTRI | |
long_reads | string (file path) | no | matches ^\S+\.f(ast)?q\.gz$ | |
long_reads_platform | string | no | OXFORD_NANOPORE, OXFORD_NANOPORE_HQ, PACBIO_CLR, PACBIO_HIFI |
--input must match ^\S+\.csv$.
Additional row validation rules from the schema:
- When
short_reads_2is set, also provideshort_reads_1. - When
short_reads_1is set, also provideshort_reads_platform. - When
long_readsis set, also providelong_reads_platform.
The samplesheet is a CSV with this header (the columns the schema requires); fill each value per the table above and reference.md (no example value is invented here):
sample,groupAny of the optional columns above may be appended to the header when your data needs them: run, short_reads_1, short_reads_2, short_reads_platform, long_reads, long_reads_platform.
From data files. --input may also be the data files — a directory (searched recursively), a quoted glob, or files (--input once per file) — from which nfclaw writes this samplesheet once you choose the read columns: the samplesheet takes reads in several column groups (short_reads_1 + short_reads_2; long_reads) and file names cannot tell which yours belong in; leave the groups you do not use empty, e.g. --sheet-set long_reads=. Set the required columns no file determines with --sheet-set COLUMN=VALUE. Preview it with --check; a run records it in <outdir>/provenance/ for replay.
Resources
Resource requests follow the pipeline's configs and selected profiles. nf-core's resource labels often assume a server — a single step can request far more memory than a workstation has, and retry rules can increase requests. If a run fails with Process requirement exceeds available memory (or CPUs), cap every request, and every retry, at what this machine actually has:
nfclaw run mag --input samplesheet.csv --outdir results -profile docker \ --limit-cpus 4 --limit-memory 15.GBnfclaw turns those into Nextflow's process.resourceLimits and passes them as a -c config — the mechanism nf-core prescribes for exactly this (docs). Set them to the machine's real capacity. --limit-time DURATION also caps how long any one task may run: a task that needs longer is stopped, and so is every retry, so set it only as a deliberate ceiling, never as a guess at how long the analysis takes. The generated config is kept in <outdir>/provenance/, so commands.sh replays the run under the same ceiling.
Nextflow engine
This release declares nextflowVersion = '!>=26.04.0'.
To pin this release's declared minimum engine — useful when a newer Nextflow has incompatible config parsing:
nfclaw run mag ... --nxf-ver 26.04.0--nxf-ver is recorded in <outdir>/provenance/, so the replay uses the same engine. See known-issues.
Demo
nfclaw run mag --demo --outdir results # adds the upstream test profile (-profile test,docker)Samplesheet
A CSV with this header (the columns the schema requires). Fill each value per the constraints below; optional columns from the table may be appended.
sample,group| Column | Type | Required | Allowed / constraints |
|---|---|---|---|
sample | string | required | matches ^\S+$ |
run | string or integer | optional | matches ^\S+$ |
group | string or integer | required | matches ^\S+$ |
short_reads_1 | string (file path) | optional | matches ^\S+\.f(ast)?q\.gz$ |
short_reads_2 | string (file path) | optional | matches ^\S+\.f(ast)?q\.gz$ |
short_reads_platform | string | optional | ILLUMINABGISEQLS454ION_TORRENTDNBSEQELEMENTULTIMAVELA_DIAGNOSTICSGENAPSYSGENEMINDTAPESTRI |
long_reads | string (file path) | optional | matches ^\S+\.f(ast)?q\.gz$ |
long_reads_platform | string | optional | OXFORD_NANOPOREOXFORD_NANOPORE_HQPACBIO_CLRPACBIO_HIFI |
Required parameters
--inputstring (file path)CSV samplesheet file containing information about the samples in the experiment.
Constraint · matches ^\S+\.csv$
--outdirstring (directory path)The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.
Parameter explorer
179 parameters · 14 groupsEvery parameter from the pinned nextflow_schema.json. nfclaw checks explicit values before launch; complex rules use the pinned pipeline's runtime validation. Search, filter by group, and expand any parameter for its default, allowed values and constraints.
157 parameters · 22 hidden
Ancient Dna Assembly9
Turn on/off the ancient DNA subworkflow
minimum genotype quality for considering a variant high quality
minimum genotype quality for considering a variant medium quality
minimum number of bases supporting the alternative allele
minimum base quality required for variant calling
minimum minor allele frequency for considering variants
Ploidy for variant calling
PyDamage accuracy threshold
deactivate damage correction of ancient contigs using variant and consensus calling
Assembly options13
Enable ALE per-base output. This output can be very large (tens of GB).
Co-assemble samples within one group, instead of assembling each sample separately.
Additional custom options for MEGAHIT.
Run PyPOLCA polishing on long-read assemblies before binning.
Skip ALE
Skip Flye assembly.
Skip MEGAHIT assembly.
Skip MetaDBG assembly.
Skip metaQUAST.
Skip Illumina-only SPAdes assembly.
Skip SPAdes hybrid assembly.
Specify whether to use contigs or scaffolds assembled by SPAdes
Additional custom options for SPAdes and SPAdesHybrid. Do not specify --meta as this will be added for you!
Bin Quality Check options22
Enable clean-up of temporary files created during BUSCO runs.
Download URL, local tar.gz archive, or local uncompressed directory for an *_odb10 or *_odb12 BUSCO lineage dataset.
Name of the BUSCO *_odb10 or *_odb12 lineage to check against. Additionally supports 'auto', 'auto_prok' and 'auto_euk' for automatic lineage selection mode.
Path to local file of an already downloaded and uncompressed CheckM2 database file (.dmnd file).
CheckM2 database version number to download (Zenodo record ID, for reference check the canonical reference zenodo.org/records/5571251, and pick the Zenodo ID of the database version of your choice).
Path to local folder containing already downloaded and uncompressed CheckM database.
Make a BIgMAG input file including GUNC results.
Specify which database to auto-download if not supplying own
Specify a path to a pre-downloaded GUNC dmnd database file
Save the used GUNC reference files downloaded when not using --gunc_db parameter.
Specify which binning output is sent for downstream annotation, taxonomic classification, bin quality control etc.
Turn on bin refinement using DAS Tool.
Specify to save contig to bin maps used for bin refinement
Specify single-copy gene score threshold for bin refinement.
Enable running BUSCO during bin QC.
Enable running CheckM during bin QC.
Enable running CheckM2 during bin QC.
Turn on GUNC genome chimerism checks
Save the used BUSCO lineage datasets provided via --busco_db.
Save the used CheckM2 reference files downloaded when not using --checkm2_db parameter.
Save the used CheckM reference files downloaded when not using --checkm_db parameter.
Disable bin QC with BUSCO, CheckM or CheckM2.
Binning options27
Specify length of sub-contigs cut up prior CONCOCT binning
Specify to not append the last contig less than sub-contig length to the last correct length contig
Specify the overlap between each sub-contig prior CONCOCT binning
Enable domain-level (prokaryote or eukaryote) classification of bins using Tiara. Processes which are domain-specific will then only receive bins matching the domain requirement.
Specify the longest length a bin should be to retain for downstream processing (in base pairs). By default no limit.
Dataset scale for MetaBinner
Specify the shortest length a bin should be to retain for downstream processing (in base pairs)
Limit the number of concurrent SEQKIT_STATS jobs used for bin size calculation.
Defines mapping strategy to compute co-abundances for binning, i.e. which samples will be mapped against the assembly.
Specify alternative Bowtie2 settings for aligning reads back against the assembly.
Exclude unbinned contigs in the post-binning steps (bin QC, taxonomic classification, and annotation steps).
Specify a minimum percent identity filter for long reads mapping back to assembled contigs.
Maximal number of contigs that are not part of any bin but treated as individual genome.
Minimum contig size to be considered for binning and for bin quality check.
Minimal length of contigs that are not part of any bin but treated as individual genome.
Save the output of mapping raw reads back to assembled contigs
Pre-trained model for SemiBin2 for single sample assemblies
RNG seed for SemiBin2.
Specify a minimum percent identity filter for short reads mapping back to assembled contigs.
Skip metagenome binning entirely
Skip COMEBin Binning
Skip CONCOCT Binning
Skip MaxBin2 Binning
Skip MetaBAT2 Binning
Skip MetaBinner Binning
Skip SemiBin2 Binning
Minimum contig length for Tiara to use for domain classification. For accurate classification, should be longer than 3000 bp.
Gene Prediction And Annotation options9
Path to either a local fasta file of protein sequences, or to a directory containing an MMseqs2-formatted database, for annotation of eukaryotic genomes.
A string containing the name of one of the databases listed in the mmseqs2 documentation. This database will be downloaded and formatted for eukaryotic genome annotation. Incompatible with --metaeuk_db.
Specify sequencing centre name required for Prokka's compliance mode.
Specify to skip CDS/product searching in Prokka runs
Turn on Prokka complicance mode for truncating contig names for NCBI/ENA compatibility.
Save the downloaded mmseqs2 database specified in --metaeuk_mmseqs_db.
[DEPRECATED] This parameter no longer has any effect and will be removed in a future release. MetaEuk only runs when --metaeuk_db or --metaeuk_mmseqs_db is supplied.
Skip Prodigal gene prediction
Skip Prokka genome annotation.
Generic options4
Display the help message.
Display the full detailed help message.
Custom MultiQC yaml file containing HTML including a methods description.
Display hidden parameters in the help message (only works when --help or --help_full are provided).
Input Output options6
Additional input CSV samplesheet containing information about pre-computed assemblies. When set, assembly is skipped and the supplied assemblies are used for downstream analysis.
Email address for completion summary.
CSV samplesheet file containing information about the samples in the experiment.
MultiQC report title. Printed as page header, used for filename if not otherwise specified.
The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.
Specifies that the input is single-end reads.
Quality Control For Long Reads options15
Filter long reads against short reads when using filtlong.
Keep reads similar to the ONT internal standard Escherichia virus Lambda genome.
Genome reference used to remove ONT Lambda contaminant reads.
Specify which long read adapter trimming tool to use.
Specify which long read filtering tool to use.
Keep this percent of bases. Only used by filtlong.
The higher the more important is read length when choosing the best reads. Only used by filtlong.
Discard any read which is shorter than this value.
Discard any read which has a mean quality score lower than this value.
Specify to save the resulting length filtered long read FASTQ files to --outdir.
Specify to save input FASTQ files with lamba reads removed to --outdir.
Specify to save the resulting clipped FASTQ files to --outdir.
Skip removing adapter sequences from long reads.
Skip filtering long reads.
Skip all default QC steps for long reads (adapter trimming, filtering, removal of lambda sequences).
Quality Control For Short Reads options27
Forward read adapter to be trimmed by AdapterRemoval.
Reverse read adapter to be trimmed by AdapterRemoval for paired end data.
The minimum base quality for low-quality base trimming by AdapterRemoval.
Turn on quality trimming by consecutive stretch of low quality bases, rather than by window.
Run BBnorm to normalize sequence depth.
Set BBnorm minimum depth to this number.
Set BBnorm target maximum depth to this number.
Specify which adapter clipping tool to use.
The mean quality requirement used for per read sliding window cutting by fastp.
Minimum phred quality value of a base to be qualified in fastp.
Save reads that fail fastp filtering in a separate file. Not used downstream.
Turn on detecting and trimming of poly-G tails
Fasta reference file for host contamination removal.
Bowtie2 index directory corresponding to --host_fasta reference file for host contamination removal.
Name of iGenomes reference for host contamination removal.
Save the read IDs of removed host reads.
Use the --very-sensitive instead of the--sensitivesetting for Bowtie 2 to map reads against the host genome.
Keep reads similar to the Illumina internal standard PhiX genome.
Genome reference used to remove Illumina PhiX contaminant reads.
The minimum length of reads must have to be retained for downstream analysis.
Save normalized read files to output directory.
Specify to save the resulting clipped FASTQ files to --outdir.
Specify to save input FASTQ files with host reads removed to --outdir.
Specify to save input FASTQ files with phiX reads removed to --outdir.
Skip read preprocessing using fastp or adapterremoval.
Skip running FastQC on short reads both before and after QC.
Skip all default QC steps for short reads (adapter trimming, phiX removal).
Reproducibility options4
Fix number of CPUs for MEGAHIT to 1. Not increased with retries.
RNG seed for MetaBAT2.
Fix number of CPUs used by SPAdes. Not increased with retries.
Fix number of CPUs used by SPAdes hybrid. Not increased with retries.
Taxonomic Profiling options17
Allow unofficial lineages in CAT classification.
Classify unbinned contigs with CAT (contig mode).
Database for taxonomic classification of metagenome assembled genomes. Can be either a zipped file or a directory containing the extracted output of such.
Generate CAT database.
Specify to turn off CAT marking in output files most probable hits (when multiple) with an asterix.
Specify the location of a GTDBTK database. Can be either an uncompressed directory or a .tar.gz archive. If not specified will be downloaded for you when GTDBTK or binning QC is not skipped.
Max. bin contamination (in %) allowed to apply GTDB-tk classification.
Min. alignment fraction to consider closest genome.
Min. bin completeness (in %) required to apply GTDB-tk classification.
Min. fraction of AA (in %) in the MSA for bins to be kept.
Specify to disable fast classification of genomes by ANI using skani in GTDB-Tk.
Number of CPUs used for the by GTDB-Tk run tool pplacer.
Speed up pplacer step of GTDB-Tk by loading to memory.
Run GTDB-Tk classification for all bins in a single job, rather than one job per sample/assembler/binner group.
Specify to have GTDBTk to use the full bacterial tree rather than the split tree (requires more memory!)
Save the CAT database generated when specified by --cat_db_generate.
Skip the running of GTDB, as well as the automatic download of the database
Virus Identification options4
Database for virus classification with geNomad
Minimum geNomad score for a sequence to be considered viral
Number of groups that geNomad's MMSeqs2 databse should be split into (reduced memory requirements)
Run virus identification.
Outputs
Results land in --outdir; the files and directory layout depend on the selected workflow, parameters and publication settings. Run metadata is normally placed in <outdir>/pipeline_info/ (execution report, software versions), when those outputs are enabled. The pinned tree includes MultiQC support; a report is produced only when its workflow step runs. nfclaw run also writes <outdir>/provenance/ with the exact params file and the run log, <outdir>/provenance/logs/run.log — the whole launch, whose last line states the outcome (Nextflow's own log is <outdir>/.nextflow.log); unless --no-provenance it adds a run manifest (pinned version, commit and exact command), input/output SHA-256 checksums, and a replayable commands.sh.
The exact output files and directory layout for this release are documented upstream: github.com/nf-core/mag/blob/5.5.0/docs/output.md
Check a run — in the foreground or the background — with nfclaw status <outdir>: success, still running, how it ended (with the error), or stopped without an outcome; exit 0 success, 3 running, 1 otherwise.
Chaining
Run mag as one stage of a chain: nfclaw chain run starts each stage only after the one before it succeeded, and prepares its inputs from that stage's outputs. Every rule is checked against both pinned schemas.
Fed by
- detaxizer — detaxizer, with --generate-downstream-samplesheets, writes downstream_samplesheets/mag-pe.csv for its filtered paired-end reads, in the column layout of mag 3.x, and mag-se.csv for samples without a second short-read file, which mag cannot take in the same run: a run with any fails this handoff instead of leaving them out; each sample gets its own group and its short reads the ILLUMINA platform detaxizer documents for them. Different samples are not pooled for binning or assembly; use an inline handoff to supply study-specific pooling groups. detaxizer records no long-read platform, and mag chooses its long-read preprocessing and assembler mode by it: samples with long reads are refused until an inline handoff sets long_reads_platform (OXFORD_NANOPORE, OXFORD_NANOPORE_HQ, PACBIO_CLR or PACBIO_HIFI) for the data.
- fetchngs — fetchngs writes a mag samplesheet whose read columns predate mag's short_reads_1/short_reads_2; the columns are renamed, each sample gets its own group, and each run keeps its own run id. fetchngs labels every run's reads ILLUMINA short reads; the short-read platform is taken from the run's ENA instrument_platform instead, so a long-read run (OXFORD_NANOPORE, PACBIO_SMRT) is refused rather than analysed as short reads. Different samples are not pooled for binning or assembly; use an inline handoff to supply study-specific pooling groups.
nfclaw chain edges mag