variantbenchmarking
nf-core/variantbenchmarking
Variant Benchmarking pipeline for germline and somatic variant callers
Key tools
Run it
nfclaw run variantbenchmarking --input samplesheet.csv --outdir results --analysis <analysis> --variant-type <variant_type> --method <method> --fasta <fasta> --fai <fai> -profile dockernextflow run pipelines/variantbenchmarking/upstream -profile docker --input samplesheet.csv --outdir results --analysis <analysis> --variant_type <variant_type> --method <method> --fasta <fasta> --fai <fai>nfclaw run variantbenchmarking --demo --outdir results # adds the upstream test profile (-profile test,docker)Inputs
| column | type | required | allowed values | constraints |
|---|---|---|---|---|
test_vcf | string (file path) | no | matches \S+\.(vcf|bcf)(\.gz)?$ | |
test_regions | string (file path) | no | matches ^\S+\.(vcf|bed|txt|cnn|csv|cns|gz_CNVs)(\.gz)?$ | |
id | string | yes | matches ^\S+$; length ≥ 1 | |
caller | string | yes | matches ^\S+$; length ≥ 1 | |
subsample | string | no | matches ^\S+$; length ≥ 1 | |
normshift | number | no | ≥ 0; ≤ 1 | |
normdist | number | no | ≥ 0; ≤ 1 | |
normsizediff | number | no | ≥ 0; ≤ 1 | |
maxdist | integer | no | ≥ 0 | |
pctsize | number | no | ≥ 0; ≤ 1 | |
pctseq | number | no | ≥ 0; ≤ 1 | |
pctovl | number | no | ≥ 0; ≤ 1 | |
refdist | integer | no | ≥ 0 | |
chunksize | integer | no | ≥ 0 | |
dup_to_ins | boolean | no | ||
typeignore | boolean | no | ||
bpDistance | integer | no | ≥ 0 | |
percentThreshold | number | no | ≥ 0 | |
absoluteThreshold | integer | no | ≥ 0 | |
maxMatches | integer | no | ||
evaluationmode | string | no | sc, cts, d | length ≥ 1 |
liftover | boolean | no | ||
fix_prefix | boolean | no | ||
missing_genotypes | boolean | no | ||
concordance_truth_id | string | no | matches ^[a-zA-Z0-9_\-]+(?:;[a-zA-Z0-9_\-]+)*$ |
--input must match ^\S+\.(csv|tsv|yaml|yml|json)$.
Additional row validation rules from the schema:
- At least one of these conditional requirements must be satisfied:
test_vcfwhenidis set;test_regionswhenidis set.
For tabular CSV/TSV input, use this header (the columns the schema requires); fill each value per the table above and reference.md (no example value is invented here):
id,callerAny of the optional columns above may be appended to the header when your data needs them: test_vcf, test_regions, subsample, normshift, normdist, normsizediff, maxdist, pctsize, pctseq, pctovl, refdist, chunksize, dup_to_ins, typeignore, bpDistance, percentThreshold, absoluteThreshold, maxMatches, evaluationmode, liftover, fix_prefix, missing_genotypes, concordance_truth_id.
From data files. --input may also be the data files — a directory (searched recursively), a quoted glob, or files (--input once per file) — from which nfclaw writes this samplesheet: id from file names; test_vcf, test_regions from files matching their patterns, by sample name. Optional columns left out take their schema defaults in the pipeline: liftover = false, fix_prefix = false, missing_genotypes = false — set them with --sheet-set where your design differs. Set the required columns no file determines with --sheet-set COLUMN=VALUE (a constant or a {sample}/{lane} template): caller (matching ^\S+$). Preview it with --check; a run records it in <outdir>/provenance/ for replay.
Reference genome
No reference genome is set by default: supply your own reference. The schema declares AWS iGenomes at s3://ngi-igenomes/igenomes/; selecting --genome <id> uses the paths in the pipeline's reference configuration. Access to that remote source is required. To supply your own reference, use e.g. --fasta; the reference_genome_options group in reference.md lists every reference option. Set --igenomes-ignore true to disable the lookup entirely. Confirm the selected reference's paths and source in the pinned upstream/nextflow.config and upstream/conf/ before running.
Resources
Resource requests follow the pipeline's configs and selected profiles. nf-core's resource labels often assume a server — a single step can request far more memory than a workstation has, and retry rules can increase requests. If a run fails with Process requirement exceeds available memory (or CPUs), cap every request, and every retry, at what this machine actually has:
nfclaw run variantbenchmarking --input samplesheet.csv --outdir results --analysis <analysis> --variant-type <variant_type> --method <method> --fasta <fasta> --fai <fai> -profile docker \ --limit-cpus 4 --limit-memory 15.GBnfclaw turns those into Nextflow's process.resourceLimits and passes them as a -c config — the mechanism nf-core prescribes for exactly this (docs). Set them to the machine's real capacity. --limit-time DURATION also caps how long any one task may run: a task that needs longer is stopped, and so is every retry, so set it only as a deliberate ceiling, never as a guess at how long the analysis takes. The generated config is kept in <outdir>/provenance/, so commands.sh replays the run under the same ceiling.
Nextflow engine
This release declares nextflowVersion = '!>=25.10.4'.
To pin this release's declared minimum engine — useful when a newer Nextflow has incompatible config parsing:
nfclaw run variantbenchmarking ... --nxf-ver 25.10.4--nxf-ver is recorded in <outdir>/provenance/, so the replay uses the same engine. See known-issues.
Demo
nfclaw run variantbenchmarking --demo --outdir results # adds the upstream test profile (-profile test,docker)Samplesheet
A CSV with this header (the columns the schema requires). Fill each value per the constraints below; optional columns from the table may be appended.
id,caller| Column | Type | Required | Allowed / constraints |
|---|---|---|---|
test_vcf | string (file path) | optional | matches \S+\.(vcf|bcf)(\.gz)?$ |
test_regions | string (file path) | optional | matches ^\S+\.(vcf|bed|txt|cnn|csv|cns|gz_CNVs)(\.gz)?$ |
id | string | required | matches ^\S+$; length ≥ 1 |
caller | string | required | matches ^\S+$; length ≥ 1 |
subsample | string | optional | matches ^\S+$; length ≥ 1 |
normshift | number | optional | ≥ 0; ≤ 1 |
normdist | number | optional | ≥ 0; ≤ 1 |
normsizediff | number | optional | ≥ 0; ≤ 1 |
maxdist | integer | optional | ≥ 0 |
pctsize | number | optional | ≥ 0; ≤ 1 |
pctseq | number | optional | ≥ 0; ≤ 1 |
pctovl | number | optional | ≥ 0; ≤ 1 |
refdist | integer | optional | ≥ 0 |
chunksize | integer | optional | ≥ 0 |
dup_to_ins | boolean | optional | — |
typeignore | boolean | optional | — |
bpDistance | integer | optional | ≥ 0 |
percentThreshold | number | optional | ≥ 0 |
absoluteThreshold | integer | optional | ≥ 0 |
maxMatches | integer | optional | — |
evaluationmode | string | optional | scctsdlength ≥ 1 |
liftover | boolean | optional | — |
fix_prefix | boolean | optional | — |
missing_genotypes | boolean | optional | — |
concordance_truth_id | string | optional | matches ^[a-zA-Z0-9_\-]+(?:;[a-zA-Z0-9_\-]+)*$ |
Required parameters
--inputstring (file path)Path to comma-separated file containing information about the samples in the experiment.
Constraint · matches ^\S+\.(csv|tsv|yaml|yml|json)$
--outdirstring (directory path)The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.
--analysisstringThe analysis type used by the input files
germlinesomatic--variant-typestringVariant types to benchmark
smallsnvindelstructuralcopynumber--methodstringThe benchmarking methods to use. For germline small variants (SNV and INDEL) use happy and/or rtgtools, for somatic small variants (SNV and INDEL) use sompy and/or rtgtools, for structural variants use wittyer, truvari and/or svanalyzer, for copy number variations use wittyer and/or truvari. Use intersect to intersect BED files. Should be a comma-separate list of one or more of the following options: truvari, svanalyzer, happy, sompy, rtgtools, wittyer, intersect
Constraint · matches ^((truvari|svanalyzer|happy|sompy|rtgtools|wittyer|intersect|bndeval|concordance|aardvark)?,?)*(?<!,)$
--fastastring (file path)Path to FASTA genome file.
Constraint · matches ^\S+\.fn?a(sta)?(\.gz)?$
--faistring (file path)Path to FAI genome file.
Constraint · matches ^\S+\.fai$
Parameter explorer
60 parameters · 4 groupsEvery parameter from the pinned nextflow_schema.json. nfclaw checks explicit values before launch; complex rules use the pinned pipeline's runtime validation. Search, filter by group, and expand any parameter for its default, allowed values and constraints.
39 parameters · 21 hidden
Generic options4
Display the help message.
Display the full detailed help message.
Custom MultiQC yaml file containing HTML including a methods description.
Display hidden parameters in the help message (only works when --help or --help_full are provided).
Input Output options26
Path to ambiguous BED. Only applicable to sompy tool.
The analysis type used by the input files
Email address for completion summary.
The pipeline filter outs missing genotypes with ./. or 0/0 by default, enable using missing genotypes for test, truth. Should be a comma-separated list of one or more of the following options: test, truth
Ensemble truth using input VCF files following the majority rule specified. This method should be used only if truth file is not available
Use bcftools expressions samtools.github.io/bcftools/bcftools.html#expressions to exclude variants
Comparison engine used by hap.py to match test variants against the truth set.
If set to true, treat regions_bed as a high confidence region rather than a restriction region.
Use bcftools expressions samtools.github.io/bcftools/bcftools.html#expressions to include variants
Path to comma-separated file containing information about the samples in the experiment.
Maximum SV size of variants to benchmark, -1 to disable
The benchmarking methods to use. For germline small variants (SNV and INDEL) use happy and/or rtgtools, for somatic small variants (SNV and INDEL) use sompy and/or rtgtools, for structural variants use wittyer, truvari and/or svanalyzer, for copy number variations use wittyer and/or truvari. Use intersect to intersect BED files. Should be a comma-separate list of one or more of the following options: truvari, svanalyzer, happy, sompy, rtgtools, wittyer, intersect
Minimum Alele Frequency of variants to benchmark, Use -1 to disable
Minimum number of read supporting variants to benchmark, Use, -1 to disable
Minimum SV size of variants to benchmark, 0 to disable
MultiQC report title. Printed as page header, used for filename if not otherwise specified.
The output directory where the results will be saved. You have to use absolute paths to storage on Cloud infrastructure.
The preprocessing steps to perform on the input files. Should be a comma-separated list of one or more of the following options: split_multiallelic, normalizate, deduplicate, filter_contigs
Path for regions of interest BED or VCF file.
Skip plots: metrics, upset, svlength
Skip reports from these tools: datavzrd, happy
The standardization methods to perform on the input files. Should be a comma-separated list of one or more of the following options: variantextractor, svync, svdecompose, svtk
Path to targets BED. Works similar to Bcftools -T.
Truth id, sample name to define truth vcf
Path to the golden set VCF files.
Variant types to benchmark
Reference Genome options9
Path to the chain file required for liftover.
The dictionary file is required for liftover process. It has to be .dict of genome file used in the workflow.
Path to FAI genome file.
Path to FASTA genome file.
Name of iGenomes reference.
Path to the ranaming chromosomes for lifting over.
The SDF file needed to run rtgtools vcfeval
Path to stratification BED files provided in a directory. This directory has to be given together with stratification_tsv, list BED files in stratification_tsv. Only applicable to happy tool.
List the stratification BED files in this file, to be used with stratification_bed
Outputs
Results land in --outdir; the files and directory layout depend on the selected workflow, parameters and publication settings. Run metadata is normally placed in <outdir>/pipeline_info/ (execution report, software versions), when those outputs are enabled. The pinned tree includes MultiQC support; a report is produced only when its workflow step runs. nfclaw run also writes <outdir>/provenance/ with the exact params file and the run log, <outdir>/provenance/logs/run.log — the whole launch, whose last line states the outcome (Nextflow's own log is <outdir>/.nextflow.log); unless --no-provenance it adds a run manifest (pinned version, commit and exact command), input/output SHA-256 checksums, and a replayable commands.sh.
The exact output files and directory layout for this release are documented upstream: github.com/nf-core/variantbenchmarking/blob/1.6.1/docs/output.md
Check a run — in the foreground or the background — with nfclaw status <outdir>: success, still running, how it ended (with the error), or stopped without an outcome; exit 0 success, 3 running, 1 otherwise.