Known issues and troubleshooting

Symptoms and their fixes. Most failures at run time are either environment problems (host, network or paths, fixed with nfclaw run flags or host settings) or upstream pipeline defects (bugs in a specific nf-core release). nf-claw never edits the pinned pipeline sources, so upstream defects, the recorded runtime corrections and their limits are documented here instead. Each entry names the version and date it was measured at; a pipeline's skill.md gives its current pin. Agents can search this page by section with read_doc(guide="known-issues", search="…").

Navigation

Reading a failure • Environment • Runs, stops and resume • Replay and verification • Inputs and samplesheets • Chains • Harmless warnings • Upstream pipeline bugs • Pipeline run notes

Reading a failure

When a run fails, nfclaw's error quotes Nextflow's own error report (with the full Caused by: chain from .nextflow.log when the console hides it) and gives the absolute paths of:

1. the run log, <outdir>/provenance/logs/run.log, whose last line is the outcome;
2. Nextflow's own log, <outdir>/.nextflow.log;
3. the failing task's .command.err, next to its .command.log and .command.sh.

[!NOTE] nfclaw run treats dashes and underscores in parameter names as the same (--skip-busco and --skip_busco are equivalent; raw Nextflow needs the pipeline's exact spelling).

A value that starts with a dash, such as a flag passed through to a tool (rnaseq's --extra_star_align_args), must be written with =, as in --extra_star_align_args='--outFilterMismatchNmax 5', or put in a --params-file. Written as two words, --extra_star_align_args '--outFilterMismatchNmax 5', it is refused with an unknown parameter error, because the value would otherwise be read as a flag.

Environment

nfclaw aborts with ModuleNotFoundError: No module named 'runner'

• Symptom: The installed nfclaw command fails immediately — before doing any work — with ModuleNotFoundError: No module named 'runner' (or 'librarian').
• Why: nfclaw is a console script that imports the repo's runner package, which relies on the pip install -e . editable install being active in the same Python the command runs under. Two things break that import:
  1. A space in the install path: Python's site module does not execute an editable install's path hook when the virtualenv/site-packages path contains a space, so runner never lands on sys.path (this repo already requires a space-free path for runs; the same applies to the install).
  2. Python interpreter mismatch: The editable install was performed with a different Python than the one in nfclaw's shebang (e.g. a --user install whose site-packages is not on that interpreter's path). This occurs before nfclaw code executes, so it cannot be caught by runtime space checks.
• Fix:
  • Use a space-free path for both the repo and the virtualenv (on macOS also avoid iCloud paths), and run pip install -e . inside the virtualenv you actually use.
  • Or skip the console script and use the no-install equivalent from the repo root: python3 -m runner <cmd> (maintenance runs the same way: make <target>, or python3 -m librarian.<module> — e.g. python3 -m librarian.write_skill --all). It needs no install — the repo root is already on sys.path — and resolves the pinned pipelines correctly.

Path contains a space — checked before the run, fails fast

• Symptom: A tool fails with a split path (e.g. cannot create /vol/draft 2/...: Permission denied, Got unexpected extra argument(s), or a module building shell commands breaks).
• Why: Many bioinformatics tools (and Nextflow's internal work directory) build shell commands without quoting paths, causing spaces to split arguments. This affects macOS and Linux alike.
• How nfclaw handles it: nfclaw run checks the repo path, the Nextflow work directory and --outdir before launching and fails fast, naming exactly which path has the space — a deterministic check, no guessing.
• Fix:
  • Move the repository to a space-free path (strongly recommended).
  • Keep the repository space-free and set a space-free work directory: --nxf-env NXF_WORK=/a/space-free/dir, and use a space-free --outdir.
  • Or, if you know your pipeline tolerates spaces, pass --allow-spaces to bypass the preflight check.

IPv6-only host — JVM can't reach GitHub

• Symptom: java.net.SocketException: Network is unreachable while Nextflow downloads https://raw.githubusercontent.com/nf-core/configs/master/nfcore_custom.config; Nextflow 26.04 prints only Unable to parse config file, and --check says the engine could not use a remote configuration.
• Why: The host has no default IPv4 route; the JVM prefers IPv4 by default and never attempts IPv6.
• Fix: --nxf-env NXF_JVM_ARGS=-Djava.net.preferIPv6Addresses=true. To skip remote config fetches entirely (offline): --nxf-env NXF_OFFLINE=true.
• Dual Requirement on IPv6 Hosts: On an IPv6-only host you usually need BOTH this and the Docker host-network config together: the JVM flag fixes Nextflow's own GitHub download, while host-network fixes DNS inside containers — neither alone is sufficient:
  bashCopy

  nfclaw run <name> --nxf-env NXF_JVM_ARGS=-Djava.net.preferIPv6Addresses=true \
                    --config host-net.config …

No network at run time — a tool downloads a database

• Symptom: A step (e.g. BUSCO) hangs then fails trying to fetch a database it requires.
• Fix: Supply the step's database locally through the pinned release's declared parameters, or deliberately skip that analysis if the release supports it. Where reference.md declares skip_busco, use --skip-busco true (or {"skip_busco": true} via --params-file params.json). Skipping a step changes the analysis; NXF_OFFLINE does not prevent tools inside containers from attempting downloads.

A samplesheet's remote file is reported as missing, but it exists

• Symptom: the run stops before any task with Validation of file failed, naming one or more https:// URLs with the file or directory '…' does not exist, or with Unable to stage foreign file … Unable to access path, while curl -IL on that URL answers 200.
• Seen: seqinspector 1.1.2 and metatdenovo 2.0.0 on 2026-11-05: one URL each on github.com, on both attempts a minute apart. drop 1.0.0 the same afternoon, on URLs of its own: the first attempt stopped on Unable to stage foreign file after Nextflow's 3 tries, and the second with does not exist on two others. Each passed when run again later. In the kept logs of all three, the URLs reported missing have no Remote redirect line, while every other URL checked has at least four and a passing run has those too, so github.com did not send its usual redirect for them. Nextflow does not log what came instead, so the answer itself is not known. Got HTTP error=404 lines in .nextflow.log are not it: metatdenovo's passing run has the same six.
• Caught once, with its answer: on 2026-11-08, demultiplex 1.8.0's UNTAR_FLOWCELL could not stage a github.com/nf-core/test-datasets/raw/… file in four chain attempts between 10:01 and 10:14 UTC, while a resume at about 10:05 staged it. In that window github.com answered this host with HTTP 504 instead of its redirect: 504 Gateway Time-out, after about 11 s, from GitHub's own front end (its x-github-request-id and x-github-edge-region: fra headers, behind its certificate). The file's raw.githubusercontent.com address, a separate service, answered 200. GitHub-hosted CI runners got HTTP 504 from GitHub downloads in the same hour, and at 10:36 githubstatus.com opened an incident, "Increase in timeouts for some unauthenticated users", resolved at 12:32; Nextflow downloads remote files unauthenticated. The 2026-11-05 windows also lacked the redirect, which fits the same cause, but their answers were not kept.
• Fix: wait and run the same command again. Nothing has been computed yet, so nothing is lost. curl -sI https://github.com/nf-core shows whether github.com is still answering 504; githubstatus.com may name the incident, though here it did so about 35 minutes after the first failure. A samplesheet of your own can name https://raw.githubusercontent.com/<org>/<repo>/<commit>/<path> instead, which does not go through github.com.

A process requests more memory than the host has — aborts before any work

• Symptom: A run aborts at scheduling time because a single step requests more RAM than the machine has, e.g. Process requirement exceeds available memory -- req: 80 GB; avail: 62.8 GB. Seen with detaxizer 1.3.0 run with --classification_bbduk: its BBMAP_BBDUK step carries nf-core's process_high label, which conf/base.config sizes at 80.GB * task.attempt — more than a typical workstation or small VM has.

• Why: An nf-core base.config sizes each process by a resource label (process_low/medium/high/high_memory) tuned for an HPC cluster; one high-memory step can exceed a small host's physical RAM, and Nextflow refuses to schedule a task it knows cannot fit.

• Fix: Put a ceiling on the whole run with --limit-cpus / --limit-memory, sized to the host machine:

  bashCopy

  nfclaw run rnaseq --input ss.csv --outdir results -profile docker \
    --limit-cpus 4 --limit-memory 15.GB

  nfclaw writes these as Nextflow's process.resourceLimits, the mechanism nf-core prescribes, so one ceiling covers every process and every retry and is replayed from <outdir>/provenance/resource_limits.config; AGENTS.md has the details, including --limit-time.

  A named override sizes only its own process, so it cannot stand in for that ceiling: you would need one for every step that might exceed the host (STAR_GENOMEGENERATE, then BBMAP_BBSPLIT, then the next). Use withName (via --config) only to tune one specific step, e.g. giving a tool less than its label implies:

  groovyCopy

  // tune-one-step.config
  process {
      withName: 'BBMAP_BBDUK' { memory = '12.GB' }
  }

  (Size any cap to the host and to what the tool actually needs — too low and the step itself fails or is OOM-killed.)

Nextflow too new for an older release

• Symptom: Unexpected input: ':', Unexpected token, Invalid include source, or import ... rejected — on Nextflow 26.04+, whose default strict parser rejects older Groovy config syntax (typed declarations, functions with params, manifest.*/validation.* accessed at parse time, import in .nf). Many older releases hit this.

• Fix: Pin an engine the release was written for: --nxf-ver 25.10.2 (must still satisfy the pipeline's declared nextflowVersion minimum). If you hit it on most pipelines, set it once for the shell: export NXF_VER=25.10.2 (nfclaw passes it through). Or keep the installed 26.04+ engine and select its legacy parser for the run: --nxf-env NXF_SYNTAX_PARSER=v1 (a chain stage's "nxf_env": {"NXF_SYNTAX_PARSER": "v1"}), which needs no other engine download and is recorded and replayed with the run. nfclaw's error for a run that fails this way names both, and nfclaw run --check (like nfclaw chain run --check) asks the run's engine to parse the configuration first, so the refusal comes before anything launches. See compatibility.md. Measured on Nextflow 26.04.3 (2026-10-31): nfclaw run <name> --check --demo without --nxf-ver refuses 25 of the 95 pins this way — atacseq, bactmap, callingcards, chipseq, circdna, crisprseq, cutandrun, denovotranscript, hgtseq, hic, hicar, marsseq, metaboigniter, metapep, methylseq, multiplesequencealign, nanoseq, nascent, pangenome, proteinfold, rangeland, readsimulator, reportho, viralintegration and viralrecon — and every one of the 95 passes on the engine its skill.md advises. The smoke workflow's summary reports the current list for each new pin. On Nextflow 26.04.3 the pinned methylseq 4.2.0 fails this way (Invalid include source: conf/aws/batch/nextflow.config, a profile file missing at its tag), as do atacseq 2.1.2 and viralrecon 3.0.0 below; nextflow config parses all three with NXF_SYNTAX_PARSER=v1 or NXF_VER=25.10.4. Confirmed-affected releases that do run with --nxf-ver 25.10.2 include hgtseq, callingcards, coproid, denovotranscript 1.2.1, chipseq 2.1.0, atacseq 2.1.2, circdna 1.1.0, detaxizer 1.4.0 (and 1.3.0) and marsseq 1.0.3, plus the earlier, non-pinned releases epitopeprediction 3.1.1, fastqrepair 1.0.0 and genomeassembler 1.1.0 (--pipeline-version). Newer releases — including the pinned epitopeprediction 3.2.0, fastqrepair 1.1.1, genomeassembler 2.0.1, fetchngs 1.13.0, sarek 3.11.0 (and 3.10.0) and scrnaseq 4.2.0 — declare !>=25.10.4, so use --nxf-ver 25.10.4 to avoid Nextflow 26 strict parser failures on them.

  Examples of what NF 26 rejects:

  • chipseq's and marsseq's def check_max(obj, type) in nextflow.config (marsseq 1.0.3 declares only !>=23.04.0, so unpinned it parses under NF 26 and fails at launch with a nextflow.cli.Launcher error — --nxf-ver 25.10.2 fixes it; note its test profile also sets genome = 'mm10', which expects prebuilt local references; its configuration maps that key to GRCm39/M27, not AWS iGenomes).
  • scrnaseq 4.1.0's Invalid include source: conf/test_multiome.config (the test_multiome profile includeConfigs a file not committed at the tag — NF 26 validates it at parse time even though the profile is unused, while the legacy parser skips it and the --demo run completes).

• Exception — a release that needs an older engine: cutandrun 3.2.2 fails before any task with Cannot invoke method optional() on null object on Nextflow 25.10.0, 25.10.2 and 25.10.4 (26.04 already fails to parse its configuration, as above), from the deprecated output syntax emit: html optional true in modules/local/for_patch/trimgalore/main.nf (sibling modules already use current emit: …, optional: true). Nextflow 25.04.8 and older accept it: its skill.md names --nxf-ver 23.04.0 (its declared minimum), with which the validation run completed, and its bundled test also completed on 25.04.8 (Linux, Docker).

• Exception — some releases need a newer engine: funcscan 4.0.0 and lsmquant 1.0.3 declare nextflowVersion = '!>=25.10.4', so Nextflow rejects --nxf-ver 25.10.2 at its version gate (Nextflow version 25.10.2 does not match workflow required version: >=25.10.4, on stderr); nfclaw's error names the fix: use --nxf-ver 25.10.4. (bactmap 1.0.0 also needs the test-data and module-option workarounds in the upstream table; with those, its test completed on 24.10.5).

• In a chain (nfclaw chain run): Pin the engine on that stage only — "nxf_ver": "25.10.4" in its spec entry — since other stages may need a newer one (fetchngs 1.13.0 requires >=25.10.4, mag 5.5.0 >=26.04.0). The chain parses every stage's config with its own engine (nextflow config) before the first stage launches, so this fails up front, not after the stages before it have run. Seen in chains on Nextflow 26.04.3: atacseq 2.1.2 (def check_max) and viralrecon 3.0.0 (Invalid include source: conf/test_full_sispa.config, a profile file missing at the pinned commit) — both parse with "nxf_ver": "25.10.4".

An engine older than 21.10.0 fails on Java 17: Unsupported class file major version 61

• Symptom: with an old engine pinned (bactmap 1.0.0 declares !>=21.04.0), Nextflow stops before reading the pipeline: Project config file is malformed -- Cause: Compile failed … caused by java.lang.IllegalArgumentException: Unsupported class file major version 61.
• Why: Java 17 support arrived in Nextflow 21.10.0. Verified on OpenJDK 17.0.20: nextflow config of a pinned pipeline fails on 21.04.0 and succeeds on 21.10.3.
• Fix: nfclaw run now refuses such a pin on Java 17+ before launching and names a later engine; the generated skill.md of a release whose declared minimum predates 21.10.0 recommends --nxf-ver 24.10.5 (the engine the acceptance gate checks those releases on) instead of that minimum. To run the old engine itself, point NXF_JAVA_HOME at a Java 11 installation.

BUSCO finishes, then the task hangs until its time limit

• Symptom: A BUSCO_BUSCO task (mag, and any pipeline running BUSCO 6) is killed at its time limit — Nextflow reports process hasn't exited — although its .command.log shows BUSCO's results and "Total running time: 6 seconds" hours earlier.
• Why: BUSCO 6 collects anonymous run statistics when it ends ("You may opt out with --opt-out-run-stats"): it looks up the host's country at ipinfo.io and posts the statistics to busco-data.ezlab.org, both without a timeout. If that HTTPS exchange cannot complete from inside the container, the task waits on it indefinitely — on the VM where this was seen, because Docker's MTU exceeded the host's (the lookup answered at once from a container on a network with the host's MTU).
• Fix: Opt out through a config given with --config (in a chain: the stage's "config"):
  groovyCopy

  process {
      withName: 'BUSCO_BUSCO' {
          ext.args = { (params.busco_db ? '--offline ' : '') + '--opt-out-run-stats' }
      }
  }

  (mag's own ext.args is --offline when --busco_db is given; the closure preserves it.) Pipelines that include the module under another name need that name: genomeqc 1.0.1 runs it as BUSCO_GENOME and BUSCO_PROTEIN, which set no ext.args, so withName: 'BUSCO_GENOME|BUSCO_PROTEIN' { ext.args = '--opt-out-run-stats' }. Fixing the container network instead (MTU) needs no per-pipeline config.

Docker bridge network has no DNS (IPv6-only host)

• Symptom: Containers cannot resolve hostnames; downloads inside a container fail even though the host has connectivity. Docker's default bridge uses IPv4 DNS 8.8.8.8, unreachable on an IPv6-only host.
• Fix: Give containers the host network via a config file passed with --config. docker.runOptions is a single string that a --config replaces (it does not merge with the pipeline's value), so preserve nf-core's default user mapping in the same string — otherwise the container reverts to running as root and outputs become root-owned:
  groovyCopy

  // host-net.config — host network + nf-core's default user mapping
  docker { runOptions = '--network host -u $(id -u):$(id -g)' }

  Run with nfclaw run <name> --config host-net.config ….
• Also when the bridge has DNS: a bridge configured with IPv6 and an IPv6 DNS server can still leave older runtimes resolving a name to its IPv4 address only. funcscan 4.0.0's DEEPARG_DOWNLOADDATA (deepARG 1.0.4, Python 2.7) resolved Zenodo to IPv4, timed out in the TLS handshake and stalled for 35 minutes, while curl on the host reached Zenodo over IPv6. Stopped and resumed with the host-network config above, the download completed in 23 minutes and the run succeeded with its cached tasks reused. Names with no IPv6 address at all, which the host reaches through DNS64/NAT64, cannot be resolved from the default bridge either: genomeqc 1.0.1's REPEATMASKER_DOWNLOADDB failed with curl: (6) Could not resolve host: www.dfam.org, and completed when resumed with the host-network config. A day later the host's own DNS64 resolver answered SERVFAIL for www.dfam.org, although its CNAME target nginx.systemsbiology.net resolved. Nextflow then could not stage rnafusion's Dfam files (No such file or directory: https://www.dfam.org/…). Files fetched with curl --resolve www.dfam.org:443:[<that target's IPv6 address>] and passed as --dfam-hmm, --dfam-h3f, --dfam-h3i, --dfam-h3m and --dfam-h3p matched the published sizes. genomeqc 1.0.1 hit the same failure later (REPEATMASKER_DOWNLOADDB, curl: (6) Could not resolve host: www.dfam.org). Its own offline option avoids the download: fetch dfam39_full.0.h5.gz and its .md5 the same way, check the MD5, gunzip it, and pass the .h5 file as --famdb-library.

Java tools in containers prefer IPv4 (IPv6-only host)

• Symptom: a Java task that contacts a server fails with java.net.ConnectException: Network is unreachable, even with the host-network config — e.g. pathogensurveillance 1.1.0's BBMAP_SENDSKETCH, which sends a sketch to JGI's server.
• Why: the JVM tries a name's IPv4 address first unless told otherwise, and a host without IPv4 egress cannot reach it. (nfclaw's NXF_JVM_ARGS=-Djava.net.preferIPv6Addresses=true advice covers Nextflow's own JVM, not the JVMs inside task containers.)
• Fix: set the preference for that process's container through --config:
  groovyCopy

  process {
      withName: 'BBMAP_SENDSKETCH' {
          containerOptions = '-e JAVA_TOOL_OPTIONS=-Djava.net.preferIPv6Addresses=true'
      }
  }

  With it, pathogensurveillance's BBMAP_SENDSKETCH, which had failed every attempt, completed on resume.

Python tools cannot download ftp:// URLs (IPv6-only host)

• Symptom: a Python task that fetches an ftp:// URL fails even with the host-network config — e.g. coproid 2.0.1's SAM2LCA_UPDATEDB: Download of taxonomy files failed. NCBI's server may be offline. (taxopy downloads ftp://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz), while curl fetches the same URL on the host.
• Why: Python's urllib FTP handler resolves the server with socket.gethostbyname, which returns IPv4 addresses only (OSError: [Errno 101] Network is unreachable). ftplib itself works over IPv6 (EPSV); only the urllib route is IPv4-bound.
• Fix: urllib honours ftp_proxy. Run a small HTTP proxy on the host that fetches ftp:// URLs with curl, and pass it to that process together with the host-network config:
  groovyCopy

  process {
      withName: 'SAM2LCA_UPDATEDB' {
          containerOptions = '-e ftp_proxy=http://127.0.0.1:38128'
      }
  }

  With such a proxy, coproid's bundled test, resumed with the host-network config and this one, completed. Avoid ports already taken by a host proxy (here tinyproxy held 3128 and answered 403 Filtered).

sra-tools cannot reach NCBI (IPv6-only host)

• Symptom: SRATOOLS_FASTERQDUMP or SRATOOLS_PREFETCH fails with Failed to call external services (exit status 64) after its retries, also with the host-network config, while curl on the same host reaches NCBI.
• Why: sra-tools 3.0.8 (ncbi-vdb) connects over IPv4 only. Traced on such a host, srapath opened only IPv4 sockets — 244 attempts to NCBI (130.14.29.113:443) and its cloud-metadata probe (169.254.169.254), each refused with ENETUNREACH — and exited 3. No configuration of the container changes this.
• Fix: fetch the runs' FastQ files from ENA over HTTPS — the ENA portal's filereport?accession=<run>&result=read_run&fields=fastq_ftp,fastq_md5 lists them and their MD5 sums — and give the pipeline local paths instead of accessions. This affects pathogensurveillance's bundled test (its sample SRR18850314 is an NCBI accession) and any pipeline that falls back to sra-tools (fetchngs' SRATOOLS_PREFETCH route).

Downloads inside task containers stall: Docker's MTU is larger than the host's

• Symptom: A step that downloads from inside its container hangs until its time limit or fails with The handshake operation timed out / Read timed out, although the same URL downloads at full speed on the host. Seen on a cloud VM (de.NBI OpenStack, link MTU 1450) for DeepARG's database from Zenodo (funcscan), sam2lca's NCBI taxonomy (coproid), and BUSCO 6's run-statistics lookup at ipinfo.io (the BUSCO hang). Downloads Nextflow stages itself (remote inputs, test data) are not affected: the JVM runs on the host.
• Why: Docker's default bridge uses MTU 1500; on a VM whose network link is smaller (overlay networks often use 1450), full-size packets to a bridged container are dropped without the "packet too big" reply that would shrink them. Connections open, but any response larger than one packet — a TLS certificate chain, a file — never arrives. Check with ip link show (host interface vs docker0).
• Fix: Match Docker's MTU to the host: "mtu": 1450 in /etc/docker/daemon.json (the bridge) or --opt com.docker.network.driver.mtu=1450 on a user-defined network, then restart Docker. Without changing the daemon, give task containers the host network with the host-net config above. With either, the downloads above completed on that VM; on its default 1500-MTU bridge they hung.

A container creates root-owned files that block publishing

• Symptom: A step writes a file/dir owned by root with restrictive permissions, and Nextflow — running as your user — cannot read or publish it. Examples: CUSTOM_SRATOOLSNCBISETTINGS and macrel (mode 600), or STAR in rnaseq (_STARgenome/ / _STARpass1/ as drwx------, failing with AccessDeniedException).
• Why: nf-core's docker profile maps your host user into the container (docker.runOptions = '-u $(id -u):$(id -g)'). Any --config that sets docker.runOptions replaces this value rather than adding to it, causing containers to fall back to root.
• Fix: Keep the user mapping folded into the same runOptions string in your --config:
  groovyCopy

  // run-as-user.config
  docker { runOptions = '-u $(id -u):$(id -g)' }

  Keep $(id ...) inside single-quoted Groovy strings so the task shell expands it; double quotes cause a config parse error. Alternatively, hardcode numeric ids (id -u / id -g): docker { runOptions = "-u 1000:1000" }.

No space left on device while writing results or run records

• Symptom: Nextflow, provenance recording or a chain state write fails with No space left on device; the final outcome may be incomplete if the log cannot be written either.
• Fix: Check free space on the output, work, temporary and engine-cache volumes (df -h). Free space or choose a larger writable work directory with --nxf-env NXF_WORK=/path/to/work. Preserve the existing run and work files needed for --resume, provenance checks and replay; moving only the output directory does not relocate recorded external dependencies.

Engine caches and temporary directories inside an input directory

Configured Nextflow engine caches and temporary directories must remain outside declared input directories. The launcher can create framework and Java-version cache entries even while probing its version. Boundary checks cover NXF_HOME, NXF_DIST, NXF_TEMP and CAPSULE_CACHE_DIR before metadata probes, including inherited settings and relative paths.

Runs, stops and resume

A successful run is missing samples

• Symptom: A run (or chain stage) ends success, but a sample has no results — e.g. no column in rnaseq's merged count tables.
• Why: Pipelines filter samples and tolerate task failures without failing the run. rnaseq skips a sample below its --min_trimmed_reads or --min_mapped_reads threshold (observed: a public paired-end run whose deposited read 2 was a 4 bp index read lost every pair in trimming), and nf-core's template ends "Pipeline completed successfully, but with errored process(es)" when failed tasks were ignored. Exit zero does not establish that every requested sample or task completed; nfclaw's additional dependency and provenance checks do not establish that completeness either.
• Behavior: nfclaw quotes the pipeline's own completion caveats and Nextflow's count of ignored task failures: as warnings after the run, in nfclaw status (status: success — with caveats the pipeline reported about its results), in run_manifest.json (completion_caveats) and per stage in the chain log and nfclaw chain status. The outcome stays success; nfclaw reports what the pipeline said and does not reinterpret it. A pipeline that drops samples without saying so is not detected: compare the samples you supplied with those in the results.
• In three demos: the bundled test data gives one tool too little to work with, and the pipeline's configuration tolerates the failure, so the demo ends success with caveats and that task's results are missing. cutandrun 3.2.2's PRESEQ_LCEXTRAP stops at max count before zero is less than min required count (4) duplicates removed for both of its samples (334 read pairs with no duplicates, and 221 with two). hicar 1.0.0's CIRCOS_PREPARE stops at No data available for plot: its script found no same-chromosome interactions to draw. mag 5.5.0's MAXBIN2 exits 255 for one assembly at Marker gene search reveals that the dataset cannot be binned. The pipelines tolerate these failures on real data too, so read the caveat there: it may not have the same cause.

Intermittent No signature of method: java.util.LinkedHashMap.subMap() is applicable…

• Symptom: A run fails in a channel operator rather than in a task: .nextflow.log shows groovy.lang.MissingMethodException: No signature of method: java.util.LinkedHashMap.subMap() is applicable for argument types: (String, String, String), although the pipeline calls the ordinary varargs meta.subMap('a', 'b', 'c').
• Observed: Once among the fastqrepair 1.1.1 native validations on Nextflow 25.10.4 (GitHub-hosted Ubuntu runner, 2026-10-13); the identical job passed in every run before and after it. The stack trace fails inside Groovy's invokedynamic call-site cache (IndyInterface.fromCache), not in pipeline or nfclaw code, and no input changed.
• Also seen: Cannot invoke "java.util.Map.get(Object)" because "self" is null, printed with no failed task, followed by Pipeline completed with errors: in hgtseq 1.1.0 on Nextflow 22.10.1, on 2026-11-05 and again on 2026-11-07. Each time its --resume and replay then passed. The cause was not established.
• Fix: Re-run the same command with --resume: completed tasks are reused. If it recurs, report it to Nextflow with the .nextflow.log stack trace.

--resume resumed the wrong session

• Status: Fixed. nfclaw run launches Nextflow from the --outdir, so each run owns its own .nextflow/ history and cache. --resume resumes this outdir's session — it can no longer pick up another pipeline's run. Always use a distinct --outdir per pipeline.

A dependency changes while the pipeline is running

• Status: Detected at completion checkpoints. On Nextflow 25.10.4 and 26.04.0, a gated task consumed a changed FASTQ while the original wrapper reported success and recorded the old input SHA-256. The mutation preserved the file size and timestamp.
• Behavior: Every real run snapshots explicitly declared local inputs, configurations and tracked pipeline source, then compares them after execution. Changed bytes, namespaces, Git revision or unreadable dependencies prevent success. The bundle records the original hashes and a dependency_stability report; results remain for inspection. The original engine error is preserved when Nextflow also failed. An invalidated bundle cannot be replayed even after restoring inputs. Resume starts fresh and archives it, since cached tasks can contain results from the intervening data. --no-provenance still performs the comparisons. Standalone replay verifies its original bundle again before reporting success.
• Scope: This checks observed launch/completion states, not every byte a task read. A temporary edit restored between checks can evade it. Remote inputs and container images are not part of this check; the bundle records them after the run (environment.json) and a replay compares its own. Keep inputs and source stable throughout execution; this check establishes neither biological accuracy nor immutability.
• Validation: Native gated tasks test input mutations with restored metadata, configuration/source edits, unchanged controls and disabled provenance on both engines. Restoring original input bytes and timestamps after a tainted attempt forces a fresh task, confirmed by an execution counter and the corrected digest; the archive retains the tainted output. Separate replay gates consume changed data and fail their post-execution dependency check. Multiline post-execution errors retain a readable final status line.

Local-input tasks can reuse stale data after --resume

• Status: Fixed for explicitly declared local inputs. Ordinary Nextflow file and directory cache keys use path, size and timestamp rather than file bytes (Nextflow caching documentation). On both Nextflow 25.10.4 and 26.04.0, a same-size FASTQ byte edit with its timestamp restored reused the old digest result. Directory inputs have an additional recursive namespace problem: Nextflow 25.10.4 and 26.04.0 reuse a cached directory-consuming task after an empty directory is renamed, or after a file in a subdirectory is edited in place (verified: the resumed task republished the old bytes), while the root size and timestamp remain unchanged. File checksums cannot detect the rename, and the namespace cannot detect the edit, so nfclaw checks both.
• Behavior: nfclaw records the full directory namespace and the SHA-256 of every declared local input file, including samplesheet data paths, before launch. Unchanged namespace and content preserve normal resume caching; a changed namespace, changed file content or an unavailable previous inventory preserves the previous run in a sibling <outdir>.previous-<id> archive, starts a fresh cache session, and recomputes all tasks. This avoids stale publications and fixed report filename conflicts; prior logs are retained and copied forward. Active engine database locks and dependencies overlapping the run directory block archiving. This also applies when the new launch disables provenance. Remote and indirectly configured inputs are outside this preflight's scope. Without a valid historical input snapshot, local-input resumes start fresh because unchanged bytes cannot be established. Whole-file path changes remain subject to Nextflow's own path cache keys.
• Validation: Real tasks enumerated sample_A before the change and sample_B after resume on both engines, with the root timestamp restored. Unchanged resumes reused the cached task. Sample-specific output files proved that old publications stayed in the archive, while a fixed trace filename was successfully regenerated. Standalone replay refused the changed inventory before launching Nextflow. Editing a nested file to same-size content with both directory timestamps restored made the resumed run recompute and publish the new bytes on both engines, and an unchanged follow-up resume reused the cache. Independent SHA-256 and execution-counter controls also verified same-size FASTQ mutations with the file timestamp restored on both engines: changed bytes recomputed, and unchanged inputs with valid snapshots reused the task. When provenance is disabled and no earlier snapshot exists, subsequent resumes recompute instead of trusting metadata. Shared physical file targets are hashed once while logical aliases remain distinct.

--resume kept a removed sample's results

• Status: Fixed. Resuming after a sample was removed from the input selection (a hand-written or generated samplesheet) reused the session, and the removed sample's earlier publications stayed in --outdir beside the new results. Any change to the recorded set of local input files (added, removed, replaced or retargeted) now starts a fresh session; the previous run moves to a sibling <outdir>.previous-<id> archive first. Adding samples therefore recomputes all tasks instead of reusing the earlier ones.

--resume after Ctrl-C or kill recomputed every task

• Status: Fixed. A run stopped with Ctrl-C or kill <nfclaw pid> skipped the post-execution dependency comparison and recorded it as not_checked; --resume read that as failed attribution, archived the run and recomputed every task, although a run that failed (or was killed with SIGKILL) kept its cache. A stopped run now compares its declared local inputs, configurations and pipeline source when it stops, as a failed run does: unchanged dependencies keep the cache for --resume, and changed ones still start a fresh session with the previous run archived. The log keeps the stop as the outcome (terminated by SIGTERM, interrupted).
• Validation: On Nextflow 25.10.4 with Docker, demo runs stopped with SIGTERM once their MultiQC task was submitted recorded unchanged dependencies and resumed with 7 (bundled remote test data) and 5 (local FastQ directory) cached tasks, no archive and a successful outcome. Regression tests cover both stop signals and a pipeline source edited before the stop, which is recorded as changed and makes the resume start fresh.

--resume after SIGKILL, out of memory or a host restart recomputed every task

• Status: Fixed. The bundle is written when a run ends, so a nfclaw that vanished, or could not finish it on a full disk, left nothing for --resume to compare the sources and inputs against: it archived the run and recomputed every task, although nfclaw status and the full-disk error promised to continue. A launch now writes provenance/launch-record.json before Nextflow starts, and --resume compares against it when no manifest exists; a changed source or input still starts fresh. A run launched with --no-provenance records nothing and still recomputes, and nfclaw status says so.
• Validation: On Nextflow 26.04.3 with Docker, a demo run whose nfclaw received SIGKILL after 4 finished tasks resumed with its finished tasks cached, no archive and a successful outcome. Its commands.sh replay succeeded and nfclaw verify reported the same files and an identical environment.

A run aborts with process hasn't exited

Caused by: process hasn't exited (in .nextflow.log, java.lang.IllegalThreadStateException) means a task ran past its time limit on the local executor. Nextflow before 26.10.0 reads the exit status of the task it has just killed without waiting for it to exit, and the exception aborts the whole run instead of letting the task's errorStrategy retry it (nextflow-io/nextflow#7569, fixed in 26.10.0). The pull of a task's container image counts toward its limit, so test profiles with minute-scale limits trip it on a cold image cache or a busy host. proteinfamilies 2.6.0's test profile gives most tasks 2–3 minutes: a fresh --demo aborted when a CLIPKIT task spent 1.5 of its 2 minutes pulling its image. nfclaw status names this cause in its fix: line. Re-run with --resume; with the image now present, the task usually completes. If it times out again, raise that process's time with --config, or run on Nextflow 26.10.0 or later, which retries the task instead. Reported to proteinfamilies as nf-core/proteinfamilies#209.

--resume fails with "Unable to acquire lock on session …"

• Symptom: After a run was interrupted (killed/timed out), re-running with --resume fails to acquire the session lock.
• Why: A hard kill leaves Nextflow's session lock behind; Nextflow refuses to resume because it cannot determine whether the lock is stale or held by a live process.
• Fix: Ensure no Nextflow process for that --outdir is active, remove the stale lock under <outdir>/.nextflow/cache/*/LOCK (or start fresh in a new --outdir), and re-run with --resume.

A run in the background: is it still going, and how did it end?

• Symptom: A run started with nohup nfclaw run … & — its run.log has no terminal line, or you need its outcome without opening the file.
• Fix: nfclaw status <outdir> answers from the log alone: running (nfclaw pid … on host …) with the last output; the outcome it ended with and the recorded error; refused before launch; or stopped without an outcome. Re-run the command with --resume. Stop a background run cleanly with kill <nfclaw pid> — never kill -9 — so logs and provenance are finalized.

Launching several pipelines in parallel

• Status: Fixed. Starting 2+ pipelines simultaneously whose submodules were uninitialised previously raced on .git/config. nfclaw run takes a per-repository file lock while it initialises submodules, so concurrent first runs do not collide.

Concurrency: another nfclaw run is active in …

• Symptom: nfclaw run aborts immediately with [environment] another nfclaw run is active in <outdir>.
• Why: To prevent session corruption, runner/locking.py holds an exclusive non-blocking sibling flock on .{outdir.name}.nfclaw.lock throughout workflow execution.
• Fix: Wait for the active run to finish (nfclaw status <outdir>) or terminate it (kill <pid>) before launching or resuming in that directory.

Replay and verification

Replaying a run: provenance/commands.sh refuses a target that holds files

• Symptom: commands.sh refuses its target directory, or a replay into the original --outdir fails on pipeline_info/execution_trace_*.txt.

• Fix: Replay into a fresh directory: commands.sh defaults to <outdir>.replay, or takes a target (./results/provenance/commands.sh /tmp/check-it); replay explains why a replay cannot publish over the original run.

  Stop it with kill <replay pid>. New bundles run the command in its own process group, allow the shutdown grace recorded at launch (10 seconds by default), then kill surviving group members before closing the log and releasing the writer lock. Set NFCLAW_SHUTDOWN_GRACE before the original run to record another margin; timeouts explains how new and historical replay scripts use it. A stopped replay returns a nonzero exit status even if the command handles the signal with exit zero. Tasks or containers detached from that group still require executor cleanup.

A replay can change bytes and timestamped metadata filenames

Some outputs embed dates or durations: execution reports, timelines, gzip headers, FastQC ZIP entries and MultiQC HTML can differ between runs. A changed checksum does not identify the cause; inspect scientific results with a format-aware comparison. Matching file inventories alone does not establish analytical agreement. Compare the two with nfclaw verify results.replay --against results; the verification engine explains its default and --strict checks.

nfclaw pins the schema's trace_report_suffix where available, but some upstream code generates other filenames directly from the clock. For example, demo 1.2.0's dumpParametersToJSON() independently names pipeline_info/params_<timestamp>.json. Every faithful replay therefore has a differently named file, so comparing exact paths failed every replay of the modern template (demo 1.2.0, ampliseq 3.0.0) and the failure no longer distinguished a broken replay. nfclaw verify pairs these run-metadata names (the execution report/timeline/trace/DAG of releases without trace_report_suffix, and the nf-co2footprint plugin's launch-stamped report/summary/trace files) one to one, lists each pair under renamed as original -> replay, and compares its bytes like any other file. The pairing is explicit, never silent; any other dated path — possibly a scientific result — keeps its exact identity. A resumed run can retain reports from earlier attempts: several candidates for one name are never paired by guess and stay missing or extra for explicit classification.

Do not diff the two outputs.sha256 files directly. Each line is hash path, so a file whose content merely changed has a different line in each bundle and shows up as both "missing" and "extra" — one changed report is counted twice, and a run whose reports simply carry a new timestamp reads as hundreds of missing and extra files. That is an artefact of the comparison, not a defect in the replay; nfclaw verify keys on the path precisely to separate the two questions.

Replay dependency guard failures (provenance/replay_guard.py)

• Symptom: Running <outdir>/provenance/commands.sh aborts with nfclaw replay: dependency check failed: ... before Nextflow is launched.
• Why: commands.sh executes replay_guard.py to ensure local input files (including paths referenced in samplesheets), configs (the bundle's copies of external -c files where it holds them), the archived pipeline source (pipeline-source.tar.gz; a bundle without one checks the checkout's commit and tracked bytes) and the recorded environment.json / containers.config match their recorded SHA-256 digests. New bundles also check directory namespaces, empty directories, entry types and symlink identities against inputs.inventory.json and configs.inventory.json. A checksum-only prelaunch snapshot cannot certify the original directory namespace and is refused.
• Fix: Restore the dependency or bundle artifact named in the diagnostic from an intact original. Replayable config copies are checked inside provenance/configs/; editing or deleting their external originals does not invalidate those copies. Configs using includeConfig still need the original paths. New source archives allow a changed or missing checkout to fall back to extracted source; older bundles still require their recorded checkout. If input changes are intentional, launch a new run into a new --outdir instead of replaying.

A replay fails after Nextflow succeeded: execution environment differs from the recording

• Symptom: A replay's run log ends with failed after Nextflow succeeded (execution environment differs from the recording), and the console lists mismatch: lines.
• Why: After a replay's Nextflow succeeds, commands.sh records the replay's own execution environment (<target>/provenance/environment.json) and compares it with the original bundle's (<target>/provenance/environment_check.json). A process that ran another image digest (an image without a recorded digest cannot be pinned, so its tag may have moved), another set of Nextflow plugins, another container engine, or a remote input whose staged bytes changed (test data on a moving branch, an updated database URL) makes the replay a different execution, whatever its outputs.
• Fix: Read the mismatch: lines and recover the recorded data or image identity where possible; a mismatch does not establish that recovery is impossible. Keep local copies of remote data needed for future reproduction. unverified: lines (an image without a comparable identity, a Singularity image converted on another host) and note: lines (another Java runtime, a process that ran in only one of the runs) do not fail standalone replay. Inspect those gaps explicitly; a successful replay need not have an identical environment comparison.

A process cannot be pinned to an image digest

• Symptom: nfclaw run ends with provenance: N process(es) are not pinned to an image digest for replay: … or provenance: N container image(s) have no recorded identity: ….
• Why: For named Docker/Podman profiles, nfclaw observes task-container start events and inspects image identities while the run is active. Observation is best effort, reinspecting a reused reference after 600 seconds; missing observations fall back to post-run inspection or an earlier attempt's record when available. Locally built images can lack registry digests, a process running several images or containerless tasks has no single pin, and ephemeral Wave references are excluded. Remote executors may leave no image in the local engine. Singularity/Apptainer files are hashed after execution and do not receive registry pins.
• Fix: Inspect environment.json (images, pinned_containers, issues) and the manifest's container_pins entry. containers.config is absent when no process qualifies. Unpinned processes retain their original container configuration, which may use tags or local image files; the replay compares the evidence it can obtain. Push locally built images to a registry and reference them by digest when registry pins are needed.
• Tasks that run no container: mcmicro 2.0.0's OMEVALIDATION is an exec: task, which runs inside Nextflow; circdna 1.1.0's CIRCLEFINDER and GETCIRCULARREADS declare no container, so they run the host's awk, sort, grep, sed, zcat and gzip; and raredisease 3.1.2's PREP_MITOSALT and SALTSHAKER_TO_HTML declare none either, so they run the host's find, sed and python3. nfclaw status lists such tasks under provenance gaps. Their replay's environment check therefore stays unverified for those processes, and nfclaw verify --require-environment fails, however faithful the replay.

A run has no task records: Nextflow holds no task records for this run

• Symptom: a successful run warns provenance: Nextflow holds no task records for this run…. Its replay reports execution environment unverified and the recording holds no task records to compare containers with, and nfclaw verify --require-environment fails with no complete process inventory was recorded.
• Why: cleanup = true makes Nextflow delete a successful run's task records along with its work files, so nfclaw cannot identify or pin any container. A replay then runs the images its references resolve to at that time, and nothing compares them. genomeqc 1.0.1's test and test_gca profiles set it, so its --demo always ends this way. Measured with the demo pipeline and cleanup = true in a --config file: 0 processes, 0 images and 0 pins recorded, and the replay succeeded unverified.
• Fix: for a complete record, pass a config holding cleanup = false with --config: on Nextflow 25.10.4, nextflow config -profile test,docker of genomeqc 1.0.1 then resolves to cleanup = false.

Replay refuses an unobserved engine or malformed environment names

• Why: New bundles pin replays to the engine version observed in the executed Nextflow banner, even when the original environment selected latest or another moving selector. If no version was observed and no concrete NXF_VER was set, the bundle cannot select that engine faithfully. Environment names that cannot be represented as shell identifiers are also refused.
• Fix: Start a fresh run with a valid environment and an explicit --nxf-ver from the pinned pipeline's documented engine requirement. Do not guess which engine an earlier unobserved run used. Private URI credentials are omitted from exported replay values; supply any needed credentials through your environment.

File-only inventories missed empty directories and changed aliases

• Status: Fixed for new recorded namespaces. Output verification previously accepted two trees with identical files but different empty-directory names. Reference handoffs could also accept changed directory shapes or retargeted aliases whose file bytes remained identical.
• Behavior: New bundles record outputs.inventory.json. Verification reports missing/extra directories and changed link identities alongside file differences. Old file-only manifests provide byte comparisons, but historical directory agreement remains unverified and cannot return a successful structural check. Live trees without historical manifests remain comparable.
• Handoffs: Published output namespaces and local reference namespaces are verified before the next stage starts. Chain status checks source namespaces and the downstream prelaunch snapshot. Complete run-directory handoffs retain all entries and record execution metadata at the boundary separately from historically published results.
• Validation: Real producer and consumer tasks exercised external references, published directories and complete run directories. Renaming an empty sample directory while retaining reference bytes blocked the consumer; restoring the original namespace allowed resume, and subsequent changes broke lineage. These execution and conservation checks do not establish biological accuracy.

Inputs and samplesheets

Legacy samplesheet contracts and local data identity

Some pinned schemas omit path formats on consumed reads, references or indexes, or declare them only inside anyOf/oneOf. The runner applies declared alternative formats to the values they match. Additional file roles are documented in runner/input_roles.json, keyed by the original schema's SHA-256 and supported by source references. Handwritten sheets and handoffs receive the same local path checks, input hashes and namespace capture as generated sheets. Metadata such as Nanostring's RCC_FILE_NAME, and absence sentinels such as Bacass's NA and Isoseq's none, are preserved as literal values.

HGTseq's pinned 1.1.0 consumer reads sample,input1 and optional input2, while its template lists fastq_1/fastq_2. Nanoseq 3.1.0 requires group,replicate,barcode,input_file,fasta,gtf in that order. The runner and generated documentation correct these stale contracts using the exact pinned consumer source, without editing upstream files. Automatic generation is refused because these formats require explicit data-role, barcode, replicate or reference assignments. Corrections apply only at the audited revision, and a changed consumer source there is refused; another release (an older one run with --pipeline-version, or a newer pin) keeps the contract its own schema declares until it has its own reviewed contract. The pinned consumers continue to enforce extension compatibility, replicate grouping and their other runtime constraints.

The audited roles also cover directory-valued sources (including Methylong POD5 inputs and microscopy datasets) and auxiliary reference sheets. Oncoanalyser FASTQ filepath values contain exactly two nonempty semicolon-delimited paths; both are validated, normalized during handoffs and hashed independently. group_id, subject_id and the library_id, lane, flowcell info values must be literal safe aliases. Info keys use the consumer's uppercase normalization. An apostrophe in read_group_overrides is unsupported by the pinned single-quoted BWA argument and is refused. Datasync output_path locations are destinations rather than consumed inputs. The composite interpretation and destination exclusion require the exact audited revision and source hashes.

Datasync passes complete source and destination strings into double-quoted command arguments. The runner refuses double quotes, backslashes, dollar signs, backticks and control characters anywhere in those strings, including parent folders. Destinations are not required to exist or inspected as input directories; apostrophes remain supported by that quoted argument.

Some upstream consumers infer companion files that are absent from the samplesheet (for example, Sarek's SnpSift database index defaults to vcf.tbi). Input hashes cover explicitly named paths; they do not comprehensively discover these inferred dependencies. Supply index paths explicitly where the consumer supports them, and preserve the complete reference dataset for reproducibility.

Runs and handoffs share the same declared and conservatively discovered local dependencies. A sheet mixing declared paths with pattern-only reads records both sets, so restoring producer bytes cannot mask different data captured by the consumer. Equal filename patterns do not establish a metadata role: an unformatted second mate remains a data path even when its first mate has a path format. Nanostring's RCC_FILE_NAME exclusion instead requires its exact audited revision and consumer hashes, and applies to input capture and handoff derivation. Untyped absolute existing-file values without a proven exclusion are recorded conservatively; this is dependency attribution, not proof that an analysis task read every recorded file.

An explicit remote input has an unsafe staged filename

Nextflow can retain literal shell characters in a remote basename. A task deriving a prefix from that name can interpolate shell syntax even when its input path object is escaped. For example, a local HTTP fixture named reads$(id).txt produced printf '%s\n' reads$(id) > name.txt on Nextflow 25.10.4 and executed command substitution. nfclaw now refuses explicit unsafe remote names in path parameters and CSV/TSV or flat JSON/YAML path fields before launching. Use a safely named remote object or download the data into a safely named local file; sample identities are preserved.

Percent escapes remain literal in the staged name and are not decoded by this guard. HTTP query suffixes are excluded from the filename check so signed URLs remain usable. A standalone HTTP fragment remains in the staged name and is checked. S3 paths retain both query and fragment characters in their object keys. The local HTTP integration oracle checks actual staged names and bytes on Nextflow 25.10.4 and 26.04.0 without interpolating their names into executable script text. Local directory inputs and directory glob matches are inspected recursively, including empty subdirectories and resolved symlink target names. Broken members, unreadable subtrees and directory cycles are refused before launch. Parent names outside the explicit directory are not staged names and retain their separate path contract.

Remote file contents, remote glob matches and values supplied only by configs/profiles are outside this explicit-value preflight.

A valid input symlink lost its filename and format suffix

• Status: Fixed. Resolving a whole input path replaced sheet.csv with its target stored.data. The CSV parameter constraint then rejected it; format-dependent samplesheet parsing and filename-derived identities could also use the target name. Local path parameters now resolve their parent directory while retaining the supplied basename. Alias and target safety checks still apply. Replay records and checks the alias identity, even when a retargeted link has identical bytes. Reference handoffs preserve the alias too; chain status can still verify older bundles that explicitly recorded the resolved target as their consumed parameter.

Output or work directories contaminated a directory input

• Status: Fixed. A real Nextflow 25.10.4 task consuming a directory observed .analysis.nfclaw.lock and analysis alongside the original index.bin when its --outdir was a child of that input. The sibling lock was created before input validation, and the output tree appeared after the prelaunch snapshot. Dependency separation now precedes lock acquisition and all generated run files. Checks cover explicit directory inputs, samplesheet data paths, directory glob matches, nested aliases, code/configuration dependencies, generated-file redirects, the configured Nextflow log, work directories and chain records. A boundary refusal does not append to an existing run log inside the input. Refusal logging for other errors acquires the writer lock so it cannot forge the outcome of a live run. Real-engine tests on 25.10.4 and 26.04.0 verify unchanged input bytes and directory entries, then execute and replay a separate safe output location with literal filename and content oracles. Previously computed task files beneath a shared work tree remain legitimate downstream inputs; consuming the whole work tree is refused.

Staging routes and checksum records disagreed about local data

• Status: Fixed. Native controls on Nextflow 25.10.4 and 26.04.0 showed that a newline or carriage return in a source parent becomes a separate shell line, and directory_symlink/../reads.fastq can select a different file after Nextflow's lexical normalization. Local input paths and resolved targets containing LF/CR are refused. Samplesheet routes with differing filesystem and lexical meanings are refused; scalar path parameters resolve their parent before staging while retaining the basename. Wildcard parents followed by .. are refused before resolution, and glob matches receive the same route checks. LF/CR in launch, output or work paths cannot be enabled with --allow-spaces.
• Ordinary absolute parent traversal remains supported. Its logical dependency keys and resolved identity are retained; relative checksum keys cannot traverse outside a bundle. Checksum records use the GNU escaped filename format for backslashes, LF and CR. Readers split records only at LF, so Unicode line separators remain filename characters, and reject invalid escapes and decoded duplicate keys. Plain legacy records and CRLF record endings remain supported.
• An absolute local parent such as /path/data://reads.fastq was also mistaken for a remote URL and omitted from input checksums. URI detection now requires a scheme at the start of the value. Native execution and relocated replay controls check supported backslash/Unicode parents and local colon parents, then confirm that changed input content is refused. These are execution and provenance controls, not proofs of biological algorithm accuracy.

[samplesheet_invalid] while generating a samplesheet from --input data files

nfclaw run <name> --input <directory | glob | files> writes the samplesheet from the files and refuses, before anything is written, whatever it cannot decide from them. Each message names the files or columns; the fixes:

Files that match no column (checksums, reports, index reads _I1/_I2, the demultiplexer's Undetermined_S0 reads) are ignored and listed before launch. _R1/_R2 files are paired as named; when their first read names differ (a swapped, truncated or desynchronised mate — what fastqrepair repairs) the pre-launch advisory says so. Two FastQ files named x_1/x_2 pair only when their first read names show they are mates; otherwise they are two single-end samples.

Existing output aliases can expose input data to execution writes

• Status: Fixed for explicitly declared local dependencies. Resolved root paths alone do not establish separation. A run log hard-linked to an external FASTQ shares its bytes, and an output directory can contain a symlink into an external input tree. The production log writer was reproduced appending header text to a hard-linked input fixture.
• Behavior: Before locks, logs or pipeline execution, nfclaw inspects existing output namespaces and compares physical file identities with consumed dependencies. It refuses hard links and symlink routes that permit writes into those inputs, including logical input slots beneath output trees whose targets are external. Declared work/cache trees retain their staged-read policy. Refusal logging leaves multiply linked logs untouched even when parameter validation fails before dependency preflight.
• Validation: Controlled fixtures preserve input bytes and directory timestamps on refusal. Separate controls retain normal staged file/directory aliases, unrelated hard links and cycle-safe traversal. This preflight checks the filesystem state it observes; independent concurrent changes outside nfclaw's output writer lock remain outside its guarantee.

A sample sheet or pipeline revision changes during preflight

• Status: Refused at preflight and launch checkpoints. A valid sheet changed during the engine probe previously launched the replacement data while recording the original dependency selection. A pipeline checkout changed to another clean commit could similarly record its source hashes under the earlier validated commit.
• Behavior: Local primary and auxiliary sample sheets are fingerprinted before validation. Their bytes and namespace must still match after dependency discovery, external probes and launch preparation, including --check. Real runs also bind the descriptor fingerprint to the captured input baseline. Pipeline identity and tracked-file cleanliness are rechecked after probes and launch preparation; the captured source revision must equal the validated revision. A changed descriptor or source is refused rather than silently adopting a new selection without validation or write-boundary checks.
• Validation and scope: Controlled fixtures replace valid descriptors, retarget them to identical-byte files and switch clean revisions or edit source during preflight and command construction. These are checkpoint comparisons; transient changes restored between checks remain outside their guarantee. Keep descriptors, inputs and source stable throughout preflight and execution.

Numeric parameters can change before Nextflow receives them

• Status: Fixed at the wrapper boundary. Binary-float conversion changed 9007199254740993e0 into 9007199254740992, changed 0.1000000000000000000001 into 0.1, and underflowed 1e-9999 to zero. Converting the float 1e23 directly to an integer also emitted 99999999999999991611392 rather than the JSON value 100000000000000000000000.
• Behavior: CLI flags and JSON/YAML parameters retain their decimal value, including nested values and chain records. Precise values are emitted as JSON numbers without quoting or expansion of extreme exponents into digit strings. Plain JSON integer literals outside the signed 64-bit range also wrapped in the tested engines, so those values are emitted with a decimal point to retain their value in Nextflow. Schema enum/type/boundary checks and numeric CSV/TSV bounds compare that emitted value. Non-finite values remain refused. Pipeline engines, validation plugins and scientific tools retain their own numerical ranges and arithmetic precision.
• Validation: Independent decimal-text controls check serialization, enum and boundary decisions, YAML aliases, and chain records/fingerprints. Native tasks on Nextflow 25.10.4 and 26.04.0 observe the original numbers from CLI, JSON and YAML and preserve them in standalone replay. Separate nf-schema 2.5.1 and 2.7.2 controls on both engines accept the exact integer enum and exact decimal bounds without changing task-observed values.

Enum preflight confused JSON types and rounded large integers

• Status: Fixed. Enum documentation text was also used for validation, so numeric 1 could match the string enum "1", and converting enum entries to floating point could make 9007199254740992.0 match the distinct integer 9007199254740993. A matching enum also bypassed independent type, pattern and numeric constraints.
• Behavior: Preflight retains typed enum values, distinguishes booleans from numbers and compares numbers without converting integer enum entries to floats. Independent constraints still apply to enum members. Whole-number CLI tokens for number parameters retain their integer precision. Enum-only schemas may contain mixed JSON types; genuinely equivalent 1 and 1.0 remain accepted, consistent with JSON Schema enum semantics and numeric semantics. Version-dependent treatment of unset values remains delegated to nf-schema.
• Validation: Regression controls cover scalar and nested enum types, large integers, empty enums, conflicting independent constraints and CLI conversion. An independent JSON Schema validator agreed on 840 decisions. Every enum in the pinned library was checked for conflicts with its declared independent constraints; none were found. These controls verify preflight decisions rather than biological accuracy or arbitrary-precision support throughout Nextflow.

Chains

Chains: [handoff_failed] — the next stage's input could not be prepared

• Symptom: nfclaw chain run stops after a stage succeeded, with [handoff_failed] and the chain log ending failed at stage NN-<stage>: handoff.
• Why: The handoff rule expected an output the upstream run did not produce (or produced multiple matching files), or the generated samplesheet failed downstream schema checks.
• Fix: Inspect the snapshot in <outdir>/chain/handoffs/; provide parameters directly in stage params, supply an inline handoff, and resume: nfclaw chain run spec.json --outdir DIR --resume.

Chains: another nfclaw chain is running in …

• Symptom: nfclaw chain aborts indicating another chain process is running.
• Why: Only one chain process is permitted per --outdir to prevent race conditions on shared stage state (chain/.lock).
• Fix: Wait for the active chain to complete (tail -n 1 <outdir>/chain/logs/chain.log) or stop it with kill <pid>.

Chains: what a demo: true stage hands over

When a stage specifies demo: true, it executes with its bundled test profile dataset:

• createtaxdb: The test profile also builds Sourmash databases, which taxprofiler does not use. The exact pinned 3.1.0 source now receives the recorded copy-sort correction in nfclaw runs and chains. Supplying "params": {"build_sourmash_dna": false, "build_sourmash_protein": false} deliberately omits those optional builders; it is no longer required to avoid the shared-list race in corrected nfclaw execution.
• detaxizer: The test profile filters reads with an empty database, removing all reads and leaving empty FastQ files. For downstream chaining, provide active classification parameters: "params": {"classification_kraken2": false, "classification_bbduk": true, "fasta_bbduk": "<host fasta>"}.
• rnaseq → differentialabundance: rnaseq's test data leave 100 genes after differentialabundance's default abundance filter — fewer than DESeq2's default variance-stabilisation subsample (--deseq2-vst-nsub, 1000), so DESEQ2_DIFFERENTIAL fails with less than 'nsub' rows. Give the stage "deseq2_vst_nsub": 50 along with its own input (a design sheet naming rnaseq's samples) and contrasts. With that, an rnaseq demo: true → differentialabundance chain completed (the failed stage resumed after editing the spec, rnaseq skipped as already succeeded) and nfclaw chain status verified the lineage; DESeq2 tested all 100 genes.
• bamtofastq → rnaseq: the bamtofastq stage needs its fasta (see the upstream table). Its test BAMs hold 2,821 and 2,718 read pairs of human chr22:1-40001, below rnaseq's --min-trimmed-reads (10,000): rnaseq then skips both samples and reports it as a caveat. Give the rnaseq stage the same chr22 test fasta and gtf (six genes), "min_trimmed_reads": 1000, and skip the steps that need more genes than that, "skip_dupradar": true and "skip_deseq2_qc": true. rnaseq's test profile also runs a Salmon pseudo-alignment against its own yeast transcript_fasta and prebuilt salmon_index, which fasta and gtf do not replace: add "skip_pseudo_alignment": true (or supply a matching salmon_index). Without it the human reads are quantified against yeast transcripts; on 2026-10-29 that step's gene-level SummarizedExperiment failed (Rows or columns different among input matrices), its matrices holding only the GFP transgene. With these settings the chain (the build handoff) completed with verified lineage and STAR-Salmon counts for both samples; rnaseq reports both as failing its strandedness check, which six genes cannot settle.
• demultiplex: Test data consists of human amplicon reads. Downstream stages requiring reference genomes need human references (fasta/gtf), and paired-end profiles: "profile": "test_pe,docker".
• taxprofiler: Test profiling on unrecognized reads fails in unhandled zero-hit modules; enable only resilient profilers (run_kraken2, run_kaiju) with "run_profile_standardisation": false. With the detaxizer parameters above and every other run_* profiler switched off, a detaxizer demo: true → taxprofiler demo: true chain completed with verified lineage; all five handed-over read sets (three short-read, two long-read) were profiled with kaiju, and the short-read sets with kraken2.

Warnings a run prints that are not faults

These messages originate in upstream configurations, plugins or tools. Some have been observed during successful runs, but a warning or an exit-zero process alone does not establish analytical correctness. Interpret each message in the context of its documented version and cause. A replay can reproduce the same message; cache, network and environment changes can also change what it prints.

• .command.trace: cannot overwrite existing file and Error: Failed to write to file: .command.trace in a task's .command.err — nf-core's newer shell settings include -C (noclobber), and Nextflow's trace writer creates that file before writing it. It was in 94 of a reportho run's 103 task directories on Nextflow 25.04.0, on tasks that succeeded, so it does not explain why a task failed.

WARN: Unrecognized config option 'validation.defaultIgnoreParams' / 'validation.monochromeLogs'

• Cause: Running a Nextflow engine newer than the release targets. Harmless and avoidable.

  Nextflow Version	Output / Result
  25.10.4 (declared minimum !>=25.10.4)	No warnings emitted
  26.04.6	WARN: Unrecognized config option 'validation.defaultIgnoreParams' + 'validation.monochromeLogs'

  Both options are set by the pipeline's own nextflow.config inside the validation scope contributed by nf-schema. Nextflow 26's strict parser checks config options before plugins load; because nf-schema registers no ConfigScope for validation, the parser warns. Tracked upstream in nextflow-io/nf-schema#117.

• Fix: Run the engine the release was authored for:

  bashCopy

  nfclaw run scrnaseq --input ss.csv --outdir results -profile docker --nxf-ver 25.10.4

WARN: Unrecognized config option 'manifest.diagram'

• Cause: Running a Nextflow engine newer than the release targets. Releases built from the current nf-core template set manifest.diagram (the path of the pipeline's metro-map image) in nextflow.config; the strict config parser of Nextflow 26.04 does not know that option and warns once per launch. Nextflow 25.10.4 prints nothing (verified with rnaseq 3.27.0 on 25.10.4 and 26.04.3). The option only names a documentation image; it changes no parameter, process or result.
• Fix: None needed. To avoid the message, run the engine the release declares, e.g. --nxf-ver 25.10.4.

WARN: Could not load / include the nf-core institutional config (a host without network access)

• Cause: Nextflow could not fetch nf-core's optional remote institutional configuration from GitHub at parse time. sarek 3.11.0 (and 3.10.0/3.9.0) includes it dynamically in nextflow.config:
  groovyCopy

  includeConfig params.custom_config_base && (!System.getenv('NXF_OFFLINE') || !params.custom_config_base.startsWith('http'))
      ? "${params.custom_config_base}/nfcore_custom.config" : "/dev/null"

  On air-gapped or IPv6-only hosts unable to reach GitHub, the remote file cannot be resolved. The configuration is purely optional tuning.
• Fix: Set NXF_OFFLINE=true to skip remote includes cleanly:
  bashCopy

  nfclaw run sarek --input ss.csv --outdir results -profile docker --nxf-env NXF_OFFLINE=true

WARN: The following invalid input values have been detected: * --igenomes_base: … (an institutional profile)

• Cause: Set by an institutional profile, not by the pipeline or nf-claw. For example, binac2.config sets params.igenomes_base for all pipelines; pipelines that do not resolve reference genomes (such as fetchngs) do not declare this parameter, causing nf-schema to flag it. It is benign and does not affect analysis.

ERROR org.pf4j.AbstractExtensionFinder - Different class loaders

• Cause: A plugin-cache artifact on the host where pf4j discovers the same extension point loaded by two class loaders.
• Fix: Clear the plugin cache and re-pin the declared engine:
  bashCopy

  rm -rf "${NXF_HOME:-$HOME/.nextflow}/plugins"

WARN: nf-core pipelines do not accept positional arguments. The positional argument \nextflow` has been detected.`

• Cause: A known defect in the pinned nf-core-utils@0.4.0 plugin used by sarek 3.11.0 (and 3.10.0/3.9.0). The plugin's validator splits session.commandLine on whitespace and incorrectly inspects the launcher binary name nextflow. It is benign (log.warn, exit 0) and does not affect execution.

WARN: --validationSchemaIgnoreParams: genomes is not a valid parameter (scrnaseq --demo)

• Cause: An upstream defect in nf-core/scrnaseq 4.2.0 test configurations referencing obsolete nf-validation 1.x parameters.
• nf-claw Mitigation: nfclaw run automatically neutralizes this warning by generating a narrow -c compatibility configuration:
  groovyCopy

  // <outdir>/provenance/nf_schema_compat.config
  validation { ignoreParams = ['validationSchemaIgnoreParams'] }

  Pipelines declaring nf-validation 1.x correctly (e.g. chipseq, atacseq, circdna) are untouched.

rnaseq --demo: --gtf with --gff, --transcript_fasta, and the first operator

• Cause: rnaseq's bundled conf/test.config intentionally sets gtf, gff, and transcript_fasta simultaneously to exercise multiple code paths, prompting Nextflow to log preference for --gtf. A real production run specifying custom references does not set both and emits no warning.

Upstream pipeline bugs (documented, not patched)

The exact-pin wrapper corrections are recorded in generated operating notes. HGTseq's bare demo now stages its immutable historical input/reference recipe; custom input, references, alignment indexes, data-base selectors, profiles or configs disable that whole recipe. Tumourevo's bare demo receives the intended sparsesignatures_k=2:3 with explicit user values preserved; its native 66-task control passed at a 24 GiB task ceiling. These corrections leave upstream sources intact. The table's explicit commands remain useful when using upstream directly or another revision.

These defects affect specific pinned releases. Workarounds and recorded runtime corrections keep upstream source trees unchanged; generated task scripts may differ when a documented correction is applied:

Pipeline-specific run notes

• demo 1.2.0 on macOS arm64 with amd64 containers — MultiQC 1.34 can return exit zero and write its HTML/data while Kaleido/Chromium fails to export plots under QEMU. Fresh and resumed local runs both reported 0/36 completed, with qemu: unknown option 'type=utility' and Chromium sandbox/GPU crashes. Check the task's .command.err and required plot files even when pipeline status is success. Validate required exports on a compatible native execution platform; no arm64 image or emulation workaround has been established yet.

• rnaseq 3.27.0 bundled test — A native Docker execution completed all 234 tasks on Nextflow 25.10.4. Downloaded bytes matched all 1,056 recorded output checksums. Independent consistency checks covered five samples and 125 transcripts in each of the Salmon and STAR/Salmon routes: finite nonnegative estimates, normalized TPM, assigned-fragment totals and merged transcript/gene tables agreed with individual results within their published rounding. This validates the bundled test's execution and internal numerical consistency, not biological expression accuracy, differential expression or every alternative aligner.

  • Replay: A --demo run on Nextflow 26.04.3 (Linux amd64, Docker) and its commands.sh replay produced the same 1,056 result files and directories (nfclaw verify: 0 missing, 0 extra). Salmon quant.sf, the merged gene count tables of both routes and featureCounts tables were byte-identical. 344 files differed in bytes: reports, logs and plots that embed timestamps or paths, and three scientific outputs whose tools are not order-stable — the deduplicated BAMs (Picard MarkDuplicates flagged a different mate as the duplicate in 6 of 38,486 duplicate pairs of WT_REP1, with identical totals: equal-score ties follow STAR's multithreaded output order), StringTie FPKM in the sixth decimal place, and the row order of RSeQC inner_distance.txt. Expect nfclaw verify --strict to fail on such a replay; compare the quantification tables instead.

• fastqrepair 1.1.1 bundled test — The original upstream fixture completes nine tasks but filters its damaged pair after gzip recovery produces an empty mate; its empty sample is also excluded. Its surviving single-end sample has 24 complete valid reads and two malformed records. For this exact pin, a bare nfclaw run fastqrepair --demo stages all three original rows from immutable test-data commit f72c633f2547fdfaf475c7f05423437a7eac0bbf and adds paired_control: 646 genuine pairs from immutable commit 800330ec8f47c93caabe8157e20dd23e243de4e1. This makes paired repair part of the managed demo; it does not claim to recover the excluded data. The generated CSV and row roles are recorded and replayed. Explicit input/test-data selectors, extra profiles or configs disable this fixture; explicit num_splits values win. Inspect discarded-sample warnings, executed tasks and repair reports alongside pipeline status. Independent reordered/orphan controls are still needed to assess pairing errors beyond this matched control.

  The managed demo and fresh replay each completed 22 native Docker tasks, including BBMap, with matching recorded environments. Both preserved all 646 paired read identities, full headers, sequences and mate orientation, zero singletons, and the original 24 valid single-end records. BBMap 39.18's default quality validation sets ambiguous-base qualities to Phred 0: exactly 211 N positions per mate changed from Phred 2 to 0; every other quality was unchanged. The pre-BBMap gathered reads retained the original qualities. An independent tiny image control confirmed this behavior; raw paired quality identity is therefore not claimed. See the run/replay assessment.

• createpanelrefs 1.0.0 nonempty antitarget control — Its bundled whole-genome CNVkit configuration deliberately has no antitarget bins. A separate controlled hybrid fixture and fresh replay each completed all three native Docker tasks, including CNVkit 0.9.12. It used a synthetic 1 Mb reference and two normal BAMs with independently checked uniform 1× and 2× aligned-base coverage. Explicit access bounds and bin sizes produced two target and three antitarget bins; all coverage coordinates/depths and the five-bin pooled reference passed, and all seven scientific files matched byte-for-byte in replay. Hybrid mode alone is insufficient: without --access, this CNVkit version guesses bounds starting at 150,000, which excluded these short target coordinates. The tested access bytes are embedded in the recorded config and checked in each task. This validates synthetic bin/count mechanics, not CNV calling accuracy or real-cohort behavior. See the assessment.

• proteinfold 2.0.0 --demo — The bundled test profile sets stubRun = true: every task runs its stub block, so the run completes in minutes with empty placeholder structures, metrics and reports. nfclaw run announces this before launch and nfclaw status reports the success with a stub run: caveat (also recorded in run_manifest.json as stub_profiles and carried by a replay). A --config that sets stubRun = true is reported the same way and recorded as stub_configs. A real prediction needs a mode's databases (see reference.md); the demo only checks the workflow's wiring and publication. circdna's non-default test_AA profile is a stub run too.

• pathogensurveillance 1.1.0 --demo on an IPv6-only host — Its one test sample is the NCBI accession SRR18850314, which sra-tools cannot fetch there, and BBMAP_SENDSKETCH needs the JVM's IPv6 preference. Download the run's two ENA FastQ files (check their MD5 sums against the ENA portal) and pass a sheet with sample_id,description,path,path_2,sequence_type,enabled,color_by,report_group_ids,ploidy,taxon to --input alongside --demo; keep taxon, which color_by refers to. ENA serves this run with simplified quality scores (only ? = Q30 and $ = Q3), from which SPAdes cannot infer the encoding (Failed to determine offset!); the pipeline then ignores the assembly after four attempts and reports success with caveats. The $ scores show Phred+33, so add process { withName: 'SPADES' { ext.args = '--phred-offset 33' } }. With these, the bundled test completed without caveats on Nextflow 25.04.6: a 2.07 Mb SPAdes assembly in 472 scaffolds, and sendsketch's top hit Limosilactobacillus at 98.4% ANI, as the test sheet expects. This establishes execution, not the accuracy of its phylogenetic placement.

• lsmquant 1.0.3 --demo without an NVIDIA GPU — The test profile gives NUMORPH_3DUNET ext.use_gpu = true and accelerator = 1, and the module calls nvidia-smi unconditionally, so on a host without a GPU the task fails with exit status 127 (nvidia-smi: command not found). Adding --nuclei-quantification false skips that step: intensity adjustment, alignment and stitching then completed on Nextflow 25.10.4. Nuclei quantification itself needs a GPU: on a GPU host add the release's gpu profile, which passes the GPU to Docker with --gpus all (nfclaw run lsmquant --demo -profile docker,gpu …; nfclaw composes test,docker,gpu).

• fetchngs — If accessions lack ENA FTP links, the pipeline falls back to SRATOOLS_PREFETCH (requiring NCBI SRA Cloud access). In network-restricted environments, run metadata-only via --skip_fastq_download. (Accepts .csv, .tsv, or .txt accession lists at pinned 1.13.0 and dev).

  • Bundled test on an IPv6-only host: three of the test accessions (SRR12848126, SRR14593545, SRR14709033) have no ENA FastQ files and take the sra-tools route, which cannot connect there: SRATOOLS_PREFETCH fails with cannot resolve remote location. With --demo and an --input list of the other seven test accessions (including the GEO GSM4907283 and GSE214215), the run completed on Nextflow 25.10.4. For a fetchngs → taxprofiler chain also leave out SRR9984183: ENA lists it as Illumina HiSeq 2500, but it holds a submitted metagenome bin (58 low-quality reads of ~34 kb), from which fastp keeps nothing, and taxprofiler's FASTQC_PROCESSED then fails on the empty file. Give the taxprofiler stage the restricted profilers below and "perform_shortread_redundancyestimation": false (its test profile turns Nonpareil on; its curve fit fails on small real samples). With these, the chain completed with verified lineage.

• coproid — Requires two samplesheets: --input (FASTQ sheet) and a separate required --genome_sheet. Each row in --genome_sheet requires genome_name,taxid,genome_size plus exactly one of igenome or fasta (mutually exclusive oneOf schema). --kraken2_db is mandatory and must point to a valid Kraken2 database.

• crisprseq — The samplesheet reference column expects a raw DNA sequence string (pattern ^[ACTGNactgn]+$), not a FASTA file path.

• circdna — Two parameters are required by schema for non-demo runs: --input_format (FASTQ or BAM) and --circle_identifier (one or more of circle_map_realign, circle_map_repeats, circle_finder, circexplorer2, ampliconarchitect).

• metapep — The --demo profile runs DOWNLOAD_PROTEINS, which fetches sequences from NCBI Entrez (download_proteins_entrez.py --email $NCBI_EMAIL) reading a Nextflow secret named NCBI_EMAIL. Set the secret before the run: nextflow secrets set NCBI_EMAIL you@example.com.

• seqsubmit 1.0.0 --demo needs an ENA Webin account — The bundled test profile sets test_upload = true, so genome_upload validates the submission without uploading it, but it still queries ENA for the study's run metadata with Webin credentials. nextflow.config passes the Nextflow secrets ENA_WEBIN and ENA_WEBIN_PASSWORD to every task through its env scope, so without them Nextflow only warns that both variables evaluate to an empty value: the FASTA validation, classification and coverage tasks complete, and CREATE_MANIFESTS then fails with CredentialsNotSetError: Credentials for ENA are not set (reproduced on Nextflow 25.10.4). Set both secrets for a Webin account (nextflow secrets set ENA_WEBIN Webin-NNNNN, then nextflow secrets set ENA_WEBIN_PASSWORD …) before any run, including --demo. Without an account, a stub run checks the rest of the workflow: pass --config a file containing stubRun = true, and every task runs its stub block, none contacting ENA (nfclaw status then reports a stub run: caveat). It stops before the submission step, whose stub inputs match no genome.

• epitopeprediction 3.1.1 queries Ensembl BioMart while it runs; the pinned 3.2.0 does not — 3.2.0 replaced the epytope/BioMart variant path with bcftools, Ensembl VEP and pVACtools (nf-core/epitopeprediction#362). Its test profile annotates with a 15 MB GRCh38 VEP cache subset that it downloads with the test data. For your own VCFs, give --vep_cache and --ref_fasta, or set --vep_download_cache, which fetches about 20 GB from the Ensembl FTP site. In 3.1.1 (--pipeline-version 3.1.1), EPYTOPE_VARIANT_PREDICTION looks up the proteins of the variants' transcripts through epytope's MartsAdapter. When BioMart is overloaded it answers with its "Service unavailable" HTML page and HTTP status 200. epytope retries only error statuses, so the page reaches pandas and the task fails with ValueError: Length mismatch: Expected axis has 1 elements, new values have 3 elements in MartsAdapter.get_dataset_attributes. On 2026-10-15, 24 of 600 requests to grch37.ensembl.org/biomart/martservice over 50 minutes got that page, and two of the three 3.1.1 --demo runs that reached this step lost a task this way. Upstream tracked this as nf-core/epitopeprediction#337. For 3.1.1, retry the task with process { withName: 'EPYTOPE_VARIANT_PREDICTION' { errorStrategy = 'retry'; maxRetries = 3 } } passed as --config, or re-run with --resume. A host that cannot reach Ensembl cannot run that step.

• oncoanalyser 3.0.0 --demo needs about 130 GB for references — The bundled test downloads about 53 GB of reference tarballs (HMF pipeline resources, GRIDSS, STAR and BWA-MEM2 indexes) plus the 3.2 GB GRCh38 FASTA and a 5.6 GB BWA image, and it extracts the tarballs inside the work directory, so a run holds both copies. Upstream's CI extracts them once instead and passes them through tests/cached_refs.config and five NXF_REF_* variables. To run it that way, download the FASTA, its .fai, .dict and .img, and extract each tarball into its own directory without its top-level directory (tar -xzf … --strip-components 1, as the pipeline's CUSTOM_EXTRACTTARBALL does; 77 GB in all). Then run nfclaw run oncoanalyser --demo --outdir results/oncoanalyser --config <repo>/pipelines/oncoanalyser/upstream/tests/cached_refs.config with --nxf-env NXF_REF_DATA_PATH=<dir>, plus NXF_REF_BWAMEM2_INDEX, NXF_REF_GRIDSS_INDEX, NXF_REF_STAR_INDEX and NXF_REF_HMF_DATA, each set to its extracted directory. On Nextflow 25.10.4 this ran the 42 tasks upstream's test expects with cached references. It was resumed once, after the sweep's disk guard stopped it. All 473 output paths and all 151 stable-content checksums in the release's nf-test snapshot (tests/profile_test.nf.test.snap) matched. On 2026-11-08, through nfclaw 1.0.0, the same route ran the 42 tasks in 40 minutes at the first attempt, replayed with an identical environment and gave nfclaw verify exit 0. Its STAR_ALIGN peaked at 29.7 GiB, as in 2026-10, so leave it more than that free.

• createtaxdb — Assign each sample a non-numeric id (e.g. seq1, chr1). Purely numeric strings ("1") are coerced to integers by nf-schema, violating string constraints.

• genomeassembler — Set at least one of --ont true or --hifi true (the pipeline aborts with At least one of params.ont, params.hifi needs to be true.).

• funcscan — Pin --nxf-ver 25.10.4 (declares !>=25.10.4). Demo profiles run all three screenings (AMP, ARG, CAZyme), which download multi-gigabyte models; pre-supply databases via --amp_ampcombi_db or disable heavy steps via --run_cazyme_screening false and --arg_skip_deeparg true. If DEEPARG_DOWNLOADDATA makes no progress until its time limit, see Docker's MTU. The test profile also limits each task to an hour, and a slow download can reach it while still progressing: RUNDBCAN_DATABASE (dbCAN-sub.hmm alone is about 5 GB) stopped with Process exceeded running time limit (1h) on one server run and took 11 minutes when --resume ran it again. A replay downloads it again: one on 2026-11-07 failed in that task with DatabaseDownloadError for dbCAN-sub.hmm, Read timed out from dbcan.s3.us-west-2.amazonaws.com. Give a downloaded copy with --cazyme-dbcan-db, or skip dbCAN with --cazyme-skip-dbcan.

• sarek 3.11.0 (sarek 3.10.0, sarek 3.9.0) — Upstream examples for cache/index-only runs show --build_only_index --input false. In raw Nextflow, this causes an nf-schema validation error because input is declared as a string path. In nfclaw run sarek --input false, nfclaw intercepts --input false and leaves input unset in params.json, allowing index-only runs (--build_only_index true --download_cache true) to succeed cleanly without schema errors. For raw Nextflow runs, omit --input instead of passing false.

• sarek 3.11.0 (sarek 3.10.0, sarek 3.9.0) — Do not use --config pipelines/sarek/upstream/conf/test.config to convert an analysis run into a test run with a custom samplesheet. Extra configs load after initial profile setup, causing validation of default S3 iGenomes paths. Run with -profile test,docker instead: nfclaw run sarek --input samplesheet_sarek.csv --outdir results -profile test,docker. For bundled test data, use nfclaw run sarek --demo --outdir results.

• ampliseq (measured at 3.0.0; the pinned 3.1.0 has the same test limits and parameters) — The bundled test profile sets its own process.resourceLimits (4 CPUs, 15 GB, 1 h); the release has no --max_memory/--max_cpus parameters (nfclaw rejects them as unknown), so cap a real run with --limit-cpus/--limit-memory/--limit-time. Its --demo completed all 130 tasks without a retry on Nextflow 26.04.3 with Docker (Linux amd64); QIIME2_EXPORT_RELTAX peaked at 1.2 GB of RSS. This establishes execution of the bundled test, not taxonomic accuracy. 3.0.0 also removed the legacy sampleID,forwardReads samplesheet format (use sample,fastq_1[,fastq_2]) and renamed --FW-primer/--RV-primer to --primer-fwd/--primer-rev; nfclaw rejects the old names.

  • Reproducibility: DADA2 taxonomy ties and QIIME2 rarefaction can change results on fresh runs despite --seed. See verified findings and workarounds.

• sopa 1.0.2 --demo cannot be replayed faithfully — Its test samplesheet (tests/samplesheet.csv) names https://github.com/nf-core/sopa/blob/dev/tests/samplesheet.csv as the sample's data_path: a GitHub HTML page, not a raw file. Nextflow stages that page as a task input, and the page's bytes change on every download (213,168 bytes in one run and 213,166 in its replay on 1.0.1; the digests differed again on 1.0.2). A replay of --demo therefore fails its environment check with remote input … changed, as it should: the replay did consume other bytes. Pass a stable data_path (a local file or a raw.githubusercontent.com URL at a commit) for a run that replays exactly.

• epitopeprediction 3.1.1 replays meet the same BioMart replies — The BioMart lookups above happen in the replay too, so a replay can fail where its original succeeded. Pass the retry config as --config when the run starts: the bundle keeps a copy of it, and the replay retries the same way. A replay with it reproduced the run's environment exactly.

• reportho 1.1.0 can drop a query's results silently, depending on live database answers — Its tasks query OMA, Ensembl, UniProt and RefSeq while running, so two runs of the same input get slightly different answers. MERGE_IDS clusters only taxon groups holding two or more sequences (DIAMOND_CLUSTER) and then joins those clusters with the single-sequence groups (MERGE_ALL); a plain join drops a query missing on either side. A query is therefore dropped unless its answer holds both kinds of taxon group, at least one with two or more sequences and at least one with a single sequence. Such a query gets no merge/, score/ or report/ outputs, while the pipeline reports "completed successfully" and nfclaw records no caveat. Measured on denbi (2026-11-04, --demo, main e88ce4a): a fresh run with 98 BicD2 ortholog IDs had no DIAMOND_CLUSTER (BicD2) task and no BicD2 report, while its replay with 96 IDs clustered one group and reported BicD2. A resumed run lost HBB the same way. On 2026-11-08 (main 8289263) a run resumed once published no BicD2 results, and its replay did: 25 files in 4 directories. Ensembl's /lookup/id also sometimes returns a non-JSON reply, failing FETCH_ENSEMBL_SEQUENCES; retry, or --resume. Check that every query has a report/<id>/ folder. Between two runs, expect nfclaw verify to show changed bytes, and missing or extra files for a dropped query.

• reportho 1.1.0 can also fail on live database answers — On 2026-11-05, 3 of 7 launches of its --demo failed in FETCH_ENSEMBL_SEQUENCES; the two read in full ended in requests.exceptions.JSONDecodeError, because the script parses whatever Ensembl answers as JSON. --resume, or a further replay into a fresh directory, then passed. On 2026-11-07 a run and its --resume a minute later both failed the same way in another task, FETCH_ENSEMBL_IDMAP, which reads https://rest.ensembl.org/info/species; on 2026-11-08 the same failure passed on its --resume. Of three requests to that address from the same host on 2026-11-08, the first got an HTTP 503 HTML page (No server is available to handle this request) after 31 s, the next two JSON after 20 s and 0.2 s. These tasks then exit 1, which the release's retry rules (conf/base.config) do not retry. One replay also failed with DIAMOND_CLUSTER past its 3-minute limit (upstream's conf/modules.config, so in every profile) although DIAMOND's own output had finished. A task's time limit counts everything from its start, including an image download and any wait on a busy disk. This one coincided with another pipeline's first tasks spending over four minutes outside their scripts, which is how an image download looks in a trace, on the same volume: other DIAMOND_CLUSTER tasks went from about 10 to about 50 seconds as that began, and the two that timed out started about a minute later and stopped exactly 3 minutes after submission. The link is inferred from the traces and unproven: during one 1.22 GB image pull on an idle host, a small container took about 1 s longer at the median and 18 s at worst, though that was not several pipelines pulling at once. It did not recur; --resume continues the run.

• Outputs that differ between a run and its replay by design — nfclaw verify compares outputs by path. Some releases name or include outputs nondeterministically, so a faithful replay (identical images, plugins, source and inputs) still reports files as missing or extra:

  • longraredisease 1.0.0 names its MultiQC report after the output directory (<outdir basename>-pipeline-report_multiqc_report.html), so a replay into <outdir>.replay names it differently.
  • createtaxdb 3.1.0: kraken2-build gives the files it adds to library/added/ random names. nfclaw verify pairs exactly these names one to one within their directory, lists them under renamed and compares their bytes, so a faithful replay no longer reports them as missing and extra.
  • hicar 1.0.0: its MultiQC report has one of samtools' stats, flagstat and idxstats sections, whichever finished first. The workflow keeps one MultiQC input per file base name (groupTuple on baseName, then the first file, in workflows/hicar.nf), and <bam>.stats, <bam>.flagstat and <bam>.idxstats share one. A report with stats has its alignment plots: on 2026-11-07, 8 files that a replay with flagstat lacked, which had 1 of its own. One .hic file under cooler/MAPS/ was also published under different names across runs.
  • nanoseq 3.1.0: NanoPlot names its log by the minute it ran (NanoPlot_<date>_<time>.log). MULTIQC also receives only one FastQC file, whichever arrives first, because the workflow does not collect that channel. A report that received sample_R1_fastqc.zip has FastQC sections and 44 more files under multiqc_data and multiqc_plots than one that received sample_R2_fastqc.html. The FastQC results themselves were identical across both runs.
  • nanostring 1.3.3: MultiQC exported four plots in one run and not in its replay (2026-11-05); on 2026-11-07 a run exported 8 of its 15 plots and its replay 13.
  • hlatyping 2.2.0: on 2026-11-07 a replay exported 22 of 30 plots, and lacked the 8 files its run had.
  • coproid 2.0.1: MultiQC exported 25 plots in one run and not in its replay, on a loaded host on 2026-11-05; an earlier replay had them all, and so did one on an idle host a day later and both of a later pair.
  • Why these can differ: they run MultiQC 1.32 or 1.33 with export_plots: true, which gives up on a batch of exports after export_plots_timeout (30 seconds by default), logs Batch plot export timed out after 30s. Only N of M plots were exported and still exits 0. On test data, a 1-second limit gave 0 of 36 plots and a 300-second limit gave all 36. On 2026-11-07 the task logs were kept, and the counts match: hlatyping's replay left 8 plots unexported and lacks 8 files, while its run logged no timeout; nanostring's run left 7 and its replay 2, and the replay has 5 files more. coproid's loss rests on the same mechanism without its log: on an idle host its whole MultiQC task took 27.4 s and logged no timeout. To allow longer, pass a file holding export_plots_timeout: 300 with the pipeline's --multiqc-config. A later config overrides the limit: this was tested in the 1.33 image, in the order the module passes them, and not run through a whole pipeline.
  • mag 5.5.0: Prokka publishes temporary protein files named by its process ID (….AMR.tmp.<pid>.faa).
  • rnafusion 4.1.3: FusionInspector publishes fi_workdir/pipeliner.<pid>.cmds.
  • rnasplice 1.0.4 and 2.0.0: MISO names its batch logs by the time they were written.
  • spatialaxe 1.0.1: Proseg's segmentation differed between runs on the tiny test data (9 and 10 cells, 2026-11-05). At 9 cells Xenium Ranger wrote analysis/note.txt, There was insufficient data to run the secondary analysis, where the 10-cell run has 26 clustering, differential expression, PCA and UMAP results.
  • A resumed run's outdir keeps the launch-stamped pipeline_info reports of its earlier attempts; its replay reproduces the last attempt only. Inspect such differences with a format-aware comparison; they are not reported as environment differences.

AmpliSeq 3.0.0: taxonomy and rarefaction reproducibility

Scope: historical AmpliSeq 3.0.0 commit 992e6df34065067185b5e34665b5861b6827981d; source inspection and a small C++ tie-breaking check, not a new end-to-end demo replay. The pinned 3.1.0 was inspected on 2026-11-06: its taxonomy module still seeds only R's generator and its core-metrics module still omits --p-random-seed, so the same mechanisms apply there.

• DADA2 claim confirmed, with precise wording. The 1.38.0 and 1.40.0 release archives contain identical src/taxonomy.cpp: get_best_genus() seeds std::mt19937 from std::random_device and randomly chooses among equal scores. This is entropy-seeded randomness, without a user-controllable seed. The function is called for both the full query and bootstrap replicates; bootstrap subsampling separately uses R's Rcpp::runif. AmpliSeq's taxonomy module calls set.seed() using params.seed (default 100), but that controls only R's RNG. Neither changing --seed nor using one thread controls the C++ ties.

• QIIME2 claim depends on version and action. The 2024.10 rarefy interface has no seed parameter; feature-table rarefy in 2025.4 exposes random_seed. AmpliSeq 3.0.0 specifies QIIME2 2026.4, whose core-metrics interface also exposes it and forwards it to rarefy. The AmpliSeq core module omits --p-random-seed and does not consume params.seed or ext.args: its sampling remains uncontrolled by AmpliSeq's --seed. Its separate alpha-rarefaction action has no seed argument and repeatedly calls rarefy without one.

• Implications and limits. Fresh executions can change taxonomy labels, bootstrap confidence and assignments near --dada-min-boot, and rarefied counts, diversity metrics and downstream analyses. These mechanisms do not imply changed ASV inference, that every output must differ, or that GTDB ties are common: their frequency and scientific impact depend on the inputs/reference and were not established here. Previously reported demo measurements were 3 of 241 changed labels, 34 changed bootstrap confidences and 4–6 label differences per pair across four isolated reruns (4 threads and FALSE). ASV sequences/counts, DADA2 statistics and QIIME2 taxonomy matched; the filtered table retained 38 ASVs/counts with changed row order. These observations were not independently repeated in this audit and are not general guarantees.

• Workarounds. Retain original taxonomy and rarefied tables for downstream reuse (the core task's diversity_core/rarefied_table.qza stays in its work directory, outside the published module outputs); --resume can reuse valid cached tasks but does not make a fresh replay deterministic. Alternatively, use --skip-dada-taxonomy true with a suitable --qiime-ref-taxonomy classifier (the prior demo's repeatability does not establish universal accuracy or determinism). For controlled diversity calculations, skip pipeline diversity with --skip-diversity-indices true, retain the matching feature table, tree and metadata, and run QIIME2 2026.4 core-metrics-phylogenetic separately with --p-random-seed 100. Skip the independent curves with --skip-alpha-rarefaction true if needed. Seeds constrain sampling with the same inputs/software; they do not guarantee byte-identical QZA/QZV archives or all downstream statistics. Compare exported scientific values by ASV/sample; nfclaw verify checks structure and --strict additionally checks bytes, neither guarantees analytical accuracy.

• nf-claw behavior. The runner currently announces taxonomy/core-metrics variability before launch through runner/reproducibility.py; it adds no seed correction. Detection scans module text and explicit skip parameters, without checking tool versions or resolving config/profile overrides; it does not detect alpha-rarefaction separately. An advisory therefore identifies a potential source of variability, not proof that a particular run changed.