End-to-end validation

Every pipeline pinned at the time of the validation was attempted through nfclaw. Of 91 bundled tests, 90 reached success, including proteinfold's stub-only test; seqsubmit could not complete without ENA credentials. nfclaw's own features were tested on the same host: replay, verification, resume, chains, sample-sheet generation, stop handling, versions and status reporting. This page records how the runs were done, what passed, and what each result does and does not show.

The versions and observations below describe the validation of 2026-10-14 to 2026-10-16; they are not repeated after every change to the repository. Since then:

• releases bumped and pipelines added, and run the same way, are listed under Releases bumped or added after the sweep;
• the final pin sweep of 2026-11-05 to 2026-11-06 ran every pin again, with its replay;
• the final-code sweep of 2026-11-07 to 2026-11-08 ran them once more on the audited code.

The automated updater itself publishes a release only after its preview, the tests, the drift checks and a real run of its bundled test on a hosted runner pass (publication policy). For the current pin and its requirements, read pipelines/<name>/skill.md.

Navigation

Environment • Method • Pipelines • Fresh single-attempt runs • Releases bumped or added after the sweep • Final pin sweep • Final-code sweep • nfclaw features • Chains • Provenance bundles • Defects found and fixed • Test suites and CI • Limits

Environment

Method

Each pipeline ran its bundled test profile with nfclaw run <name> --demo. The engine was pinned to the version its skill.md names, and provenance was written as usual. A run counted only when nfclaw status reported success. Its last execution trace was then checked for failed or ignored tasks, and its completion caveats were read.

Some bundled tests could not run as shipped. For each one, the cause was found first: an upstream defect, test data that changed after the release, or a property of this host. The run was then repeated with the smallest documented change. The table lists every such change, and known-issues explains it. This host is IPv6-only, so the later part of the sweep used the documented host-network config (--config host-net.config). That came after containers on the default bridge failed to reach IPv4-only servers.

A resumed run (--resume) continued an attempt that had stopped. The sweep's disk guard stopped some attempts; others stopped on a failure the table explains. The trace of a resumed run counts both cached and newly completed tasks.

Pipelines

90 of the 91 bundled tests reached success; proteinfold executed only stubs, not protein prediction. seqsubmit could not complete: its test queries ENA with a Webin account. "Attempts" counts every launch, including runs the sweep stopped and resumed. The notes give the reason for each extra attempt and every setting beyond --demo. "Host network" means the documented host-network config. Three releases needed more than one run each: marsseq (a reference build, then the analysis on those references), oncoanalyser (references prepared as upstream's CI prepares them) and rnafusion (the test analysis on upstream's hosted references, split over three runs that together cover every tool).

The 21 releases that passed only on a --resume here were also run again from scratch, each in one uninterrupted run; see Fresh single-attempt runs. Some runs were also checked against independent expectations. oncoanalyser's outputs matched every path and checksum in the release's own nf-test snapshot. STAR-Fusion reproduced the fusions and breakpoints that its tutorial documents for the tutorial reads. rnafusion's three fusion callers called 12 fusions in common on the test reads; FusionCatcher called all 14 of Arriba's.

Fresh single-attempt runs

In the sweep above, 21 releases reached success only on a --resume. The sweep's disk guard, a shared-disk outage or a transient failure had stopped an earlier attempt. Each of the 21 was run again from scratch into a new outdir, with the same settings and no --resume, so that one uninterrupted run covers the whole test. All of them succeeded. "Launches" counts the fresh launches it took. Extra launches were stopped by a full shared disk or by the guard, or needed the change in the notes; each started over in a new outdir.

Two of these needed a setting the first sweep had not:

• epitopeprediction: a retry for EPYTOPE_VARIANT_PREDICTION, whose BioMart lookups failed 16 times on Ensembl's HTTP-200 outage page (run note).
• genomeqc: its own --famdb-library offline option, once www.dfam.org stopped resolving on this host (IPv6 host).

proteinfamilies ran as shipped once its container images were present. Its 2–3 minute test limits include the image pull (known issue).

Releases bumped or added after the sweep

Each release below was validated separately on the same host, in one uninterrupted nfclaw run <name> --demo on the engine its skill.md names: before its pin was updated or its pipeline was added, or, for a release the automated updater published itself (rnastructurome 1.1.0, genomeassembler 2.0.1 and those dated 2026-11-05), after it. This execution evidence is additional to the automated updater's gates.

epitopeprediction 3.2.0 replaced 3.1.1's epytope and Ensembl BioMart variant path with bcftools, Ensembl VEP and pVACtools, and its test annotates with a bundled VEP cache subset, so the BioMart failures above no longer apply (run note). It ran twice, on the bump and again once the provenance bundles change had merged. Both runs matched the release's own -profile test nf-test: 21 succeeded tasks, every software version, all 78 stable output paths, all 17 checksummed files, and the missense, in-frame deletion and frameshift peptides its assertions require. The second run's bundle pinned all 17 containerized processes to the digests of their 9 images and recorded 4 remote inputs by SHA-256. Its replay from provenance/commands.sh succeeded with an identical execution environment. nfclaw verify exited 0: 36 files identical, 39 changed only in MultiQC and pipeline_info reports, 1 launch-time-named file paired, 0 missing and 0 extra. Every analytical output (variant_peptides/, variant_fasta/, mhcnuggets/ and predictions/) was byte-identical across both runs and the replay.

The five releases of 2026-10-22 were checked the same way: each run against its release's own -profile test nf-test snapshot (tests/default.nf.test.snap), then a replay from provenance/commands.sh and nfclaw verify. Every run reported every software version its snapshot records, produced every stable output path it lists, and matched all of its checksummed files: 15 for magmap, 111 for rnasplice, 1 for bacmodel, 9 for plasmodiumdrugres and the 7 of provenancereport's 8 that its tests/.nftignore does not exclude. Every containerized process was pinned to its image digest, and every replay succeeded with an identical execution environment and again matched all of those checksums.

• magmap: 48 of the changed files are MultiQC and pipeline_info reports, 2 are gzip files whose decompressed content is identical, and 3 are BBMap logs that differ only in their timings.
• bacmodel and plasmodiumdrugres: only reports changed, apart from bacmodel's Prokka log.
• provenancereport: besides reports, the files that changed are the ones its own tests/.nftignore marks as unstable: the published resolved nextflow.config, artifacts.csv, notebook.csv, reports.csv and ro-crate-metadata.json, which record the output path and launch time. The eighth snapshot checksum is that nextflow.config; it also differs under upstream's nf-test, whose two tests passed on the same host.
• rnastructurome 1.1.0 (2026-10-23) was checked the same way. Its run and its replay matched all 56 checksummed files and every one of the 108 stable paths of its -profile test snapshot, and every software version but R2DT's, which upstream's test leaves out because R2DT runs only with containers. The 15 changed files are ones its tests/.nftignore marks as unstable: reports, FastQC archives, STAR logs and R2DT's log.
• genomeassembler 2.0.1 (2026-10-23) was checked the same way. Its run and its replay matched all 113 checksummed files, all 427 stable paths and every software version of its -profile test snapshot, which also records the 88 succeeded tasks. Of the 78 changed files, 25 are reports and metadata and 8 are gzip files whose decompressed content is identical. The other 45 are all files its tests/.nftignore excludes from that comparison: hifiasm's binary indexes and logs, LongStitch's scaffolds, Flye's logs, read-trimming and Hi-C logs and metrics, the genomeqc samplesheet (it records absolute output paths) and the test_ont_flye sample's assembly. Flye assembled that sample's ONT test reads into 4 contigs (2,083,676 bp) in the run and 3 (2,068,682 bp) in the replay, so its graph, assembly info and Liftoff annotation differ too.
• stableexpression 1.1.0 (2026-10-25) was added by hand: the weekly discovery rejects it because its default preview never exits (known issues). Its run and its replay matched all 27 checksummed files and all 72 stable paths of its -profile test snapshot, and the known answers that test asserts: reporting/all_genes_summary.csv holds 27,849 genes in 35 columns, with the expected stability scores for ENSRNA050004025, PRUPE_1G169400 and PRUPE_1G195800. The 5 changed files are the launch-time pipeline_info reports and the MultiQC report, which its tests/.nftignore excludes.
• rnasplice: as with 1.0.4, MISO names its logs by the clock (the 20 missing and 20 extra files, so nfclaw verify exits 1) and samples by MCMC, so 1,223 of the changed files are MISO's data. Salmon's multi-threaded quantification also differs slightly between runs (948 of 5,895 transcripts of ERR188383, by at most 4.4 of 46,570 reads), in both its salmon and star_salmon routes, and so do the SUPPA, tximport, DEXSeq DTU and IsoformSwitchAnalyzeR results computed from it. The other changed files are reports, PDFs, FastQC archives and logs that embed dates or timings. The snapshot's 111 checksummed files, its stable outputs, were identical in both runs. 2.0.0's new LeafCutter and RegTools steps belong to upstream's separate leafcutter test, not to -profile test, and were not run.
• createpanelrefs 1.1.0 (2026-11-03) was bumped by hand. The updater ran the same bump that day and refused it at its preview gate: like 1.0.0, the release's default preview prints "Pipeline completed successfully" and never exits (known issues). A preview of its commit b9d0e44 on 26.04.0 reproduced that timeout with the documented evidence (known-timeout). The bundled test (CNVkit on two BAMs) completed: CNVKIT_BATCH and MultiQC. Its outputs matched the paths of the release's closest -profile test snapshot (--cnvkit_method hybrid --cnvkit_targets), apart from the reference BEDs named after their source (genome.*.bed instead of multi_intervals.*.bed), and the versions it records (cnvkit 0.9.14, samtools 1.24). As with 1.0.0, the antitarget coverage tables hold only their header. The replay from provenance/commands.sh succeeded with an identical environment: nfclaw verify reported 13 identical, 9 changed, 0 missing, 0 extra, and the panel of normals, both target coverage tables and the reference BEDs were byte-identical. The changed files were run metadata and MultiQC reports.
• The six releases of 2026-11-05 ran for the final audit, with --limit-cpus 8 --limit-memory 24.GB, and were not compared with their nf-test snapshots. The replays of ampliseq, bacmodel, detaxizer and pixelator succeeded with an identical execution environment, and nfclaw verify reported 0 missing and 0 extra: 1,288 identical and 322 changed, 15 and 5, 80 and 42, 21 and 97. The volume was too full to replay sarek and sopa then, and to finish bacass 2.7.0 and variantbenchmarking 1.6.0, which a disk watchdog stopped after 9 and 53 tasks. With room made, all four ran with their replays in the final pin sweep.

Final pin sweep

For the final audit, 90 of the 95 pins ran again from 2026-11-05 to 2026-11-06, at one commit of the runner (a23a092): the bundled test, its replay from provenance/commands.sh, then nfclaw verify. coproid's last rerun ran on 85313d4 and marsseq's on ebc2a96, which has the runner of the last code change, and their rows say so. Three jobs ran at a time, each on the engine its skill.md names and the host network, with --limit-cpus 8 --limit-memory 20.GB on every engine that applies them (77 of the 91 jobs: marsseq has one job to build its references and one to analyse). funcscan and marsseq ran alone at the end. The per-job record is audit/evidence/2026-11-05-pin-sweep.json.

This repeats execution and replay. It does not compare outputs with nf-test snapshots or assess their science, except that the tumourevo and hgtseq demos also passed their output contracts. Runs that came after a23a092 and what they cover are in the audit.

The other five pins: ampliseq, bacmodel, detaxizer and pixelator ran separately for the same audit, as listed above. oncoanalyser did not run: its references were deleted on 2026-10-16, and its STAR_ALIGN peaked just above the memory this host had available then. It ran on 2026-11-08, after the final-code sweep. variantbenchmarking moved to 1.6.1 after the sweep began, so both versions have a row.

Of the 91 jobs, 90 runs succeeded; the other is seqsubmit. Of their replays, 88 succeeded, and the environment of 84 was identical to the recording. nfclaw verify exited 0 for 81 and 1 for 7. On their first attempt with no retry or rerun, 69 jobs ended with success, an identical environment and nfclaw verify 0, and 3 of those carry caveats. No job's final attempt left a container behind.

What did not come out clean, by kind:

• The run did not finish. seqsubmit needs ENA Webin credentials.
• marsseq ran without STAR's index. That index takes more memory than this host has: about 48 GiB before exit 137 under a 48.GB cap and 46.2 GiB on the retry that completed, in October, against about 30 GiB available. The sweep's guards stopped the build at 2.6 GB of memory and then at 2.8 GB of disk, and with more room a 26 GB cap killed STAR with exit 137. The bundled analysis reads that index only with --velocity, which its test profile leaves off, though it checks that the path exists, so the build ran without --velocity and the analysis named an empty directory as --star-index. Both ran, replayed with an identical environment and exited 0 in nfclaw verify. The analysis has the 75 identical and 67 changed files of its earlier row, where the index was real. The build with STAR's index, and the analysis with --velocity, ran on 2026-11-08, after the final-code sweep.
• The replay did not complete. sopa's test names a GitHub HTML page as its data, so the environment check refuses the replay. reportho's tasks query live databases: 3 of 7 launches failed on Ensembl's answers. Two replays failed, one of them because two DIAMOND_CLUSTER tasks hit upstream's 3-minute limit. The traces suggest a first image download slowing the shared volume, but one pull on an idle host did not reproduce that, so the link is unproven. A third, from the kept bundle, passed: environment identical, 30 identical and 26 changed.
• The environment check stays unverified. mcmicro, circdna and raredisease run tasks with no container, and genomeqc's test profile leaves no task records. Their replays succeeded and nfclaw verify exits 0. The table of 2026-10-16 shows identical for all four. That fits runs from before these checks existed, inferred from the code rather than from running the old code again.
• nfclaw verify exits 1: a file exists on one side only. All of these replays have an identical environment. Their differences are listed: rnafusion, nanostring, longraredisease, mag, spatialaxe (9 and 10 cells), nanoseq and rnasplice. rnasplice's 20 missing and 20 extra files were not read, since the run directory was gone; the count matches 2.0.0's earlier row.
• A first attempt failed and a rerun passed. seqinspector, metatdenovo and drop stopped on a remote file reported as missing, and drop's left one container behind. coproid's failed downloading NCBI's taxonomy, and its rerun's replay, under load, lost 25 plot exports. A further rerun on an idle host, on 85313d4, kept them all and nfclaw verify exited 0. That fits MultiQC's 30-second limit on exports: the whole task took 27.4 s there. The link is inferred, because the logs of the replay that lost them were not kept. funcscan's RUNDBCAN_DATABASE hit the test profile's one-hour limit on a slow download and took 11 minutes on --resume. hgtseq, rerun with the references its contract needs, stopped once with Pipeline completed with errors and passed on --resume. No cause was read for that stop or for coproid's. reportho's run also needed its resume.
• Success with caveats. proteinfold's test is a stub run, so it computes nothing. cutandrun, hicar and mag succeeded with failures their own configuration ignores. The guide quotes the three failures: each tool stopped saying it had too little to work with.

Tasks in pathogensurveillance (DOWNLOAD_ASSEMBLIES, exit 1) and proteinfamilies (exit 137) failed and passed on Nextflow's retry, so these runs carry no caveat.

Final-code sweep

The final pin sweep ran on a23a092, before the audit's last runner changes. So the same harness ran every pin but oncoanalyser again, from 2026-11-07 01:59 to 2026-11-08 01:40 UTC, on 8d8c8b8, whose tree main has at bec857d: 95 jobs, with marsseq's two. Engines, limits and host configuration were those above, with --limit-cpus 8 --limit-memory 20.GB on the 81 jobs whose engine applies them. The per-job record is audit/evidence/2026-11-07-final-code-sweep.json. Every row's version and commit match the pins at 8d8c8b8.

• Runs. 93 of 95 succeeded. seqsubmit needs ENA Webin credentials. reportho's run and its --resume both failed in FETCH_ENSEMBL_IDMAP on an answer from Ensembl that is not JSON. hgtseq and stableexpression passed on their --resume: hgtseq after its intermittent Groovy exception, stableexpression after EBI's FTP server refused a connection.
• Replays. 91 of 93 succeeded. sopa's is refused, as before. funcscan's failed when the download of dbCAN's database from S3 timed out.
• Environment. Identical in 87, differs in sopa, and unverified in mcmicro, circdna, raredisease and genomeqc. Their bundles now record why, a process with no container or no task records, where at a23a092 none did.
• nfclaw verify. It exited 0 for 82 replays and 1 for 9: the seven of the final pin sweep, hicar and hlatyping.
• First attempts. 73 jobs ended with success, an identical environment, nfclaw verify 0 and no container left at their first attempt; 2 of those carry caveats (cutandrun, proteinfold).
• The disk guard stopped five jobs when the volume fell below 4 GB free, and reran each alone at the end: hadge, differentialabundance and lsmquant then came out clean, genomeqc and mcmicro unverified as above.
• No job left a container behind, and tumourevo's and hgtseq's demos passed their output contracts.
• marsseq built its references without --velocity in 3,395 s (8 tasks), and its analysis, with an empty --star-index, took 332 s (34 tasks). Both replayed with an identical environment and nfclaw verify 0, with the 11 identical and 3 changed, and the 75 identical and 67 changed files of their rows on ebc2a96.

What came out differently from the final pin sweep:

Logs of every MultiQC and failed task were kept this time. The plot-export timeout appears in three pipelines' logs: hlatyping's replay, nanostring's run and replay, and stableexpression's run and replay, which exported 64 and 60 of 93 plots and still verified with exit 0, so its published files hold none of them. coproid's run and replay logged none and verified with exit 0.

After the sweep, on 2026-11-08, three more jobs ran the same way on 8289263, which differs from 8d8c8b8 only in drugresponseeval's pin. They are under later_head and retries in the record; the sweep's rows stand.

reportho's first attempt failed again in FETCH_ENSEMBL_IDMAP on a reply that was not JSON. The 25 extra files are BicD2's merge/, score/ and report/ results: the run, though successful, had none, as reportho can drop a query.

Later on 2026-11-08, /dev/shm was cleared of leftovers from other work, and about 63 GB of the host's memory was available instead of about 31 GB. The two jobs this sweep had left out then ran the same way: oncoanalyser through the deployed nfclaw 1.0.0 (5d14ed4), and marsseq's reference build with --velocity, which builds STAR's index, then its analysis with --velocity, on 453d68e, whose runner is 1.0.0's. The per-job record is audit/evidence/2026-11-08-oncoanalyser-and-marsseq.json.

• oncoanalyser ran its bundled test with references extracted once and passed through upstream's tests/cached_refs.config, as its run note describes, in 40 minutes. Its STAR_ALIGN peaked at 29.7 GiB and each BWAMEM2_ALIGN at 17.4 GiB. nfclaw verify --require-environment exited 1 only because its samplesheet is a remote validation input, whose refetched bytes do not establish the bytes the run read.
• marsseq's build ran with --max-memory 48.GB --max-cpus 16, as its row documents. STAR peaked at 47.9 GiB in 27 minutes at its first attempt, and at 44.1 GiB in the replay. The run and its replay published the same bytes but for STAR's Log.out and nf-core's run metadata. The analysis ran the four tasks of VELOCITY; VELOCITY_STARSOLO peaked at 26.5 GiB.
• Before /dev/shm was cleared, the same pair ran on dbd598d with a sparse STAR index, every third suffix (--genomeSAsparseD 3), under a 26 GB cap. The analysis ran, replayed with an identical environment and verified with exit 0, but the build's replay was killed twice by the kernel: Nextflow counted the host's 62 GB and ran STAR_GENOMEGENERATE beside BOWTIE2_BUILD. With executor.memory = 28.GB the two ran in turn, and both jobs ran, replayed with an identical environment and verified with exit 0 (STAR 25.2 GiB, VELOCITY_STARSOLO 13 GiB). These rows are in the same record.

nfclaw features

These checks ran on the same host, apart from the pipeline sweep:

Chains

Each chain ran with nfclaw chain run and ended with nfclaw chain status. Its lineage check verified every handoff against the checksums the upstream stage recorded.

chaining.md and the chain notes in known-issues explain these settings. Run again on 2026-10-29 with exactly these settings, bamtofastq → rnaseq failed in rnaseq's Salmon pseudo-alignment: its test profile's prebuilt yeast index quantified the human reads. With skip_pseudo_alignment added, as the chain notes now say, it completed with verified lineage (deep audit, B09).

The createtaxdb stage's outputs were deleted after its lineage had verified, to free disk for the sweep. From then on, nfclaw chain status reported the chain's lineage as BROKEN. It named the missing checksum record, the missing Bracken database, and each database file taxprofiler had consumed. This is the behaviour the lineage check is designed to have.

Provenance bundles

A second sweep on the same host (2026-10-15 to 2026-10-16) checked the bundle of every pipeline by replaying it. Each pipeline ran with the settings the pipeline sweep needed. Its provenance/commands.sh then replayed it into a fresh directory, and nfclaw verify compared the replay's outputs with the original's.

Since this sweep, a bundle records the run's execution environment in environment.json:

• each process's container image, by registry digest and image ID;
• the Nextflow plugins and Java runtime;
• the SHA-256 of every remote file Nextflow staged.

For these Docker runs, containers.config pins eligible processes to their digests. The bundle holds the pipeline source as an archive, and copies of the external configs. A replay records its own environment and compares it with the recording (environment_check.json). Detected mismatches fail standalone replay; unverified comparisons are reported without failing it. The successful comparisons in the table below were recorded as identical, rather than merely passing with unverified entries.

92 runs of 91 pipelines were replayed. 90 replays reported an identical execution-environment comparison. 2,496 of the 2,496 containerized processes were pinned to the registry digest of the image they ran, no image was left without an identity, and 673 remote inputs were recorded by SHA-256. The replays that did not pass are explained below.

"Pinned / containerized processes" counts the processes containers.config pins to the digest of the image they ran. Every replay ran those digests, not the tags. "Remote inputs" counts the remote files Nextflow staged, recorded by SHA-256. nfclaw verify compares outputs by path. Its "changed" files are reports, logs and plots that embed timestamps or paths. Notes on the remaining rows:

• sopa: the replay failed its environment check, correctly. The release's test samplesheet names a GitHub HTML page as the sample's data file, and the page's bytes change on every download (known issue).

• seqsubmit: needs an ENA Webin account. The run failed where its test queries ENA, and its replay failed the same way. On 2026-10-23 its bundled test also ran in stub mode (--demo with a --config setting stubRun = true, Nextflow 25.10.4), which needs no ENA access. All 17 tasks completed, among them every step up to and including the manifests (CREATE_MANIFESTS). nfclaw status reported the success with its stub run: caveat. The replay ran with an identical execution environment (13 processes, 12 images), and nfclaw verify exited 0 (20 identical, 4 changed reports). Stub mode cannot reach the submission (SUBMIT): the stub manifest's name matches no genome, so nothing is handed on. The ENA query and submission themselves remain unverified without a Webin account.

• epitopeprediction 3.1.1: the plain run's replay met a malformed Ensembl BioMart reply, the upstream run note, and failed. With the retry config passed as --config, which the bundle copies and the replay passes on, the run and its replay succeeded, and the environments were identical.

• Different file sets: these were the only differences nfclaw verify reported. Tools named outputs by the clock or by a process ID:

  • kraken2-build in createtaxdb
  • Prokka in mag
  • FusionInspector in rnafusion
  • MISO in rnasplice
  • NanoPlot in nanoseq

  longraredisease names its MultiQC report after the output directory. hicar's MultiQC report and nanostring's plot exports varied between runs. The isoseq and raredisease originals were resumed after an attempt stopped by the shared host, so their outdirs also hold the first attempt's launch-stamped reports (known issue).

• Reruns: the table shows each pipeline's last run. The shared host's full disk and image removals stopped some first attempts, and pipelines whose bundles came from an nfclaw version with a defect listed below were run again.

• genomeqc: the table's 0, 0/0 and identical mean no images were compared. Its test profile sets cleanup = true, which removes a successful run's task records (known issue). The check now reports this as unverified. An earlier version of this page blamed one replay's Nextflow cache.

The nfclaw feature paths were replayed the same way:

• a generated samplesheet
• a YAML params file
• a failed run, whose replay failed the same way
• a resumed run
• an older release (demo 1.0.2) and dev
• a bundle moved to another directory
• both stages of a demultiplex → seqinspector chain, with the lineage verified
• --no-provenance, which writes no bundle

Every replay reported as successful in these final results had an identical environment comparison. This describes the recorded comparison scope in the architecture guide, not complete machine identity or proof of analytical agreement.

On GitHub Actions, the demo-validation workflow replays the demo with Docker twice: in place, and from the bundle's source archive with the checkout moved away. It also runs the demo with Apptainer, which records image files by SHA-256.

The sweep found these defects in the new recording. Each is fixed and covered by tests:

• Engines before Nextflow 23 recorded no remote inputs. They stage remote files in a shared work/stage/ directory, not work/stage-<session>/. The recorder now selects Nextflow's logs by the session each one names.
• Images were missed on a busy host. The watcher inspected every container started on the host to find the run's own, and short tasks exited before their turn. It now inspects each image reference once, when the first task container starts from it.
• An nfclaw upgrade during a long run left a bundle that could not replay. The bundled scripts were read from disk when the run ended, but commands.sh came from the code loaded at the start. The scripts are now read with that code.
• Replaying from the source archive changed ${projectDir} paths. A replay now runs the recorded checkout while it still holds exactly the recorded source. It extracts the archive only when the checkout has changed or is gone.
• An official Docker Hub image named with its registry (docker.io/ubuntu) was not pinned.
• nfclaw verify did not pair the nf-co2footprint plugin's launch-stamped reports.
• A samplesheet the pipeline read from a URL itself was not recorded. nf-schema reads such a file without Nextflow staging it. The bundle now reads each file that nf-schema validated again, right after the run, and records its SHA-256. This was added at the end of the sweep and checked on demo, bamtofastq, datasync and fastqrepair, whose rows show the final runs. Their test samplesheets sit on moving test-datasets branches.

Defects found and fixed

These nfclaw changes came out of the validation. Each is covered by tests, and the runs that showed the problem were repeated:

• --resume after Ctrl-C or kill recomputed every task. On a stop, nfclaw skipped the dependency-stability check that records which inputs a run may reuse, so the next --resume treated the whole cache as stale. The check now runs on a stop too, and a resumed run reuses the cache.
• Stub-run test profiles reported an unqualified success. proteinfold's test profile sets stubRun = true, so its demo computes nothing. nfclaw run now says so before launch, nfclaw status reports a stub run: caveat, and the run manifest, replay header and skill.md record it.
• Nextflow engines older than 21.10.0 failed on Java 17. They cannot compile a pipeline config there (Unsupported class file major version 61). nfclaw now refuses such an engine before launch, and skill.md recommends Nextflow 24.10.5 for the releases that declare one (bactmap).
• A failed run's error was misquoted when the nf-co2footprint plugin relayed the console. The plugin relays log records without line breaks, so nfclaw status quoted a stray Tip: line. It now splits those records and quotes Nextflow's error report.
• A run that aborted with process hasn't exited gave no cause. That is what Nextflow before 26.10.0 prints when a local task is killed at its time limit (nextflow-io/nextflow#7569). nfclaw status now names the time-limit kill and the remedies in its fix: line.
• "Stopped without an outcome" did not mention a full disk, although a full disk stopped runs here. The message now names it among the causes.
• skill.md did not say which Nextflow secrets a release reads. It now lists them, and where they are read (seqsubmit, metapep, sarek, raredisease, rnaseq, pathogensurveillance).
• A hidden directory beside the pipelines was taken for a pipeline. A Nextflow launched from pipelines/ leaves pipelines/.nextflow/; nfclaw list showed it, and write_skill --all crashed. Hidden entries are now skipped.
• createtaxdb 3.1.0's sourmash sort race is an upstream defect. nfclaw now warns about it before launch.

Upstream defects and host limits found along the way are documented in known-issues, with the narrowest workaround that ran.

Test suites and CI

These are results recorded during that validation, not current test counts:

• pytest: 2730 passed and 2 skipped on the Linux validation host. On macOS 2564 passed and 168 were skipped: 167 need the pipeline submodules, which that checkout did not initialize, and one reads Linux's /proc.
• Website: npm audit found 0 vulnerabilities; type check and build passed.
• GitHub Actions on the pull request: every check passed.

For the provenance bundle sweep:

• pytest: 2898 passed and 2 skipped on the Linux validation host, with every pipeline submodule. On macOS 2730 passed and 170 were skipped: they need the pipeline submodules, which that checkout did not initialize, or Linux's /proc.
• The demo-validation workflow on GitHub Actions also checks the recorded execution environment. Its Docker job checks the digest pins, an in-place replay and a replay from the source archive. Its Apptainer job checks that image files are recorded by SHA-256.

Limits

• A bundled test profile shows that a release runs end to end on small data. It does not show that the results are scientifically valid for real data. Where a stronger check was possible, it was done: the replay comparisons, the fastqrepair known-answer cell, oncoanalyser's snapshot, the STAR-Fusion tutorial and the agreement of rnafusion's three fusion callers.
• Every run used Docker on Linux amd64. Singularity, Conda, other executors and arm64 were not part of this sweep; known-issues records the arm64 MultiQC export failure.
• rnafusion ran on upstream's hosted GRCh38 references instead of building them, as its test profile would. The reference build itself ran only on the STAR-Fusion tutorial's minigenome (upstream's test_build profile).
• seqsubmit was not run to completion: its test queries ENA with a Webin account, which this validation did not have. The tasks before that query completed (details), and a stub run covered its workflow up to the ENA submission (above).
• The host's IPv6-only network shaped several runs: host networking, a local FTP relay, and local copies of ENA reads. A host with IPv4 connectivity should need none of these.
• A bundle records the images Nextflow runs and the remote files Nextflow stages. It does not record files a task downloads by itself: genomeqc's Dfam database, epitopeprediction 3.1.1's BioMart replies, or databases fetched inside a task. A replay can therefore meet other bytes there. Conda environments are not recorded either.
• Singularity and Apptainer image hashing ran only in CI, on the demo pipeline. Podman ran only in tests.