Replaying and verifying runs

How a run's provenance bundle replays it, what that replay checks before and after Nextflow runs, and how nfclaw verify and nfclaw assess compare a replay with the run it reproduces. The agent guide (AGENTS.md) gives the commands; this page gives the guarantees and their limits.

Replaying • What the bundle carries • Dependency stability • Local inputs and resume • Verifying • Output contracts

Replaying a run

<outdir>/provenance/commands.sh re-runs the recorded command. It reproduces the run into a fresh directory (default <outdir>.replay, or pass one: ./commands.sh /path/to/fresh-dir) and refuses a target that already holds files. That is deliberate: an nf-core pipeline publishes into --outdir and cannot re-publish over a previous run's files, so replaying in place fails immediately on pipeline_info/execution_trace_*.txt (and on sarek's manifest_*.bco.json). A replay runs the pipeline again (it is a reproduction, not a --resume), and its result can be compared with the original bundle's outputs.sha256. It logs itself the same way, to <target>/provenance/logs/run.log, ending with its outcome: nfclaw status <target> reads it, and kill <replay pid> stops Nextflow with it.

Before launching Nextflow, commands.sh runs provenance/replay_guard.py against the original bundle. The guard checks the recorded pipeline revision and source identity, declared local input content including data paths referenced in samplesheets (inputs.sha256), and parameters and configuration (configs.sha256). Source identity version 2 binds regular-file bytes (pipeline.sha256), permissions, directories, internal relative symlinks and hard-link topology in pipeline-source-identity.json, including untracked/ignored source files and excluding root .git administration and Python __pycache__ bytecode caches. The inputs.inventory.json and configs.inventory.json records also preserve directory entries (including empty directories), entry types and symlink identities. Changed or missing dependencies and incomplete bundles are refused, rather than producing silently divergent scientific results. Replay targets inside a recorded input directory are refused before creating their locks or logs. New replay bundles acquire the same output-directory writer lock as nfclaw run, so overlapping runs and replays cannot write into that directory concurrently.

Invoke commands.sh by an absolute or relative path. A moved bundle replays its own params, configs, config copies, pin config and source archive; external input data must still exist at their recorded paths and pass their checksum checks, as must a config that uses includeConfig.

Replay runs the recorded analysis into a fresh directory, removes -resume, and pins the observed engine version; warning text can vary with plugin resolution and cached configuration even when scientific outputs agree. Upstream warnings that are not faults are catalogued in known issues.

What the bundle carries

New ordinary bundles archive the complete recorded source namespace and copy replayable external configs; they do not embed scientific input data, container images or plugins. Replay still needs Python, Bash, Nextflow and its compatible Java runtime, the selected execution tools, and access to recorded external data and software. Inspect the manifest for recording gaps; configs using includeConfig retain the original-path requirement below.

Pipeline source: pipeline-source.tar.gz, a deterministic archive bound to the versioned source identity. Version 2 retains file permissions, empty directories, internal relative symlinks and hard links; it does not dereference links into other source bytes. Its archive follows the results' umask, further restricted when source-file or directory permissions disallow group/other reads; extraction always starts under a private 0700 root. A replay runs the recorded checkout while it retains the exact revision and source identity, so paths embedding the pipeline directory (nf-core's ${projectDir}/assets/... defaults) keep their values. Otherwise it extracts the archive into <target>/provenance/pipeline, checks its identity, and runs that copy. <target>/provenance/pipeline-source.json records the choice and is required for replays that select a source. Both archived and working source are checked after Nextflow finishes. Legacy version 1 records only contents/revision and preserves its older, narrower guarantees; it is not upgraded implicitly. Archive fallback changes ${projectDir}; absolute paths already stored in params are not rewritten. See the source boundaries.
Configuration: the params file, generated configs and a copy of each external --config file (configs/). A config that uses includeConfig resolves relative includes against its own directory, so it replays from its original path. For recognized nf-core/configs includes, the release's default branch is resolved before launch and passed as custom_config_version, recorded in params.json and the manifest (institutional_configs). Explicit custom_config_version or custom_config_base values, a nonempty NXF_OFFLINE, or an unrecognized include bypass automatic pinning; a resumed run reuses its prior commit when applicable. A failed lookup is announced and leaves the release default.
Execution environment: environment.json, read after the run from Nextflow's own records (nextflow log, .nextflow.log): available container identities by registry digest and local image ID (Docker, Podman) or image-file SHA-256 (Singularity, Apptainer), the engine version, the plugins and Java runtime Nextflow loaded, and remote inputs recovered from its staging and validation logs. Staged local copies are hashed after execution; supported validation-only HTTP/HTTPS/FTP URLs are fetched again, up to 100 MiB, so those hashes describe the refetched bytes rather than proving what validation originally consumed. containers.config is generated only for eligible Docker/Podman processes with one known registry digest and no containerless tasks, and commands.sh passes it last. Multiple or unresolved images and ephemeral Wave references have no pin; Singularity/Apptainer image files are hashed rather than pinned this way. Inspect unresolved entries and issues for recording gaps; recorded issues are also printed as advisories.

After Nextflow succeeds, a replay records its own environment.json and compares it with the recording in <target>/provenance/environment_check.json. Another image digest, plugin set, container engine, or remote file content or set fails the replay with failed after Nextflow succeeded (execution environment differs from the recording); entries no record can decide (an image without a comparable identity, a Singularity image converted on another host) are reported as unverified and do not fail standalone replay. A differing Java runtime and processes present in only one run are notes. The comparator checks the container engine name, plugin list, images of shared processes and recorded remote URLs/hashes; Nextflow build, container engine version, host platform and task counts are not equality gates.

Dependency stability during execution

Replay restores the recorded Nextflow environment, including the absence of unrecorded NXF_* settings. Sensitive recorded values remain redacted; export every required redacted variable before replay. Newly inherited Nextflow settings are removed before launching. A production bundle with incomplete pipeline identity or checksum metadata is refused.

Before declaring success, nfclaw compares declared local input bytes and namespaces, generated/user configuration files, versioned pipeline source identity and its Git revision with the launch snapshots. A visible change or unreadable dependency fails the run even when Nextflow exits zero. Results and launch hashes remain available for inspection; provenance records dependency_stability. Restoring the original bytes does not validate cached tasks from that attempt: resume archives the failed run and recomputes, and standalone replay refuses its tainted bundle. The check also applies with --no-provenance; that flag skips the bundle rather than the dependency checks. Standalone replay rechecks its original bundle after Nextflow finishes. These are checkpoint comparisons, not filesystem locks or immutable input copies: transient edits restored between checks, remote inputs and indirect configuration dependencies are outside their scope. Keep scientific inputs stable for the full execution.

Local inputs and resume

Nextflow's own cache can miss a change to a file whose size and timestamp stayed the same, and a change deep inside a directory whose top-level metadata did not change. So before a real --resume, nfclaw compares two things with the previous run:

The inputs: the directory listing taken before launch and the SHA-256 of every declared local input file, samplesheet data paths included.
The pipeline source: its bytes, permissions, links, ignored files, path and Git revision.

If both are unchanged, Nextflow's cache is reused as usual. If either changed, or the previous record is missing or incomplete, nfclaw moves the whole previous run to <outdir>.previous-<id>, says where, and recomputes every task in a fresh Nextflow cache session. The earlier logs stay in the archive and are copied forward, so the run log keeps appending, and earlier published files cannot leak into the new result inventory. Archiving refuses while the engine's database is locked or when a dependency lies inside the run directory. A previous bundle without the complete version 2 source identity also starts fresh, because the older content-only checks cannot vouch for permissions or ignored runtime code. These checks also apply when the new launch uses --no-provenance. They cover declared local inputs, not remote or indirectly configured ones.

Where the previous record comes from:

A run that failed, timed out or was stopped with kill <pid> writes its bundle and resumes from it.
A run whose nfclaw vanished before writing the bundle (SIGKILL, out of memory, a host restart) or could not finish it (a full disk) leaves provenance/launch-record.json, written before Nextflow starts, with the source identity and the input bytes and listing it launched with. --resume compares against that and reuses the finished tasks if they agree. The record cannot reveal a change made and undone while that run ran, and it does not replace the check a run makes when it ends normally. The bundle replaces the record once it is written.
A run launched with --no-provenance records neither, so there is nothing to compare: --resume after it archives the earlier run and recomputes every task. Keep provenance on for long runs you may need to resume.

Checking a replay (`nfclaw verify`)

bash

nfclaw verify <replay-outdir> --against <original-outdir>
nfclaw verify <replay-outdir> --against <original-outdir> --strict

Hashes the replay's live files and compares them by path with the original recorded outputs.sha256 (or live originals when no manifest exists). It reports identical / changed / missing / extra for files, missing/extra directories, and changed link identities from outputs.inventory.json. Missing or extra files or directories fail the structural check.

Paired names. The nf-core run-metadata files the template names from its launch clock (pipeline_info/params_<time>.json, the execution report/timeline/trace/DAG of releases without trace_report_suffix, and the nf-co2footprint plugin's co2footprint_{report,summary,trace}_<time> files) are paired one to one, listed under renamed as original -> replay, and compared by bytes; so are the files kraken2-build adds to library/added/ under random mktemp names (XXXXXXXXXX.fna, its .fna.masked copy and prelim_map_XXXXXXXXXX.txt, as in createtaxdb's Kraken2 and Bracken databases), within their own directory. Any other dated or differently named path keeps its exact identity.

Older bundles. A historical file-only manifest cannot certify its original directory structure; verification reports that gap and exits unsuccessfully. Without any historical checksum manifest, both live directory trees can still be compared.

Reading the result. Changed bytes may reflect metadata or different scientific results; inspect them using a format-aware comparison. File structure alone establishes no analytical agreement. Links into independently generated task work directories can differ; inspect those changes explicitly.

Environment. When both execution-environment records (provenance/environment.json) can be read as JSON objects, it also reports the comparison of container images, plugins, the container engine name and remote inputs (environment: identical, differs with the mismatches, or unverified). Add --strict to require identical file bytes, recorded link identities and an identical environment comparison when available. If either environment record is genuinely absent and any remaining record has usable evidence, verification reports not recorded by both runs and strict mode enforces only the output checks; it cannot establish environment agreement. An existing record that is unreadable, malformed, empty or declares an unsupported version is instead unverified, so strict mode refuses it. Missing task/container, plugin or container-engine evidence also prevents an identical verdict; valid legacy records may omit newer optional fields. Use --require-environment to require complete evidence from both runs.

When the original was a resumed launch (-resume in its recorded command), its pipeline_info/ also holds the run metadata of the launches before it: a stopped or failed attempt's report, timeline, trace, DAG and params file. A replay makes one launch, so those files are listed separately as earlier attempts and not compared; only launch-stamped run metadata older than the newest of its kind is set apart this way, never a result. Without a recorded resume, two such files for one remain unexplained and fail the structural check.

For a stronger check of the recorded environment, add --require-environment (and --strict for output bytes). It refuses missing or incomplete environment records, tasks without a container, changed sets of processes, platforms or engine builds, and validation inputs that were only hashed by fetching them again after the run. A replay that succeeded on its own can still fail this check. It verifies the recorded evidence and cannot discover dependencies that were never recorded.

A replay needs python3 3.9 or newer (the guard scripts use only the standard library; nfclaw itself is not needed). Keep the original bundle: a replay writes its logs and source and environment records into the target, not a new complete bundle. Files a task downloads by itself, Conda environments and a config's own includeConfig targets are not frozen.

Scientific output contracts (`nfclaw assess`)

bash

nfclaw assess results.replay --contract runner/output_contracts/tumourevo-1.0.0-demo.json
nfclaw assess results.replay --against results --contract runner/output_contracts/hgtseq-1.1.0-historical-demo.json

This is separate from file/environment verification. The output-contract guide defines sample coverage, exact parsed comparisons, reasoned numeric tolerances and explicit exclusions for tables/BAM/VCF/FASTA and selected H5AD matrices. Two exact-pin demo contracts are supplied; other branches need their own.