Validation scope and samplesheet generation

What nfclaw checks before a pipeline starts, what it leaves to the pipeline, how it writes a samplesheet from data files and for which pipelines, and which inputs it refuses.

nfclaw run --check prepares the command without running any scientific task. Its three preflight lines list the checks it completed, those left to the pipeline at launch, and those it does not support. A real run keeps the detailed, versioned report as validation_scope in provenance/run_manifest.json. When preparation fails, the error names the failing check and does not claim that later checks ran.

host_requirements means the run's tools are on the host: Git, Nextflow, Java and the profile's container engine. For a Docker profile the daemon must answer docker info, so start Docker before checking. --check also runs one engine probe: it asks the Nextflow the run would use (with its --nxf-ver, --nxf-env and every -c, generated ones included) to resolve the pipeline's configuration and selected profiles with nextflow config, from a scratch directory. A release that engine cannot parse, such as an older one under Nextflow 26.04's strict parser, is refused with the engine or parser to use instead; otherwise profile_config is reported as checked. The probe resolves no parameter values, and a pinned engine that is not on the host yet is downloaded first.

Generated samplesheets are validated like hand-written ones. --check looks at local data paths but does not read every data file, so local_data_bytes stays deferred until a real run takes its input snapshots: a file that cannot be read can still stop a run that passed --check.

Navigation

Schema subset • Replay integrity limits • Generation boundaries • Support matrix • Inputs refused before launch

Wrapper schema subset

The supported path formats and exact-source annotations are shared by handwritten tables, generated tables and handoffs. The architecture describes their boundaries. Before/after local hashes can detect a lasting change but cannot prove continuous immutability or establish every byte a task consumed. Implicit indexes, transitive config includes, Conda environments and task-managed downloads are not comprehensively discovered.

Replay integrity limits

New ordinary runs use source identity version 2: pipeline-source-identity.json binds the full source namespace, permissions, regular-file bytes, internal relative symlinks and internal hard-link topology. Untracked and ignored files are included; only the root .git administration entry and Python bytecode caches (__pycache__ directories, which tasks running a bin/ script that imports a sibling module write into the source, as circdna, reportho and viralmetagenome do) are excluded. Symlink permission bits are recorded as 0777: Linux cannot set them and neither platform uses them, so a bundle replays from its archive on either. Source links that escape the tree, aliases into excluded Git metadata, outside hard-link aliases, privileged executable bits and special objects are refused. Archives retain this identity and extract under a private root. Source modes, links and new files are checked after execution too, including with --no-provenance. Put other generated files and writes outside the source tree; their appearance inside it is a source change.

Legacy source identity version 1 authenticates the recorded revision and file contents only; its source archive dereferences tracked symlinks and does not independently bind their identity, permissions or an ordinary Git checkout's untracked/ignored files. Reading an old bundle does not upgrade this evidence. Neither version establishes continuous immutability or captures transitive dependencies outside its recorded namespace, filesystem ownership/ACLs or Git administration data.

The three exact-pin recorded source corrections (Createtaxdb, Funcscan, and Bactmap) use independent regular-file copies with a separately bound inventory, executable modes and complete directory namespace. They reject original untracked/ignored files and source symlinks, and verify the actual corrected copy or extracted archive before/after execution. These stronger checks apply to those corrected copies; they do not retroactively strengthen ordinary historical bundles or establish continuous filesystem immutability.

Generation boundaries

Generation consumes local directories, quoted local globs or explicit local files. Remote dataset inventories require an explicit samplesheet. File patterns select typed columns; sample names remain unchanged. Values that files cannot determine still require schema defaults or --sheet-set. A "direct" classification does not promise that every arbitrary filename or mixed layout can be inferred; orphan mates, duplicate/ambiguous roles and missing assignments are refused.

_R1/_R2 filenames pair directly, with an advisory if the first read names disagree. _1/_2 pair only when their first read identities establish mates. This inspection examines the first read, not every record, so it does not establish complete read pairing or FASTQ validity. Zstd read-name inspection is unavailable: use unambiguous _R1/_R2 names (with an advisory) or a manual sheet. Unfamiliar naming and mixed layouts may need manual assignment. Unmatched unrelated files may be ignored with a warning and recorded inventory; inspect the warnings before running.

Support matrix

Classified on 2026-11-06 at commit a3e8cb3, over the 95 pins of that commit, with the same sheetgen.plan the runtime and the generated pipeline skills use: 71 direct, 3 requiring column selection, 12 manual, 9 without a consumed samplesheet schema. It classifies schemas rather than executing analyses. The machine-readable inventory records every source commit and the classifier's digest.

The maintenance bots move pins without editing this table, and another release may have another contract: each pipeline's generated Inputs section in skill.md is what stays current, and is authoritative for assignments and required values. To reclassify, load each pin's schema and read sheetgen.plan: choice means choose columns, another problem means a manual sheet, neither means direct, and inputs.samplesheet_schema returning nothing means no schema.

Inputs refused before launch

nfclaw run refuses these before it creates a lock or writes anything into --outdir; the agent guide gives the one-line rules.

Output directories and locking

On POSIX systems, nfclaw run holds an exclusive, non-blocking lock on a file beside the output directory (.{outdir.name}.nfclaw.lock) for the whole run. A second run into the same --outdir fails at once with an environment error instead of mixing its output with the first.

Before creating that lock or any run files, explicit local dependencies and samplesheet data paths are checked against output, work, engine-state and log destinations:

• Keep execution writes outside consumed directories, including nested aliases and directory glob matches; keep data, pipeline code and configuration outside the output tree. Chain preflight also checks its whole output tree before writing its record.
• Existing output aliases and hard-linked files are inspected too: a different pathname does not establish different data. Declared work/cache trees retain their staged-read policy; input slots within output trees remain protected even when their symlinks point outside.
• Boundary refusals preserve input files, directory entries and existing logs.

Input filenames

Explicit local and remote input path fields and schema-declared path parameters are checked for filenames that pinned tasks could interpret as shell syntax.

• Refused: unsafe staged basenames, including literal wildcards, whitespace, shell operators and leading - or ~. Choose a safely named copy rather than renaming a scientific sample.
• Supported: Unicode letters, digits and combining marks and literal ._-+@%=,:~ characters (with ~ only internally).
• Directories and links: local directories and directory glob matches are inspected recursively, including empty subdirectories; unreadable entries, broken links and cycles are refused. Symlink targets and local glob matches receive the same name check.
• Remote names: Nextflow preserves percent escapes in remote basenames; nfclaw preserves them too. HTTP/FTP query suffixes are excluded from the staged name; standalone fragments remain literal filename characters. Object-store keys such as S3 retain query and fragment characters and receive the literal-name check.
• Parents: parent directory names are not staged as the input basename, but LF/CR in local input paths or resolved targets are refused because Nextflow can turn them into shell lines. Samplesheet paths crossing a directory symlink with .. must use an explicit absolute parent: Nextflow's lexical normalization may select different data. Ordinary parent traversal is supported; wildcard parents followed by .. are refused before resolution.
• URIs: a URI scheme must start the value, so a local colon parent remains a checksummed local dependency. Local file: URI inputs must be written as absolute filesystem paths so the same filename and replay-dependency checks apply.
• Run directories: the separate run-directory space check still applies, and --allow-spaces cannot permit LF/CR in launch, output or work paths.
• Not covered: values supplied indirectly through configs/profiles, remote glob inventories and remote file contents remain subject to upstream validation.

Audited output-only directory parameters retain basename checks while excluding unrelated existing children from input snapshots. These roles are declared in runner/path_roles.json for exact clean source revisions, never inferred from names or descriptions. Whole-path arguments in audited task scripts also validate parent folders and resolved symlink paths. HTTP/HTTPS/FTP basenames remain literal even in schema-declared file patterns; object-store patterns retain their upstream expansion semantics.

Sample identities

Metadata identifiers (meta.id, patient/sample IDs, genome/database names and legacy sample/patient columns) must start with a letter, digit or underscore and contain only letters, digits, combining marks, dots, underscores or hyphens. Upstream tasks use these values in shell commands and filenames. Choose explicit safe aliases for other identifiers; nfclaw never silently renames scientific samples.

This preflight applies to CSV/TSV and flat JSON/YAML records; YAML requires pip install -e ".[yaml]". Local path fields in these records must be absolute, including glob patterns, and their content is included in replay dependency checks. Explicitly supplied auxiliary tables such as --genome-sheet and --databases receive their own schema validation and local-data dependency checks too. Values supplied by profiles/configs and future chain handoffs are validated by Nextflow when available. Nested input structures remain subject to the pinned nf-schema validation.

Numeric parameter values

Numeric values supplied through CLI flags, JSON/YAML parameter files and chain specs retain their decimal value when nfclaw writes the parameters Nextflow reads. Decimal notation and scientific notation do not round large integers or underflow small nonzero values to zero. Schema enum/boundary comparisons and numeric CSV/TSV column checks use the emitted decimal value. JSON/YAML nested numeric parameters and recorded chain specs retain it too. Integer values beyond Nextflow's signed 64-bit JSON parser range use decimal notation to prevent wraparound. NaN and infinity remain refused. Nextflow, nf-schema and each scientific tool still determine their own supported ranges and arithmetic precision.