Scientific output contracts

nfclaw assess checks a run's outputs against expectations written down in a contract. It prints JSON and exits non-zero if any declared file, sample, format, binding or comparison fails. It does not take a zero exit status as proof that the outputs are complete. nfclaw verify, with or without --strict or --require-environment, is a separate check of files, listings and environment.

Navigation

Usage • Contract format • Formats • Tolerances and exclusions • Supplied contracts

Usage and setup

pip install -e '.[scientific]'
nfclaw assess results/hgtseq.replay --against results/hgtseq \
  --contract runner/output_contracts/hgtseq-1.1.0-historical-demo.json

Without --against, nfclaw assess checks one run against the contract. With --against, the original run's selected files must still match their recorded provenance/outputs.sha256 before they serve as the reference. The candidate's files are read as they are. Both sides are hashed again after parsing; that shows they did not change between the two readings, not that they never changed.

The BAM/VCF and H5AD readers are optional (the scientific extra). h5py 3.16 has no wheel for Python 3.15, so installing it there needs a source build with the HDF5 development headers and libraries (and pkg-config, or explicit HDF5 include and library paths). On the validation server, the Python 3.15 reader tests passed with a private HDF5 1.10.10 build. CI prepares HDF5 for Linux with 3.15; macOS with 3.15 tests the runner and the BAM/VCF readers but skips H5AD until a wheel exists. The other CI Python versions install every reader. That is how CI is configured; each workflow run is the evidence for its own lanes.

Contract specification

The version-1 contract contains name, version: 1, a nonempty artifacts list, and an optional pipeline binding with an exact name, 40-character commit, and explicit params conditions. Unknown or duplicate fields are refused. Parameter conditions preserve JSON types (false is not zero), and parameter bytes must match configs.sha256. Profile/config defaults absent from the recorded params file cannot be bound through a parameter condition.

With --against, the binding applies only to the recorded original. The candidate's software and parameters are not checked: passing means its selected content meets the expectations, not that it ran the same pipeline or environment. Check the environment separately with nfclaw verify. When one run is assessed on its own, its own records must satisfy the binding.

Each artifact names one exact relative path and a supported format. It may declare min_records, expected_records and a coordinates description. Missing, external-link, nonregular, malformed or unsupported artifacts fail. Internal links must resolve inside the run. Scratch SQLite files are created outside both runs; set TMPDIR to another writable filesystem if necessary. Large record inventories use disk, while table rows, individual sequence lines, headers and H5AD axis identifiers still need memory. Reports cap mismatch examples and list how many result files were not assessed. No unmatched result is implicitly accepted as scientifically validated. The contract SHA-256 and comparator/adapter versions identify the assessment. Exact BAM/VCF header comparison uses canonical header digests in the report; thousands of declarations are not repeated in its JSON. The complete header remains in the selected data file.

Supported formats and comparison scope

Tolerances, samples, and exclusions

Tables can require sample columns with sample_columns: ["A", "B"], or exact row-membership with sample_column: "sample_id", expected_samples: ["A", "B"]. Every artifact is assessed separately; samples found in one file do not fill omissions in another. Declare all matrix columns, not just a convenient subset. BAM expected_samples checks sample names observed through record read groups; records without an applicable read group cannot establish that identity. VCF expected_samples checks the sample columns, not the presence of alternate calls per sample.

Table tolerances default to zero. To permit a justified difference in a named numeric column:

"tolerances": {
  "abundance": {"absolute": "0.000001", "relative": "0", "reason": "Documented output precision"}
}

The rule is abs(a-b) <= absolute + relative * max(abs(a), abs(b)), using exact rational arithmetic from finite decimals, without the ambient Decimal rounding context. Decimal exponents outside +/-10000 are unsupported. Nonzero tolerances need a reason; no tolerances apply to BAM, VCF, FASTA or H5AD matrices. A table may explicitly exclude a non-scientific column with ignore_columns: {"": "Pandas positional index; stable keys are declared separately"}. Key, numeric and sample-identity columns cannot be excluded, and the exclusion is reported.

BAM can explicitly exclude PG/CO header records with a reason. VCF can explicitly exclude named metadata header records, but cannot exclude contig, INFO, FORMAT, FILTER, reference or fileformat declarations. Every exclusion is reported. Numeric VCF values retain decoded bits, including INFO/FORMAT values whose shorter printed strings would otherwise hide differences. HTSlib/BCF precision applies; this is not arbitrary textual-decimal identity. Supported field cardinalities are fixed, A, R, variable (.), and FORMAT G with known genotype ploidy. Unsupported cardinalities, undeclared fields and nonfinite numeric data fail. A single missing value is allowed; an explicitly supplied missing vector must still have the correct length.

GATK's AS_SB_TABLE deliberately serializes per-allele forward/reverse counts with commas and pipes inside a Number=1,Type=String annotation. A VCF contract can explicitly declare field_encodings: {"INFO/AS_SB_TABLE": "gatk-strand-counts"}. This validates two nonnegative integer counts per allele and compares the full decoded annotation; it does not ignore the field or relax any other declaration. See the GATK encoder/parser and VCF specification.

H5AD checks the declared AnnData/dataframe/array encodings and axis/matrix dimensions. Sparse pointers, bounds, value arrays and duplicate coordinates are checked. Selected external links, soft links, external-storage datasets and virtual datasets are refused before reading their data. See the AnnData format.

Shipped contracts and coverage

Other pins and branches need their own reviewed contracts. The format readers and these two demo contracts do not validate any other pipeline scientifically. H5AD in particular is covered by format tests built to break the reader, not by a biological benchmark of a pipeline run. Exact assembly comparison deliberately flags a lost Flye contig: output that changes with a random seed, reports or file names is not accepted just because it varies.