Core¶
Workflows¶
Workflow is the discriminated union NfPipeline | NfModule, selected on each entry's
type: field. Internal plumbing (the graph, events, runners) is typed against Workflow;
construct a concrete NfPipeline/NfModule directly, or build one from raw config data via
validated_workflow.
NfPipeline
¶
Bases: BaseWorkflow
A Nextflow pipeline step executed with 'nextflow run'.
NfModule
¶
Bases: BaseWorkflow
A Nextflow module step executed with 'nextflow module run'.
WorkflowType
¶
Bases: StrEnum
Discriminates the two workflow shapes a config's type: key selects
between: a full nextflow run pipeline, or a single nextflow module run.
Globals and transitions¶
GlobalOptions
¶
Bases: BaseModel
Settings applied across every workflow in a metapipeline (the config's
globals: key). params is merged under each workflow's own params at
run time, with the workflow's own values winning on key collision;
profile/config_file likewise act as a fallback a workflow can
override, not a value forced onto it.
nextflow_version_tuple
property
¶
nextflow_version as (major, minor, patch, is_edge), or None
if unset. Missing minor/patch components default to 0.
Transition
¶
Bases: BaseModel
A directed source -> target edge between two workflow ids.
The legacy top-level transitions: representation; equivalent to the
target workflow declaring from: source. Prefer from: in new configs.
The graph¶
MetaworkflowGraph
¶
The in-memory representation of a metapipeline: a directed graph of
Workflow nodes plus shared GlobalOptions.
Mutating methods (add_workflow, add_transition, ...) validate the
change immediately and raise GraphValidationError/ConfigValidationError
rather than leaving the graph in an invalid state -- a failed mutation
has no effect. Wrap a batch of mutations in deferred_validation() to
validate once at the end instead of after each call.
Construct one via nf_meta.core.config.from_file/validated_graph
rather than composing an empty graph by hand, unless you're building it
up programmatically from scratch.
Source code in src/nf_meta/core/graph.py
pop_events
¶
Return and clear the events emitted by mutations since the last call.
Used by nf_meta.core.session.Session to drive undo/redo history;
most consumers don't need to call this directly.
Source code in src/nf_meta/core/graph.py
add_workflow
¶
Add a new workflow node.
Runs dynamic (network-backed) validation immediately unless inside
deferred_validation(). Raises ConfigValidationError and leaves the
graph unchanged if wf's id already has unresolved param references,
or if dynamic validation reports an error.
Source code in src/nf_meta/core/graph.py
update_workflow
¶
Replace the workflow with wf.id with wf.
Same validation behavior as add_workflow. Raises ValueError if no
workflow with that id exists yet -- use add_workflow to create one.
Source code in src/nf_meta/core/graph.py
remove_workflow
¶
Remove the workflow wf_id and every transition touching it.
Raises ValueError if wf_id doesn't exist. Does not remove
workflows that only reference the removed one via ${wf_id:params:...}
-- those become dangling references, caught on their own next
validation.
Source code in src/nf_meta/core/graph.py
add_transition
¶
add_transition(tr: Transition)
Add an edge marking tr.source as a predecessor of tr.target.
Raises GraphValidationError if either endpoint doesn't exist yet,
or if adding the edge would introduce a cycle.
Source code in src/nf_meta/core/graph.py
remove_transition
¶
Remove the edge source -> target, if present.
No-op (returns without raising) if the edge doesn't exist.
Source code in src/nf_meta/core/graph.py
update_global_options
¶
update_global_options(glob: GlobalOptions)
Replace the graph's GlobalOptions wholesale. Not validated against
individual workflows -- per-workflow effects (e.g. a shared profile)
are only checked when that workflow is itself next validated.
Source code in src/nf_meta/core/graph.py
validate_param_references
¶
Check that every ${wf_id:params:key} reference in wf.params
points at a real, already-resolvable predecessor param.
Raises ConfigValidationError (carrying one Diagnostic per bad
reference) if any reference targets an unknown workflow, a workflow
that isn't actually a predecessor of wf, or a param with no value.
Source code in src/nf_meta/core/graph.py
validate_graph_topology
¶
Raise GraphValidationError if the graph contains a cycle.
Source code in src/nf_meta/core/graph.py
validate_edge
¶
Raise GraphValidationError if src or tgt isn't a node in the graph.
validate
¶
Fully re-validate the graph: topology, edge endpoints, workflow ids, and every workflow's param references.
Pass touched (a set of workflow ids) to also run a single batched
dynamic (network-backed) validation pass scoped to just those
workflows -- this is what deferred_validation() uses on exit, rather
than dynamic-validating the whole graph on every batch. Raises
GraphValidationError/ConfigValidationError on the first failure
found; does not collect all errors across the whole graph in one
call.
Source code in src/nf_meta/core/graph.py
get_workflow_by_id
¶
Return the workflow with this id, or None if it doesn't exist.
get_transition
¶
get_transition(source: str, target: str) -> Optional[Transition]
Return the source -> target transition, or None if no such edge exists.
Source code in src/nf_meta/core/graph.py
get_transitions
¶
get_transitions() -> list[Transition]
get_workflows
¶
get_workflows_sorted
¶
Returns workflows in valid execution order.
Source code in src/nf_meta/core/graph.py
subset_workflows
¶
subset_workflows(start: Optional[str] = None, target: Optional[str] = None, workflows: Optional[list[Workflow]] = None)
Filter workflows (default: get_workflows_sorted()) down to the
sub-DAG reachable from start and able to reach target (inclusive
on both ends). Omit either bound to leave that side unrestricted.
Source code in src/nf_meta/core/graph.py
get_start_workflow
¶
Return the first order in topological sorting or None if no nodes exist
Source code in src/nf_meta/core/graph.py
successors
¶
Return wf's direct successors (workflows with from: wf.id).
predecessors
¶
Return wf's direct predecessors (the workflows in wf.from).
deferred_validation
¶
Suspend per-operation validation; on exit, run a full graph validation plus a single batched, concurrent dynamic-check pass scoped to the workflows touched while suspended (not the whole graph).
Source code in src/nf_meta/core/graph.py
Parsing, construction, and serialization¶
Free functions at the config-file-format boundary: parsing raw YAML-shaped dicts, constructing/validating individual workflows or a whole graph from them, and serializing a graph back to a dict or file.
from_file
¶
from_file(cfg_file: Path, *, collect_errors: bool = False) -> MetaworkflowGraph
Read cfg_file as YAML and build+validate a MetaworkflowGraph from
it. Reads the whole file into memory; makes no other side effects. See
validated_graph for the validation contract and collect_errors.
Source code in src/nf_meta/core/config.py
to_file
¶
to_file(g: MetaworkflowGraph, file: Path | str) -> None
Write g to file as YAML (via dump_config), prefixed with a
# yaml-language-server: $schema=... comment pointing at the published
JSON Schema for editor autocompletion. Overwrites file if it exists.
Source code in src/nf_meta/core/config.py
dump_config
¶
dump_config(g: MetaworkflowGraph) -> dict
Project a MetaworkflowGraph back to the config's dict shape (the
inverse of validated_graph): config_version, globals (omitted if
empty), and workflows keyed by id with each workflow's own from: list
reconstructed from the graph's edges. Always writes the current
dict-keyed workflows: + per-workflow from: form, never the legacy
list/top-level-transitions: shapes, even if the graph was loaded from
one of those.
Source code in src/nf_meta/core/config.py
validated_graph
¶
validated_graph(data: dict, *, collect_errors: bool = False) -> MetaworkflowGraph
Construct a MetaworkflowGraph from a raw, parsed-YAML config dict.
Workflows can be declared as a dict keyed by id (workflows: {<id>: {...}},
the preferred form) or as a legacy list of definitions each carrying their
own id. Transitions are declared per-workflow via from: <id> or
from: [<id>, ...]; a top-level transitions: key is also still accepted,
but mixing both styles in the same file raises rather than merging them.
Both the list-format workflows: and the top-level transitions: key are
deprecated -- reading either logs a CLI warning. Fix by opening and saving
the config in the editor, which always writes the current dict-keyed
workflows: + per-workflow from: form.
By default (collect_errors=False) this fails fast, raising on the first
problem found. Pass collect_errors=True for a best-effort pass that
collects every problem instead of stopping at the first -- used by the
CLI's validate command.
Source code in src/nf_meta/core/config.py
validated_workflow
¶
Construct and fully validate a single workflow from a flat, raw dict
(id optional -- the model's own default_factory generates one if
omitted). Runs the same tier-0/1 construction plus tier-2 dynamic checks
a workflow gets as part of a full config build. Returns the validated
Workflow, or raises GraphValidationError/ConfigValidationError.
Source code in src/nf_meta/core/config.py
validated_globals
¶
validated_globals(data: dict) -> GlobalOptions
Construct and validate a GlobalOptions block from a flat, raw dict.
Returns the validated GlobalOptions, or raises
GraphValidationError/ConfigValidationError.
Source code in src/nf_meta/core/config.py
Errors and diagnostics¶
Diagnostic
dataclass
¶
Diagnostic(workflow_id: str | None, field: str, message: str, severity: Severity, yaml_path: str | None = None)
One validation problem with a single field of a single workflow (or of
globals, when workflow_id is None). The common vocabulary every
validation tier (pydantic construction, dynamic/network checks,
cross-workflow reference checks) reports through, so callers handle one
shape regardless of where a problem originated.
Severity
¶
Bases: StrEnum
A Diagnostic's severity: ERROR blocks (raises
ConfigValidationError); WARNING is logged but doesn't stop
construction.
ConfigValidationError
dataclass
¶
ConfigValidationError(errors: list[Diagnostic], graph_messages: list[str] = list())
Bases: Exception
A workflow/config-definition-level validation failure, whether from tier-0/1 (pydantic construction), tier-2 (network-backed) checks, or cross-workflow reference validation. Every consumer treats all of these identically, so they share one exception type.
GraphValidationError
¶
Bases: Exception
A structural problem with the graph itself: a cycle, an edge to a
nonexistent node, or a malformed workflows:/from:/transitions:
shape. Distinct from ConfigValidationError, which carries per-field
Diagnostics about individual workflow/globals content.