acia.analysis.stage#

Shared run context for a chain of analysis notebooks (“stages”).

A staged workflow splits one analysis into several notebooks (segment -> track -> measure) that hand their results over as files in a shared output folder, one folder per imaged population. Every stage therefore has to answer the same three questions before it can do anything: where do results go, which population is this, and did the stage I depend on actually run here?

StageContext answers them once:

from acia.analysis import StageContext

ctx = StageContext.for_image(image_id, "./output", stage="Track")

seg = ctx.input_path("segmentation.npz")            # fail early, actionably
ctx.log_params(mode="greedy")                       # on disk immediately
write_units_csv(ctx.keyed(props), ctx.output_path("cell_properties.csv"))
ctx.log_metrics(n_tracklets=len(graph))
ctx.finish()

ctx                                                 # -> the folder, rendered

Naming the stage up front is what lets the record be written as the stage runs. Everything logged is on disk the moment it happens, so a notebook that dies half-way still says what it was doing and under which settings – which is exactly the run whose record is wanted, and the one that used to record nothing. An unfinished stage is marked "running" rather than passing for a complete one. A context built without a stage= name behaves as it always did: nothing is written until record().

The recorded stage_manifest.json makes a finished run self-describing – which stages ran, what each produced and under which settings – and is what acia.analysis.scale() batches read back to summarise a fan-out. Stages append to it, so a chain that grows a stage later keeps the earlier entries.

Beyond what a stage states, the context records what it did: the files it was observed to read and write, when it ran, and a digest of the notebook that ran (see acia.analysis._stage_io). Three things follow, none of which a notebook has to ask for:

the dependency graph is derived – tracking read the file segmentation wrote, so there is an edge, and it cannot fall out of date the way a written-down one does (stage_graph());
a stage whose input has changed since it ran says so (check_stale()), which is the one thing a folder of results could not tell you before – re-segmenting leaves stale tracking output looking perfectly current;
a whole batch becomes searchable as one table (stage_table()).

Provenance is a bonus and never a risk: if capture fails, record() writes exactly the manifest it would have written without it.

acia.analysis.stage.MANIFEST_NAME = 'stage_manifest.json'#: file name of the per-population manifest inside the output folder

acia.analysis.stage.IO_SCHEMA = 'acia.stage_io/v1'#: schema of the recorded-I/O block inside a stage entry, following the same convention as acia.selection/v1 and acia.registration/v1

acia.analysis.stage.DEFAULT_KEY_PATTERN = 'pos(?P<position>\\d+)_roi(?P<roi>\\d+)'#

pos001_roi002 -> position=1, roi=2. It is only a naming convention – pass your own key_pattern (named groups) or None for sources that don’t follow it; unmatched names simply become None.

Type:: default identity pattern

acia.analysis.stage.STAGE_SCHEMA = 'acia.stage/v2'#: schema of one stage entry. Versioned on the entry rather than the manifest, like IO_SCHEMA, because one folder legitimately holds entries written by different acia versions. An entry with no schema key is a v1 entry.

acia.analysis.stage.CALIBRATION_SCHEMA = 'acia.calibration/v1'#: schema of the recorded calibration block inside a stage entry

acia.analysis.stage.population_id_of(path)[source]#

Identity of one imaged population, derived from its source path.

A folder source (one image per timepoint) keeps its full name; a file source drops the extension. Stripping a .tiff-like suffix only makes sense for a file – folder names may legitimately contain dots, which a blind .stem would truncate.

Parameters:: path (str | Path)
Return type:: str

class acia.analysis.stage.StageContext[source]#

Bases: object

Where one stage writes, which population it is on, and what it may read.

Build it with for_image(); the fields are derived, not meant to be assembled by hand.

Variables:

image_id (str) – the source this run analyses, exactly as it was passed in.
output_dir (pathlib.Path) – the folder every stage of this population reads and writes.
population_id (str) – identity of the population (see population_id_of()).
keys (collections.abc.Mapping[str, Any]) – the columns that key this population’s rows in exported tables – population_id plus whatever key_pattern extracted (by default position and roi). Splat it into a table: df.assign(**ctx.keys).
stage – name of the stage this context is running, when it was given one. Naming it up front is what lets the record be written as it happens rather than in one call at the end – see log_params().

image_id: str#

output_dir: Path#

population_id: str#

keys: Mapping[str, Any]#

stage_name: str | None = None#

classmethod for_image(image_id=None, output_folder='./output', stage=None, *, key_pattern='pos(?P<position>\\d+)_roi(?P<roi>\\d+)', create=True, track_io=True, track_roots=())[source]#

Resolve the run context for one source.

Parameters:

image_id (str | Path | None) – path to the movie this run analyses – a single stack file or a folder of per-timepoint images. Optional: a folder that already holds a manifest knows its own source, so a downstream stage can be re-run there without being told it again. Passing it anyway is verified against what the folder records, and warns on a mismatch – running the wrong source in an existing folder mixes two populations into one set of results.
output_folder (str | Path) – where this population’s artifacts live. A relative value resolves against the working directory, which is what makes a stage chain work unchanged both interactively (the notebook folder) and under acia.analysis.scale() (each population’s own execution folder).
stage (str | None) – name of the stage about to run, e.g. "02_Track". Naming it here opens its manifest entry immediately, so everything logged afterwards (log_params(), log_metrics(), log_figure()) is on disk the moment it happens rather than waiting for a call at the end that a failing notebook never reaches. Omitted, the context behaves exactly as before and the stage is named by the deprecated record().
key_pattern (str | None) – regex with named groups, matched against the population id to derive grouping keys. None disables key extraction.
create (bool) – create output_folder if it does not exist.
track_io (bool) – record the files this stage reads and writes (see record()). On by default and free – notebooks need no change. Set it to False to capture only inside explicit track() regions, e.g. when exploratory cells should not count.
track_roots (Iterable[str | Path]) – extra directories whose reads should be recorded. Reads are otherwise limited to the working directory, the output folder and the source, without which a notebook’s thousands of site-packages opens would drown the record.

Return type:

StageContext

track()[source]#

Capture reads inside this block – advanced, rarely needed.

With the default track_io=True the whole stage is already captured and this is a no-op. It earns its keep with track_io=False, where nothing is recorded except inside these regions – the way to keep exploratory cells out of a stage’s record:

ctx = StageContext.for_image(image_id, "./output", track_io=False)
...                                      # scratch work, not recorded
with ctx.track():
    overlay = segment(source)            # this is the real analysis

Re-entrant, because a with block cannot span notebook cells: open one per cell and everything accumulates into the next record().

Return type:: Any

path(name)[source]#

Path of an artifact in this population’s output folder.

Parameters:: name (str | Path)
Return type:: Path

has(name)[source]#

Whether an artifact is already present (e.g. an optional upstream stage).

Parameters:: name (str | Path)
Return type:: bool

require(name, produced_by=None)[source]#

Path of an artifact that must exist, or an actionable error.

Parameters:

name (str | Path) – artifact inside this population’s output folder.
produced_by (str | None) – optional hint naming the notebook that makes it. The producer recorded in the manifest is always derived from what the earlier stages actually wrote, never from this string – but the error path cannot use that, because a missing artifact usually means the producing stage never ran here and the manifest cannot know either. So the message degrades by what is genuinely known: the recorded producer, else which stages did run here (which distinguishes “wrong folder” from “not run yet”), else this hint.

Return type:

Path

open(name, mode='r', **kwargs)[source]#

Open an artifact in this population’s output folder.

A convenience only – it joins the path and notes the access. It is not how tracking works and is never required: almost nothing in this stack takes a file handle (np.savez, pd.read_csv, tifffile.imwrite and cv2.VideoWriter all take paths), so capture watches the filesystem rather than handles.

Parameters:

name (str | Path)
mode (str)
kwargs (Any)

Return type:

Any

input_path(name, produced_by=None)[source]#

Path of an artifact this stage reads, which must already exist.

The reading half of the pair with output_path(). Beyond what require() has always done – failing early with a message that says which stage makes the file – it records the name as a declared input, so the stage says what it meant to read next to what it was observed to read. Direction itself is still observed, not taken from this call.

Parameters:

name (str | Path) – artifact inside this population’s output folder.
produced_by (str | None) – optional name of the stage that makes it – the name passed to for_image() as stage=, not the notebook file. It is checked: naming a stage that never ran here, or one that did not write this file, warns. It never decides anything – the producer recorded in the manifest stays derived from what stages actually wrote – and it still shapes the error when the artifact is missing (see require()).

Return type:

Path

output_path(name, *, parents=True)[source]#

Path of an artifact this stage writes, with its folder ready.

The writing half of the pair with input_path(). Creates the parent directory and records the name as a declared output, which is then checked against what was actually written – a name declared here but never written shows up in the manifest’s io.missing, which is usually a typo.

Nothing is created for the file itself, and the path is an ordinary Path: write to it with whatever writer you already use.

Parameters:

name (str | Path)
parents (bool)

Return type:

Path

log_params(**params)[source]#

Record the settings this stage is running with, right away.

Parameters are the part of a run that cannot be recovered by re-running it, so they are written to the manifest the moment they are known rather than at the end – a notebook that dies half-way still says what it was doing. Values that are not JSON-serialisable (pint quantities, paths, numpy scalars) are stored as their string form.

Parameters:: params (Any)
Return type:: None

log_metrics(**metrics)[source]#

Record numbers this stage obtained – counts, rates, scores.

Same immediacy as log_params(); the split is only about meaning, and both flatten into stage_table() columns.

Parameters:: metrics (Any)
Return type:: None

log_artifact(path, *, kind=None, caption=None)[source]#

Note that a file belongs to this stage, optionally describing it.

The observed writes already catch the file; this adds the intent and the description a bare filename cannot carry.

Parameters:

path (str | Path)
kind (str | None)
caption (str | None)

Return type:

Path

log_figure(figure, name, *, caption=None, dpi=150, subfolder='figures')[source]#

Save a figure into the output folder and record it with its caption.

A bare savefig lands in the folder anonymously: the diff notices a new file and nothing says what it shows or which step produced it. This writes it, records it in order with its caption, and keeps a small thumbnail for the context’s own display.

Parameters:

figure (Any) – a matplotlib Figure (anything with savefig), or the path of an image already written.
name (str) – file name; .png is appended when it has no suffix.
caption (str | None) – what the figure shows.
dpi (int) – resolution for the saved file (not the thumbnail).
subfolder (str) – where figures go inside the output folder.

Returns:

The path written.

Return type:

Path

property manifest_path: Path#: Path of this population’s stage_manifest.json.

manifest()[source]#

The manifest as recorded so far ({} before the first stage).

Return type:: dict[str, Any]

flush()[source]#

Persist everything recorded so far, including the observed I/O.

Rarely needed – logging already persists, and a cell boundary collects the rest. Useful before a long step whose record you want on disk first.

Return type:: Path | None

finish(status='ok', **extra)[source]#

Close this stage: stamp it finished and collect its I/O one last time.

Parameters:

status (str) – how the stage ended, "ok" unless you know better.
**extra (Any) – last settings/counts to record, same rules as record().

Return type:

Path | None

record(stage=None, artifacts=(), **extra)[source]#

Append this stage’s entry to the manifest and return its path.

The explicit finalize. Equivalent to logging **extra and calling finish(); still the whole story for a context built without a stage= name, where nothing is written until this call.

Besides what the caller states, the entry carries what was observed: the files this stage read and wrote (each with a (size, mtime) fingerprint), when it ran and how long it took, and which version of the notebook produced it. That is what lets a later run notice that an input has changed underneath a result, and what makes “these two populations disagree” answerable with “they ran different code”.

Parameters:

stage (str | None) – name of the stage, e.g. "Segment". Optional when the context was built with stage=. Re-running a stage replaces its own entry and leaves the others alone.
artifacts (Iterable[str]) – what this stage produced, as names inside the output folder. Optional – the observed writes are recorded regardless, and output_path() adds to this list on its own; pass it only to state an intent that can then be checked against reality.
**extra (Any) – whatever makes the run reproducible – settings used, counts obtained. Stored verbatim at the top of the entry, so keep it JSON-serialisable.

Return type:

Path

stage(name)[source]#

The recorded entry of a stage in this folder, or None.

Lets a downstream notebook read a setting off an upstream one – the pixel size segmentation actually used, say – instead of re-deriving it and risking a different answer.

Parameters:: name (str)
Return type:: dict[str, Any] | None

clear(stage)[source]#

Delete a stage’s recorded outputs and drop its manifest entry.

The honest undo for a stage that failed half-way: scale(exist_skip=True) keys on the copied notebook existing, so a half-finished stage is skipped on every later run until its traces are gone. Removing exactly what the stage recorded is more precise than deleting a folder – it also catches whatever it wrote elsewhere – and it is a no-op for a stage that never ran.

Parameters:: stage (str)
Return type:: list[Path]

log_calibration(source, *, origin=None, check=True)[source]#

Record the pixel size and frame interval this stage actually resolved.

The movie stays the single authority on calibration – this records what was resolved, it does not become a second place to read it from. That distinction matters: a stage that took its interval from an override parameter and one that read it from the file’s OME-XML currently leave indistinguishable records, so “what interval did this run use?” has no answer, and two stages of one population silently disagreeing has nothing to notice it.

With check (the default), a value that disagrees with what an earlier stage of this population recorded warns – the same reports-never-decides stance as check_stale(), because the caller may know exactly what they changed.

Parameters:

source (Any) – the opened image sequence, after calibration is resolved.
origin (str | None) – where the calibration came from, when the source cannot say (it is read from source.calibration_source otherwise).
check (bool) – warn when this disagrees with an earlier stage’s record.

Returns:

The recorded calibration block.

Return type:

dict[str, Any]

calibration(stage=None)[source]#

The calibration recorded by a stage of this population.

With no argument, the most recent one recorded here. Use it to check a value, not to replace opening the source – see log_calibration().

Parameters:: stage (str | None)
Return type:: dict[str, Any] | None

keyed(df)[source]#

df with this population’s key columns added, attrs preserved.

Exported tables carry their keys so many populations concatenate cleanly. DataFrame.assign drops attrs, which would silently lose the unit map (df.attrs["units"]) that the CSV writers rely on – this restores it.

Parameters:: df (DataFrame)
Return type:: DataFrame

summary(*, figures=True)[source]#

This population’s state as a rich display object.

The same panel _repr_html_() renders, as a value you can display from anywhere in a notebook rather than only as a cell’s last expression.

Parameters:: figures (bool)
Return type:: Any

__init__(image_id, output_dir, population_id, keys, stage_name=None, _recorder=None, _started_at=None, _run=None)#

Parameters:

image_id (str)
output_dir (Path)
population_id (str)
keys (Mapping[str, Any])
stage_name (str | None)
_recorder (Any | None)
_started_at (str | None)
_run (Any | None)

Return type:

None

acia.analysis.stage.read_manifest(output_dir)[source]#

The stage_manifest.json of one population’s output folder, or {}.

Parameters:: output_dir (str | Path)
Return type:: dict[str, Any]

acia.analysis.stage.stages_run(output_dir)[source]#

Names of the stages recorded in one population’s output folder, in order.

Parameters:: output_dir (str | Path)
Return type:: list[str]

acia.analysis.stage.check_stale(output_dir)[source]#

Recorded stages whose inputs have changed since they ran.

This is the gap that makes a staged workflow quietly wrong: re-segmenting with a new filter leaves the tracking output looking current forever, because exist_skip keys on a notebook file existing, not on freshness. Comparing each recorded input against the file on disk surfaces it.

Reports, never decides – like the curation manifest’s fingerprint check, a mismatch is a warning, because the caller may know exactly what they changed.

Parameters:: output_dir (str | Path)
Return type:: list[dict[str, Any]]

acia.analysis.stage.stage_graph(output_dir)[source]#

(upstream, artifact, downstream) edges derived from what stages read.

Nothing declares this graph – it is what the recorded inputs and outputs say actually happened.

Parameters:: output_dir (str | Path)
Return type:: list[tuple[str, str, str]]

acia.analysis.stage.stage_table(root, pattern='*/output', stale=True)[source]#

Every recorded stage run under root, as one table – one row per run.

Turns a folder of results into something searchable without a query language: the population keys, the timings, the code that ran and every setting a stage recorded become columns, so questions are pandas.

runs = stage_table("automated_executions_stages")
runs[runs.stale]                                  # what needs redoing
runs[runs.stage == "Segment"].pixel_size.value_counts()
runs.groupby(["stage", "code_sha256"]).size()     # did the batch run one code?

Parameters:

root (str | Path) – folder holding one subfolder per population.
pattern (str) – glob from root to each population’s output folder. The default matches the layout acia.analysis.scale() produces.
stale (bool) – check each run’s inputs against the files on disk and add a stale column. Set it to False for a large tree where the extra stat calls are not worth it.

Returns:

pd.DataFrame –

one row per (population, stage). Besides the population’s key: columns it carries status and, for a stage that failed, error_type/error_message; the timing, notebook and code_sha256 of the run; n_inputs/n_outputs/n_figures; the resolved pixel_size_um/frame_interval_s; and every parameter and metric the stages logged, flattened into columns of its own name so a whole fan-out can be compared in one table.

Return type:

DataFrame