Skip to content

Speed shared measurement cell assembly and CSV export with stable-ABI primitives #239

Closed

Description

@trissim

Current wide-producer spreadsheet export for the representative one-well ImagingFlowCytometry pipeline spends about 0.94 s assembling tables and 0.79 s serializing CSV, ~2.04 s overall. This cost repeats with wells and selected cells.

Saved current producer input: 14 MB, 31 measurement payloads, selected output 1,800 rows / 11,550,064 CSV bytes, SHA256 3e00436ae1500047fd2605021ae055492b17dbe0b4aa4ddbe2f62af6f8e073be. This is current wide-form data; the old eight-well long-form artifact is not the current frontier.

A shared compiled cell-assignment / CSV prototype preserves all nine saved-input observations byte for byte. Alternating replays: current Python ~1.93 s, compiled assembly alone ~1.80 s, compiled assembly plus CSV ~1.445 s. The bounded improvement is ~25% export time, not an established whole-pipeline speedup and not a fix for first-ever cold compilation. The existing accumulator must retain row identity, scope and qualifier ownership, with one shared conflict-assignment implementation and existing scalar-equality authority. Keep Python numeric string rendering and CSV quoting, nonfinite handling, source encodings and field order exact; do not add an import fallback or duplicate a backend roster.

Production admission requires meaningful adversarial equality/sparsity/projection-order and CSV parity tests, stable-ABI wheel consumption, and ordinary fresh endpoint benchmark with exact output checks. Current release-floor test also assumes setup() has only cmdclass, although main already builds the native granularity extension through ext_modules; that stale guard must admit build-only extensions while continuing to reject dependency/metadata duplication.

Related: #162 remains open for kernel/timing scope. Prior #194 and #216/#219 addressed different long-form/wide-producer bottlenecks and are already merged.

Activity

  1. added 2 commits that reference this issue on Sep 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions