diff --git a/benchmark/results/perf_plate_input_transport_20260929/README.md b/benchmark/results/perf_plate_input_transport_20260929/README.md new file mode 100644 index 0000000000..e152b75d4e --- /dev/null +++ b/benchmark/results/perf_plate_input_transport_20260929/README.md @@ -0,0 +1,43 @@ +# Transfer only declared plate inputs from worker lanes (2026-09-29) + +This report supports [OpenHCS issue #192](https://github.com/OpenHCSDev/openhcs/issues/192) and [PR #193](https://github.com/OpenHCSDev/openhcs/pull/193). The control is integrated commit `cc0615a9f`, which contains the single-axis compilation fix in [PR #191](https://github.com/OpenHCSDev/openhcs/pull/191) and separate pending performance changes from PRs #157, #170, and #175. The candidate adds only the plate-input transport change to that integration checkout. The review branch includes #157 as a prerequisite because #191 is not based on it. Four-worker observations use `fork`, 16 wells, one native thread per worker, and a fresh client-owned execution server. One-well observations also use a fresh server. The change affects execution and total time; it does not change compilation. + +`ExampleColocalization` exposed a **18.8–25.6-second gap** between the last worker completing and the plate spreadsheet starting. Instrumenting the worker pipe showed sequential result serialization and receipt. One four-well lane sent about **1.54 GB** of pickled runtime records: 1.11 GB object labels, 382 MB images, 41 MB measurements, and less than 1 MB relationships. The spreadsheet's compiled artifact inputs require measurements and relationships. The new `MERGE_PLATE_INPUTS` observation mode derives retained artifact types from those compiled inputs. Explicit full-value observation and analysis consolidation still retain every record. Runtime export paths are found before filtering. + +| Fresh-server official30, one well per case (summed seconds) | Control | Candidate | Change | +| --- | ---: | ---: | ---: | +| Server compilation jobs | 70.193 | 70.310 | +0.117 | +| Server execution jobs | 85.869 | 80.329 | **-5.539** | +| Other measured time | 63.944 | 64.069 | +0.125 | +| Per-observation total | 220.006 | 214.708 | **-5.298** | + +All **30/30** cases succeeded; **110/110** generated result files matched by relative path and bytes. Execution improved in 21 cases, with the largest reductions in ImagingFlow (-2.512 seconds), TrackObjects (-0.820 seconds), and Colocalization (-0.530 seconds). The full sweep followed a candidate warmup through the same corpus so cold JIT compilation was outside the paired measurement. + +![Official30 compile, execution, and total times](official30_phase_totals.png) + +| 16 wells, four fork workers | Control execution | Candidate execution | Control total | Candidate total | Post-worker gap | +| --- | ---: | ---: | ---: | ---: | ---: | +| Colocalization | 51.177 s | **33.491 s** | 61.983 s | **43.263 s** | 18.804 → 0.726 s | +| ImagingFlow | 212.827 s | **126.078 s** | 227.382 s | **139.305 s** | 88.090 → 3.171 s | + +The plate-level export remained near 15 seconds for Colocalization and 52 seconds for ImagingFlow; the gain comes from worker-result transfer. Both 16-well plate exports matched the control byte for byte. The ImagingFlow candidate's measured process-tree peak memory was slightly higher (14.14 versus 13.72 GB), so this result does not establish a memory reduction. + +The earlier native CellProfiler 4.2.8.1 single-sample `EXECUTE_NATIVE_CP` baseline includes a fresh subprocess launch. Multiplying it by 16 gives **cold-launch projections** of 592.570 seconds for Colocalization and 985.625 seconds for ImagingFlow. Those are not measured native plate runs. [PR #197](https://github.com/OpenHCSDev/openhcs/pull/197) instead measured warm native four-job makespans of **451.878** and **388.431 seconds**, respectively, on the same 16 synthetic wells. Relative to this PR's OpenHCS candidate execution observations, the measured warm native ratios are **13.49×** for Colocalization and **3.08×** for ImagingFlow. The later spreadsheet-folding improvement in PR #195 lowers ImagingFlow OpenHCS execution further to 115.808 seconds, giving a **3.35×** measured ratio. The figure below retains the older projection as a labeled historical reference; PR #197 contains the measured scaling figure and timing-boundary details. + +![Multiwell execution and worker-transfer gaps](multiwell_execution_and_transfer.png) + +`official30_control.csv`, `official30_candidate.csv`, `multiwell.csv`, and `transport_payload.csv` retain the observations; `plot.py` regenerates the figures. The raw runs are under `/home/ts/code/projects/openhcs-benchmark-runs/perf-single-axis-full-candidate-20260929`, `perf-plate-transport-full-candidate-20260929`, `perf-single-axis-multiwell-candidate-20260929`, `perf-plate-transport-candidate-16w-repeat-20260929`, and `perf-plate-transport-imaging-{control,candidate}-20260929`. The saved native CP reference is `benchmark/results/perf_lazy_catalog_20260929/native_cp_summary.csv`. + +Seventy-six focused unit tests passed on the exact stacked review branch, including full-observation and plate-input retention policies. A real ZMQ full-value export and an analysis-consolidation integration test passed there too. A warmed 16-well Colocalization run on that exact branch measured 7.539 seconds compilation, 33.217 seconds execution, and 43.172 seconds total; its plate CSV matched the integration candidate byte for byte. The official30 and paired 16-well cases used the ordinary ZMQ benchmark route on the integration checkout. + +Reproduce an observation with: + +```bash +OPENHCS_CPU_ONLY=true \ +OPENHCS_REFERENCE_EXPORT_PIPELINES_ROOT=/absolute/path/to/benchmark/reference_exports/official30_value_completion_20260914 \ +.venv/bin/python scripts/benchmark_cppipe_well_throughput.py \ + --manifest benchmark/manifests/official30_portable_axis1.json \ + --output-dir /path/to/output \ + --native-summary-csv benchmark/results/perf_lazy_catalog_20260929/native_cp_summary.csv \ + --mode 16w_4c --case ExampleColocalization +``` diff --git a/benchmark/results/perf_plate_input_transport_20260929/multiwell.csv b/benchmark/results/perf_plate_input_transport_20260929/multiwell.csv new file mode 100644 index 0000000000..696ef43175 --- /dev/null +++ b/benchmark/results/perf_plate_input_transport_20260929/multiwell.csv @@ -0,0 +1,5 @@ +variant,case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,post_worker_gap_seconds,plate_export_seconds +control,ExampleColocalization,16w_4c,4,16,7.391266584396362,0.0,51.176520586013794,61.98291008686647,37.035618890076876,592.56990224123,11.57894080050404,14075.07421875,success,18.804231,14.916857 +candidate,ExampleColocalization,16w_4c,4,16,7.330875873565674,0.0,33.491028785705566,43.26270653307438,37.035618890076876,592.56990224123,17.69339204336855,8422.80859375,success,0.726061,15.118663 +control,ExampleImagingFlowCytometryObjectsInGrid,16w_4c,4,16,10.196629524230957,0.0,212.82674908638,227.38193692686036,61.60158259188756,985.625321470201,4.631115805232571,13723.98828125,success,88.090158,52.358813 +candidate,ExampleImagingFlowCytometryObjectsInGrid,16w_4c,4,16,10.152604818344116,0.0,126.07837319374084,139.30536010675132,61.60158259188756,985.625321470201,7.8175605895201405,14144.87890625,success,3.171111,51.494128 diff --git a/benchmark/results/perf_plate_input_transport_20260929/multiwell_execution_and_transfer.png b/benchmark/results/perf_plate_input_transport_20260929/multiwell_execution_and_transfer.png new file mode 100644 index 0000000000..9f90103a27 Binary files /dev/null and b/benchmark/results/perf_plate_input_transport_20260929/multiwell_execution_and_transfer.png differ diff --git a/benchmark/results/perf_plate_input_transport_20260929/official30_candidate.csv b/benchmark/results/perf_plate_input_transport_20260929/official30_candidate.csv new file mode 100644 index 0000000000..d25f951d7a --- /dev/null +++ b/benchmark/results/perf_plate_input_transport_20260929/official30_candidate.csv @@ -0,0 +1,31 @@ +case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status +ExampleColocalization,1w_1t,1,1,3.2597548961639404,0.0,3.1273117065429688,8.721822986844927,37.035618890076876,37.035618890076876,11.842637499994282,1244.52734375,success +ExampleCometAssay,1w_1t,1,1,2.58327579498291,0.0,2.230452299118042,6.923803344368935,13.304211451206356,13.304211451206356,5.964804293939424,1095.73828125,success +ExampleFly,1w_1t,1,1,2.9442410469055176,0.0,2.0377795696258545,7.328427569940686,8.094948755111545,8.094948755111545,3.9724359178838045,801.87890625,success +ExampleFlyURL,1w_1t,1,1,2.084643840789795,0.0,1.0988714694976807,5.346818874124438,4.900778113398701,4.900778113398701,4.459828332460901,717.64453125,success +ExampleHuman,1w_1t,1,1,2.023411989212036,0.0,1.1636254787445068,5.250995545182377,4.420192994643003,4.420192994643003,3.7986388880141813,688.3671875,success +ExampleIlluminationCorrection_Example1_AllMethod,1w_1t,1,1,1.9811787605285645,0.0,2.3273584842681885,6.1548775136470795,946.6214662808925,946.6214662808925,406.73642358046396,1103.76953125,success +ExampleIlluminationCorrection_Example1_EachMethod,1w_1t,1,1,1.915731430053711,0.0,0.1673119068145752,3.9306574533693492,4.54475614009425,4.54475614009425,27.163375438252665,635.22265625,success +ExampleIlluminationCorrection_Example2,1w_1t,1,1,2.2534866333007812,0.0,1.775156021118164,6.069607189390808,48.04419008316472,48.04419008316472,27.064770370382355,867.40234375,success +ExampleIlluminationCorrection_Example3,1w_1t,1,1,2.3206868171691895,0.0,0.7344753742218018,5.070914588402957,2.1520505752414465,2.1520505752414465,2.930051368327505,678.76171875,success +ExampleImagingFlowCytometryObjectsInGrid,1w_1t,1,1,3.151324987411499,0.0,19.375733852386475,25.135063129942864,61.60158259188756,61.60158259188756,3.179316100293162,2356.98828125,success +ExampleNeighbors,1w_1t,1,1,1.9026966094970703,0.0,1.42476224899292,5.314375401940197,4.5466188420541584,4.5466188420541584,3.191142132849108,726.25390625,success +ExamplePercentPositive,1w_1t,1,1,2.4588496685028076,0.0,0.5351963043212891,5.175749351270497,2.5050158216618,2.5050158216618,4.680555155997469,709.015625,success +ExampleSpeckles,1w_1t,1,1,1.7379801273345947,0.0,0.687899112701416,4.380787087138742,3.179383432958275,3.179383432958275,4.62187459505896,774.61328125,success +ExampleTrackObjects,1w_1t,1,1,1.885824203491211,0.0,2.172006607055664,5.999519484117627,9.587221414782107,9.587221414782107,4.413992749211009,1475.078125,success +ExampleTumor,1w_1t,1,1,1.7517242431640625,0.0,0.8000016212463379,4.5107450829818845,4.360487373080105,4.360487373080105,5.450598170397227,853.3125,success +ExampleUntangleAndStraightenWorms,1w_1t,1,1,2.5558722019195557,0.0,1.2160651683807373,5.911333395168185,3.4416790287941694,3.4416790287941694,2.8301764726778322,750.53515625,success +ExampleUntangleWorms,1w_1t,1,1,2.7672553062438965,0.0,2.044220209121704,6.991439655888826,4.618807300925255,4.618807300925255,2.2594470401550906,805.34765625,success +ExampleUntangleWormsBrightField,1w_1t,1,1,2.7660014629364014,0.0,3.6838223934173584,8.677123927045614,7.377188352867961,7.377188352867961,2.0025906694226894,919.84765625,success +ExampleVitra,1w_1t,1,1,2.87866473197937,0.0,1.3409881591796875,6.444980601314455,4.122247557155788,4.122247557155788,3.074037253004127,844.765625,success +ExampleWoundHealing,1w_1t,1,1,1.626863718032837,0.0,1.0248918533325195,4.569093783851713,4.702593941241503,4.702593941241503,4.5883806432362935,1127.16015625,success +ExampleYeastColonies,1w_1t,1,1,3.194967269897461,0.0,1.9961652755737305,7.521727431099862,12.096297012176365,12.096297012176365,6.05976727488143,1036.7109375,success +ExampleYeastPatches,1w_1t,1,1,3.2484090328216553,0.0,2.507408618927002,8.123654478229582,5.990533472970128,5.990533472970128,2.389133317860917,993.30078125,success +cp_tutorial_3d_monolayer,1w_1t,1,1,2.690370559692383,0.0,9.333230018615723,14.425248390063643,14.058660756796598,14.058660756796598,1.5063017549932556,1291.5859375,success +cp_tutorial_advanced_segmentation_final,1w_1t,1,1,3.3375144004821777,0.0,9.285533428192139,15.477915027644485,36.253466514870524,36.253466514870524,3.904295514655096,1004.75390625,success +cp_tutorial_quality_control,1w_1t,1,1,1.2691724300384521,0.0,0.6221444606781006,3.73303104005754,4.787729676347226,4.787729676347226,7.6955272914089505,662.765625,success +cp_tutorial_beginner_segmentation_final,1w_1t,1,1,2.4608864784240723,0.0,5.43627405166626,10.31802385719493,16.074153264053166,16.074153264053166,2.956832770254898,863.7265625,success +cp_tutorial_pixel_based_classification,1w_1t,1,1,1.845350742340088,0.0,0.7938871383666992,4.585539441090077,4.208680049050599,4.208680049050599,5.301358147342343,911.42578125,success +cp_tutorial_translocation_final,1w_1t,1,1,2.7141711711883545,0.0,0.8469216823577881,5.813108752015978,3.004684013314545,3.004684013314545,3.54777079853435,798.50390625,success +cp4_supplement_combine_objects,1w_1t,1,1,1.273928165435791,0.0,0.20880627632141113,3.234272418078035,1.8334413380362093,1.8334413380362093,8.780585384387734,643.35546875,success +cp_tutorial_translocation_start,1w_1t,1,1,1.4258105754852295,0.0,0.3311126232147217,3.5676592998206615,2.1760300607420504,2.1760300607420504,6.5718728558739565,679.41796875,success diff --git a/benchmark/results/perf_plate_input_transport_20260929/official30_control.csv b/benchmark/results/perf_plate_input_transport_20260929/official30_control.csv new file mode 100644 index 0000000000..360caad3d0 --- /dev/null +++ b/benchmark/results/perf_plate_input_transport_20260929/official30_control.csv @@ -0,0 +1,31 @@ +case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status +ExampleColocalization,1w_1t,1,1,3.2704761028289795,0.0,3.656903028488159,9.323421066626906,37.035618890076876,37.035618890076876,10.127591188926926,1249.15625,success +ExampleCometAssay,1w_1t,1,1,2.5978925228118896,0.0,2.4985811710357666,7.230326514691114,13.304211451206356,13.304211451206356,5.3247065196169725,1130.08984375,success +ExampleFly,1w_1t,1,1,2.928973913192749,0.0,2.1651382446289062,7.507112581282854,8.094948755111545,8.094948755111545,3.738767616891354,813.9453125,success +ExampleFlyURL,1w_1t,1,1,2.0594329833984375,0.0,1.0964572429656982,5.262936326675117,4.900778113398701,4.900778113398701,4.469648173551276,728.890625,success +ExampleHuman,1w_1t,1,1,2.0272867679595947,0.0,1.1660683155059814,5.281382285989821,4.420192994643003,4.420192994643003,3.7906809882961174,699.44921875,success +ExampleIlluminationCorrection_Example1_AllMethod,1w_1t,1,1,1.958434820175171,0.0,2.668771266937256,6.5271095228381455,946.6214662808925,946.6214662808925,354.70310925793854,1178.74609375,success +ExampleIlluminationCorrection_Example1_EachMethod,1w_1t,1,1,1.8953959941864014,0.0,0.16789460182189941,3.8588457694277167,4.54475614009425,4.54475614009425,27.069102227094074,644.79296875,success +ExampleIlluminationCorrection_Example2,1w_1t,1,1,2.2534286975860596,0.0,1.8507678508758545,6.131418786942959,48.04419008316472,48.04419008316472,25.959058052812168,856.3046875,success +ExampleIlluminationCorrection_Example3,1w_1t,1,1,2.3259546756744385,0.0,0.7421314716339111,5.059442943893373,2.1520505752414465,2.1520505752414465,2.8998238957625553,689.515625,success +ExampleImagingFlowCytometryObjectsInGrid,1w_1t,1,1,3.1455113887786865,0.0,21.887840747833252,27.72792947711423,61.60158259188756,61.60158259188756,2.8144202665576183,2394.8515625,success +ExampleNeighbors,1w_1t,1,1,1.9047722816467285,0.0,1.4113435745239258,5.292001169640571,4.5466188420541584,4.5466188420541584,3.221482652505662,736.76953125,success +ExamplePercentPositive,1w_1t,1,1,2.4342103004455566,0.0,0.5324540138244629,5.061868476681411,2.5050158216618,2.5050158216618,4.7046613540745,721.30078125,success +ExampleSpeckles,1w_1t,1,1,1.7089776992797852,0.0,0.6968872547149658,4.287914528977126,3.179383432958275,3.179383432958275,4.562263711163259,814.83984375,success +ExampleTrackObjects,1w_1t,1,1,1.900160312652588,0.0,2.9917476177215576,6.925043139141053,9.587221414782107,9.587221414782107,3.204555544054716,1477.71484375,success +ExampleTumor,1w_1t,1,1,1.7615485191345215,0.0,0.8002479076385498,4.493931999895722,4.360487373080105,4.360487373080105,5.448920680027093,857.25,success +ExampleUntangleAndStraightenWorms,1w_1t,1,1,2.660576820373535,0.0,1.2797966003417969,6.081565018277615,3.4416790287941694,3.4416790287941694,2.6892390774244874,761.98828125,success +ExampleUntangleWorms,1w_1t,1,1,2.712616205215454,0.0,2.1271276473999023,7.022926961071789,4.618807300925255,4.618807300925255,2.171382289431977,816.29296875,success +ExampleUntangleWormsBrightField,1w_1t,1,1,2.740830183029175,0.0,3.6641621589660645,8.660505207721144,7.377188352867961,7.377188352867961,2.0133356638751,928.48046875,success +ExampleVitra,1w_1t,1,1,2.868414878845215,0.0,1.3481652736663818,6.441189814824611,4.122247557155788,4.122247557155788,3.057672258494831,855.33203125,success +ExampleWoundHealing,1w_1t,1,1,1.6260030269622803,0.0,1.2126474380493164,4.643486723769456,4.702593941241503,4.702593941241503,3.877956439512352,1131.7890625,success +ExampleYeastColonies,1w_1t,1,1,3.2015538215637207,0.0,2.005239248275757,7.525242286268622,12.096297012176365,12.096297012176365,6.032346026828268,1040.9765625,success +ExampleYeastPatches,1w_1t,1,1,3.22324800491333,0.0,2.621635913848877,8.204678097739816,5.990533472970128,5.990533472970128,2.2850363932401674,1011.1171875,success +cp_tutorial_3d_monolayer,1w_1t,1,1,2.6850473880767822,0.0,9.711158752441406,14.777937124948949,14.058660756796598,14.058660756796598,1.4476810764999821,1347.82421875,success +cp_tutorial_advanced_segmentation_final,1w_1t,1,1,3.3439583778381348,0.0,9.420271396636963,15.600868972949684,36.253466514870524,36.253466514870524,3.8484524477514537,1015.14453125,success +cp_tutorial_quality_control,1w_1t,1,1,1.2613310813903809,0.0,0.6129217147827148,3.709976974874735,4.787729676347226,4.787729676347226,7.811323307487173,672.83984375,success +cp_tutorial_beginner_segmentation_final,1w_1t,1,1,2.4626216888427734,0.0,5.456287384033203,10.324893586803228,16.074153264053166,16.074153264053166,2.9459872863535645,897.73828125,success +cp_tutorial_pixel_based_classification,1w_1t,1,1,1.8467652797698975,0.0,0.7934067249298096,4.601227829232812,4.208680049050599,4.208680049050599,5.304568157552394,918.87109375,success +cp_tutorial_translocation_final,1w_1t,1,1,2.631880521774292,0.0,0.8159711360931396,5.705801505129784,3.004684013314545,3.004684013314545,3.682341053999701,808.73828125,success +cp4_supplement_combine_objects,1w_1t,1,1,1.251295566558838,0.0,0.206801176071167,3.2015029741451144,1.8334413380362093,1.8334413380362093,8.86572007407377,655.7734375,success +cp_tutorial_translocation_start,1w_1t,1,1,1.5043699741363525,0.0,0.26001715660095215,3.533711294643581,2.1760300607420504,2.1760300607420504,8.368794156462528,700.0546875,success diff --git a/benchmark/results/perf_plate_input_transport_20260929/official30_phase_totals.png b/benchmark/results/perf_plate_input_transport_20260929/official30_phase_totals.png new file mode 100644 index 0000000000..56fc1dc4bc Binary files /dev/null and b/benchmark/results/perf_plate_input_transport_20260929/official30_phase_totals.png differ diff --git a/benchmark/results/perf_plate_input_transport_20260929/plot.py b/benchmark/results/perf_plate_input_transport_20260929/plot.py new file mode 100644 index 0000000000..aa68f2037f --- /dev/null +++ b/benchmark/results/perf_plate_input_transport_20260929/plot.py @@ -0,0 +1,120 @@ +"""Regenerate the plate-input transport benchmark figures from committed CSVs.""" + +from __future__ import annotations + +import csv +from pathlib import Path + +import matplotlib + +matplotlib.use("Agg") +import matplotlib.pyplot as plt +import numpy as np + +ROOT = Path(__file__).resolve().parent +CONTROL_COLOR = "#4263a6" +CANDIDATE_COLOR = "#159c85" + + +def read_rows(name: str) -> list[dict[str, str]]: + with (ROOT / name).open(newline="") as stream: + return list(csv.DictReader(stream)) + + +def plot_official30() -> None: + rows = { + label: read_rows(f"official30_{label}.csv") + for label in ("control", "candidate") + } + totals = {} + for label, observations in rows.items(): + compile_seconds = sum(float(row["compile_seconds"]) for row in observations) + execute_seconds = sum(float(row["execute_seconds"]) for row in observations) + total_seconds = sum(float(row["total_seconds"]) for row in observations) + totals[label] = ( + compile_seconds, + execute_seconds, + total_seconds - compile_seconds - execute_seconds, + total_seconds, + ) + + fig, ax = plt.subplots(figsize=(9, 4.8)) + positions = np.arange(4) + width = 0.35 + for offset, label, color in ( + (-width / 2, "control", CONTROL_COLOR), + (width / 2, "candidate", CANDIDATE_COLOR), + ): + bars = ax.bar( + positions + offset, + totals[label], + width, + label=label.title(), + color=color, + ) + ax.bar_label(bars, fmt="%.1f", padding=3, fontsize=8) + ax.set_xticks(positions, ("Compile", "Execute", "Other", "Total")) + ax.set_ylabel("Summed seconds over 30 one-well observations") + ax.set_title("Fresh-server official30: plate-input record transport") + ax.legend(frameon=False) + ax.set_ylim(0, 245) + ax.spines[["top", "right"]].set_visible(False) + fig.tight_layout() + fig.savefig(ROOT / "official30_phase_totals.png", dpi=170) + plt.close(fig) + + +def plot_multiwell() -> None: + rows = read_rows("multiwell.csv") + cases = ("ExampleColocalization", "ExampleImagingFlowCytometryObjectsInGrid") + labels = ("Colocalization", "ImagingFlow") + by_case = {(row["case_name"], row["variant"]): row for row in rows} + fig, axes = plt.subplots(1, 2, figsize=(11, 4.8)) + positions = np.arange(len(cases)) + width = 0.36 + for axis, field, title in ( + (axes[0], "execute_seconds", "Server execution"), + (axes[1], "post_worker_gap_seconds", "Worker finish to plate export"), + ): + for offset, variant, color in ( + (-width / 2, "control", CONTROL_COLOR), + (width / 2, "candidate", CANDIDATE_COLOR), + ): + values = [float(by_case[(case, variant)][field]) for case in cases] + bars = axis.bar( + positions + offset, + values, + width, + color=color, + label=variant.title(), + ) + axis.bar_label(bars, fmt="%.1f", padding=3, fontsize=8) + axis.set_xticks(positions, labels) + axis.set_title(title) + axis.set_ylabel("Seconds") + axis.spines[["top", "right"]].set_visible(False) + axis.set_ylim(0, max(float(row[field]) for row in rows) * 1.18) + native_speedups = tuple( + float(by_case[(case, "candidate")]["projected_native_execution_seconds"]) + / float(by_case[(case, "candidate")]["execute_seconds"]) + for case in cases + ) + axes[0].set_xticks( + positions, + tuple( + f"{label}\nCP cold projection: {speedup:.1f}×" + for label, speedup in zip(labels, native_speedups, strict=True) + ), + ) + axes[0].legend(frameon=False) + fig.suptitle( + "16 wells, four fork workers; native CP is a 16× cold-launch projection" + ) + fig.tight_layout() + fig.savefig(ROOT / "multiwell_execution_and_transfer.png", dpi=170) + plt.close(fig) + + +if __name__ == "__main__": + plot_official30() + plot_multiwell() diff --git a/benchmark/results/perf_plate_input_transport_20260929/transport_payload.csv b/benchmark/results/perf_plate_input_transport_20260929/transport_payload.csv new file mode 100644 index 0000000000..03de92c4f8 --- /dev/null +++ b/benchmark/results/perf_plate_input_transport_20260929/transport_payload.csv @@ -0,0 +1,7 @@ +artifact_type,records,pickle_bytes,data_types +ImageArtifactType,32,382249189,"ImageMetadataPayload,MaskedImagePayload" +MeasurementsArtifactType,68,40925527,MeasurementTable +ObjectLabelsArtifactType,32,1112183351,ObjectLabelSet +RelationshipsArtifactType,16,285033,ObjectRelationship +ObjectLineageArtifactType,8,145358,ObjectRelationship +total,156,1540124588, diff --git a/openhcs/core/compiled_execution.py b/openhcs/core/compiled_execution.py index 502c12b7e4..986a7157af 100644 --- a/openhcs/core/compiled_execution.py +++ b/openhcs/core/compiled_execution.py @@ -147,16 +147,23 @@ def axis_ids(self) -> tuple[str, ...]: def requires_parent_runtime_observation(self) -> bool: """Return whether compiled scopes require worker records in the parent.""" - return any( + return self.requires_full_parent_runtime_observation or any( plan.execution_scope.requires_parent_runtime_observation - or ( - context.analysis_consolidation_config.enabled - and plan.runtime_artifact_materialization.has_persistent_target - and any( - output.materialization is not None - and output.materialization.participates_in_runtime_export_observation() - for output in plan.artifact_outputs.values() - ) + for context in self.runtime_contexts.values() + for plan in context.step_plans.values() + ) + + @property + def requires_full_parent_runtime_observation(self) -> bool: + """Return whether consolidation needs records beyond plate inputs.""" + + return any( + context.analysis_consolidation_config.enabled + and plan.runtime_artifact_materialization.has_persistent_target + and any( + output.materialization is not None + and output.materialization.participates_in_runtime_export_observation() + for output in plan.artifact_outputs.values() ) for context in self.runtime_contexts.values() for plan in context.step_plans.values() diff --git a/openhcs/core/orchestrator/execution_result.py b/openhcs/core/orchestrator/execution_result.py index b5c4af3471..4f691525ce 100644 --- a/openhcs/core/orchestrator/execution_result.py +++ b/openhcs/core/orchestrator/execution_result.py @@ -10,11 +10,14 @@ from enum import Enum from pathlib import Path from types import MappingProxyType -from typing import Mapping, Optional +from typing import TYPE_CHECKING, Mapping, Optional from openhcs.core.context.processing_context import ProcessingContext from openhcs.core.runtime_stores import StoredRuntimeValue +if TYPE_CHECKING: + from openhcs.core.compiled_execution import CompiledExecutionBundle + class RuntimeExecutionTransportSerialization: """Pickle reducers required for worker-to-parent execution results.""" @@ -78,11 +81,12 @@ class RuntimeObservationMode(Enum): """Controls whether worker runtime records are returned to the parent.""" MERGE_INTO_PARENT = "merge_into_parent" + MERGE_PLATE_INPUTS = "merge_plate_inputs" OMIT = "omit" @property def collects_records(self) -> bool: - return self is RuntimeObservationMode.MERGE_INTO_PARENT + return self is not RuntimeObservationMode.OMIT @property def releases_worker_records(self) -> bool: @@ -94,13 +98,47 @@ def from_parent_requirement(cls, required: bool) -> "RuntimeObservationMode": return cls.MERGE_INTO_PARENT if required else cls.OMIT + @classmethod + def for_compiled_bundle( + cls, bundle: "CompiledExecutionBundle" + ) -> "RuntimeObservationMode": + """Retain only plate inputs unless another parent consumer needs all records.""" + + if bundle.requires_full_parent_runtime_observation: + return cls.MERGE_INTO_PARENT + if bundle.requires_parent_runtime_observation: + return cls.MERGE_PLATE_INPUTS + return cls.OMIT + def including_parent_requirement( self, required: bool, ) -> "RuntimeObservationMode": """Return this mode strengthened by an additional retention requirement.""" - return type(self).from_parent_requirement(self.collects_records or required) + return type(self).MERGE_INTO_PARENT if required else self + + def retain_records( + self, + records: tuple[StoredRuntimeValue, ...], + context: ProcessingContext, + ) -> tuple[StoredRuntimeValue, ...]: + """Keep records consumed by the selected parent-side execution route.""" + + if self is RuntimeObservationMode.OMIT: + return () + if self is RuntimeObservationMode.MERGE_INTO_PARENT: + return records + required_types = { + spec.artifact_type + for plan in context.step_plans.values() + if plan.execution_scope.requires_parent_runtime_observation + for invocation in plan.compiled_function_pattern.default_group.invocations + for spec in invocation.contract.artifact_inputs + } + return tuple( + record for record in records if record.key.artifact_type in required_types + ) @dataclass(frozen=True) diff --git a/openhcs/core/orchestrator/orchestrator.py b/openhcs/core/orchestrator/orchestrator.py index 09d3836fb2..2169f6ab89 100644 --- a/openhcs/core/orchestrator/orchestrator.py +++ b/openhcs/core/orchestrator/orchestrator.py @@ -710,9 +710,7 @@ def execute_compiled_plate( log_file_base=log_file_base, progress_queue=progress_queue, runtime_observation_mode=( - RuntimeObservationMode.from_parent_requirement( - execution_bundle.requires_parent_runtime_observation - ) + RuntimeObservationMode.for_compiled_bundle(execution_bundle) if runtime_observation_mode is None else runtime_observation_mode ), diff --git a/openhcs/core/orchestrator/worker_execution.py b/openhcs/core/orchestrator/worker_execution.py index 2c2beaff03..7f82c7bafc 100644 --- a/openhcs/core/orchestrator/worker_execution.py +++ b/openhcs/core/orchestrator/worker_execution.py @@ -818,8 +818,9 @@ def _execute_axis_with_sequential_combinations( frozen_context.release_execution_image_cache() if release_axis_resources: _release_runtime_resources((frozen_context,), owner=f"axis {axis_id}") - retained_records = ( - observed_records if runtime_observation_mode.collects_records else () + retained_records = runtime_observation_mode.retain_records( + observed_records, + frozen_context, ) if retained_records or runtime_export_paths: runtime_observations.append( diff --git a/openhcs/runtime/zmq_execution_signature.py b/openhcs/runtime/zmq_execution_signature.py index d002f5f23e..db76c61f2f 100644 --- a/openhcs/runtime/zmq_execution_signature.py +++ b/openhcs/runtime/zmq_execution_signature.py @@ -73,8 +73,8 @@ def runtime_observation_mode_for( ) -> RuntimeObservationMode: """Resolve retention from compiled needs plus an explicit export request.""" - return RuntimeObservationMode.from_parent_requirement( - execution_bundle.requires_parent_runtime_observation + return RuntimeObservationMode.for_compiled_bundle( + execution_bundle ).including_parent_requirement( self.runtime_observation_export_path is not None and self.runtime_observation_export_scope diff --git a/tests/unit/test_compiled_execution.py b/tests/unit/test_compiled_execution.py index e795860a76..29f7bbd1c4 100644 --- a/tests/unit/test_compiled_execution.py +++ b/tests/unit/test_compiled_execution.py @@ -114,7 +114,7 @@ def test_compiled_execution_bundle_derives_axis_ids_from_contexts() -> None: FunctionStepExecutionScope.AXIS, FunctionStepExecutionScope.PLATE, ), - RuntimeObservationMode.MERGE_INTO_PARENT, + RuntimeObservationMode.MERGE_PLATE_INPUTS, ), ), ) @@ -141,12 +141,7 @@ def test_compiled_execution_bundle_derives_runtime_observation_mode( runtime_environment=_runtime_environment(), ) - assert ( - RuntimeObservationMode.from_parent_requirement( - bundle.requires_parent_runtime_observation - ) - is expected_mode - ) + assert RuntimeObservationMode.for_compiled_bundle(bundle) is expected_mode @pytest.mark.parametrize( @@ -193,6 +188,43 @@ def test_consolidation_retains_declared_persistent_export_records( ) assert bundle.requires_parent_runtime_observation is expected + assert bundle.requires_full_parent_runtime_observation is expected + assert RuntimeObservationMode.for_compiled_bundle(bundle) is ( + RuntimeObservationMode.MERGE_INTO_PARENT + if expected + else RuntimeObservationMode.OMIT + ) + + +def test_plate_input_retention_uses_compiled_consumer_types() -> None: + plate_input = SimpleNamespace(artifact_type=MeasurementsArtifactType) + plate_plan = SimpleNamespace( + execution_scope=FunctionStepExecutionScope.PLATE, + compiled_function_pattern=SimpleNamespace( + default_group=SimpleNamespace( + invocations=( + SimpleNamespace( + contract=SimpleNamespace(artifact_inputs=(plate_input,)) + ), + ) + ) + ), + ) + context = ProcessingContext(axis_id="A01", step_plans={0: plate_plan}) + image_record = SimpleNamespace(key=SimpleNamespace(artifact_type=ImageArtifactType)) + measurement_record = SimpleNamespace( + key=SimpleNamespace(artifact_type=MeasurementsArtifactType) + ) + records = (image_record, measurement_record) + + assert RuntimeObservationMode.MERGE_PLATE_INPUTS.retain_records( + records, context + ) == (measurement_record,) + assert ( + RuntimeObservationMode.MERGE_INTO_PARENT.retain_records(records, context) + == records + ) + assert RuntimeObservationMode.OMIT.retain_records(records, context) == () def test_runtime_observation_mode_can_only_be_strengthened() -> None: diff --git a/tests/unit/test_zmq_execution_server.py b/tests/unit/test_zmq_execution_server.py index 51ab963ed3..45e8070fa3 100644 --- a/tests/unit/test_zmq_execution_server.py +++ b/tests/unit/test_zmq_execution_server.py @@ -97,6 +97,16 @@ def test_zmq_execution_context_seeds_saved_global_config_for_compilation() -> No "/tmp/runtime-observation.pkl", RuntimeObservationMode.MERGE_INTO_PARENT, ), + ( + RuntimeObservationMode.MERGE_PLATE_INPUTS, + None, + RuntimeObservationMode.MERGE_PLATE_INPUTS, + ), + ( + RuntimeObservationMode.MERGE_PLATE_INPUTS, + "/tmp/runtime-observation.pkl", + RuntimeObservationMode.MERGE_INTO_PARENT, + ), ( RuntimeObservationMode.MERGE_INTO_PARENT, None, @@ -115,7 +125,10 @@ def test_zmq_auxiliary_params_strengthen_compiled_observation_requirement( else None ) execution_bundle = SimpleNamespace( - requires_parent_runtime_observation=compiled_mode.collects_records + requires_parent_runtime_observation=compiled_mode.collects_records, + requires_full_parent_runtime_observation=( + compiled_mode is RuntimeObservationMode.MERGE_INTO_PARENT + ), ) assert params.runtime_observation_mode_for(execution_bundle) is expected_mode @@ -126,7 +139,10 @@ def test_outcome_export_does_not_strengthen_worker_runtime_value_retention() -> runtime_observation_export_path=Path("/tmp/outcomes.pkl.gz"), runtime_observation_export_scope=ZMQRuntimeObservationExportScope.OUTCOMES, ) - execution_bundle = SimpleNamespace(requires_parent_runtime_observation=False) + execution_bundle = SimpleNamespace( + requires_parent_runtime_observation=False, + requires_full_parent_runtime_observation=False, + ) assert ( params.runtime_observation_mode_for(execution_bundle)