Avoid transporting unused worker artifacts into plate exports

benchmark/results/perf_plate_input_transport_20260929/README.md

@@ -0,0 +1,43 @@
+# Transfer only declared plate inputs from worker lanes (2026-09-29)
+
+This report supports [OpenHCS issue #192](https://github.com/OpenHCSDev/openhcs/issues/192) and [PR #193](https://github.com/OpenHCSDev/openhcs/pull/193). The control is integrated commit `cc0615a9f`, which contains the single-axis compilation fix in [PR #191](https://github.com/OpenHCSDev/openhcs/pull/191) and separate pending performance changes from PRs #157, #170, and #175. The candidate adds only the plate-input transport change to that integration checkout. The review branch includes #157 as a prerequisite because #191 is not based on it. Four-worker observations use `fork`, 16 wells, one native thread per worker, and a fresh client-owned execution server. One-well observations also use a fresh server. The change affects execution and total time; it does not change compilation.
+
+`ExampleColocalization` exposed a **18.8–25.6-second gap** between the last worker completing and the plate spreadsheet starting. Instrumenting the worker pipe showed sequential result serialization and receipt. One four-well lane sent about **1.54 GB** of pickled runtime records: 1.11 GB object labels, 382 MB images, 41 MB measurements, and less than 1 MB relationships. The spreadsheet's compiled artifact inputs require measurements and relationships. The new `MERGE_PLATE_INPUTS` observation mode derives retained artifact types from those compiled inputs. Explicit full-value observation and analysis consolidation still retain every record. Runtime export paths are found before filtering.
+
+| Fresh-server official30, one well per case (summed seconds) | Control | Candidate | Change |
+| --- | ---: | ---: | ---: |
+| Server compilation jobs | 70.193 | 70.310 | +0.117 |
+| Server execution jobs | 85.869 | 80.329 | **-5.539** |
+| Other measured time | 63.944 | 64.069 | +0.125 |
+| Per-observation total | 220.006 | 214.708 | **-5.298** |
+
+All **30/30** cases succeeded; **110/110** generated result files matched by relative path and bytes. Execution improved in 21 cases, with the largest reductions in ImagingFlow (-2.512 seconds), TrackObjects (-0.820 seconds), and Colocalization (-0.530 seconds). The full sweep followed a candidate warmup through the same corpus so cold JIT compilation was outside the paired measurement.
+
+![Official30 compile, execution, and total times](official30_phase_totals.png)
+
+| 16 wells, four fork workers | Control execution | Candidate execution | Control total | Candidate total | Post-worker gap |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Colocalization | 51.177 s | **33.491 s** | 61.983 s | **43.263 s** | 18.804 → 0.726 s |
+| ImagingFlow | 212.827 s | **126.078 s** | 227.382 s | **139.305 s** | 88.090 → 3.171 s |
+
+The plate-level export remained near 15 seconds for Colocalization and 52 seconds for ImagingFlow; the gain comes from worker-result transfer. Both 16-well plate exports matched the control byte for byte. The ImagingFlow candidate's measured process-tree peak memory was slightly higher (14.14 versus 13.72 GB), so this result does not establish a memory reduction.
+
+The earlier native CellProfiler 4.2.8.1 single-sample `EXECUTE_NATIVE_CP` baseline includes a fresh subprocess launch. Multiplying it by 16 gives **cold-launch projections** of 592.570 seconds for Colocalization and 985.625 seconds for ImagingFlow. Those are not measured native plate runs. [PR #197](https://github.com/OpenHCSDev/openhcs/pull/197) instead measured warm native four-job makespans of **451.878** and **388.431 seconds**, respectively, on the same 16 synthetic wells. Relative to this PR's OpenHCS candidate execution observations, the measured warm native ratios are **13.49×** for Colocalization and **3.08×** for ImagingFlow. The later spreadsheet-folding improvement in PR #195 lowers ImagingFlow OpenHCS execution further to 115.808 seconds, giving a **3.35×** measured ratio. The figure below retains the older projection as a labeled historical reference; PR #197 contains the measured scaling figure and timing-boundary details.
+
+![Multiwell execution and worker-transfer gaps](multiwell_execution_and_transfer.png)
+
+`official30_control.csv`, `official30_candidate.csv`, `multiwell.csv`, and `transport_payload.csv` retain the observations; `plot.py` regenerates the figures. The raw runs are under `/home/ts/code/projects/openhcs-benchmark-runs/perf-single-axis-full-candidate-20260929`, `perf-plate-transport-full-candidate-20260929`, `perf-single-axis-multiwell-candidate-20260929`, `perf-plate-transport-candidate-16w-repeat-20260929`, and `perf-plate-transport-imaging-{control,candidate}-20260929`. The saved native CP reference is `benchmark/results/perf_lazy_catalog_20260929/native_cp_summary.csv`.
+
+Seventy-six focused unit tests passed on the exact stacked review branch, including full-observation and plate-input retention policies. A real ZMQ full-value export and an analysis-consolidation integration test passed there too. A warmed 16-well Colocalization run on that exact branch measured 7.539 seconds compilation, 33.217 seconds execution, and 43.172 seconds total; its plate CSV matched the integration candidate byte for byte. The official30 and paired 16-well cases used the ordinary ZMQ benchmark route on the integration checkout.
+
+Reproduce an observation with:
+
+```bash
+OPENHCS_CPU_ONLY=true \
+OPENHCS_REFERENCE_EXPORT_PIPELINES_ROOT=/absolute/path/to/benchmark/reference_exports/official30_value_completion_20260914 \
+.venv/bin/python scripts/benchmark_cppipe_well_throughput.py \
+  --manifest benchmark/manifests/official30_portable_axis1.json \
+  --output-dir /path/to/output \
+  --native-summary-csv benchmark/results/perf_lazy_catalog_20260929/native_cp_summary.csv \
+  --mode 16w_4c --case ExampleColocalization
+```

benchmark/results/perf_plate_input_transport_20260929/multiwell.csv

@@ -0,0 +1,5 @@
+variant,case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,post_worker_gap_seconds,plate_export_seconds
+control,ExampleColocalization,16w_4c,4,16,7.391266584396362,0.0,51.176520586013794,61.98291008686647,37.035618890076876,592.56990224123,11.57894080050404,14075.07421875,success,18.804231,14.916857
+candidate,ExampleColocalization,16w_4c,4,16,7.330875873565674,0.0,33.491028785705566,43.26270653307438,37.035618890076876,592.56990224123,17.69339204336855,8422.80859375,success,0.726061,15.118663
+control,ExampleImagingFlowCytometryObjectsInGrid,16w_4c,4,16,10.196629524230957,0.0,212.82674908638,227.38193692686036,61.60158259188756,985.625321470201,4.631115805232571,13723.98828125,success,88.090158,52.358813
+candidate,ExampleImagingFlowCytometryObjectsInGrid,16w_4c,4,16,10.152604818344116,0.0,126.07837319374084,139.30536010675132,61.60158259188756,985.625321470201,7.8175605895201405,14144.87890625,success,3.171111,51.494128

benchmark/results/perf_plate_input_transport_20260929/official30_candidate.csv

@@ -0,0 +1,31 @@
+case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status
+ExampleColocalization,1w_1t,1,1,3.2597548961639404,0.0,3.1273117065429688,8.721822986844927,37.035618890076876,37.035618890076876,11.842637499994282,1244.52734375,success
+ExampleCometAssay,1w_1t,1,1,2.58327579498291,0.0,2.230452299118042,6.923803344368935,13.304211451206356,13.304211451206356,5.964804293939424,1095.73828125,success
+ExampleFly,1w_1t,1,1,2.9442410469055176,0.0,2.0377795696258545,7.328427569940686,8.094948755111545,8.094948755111545,3.9724359178838045,801.87890625,success
+ExampleFlyURL,1w_1t,1,1,2.084643840789795,0.0,1.0988714694976807,5.346818874124438,4.900778113398701,4.900778113398701,4.459828332460901,717.64453125,success
+ExampleHuman,1w_1t,1,1,2.023411989212036,0.0,1.1636254787445068,5.250995545182377,4.420192994643003,4.420192994643003,3.7986388880141813,688.3671875,success
+ExampleIlluminationCorrection_Example1_AllMethod,1w_1t,1,1,1.9811787605285645,0.0,2.3273584842681885,6.1548775136470795,946.6214662808925,946.6214662808925,406.73642358046396,1103.76953125,success
+ExampleIlluminationCorrection_Example1_EachMethod,1w_1t,1,1,1.915731430053711,0.0,0.1673119068145752,3.9306574533693492,4.54475614009425,4.54475614009425,27.163375438252665,635.22265625,success
+ExampleIlluminationCorrection_Example2,1w_1t,1,1,2.2534866333007812,0.0,1.775156021118164,6.069607189390808,48.04419008316472,48.04419008316472,27.064770370382355,867.40234375,success
+ExampleIlluminationCorrection_Example3,1w_1t,1,1,2.3206868171691895,0.0,0.7344753742218018,5.070914588402957,2.1520505752414465,2.1520505752414465,2.930051368327505,678.76171875,success
+ExampleImagingFlowCytometryObjectsInGrid,1w_1t,1,1,3.151324987411499,0.0,19.375733852386475,25.135063129942864,61.60158259188756,61.60158259188756,3.179316100293162,2356.98828125,success
+ExampleNeighbors,1w_1t,1,1,1.9026966094970703,0.0,1.42476224899292,5.314375401940197,4.5466188420541584,4.5466188420541584,3.191142132849108,726.25390625,success
+ExamplePercentPositive,1w_1t,1,1,2.4588496685028076,0.0,0.5351963043212891,5.175749351270497,2.5050158216618,2.5050158216618,4.680555155997469,709.015625,success
+ExampleSpeckles,1w_1t,1,1,1.7379801273345947,0.0,0.687899112701416,4.380787087138742,3.179383432958275,3.179383432958275,4.62187459505896,774.61328125,success
+ExampleTrackObjects,1w_1t,1,1,1.885824203491211,0.0,2.172006607055664,5.999519484117627,9.587221414782107,9.587221414782107,4.413992749211009,1475.078125,success
+ExampleTumor,1w_1t,1,1,1.7517242431640625,0.0,0.8000016212463379,4.5107450829818845,4.360487373080105,4.360487373080105,5.450598170397227,853.3125,success
+ExampleUntangleAndStraightenWorms,1w_1t,1,1,2.5558722019195557,0.0,1.2160651683807373,5.911333395168185,3.4416790287941694,3.4416790287941694,2.8301764726778322,750.53515625,success
+ExampleUntangleWorms,1w_1t,1,1,2.7672553062438965,0.0,2.044220209121704,6.991439655888826,4.618807300925255,4.618807300925255,2.2594470401550906,805.34765625,success
+ExampleUntangleWormsBrightField,1w_1t,1,1,2.7660014629364014,0.0,3.6838223934173584,8.677123927045614,7.377188352867961,7.377188352867961,2.0025906694226894,919.84765625,success
+ExampleVitra,1w_1t,1,1,2.87866473197937,0.0,1.3409881591796875,6.444980601314455,4.122247557155788,4.122247557155788,3.074037253004127,844.765625,success
+ExampleWoundHealing,1w_1t,1,1,1.626863718032837,0.0,1.0248918533325195,4.569093783851713,4.702593941241503,4.702593941241503,4.5883806432362935,1127.16015625,success
+ExampleYeastColonies,1w_1t,1,1,3.194967269897461,0.0,1.9961652755737305,7.521727431099862,12.096297012176365,12.096297012176365,6.05976727488143,1036.7109375,success
+ExampleYeastPatches,1w_1t,1,1,3.2484090328216553,0.0,2.507408618927002,8.123654478229582,5.990533472970128,5.990533472970128,2.389133317860917,993.30078125,success
+cp_tutorial_3d_monolayer,1w_1t,1,1,2.690370559692383,0.0,9.333230018615723,14.425248390063643,14.058660756796598,14.058660756796598,1.5063017549932556,1291.5859375,success
+cp_tutorial_advanced_segmentation_final,1w_1t,1,1,3.3375144004821777,0.0,9.285533428192139,15.477915027644485,36.253466514870524,36.253466514870524,3.904295514655096,1004.75390625,success
+cp_tutorial_quality_control,1w_1t,1,1,1.2691724300384521,0.0,0.6221444606781006,3.73303104005754,4.787729676347226,4.787729676347226,7.6955272914089505,662.765625,success
+cp_tutorial_beginner_segmentation_final,1w_1t,1,1,2.4608864784240723,0.0,5.43627405166626,10.31802385719493,16.074153264053166,16.074153264053166,2.956832770254898,863.7265625,success
+cp_tutorial_pixel_based_classification,1w_1t,1,1,1.845350742340088,0.0,0.7938871383666992,4.585539441090077,4.208680049050599,4.208680049050599,5.301358147342343,911.42578125,success
+cp_tutorial_translocation_final,1w_1t,1,1,2.7141711711883545,0.0,0.8469216823577881,5.813108752015978,3.004684013314545,3.004684013314545,3.54777079853435,798.50390625,success
+cp4_supplement_combine_objects,1w_1t,1,1,1.273928165435791,0.0,0.20880627632141113,3.234272418078035,1.8334413380362093,1.8334413380362093,8.780585384387734,643.35546875,success
+cp_tutorial_translocation_start,1w_1t,1,1,1.4258105754852295,0.0,0.3311126232147217,3.5676592998206615,2.1760300607420504,2.1760300607420504,6.5718728558739565,679.41796875,success

benchmark/results/perf_plate_input_transport_20260929/official30_control.csv

@@ -0,0 +1,31 @@
+case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status
+ExampleColocalization,1w_1t,1,1,3.2704761028289795,0.0,3.656903028488159,9.323421066626906,37.035618890076876,37.035618890076876,10.127591188926926,1249.15625,success
+ExampleCometAssay,1w_1t,1,1,2.5978925228118896,0.0,2.4985811710357666,7.230326514691114,13.304211451206356,13.304211451206356,5.3247065196169725,1130.08984375,success
+ExampleFly,1w_1t,1,1,2.928973913192749,0.0,2.1651382446289062,7.507112581282854,8.094948755111545,8.094948755111545,3.738767616891354,813.9453125,success
+ExampleFlyURL,1w_1t,1,1,2.0594329833984375,0.0,1.0964572429656982,5.262936326675117,4.900778113398701,4.900778113398701,4.469648173551276,728.890625,success
+ExampleHuman,1w_1t,1,1,2.0272867679595947,0.0,1.1660683155059814,5.281382285989821,4.420192994643003,4.420192994643003,3.7906809882961174,699.44921875,success
+ExampleIlluminationCorrection_Example1_AllMethod,1w_1t,1,1,1.958434820175171,0.0,2.668771266937256,6.5271095228381455,946.6214662808925,946.6214662808925,354.70310925793854,1178.74609375,success
+ExampleIlluminationCorrection_Example1_EachMethod,1w_1t,1,1,1.8953959941864014,0.0,0.16789460182189941,3.8588457694277167,4.54475614009425,4.54475614009425,27.069102227094074,644.79296875,success
+ExampleIlluminationCorrection_Example2,1w_1t,1,1,2.2534286975860596,0.0,1.8507678508758545,6.131418786942959,48.04419008316472,48.04419008316472,25.959058052812168,856.3046875,success
+ExampleIlluminationCorrection_Example3,1w_1t,1,1,2.3259546756744385,0.0,0.7421314716339111,5.059442943893373,2.1520505752414465,2.1520505752414465,2.8998238957625553,689.515625,success
+ExampleImagingFlowCytometryObjectsInGrid,1w_1t,1,1,3.1455113887786865,0.0,21.887840747833252,27.72792947711423,61.60158259188756,61.60158259188756,2.8144202665576183,2394.8515625,success
+ExampleNeighbors,1w_1t,1,1,1.9047722816467285,0.0,1.4113435745239258,5.292001169640571,4.5466188420541584,4.5466188420541584,3.221482652505662,736.76953125,success
+ExamplePercentPositive,1w_1t,1,1,2.4342103004455566,0.0,0.5324540138244629,5.061868476681411,2.5050158216618,2.5050158216618,4.7046613540745,721.30078125,success
+ExampleSpeckles,1w_1t,1,1,1.7089776992797852,0.0,0.6968872547149658,4.287914528977126,3.179383432958275,3.179383432958275,4.562263711163259,814.83984375,success
+ExampleTrackObjects,1w_1t,1,1,1.900160312652588,0.0,2.9917476177215576,6.925043139141053,9.587221414782107,9.587221414782107,3.204555544054716,1477.71484375,success
+ExampleTumor,1w_1t,1,1,1.7615485191345215,0.0,0.8002479076385498,4.493931999895722,4.360487373080105,4.360487373080105,5.448920680027093,857.25,success
+ExampleUntangleAndStraightenWorms,1w_1t,1,1,2.660576820373535,0.0,1.2797966003417969,6.081565018277615,3.4416790287941694,3.4416790287941694,2.6892390774244874,761.98828125,success
+ExampleUntangleWorms,1w_1t,1,1,2.712616205215454,0.0,2.1271276473999023,7.022926961071789,4.618807300925255,4.618807300925255,2.171382289431977,816.29296875,success
+ExampleUntangleWormsBrightField,1w_1t,1,1,2.740830183029175,0.0,3.6641621589660645,8.660505207721144,7.377188352867961,7.377188352867961,2.0133356638751,928.48046875,success
+ExampleVitra,1w_1t,1,1,2.868414878845215,0.0,1.3481652736663818,6.441189814824611,4.122247557155788,4.122247557155788,3.057672258494831,855.33203125,success
+ExampleWoundHealing,1w_1t,1,1,1.6260030269622803,0.0,1.2126474380493164,4.643486723769456,4.702593941241503,4.702593941241503,3.877956439512352,1131.7890625,success
+ExampleYeastColonies,1w_1t,1,1,3.2015538215637207,0.0,2.005239248275757,7.525242286268622,12.096297012176365,12.096297012176365,6.032346026828268,1040.9765625,success
+ExampleYeastPatches,1w_1t,1,1,3.22324800491333,0.0,2.621635913848877,8.204678097739816,5.990533472970128,5.990533472970128,2.2850363932401674,1011.1171875,success
+cp_tutorial_3d_monolayer,1w_1t,1,1,2.6850473880767822,0.0,9.711158752441406,14.777937124948949,14.058660756796598,14.058660756796598,1.4476810764999821,1347.82421875,success
+cp_tutorial_advanced_segmentation_final,1w_1t,1,1,3.3439583778381348,0.0,9.420271396636963,15.600868972949684,36.253466514870524,36.253466514870524,3.8484524477514537,1015.14453125,success
+cp_tutorial_quality_control,1w_1t,1,1,1.2613310813903809,0.0,0.6129217147827148,3.709976974874735,4.787729676347226,4.787729676347226,7.811323307487173,672.83984375,success
+cp_tutorial_beginner_segmentation_final,1w_1t,1,1,2.4626216888427734,0.0,5.456287384033203,10.324893586803228,16.074153264053166,16.074153264053166,2.9459872863535645,897.73828125,success
+cp_tutorial_pixel_based_classification,1w_1t,1,1,1.8467652797698975,0.0,0.7934067249298096,4.601227829232812,4.208680049050599,4.208680049050599,5.304568157552394,918.87109375,success
+cp_tutorial_translocation_final,1w_1t,1,1,2.631880521774292,0.0,0.8159711360931396,5.705801505129784,3.004684013314545,3.004684013314545,3.682341053999701,808.73828125,success
+cp4_supplement_combine_objects,1w_1t,1,1,1.251295566558838,0.0,0.206801176071167,3.2015029741451144,1.8334413380362093,1.8334413380362093,8.86572007407377,655.7734375,success
+cp_tutorial_translocation_start,1w_1t,1,1,1.5043699741363525,0.0,0.26001715660095215,3.533711294643581,2.1760300607420504,2.1760300607420504,8.368794156462528,700.0546875,success