Reuse exact Costes thresholds across object sets in one step

benchmark/results/perf_coloc_step_cache_20260929/README.md

@@ -0,0 +1,44 @@
+# Step-scoped Costes threshold reuse: official30 1w_1t (2026-09-29)
+
+This report accompanies [issue #185](https://github.com/OpenHCSDev/openhcs/issues/185). It compares the same CPU-only integration code with and without a step-scoped Costes threshold cache. The control is commit `30e1e905c`, including the faster completion polling from [PR #183](https://github.com/OpenHCSDev/openhcs/pull/183). Each benchmark observation owns a fresh ZMQ execution server and uses one well and one OpenHCS thread with the `fork` start method. `total_seconds` includes server startup, compilation, execution, and shutdown; `execute_seconds` is the server execution job and can include first-use JIT work. The native CellProfiler 4.2.8.1 reference is retained in the [native summary](../perf_lazy_catalog_20260929/native_cp_summary.csv).
+
+An instrumentation-only probe on the two segmentation tutorials found **180 object-colocalization calls but only 40 distinct selected image-pair pixel inputs**: 20 pairs repeated five times and 20 repeated four times across object sets. The image payloads are reconstructed between sets, so object identity cannot establish reuse. The candidate fingerprints the exact selected pixels and keeps method, scale, and backend selection in its key. The cache lives only for one FunctionStep invocation.
+
+An interleaved control/candidate/candidate/control experiment measured the two affected pipelines. The first control had a cold compilation run and is retained as raw data but excluded from the warmed comparison. The later control and the mean of both candidates give:
+
+| Pipeline | Control execution (s) | Candidate execution (s) | Execution saved (s) | Control total (s) | Candidate total (s) |
+| --- | ---: | ---: | ---: | ---: | ---: |
+| Advanced segmentation | 10.815 | 9.384 | 1.431 | 18.642 | 17.286 |
+| Beginner segmentation | 6.778 | 5.607 | 1.171 | 13.362 | 12.060 |
+| **Both** | **17.593** | **14.991** | **2.602** | **32.004** | **29.347** |
+
+All 15 produced result files in both candidate runs matched the control byte-for-byte. A separate instrumented run confirmed the mechanism: `coloc_object_costes_thresholds` fell from **3.999 to 1.400 seconds** over 180 calls, and `coloc_object_total` fell from **5.217 to 2.620 seconds**. These are phase sums from separate profiled runs, not additional time to subtract from the benchmark.
+
+![Native CellProfiler and OpenHCS execution time for the affected tutorials](affected_case_execution.png)
+
+The full official30 sweeps showed material run-order variation, including in cases this cache cannot affect. All 30 cases succeeded in every sweep; the two warmed sweeps produced the same 110 result paths with byte-identical contents.
+
+| Sweep order | Compile sum (s) | Execution sum (s) | Total sum (s) |
+| --- | ---: | ---: | ---: |
+| First candidate | 84.079 | 97.248 | 270.468 |
+| First control, cold checkout | 99.153 | 100.178 | 288.514 |
+| Warmed control | 83.701 | 89.286 | 261.691 |
+| Warmed candidate | 84.095 | 87.857 | 260.928 |
+
+The warmed candidate saved **1.429 seconds in summed execution** and **0.763 seconds in summed total** against the immediately preceding warmed control. The two affected tutorials saved 2.654 seconds in that full-suite comparison; unrelated cases varied in the other direction. The pair-specific improvement and its profiled mechanism are reproducible. The small full-suite net difference is not a stable estimate of a global speedup. It is also much smaller than the remaining end-to-end startup and compilation terms, so this cache is a targeted improvement, not the dominant route to faster official30 total time.
+
+![Per-case execution changes in the warmed full sweep](full_suite_execution_delta.png)
+
+The warmed full-sweep native execution ratios for the affected tutorials are 3.34× to 3.84× for advanced segmentation and 2.37× to 2.90× for beginner segmentation. These use the saved native execution reference and OpenHCS's `execute_seconds`; they do not include either system's total startup boundary.
+
+The retained CSVs are `abba_*.csv`, `full_first_*.csv`, and `full_warm_*.csv`. `plot.py` regenerates both figures. Raw result trees and profiling events are under `/home/ts/code/projects/openhcs-benchmark-runs/perf-coloc-step-cache-*20260929`. Validation also included 624 focused unit tests and byte parity in the full sweep. The separate [batch executor metadata issue #176](https://github.com/OpenHCSDev/openhcs/issues/176) remains open; this cache does not restore those declarations.
+
+Reproduce the fresh-server sweep with the native reference:
+
+```bash
+OPENHCS_CPU_ONLY=true .venv/bin/python scripts/benchmark_cppipe_well_throughput.py \
+  --manifest benchmark/manifests/official30_portable_axis1.json \
+  --output-dir /path/to/output \
+  --native-summary-csv benchmark/results/perf_lazy_catalog_20260929/native_cp_summary.csv \
+  --mode 1w_1t
+```

benchmark/results/perf_coloc_step_cache_20260929/abba_1_control.csv

@@ -0,0 +1,3 @@
+case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,wells_per_second,successful_wells,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,memory_limit_mb,error_message,execution_route,server_lifecycle,run_input_sha256
+cp_tutorial_advanced_segmentation_final,1w_1t,1,1,50.092376708984375,0.0,12.277675151824951,66.6460295310244,0.08144864460364552,1,36.253466514870524,36.253466514870524,2.9527957098198527,1523.5390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,45578cef16d0f9006b11f3172f9a33471622f9c1ab3a461e2dd3316561930482
+cp_tutorial_beginner_segmentation_final,1w_1t,1,1,3.277191638946533,0.0,7.9185473918914795,14.549100143834949,0.1262857883535547,1,16.074153264053166,16.074153264053166,2.029937117066819,1014.4140625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,45578cef16d0f9006b11f3172f9a33471622f9c1ab3a461e2dd3316561930482

benchmark/results/perf_coloc_step_cache_20260929/abba_2_candidate.csv

@@ -0,0 +1,3 @@
+case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,wells_per_second,successful_wells,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,memory_limit_mb,error_message,execution_route,server_lifecycle,run_input_sha256
+cp_tutorial_advanced_segmentation_final,1w_1t,1,1,4.238177299499512,0.0,9.434734344482422,17.30373717378825,0.10599132561531162,1,36.253466514870524,36.253466514870524,3.842552974061438,1089.796875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,56a334216fae7002dfa87fe7431ab12e1df13742d256f30887eb2a2446bd4041
+cp_tutorial_beginner_segmentation_final,1w_1t,1,1,3.1875345706939697,0.0,5.5730133056640625,12.019987728912383,0.17943614076493636,1,16.074153264053166,16.074153264053166,2.8842840277658053,960.5625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,56a334216fae7002dfa87fe7431ab12e1df13742d256f30887eb2a2446bd4041

benchmark/results/perf_coloc_step_cache_20260929/abba_3_candidate.csv

@@ -0,0 +1,3 @@
+case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,wells_per_second,successful_wells,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,memory_limit_mb,error_message,execution_route,server_lifecycle,run_input_sha256
+cp_tutorial_advanced_segmentation_final,1w_1t,1,1,4.3331286907196045,0.0,9.333743572235107,17.268264922779053,0.1071381479746968,1,36.253466514870524,36.253466514870524,3.8841292600659134,1090.54296875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,56a334216fae7002dfa87fe7431ab12e1df13742d256f30887eb2a2446bd4041
+cp_tutorial_beginner_segmentation_final,1w_1t,1,1,3.1799824237823486,0.0,5.640503883361816,12.100588019937277,0.17728912534743022,1,16.074153264053166,16.074153264053166,2.8497725728845262,977.95703125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,56a334216fae7002dfa87fe7431ab12e1df13742d256f30887eb2a2446bd4041

benchmark/results/perf_coloc_step_cache_20260929/abba_4_control.csv

@@ -0,0 +1,3 @@
+case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,wells_per_second,successful_wells,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,memory_limit_mb,error_message,execution_route,server_lifecycle,run_input_sha256
+cp_tutorial_advanced_segmentation_final,1w_1t,1,1,4.226243019104004,0.0,10.815070152282715,18.641795020084828,0.09246357036241064,1,36.253466514870524,36.253466514870524,3.352124951979029,1088.5390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,45578cef16d0f9006b11f3172f9a33471622f9c1ab3a461e2dd3316561930482
+cp_tutorial_beginner_segmentation_final,1w_1t,1,1,3.3164355754852295,0.0,6.7775774002075195,13.362209782004356,0.1475453456229628,1,16.074153264053166,16.074153264053166,2.3716664989411997,957.03125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,45578cef16d0f9006b11f3172f9a33471622f9c1ab3a461e2dd3316561930482

benchmark/results/perf_coloc_step_cache_20260929/full_first_candidate.csv

@@ -0,0 +1,31 @@
+case_name,mode_name,worker_count,well_count,compile_seconds,prepare_seconds,execute_seconds,total_seconds,wells_per_second,successful_wells,native_single_sample_execution_seconds,projected_native_execution_seconds,projected_execution_speedup,peak_memory_mb,status,memory_limit_mb,error_message,execution_route,server_lifecycle,run_input_sha256
+ExampleColocalization,1w_1t,1,1,3.3314623832702637,0.0,4.047102689743042,10.49303712695837,0.2470903450348308,1,37.035618890076876,37.035618890076876,9.151143850127593,1313.76171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleCometAssay,1w_1t,1,1,3.0678517818450928,0.0,3.382840394973755,9.415838974993676,0.29560957161496776,1,13.304211451206356,13.304211451206356,3.9328522477660592,1233.125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleFly,1w_1t,1,1,2.9995768070220947,0.0,2.528999090194702,8.652359592262655,0.3954133490506761,1,8.094948755111545,8.094948755111545,3.200850797652257,911.796875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleFlyURL,1w_1t,1,1,2.6812729835510254,0.0,1.1671361923217773,6.791301331017166,0.8567980382912347,1,4.900778113398701,4.900778113398701,4.198977073660625,824.68359375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleHuman,1w_1t,1,1,2.620032548904419,0.0,1.2181837558746338,6.832155702169985,0.8208942166381279,1,4.420192994643003,4.420192994643003,3.628510865726809,795.67578125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleIlluminationCorrection_Example1_AllMethod,1w_1t,1,1,2.5418341159820557,0.0,4.662762403488159,9.888185085728765,0.2144651417906071,1,946.6214662808925,946.6214662808925,203.017306987964,1312.375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleIlluminationCorrection_Example1_EachMethod,1w_1t,1,1,2.4701032638549805,0.0,0.16818952560424805,5.320872409734875,5.945673468114845,1,4.54475614009425,4.54475614009425,27.021636001210418,757.953125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleIlluminationCorrection_Example2,1w_1t,1,1,2.85026478767395,0.0,2.3008546829223633,8.016120202839375,0.4346211029415727,1,48.04419008316472,48.04419008316472,20.88101888387962,973.43359375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleIlluminationCorrection_Example3,1w_1t,1,1,2.956389904022217,0.0,0.8741958141326904,6.6492008520290256,1.1439084743183343,1,2.1520505752414465,2.1520505752414465,2.4617488901803366,796.0390625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleImagingFlowCytometryObjectsInGrid,1w_1t,1,1,3.4203906059265137,0.0,23.51734471321106,30.361198461148888,0.042521807295627295,1,61.60158259188756,61.60158259188756,2.619410624077912,2453.83984375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleNeighbors,1w_1t,1,1,2.4734487533569336,0.0,1.379551649093628,6.744648599997163,0.724873186630602,1,4.5466188420541584,4.5466188420541584,3.2957220884345353,833.3671875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExamplePercentPositive,1w_1t,1,1,2.445389747619629,0.0,0.5640556812286377,5.9112587329000235,1.772874617310439,1,2.5050158216618,2.5050158216618,4.441078966185259,798.546875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleSpeckles,1w_1t,1,1,2.3372161388397217,0.0,1.4638993740081787,6.56630947580561,0.6831070616978165,1,3.179383432958275,3.179383432958275,2.171859274898844,916.71875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleTrackObjects,1w_1t,1,1,2.4260778427124023,0.0,5.029005765914917,10.31930301617831,0.19884646121857688,1,9.587221414782107,9.587221414782107,1.9063850512483798,1578.8359375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleTumor,1w_1t,1,1,2.303354501724243,0.0,1.0990476608276367,6.154544240795076,0.9098786482534806,1,4.360487373080105,4.360487373080105,3.9675143567444966,1030.734375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleUntangleAndStraightenWorms,1w_1t,1,1,3.006577253341675,0.0,1.2314109802246094,7.1993993683718145,0.8120765658737265,1,3.4416790287941694,3.4416790287941694,2.794906886542791,865.18359375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleUntangleWorms,1w_1t,1,1,3.2606966495513916,0.0,2.211594343185425,8.507513043936342,0.4521624877009182,1,4.618807300925255,4.618807300925255,2.088451399397527,924.35546875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleUntangleWormsBrightField,1w_1t,1,1,3.3257691860198975,0.0,3.895765781402588,10.349423209670931,0.25668894284501137,1,7.377188352867961,7.377188352867961,1.8936426794662078,1020.09765625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleVitra,1w_1t,1,1,2.9028685092926025,0.0,1.5386874675750732,7.410238386131823,0.649904558965422,1,4.122247557155788,4.122247557155788,2.6790674805796204,996.34765625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleWoundHealing,1w_1t,1,1,2.023697853088379,0.0,2.044299364089966,6.776258641853929,0.4891651475150544,1,4.702593941241503,4.702593941241503,2.3003450589708008,1167.01953125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleYeastColonies,1w_1t,1,1,3.223799705505371,0.0,2.2835490703582764,8.649375961627811,0.43791482871139065,1,12.096297012176365,12.096297012176365,5.297147834129319,1149.1171875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+ExampleYeastPatches,1w_1t,1,1,3.211822986602783,0.0,2.7104239463806152,8.996431880164891,0.368945972948386,1,5.990533472970128,5.990533472970128,2.2101832006648374,1113.39453125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+cp_tutorial_3d_monolayer,1w_1t,1,1,3.5883374214172363,0.0,10.085208892822266,16.968512172345072,0.09915511028350726,1,14.058660756796598,14.058660756796598,1.3939880577785826,1430.68359375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+cp_tutorial_advanced_segmentation_final,1w_1t,1,1,4.423793315887451,0.0,9.268096923828125,17.375662088859826,0.10789701577559212,1,36.253466514870524,36.253466514870524,3.9116408484748857,1100.3125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+cp_tutorial_quality_control,1w_1t,1,1,2.0238704681396484,0.0,0.6121540069580078,5.509663320146501,1.6335758463288101,1,4.787729676347226,4.787729676347226,7.82111955803248,807.65625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+cp_tutorial_beginner_segmentation_final,1w_1t,1,1,3.170426845550537,0.0,5.66499924659729,12.072373040020466,0.17652253009577273,1,16.074153264053166,16.074153264053166,2.8374502033178883,970.3984375,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+cp_tutorial_pixel_based_classification,1w_1t,1,1,2.3951919078826904,0.0,0.9567317962646484,6.207462176214904,1.0452250086223567,1,4.208680049050599,4.208680049050599,4.399017640557654,1020.7890625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+cp_tutorial_translocation_final,1w_1t,1,1,2.5509088039398193,0.0,0.9266119003295898,6.385702965315431,1.079200471787926,1,3.004684013314545,3.004684013314545,3.242656404742696,879.53515625,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+cp4_supplement_combine_objects,1w_1t,1,1,1.9912192821502686,0.0,0.15514206886291504,4.98156897816807,6.445704942117339,1,1.8334413380362093,1.8334413380362093,11.817821893662222,777.11328125,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2
+cp_tutorial_translocation_start,1w_1t,1,1,2.0552892684936523,0.0,0.2602684497833252,4.96244603022933,3.8421867914935715,1,2.1760300607420504,2.1760300607420504,8.36071395727606,816.0546875,success,,,ordinary-zmq-outcomes-v1,fresh-per-observation,2712686a312b6cd048a8c3f2c24844a05fd8dc668390809299286250a672ceb2