Skip to content

fix(pyamber): add_sub_queue replaces existing sub-queue instead of using putIfAbsent semantics - #7850

Open
Musxeto wants to merge 2 commits into
apache:mainfrom
Musxeto:fix/add-sub-queue-put-if-absent
Open

fix(pyamber): add_sub_queue replaces existing sub-queue instead of using putIfAbsent semantics#7850
Musxeto wants to merge 2 commits into
apache:mainfrom
Musxeto:fix/add-sub-queue-put-if-absent

Conversation

@Musxeto

@Musxeto Musxeto commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

LinkedBlockingMultiQueue.add_sub_queue diverged from the upstream Java
LinkedBlockingMultiQueue 0.6.0 it ports. The Java implementation registers
sub-queues with subQueues.putIfAbsent(key, subQueue), which leaves the map
untouched when the key already exists. The Python port unconditionally
overwrote self.sub_queues[key] with a freshly constructed SubQueue before
checking whether the key was already present.

On a repeated key the method still returned the old SubQueue (matching its
docstring), but an unattached replacement (priority_group = None) had
already been installed in the map. The failure chain:

Before: add_sub_queue("k") called twice → orphan SubQueue in map (priority_group = None)
After:  put("k", item) → total_count incremented, blocked get() woken
        → get_next() walks priority_groups, finds no non-empty group, returns None
        → None.dequeue() → AttributeError while holding take_lock

The fix: early-return from add_sub_queue when the key is already registered,
leaving the map and all priority groups untouched. The SubQueue object is
now constructed only when the key is absent, matching upstream Java semantics.

Any related issues, documentation, discussions?

Closes #7810

Related to the same family of port divergences as #6903, found during review
of #6906.

How was this PR tested?

Added TestAddSubQueue to
amber/src/test/python/core/util/customized_queue/test_linked_blocking_multi_queue.py
with 5 tests written red-first (TDD):

  • test_new_key_returns_none — positive path: first registration returns None
  • test_repeated_key_returns_existing_sub_queue — duplicate call returns the original object
  • test_repeated_key_keeps_existing_queue_in_map — map still holds the original after a second add
  • test_repeated_key_priority_group_is_not_nonepriority_group is not None after a duplicate add
  • test_repeated_key_put_then_get_does_not_crash — exact reproduction from the issue: no AttributeError

All 47 tests (5 new + 42 pre-existing) pass with no regressions:

pytest src/test/python/core/util/customized_queue/test_linked_blocking_multi_queue.py -v
# 47 passed in 3.64s

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Antigravity (Google DeepMind), powered by Claude Sonnet 4.6

@Yicong-Huang Yicong-Huang added the release/v1.2 back porting to release/v1.2 label Aug 22, 2026
@github-actions
github-actions Bot requested a review from xuang7 August 22, 2026 13:52
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @mengw15
    You can notify them by mentioning @mengw15 in a comment.

@github-actions

github-actions Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

Backport auto-label report

This fix: PR was checked against each actively-supported release branch. release/* labels drive the post-merge backport, so add or remove one to change where this fix lands.

Release branch Analysis
release/v1.2 Already labeled — this fix is queued to backport here.

Auto-label run.

@github-actions

github-actions Bot commented Aug 22, 2026

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 2 better · 🔴 2 worse · ⚪ 11 noise (<±5%) · 0 without baseline

Compared against main b7c33b0 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 371 0.226 26,679/37,591/37,591 us 🔴 +20.7% / 🔴 +161.4%
🔴 bs=100 sw=10 sl=64 798 0.487 125,730/143,366/143,366 us 🟢 -14.0% / 🔴 +44.0%
bs=1000 sw=10 sl=64 927 0.566 1,083,659/1,115,887/1,115,887 us ⚪ within ±5% / 🔴 +19.2%
Baseline details

Latest main b7c33b0 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 371 tuples/sec 384 tuples/sec 833.79 tuples/sec -3.4% -55.5%
bs=10 sw=10 sl=64 MB/s 0.226 MB/s 0.234 MB/s 0.509 MB/s -3.4% -55.6%
bs=10 sw=10 sl=64 p50 26,679 us 22,112 us 11,864 us +20.7% +124.9%
bs=10 sw=10 sl=64 p95 37,591 us 36,531 us 14,381 us +2.9% +161.4%
bs=10 sw=10 sl=64 p99 37,591 us 36,531 us 18,035 us +2.9% +108.4%
bs=100 sw=10 sl=64 throughput 798 tuples/sec 823 tuples/sec 1,083 tuples/sec -3.0% -26.3%
bs=100 sw=10 sl=64 MB/s 0.487 MB/s 0.502 MB/s 0.661 MB/s -3.0% -26.4%
bs=100 sw=10 sl=64 p50 125,730 us 116,626 us 93,077 us +7.8% +35.1%
bs=100 sw=10 sl=64 p95 143,366 us 166,783 us 99,553 us -14.0% +44.0%
bs=100 sw=10 sl=64 p99 143,366 us 166,783 us 108,604 us -14.0% +32.0%
bs=1000 sw=10 sl=64 throughput 927 tuples/sec 925 tuples/sec 1,119 tuples/sec +0.2% -17.1%
bs=1000 sw=10 sl=64 MB/s 0.566 MB/s 0.565 MB/s 0.683 MB/s +0.2% -17.1%
bs=1000 sw=10 sl=64 p50 1,083,659 us 1,074,248 us 909,247 us +0.9% +19.2%
bs=1000 sw=10 sl=64 p95 1,115,887 us 1,128,446 us 952,561 us -1.1% +17.1%
bs=1000 sw=10 sl=64 p99 1,115,887 us 1,128,446 us 985,186 us -1.1% +13.3%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,539.73,200,128000,371,0.226,26678.79,37591.14,37591.14
1,100,10,64,20,2505.24,2000,1280000,798,0.487,125729.80,143366.24,143366.24
2,1000,10,64,20,21566.07,20000,12800000,927,0.566,1083658.56,1115887.48,1115887.48

@codecov-commenter

codecov-commenter commented Aug 22, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.88%. Comparing base (b7c33b0) to head (a11b35c).

Additional details and impacted files
@@            Coverage Diff             @@
##               main    #7850    +/-   ##
==========================================
  Coverage     91.87%   91.88%            
  Complexity     4510     4510            
==========================================
  Files          1173     1173            
  Lines         47350    48067   +717     
  Branches       5306     5306            
==========================================
+ Hits          43502    44165   +663     
- Misses         2204     2258    +54     
  Partials       1644     1644            
Flag Coverage Δ *Carryforward flag
access-control-service 81.00% <ø> (ø) Carriedforward from b7c33b0
agent-service 98.62% <ø> (ø) Carriedforward from b7c33b0
amber 88.38% <ø> (ø) Carriedforward from b7c33b0
computing-unit-managing-service 73.67% <ø> (ø) Carriedforward from b7c33b0
config-service 86.73% <ø> (ø) Carriedforward from b7c33b0
file-service 75.74% <ø> (ø) Carriedforward from b7c33b0
frontend 93.79% <ø> (ø) Carriedforward from b7c33b0
notebook-migration-service 79.13% <ø> (ø) Carriedforward from b7c33b0
pyamber 96.88% <100.00%> (-0.69%) ⬇️
workflow-compiling-service 77.19% <ø> (ø) Carriedforward from b7c33b0

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

fix pyamber release/v1.2 back porting to release/v1.2

Projects

None yet

Development

Successfully merging this pull request may close these issues.

LinkedBlockingMultiQueue.add_sub_queue replaces an existing sub-queue instead of keeping it (putIfAbsent divergence)

3 participants