Skip to content

Fix admin crashes and Resonate worker going idle - #4859

Merged
marcoacierno merged 6 commits into
mainfrom
fix-admin-crashes
Nov 9, 2026
Merged

marcoacierno merged 6 commits into
mainfrom
fix-admin-crashes

Conversation

@marcoacierno

Copy link
Copy Markdown
Member

Summary

While testing the Django 6.1 upgrade (#4858) end to end on a local instance, I hit six bugs that were already on main and are unrelated to that upgrade. I reproduced the first one on Django 5.2.18 as well. This PR fixes them, one commit per bug, and each fix comes with a test that fails without it.

Fixes

  1. Deleting a conference that has grants crashed with RecursionError.
    • Grant.__init__ snapshots status, pending_status and country_type so it can detect changes. When the deletion collector loads the grants with .only(), reading a deferred field reloads the row, which builds another Grant, whose __init__ reads the next deferred field, and so on.
    • Any deferred load hit this, for example Grant.objects.only("id").first().
    • The snapshots are now read from __dict__, so deferred fields are never loaded here.
  2. Grant CSV export from the admin returned a 500, and export dropped the form-question columns.
    • GrantResource.export_field() did not accept the **kwargs (export_fields) that django-import-export 4 passes, so every admin export raised TypeError.
    • Once that was fixed, a second problem showed up. The admin export form lists the fields from get_export_order(), before the per-conference question columns are added. Those columns would have been left out, so they are now always exported.
    • The export tests now pass the field selection the same way the admin does.
  3. create_grant_vouchers returned a 500 when Pretix failed. It now reports each failing grant with an admin error message and keeps creating vouchers for the remaining grants. Before, a Pretix error stopped the action halfway through.
  4. Invalid deadline dates in the conference admin returned a 500.
    • validate_deadlines_form read cleaned_data["start"], ["end"] and ["type"] even when those fields had failed validation, which raised KeyError.
    • An empty date passed field validation, because TimeFramedModel makes both dates optional, and then crashed when compared with None.
    • Rows with field errors are now skipped by that check, so their own field errors show. Both dates are now required.
  5. Changelist search returned a 500 for files and video import requests. Their search_fields listed a foreign key directly (uploaded_by, conference), and Django can't run an icontains lookup on that. Search now uses the uploader's email and the conference code.
  6. Schedule invitations list and CSV export returned a 500 when Pretix failed. They now log the error and show the ticket status as unknown, the same way the grants list handles it.
  7. The Resonate worker stopped receiving work after 5 idle minutes. This affects production too.
    • The cause is in resonate-sdk 0.8.1, the latest release. It opens its long-lived SSE poll stream with aiohttp's default session timeout, total=300s.
    • When that timeout expires, aiohttp raises a bare TimeoutError. That isn't an aiohttp.ClientError, so the SDK's reconnect loop doesn't catch it and the listener task dies without logging anything.
    • The server then removes the worker's connection ("Poll connection deregistered"), and workflows such as clone conference stay pending until the worker restarts.
    • The worker now uses its own SSE source with no total timeout (the connect timeout stays at 30s), and it shares the process id with the SDK.
    • This depends on the SDK's private _ensure_session. The test streams one event before aiohttp's default timeout runs out (the timeout is shortened to 0.5s in the test) and one after. That way it fails if the workaround stops having any effect.
    • It's worth reporting upstream.

Test plan

  • Full backend suite on main's dependencies (Django 5.2): 1376 passed
  • The touched test modules also on the Upgrade Django to 6.1.3 #4858 dependency set (Django 6.1.3 / Wagtail 8.1): 293 passed
  • Checked each new test fails without its fix: RecursionError, TypeError, missing question columns, KeyError/TypeError, FieldError, HTTPError, and the SSE listener dying with TimeoutError
  • Live check of the Resonate fix:
    • Ran a worker from this branch against the local Resonate server. Seven minutes after it connected, the server had still not dropped its connection.
    • Started a clone conference workflow, and the worker ran it immediately.
    • With the current code, the same worker loses its connection after exactly 5 minutes, and workflows stay pending.
  • Grant.objects.only("id") against the local dev database now returns the grant with its fields still deferred
  • pre-commit hooks pass
  • Check after deploy that the production Resonate worker keeps picking up workflows after more than 5 minutes idle

This merges cleanly with #4858.

Grant.__init__ snapshots status, pending_status and country_type to detect
changes on save. When a field is deferred (for example when the deletion
collector loads related grants with .only()), reading it refetches the row,
which instantiates another Grant whose __init__ reads the next deferred
field, recursing until RecursionError. Deleting a conference that has
grants crashed this way.

Read the snapshots from __dict__ so deferred fields stay unloaded.
- GrantResource.export_field did not accept the keyword arguments that
  django-import-export 4 passes (export_fields), so every export from the
  admin raised TypeError. With that fixed, the admin's field selection,
  built from get_export_order(), would also have dropped the per-question
  columns added in before_export; they are now always exported.
- create_grant_vouchers let a Pretix error abort the whole action with a
  500. Failures are now reported per grant and the remaining grants still
  get their vouchers.

The export tests now pass the field selection the way the admin does.
validate_deadlines_form read cleaned_data["start"], ["end"] and
["type"] unconditionally, so a deadline row whose fields failed
validation raised KeyError. Empty dates passed field validation (they are
optional on TimeFramedModel) and then failed comparing None. Rows with
field errors are now left to those errors, and both dates are required.
Both admins listed a foreign key directly in search_fields, which Django
cannot search with icontains, so any search returned a 500. Search the
uploader's email and the conference code instead.
The speaker ticket column in the invitations list and the invitations CSV
export called Pretix without handling errors, so a failing request turned
the whole page or export into a 500. Log the error and show the ticket
status as unknown, as the grants list already does.
resonate-sdk opens its long-lived SSE poll stream with aiohttp's default
ClientSession timeout (total=300s). When it expires aiohttp raises a bare
TimeoutError, which is not an aiohttp.ClientError, so it escapes the
SDK's reconnect loop and the listener task dies silently. The server then
deregisters the worker and pending workflows are never dispatched until
the process restarts.

Give the worker its own SSE source with no total timeout (connect timeout
kept at 30s), sharing the process id with the SDK.
@vercel

vercel Bot commented Nov 9, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated
pycon Error Error Nov 9, 2026 2:44am UTC

@claude

claude Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

This PR fixes several admin 500s (grant deletion recursion, grant export, voucher creation, deadline validation, changelist search, schedule invitations) and works around the Resonate SSE 5-minute timeout with a custom poll connection.

Issues

  • grants/models.py:238-240: for a deferred field, _original_status, _original_pending_status and _original_country_type are now None. If the field is loaded later, any comparison against _original_* sees a change that didn't happen. grants/admin.py:663-673 does exactly this and would log a bogus "Status changed from 'None' to …". The admin loads full rows, so this doesn't happen today. Consider a sentinel, or only comparing when the original was loaded.
  • pycon/resonate_app.py: WorkerPollConnection relies on the SDK's private _ensure_session and _session/_stopped, and resonate-sdk is not pinned to an exact version. An SDK update could silently stop the override from working. The test catches that. Pinning the version, or adding a comment saying to re-check on upgrade, would make it harder to miss.
  • grants/admin.py, schedule/admin.py: from requests import RequestException is placed among the third-party imports out of order (before import_export in grants, and between django.db and custom_admin in schedule). Ruff's import sorting may flag it.

@codecov

codecov Bot commented Nov 9, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 78.26087% with 5 lines in your changes missing coverage. Please review.
✅ Project coverage is 92.41%. Comparing base (2d2173c) to head (e9e7df6).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4859      +/-   ##
==========================================
- Coverage   92.43%   92.41%   -0.02%     
==========================================
  Files         360      360              
  Lines       10938    10958      +20     
  Branches      949      953       +4     
==========================================
+ Hits        10110    10127      +17     
+ Misses        706      705       -1     
- Partials      122      126       +4     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@marcoacierno
marcoacierno merged commit f0bb858 into main Nov 9, 2026
8 of 9 checks passed
@marcoacierno
marcoacierno deleted the fix-admin-crashes branch November 9, 2026 02:50

This branch had an error being deployed

1 failed deployment
Preview — e9e7df64 Deployed Nov 9, 2026 by vercel[bot]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant