fix(automerge): requeue transiently-failed evaluations instead of dropping them

[willemkokke]

Fixes #38969

The pr_auto_merge queue handler always returned nil, telling the queue every item was handled even when handlePullRequestAutoMerge bailed out on a transient failure (a database read, a git repository open). The queue's retry mechanism — workergroup.go re-pushes items the handler returns as unhandled — was never engaged, so one hiccup at the wrong moment permanently lost a scheduled merge: the pull request stayed green with auto-merge armed and never merged, with nothing on the PR page and only a single generic server log line. On a PR whose CI has already settled, no further event ever re-enqueues the evaluation. Issue #38969 has the full analysis with line-by-line citations.

What this changes

• handlePullRequestAutoMerge now returns an error for transient infrastructure failures (DB reads, git repository access) and nil for everything genuinely handled — merged, or refused for a reason retrying cannot cure (pull request deleted, head branch gone, SHA superseded, checks not passing, CheckPullMergeable sentinel refusals such as ErrHasMerged/ErrIsClosed/ErrNotReadyToMerge).
• handler returns transiently-failed items to the queue so its existing retry path engages, bounded per item (maxTransientRetries, in-memory): the queue's requeue loop has no retry limit and no dead-letter, so unbounded requeueing would trade a silent drop for a permanent spin — we have watched a poison item in another queue do exactly that for days. When the bound is reached the item is dropped with a loud log.Error; the pull_auto_merge row stays in place, so the next event for the PR re-evaluates it.
• The ErrNotReadyToMerge branch now logs the precise refusal reason the error already carries (status checks / approvals / requested changes / official review requests / behind base branch / code owner reviews / protected files) instead of the misleading ... was scheduled to automerge by an unauthorized user, which sends anyone debugging from logs chasing a permissions problem that may not exist.
• handlePullRequestAutoMerge becomes a function variable (same pattern as automergequeue.AddToQueue and pull.AddPullRequestToCheckQueue) so the handler's requeue contract is unit-testable.

pull_service.Merge failures deliberately stay terminal (unchanged behaviour, existing FIXME kept): retrying a failed merge is riskier than retrying a read, and surfacing merge errors on the PR page is tracked separately (#30691).

Testing

• go test ./services/automerge/... ./modules/queue/...
• Regression test proven red on the old code: TestHandlerRequeuesOnRepositoryAccessFailure exercises the full evaluation with nothing mocked but the repository storage location — a fixture PR with a scheduled auto merge whose git repository is inaccessible. Run against the unpatched handler at main (83af7aa) it fails (handler returns nil, the scheduled merge is dropped); with this PR it passes (the item is returned for requeue). Everything except the one clearTransientFailures cleanup line compiles against the old code as-is.
• Further unit tests via the handler seam: handled and unparsable items are not requeued; the per-item bound drops a persistently failing item and counts afresh on a later event; a deleted pull request is terminal (fixture DB).
• go vet, gofmt clean.

A complementary, heavier hardening — a periodic reconciler over pull_auto_merge rows that re-enqueues any still-open scheduled merge, which would also self-heal losses from restarts and missed events — is sketched in #38969 and intentionally not part of this PR.

AI assistance

AI assistance (Claude) was used to prepare this patch, its tests, and this description, per the AI Contribution Policy. The analysis and the change have been reviewed and tested by me, I understand and can defend every part of it, and I will answer review questions myself.

[bircni]

sorry but this looks like completely AI generated (which is fine)
But please follow the guidelines and state so https://github.com/go-gitea/gitea/blob/main/CONTRIBUTING.md#ai-contribution-policy

Also we have a guideline for https://github.com/go-gitea/gitea/blob/main/CONTRIBUTING.md#pr-title-and-summary

[willemkokke]

My apologies, I messed up. I'll fix it.

[wxiaoguang]

The retry count should be saved in the automerge's database record?

[willemkokke]

Hi @wxiaoguang

I did consider this, and I intentionally decided against it for this pr.

I fixed this to patch an active production issue (this has been running in production for over a week running the same workloads it ran before but without the occasional auto merge never being triggered)

I wanted to avoid a migration and just have a drop-in binary. (data was safely backed up regardless)

There are benefits to it for sure, the retries would survive restarts, you could inspect it on a running system, and it might be a good time to do #30691 (adding a field to display a merge error message on the page).

It was also not super clear what the semantics should be. If the retry count is on that row, it is shared between all (PR, sha) pairs. Do we reset when the sha moves? Keeping the count in memory also allows it to be trivially back-ported if that is a thing your organisation does, or anyone else needs.

This PR is just a bug fix, the follow-up(#38972) is auto merge hardening, a reconciler that re-queues merges after a restart and when lost due to other failures.

That would reduce the need to persist the data, as there is no risk of permanently losing an auto merge anymore.
It would also be a better place for me to implement this, if you still truly want this, considering that is the code that actually interacts with the database.

[wxiaoguang]

handlePullRequestAutoMerge bailed out on a transient failure (a database read, a git repository open).

Could you clarify why such transient failure (a database read, a git repository open) can happen? If such error happens, why the queue is still working?

---

Update: OK, I see the reason, "database restart"

[wxiaoguang]

I think the design can be simplified like this (with #38972)

1. Add a LastTryTime, TriedCount, LastTryFailureMessage to table AutoMerge
2. In handlePullRequestAutoMerge: update LastTryTime=now, TriedCount++, LastTryFailureMessage='' first
   • if the PR is closed or broken: delete the AutoMerge record
   • if TriedCount > N (maybe 3 or 5): do nothing and just return
3. If any error happens in handlePullRequestAutoMerge: record LastTryFailureMessage (ignore error), then debounce 30 seconds to requeue
4. On Gitea startup, populate the "recent" (e.g.: create time < 3 days) AutoMerge records to the queue