Keep concurrency locks across non-graceful job release

[rafael-pissardo]

Summary

• Extend a job's concurrency semaphore expiry when a claimed execution is released back to ready, so the slot stays owned across restart
• Skip expiring semaphores whose key still has a ready job, and run concurrency maintenance synchronously at supervisor boot and dispatcher start before workers can poll
• Add regression tests for the ready-job lock race from non-graceful shutdown

Fixes #735.

Test plan

• TARGET_DB=sqlite bundle exec ruby -Itest test/unit/concurrency_maintenance_test.rb
• TARGET_DB=sqlite bundle exec ruby -Itest test/integration/concurrency_controls_test.rb
• TARGET_DB=sqlite bundle exec ruby -Itest test/models/solid_queue/claimed_execution_test.rb
• TARGET_DB=sqlite bundle exec ruby -Itest test/unit/dispatcher_test.rb
• TARGET_DB=sqlite bundle exec ruby -Itest test/unit/fork_supervisor_test.rb
• RuboCop on changed files

[rosa]

Hey @rafael-pissardo, thanks for this, but I'm not sure these changes are compatible with the way concurrency controls work. What they fix is intentional behaviour.

Extend a job's concurrency semaphore expiry when a claimed execution is released back to ready, so the slot stays owned across restart

The expiration is based on the max duration specified in the job's concurrency settings. This means that after that time has passed, we can no longer guarantee that other jobs will run. This is the case no matter what the job is doing. If the job went back to ready, that means that either the worker crashed, or it was gracefully terminated with the job going over the shutdown timeout, or something similar. If the lock expires during that time, we consider that to be the time the job takes to run, and it counts towards the concurrency duration. The concurrency duration is a self-healing mechanism to avoid having jobs blocked forever. The trade-off is that in some cases, depending on the jobs and the duration configured, it might happen that some jobs overlap. This is preferable to having jobs blocked forever.

Skip expiring semaphores whose key still has a ready job, and run concurrency maintenance synchronously at supervisor boot and dispatcher start before workers can poll

This is the same idea. If we have ready jobs when their locks have already expired, then they need to overlap with other jobs, that's the way it's intended to work. From the README:

Since something can happen that prevents the first job from releasing the semaphore and unblocking the next job (for example, someone pulling a plug in the machine where the worker is running), we have the duration as a failsafe. Jobs that have been blocked for more than duration are candidates to be released, but only as many of them as the concurrency rules allow, as each one would need to go through the semaphore dance check. This means that the duration is not really about the job that's enqueued or being run, it's about the jobs that are blocked waiting, or about the jobs that would get discarded while the semaphore is closed.

Thank you!