[GalihFajar]

Problem

findSubscribedProjectSettingsById took s.RLock() and then called findProjectSettingsById, which took s.RLock() again on the same goroutine. In Go, a waiting writer blocks every new RLock(). So if a writer asked for Lock() between the two RLock() calls, the nested RLock() waited for the writer and the writer waited for the outer RLock(). Both waited forever. Writers include the poller's Init() every 30s and the pubsub update handlers.

After that, every read of LocalStorage on the pod blocked. In production, one Turing router pod returned 408 for all requests while goroutines and memory grew, until the pod was OOMKilled after about 10 minutes. This happened about every 1 to 3 weeks.

Fix

• Add findProjectSettingsByIdLocked. It does the lookup and takes no lock. The caller must hold the lock.
• findProjectSettingsById takes RLock and calls it.
• findSubscribedProjectSettingsById calls it directly, because it already holds RLock.

Behavior does not change. I checked all lock sites in LocalStorage, and no other code path takes the lock again in the same goroutine.

This MR replaces the lock-fix part of #93. The LocalStorage metrics from #93 are not included and can go in a separate MR.

Test

TestFindProjectSettingsWithId_ConcurrentWithWriter_NoDeadlock runs 50 readers that call FindProjectSettingsWithId and 1 writer that calls UpdateProjectSettings 2000 times, with a 5s deadline.

• Before the fix: 10/10 runs failed with -race, and 3/3 failed without it.
• After the fix: 30/30 runs passed with -race, and 30/30 passed without it.

If the test times out, its goroutines stay blocked. This is expected, because they are stuck in the deadlock that the test detects.

Follow-ups (not in this MR)

• FindProjectSettingsWithId: for a project that is not subscribed, or not found, each request calls the management API and appends to s.ProjectSettings with no dedup. The slice grows until the next Init().
• fetchProjectSettingsWithId and getProjectSettings use JSON200 without a status or nil check. A non-200 response causes a panic.
• InsertProjectSettings checks and inserts in two separate lock sections. Concurrent inserts can add duplicates.
• UpdateProjectSegmenters writes into s.ProjectSegmenters[pid] and panics if that map is nil.
• fetchProjectSettingsWithId can append to the old slice while Init() swaps in a new one. The appended entry is lost.

🤖 Generated with Claude Code