Skip to content

bug(gateway): ssh connect occasionally fails with unique violation on objects_name_uq #5661

Open

Description

@FrostGod

User Story

I ran a mini scale test of OpenShell with many concurrent ssh connections into sandboxes. About once in 1,700 connections, the connection failed with a database "unique violation" error, and a retry worked. I need SSH connections to sandboxes to be reliable under load.

Problem Statement

Opening an SSH connection to a sandbox sometimes fails with:

persist ssh session failed: unique violation on objects_name_uq

We saw this once in about 1,700 ssh calls with 144 concurrent workers. The client sees a failed connection, and an immediate retry succeeds.

Why This Happens

Each SSH connection creates an SSH session record in the gateway database. The record gets a random name of only 6 lowercase letters (generate_name, used here), which gives about 309 million possible names. The database requires names to be unique within a workspace.

Session records stay in the database for up to about 2 hours: a 1-hour lifetime, plus a cleanup job that runs once an hour (lib.rs). The more records there are, the more likely a new random name matches an existing one. When it does, the insert fails and the gateway returns an error instead of trying another name (sandbox.rs).

This isn't a concurrency race. The chance of a collision grows with how many session records exist, not with how many connections run at once:

Session records in the workspace Chance a new connection fails
10,000 about 1 in 31,000
100,000 about 1 in 3,100

Impact / Why This Matters

  • At scale, a steady share of SSH connections fails for no visible reason.
  • Busy workspaces, with many short SSH or file-transfer sessions, see it more often.
  • Users and agents need their own retry logic for a failure that OpenShell could avoid.

Acceptance Criteria

  • Creating an SSH session never fails because of a name collision. For example, the gateway could retry with a new name, or derive the name from the session's unique ID.
  • A test covers a name collision during SSH session creation.

Reproduction Steps

  1. Create a sandbox and open many short SSH connections to it in a loop, for example ssh openshell-mybox.default true, so that tens of thousands of session records build up within the 1-hour lifetime.
  2. A small share of connections fails with unique violation on objects_name_uq.

Environment

  • OpenShell: mini scale test build; code path confirmed on main at 624acee
  • Deployment: Kubernetes (Helm), multi-replica gateway, PostgreSQL

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions