API 524 on non-streaming requests that take too long (~180s); streaming works fine

[sassquatch-cloud]

Summary

API 524s on non-streaming requests that take too long.

Expected Behavior

API should return the response even if streaming is turned off.

Actual Behavior

API returns 524 on non-streaming requests that take longer than ~180s. Streaming works fine on the same task. The 524 body is the relay's own envelope: {"error":{"type":"server_error","message":"Upstream model provider is temporarily unavailable..."}}. The ceiling appears variable: a non-streaming 16K request succeeded at 109.5s while a 12K request died at 180.2s. This looks like the relay buffering the full non-streaming response and timing out on long generations.

Steps to reproduce the issue

1. Send a non-streaming chat/completions request with high reasoning_effort and a prompt that produces a long generation (>~2 min).
2. The relay holds the connection open while it buffers the response.
3. After ~180s the request dies with HTTP 524.
4. The same request with stream: true succeeds, even past 204s.

Command Code Version

n/a - api

Operating System

Linux

Terminal/IDE

Hermes

Shell

No response

Session file (optional)

No response

Fix prompt (optional)

No response

Additional context

same model (deepseek/deepseek-v4-flash), same prompt, same reasoning_effort=max, direct to api.commandcode.ai/provider/v1:

Non-streaming, max_tokens 12288 → 524 at 180.2s
Non-streaming, max_tokens 16384 → 200 OK at 109.5s
Non-streaming, max_tokens 8192 (hard prompt) → 200 OK at 100.5s
Non-streaming, max_tokens 8192 (easy prompt) → 200 OK at 1.9s
Streaming, max_tokens 16384 (same hard prompt) → 200 OK at 204.6s

commandcode relay A/B test — 2026-08-16 19:00-20:00 PDT

Endpoint: https://api.commandcode.ai/provider/v1/chat/completions

Model: deepseek/deepseek-v4-flash · reasoning_effort=max

Hard prompt: "Think very carefully and at length about the history of computing, step by step, in extreme detail. Cover every era in depth. Write a very long essay."

Easy prompt (control): "Write a haiku about the ocean."

Test scripts: run_tests.py (round 1), run_tests2.py (round 2) — same directory.

{"label": "T8k_control", "status": 200, "elapsed_s": 1.9, "bytes": 988, "ok": true, "finish": "stop", "content_len": 71, "reasoning_len": 0, "usage": {"prompt_tokens": 104, "completion_tokens": 52, "total_tokens": 156, "completion_tokens_details": {"reasoning_tokens": 36}}}
{"label": "T12k_boundary", "status": 524, "elapsed_s": 180.2, "bytes": 127, "ok": false, "api_error": "{'message': 'Upstream model provider is temporarily unavailable. Please try again in a moment.', 'type': 'server_error'}"}
{"label": "T16k_repeat", "status": 200, "elapsed_s": 109.5, "bytes": 68903, "ok": true, "finish": "stop", "content_len": 3491, "reasoning_len": 0, "usage": {"prompt_tokens": 128, "completion_tokens": 8850, "total_tokens": 8978, "completion_tokens_details": {"reasoning_tokens": 8057}}}
{"label": "Tstream_16k", "status": 200, "elapsed_s": 204.6, "bytes": 5421554, "ok": true, "raw_head": "data: {"id":"gen_01M06TNW59PYEZ36TQ5CBYDB5S","object":"chat.completion.chunk","created":1786935702,"model":"deepseek/dee"}
{"label": "T8k_hard", "status": 200, "elapsed_s": 100.5, "bytes": 48345, "ok": true, "finish": "length", "content_len": 28296, "reasoning_len": 0, "usage": {"prompt_tokens": 128, "completion_tokens": 8192, "total_tokens": 8320, "completion_tokens_details": {"reasoning_tokens": 2357}}}

[vipulgupta2048]

Thanks for opening an issue.
Any reason why you don't want to stream API requests and rather than non-streaming requests? Trying to understand the usecase.

[sassquatch-cloud]

Background cron jobs in hermes aren't designed to use streaming.

Most of the time the relay timeout isn't an issue, but I have some data processing crons that give the LLM a lot of data so usually it would think for a couple minutes. As a work around I have a script that streams the responses and forwards it.

I haven't run into this issue with other inference providers so not sure if it's intended behavior. Thank you for looking into this!