Skip to content
19 changes: 19 additions & 0 deletions workers/control-api/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -58,3 +58,22 @@ No production routes are attached. This Worker is isolated from existing OCodeJS
## Structured request telemetry

Every request emits one JSON event to Cloudflare Workers Logs with `event`, `requestId`, HTTP method, normalized URL path, response status, duration in milliseconds and timestamp. The query string, request body, cookies, authorization headers and upstream payloads are deliberately excluded. Responses also return `X-Request-Id` and `Server-Timing` for correlation. This is log-based observability. The current branch additionally supports an optional Workers Analytics Engine binding named `OBSERVABILITY`, which writes bounded request dimensions and duration when attached. The binding is not configured on staging by this code change; without it, the Worker continues to operate using structured logs only. No public metrics query, alert evaluator, durable audit record store or role-based explorer is implemented yet.


## Cloudflare staging service binding

The Control API staging Worker should use an internal Cloudflare service binding to the MCP Worker in the same Cloudflare account. Configure this binding only on the staging Worker:

- Binding name: `MCP_SERVER`
- Binding type: `service`
- Service: `ocodejs-mcp-staging`

The API uses this binding for health probes when present and falls back to the public `MCP_ENDPOINT` URL otherwise. The public endpoint remains in the service catalog for clients. Do not copy this binding into Production without a separate review.

The same staging Worker can also use an internal binding for the configured Auth backend in the Huynh Cloudflare account:

- Binding name: `CLI_AUTH`
- Binding type: `service`
- Service: `ocodejs-auth`

This reports the internal backend health only. It does not prove that the public `auth.ocodejs.dev` hostname routes to this Worker or that an OAuth login succeeds. Keep the binding staging-only until the canonical public Auth Worker/domain ownership is resolved.
62 changes: 39 additions & 23 deletions workers/control-api/worker.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -166,11 +166,15 @@ function errorResponse(code, message, status, id, headers) {
}, status, headers);
}

async function probeCliAuth() {
async function probeCliAuth(env = {}) {
const startedAt = Date.now();
const checkedAt = new Date().toISOString();
const viaServiceBinding = env.CLI_AUTH && typeof env.CLI_AUTH.fetch === "function";
const authFetch = viaServiceBinding
? (init) => env.CLI_AUTH.fetch(new Request(CLI_AUTH_HEALTH_URL, init))
: (init) => fetch(CLI_AUTH_HEALTH_URL, init);
try {
const response = await fetch(CLI_AUTH_HEALTH_URL, {
const response = await authFetch({
method: "GET",
headers: { Accept: "application/json" },
cache: "no-store",
Expand Down Expand Up @@ -198,26 +202,33 @@ async function probeCliAuth() {
return {
state: configured ? "healthy" : "degraded",
description: configured
? "Health endpoint responds and required GitHub OAuth/PAT bindings are present; end-to-end login is not verified."
? (viaServiceBinding
? "Internal Auth Worker health responds and required GitHub OAuth/PAT bindings are present; public auth hostname and end-to-end login remain unverified."
: "Health endpoint responds and required GitHub OAuth/PAT bindings are present; end-to-end login is not verified.")
: "Health endpoint responds, but one or more GitHub OAuth/PAT bindings are missing.",
checkedAt,
latencyMs: Date.now() - startedAt
};
} catch {
} catch (error) {
console.warn(JSON.stringify({ event: "control_api.cli_auth_probe_failed", errorType: error?.name || "Error" }));
return {
state: "unavailable",
description: "CLI Auth health endpoint could not be reached.",
description: "CLI Auth health endpoint could not be reached (" + (error?.name || "Error") + ").",
checkedAt,
latencyMs: Date.now() - startedAt
};
}
}

async function probeMcpServer() {
async function probeMcpServer(env = {}) {
const startedAt = Date.now();
const checkedAt = new Date().toISOString();
let phase = "initialize";
const mcpFetch = env.MCP_SERVER && typeof env.MCP_SERVER.fetch === "function"
? (init) => env.MCP_SERVER.fetch(new Request("https://mcp-server/mcp", init))
: (init) => fetch(MCP_ENDPOINT, init);
try {
const initializeResponse = await fetch(MCP_ENDPOINT, {
const initializeResponse = await mcpFetch({
method: "POST",
headers: { Accept: "application/json, text/event-stream", "Content-Type": "application/json" },
body: JSON.stringify({
Expand All @@ -231,41 +242,45 @@ async function probeMcpServer() {
}
}),
cache: "no-store",
redirect: "error",
signal: AbortSignal.timeout(4000)
redirect: "follow",
signal: AbortSignal.timeout(12000)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Public probes hold capacity longer

When anonymous clients repeatedly call the public probe routes while the MCP endpoint responds slowly but within each request timeout, handleRequest calls probeMcpServer on public MCP connection routes without authentication or throttling. Its three serial fetches can now keep each Worker request and its upstream requests open for up to 36 seconds, allowing repeated callers to exhaust capacity during a slow MCP response.

Apply a rate limit or cache probe results, and enforce an overall deadline for the handshake.

Also at workers/control-api/worker.mjs:252, workers/control-api/worker.mjs:263.

Prompt for AI agents
In `workers/control-api/worker.mjs` at line 235:

**Public probes hold capacity longer**

When anonymous clients repeatedly call the public probe routes while the MCP endpoint responds slowly but within each request timeout, `handleRequest` calls `probeMcpServer` on public MCP connection routes without authentication or throttling. Its three serial fetches can now keep each Worker request and its upstream requests open for up to 36 seconds, allowing repeated callers to exhaust capacity during a slow MCP response.

Apply a rate limit or cache probe results, and enforce an overall deadline for the handshake.

Also at `workers/control-api/worker.mjs:252`, `workers/control-api/worker.mjs:263`.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Staging validation aborts early

When the sequential MCP handshake takes longer than the staging checker’s 10-second /services budget, probeMcpServer can spend up to 36 seconds on its three sequential requests, so check-control-api-staging.mjs aborts before the probe returns and reports the staging result as unverified or failed.

Bound the total probe duration so it completes within the staging checker’s 10-second request budget.

Also at workers/control-api/worker.mjs:251, workers/control-api/worker.mjs:263.

Prompt for AI agents
In `workers/control-api/worker.mjs` at line 235:

**Staging validation aborts early**

When the sequential MCP handshake takes longer than the staging checker’s 10-second `/services` budget, `probeMcpServer` can spend up to 36 seconds on its three sequential requests, so `check-control-api-staging.mjs` aborts before the probe returns and reports the staging result as unverified or failed.

Bound the total probe duration so it completes within the staging checker’s 10-second request budget.

Also at `workers/control-api/worker.mjs:251`, `workers/control-api/worker.mjs:263`.

});
if (!initializeResponse.ok) {
return { state: "unavailable", description: "MCP initialize returned HTTP " + initializeResponse.status + ".", checkedAt, latencyMs: Date.now() - startedAt };
}
const initialized = await initializeResponse.json();
phase = "validate-initialize";
if (initialized?.result?.protocolVersion !== "2025-03-26" || initialized?.result?.serverInfo?.name !== "ocodejs-control-center") {
return { state: "unknown", description: "MCP initialize response did not match the expected protocol contract.", checkedAt, latencyMs: Date.now() - startedAt };
}

const notificationResponse = await fetch(MCP_ENDPOINT, {
phase = "initialized-notification";
const notificationResponse = await mcpFetch({
method: "POST",
headers: { Accept: "application/json, text/event-stream", "Content-Type": "application/json", "MCP-Protocol-Version": "2025-03-26" },
body: JSON.stringify({ jsonrpc: "2.0", method: "notifications/initialized" }),
cache: "no-store",
redirect: "error",
signal: AbortSignal.timeout(4000)
redirect: "follow",
signal: AbortSignal.timeout(12000)
});
if (notificationResponse.status !== 202) {
return { state: "unknown", description: "MCP initialized notification was not accepted.", checkedAt, latencyMs: Date.now() - startedAt };
}

const toolsResponse = await fetch(MCP_ENDPOINT, {
phase = "tools-list";
const toolsResponse = await mcpFetch({
method: "POST",
headers: { Accept: "application/json, text/event-stream", "Content-Type": "application/json", "MCP-Protocol-Version": "2025-03-26" },
body: JSON.stringify({ jsonrpc: "2.0", id: 2, method: "tools/list" }),
cache: "no-store",
redirect: "error",
signal: AbortSignal.timeout(4000)
redirect: "follow",
signal: AbortSignal.timeout(12000)
});
if (!toolsResponse.ok) {
return { state: "unavailable", description: "MCP tools/list returned HTTP " + toolsResponse.status + ".", checkedAt, latencyMs: Date.now() - startedAt };
}
const listed = await toolsResponse.json();
phase = "validate-tools-list";
const toolNames = (listed?.result?.tools || []).map((tool) => tool.name).sort();
if (JSON.stringify(toolNames) !== JSON.stringify(["get_agent_definition", "get_control_api_health", "get_provider_definition", "list_agents", "list_providers", "list_services"])) {
return { state: "unknown", description: "MCP tools/list did not return the expected read-only tool allowlist.", checkedAt, latencyMs: Date.now() - startedAt };
Expand All @@ -277,22 +292,23 @@ async function probeMcpServer() {
latencyMs: Date.now() - startedAt,
toolCount: toolNames.length
};
} catch {
} catch (error) {
console.warn(JSON.stringify({ event: "control_api.mcp_probe_failed", errorType: error?.name || "Error" }));

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Medium · Failed handshakes lack diagnostics

When the endpoint returns an HTTP error or an invalid handshake response, the HTTP-status and protocol-mismatch branches return unavailable or unknown without throwing, so they bypass the catch that logs control_api.mcp_probe_failed; callers see a failed probe without the failure diagnostic.

Emit the failure warning on the HTTP-status and protocol-mismatch return branches as well as in the catch.

Prompt for AI agents
In `workers/control-api/worker.mjs` at line 281:

**Failed handshakes lack diagnostics**

When the endpoint returns an HTTP error or an invalid handshake response, the HTTP-status and protocol-mismatch branches return `unavailable` or `unknown` without throwing, so they bypass the `catch` that logs `control_api.mcp_probe_failed`; callers see a failed probe without the failure diagnostic.

Emit the failure warning on the HTTP-status and protocol-mismatch return branches as well as in the `catch`.

return {
state: "unavailable",
description: "MCP staging endpoint could not complete the protocol handshake.",
description: "MCP staging probe failed during " + phase + " (" + (error?.name || "Error") + ").",
checkedAt,
latencyMs: Date.now() - startedAt
};
}
}

async function serviceSnapshot(service, checkedAt) {
async function serviceSnapshot(service, checkedAt, env = {}) {
if (service.id === "cli-auth") {
return { ...service, ...(await probeCliAuth()) };
return { ...service, ...(await probeCliAuth(env)) };
}
if (service.id === "mcp-server") {
return { ...service, ...(await probeMcpServer()) };
return { ...service, ...(await probeMcpServer(env)) };
}
return {
...service,
Expand Down Expand Up @@ -383,7 +399,7 @@ async function handleRequest(request, env, url, headers, id) {

if (url.pathname === `${API_PREFIX}/services`) {
const checkedAt = new Date().toISOString();
const services = await Promise.all(SERVICES.map((service) => serviceSnapshot(service, checkedAt)));
const services = await Promise.all(SERVICES.map((service) => serviceSnapshot(service, checkedAt, env)));
return json({
services,
count: services.length,
Expand All @@ -409,7 +425,7 @@ async function handleRequest(request, env, url, headers, id) {
}

return json({
service: await serviceSnapshot(service, new Date().toISOString()),
service: await serviceSnapshot(service, new Date().toISOString(), env),
requestId: id
}, 200, headers);
}
Expand Down Expand Up @@ -484,7 +500,7 @@ async function handleRequest(request, env, url, headers, id) {
}

if (url.pathname === API_PREFIX + "/mcp/connections") {
const health = await probeMcpServer();
const health = await probeMcpServer(env);
const connections = MCP_CONNECTIONS.map((connection) => ({
...connection,
...health
Expand Down
65 changes: 65 additions & 0 deletions workers/control-api/worker.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -441,3 +441,68 @@ test("runtime readiness explicitly keeps provider inference and agent execution
assert.ok(body.checkedAt);
assert.ok(body.requestId);
});


test("MCP probe uses the internal Cloudflare service binding when configured", async () => {
const methods = [];
const tools = [
"get_control_api_health",
"get_provider_definition",
"get_agent_definition",
"list_agents",
"list_providers",
"list_services"
];
const MCP_SERVER = {
fetch: async (request) => {
assert.equal(new URL(request.url).pathname, "/mcp");
const message = await request.json();
methods.push(message.method);
if (message.method === "initialize") {
return new Response(JSON.stringify({ jsonrpc: "2.0", id: message.id, result: {
protocolVersion: "2025-03-26",
serverInfo: { name: "ocodejs-control-center" }
}}), { status: 200, headers: { "Content-Type": "application/json" } });
}
if (message.method === "notifications/initialized") return new Response(null, { status: 202 });
if (message.method === "tools/list") {
return new Response(JSON.stringify({ jsonrpc: "2.0", id: message.id, result: {
tools: tools.map((name) => ({ name }))
}}), { status: 200, headers: { "Content-Type": "application/json" } });
}
throw new Error("Unexpected MCP method: " + message.method);
}
};
const response = await worker.fetch(
new Request("https://api-staging.example.workers.dev/api/v1/services/mcp-server"),
{ ...env, MCP_SERVER }
);
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(body.service.state, "healthy");
assert.equal(body.service.toolCount, 6);
assert.deepEqual(methods, ["initialize", "notifications/initialized", "tools/list"]);
});


test("CLI Auth probe uses the internal Cloudflare service binding when configured", async () => {
const CLI_AUTH = {
fetch: async (request) => {
assert.equal(new URL(request.url).pathname, "/health");
return new Response(JSON.stringify({
ok: true,
service: "ocodejs-auth-worker",
patConfigured: true,
oauthConfigured: true
}), { status: 200, headers: { "Content-Type": "application/json" } });
}
};
const response = await worker.fetch(
new Request("https://api-staging.example.workers.dev/api/v1/services/cli-auth"),
{ ...env, CLI_AUTH }
);
const body = await response.json();
assert.equal(response.status, 200);
assert.equal(body.service.state, "healthy");
assert.match(body.service.description, /Internal Auth Worker health responds/);
});
Loading