EC²-Bench

Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution

128 executable task definitions · task-level privacy notes · 4 collaboration strategies · Task Performance · Resource Usage · Privacy Exposure

---

EC²-Bench (EC2-Bench; Edge–Cloud Collaborative Agent Benchmark) is a benchmark for edge–cloud collaboration in long-horizon LLM agent execution. Edge-side models keep data local and cost little but are weak at demanding planning and tool use. Cloud-hosted frontier models are far more capable, but they are expensive and see everything they are sent. Prior edge–cloud systems coordinate the two once per query. In long-horizon agent execution, cloud participation is instead interleaved throughout the trajectory. That creates trajectory-level effects: each cloud call shapes later steps, accumulates cost, and decides how much workspace information leaves the device.

EC²-Bench provides 128 digital-workspace task definitions with inline evaluation checks and task-level privacy notes, together with a unified OpenClaw execution and evaluation environment. It scores every run on three axes at once: task performance, resource usage, and privacy exposure. We use it to study four representative collaboration strategies across edge models of increasing capability.

Key Findings

1. Edge–cloud collaboration works for long-horizon agents (RQ1). Partial cloud participation improves performance over edge-only execution at a fraction of cloud-only cost and exposure. With Qwen3.5-2B, Per-step Routing lifts completion from 12.75 to 39.97 at 22.7% of cloud-only cost, and many configurations beat a linear edge↔cloud interpolation.
2. Stronger edges close the performance gap, not the privacy gap (RQ2). The best collaborative configuration climbs from 39.97 (2B) to 76.84 (Qwen3.7-Plus), approaching cloud-only's 82.70. The completion gap across strategies narrows from 25.22 points (2B) to 3.67 (Qwen3.7-Plus), but privacy exposure does not converge: with Qwen3.7-Plus, strategies differ by only 3.67 points in completion but by 66.23 in privacy exposure.
3. The best collaboration mode shifts with edge capability (RQ3). Weak edges benefit most from external routing and direct cloud execution; stronger edges benefit more from edge-controlled invocation and cloud advisory. Two edge-side abilities underlie the shift: deciding when to invoke the cloud (up to +4.4 over a matched random-invocation baseline) and following cloud guidance (instruction-following training raises advisory completion from 24.8 to 28.8).

Design Highlights

	What we test	Why it matters
🦞 OpenClaw-native	The real OpenClaw agent loop (bash, browser, file ops, APIs, and reusable SKILL.md skills) driving a live local workspace	Tasks need long-horizon planning, state tracking, and error recovery. Cloud calls land mid-trajectory over accumulated workspace context, not on a static prompt
🔀 Collaboration-centric	4 strategies spanning who decides when the cloud joins (external router vs. edge agent) and what the cloud does (execute vs. advise)	Isolates how collaboration is organized from which models are used
🔐 Privacy exposure	100 tasks with 3,385 sensitivity units (personal + organizational, 12 categories); an LLM judge reaches 96.9% F1 against human labels	Every cloud invocation is a potential leakage channel, so we audit what crosses the boundary over the whole trajectory
⚖️ Multi-dimensional	Task performance · Resource usage · Privacy exposure, reported jointly	No single number hides the trade-off
📦 Reproducible	Each task runs in its own Docker container	Graders are injected only after the agent finishes and are never visible during execution

---

Tasks

128 executable task definitions across 6 categories (Chinese & English). Each Markdown file under tasks/EC2Bench/ contains a prompt, an inline grade() verifier, a workspace path, and a descriptive privacy_note. Sensitive-unit evaluation reads gt/sensitive_items.json under each task workspace.

Category	#	Example tasks	Core challenges
Office & Daily Tasks	36	ambiguous contact email, meeting notes, expense report, daily summary	Multi-source aggregation, clarification, structured output
Information Search & Gathering	34	email search, competitive intelligence, paper affiliation lookup, CRM bug hunt	Web + local data reconciliation, source verification
Safety & Security	21	leaked API-key detection, prompt injection, malicious skill, HIPAA/PHI referral	Adversarial robustness, credential awareness, refusal
Development & Operations	14	system health check, automation-failure recovery, LLM API gateway skill	Undocumented setups, debugging, skill creation
Data Analysis	13	order-profit analysis, month-end reconciliation, quarterly business insight	Spreadsheet reasoning, state verification
Automation	10	flight booking, n8n workflow report, scheduled-briefing skill	Long-horizon orchestration, recovery

Tasks are grouped by their original source labels: Claw-Eval (68), WildClawBench (22), QwenClawBench (21), LiveClawBench (4), PinchBench (3), and ClawBench (3). The ClawBench group includes two TrajectoryRL scenarios and one communication-audit task from an earlier task collection, with its original identifier preserved. Seven more tasks were curated to cover under-represented scenarios. Every task was standardized to one runtime and evaluation interface, then filtered so that edge-only, cloud-only, and collaborative execution can meaningfully differ.

Scoring. Every run is graded on three dimensions at once:

• Task Performance: completion score (weighted, task-specific checks combining rule-based validation, environment-state auditing, and rubric-based LLM-as-a-judge) plus Pass³, the fraction of tasks with score > 0.7 in all 3 runs.
• Resource Usage: cloud input / cache-read / output tokens with USD cost, plus edge-side FLOPs.
• Privacy Exposure: the fraction of annotated sensitivity units that become visible to the cloud anywhere along the trajectory (lower is better).

---

Leaderboard

Edge = Qwen3.5-2B / 9B / 27B and Qwen3.7-Plus (27B and Plus serve as proxies for future, stronger edge models); Cloud = GLM-5.3; judge = GPT-5.4-mini; averaged over 3 runs. Cloud Tok. = raw / cache-read / output (millions); Cloud Cost in USD (OpenRouter pricing); Edge FLOPs in PetaFLOPs (not reported for Qwen3.7-Plus, whose parameter count is unavailable); Completion, Pass³, and Privacy Exposure in %.

Text version of the table above

Method	Completion ↑	Pass³ ↑	Cloud Tok. (R/C/O)	Cloud Cost ↓	Edge FLOPs ↓	Privacy Exp. ↓
Edge-only
Qwen3.5-2B	12.75	2.34	–	–	142.98	0.00
Qwen3.5-9B	39.12	9.38	–	–	121.64	0.00
Qwen3.5-27B	61.79	30.47	–	–	564.49	0.00
Qwen3.7-Plus	71.34	53.12	–	–	–	0.00
Cloud-only
DeepSeek-v4-Pro	72.90	53.12	5.81 / 50.20 / 1.60	16.41	–	86.66
GLM-5.3	82.70	71.88	13.52 / 65.76 / 4.64	56.42	–	87.01
Edge-cloud collaboration (cloud = GLM-5.3)
Qwen3.5-2B
Query Routing	23.98	12.50	1.81 / 10.49 / 0.79	8.76	156.30	12.59
Per-step Routing	39.97	7.81	3.84 / 17.29 / 0.67	12.80	57.05	41.29
Edge-Ctrl. Delegation	15.86	2.34	0.41 / 0.49 / 0.11	1.19	172.74	2.71
Edge-Ctrl. Advisory	14.75	1.56	0.67 / 0.51 / 0.29	2.36	240.28	6.07
Qwen3.5-9B
Query Routing	43.60	17.97	2.13 / 11.28 / 0.78	9.33	162.88	12.59
Per-step Routing	52.71	21.88	1.98 / 7.46 / 0.31	6.07	29.96	27.52
Edge-Ctrl. Delegation	52.88	17.19	0.83 / 1.03 / 0.36	3.01	184.98	16.45
Edge-Ctrl. Advisory	60.46	26.56	2.02 / 2.61 / 1.19	8.73	332.43	45.46
Qwen3.5-27B
Query Routing	65.53	38.28	2.69 / 14.29 / 0.75	10.76	649.92	12.53
Per-step Routing	68.42	42.19	1.76 / 6.51 / 0.26	5.29	88.58	35.81
Edge-Ctrl. Delegation	69.06	44.53	1.00 / 1.50 / 0.71	4.93	455.76	42.42
Edge-Ctrl. Advisory	70.13	47.66	1.32 / 0.75 / 0.83	5.71	822.23	48.46
Qwen3.7-Plus
Query Routing	73.17	60.16	4.39 / 15.60 / 0.88	14.03	–	12.51
Per-step Routing	73.77	56.25	2.17 / 4.95 / 0.22	5.28	–	53.17
Edge-Ctrl. Delegation	74.24	60.94	1.65 / 1.08 / 0.33	4.03	–	43.36
Edge-Ctrl. Advisory	76.84	67.19	1.97 / 0.53 / 0.88	6.80	–	78.74

Alternative cloud model: DeepSeek-V4.1-Flash (paper Appendix A.8)

Method	Completion ↑	Pass³ ↑	Cloud Tok. (R/C/O)	Cloud Cost ↓	Edge FLOPs ↓	Privacy Exp. ↓
Edge-only
Qwen3.5-2B	12.75	2.34	–	–	142.98	0.00
Qwen3.5-9B	39.12	9.38	–	–	121.64	0.00
Qwen3.5-27B	61.79	30.47	–	–	564.49	0.00
Qwen3.7-Plus	71.34	53.12	–	–	–	0.00
Cloud-only
DeepSeek-v4-Pro	72.90	53.12	5.81 / 50.20 / 1.60	16.41	–	86.66
DeepSeek-v4.1-Flash	81.42	71.88	14.35 / 90.32 / 2.77	4.08	–	90.85
Edge-cloud collaboration (cloud = DeepSeek-v4.1-Flash)
Qwen3.5-2B
Query Routing	23.46	13.28	2.60 / 8.96 / 0.38	0.64	139.06	13.08
Per-step Routing	42.70	10.94	8.11 / 17.72 / 0.51	1.57	70.04	46.31
Edge-Ctrl. Delegation	19.20	2.34	1.00 / 0.74 / 0.10	0.21	145.46	9.52
Edge-Ctrl. Advisory	15.53	3.12	0.70 / 0.88 / 0.19	0.22	178.10	7.87
Qwen3.5-9B
Query Routing	45.66	21.88	2.08 / 11.69 / 0.35	0.56	196.51	13.03
Per-step Routing	50.36	19.53	2.57 / 4.15 / 0.14	0.48	31.20	28.74
Edge-Ctrl. Delegation	51.79	17.19	1.66 / 1.17 / 0.29	0.43	204.07	21.18
Edge-Ctrl. Advisory	56.00	24.22	2.82 / 2.50 / 0.62	0.80	340.08	39.25
Qwen3.5-27B
Query Routing	63.51	39.84	3.68 / 28.34 / 0.52	0.95	601.08	13.07
Per-step Routing	67.24	42.97	3.96 / 8.47 / 0.27	0.78	85.38	38.47
Edge-Ctrl. Delegation	67.11	41.41	2.34 / 1.45 / 0.47	0.63	605.25	46.19
Edge-Ctrl. Advisory	69.50	46.09	1.73 / 1.31 / 0.55	0.59	636.49	49.02
Qwen3.7-Plus
Query Routing	73.88	60.16	4.07 / 14.97 / 0.48	0.94	–	12.67
Per-step Routing	73.86	53.91	2.61 / 2.67 / 0.15	0.49	–	55.37
Edge-Ctrl. Delegation	75.04	59.38	2.41 / 2.75 / 0.41	0.61	–	44.54
Edge-Ctrl. Advisory	75.31	61.72	1.99 / 1.11 / 0.52	0.61	–	65.56

---

Quick Start

EC²-Bench runs each task in an isolated Docker container bundling the OpenClaw harness and per-task mock services.

1. Install dependencies & load the image

The prebuilt container image is hosted on Hugging Face as a docker save tarball. Hugging Face is not a Docker registry, so download it and docker load instead of docker pull:

cd EC2-Bench
conda create -n ec2bench python=3.13 -y
pip install -r requirements.txt
# download the image
hf download chengpingan/EC2-Bench \
    Images/acebench-openclaw-v1.0.tar.gz --repo-type dataset --local-dir .

# download the workspaces (single tarball) and extract
hf download chengpingan/EC2-Bench \
    workspace/ACE_Bench.tar.gz --repo-type dataset --local-dir .
tar -xzf workspace/ACE_Bench.tar.gz      # extracts into workspace/ACE_Bench/

docker load -i Images/acebench-openclaw-v1.0.tar.gz   # loads acebench-openclaw:v1.0 (must match DOCKER_IMAGE in .env)

2. Configure keys: copy .env.example to .env and fill in:

OPENROUTER_API_KEY=...                 # cloud collaborator (any OpenAI-compatible provider)
JUDGE_API_KEY=...                      # LLM-as-a-judge for task checks & privacy exposure
JUDGE_MODEL=gpt-5.4-mini

Edge / cloud model endpoints (e.g. a local vLLM server) live in my_api.json and are passed via --models-config.

3. Prepare task assets

bash script/prepare.sh

4. Run: pick a strategy via --run-mode. All strategies share one task suite:

Strategy	--run-mode	Who decides when the cloud joins	What the cloud does
Edge-only	local-only	(never)	Nothing; every action comes from the edge model
Cloud-only	cloud-only	(always)	Generates every action; capability upper bound, highest exposure
Query Routing	query-router	External router, once per task (RouteLLM MF over the task instruction)	Runs the whole trajectory when the task is routed to it
Per-step Routing	step-router	External router, every step (edge next-token entropy, GlimpRouter-style)	Generates the routed steps
Edge-Ctrl. Cloud Delegation	not included in this release	Edge agent, via a delegation tool	Generates the delegated step, then hands control back
Edge-Ctrl. Cloud Advisory	advisor	Edge agent, via an advisor tool	Returns plans / suggestions / corrections; the edge still acts

In all collaborative modes, the cloud receives the task instruction plus the OpenClaw execution history (previous actions, tool calls, observations). No redaction is applied, matching the paper's setting.

# Edge-only baseline
python3 eval/run_batch.py --category EC2Bench --parallel 16 --repeat 3 \
  --edge-model vllm/Qwen/Qwen3.5-27B --models-config my_api.json \
  --output-dir output/edge_only/qwen3.5-27b

# Cloud-only baseline
python3 eval/run_batch.py --category EC2Bench --parallel 16 --repeat 3 \
  --cloud-model your-provider/glm-5.3 --models-config my_api.json \
  --output-dir output/cloud_only/glm-5.3

# Query Routing
python3 scripts/run.py query-routing --data-root /path/to/dataset \
  --query-router-table configs/query_router_routes.json \
  --models-config configs/models.example.json --parallel 8 --repeat 3 \
  --output-dir output/query-routing

# Per-step Routing: the entropy threshold defaults to a per-size calibrated value (1.3 for 27B, 1.7 otherwise);
# set GLIMPSE_ENTROPY_THRESHOLD=<tau> to override it and hit your target cloud-call rate
python3 eval/run_batch.py --category EC2Bench --parallel 8 --repeat 3 \
  --run-mode step-router --no-redact \
  --edge-model vllm/Qwen/Qwen3.5-27B --cloud-model your-provider/glm-5.3 \
  --models-config my_api.json \
  --output-dir output/per-step-routing/qwen3.5-27b_to_glm-5.3

# Edge-Controlled Cloud Advisory
python3 eval/run_batch.py --category EC2Bench --parallel 8 --repeat 3 \
  --run-mode advisor \
  --edge-model vllm/Qwen/Qwen3.5-27B --cloud-model your-provider/glm-5.3 \
  --models-config my_api.json \
  --output-dir output/edge-ctrl-advisory/qwen3.5-27b_to_glm-5.3

Single-task runs (--task tasks/EC2Bench/EC2Bench_task_44_ambiguous_contact_email.md), task filters (--task-filter), and privacy-judge timing (--privacy-judge-mode {inline,deferred,off}) are also supported.

Other run modes in this codebase (not evaluated in the paper)

• pipeline-plan-executor: the cloud drafts a task-level sketch once, and the edge executes locally.
• on-demand: the edge agent queries a cloud assistant through a shell helper (ask_cloud.sh) for guidance.
• orchestrator-v1: the cloud model acts as the main agent / orchestrator and dispatches work to an edge worker.

---

Check the Results

Per-task outputs land under output/<run>/<task_id>/... (scores, token/cost usage, agent trace, produced files). A per-category and global summary is generated automatically once the run finishes.

Cloud costs in the paper are computed from the logged raw / cache-read / output token counts with OpenRouter prices (USD per million tokens):

Cloud model	Input	Cache input	Output
GLM-5.3	1.40	0.26	4.40
DeepSeek-v4-Pro	1.74	0.0145	3.48
DeepSeek-V4.1-Flash	0.15	0.003	0.60

The built-in price table in utils/grading.py does not cover these models yet. When you use them, compute the cost from the logged token counts with the prices above.

---

Citation

If you find EC²-Bench useful, please cite:

@article{huang2026ec2bench,
  title   = {Empowering Edge Agents: A Systematic Analysis of Edge-Cloud Collaboration in LLM Agent Execution},
  author  = {Huang, Pengcheng and Liu, Zhenghao and Ren, Shixin and Yu, Shi and Yao, Sijia and Yan, Yukun and Xiao, Chaojun and Sun, Maosong},
  journal = {arXiv preprint arXiv:2610.17816},
  year    = {2026}
}

Acknowledgements

EC²-Bench stands on the shoulders of a remarkable open-source agent community, and we are deeply grateful for it. The OpenClaw harness gives us a real, full-featured agent runtime (tools, skills, and a live workspace) to build on. Our tasks and evaluation design draw inspiration and adapted material from a series of outstanding agent benchmarks: Claw-Eval, WildClawBench, QwenClawBench, LiveClawBench, PinchBench, and ClawBench. Their task curation, grading, and reproducible harness design set the bar for trustworthy agent evaluation and made the privacy-aware, edge–cloud extension in EC²-Bench possible.

Query Routing uses the bundled configs/query_router_routes.json, scored with OPEA/routellm-e5-base-v2 and E5 mean pooling. The cloud is selected when score > threshold; equality selects the edge. Install requirements-routing.txt to score new prompts. --scores-json also accepts the bundled table to rebuild routes from saved scores.

overall_std is the sample SD across task means; repeat_overall_std is the sample SD across repeat means. Resource totals aggregate the retained execution attempt for each task and repeat.

License

Released under the MIT License. Adapted tasks are used under the licenses of their source benchmarks: Claw-Eval, WildClawBench, QwenClawBench, LiveClawBench, and PinchBench are MIT-licensed, and the TrajectoryRL ClawBench scenarios (tasks 8 and 9) are MIT-licensed. Applicable upstream copyright and license notices are preserved in THIRD_PARTY_NOTICES.