CyberGym: Evaluating AI Agents' Real-World Cybersecurity Capabilities at Scale

CyberGym is a large-scale, high-quality cybersecurity evaluation framework designed to rigorously assess the capabilities of AI agents on real-world vulnerability analysis tasks.

Note

• FAQ.md: frequently asked questions about benchmark setups, such as network access, local deployment, pre-/post-patch versions, interpreting outcomes.
• SUBMISSION.md: guidelines for leaderboard submissions, including cost reporting, writeup requirements, etc.

Warning

Deploy everything locally. Do not expose any part of CyberGym to the public internet.

Installation

Require python and docker environment.

Install the dependencies for the server and the task generation

pip3 install -e '.[dev,server]'

Download the benchmark data (~240GB)

git lfs install
git clone https://huggingface.co/datasets/sunblaze-ucb/cybergym cybergym_data

Download Server Data (binary only mode)

If you only need static analysis and don't require the dynamic compilation environment, you can use this binary mode which only takes ~130GB.

python scripts/server_data/download_binary_only_runners.py
wget https://huggingface.co/datasets/sunblaze-ucb/cybergym-server-binary/resolve/main/cybergym-server-data.7z
7z x cybergym-server-data.7z

Download Server Data (full data)

Download the docker images with the compilation environment:

1. Full data

python scripts/server_data/download.py --tasks-file ./cybergym_data/tasks.json

2. Subset data

The full server data is large (~10TB). We provide a subset with the following 10 tasks, which include 5 tasks that the agent can successfully generate the PoC and 5 tasks that are not easy for the agent.

arvo:47101
arvo:3938
arvo:24993
arvo:1065
arvo:10400
arvo:368
oss-fuzz:42535201
oss-fuzz:42535468
oss-fuzz:370689421
oss-fuzz:385167047

Download the subset data

python scripts/server_data/download_subset.py

Evaluation

The server only needs to be reachable from the agent containers and from the host. 0.0.0.0 binds every interface, which on a machine with a public IP exposes a partly unauthenticated service to the internet.

Pick the bind address

The right address is the gateway of the Docker network your agent containers run on — that address is a host-side interface reachable from those containers and from the host, but not routable from outside the machine.

# If you use the firewall (agents on the isolated `cybergym-internal` network):
HOST=$(docker network inspect cybergym-internal -f '{{(index .IPAM.Config 0).Gateway}}')

# If agents run on the default bridge instead (no firewall):
HOST=$(docker network inspect bridge -f '{{(index .IPAM.Config 0).Gateway}}')

# If the agent runs directly on the host with no container at all:
HOST=127.0.0.1

echo $HOST  # e.g. 192.168.48.1 for cybergym-internal, 172.17.0.1 for the default bridge

Notes:

• The two gateways are not interchangeable. cybergym-internal is created with internal=True, so containers on it have no route to the default bridge — a server bound to 172.17.0.1 is unreachable from firewalled agents.
• Docker assigns the cybergym-internal subnet when the network is created, so the gateway is host-specific. Query it rather than hardcoding it. python3 -m cybergym.firewall start and status both print it as host_gateway.
• FirewallProxyManager.start() already adds this gateway to NO_PROXY and to the proxy's IP allowlist, so agent traffic to the server bypasses Squid.
• Use the same $HOST value in --server when generating tasks and verifying PoCs below.

Start the PoC submission server:

PORT=8666 # port of the server
POC_SAVE_DIR=./server_poc # dir to save the pocs
python3 -m cybergym.server \
    --host $HOST --port $PORT --mask_map_path mask_map.json \
    --log_dir $POC_SAVE_DIR --db_path $POC_SAVE_DIR/poc.db

Start the PoC submission server (binary only mode):

PORT=8666 # port of the server
POC_SAVE_DIR=./server_poc # dir to save the pocs
CYBERGYM_SERVER_DATA_DIR=./cybergym-server-data
python3 -m cybergym.server \
    --host $HOST --port $PORT --mask_map_path mask_map.json \
    --log_dir $POC_SAVE_DIR --db_path $POC_SAVE_DIR/poc.db \
    --binary_dir $CYBERGYM_SERVER_DATA_DIR

Test:

# generate the task
SERVER_IP=$HOST # server ip — same bind address chosen above
SERVER_PORT=8666 # server port
TASK_ID='arvo:10400'
OUT_DIR=./cybergym_tmp
CYBERGYM_DATA_DIR=./cybergym_data/data
python3 -m cybergym.task.gen_task \
    --task-id $TASK_ID \
    --out-dir $OUT_DIR \
    --data-dir $CYBERGYM_DATA_DIR \
    --server "http://$SERVER_IP:$SERVER_PORT" \
    --mask-map mask_map.json \
    --difficulty level1

# ./cybergym_tmp
# ├── description.txt
# ├── README.md
# ├── repo-vul.tar.gz
# └── submit.sh

# try the submission
echo -en "\x00\x01\x02\x03" > $OUT_DIR/poc
bash $OUT_DIR/submit.sh $OUT_DIR/poc

# example return
# {"task_id":"arvo:3848","exit_code":0,"output":"INFO: Seed: 779112339\nINFO: Loaded 1 modules   (6096 guards): 6096 [0x965580, 0x96b4c0), \n/out/pe_fuzzer: Running 1 inputs 1 time(s) each.\nRunning: /tmp/poc\nExecuted /tmp/poc in 3 ms\n***\n*** NOTE: fuzzing was not performed, you have only\n***       executed the target code on a fixed set of inputs.\n***\n","poc_id":"8f20a76a34d0482a82da247f96b39f01"}

Verify the PoCs Submitted by the Agent

After running the agent, you can get the agent_id from the logs/args.json. You can verify the PoCs submitted by:

# NOTE: this is the public default placeholder key. Set your own via CYBERGYM_API_KEY
# on both the server and the client. It is not a substitute for keeping the server private.
export CYBERGYM_API_KEY=cybergym-030a0cd7-5908-4862-8ab9-91f2bfc7b56d
python3 scripts/verify_agent_result.py \
    --server http://$SERVER_IP:$SERVER_PORT \
    --pocdb_path $POC_SAVE_DIR/poc.db \
    --agent_id 8113f33401d34ee3ae48cf823b757ac7

# example output
# {'agent_id': '8113f33401d34ee3ae48cf823b757ac7', 'task_id': 'arvo:3848', 'poc_id': '8f20a76a34d0482a82da247f96b39f01', 'poc_hash': '714f093fe3c90135c2845fa8bbc7dfa429051e7f91d8ce398b3cd011cea15f59', 'poc_length': 662, 'vul_exit_code': 0, 'fix_exit_code': 0, 'created_at': datetime.datetime(2025, 5, 15, 23, 39, 48, 449451), 'updated_at': datetime.datetime(2025, 5, 15, 23, 39, 49, 435333)}

Firewall (Restrict Agent Internet Access)

CyberGym includes a domain-allowlist proxy that restricts agent containers to approved destinations only. It creates an isolated Docker network with no direct internet route — all outbound traffic must pass through a Squid proxy filtered by domain.

Start the proxy (uses the built-in allowlist for apt, pip, and LLM APIs):

python3 -m cybergym.firewall start

Check status, stop the proxy, or tear down everything:

python3 -m cybergym.firewall status
python3 -m cybergym.firewall stop       # stop proxy only
python3 -m cybergym.firewall stop-all   # stop proxy and remove network

Agent containers must be started on the cybergym-internal network and use the proxy env vars (see ProxyManager.env_vars()). The default allowlist is at src/cybergym/firewall/default_allowlist.txt.

Note that the firewall only filters outbound traffic from agent containers. It does nothing about inbound reachability of the submission server — that is still up to how you bind and firewall the host.

See python3 -m cybergym.firewall --help for more usage.

Example Agents

The four example agents can be installed as:

git submodule update --init --recursive examples/agents

Then check the instructions in the folder: Example Agents

Citation

If you use this project in your research, please cite:

@inproceedings{wang2026cybergym,
    title={CyberGym: Evaluating {AI} Agents' Real-World Cybersecurity Capabilities at Scale},
    author={Zhun Wang and Tianneng Shi and Jingxuan He and Matthew Cai and Jialin Zhang and Dawn Song},
    booktitle={The Fourteenth International Conference on Learning Representations},
    year={2026},
    url={https://openreview.net/forum?id=2YvbLQEdYt}
}

License

This project is licensed under the Apache License 2.0.