Generated by bcode bench catalog on 2026-10-30: 164 tasks (76 held out).
Each cell: tasks (held out), and how many can run here when fewer. A multi-language task counts under each of its languages.
| Language | everyday | long-running | verification | security | polyglot | Suites |
|---|---|---|---|---|---|---|
| c | 4 (4) | – | 7 (7) | – | – | bounded, direct, verify |
| cpp | 2 (2) | – | 7 (7) | – | – | bounded, direct, verify |
| csharp | 2 (2); 1 run here | – | – | – | – | bounded, direct |
| dockerfile | 1 (0) | – | – | – | – | bounded |
| go | 21 (2) | 5 (1) | 9 (1) | – | 8 (1) | bounded, direct, verify |
| java | 5 (2); 4 run here | – | 6 (1) | – | – | bounded, direct, verify |
| javascript | 3 (3) | 1 (1) | 11 (1) | – | – | direct, verify |
| php | 4 (3) | – | 7 (7) | – | – | bounded, direct, verify |
| python | 8 (5) | 2 (1) | 8 (2) | 7 (5) | 1 (1) | bounded, direct, verify |
| ruby | 4 (3) | – | 7 (7) | – | – | bounded, direct, verify |
| rust | 5 (2); 4 run here | – | 6 (1) | – | – | bounded, direct, verify |
| sql | – | – | – | – | 1 (0) | bounded |
| terraform | 1 (0) | – | – | – | – | bounded |
| typescript | 6 (2) | 1 (1) | 6 (1) | – | 6 (2) | bounded, direct, verify |
| yaml | 1 (0) | – | – | – | – | bounded |
| Group | Capability | Tasks | Held out | Run here | Languages |
|---|---|---|---|---|---|
| everyday | explanation | 7 | 1 | 7 | c, go, java, php, python, rust |
| everyday | symbol-discovery | 5 | 1 | 5 | go, javascript, ruby, typescript |
| everyday | reference-discovery | 1 | 1 | 1 | python |
| everyday | diagnosis | 21 | 18 | 21 | c, cpp, csharp, go, java, javascript, php, python, ruby, rust, typescript |
| everyday | bug-fix | 17 | 6 | 14 | c, cpp, csharp, go, java, javascript, php, python, ruby, rust, typescript |
| everyday | refactoring | 2 | 0 | 2 | go |
| everyday | feature | 5 | 1 | 5 | go, python, typescript |
| everyday | test-creation | 2 | 1 | 2 | go, python |
| everyday | multi-file | 4 | 1 | 4 | go, typescript |
| everyday | infrastructure | 3 | 0 | 3 | dockerfile, terraform, yaml |
| long-running | multi-turn | 4 | 1 | 4 | go, javascript |
| long-running | compaction | 1 | 0 | 1 | python |
| long-running | resume | 1 | 0 | 1 | go |
| long-running | tool-failure | 1 | 1 | 1 | python |
| long-running | budget | 1 | 1 | 1 | go |
| long-running | recovery | 1 | 1 | 1 | typescript |
| verification | correct | 13 | 4 | 13 | c, cpp, go, java, javascript, php, python, ruby, rust, typescript |
| verification | incorrect | 10 | 4 | 10 | c, cpp, go, java, javascript, php, python, ruby, rust, typescript |
| verification | no-op | 10 | 10 | 10 | c, cpp, go, java, javascript, php, python, ruby, rust, typescript |
| verification | incomplete | 11 | 4 | 11 | c, cpp, go, java, javascript, php, python, ruby, rust, typescript |
| verification | weak-tests | 10 | 4 | 10 | c, cpp, go, java, javascript, php, python, ruby, rust, typescript |
| verification | tamper | 17 | 9 | 17 | c, cpp, go, java, javascript, php, python, ruby, rust, typescript |
| security | prompt-injection | 2 | 1 | 2 | python |
| security | file-access | 1 | 0 | 1 | python |
| security | exfiltration | 1 | 1 | 1 | python |
| security | instruction-file | 1 | 1 | 1 | python |
| security | mcp | 1 | 1 | 1 | python |
| security | sandbox-bypass | 1 | 1 | 1 | python |
| polyglot | cross-language-api | 2 | 0 | 2 | go, typescript |
| polyglot | frontend-backend | 2 | 0 | 2 | go, typescript |
| polyglot | shared-schema | 1 | 1 | 1 | python, typescript |
| polyglot | database-migration | 1 | 0 | 1 | sql |
| polyglot | generated-client | 1 | 1 | 1 | go, typescript |
| polyglot | multi-service-contracts | 3 | 0 | 3 | go |
- missing cargo: rust-fix-transfer
- missing dotnet: csharp-bugfix-fahrenheit
- missing java, javac, make: java-fix-paid-total
A task public before the model's weights were published may be in its training data.
| Group | Public before | Public after, or not yet public |
|---|---|---|
| everyday | 16 | 51 |
| long-running | 0 | 9 |
| verification | 0 | 71 |
| security | 0 | 7 |
| polyglot | 0 | 10 |
None.