Skip to content

test(vela): clarify #169 span coordinates and contract acceptance - #206

Draft
psv666 wants to merge 2 commits into
mainfrom
codex/l20-b-vela-contract
Draft

psv666 wants to merge 2 commits into
mainfrom
codex/l20-b-vela-contract

Conversation

@psv666

@psv666 psv666 commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Required human contributor answers

The contributor confirmed this single stage on 2026-11-19: Vela #169 contract audit and acceptance wording. #137 and the native PRs provide background/evidence; implementation of the native stack and other models is outside this draft's scope. Detailed human confirmation is retained in the contributor's task record.

  1. Outcome: A consistent Vela contract and reproducible, reviewable acceptance record.
  2. Highest-leverage variable: Reliable model behavior and numerical/contract consistency.
  3. How this moves it: Reuse the existing reference, resolve its wording/evidence gaps, and add focused documentation and acceptance cases.
  4. User and need: Model, application and agent developers who need stable interfaces, reliable outputs and reproducible deployment.

Is a PR needed? Yes. The contributor authorized necessary contract documentation/tests and a stage draft PR.

Observed gaps: #169 records the conflict between #137's old span/set rejection/no-truncation proposal and the shipped reference. The recipe suggested token usage as a tail-coverage check, which the actual assembly/window policy does not guarantee. Source inspection and a preparation-only repro also show distinct span coordinate origins and an unresolved cross-part-list failure. This PR changes no inference or transport behavior.

Summary

Document Vela's existing five-kind policy, clarify span coordinate origins and pending decisions, and bind a real non-BMP CPU oracle to maintained acceptance tests.

over: "request"    -> field-local code-point offsets
over: ["request"]  -> rendered-part offsets when fields share a part
over: "user"       -> rendered-part offsets
cross-part list   -> unresolved upstream preparation failure

Preserve span/set extensions, upstream state truncation, model-defined confidence and actual rendered-row token accounting. The shared RFC #61 remains open. Related: #169, #137, merged #151/#174. This does not mark native/family acceptance complete.

Evidence

  • The real five-kind CJK/non-BMP oracle retains EMAIL_ADDRESS [10,30) after an emoji; slicing the named request field gives zhang.wei@example.cn. Independent set probabilities sum to more than1.3. The actual partial PERSON prediction is preserved without a task-accuracy claim.
  • Opt-in tests exercise the hash-pinned upstream code: validation/precedence, field versus part coordinate origins, actual8192-token assembly/truncation and schema-only overflow. Controlled outputs/token IDs are explicitly distinguished from model-forward evidence.
  • Cross-part reproduction: state has request and source; a span uses over: ["request","source"]. Mapping resolves target_part=["user","context"]; upstream _to_generic -> _prep raises TypeError: unhashable type: 'list' before a model forward. This is recorded as unresolved scope/error behavior, not blessed as a client-error contract.

Personally executed on L20 Linux, baseline f1e674f2d324f5bb4e1ba2162551d0b633f20be0, final candidate bfc766c103ccf529b3f277cba38550c42f04ae2e:

Check Result
Original main ordinary Vela suite 135 passed
Final candidate with pinned-package opt-in 161 passed,0 failed
Hash-pinned upstream contract cases 14 cases included in the161
Original reference-vs-worker comparison All9 original cases exact in status/body/key order
Extra Unicode reference comparison HTTP200 exact;6 ordered answer entries; usage155/0
Actual worker/current-main Rust frontend First Unicode200, invalid span422, invalid set422, state-error precedence422, schema-only overflow413:5/5 byte-equal status/content-type/body
Strict MkDocs, Python compileall, whitespace and model-README links PASS

The real-model/HTTP runs preceded the final documentation/controlled-coordinate follow-up; their inference/transport source is unchanged. With no package env, the14 upstream checks skip explicitly; ordinary tests need no weights/torch/GPU and perform no download.

case: example-request-unicode.json
both paths: HTTP 200
answers: 6 key(s) in pinned order
input_tokens: 155 == reference
exact match (status, body, key order): PASS

first-unicode PASS 200
invalid-span PASS 422
invalid-set PASS 422
error-precedence PASS 422
schema-only-overflow PASS 413

161 passed in 10.35s

Purpose

Provide #169's Vela-specific current-state/acceptance record, correct the tail-coverage advice and relative recipe links, and name the remaining decisions without changing external behavior.

Pending decisions

  • Formal [RFC]: Write down which parts of the /v1/systemone contract all workers share #61 adoption of model-defined extension fields, confidence/usage/truncation and reconciliation of [New Model]: Support the Vela 2.0 series #137's old wording. Discussion favors following the model reference, but is not a completed shared-contract decision.
  • Whether the accepted span contract permits part/list coordinates, requires a single named field, or needs explicit origin metadata. The current response correlates target through the request; no new response field is introduced.
  • Supported behavior and client-error status for cross-part span lists; the current upstream preparation failure remains unfixed.
  • Shared error-envelope policy versus Vela's reference detail list and documented transport differences.
  • Native member/oracle/precision, router/broad execution and structural parity remain independently assigned acceptance work.

Test Plan

  • Pinned reference/runtime/tokenizer/calibration: vllm-sr/Vela-2.0-0.3B@d6f03aa9baca4f9017fa8f8e9c2764d3bed3f409; repository package gate plus full model LFS SHA256 bd37dd0db177d766bc554eded78ae8493bc248172d0480f6a543b30932a0cc8e.
  • CPU oracle: Python3.12.13, official torch2.11.0+cpu, transformers5.19.0, tokenizers0.23.2, numpy2.5.3, safetensors0.8.0; OMP/MKL8 threads, no FLA. No GPU execution is needed for this contract stage.
  • Actual HTTP: one warmed Python CPU worker and current-main Rust frontend, serial loopback requests; Rust1.100.0 binary SHA256 a56fc7bebdd3bfbb4edbbb80083c6a0421a7d51ab232533e869a14348f02bd0f.
  • Recipe documents all reproduction commands. Original four oracle files remain unchanged. Only the small consumed Unicode request/oracle is added.
  • Original raw archive vela03-evidence.tar.gz, SHA256 bcdaf65b82cfc10e9fd0ca8e91cae4e03b9ff077c960e8b5c1914e97ad12b1a2, is retained on contributor-local and L20 storage, with final coordinate repro/test/docs output retained separately. Raw machine logs remain outside product Git.

Test Result

Scoped CPU/reference/HTTP/documentation checks pass. The earlier Windows ordinary suite had128 pass/7 failures from Linux-extension assumptions, symlink privileges and default CJK decoding; the original suite passed on Linux. New fixture consumers use explicit UTF8; unrelated platform tests are unchanged.

No external behavior, calibration or numerical gate is changed. Formal contract adoption, issue closure and native acceptance are not claimed.

Demo / evidence

The committed oracle is an actual CPU prediction captured by the documented exporter. Real direct/frontend comparisons preserve complete response bodies with no timing-field exemption. Video and performance A/B are inapplicable to this docs/contract-test change.

Merge Danger

Door: two-way.

Blast Radius: contract.

230 added Python test lines; total+616/-10, including a215-line consumed oracle. Tests import only a package accepted by the existing hash gate. The docs describe current behavior and pending policy; they do not standardize other workers.

Self-review

Agent precheck reviewed the six-file diff, owning upstream functions, production response gate, fixture provenance/consumers, ordinary-test independence and docs links. All test bodies/fixtures are under root tests. The contributor checklist remains for human review; keep draft.

  • I have reviewed the full diff and addressed the issues I found.
  • I have checked that the change follows the project's architecture and stays focused on the stated purpose.
  • I have run the checks appropriate to this change and reported commands, results, and anything that I could not verify above.
  • I have checked that the PR description, documentation, and any accuracy or performance claims match the implementation and available evidence.

@psv666 psv666 changed the title test(vela): record span/set policy and pin Unicode contract acceptance test(vela): record #169 contract decisions and Unicode acceptance Nov 19, 2026
@psv666 psv666 changed the title test(vela): record #169 contract decisions and Unicode acceptance test(vela): clarify #169 span coordinates and contract acceptance Nov 19, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant