Skip to content

refactor(mcp): count UTF-16 units with unicode/utf16.RuneLen in the toolkit - #1488

Merged
HuiJun merged 1 commit into
developfrom
refactor/toolkit-utf16-runelen
Oct 25, 2026
Merged

HuiJun merged 1 commit into
developfrom
refactor/toolkit-utf16-runelen

Conversation

@devin-ai-integration

Copy link
Copy Markdown
Contributor

What and why

internal/frontend/mcp/toolkit carried its own utf16Len(r rune) uint32 (2 for a rune at or above U+10000, otherwise 1), which is unicode/utf16.RuneLen — the function internal/syntax/semtok/encode.go already uses for the same count. Position.ToProtocol and FromProtocol now call utf16.RuneLen directly and the local helper is gone.

Behaviour is identical. Both callers range over the runes of a Go string, which decodes invalid bytes to U+FFFD and never yields a surrogate, so RuneLen is always 1 or 2 here and never its −1 sentinel. The two uint32(...) conversions carry a #nosec G115 line for that reason, as lineNumber in the same file already does.

How it was verified

TestPositionRoundTripsThroughUTF16 and TestColumnsCountCharactersNotBytes in internal/frontend/mcp/toolkit/toolkit_test.go pin the emoji (two-unit) case and the round trip through both functions; they pass unchanged.

go build ./...          ok
go vet ./...            ok
gofmt -l .              (nothing)
staticcheck ./internal/frontend/mcp/toolkit/...   ok
gosec ./internal/frontend/mcp/toolkit/...         0 issues
python3 scripts/changelog.py check                ok
go test ./...           all packages ok except cmd/sysml and tests/codegen, which hit Go's
                        10 m per-package default under full-suite load (goroutine dump,
                        no assertion failure); rerun alone:
go test -count=1 -timeout 40m ./cmd/sysml ./tests/codegen
ok  github.com/Open-MBEE/OpenSysML/cmd/sysml      348.887s
ok  github.com/Open-MBEE/OpenSysML/tests/codegen  373.585s

Checklist

  • make test and make lint pass locally (see above; lint run on the touched package)
  • Tests added or updated for the change — existing toolkit tests already pin the preserved behaviour
  • Documentation extended where it already covers the surface (nothing documents the helper)
  • Changelog entry added as changes/unreleased/toolkit-utf16-runelen.changed.md, not as an edit to CHANGELOG.md
  • baselines regenerated and make docs-counts run if a gate count moved (none moved)
  • No internal work-item labels (waves, slices, F4, K5) in the body, docs, or changelog

Link to Devin session: https://nasa-jpl-demo.devinenterprise.com/sessions/dd67c09df22146688969e4a7459a5eed
Open in Devin Desktop: https://nasa-jpl-demo.devinenterprise.com/desktop/session/dd67c09df22146688969e4a7459a5eed?variant=devin
Requested by: @HuiJun

…oolkit

Co-Authored-By: jason.han <hanhuijun@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@devin-ai-integration
devin-ai-integration Bot marked this pull request as ready for review October 25, 2026 03:45

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@HuiJun
HuiJun merged commit 0cb6e4d into develop Oct 25, 2026
53 checks passed
@HuiJun
HuiJun deleted the refactor/toolkit-utf16-runelen branch October 25, 2026 07:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant