[pmachapman]

Fixes #414.

This PR fixes the following crash that occurs when a source string contains a broken surrogate:

TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]

---

This change is 

[codecov-commenter]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.30%. Comparing base (6be62fd) to head (a10a2aa).

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #418   +/-   ##
=======================================
  Coverage   92.30%   92.30%           
=======================================
  Files         399      399           
  Lines       25488    25509   +21     
=======================================
+ Hits        23526    23547   +21     
  Misses       1962     1962           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[claude]

Verdict: approve. A split surrogate pair is joined before it reaches the engine and the UTF-8 alignment spool. The test now pins a lone surrogate as a build failure, which the author kept on purpose.

Most important: F1 (accepted), machine/jobs/nmt_engine_build_job.py:260. A single lone or reversed surrogate still fails the whole build with a UnicodeDecodeError that names no segment.

Counts: Critical 0, Important 0 open (F1 accepted), Low 0 open. No new findings this round.

Ran: the compare from 86b78fb to a10a2aa reports diverged, so I re-read the whole PR diff. uv run pytest tests/jobs/test_nmt_engine_build_job.py: 7 passed. /code-review high: its candidates were either outside this diff or duplicated F1. source_segments is the list passed to both translate_batch and the spool, so the sourceTokens assertion also covers what the engine receives. CI from gh pr checks 418: all 15 builds are pending, and comment hygiene passes (advisory).

Not verified: whether Serval sends lone surrogates, and the fix against a real MBart tokenizer, since the engine is mocked.

Public API, published wheel, optional dependencies, and sillsdev/machine parity: None verified. The helper is private, and machine/jobs has no C# counterpart.

Findings: F1 accepted, F2 addressed, F3 addressed.

Reviewed at a10a2aa

[Enkidu93]

@Enkidu93 reviewed 2 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on pmachapman).

---

tests/jobs/test_nmt_engine_build_job.py line 93 at r2 (raw file):

def test_run_with_lone_surrogate(decoy: Decoy) -> None:

I think I'd prefer to just ignore the lone surrogate. We have seen some of these sneak through and I'd hate to fail an entire build because one character is off. Is there a reason you figured failing it would be preferable?