[b84330808]

Description

The shell step decodes captured output with locale.getpreferredencoding(False), as text=True would (#5064). In Python's UTF-8 mode (PYTHONUTF8=1, -X utf8, and the default from Python 3.15) that is always "utf-8". But cmd.exe and other native Windows programs still write in the ANSI code page, so on a cp932 or cp936 system any shell step that prints non-ASCII text fails.

Reproduction on Windows (code page 932, Python 3.11), running ShellStep().execute({"id": "s", "type": "shell", "run": "echo テスト"}, StepContext(project_root=".")):

• python -X utf8, on main: FAILED, Shell command stdout could not be decoded: 'utf-8' codec can't decode byte 0x83 in position 0: invalid start byte
• python without UTF-8 mode, on main: COMPLETED, stdout 'テスト\n'
• python -X utf8, with this change: COMPLETED, stdout 'テスト\n'

With this change, output that is not valid in the preferred encoding is decoded again with locale.getencoding(), the code page text=True used outside UTF-8 mode. Nothing changes when the two encodings are the same (outside UTF-8 mode, or on a UTF-8 locale). Output that is invalid in both still fails the step with the original error, so #5064's behavior is kept.

Testing

New tests in tests/workflows/test_shell_step.py, run in a fresh Python 3.11 venv on Windows:

• test_code_page_output_is_decoded_in_utf8_mode[text] and [json] simulate UTF-8 mode on a cp936 system with a child process that writes cp936 bytes. Both fail on main (assert StepStatus.FAILED == StepStatus.COMPLETED) and pass with the change.

• test_output_invalid_in_code_page_too_fails_step (negative case) checks that bytes invalid in both encodings still fail the step with the UTF-8 error. It passes before and after.

• tests/workflows/test_shell_step.py: 73 passed, 21 skipped, both normally and under -X utf8.

• tests/workflows and tests/test_workflows.py: 1412 passed, 28 skipped, 12 failed. The same 12 symlink tests fail on main on this machine (no symlink privilege).

• ruff check passes on both changed files.

• Tested locally with uv run specify --help

• Ran existing tests with uv sync && uv run pytest

• Tested with a sample project (if applicable)

AI Disclosure

• I did not use AI assistance for this contribution
• I did use AI assistance (fill in the disclosure below)

AI disclosure: Claude Code with Claude Opus 5.5, running autonomously on behalf of @b84330808 with default settings. It wrote the code change, the tests, the commit message and this description, and ran the tests and the reproduction above. No human line-by-line review or manual testing is claimed. The commit has an Assisted-by: trailer.

I used Claude Code to write this change.

[Copilot]

🟢 Approval recommended

There are no active review issues for the shell-output decoding fallback.

No open issues 🎉

What changed in this PR

This PR fixes shell-step output decoding when Python’s UTF-8 mode differs from the native system code page, while preserving structured failures for undecodable output.

Changes:

• Retry failed decoding with the locale encoding when it differs from the preferred encoding.
• Preserve universal-newline handling and the original decoding error if both attempts fail.
• Add regression coverage for code-page text and JSON output, plus invalid-byte failure coverage.

File	Description
tests/​workflows/​test_shell_step.py	Tests code-page decoding in simulated UTF-8 mode and failure when neither encoding works.
src/​specify_cli/​workflows/​step/​shell/​__init__.py	Adds locale-encoding fallback for captured stdout and stderr.

_{Review effort: Balanced (auto)}

[mnriem]

Thank you!