Repository navigation
execProcess garbles a multi-byte character that falls across two output chunks #15341
Description
I have:
- searched the issue tracker for similar issues
- installed the latest version of Quarto CLI
- formatted my issue following the Bug Reports guide
Bug description
execProcess decodes each chunk of a subprocess output on its own.
When a multi-byte UTF-8 character is split across two chunks, both halves turn into U+FFFD.
The code is in processOutput in src/core/process.ts:
const decoder = new TextDecoder();
let outputText = "";
for await (const chunk of iterator) {
// ...
const text = decoder.decode(chunk);
outputText += text;
}filteredAsyncIterator in the same file does the same with decoder.decode(data).
Neither call passes { stream: true }, so the decoder cannot hold back the incomplete bytes at the end of a chunk.
I found it through quarto dev-call build-artifacts.
After I added a few lines of description text to src/resources/schema/definitions.yml, the CSL event-place description came out as ��Amsterdam, The Netherlands” in vs-code.mjs, web-worker.js and yaml-intelligence-resources.json, on every run.
createHtmlDescriptions in src/core/schema/build-schema-file.ts sends all descriptions to Pandoc through execProcess, so any change in the text length moves the chunk boundaries.
Steps to reproduce
Save this as repro.ts at the root of a quarto-cli checkout:
import { execProcess } from "./src/core/process.ts";
for (const n of [16382, 65534]) {
const result = await execProcess({
cmd: "python3",
args: ["-c", `import sys; sys.stdout.buffer.write(b"a" * ${n} + "“".encode())`],
stdout: "piped",
});
const tail = result.stdout!.slice(-3);
console.log(n, [...tail].map((c) => c.codePointAt(0)!.toString(16)));
}Run it with the bundled Deno:
package/dist/bin/tools/aarch64/deno run --allow-all --import-map=src/import_map.json --no-lock repro.tsActual behavior
16382 [ "61", "61", "201c" ]
65534 [ "61", "fffd", "fffd" ]With 65,534 bytes before it, the three bytes of “ cross the 64 KiB read boundary, and the string ends with two U+FFFD.
Expected behavior
The output ends with U+201C in both cases.
Decoding with decoder.decode(chunk, { stream: true }) and calling decoder.decode() once after the loop would keep split characters whole.
The same applies to filteredAsyncIterator.
Your environment
- Quarto: development build from
main(ac310a4). - OS: macOS 27.2.0 (arm64).
- Deno 2.9.8, Pandoc 3.10.0.
Quarto check output
Quarto 99.9.9
[✓] Checking environment information...
Quarto cache location: /Users/mcanouil/Library/Caches/quarto
Project: none found (single-file mode)
[✓] Checking versions of quarto binary dependencies...
Pandoc version 3.10.0: OK
Dart Sass version 1.107.1: OK
Deno version 2.9.8: OK
Typst version 0.15.1: OK
[✓] Checking versions of quarto dependencies......OK
[✓] Checking Quarto installation......OK
Version: 99.9.9
Path: /Users/mcanouil/Projects/quarto-dev/quarto-cli/package/dist/bin
[✓] Checking tools....................OK
TinyTeX: v2026.11
Chrome Headless Shell: (not installed)
VeraPDF: (not installed)
[✓] Checking LaTeX....................OK
Using: TinyTex
[✓] Checking basic markdown render....OK
Activity
cscheid commentedon Nov 11, 2026
It's really wild that it took 5 years for this bug to surface!
mcanouil commentedon Nov 11, 2026
Heavy work on listings these days on my side (rework of #12667 as #15342, and #15339 for accessibility). I was puzzled at the diff I got and let Claude investigate what was happening.
cderv commentedon Nov 12, 2026
Thanks for the report and the repro. I can reproduce on Windows too.
It is not limited to piped stdout. stderr, stderr with a filter and merged output all end with fffd fffd once a character straddles the 64 KiB read boundary. So any subprocess output going through execProcess can be affected, including the knitr output we print to the console during render.
I'll look into it.
Keep multi-byte characters whole across execProcess output chunks (qu…
cderv commentedon Nov 12, 2026
The fix is merged in #15346, which closed this issue. It is in the v1.12 milestone.
Besides the streaming decode and the final flush suggested above, the fix covers two cases the report did not mention. Merged stdout and stderr now go through the decoder before they are combined, so a pending character on one stream cannot be joined with bytes from the other. The inherit branch of
processOutputalso decoded each chunk twice (once for printing, once for accumulation), which a streaming decoder cannot handle, so each chunk is now decoded once and the text is reused.