fix(app): declare utf-8 on text responses - #10492
Merged
Merged
Conversation
llms.txt and robots.txt carry em dashes and arrows but were served as bare text/plain (live-verification sweep 2026-08-19) - a strict client decodes them as Latin-1 mojibake. charset utf-8 in both server blocks; the daily bot-serving monitor asserts it on llms.txt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Contributor
There was a problem hiding this comment.
Pull request overview
This PR ensures static machine-facing text files are served with an explicit UTF-8 charset so strict clients decode llms.txt / robots.txt correctly (avoiding Latin‑1 mojibake), and extends the daily bot-serving monitor to enforce that contract going forward.
Changes:
- Add
charset utf-8;to both nginx server blocks sotext/plainresponses includecharset=utf-8. - Extend the daily bot-serving GitHub Actions workflow to assert
llms.txt’sContent-Typeincludescharset=utf-8. - Add a corresponding
[Unreleased]changelog entry describing the fix.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
CHANGELOG.md |
Documents the UTF-8 charset fix under [Unreleased]. |
app/nginx.conf |
Declares charset utf-8; in both nginx server blocks to ensure text responses include a UTF-8 charset. |
.github/workflows/bot-serving-check.yml |
Adds a monitor assertion that llms.txt is served with charset=utf-8 and updates the timeout commentary accordingly. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
1 task
MarkusNeusinger
added a commit
that referenced
this pull request
Aug 19, 2026
## Summary - Documents the `anyplot-images` IAM change applied 2026-08-19: `allUsers` swapped from `roles/storage.objectViewer` (bundles `storage.objects.list` — the whole bucket, including the unreviewed `staging/` prefix, was publicly enumerable) to `roles/storage.legacyObjectReader` (get-only). Verified live: every render URL still 200 incl. CORS; both listing APIs now denied (401/AccessDenied). - The runbook entry sits next to the bucket CORS section in `agentic/docs/project-guide.md` with the re-apply command — bucket IAM does not deploy with the repo, so a rebuild would silently drop it. Changelog entry included. ## Plan Docs-only. The commit raced #10492's squash-merge (same recovery as before: cherry-picked onto merged main, stale branch deleted). ## Test plan - [x] Live verification already done: object GET 200 (light/dark/webp/R/Julia), JSON + XML listing denied, `access-control-allow-origin: *` intact, `allUsers` holds only `legacyObjectReader` --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
llms.txtandrobots.txtcarry UTF-8 punctuation (em dashes, arrows) but were served as baretext/plainwithout a charset — a strict client decodes them as Latin-1 mojibake (found by the machine-files verifier in the 2026-08-19 live sweep;llms-full.txt, generated by the API, already declarescharset=utf-8).charset utf-8;in both nginx server blocks (main + python.anyplot.ai). It applies to nginx's defaultcharset_types; proxied responses that already declare a charset keep theirs.llms.txt(timeout comment recomputed: 22 retried checks + two non-retried probes).Plan
One of the three "for later" items from the AI-access verification sweep, done on request. Independent of #10490 and #10491.
Test plan
curl -sI https://anyplot.ai/llms.txt | grep -i content-type→text/plain; charset=utf-8; same forrobots.txt; the extended daily monitor guards it from then on