Skip to content

Improve size-limit handling for pinned repository searches #63

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Changyong Gong (chagong) merged 1 commit into from
Nov 10, 2026
Merged

Improve size-limit handling for pinned repository searches #63

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 34 additions & 2 deletions github_app_mcp/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,9 +137,20 @@ access:
| `list_merged_pull_requests` | Pull requests: read |

With a supplied `ref`, `search_repository_content` scans an immutable snapshot:
at most 64 regular files, 256 KiB of eligible content (64 KiB per file), and
optional `path` selects an exact file or directory subtree at that commit.
Use a canonical repository-relative POSIX path without a trailing slash,
at most 240 characters and 16 components; globs and textual prefixes are not
supported. For example, `ref=<full SHA>, path="src/test"` searches only that
subtree, while `path="src/main.ts"` searches one file. Path resolution reads
nonrecursive trees from the pinned root, never the entire recursive root
inventory, and never follows symlinks or submodules. Missing paths fail
explicitly. `path` without `ref` is rejected, not translated into indexed search.

Within the selected scope, at most 64 regular files, 256 KiB of eligible content
(64 KiB per file), and
66 content API requests for a full SHA (one lightweight `/git/commits` object,
one tree, and up to 64 blobs). Branch/tag inputs resolve through fixed
one tree, and up to 64 blobs) are allowed, plus at most one nonrecursive tree read
per path component. Branch/tag inputs resolve through fixed
`/git/ref/heads`, `/git/ref/tags`, and at most eight `/git/tags` reads before the
immutable commit-object read: at most ten additional requests, still inside
the same scan deadline. `HEAD` first reads the repository's default branch.
Expand All @@ -160,6 +171,27 @@ the scan. Anonymous fallback remains read-only and never authorizes writes.
Without `ref`, search uses GitHub's indexed default-branch code search with its
existing single-request 30-second HTTP timeout, not the scan budget.

Internal search-tree metadata has its own 1-MiB HTTP-response budget and is
never returned directly to the model. The generic 100,000-byte result guard
therefore does not reject a larger intermediate tree before matching.
Other HTTP responses remain capped at 128 KiB; model-facing match pages
and their MCP envelopes remain capped at 100,000 bytes. Truncated or oversized
trees and file/content/time budget exhaustion fail explicitly without partial
results or indexed fallback. Narrow `path` after an inventory or scan-limit
error, or use `get_file` at a full SHA if even a shallow tree exceeds its budget.
Do not just reduce `per_page`: that paginates matches, not the inventory scan.

Pinned results return the selected `path` (`null` for the repository root),
`resolved_ref`, and repository-relative, commit-pinned matches sorted by path.
`total_count` counts matching files only within that scope.
`incomplete_results`, `skipped_files`, and `skipped_reasons` describe skipped
content within the chosen scope, not excluded paths elsewhere in the repository.
Even zero matches with `incomplete_results=false` establish coverage only for
that scope. Files over 64 KiB, binary/non-UTF-8 files, unsupported encodings,
symlinks, and submodules are explicitly skipped. Reuse the same `path`,
`resolved_ref`, and `per_page` for subsequent pages; restart at page 1 if changing
`per_page`. Reduce `per_page` only for an oversized match-page result.

### Paged PR and commit reads

Large changes use existing tools and ordinary Copilot tool/model turns, not
Expand Down
167 changes: 124 additions & 43 deletions github_app_mcp/src/issuelens_github_mcp/github.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,8 @@
_MAX_COMMIT_HTTP_RESPONSE_BYTES = 1024 * 1024
_MAX_FILE_BYTES = 64 * 1024
_MAX_QUERY_CHARS = 512
_MAX_SEARCH_TREE_BYTES = 1024 * 1024
_MAX_SEARCH_PATH_DEPTH = 16
_MAX_SEARCH_FILES = 64
_MAX_SEARCH_BYTES = 256 * 1024
_MAX_SEARCH_SECONDS = 60
Expand Down Expand Up @@ -448,7 +450,7 @@ async def list_repository_tree(self, repository: str, ref: str, *, recursive: bo

async def search_repository_content(
self, repository: str, query: str, *, ref: str | None = None,
per_page: int = 30, page: int = 1,
path: str | None = None, per_page: int = 30, page: int = 1,
) -> Any:
repository = self._authorize(repository)
if not isinstance(query, str) or any(
Expand All @@ -459,6 +461,19 @@ async def search_repository_content(
query = query.strip()
if not query or len(query) > _MAX_QUERY_CHARS or _SEARCH_QUALIFIER.search(query):
raise GitHubAppError("query must be plain text and within the bounded limit")
if path is not None:
if ref is None:
raise GitHubAppError("path requires ref for a pinned search", error_type="invalid_input")
path = _repository_path(path)
parts = pathlib.PurePosixPath(path).parts
if path != "/".join(parts) or any(
unicodedata.category(char).startswith("C") for char in path
):
raise GitHubAppError("path must be a canonical repository-relative POSIX path", error_type="invalid_input")
if len(parts) > _MAX_SEARCH_PATH_DEPTH:
raise GitHubAppError(
f"path must have at most {_MAX_SEARCH_PATH_DEPTH} components", error_type="invalid_input",
)
pagination = _pagination(per_page, page)
if ref is None:
return await self._request(
Expand All @@ -478,22 +493,23 @@ async def search_repository_content(
)
stack.push_async_callback(client.aclose)
result = await self._search_repository_content_at_ref(
repository, query, ref, per_page=per_page, page=page, client=client,
repository, query, ref, search_path=path,
per_page=per_page, page=page, client=client,
)
if self._public_only:
await self._require_public_repository(repository)
deadline = scan_timeout.when()
if deadline is not None and asyncio.get_running_loop().time() >= deadline:
raise GitHubAppError(budget_error)
raise GitHubAppError(budget_error, error_type="limit_exceeded")
return result
except TimeoutError:
if not scan_timeout.expired():
raise
raise GitHubAppError(budget_error) from None
raise GitHubAppError(budget_error, error_type="limit_exceeded") from None

async def _search_repository_content_at_ref(
self, repository: str, query: str, ref: str, *,
per_page: int, page: int, client: httpx.AsyncClient,
search_path: str | None, per_page: int, page: int, client: httpx.AsyncClient,
) -> Any:
if self._public_only:
await self._require_public_repository(repository)
Expand All @@ -516,36 +532,46 @@ async def _search_repository_content_at_ref(
tree_sha = _search_oid(commit["tree"].get("sha"))
if _SEARCH_OID_PATTERN.fullmatch(ref) and ref.lower() != resolved_ref:
raise GitHubAppError("GitHub returned a different search commit")
tree = await self._request(
"GET", repository, f"/git/trees/{tree_sha}",
permissions={"contents": "read"}, params={"recursive": "1"},
content_read_auth=content_read_auth,
_client=client,
)
if (
not isinstance(tree, Mapping)
or not isinstance(tree.get("tree"), list)
or type(tree.get("truncated")) is not bool
or _search_oid(tree.get("sha")) != tree_sha
):
raise GitHubAppError("GitHub returned an invalid search tree")
if tree["truncated"]:
raise GitHubAppError("Repository search tree is truncated; scan cannot be complete")
prefix = ""
if search_path is not None:
parts = search_path.split("/")
for index, part in enumerate(parts):
entries = await self._read_search_tree(
repository, tree_sha, recursive=False,
content_read_auth=content_read_auth, client=client,
)
selected = next((entry for entry in entries if entry["path"] == part), None)
if selected is None:
raise GitHubAppError(
"Repository search path was not found at the resolved commit", error_type="not_found",
)
if selected["type"] == "tree":
tree_sha = _search_oid(selected["sha"])
continue
if index != len(parts) - 1:
raise GitHubAppError(
"Repository search path cannot descend through a file, symlink, or submodule",
error_type="invalid_input",
)
entries = [selected]
prefix = search_path.rpartition("/")[0]
break
else:
entries = await self._read_search_tree(
repository, tree_sha, recursive=True,
content_read_auth=content_read_auth, client=client,
)
prefix = search_path
else:
entries = await self._read_search_tree(
repository, tree_sha, recursive=True,
content_read_auth=content_read_auth, client=client,
)

files = []
seen_paths = set()
skipped_reasons: dict[str, int] = {}
for entry in tree["tree"]:
if not isinstance(entry, Mapping):
raise GitHubAppError("GitHub returned an invalid search tree entry")
path = _repository_path(entry.get("path"))
if (
path in seen_paths
or path != "/".join(pathlib.PurePosixPath(path).parts)
or any(unicodedata.category(char).startswith("C") for char in path)
):
raise GitHubAppError("GitHub returned an invalid search tree path")
seen_paths.add(path)
for entry in entries:
path = _repository_path(f"{prefix}/{entry['path']}" if prefix else entry["path"])
blob_sha = _search_oid(entry.get("sha"))
kind, mode = entry.get("type"), entry.get("mode")
if kind == "tree" and mode == "040000":
Expand All @@ -554,16 +580,17 @@ async def _search_repository_content_at_ref(
reason = "symlink" if kind == "blob" else "submodule"
skipped_reasons[reason] = skipped_reasons.get(reason, 0) + 1
continue
if kind != "blob" or mode not in ("100644", "100755"):
raise GitHubAppError("GitHub returned an unsupported search tree mode")
size = entry.get("size")
if type(size) is not int or size < 0:
raise GitHubAppError("GitHub returned an invalid search file size")
files.append((path, blob_sha, size))
files.append((path, blob_sha, entry["size"]))
if len(files) > _MAX_SEARCH_FILES:
raise GitHubAppError(f"Repository search exceeds {_MAX_SEARCH_FILES} regular files")
raise GitHubAppError(
f"Repository search exceeds {_MAX_SEARCH_FILES} regular files; narrow path, not per_page",
error_type="limit_exceeded",
)
if sum(size for _, _, size in files if size <= _MAX_FILE_BYTES) > _MAX_SEARCH_BYTES:
raise GitHubAppError(f"Repository search exceeds {_MAX_SEARCH_BYTES} content bytes")
raise GitHubAppError(
f"Repository search exceeds {_MAX_SEARCH_BYTES} content bytes; narrow path, not per_page",
error_type="limit_exceeded",
)

items = []
for path, blob_sha, size in sorted(files):
Expand Down Expand Up @@ -591,12 +618,62 @@ async def _search_repository_content_at_ref(
result = {
"items": items[start:start + per_page], "total_count": len(items),
"incomplete_results": bool(skipped_reasons), "resolved_ref": resolved_ref,
"path": search_path,
"skipped_files": sum(skipped_reasons.values()), "skipped_reasons": skipped_reasons,
}
if len(json.dumps(result, ensure_ascii=True).encode("utf-8")) > _MAX_RESULT_BYTES:
raise GitHubAppError("GitHub search result is too large; reduce per_page")
raise GitHubAppError("GitHub search result is too large; reduce per_page", error_type="limit_exceeded")
return result

async def _read_search_tree(
self, repository: str, tree_sha: str, *, recursive: bool,
content_read_auth: _ContentReadAuth, client: httpx.AsyncClient,
) -> list[Mapping[str, Any]]:
tree = await self._request(
"GET", repository, f"/git/trees/{tree_sha}",
permissions={"contents": "read"}, params={"recursive": "1"} if recursive else None,
content_read_auth=content_read_auth, _client=client, _search_tree=True,
)
if (
not isinstance(tree, Mapping)
or not isinstance(tree.get("tree"), list)
or type(tree.get("truncated")) is not bool
or _search_oid(tree.get("sha")) != tree_sha
):
raise GitHubAppError("GitHub returned an invalid search tree")
if tree["truncated"]:
raise GitHubAppError(
"Repository search tree is truncated; narrow path or use get_file with a full SHA",
error_type="limit_exceeded",
)
entries = []
seen_paths = set()
for entry in tree["tree"]:
if not isinstance(entry, Mapping):
raise GitHubAppError("GitHub returned an invalid search tree entry")
path = _repository_path(entry.get("path"))
if (
path in seen_paths
or path != "/".join(pathlib.PurePosixPath(path).parts)
or any(unicodedata.category(char).startswith("C") for char in path)
or (not recursive and "/" in path)
):
raise GitHubAppError("GitHub returned an invalid search tree path")
seen_paths.add(path)
_search_oid(entry.get("sha"))
kind, mode = entry.get("type"), entry.get("mode")
if (kind, mode) not in (
("tree", "040000"), ("blob", "120000"), ("commit", "160000"),
("blob", "100644"), ("blob", "100755"),
):
raise GitHubAppError("GitHub returned an unsupported search tree mode")
if kind == "blob" and mode in ("100644", "100755"):
size = entry.get("size")
if type(size) is not int or size < 0:
raise GitHubAppError("GitHub returned an invalid search file size")
entries.append(entry)
return entries

async def _resolve_search_commit(
self, repository: str, ref: str, *,
content_read_auth: _ContentReadAuth, client: httpx.AsyncClient,
Expand Down Expand Up @@ -1221,6 +1298,7 @@ async def _request(
_client: httpx.AsyncClient | None = None,
_commit_detail: CommitDetail | None = None,
_pull_request_file_manifest: bool = False,
_search_tree: bool = False,
_api_version: str = _API_VERSION,
_visibility_probe: bool = False,
) -> Any:
Expand Down Expand Up @@ -1287,6 +1365,7 @@ async def _request(
response.raise_for_status()
content = bytearray()
limit = (
_MAX_SEARCH_TREE_BYTES if _search_tree else
_MAX_COMMIT_HTTP_RESPONSE_BYTES
if _commit_detail is not None or _pull_request_file_manifest
else _MAX_HTTP_RESPONSE_BYTES
Expand All @@ -1295,7 +1374,8 @@ async def _request(
content.extend(chunk)
if len(content) > limit:
raise GitHubAppError(
"GitHub response is too large; narrow the request",
"Repository search tree metadata is too large; narrow path or use get_file with a full SHA"
if _search_tree else "GitHub response is too large; narrow the request",
error_type="limit_exceeded",
outcome="unknown" if write else "not_applied",
)
Expand Down Expand Up @@ -1370,7 +1450,8 @@ async def _request(
}
for item in payload
]
if len(json.dumps(payload, ensure_ascii=True).encode("utf-8")) > _MAX_RESULT_BYTES:
# Search trees are internal metadata; only the constructed match page reaches the model.
if not _search_tree and len(json.dumps(payload, ensure_ascii=True).encode("utf-8")) > _MAX_RESULT_BYTES:
raise GitHubAppError(
"GitHub response is too large; narrow the request", error_type="limit_exceeded",
outcome="unknown" if write else "not_applied",
Expand Down
Loading
Loading