Implement TokenJuice pipeline and policy primitives - #7
Conversation
|
Important Approval pendingCodeRabbit has no unresolved comments, but it has not reviewed the latest commit. Use the checkbox below to review the latest commit. CodeRabbit will approve the changes if it finds no blocking issues.
Comment |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 92fbb2e198
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
# Conflicts: # Cargo.toml # src/compress.rs # src/compressors/json.rs # src/types.rs
Extend the verify command to flag invalid regex patterns in rules, and refine the Anthropic cache hint logic to only cache the first message when it is a system prompt. These changes improve rule quality checks and reduce unnecessary cache usage for non-system initial messages. Auto-committed-on: dragonfly
Auto-committed-on: dragonfly
…sages as system Add a test to verify that anthropic cache hints correctly avoid marking user messages as system prompts, ensuring proper message type classification. test(rules): add test for discovery normalizing command-only inputs before classification Add a test to confirm that the discovery function normalizes command-only tool execution inputs before classification, preventing false positive fallback outputs. Auto-committed-on: dragonfly
Reformatted the assertion in the anthropic cache hints test to improve readability by wrapping the chained method calls across multiple lines. No functional change was made. Auto-committed-on: dragonfly
When the verify command encounters invalid regexes in rule files, it now prints each error with its path, rule ID, field, index, and the specific error message. This makes debugging configuration issues easier by surfacing the exact location and cause of each regex problem. Auto-committed-on: dragonfly
Replace the negative lookahead patterns in all vendor rule counters with a simpler pattern that matches any line containing at least one non-whitespace character. The previous patterns attempted to exclude header lines but were fragile and caused regex validation failures, as seen in the updated test assertion that now expects no invalid regexes. Auto-committed-on: dragonfly
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 960424c153
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| let host = after_scheme | ||
| .split(['/', '?', '#']) | ||
| .next() | ||
| .unwrap_or_default() |
There was a problem hiding this comment.
When a fetched URL contains userinfo, such as https://alice:password@example.com/page, this parser returns alice:password@example.com as source_host. That value is serialized in WebExtractReduction, defeating the URL-hashing/redaction design and exposing credentials downstream; parse or strip the userinfo@ portion before returning host metadata.
Useful? React with 👍 / 👎.
| let per_page_limit = (page_options.max_combined_inline_chars / input.pages.len()) | ||
| .max(page_options.min_char_limit) | ||
| .min(page_options.char_limit); |
There was a problem hiding this comment.
When a batch exceeds max_combined_inline_chars, this recalculates only each page's content limit, clamps it upward to min_char_limit, and does not reserve space for omission markers or recovery footers. Consequently, even the included two-page case with a combined limit of 80 returns substantially more than 80 inline characters, and the excess grows with page count, so hosts cannot rely on this option to protect the model context window.
Useful? React with 👍 / 👎.
| let (body, recovery_footer, ccr_token) = if ccr_for_call { | ||
| let put = store.put(content); |
There was a problem hiding this comment.
When a compressor body is smaller but the recovery footer makes the final result at least as large as the original, this inserts the original into the bounded FIFO store and only then returns passthrough at the later size check. Near the cache limits, repeated marginal attempts can evict CCR entries referenced by previously emitted recovery tokens even though these new entries are never exposed; derive the token/footer and reject the result before mutating the store, as the previous path did.
Useful? React with 👍 / 👎.
How this change flows2 changed behaviours across 1 relationship. No surrounding behaviour was found (60 graph nodes walked). 97 further behaviours left out to keep the diagram readable. flowchart LR
n0["CompressedOutput<br/>changed"]:::changed
n1["CompressorKind<br/>changed"]:::changed
n0 -->|uses| n1
classDef changed fill:#0d4429,stroke:#238636,color:#e6edf3
classDef impacted fill:#161b22,stroke:#6e7681,color:#c9d1d9
classDef flagged fill:#5a1e02,stroke:#d93f0b,color:#ffffff
classDef blocking fill:#67060c,stroke:#f85149,color:#ffffff
Green: changed behaviour. Grey: surrounding behaviour. Arrows name the call, use, implementation, or test relationship. Orange: has findings. Red: has a finding that blocks the merge. |
tinysweeper
Bot
added
the
priority: p3
1 participant
Summary
API Or Behavior Changes
CompressedOutputexposes separatebodyandrecovery_footerfields while preserving compatibilitytext.CcrStoreimplementations are available for isolated tests and host adapters.Validation
cargo fmt --checkcargo clippy --all-targets -- -D warningscargo testcargo fmt --checkcargo check --libDocumentation
README.md/docs and implementation plans to reflect the implemented TokenJuice/TinyJuice surface without unsupported compression-percentage claims.