Repository navigation
LLM Client - #417
LLM Client#417Enkidu93 wants to merge 6 commits into
Conversation
This comment has been minimized.
This comment has been minimized.
Enkidu93
left a comment
There was a problem hiding this comment.
@Enkidu93 made 10 comments and resolved 9 discussions.
Reviewable status: 0 of 7 files reviewed, 1 unresolved discussion (waiting on benjaminking, ddaspit, and mshannon-sil).
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #417 +/- ##
==========================================
+ Coverage 92.27% 92.28% +0.01%
==========================================
Files 398 402 +4
Lines 25461 25652 +191
==========================================
+ Hits 23494 23673 +179
- Misses 1967 1979 +12 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
This comment has been minimized.
This comment has been minimized.
Enkidu93
left a comment
There was a problem hiding this comment.
@Enkidu93 made 2 comments.
Reviewable status: 0 of 8 files reviewed, 1 unresolved discussion (waiting on benjaminking, ddaspit, and mshannon-sil).
This comment has been minimized.
This comment has been minimized.
|
Verdict: approve. Every finding is now settled. The only change since 934d640 restores The one point that still matters is F2 at Counts: 0 Critical, 0 Important, 0 Low new. F2 (Important) is accepted. Ran: Not verified: live OpenRouter behavior for Surface changed: new public Findings: F1 addressed, F2 accepted (truncation part unverified), F3 addressed, F4 addressed, F5 addressed, F6 addressed, F7 addressed, F8 accepted, unverified, F9 addressed, F10 addressed, F11 withdrawn. Reviewed at 375745b |
mshannon-sil
left a comment
There was a problem hiding this comment.
@mshannon-sil reviewed 8 files and all commit messages, and made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on benjaminking, ddaspit, and Enkidu93).
machine/translation/llm/openrouter_llm_completion_client.py line 13 at r1 (raw file):
self, model: str, temperature: float = 0.0,
Will there be a way to pass in additional parameters or do we need to add them here? Some things I'd want to be able to specify are seed, provider, and thinking level.
benjaminking
left a comment
There was a problem hiding this comment.
A few thoughts about exceptional behavior:
-
It looks like there are quite a few different types of errors that could occur when you call
send(). Some of them may be worth handling here, other probably at the level of the callers. -
Are you thinking that concurrency and retries will exist in a layer above this? Those are important, but fine to leave out for now, if they're going to be implemented in a wrapper.
-
Timeouts are another thing we probably need to support. That probably is best handled in this class.
@benjaminking reviewed all commit messages and made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on ddaspit and Enkidu93).
Enkidu93
left a comment
There was a problem hiding this comment.
Thank you, Ben!
- What do you have in mind here? HTTP response status codes? Malformed response content of some kind? I've only had experience with the API in testing and in reading the docs, so I'm not perhaps familiar enough with the kinds of things that could go wrong. I know you've recently done some work in silnlp to handle some of these sorts of things more gracefully, so maybe you have some particular ideas?
- Concurrency I expect to be handled at the layer above this. As for retries, the openrouter python client seems to have pretty sensible default retry behavior unless I'm misreading things. If you think we need a custom behavior, I can override that obviously.
- The provided client also claims to gracefully handle timeouts. Is there a particular situation you had in mind for us to address expressly here?
Let me know if I'm misunderstanding something.
@Enkidu93 made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on ddaspit).
Enkidu93
left a comment
There was a problem hiding this comment.
@Enkidu93 made 1 comment.
Reviewable status: all files reviewed, 1 unresolved discussion (waiting on ddaspit and mshannon-sil).
machine/translation/llm/openrouter_llm_completion_client.py line 13 at r1 (raw file):
Previously, mshannon-sil wrote…
Will there be a way to pass in additional parameters or do we need to add them here? Some things I'd want to be able to specify are seed, provider, and thinking level.
Good call. Yeah, I wasn't sure what all you need. I'll separate these parameters out into a separate configuration option. If you think of others, let me know. Alternatively, we could allow passing arbitrary options through kwargs if you expect to need that flexibility.
Resolves #398.
This change is