Repository navigation
to_openeval()/from_openeval(): interop between EvalSet and the EvalPort portable format #6871
Description
Hi maintainers,
ADK's own description is "building, evaluating, and deploying" agents, and google.adk.evaluation is already one of the most complete eval subsystems I've looked at across agent SDKs — so this seemed like a particularly good fit to raise.
I maintain EvalPort (https://github.com/adhabnr-ux/evalport), an open, provider-neutral JSON Schema spec for portable LLM eval test suites and result sets, meant as a common interchange format so eval data isn't locked to one framework. Spec: https://github.com/adhabnr-ux/evalport/blob/main/SPEC.md
Related precedent: openai/openai-python#3619 (openai/openai-python#3619 — open, not yet merged) is adding native to_openeval() / from_openeval() helpers onto that SDK's dataset/response types for the same reason.
Looking at src/google/adk/evaluation/eval_case.py and eval_set.py, an EvalSet.eval_cases is a list[EvalCase], each with a conversation: list[Invocation] where Invocation.user_content / final_response are genai_types.Content and intermediate_data carries tool calls. You already ship get_all_tool_calls() for pulling tool-call trajectories out of that, which makes a converter pretty direct:
# sketch, not a PR
def eval_case_to_openeval_testcase(case: EvalCase) -> dict:
inv = case.conversation[0] # simplest case: single-invocation, static conversation
return {
"id": case.eval_id,
"input": inv.user_content.parts[0].text if inv.user_content.parts else "",
"expected_output": (
inv.final_response.parts[0].text
if inv.final_response and inv.final_response.parts else None
),
"expected_tools": [tc.name for tc in get_all_tool_calls(inv.intermediate_data)],
"graders": ["exact_match"], # or derived from case.rubrics
}which maps onto an EvalPort test case: https://github.com/adhabnr-ux/evalport/blob/main/spec/schemas/testcase.json
Since ADK already reads/writes eval sets as JSON via local_eval_sets_manager.py, this would mostly be a format bridge rather than new infrastructure — useful for anyone moving eval sets between ADK and other tooling. No pressure if this isn't a priority; happy to sketch a fuller PR if there's interest. (Per CONTRIBUTING.md, opening this as an issue first per your "large or complex changes" guidance.)
Activity
adhabnr-ux commentedon Aug 23, 2026
Sounds good — no one else is working on this from my side, so it's all yours. Happy to review once you have a PR up, or answer any questions about the EvalPort
TestCase/ResultSetschemas in the meantime.sanketpatil06 commentedon Aug 24, 2026
Hi @adhabnr-ux , Thanks for the update. The request is currently being reviewed by our team, and we will let you know if anything further is needed. Thank you for your understanding.
adhabnr-ux commentedon Aug 26, 2026
Sounds good, thanks for the update — no rush on my end, and happy to answer any questions about the EvalPort
TestCase/ResultSetschemas whenever it's being reviewed.— Sahi
ALDRIN121 commentedon Aug 28, 2026
Claim check before contributing — per CONTRIBUTING.md I want to ask first, since the 08-24 note says this request is being reviewed by the team. Is an internal implementation planned (in which case I stand down), or would a community PR be welcome?
If welcome, here is the scope I would build, kept to one concern:
EvalSet.to_openeval()/EvalSet.from_openeval()— a format bridge insrc/google/adk/evaluation/, mappingEvalSet→ EvalPort EvalSuite (version, id, test_cases, graders) and back:eval_id→id;conversation[i].user_contentparts →input(string for single-invocation, array of strings for multi-invocation — the schema oneOf already supports both);final_responseparts text →expected_output;get_all_tool_calls(intermediate_data)→expected_tools;rubrics→ suite-levelgraderdefinitions referenced by id, with unmappable rubric kinds preserved viametadatarather than dropped.EvalCase.conversation_scenariocases cannot round-trip losslessly — proposal is to map them toinputwith ametadatamarker, or fail loudly; maintainers can pick.EvalResult→ EvalPort ResultSet interop would be a separate follow-up PR to keep this one focused.@adhabnr-ux — as the spec author, would you be open to reviewing the mapping against your conformance suite once a PR is up?
adhabnr-ux commentedon Aug 31, 2026
@ALDRIN121 Thanks for asking first, and for the detailed scope — that's a solid, well-scoped read of the mapping (the
conversation_scenariolosslessness caveat in particular is exactly the kind of edge case a converter like this needs to call out explicitly).To be clear on where I actually stand, though: I'm the EvalPort spec author who filed this issue, not an adk-python maintainer, so I can't claim or assign this for you — that call belongs to @sanketpatil06 and the adk-python team, since it's their internal-review note from 08-24 that's still open. I'd genuinely welcome a community PR here, but I don't want to imply an authorization I don't have. Best move is to ask them directly in this thread whether it's open for a community contributor to pick up, or whether they're planning to build it internally.
On the EvalPort side, two things that should help regardless of who ends up building it:
spec/schemas/testcase.jsonfor the exactTestCase/EvalSuitefield shapes your mapping above targets)adapters/opik-openeval-adapteras a reference for what a complete adapter package looks like structurally (README, pyproject, src/, tests/): https://github.com/adhabnr-ux/evalport/tree/main/adapters/opik-openeval-adapterAnd yes — happy to review the mapping against the conformance suite once there's a PR up, whoever ends up writing it.
ALDRIN121 commentedon Sep 6, 2026
@sanketpatil06, could you please confirm whether this is open for a community implementation or planned internally? I will hold off until the adk-python team gives an explicit go-ahead.
docs: fix root SPEC.md adoption-claim drift vs spec/SPEC.md
GWeale commentedon Oct 26, 2026
hi @adhabnr-ux, thanks for proposing this, and thanks @ALDRIN121 for checking in before building it. we've decided to keep converters for third-party eval formats out of the core library. since EvalSet is just a pydantic model, a converter works well as a standalone package, for instance alongside evalport's other adapters, and you can propose it for https://adk.dev/integrations/ with a pull request to google/adk-docs. i'm closing this out, though if anything in EvalSet makes that hard to build, please open a new issue and we'll take a look.