fix: Scope custom eval metrics per registry and resolve their function path from the eval config (v1) - #6792
Merged
Merged
Conversation
… path (v1) Ports "fix: scope custom metrics per registry and trust only the config path" to the v1 branch. MetricEvaluatorRegistry declared _registry in the class body, so every instance ever constructed shared one dict and a custom metric registered anywhere in the process resolved from every other registry. It is now built in __init__, and each new instance is seeded with the standard metrics. get_evaluator read the module path for a custom metric off the EvalMetric it was handed, which on a serving deployment is built from the request body and then passed to importlib.import_module. It now uses only a path recorded on the registry at registration time, or one carried on the metric as a private attribute written by get_eval_metrics_from_config. The public custom_function_path field is no longer consulted. Adapted for v1: upstream reworks register_custom_metrics_from_config, which does not exist on this branch. The equivalent registration loop in cli_tools_click.py is updated instead, so `adk eval` passes the config's declared path through when it registers a custom metric. The RubricBasedMultiTurnTrajectory registration is omitted, as that evaluator is not on v1. Breaking change. A custom metric whose function path was not declared in an eval config now raises NotFoundError from get_evaluator instead of importing the path supplied with the metric. Code that reads or writes MetricEvaluatorRegistry._registry on the class, or that relies on one registry instance seeing another's registrations, also breaks.
GWeale
changed the title
xuanyang15
approved these changes
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Ports one eval metric fix to the
v1branch:5d2aca08)MetricEvaluatorRegistryholds_registryper instance, seeded with the twelve standardv1metrics.get_evaluatortakes a custom metric's function path from the registration or from the metric's config-set private attribute, and raisesNotFoundErrorwhen neither has one.adk evalpasses the path declared in the eval config when registering.Breaking change. Custom metrics must be declared in an eval config; a path carried only on an
EvalMetricraisesNotFoundError. Registrations are scoped to a single registry instance rather than theMetricEvaluatorRegistry._registryclass attribute.Re-implemented rather than ported: the config registration loop, which lives inline in
cli/cli_tools_click.pyonv1.