-
Notifications
You must be signed in to change notification settings - Fork 21
Open
Description
Suppose we have an evaluator:
{
"fileName": "evaluator-default.json",
"id": "038038f6-46d0-4aa3-b19c-9b68fb7395b4",
"name": "My LLM Judge Evaluator",
"description": "",
"category": 1,
"type": 5,
"prompt": "ExpectedOutput:\n{{ExpectedOutput}}\n----\nActualOutput:\n{{ActualOutput}}\n",
"model": "same-as-agent",
"targetOutputKey": "*",
"createdAt": "2025-11-07T01:16:33.467Z",
"updatedAt": "2025-11-07T01:16:33.467Z"
}
If we load the evaluator and access it's name, it always shows "LegacyLlmAsAJudgeEvaluator".
data = {...} # load from file
evaluator = EvaluatorFactory().create_evaluator(data, None, "")
assert evaluator.name == "My LLM Judge Evaluator"
Metadata
Metadata
Assignees
Labels
No labels