Fix inline rubrics parsing to support score_ranges #146
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Align CLI and rubric generator output with expected_outcome and add score-range rubric proposal + mapping analysis.
…lias - Add description as backward-compatible alias for expected_outcome in all parsers (evaluator-parser, yaml-parser, jsonl-parser) - Add ScoreRange and extended RubricItem types with score_ranges and required_min_score - Implement score_ranges parsing with validation: - Ranges must be integers 0-10 - Ranges must not overlap - Ranges must cover 0-10 inclusive - Each range requires non-empty expected_outcome - Implement score-range evaluation in llm-judge: - Detect score_ranges rubrics automatically - Build specialized prompt for 0-10 integer scoring - Normalize scores to 0-1 (divide by 10) - Apply required_min_score gating (legacy required: true = required_min_score: 10) - Update test to use expected_outcome field name
- Add 4 minimal unit tests for score_ranges validation: - Valid score_ranges parsing with required_min_score - Overlapping ranges validation error - Incomplete coverage validation error - Backward-compatible description alias - Update rubric-evaluator.md skill documentation: - Add Score-Range Rubrics (Analytic Mode) section - Document score_ranges validation rules - Add required_min_score field to table - Add "When to Use Each Mode" guidance - Note description as backward-compatible alias
The inline rubrics parsing in yaml-parser.ts and jsonl-parser.ts was not handling score_ranges - they were being silently ignored and filtered out. - Parse score_ranges and required_min_score in inline rubrics - Fix filter to keep rubrics that have score_ranges (even without expected_outcome) - Use proper TypeScript casts for score_range tuples
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Fixes a bug where inline rubrics with
score_rangeswere silently ignored.Problem
The inline rubrics parsing in
yaml-parser.tsandjsonl-parser.tswas not handling:score_rangesfieldrequired_min_scorefieldscore_rangesbut no top-levelexpected_outcomewere filtered outSolution
score_rangesandrequired_min_scorein inline rubricsscore_ranges(even withoutexpected_outcome)Test plan
summary-multi-criteria-score-ranges-proposednow uses score_ranges evaluation:factual_accuracy: 10/10 (weight 2.0, required_min_score: 8) ✅brevity_and_clarity: 9/10 (weight 1.0)🤖 Generated with Claude Code