Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
28 changes: 28 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,34 @@ All notable changes to this project are documented here. The format follows
[Semantic Versioning](https://semver.org/). Versions are `major.minor.height`, where the height comes from
Nerdbank.GitVersioning.

## [1.1] - Unreleased

Adds rubrics: whole configurations written as JSON, with a built-in Panoramic Data ticket-quality rubric. Everything
is additive; 1.0 code compiles and behaves as before.

### Added

- `AssessmentRubric`: load options from rubric JSON (string or stream) and write them back; code-defined
definitions plug in by name through `NamedAssessmentDefinition`.
- `TieredAssessment`: ordered tiers of condition, score, finding and recommendation over named measures.
- Conditions (`all`, `any`, `not`, `ref`, `measure`, `textMatches`, `fieldMatches`, `fieldEquals`, `fieldPresent`,
`sectionPresent`, `placeholder`, `assessmentScore`) and measures (`patternMatchCount`, `occurrenceCount`,
`wordCount`, `alphanumericCount`, `length`, `itemCount`, `number`, `criteriaHeadingCount`) over `TextSource`s
that can narrow to a headed section.
- `AppliesWhen` and `NotApplicableExplanation` on assessments, `AppliesWhen` and `Weight` on categories; work that
does not apply has the `NotApplicable` outcome and is left out of the score.
- Named conditions, score caps, grade bands and classifications on the options, and `IAssessmentResult.Overall`
(`AssessmentOverall`: score, uncapped score, grade, applied caps, flags and classifications).
- `Recommendation` on assessments and results; `Outcome` on assessment and category results.
- `AssessmentReport`: a flat, serializable summary of a result, with `ToJson()`.
- `BuiltInRubrics.PanoramicDataTicketQuality`, matching Yoda's `IssueQualityCore.ps1` on its test fixtures and on
320 real Jira tickets.
- `NormalizedIssue` with `JiraIssueMapper` (REST v2 and v3 / ADF), `GitHubIssueMapper` and
`ServiceNowRecordMapper`.
- `IssueText`: section bodies in Jira wiki and Markdown, criteria headings, placeholder detection, word and
alphanumeric counts.
- Validation of every rubric rule at construction, including undefined and cyclic condition references.

## [1.0.19] - 2026-10-13

First release.
Expand Down
56 changes: 54 additions & 2 deletions DESIGN.md
Original file line number Diff line number Diff line change
Expand Up @@ -316,12 +316,64 @@ used and was removed: it pulled in a vulnerable NCalc version.

---

## Out of scope for 1.0
## Rubrics (1.1)

1.1 exists so that two consumers can share one definition of ticket quality: Yoda's PowerShell skills
(`IssueQualityCore.ps1`) and the QA Dashboard, which carried a hand-ported C# copy whose weights had drifted.
Rather than port that engine's code, 1.1 adds the primitives it needed and expresses its rules as data.

### Why data, not code

A rubric is a JSON file that `AssessmentRubric.Load` turns into ordinary `AssessmentOptions`. Every rule type is a
record that implements `ICondition` or `IMeasure` and is serialized polymorphically through a `type` discriminator.
This means a non-C# consumer can read and edit the rubric, and a change to it is a reviewable diff. It also means
the rubric is validated as strictly as hand-written options are.

### Tiered assessments

Most of the ported rules have the shape "if this, score 100 with this finding; else if that, score 50; else 0".
`TieredAssessment` is exactly that: named measures, then tiers tried in order, the last unconditional. Measures are
evaluated lazily and once per assessment; named conditions are memoised once per issue in the `EvaluationContext`,
which also caches joined text sources, so twenty assessments that all read the description's "steps" section parse
it once.

### Applicability

The original engine scored rules that did not apply (a rollback plan on a bug) as neither pass nor fail. 1.1 makes
that explicit: an assessment or category with an `AppliesWhen` that does not hold has the `NotApplicable` outcome,
and its weight is removed from both the achieved and possible score, so the score is renormalised over what applies.
A category whose assessments all fail to apply is itself not applicable and leaves the overall average.

### Overall verdict

`AssessmentOverall` is computed after all categories: the weighted average of applicable category scores, then
score caps (the lowest applicable cap wins; caps carry a group so a consumer can report "mandatory" ones), then grade
bands, classifications and the values of every named condition as flags. Caps and classifications can use
`assessmentScore`, which reads finished results, so validation rejects it anywhere else.

### Parity

The built-in rubric was checked against the PowerShell engine rather than reasoned about: 74 of its test fixtures
are committed as golden tests, and 320 real Jira tickets (not committed) matched on score, grade, findings in
order, recommendations, flags and classifications, through both that engine's ticket conversion and
`JiraIssueMapper`. Two features exist only because parity needed them: `IssueText.SectionBody` reproduces the
original's section rules for both wiki and Markdown dialects, and placeholders such as `{$.path~pattern}` reproduce
its finding text.

### Compatibility

Everything is additive. New members on `IAssessmentOptions` and `IAssessmentResult` are default interface members
returning empty values, existing definitions do not see the `EvaluationContext`, and options without rubric rules
behave exactly as in 1.0 (with an `Overall` that has no grade).

---

## Out of scope

- LLM-backed assessments (a `CustomCallbackAssessment` can call one today).
- Cross-ticket assessments, such as duplicate or anomaly detection.
- Serialization of results; results are records that `System.Text.Json` can serialize if needed.
- A full JSONPath implementation.
- A typed Jira client mapper; `NormalizedIssue` can be filled directly from any client's objects.

---

Expand Down
45 changes: 42 additions & 3 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,6 +89,45 @@ Needs, in order:
6. [x] Run the NugetManagement governance assessment and fix what it reports
7. [x] `./Publish.ps1` on a clean, synced `main`; confirm the version on nuget.org

## Phase 9: Consumers (optional)

- [ ] Decide whether a PanoramicData.Skills skill or a consuming dashboard should use the package
## Phase 9: Consumers

Decided 2026-10-13: the package replaces both existing ticket-quality engines, QaDashboard's
`TicketQualityService.cs` (MagicSuite) and Yoda's `_common/IssueQualityCore.ps1` (PanoramicData.Skills), which already
disagree (different weights; QaDashboard lacks business case, rollback plan and the mandatory-element cap).

Decisions:
- Version 1.1: everything below is additive, so existing code keeps compiling.
- The canonical Panoramic Data rubric ships inside this package as JSON, versioned with it.
- Yoda calls the library through a file-based C# skill (`#:package`), emitting the JSON its PowerShell wrappers read.
- Rollout end to end: this package, then PRs in PanoramicData.Skills and MagicSuite.
- Parity spec: Yoda's `Test-IssueQuality*.ps1` tests, plus all engines scored side by side on real MS tickets.

### 9.1 Scoring core
- [ ] Applicability: `AppliesWhen` conditions on assessments and categories; a not-applicable assessment is left out
of the category's score and bounds, and a not-applicable category out of the overall score
- [ ] Category `Weight` in the overall score
- [ ] `TieredAssessment`: the first matching tier's score (partial credit)
- [ ] Mandatory assessments and an overall score cap when one is unmet
- [ ] Overall result: score in [0, 1], 0–100, grade from configurable bands, and the caps applied

### 9.2 Findings and recommendations
- [ ] Assessment outcome: Passed, Partial, Failed, NotApplicable, Error
- [ ] `Recommendation` on assessments and tiers, reported when not passed

### 9.3 Ticket text tools
- [ ] Text from several fields (`AlsoFrom`)
- [ ] Heading sections (Jira wiki and Markdown, label lines), CRLF-safe, with placeholder detection

### 9.4 Rubrics, results and mappers
- [ ] JSON rubric load and save (polymorphic definitions and conditions; named custom definitions)
- [ ] Serialisable `AssessmentReport`
- [ ] Normalized issue v2 (type, priority, status, project, components, labels, fix versions, attachments) and
mappers for Jira REST (v2 wiki and v3 ADF), GitHub and ServiceNow
- [ ] `PanoramicData.IssueAssessment.Jira`: Jira.Api `Issue` mapper
- [ ] Built-in Panoramic Data ticket-quality rubric, ported from Yoda's engine; parity with its tests

### 9.5 Adoption
- [ ] Publish 1.1
- [ ] PanoramicData.Skills: file-based C# skill; IssueQualityCore.ps1 delegates to it; Yoda's tests pass; PR
- [ ] MagicSuite: QaDashboard's `TicketQualityService` uses the package and rubric; PR
- [ ] Side-by-side parity report on real MS tickets
1 change: 1 addition & 0 deletions PanoramicData.IssueAssessment.Tests/GlobalUsings.cs
Original file line number Diff line number Diff line change
Expand Up @@ -5,5 +5,6 @@
global using PanoramicData.IssueAssessment.Configuration;
global using PanoramicData.IssueAssessment.Core;
global using PanoramicData.IssueAssessment.Results;
global using PanoramicData.IssueAssessment.Rules;
global using PanoramicData.IssueAssessment.Utilities;
global using Xunit;
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ public async Task ExecuteAsync_NullInput_ThrowsArgumentNullException()
{
AssessmentCategory category = CreateSeriesCategory("Category", CreateAssessment("Assessment", new StubDefinition()));

Func<Task> act = () => CategoryExecutor.ExecuteAsync(category, null!, null, FailureHandlingStrategy.CollectAll, CancellationToken);
Func<Task> act = () => CategoryExecutor.ExecuteAsync(category, (JsonDocument)null!, null, FailureHandlingStrategy.CollectAll, CancellationToken);

(await act.Should().ThrowAsync<ArgumentNullException>()).WithParameterName("input");
}
Expand Down
Loading
Loading