diff --git a/CHANGELOG.md b/CHANGELOG.md index e375021..60fac5f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,34 @@ All notable changes to this project are documented here. The format follows [Semantic Versioning](https://semver.org/). Versions are `major.minor.height`, where the height comes from Nerdbank.GitVersioning. +## [1.1] - Unreleased + +Adds rubrics: whole configurations written as JSON, with a built-in Panoramic Data ticket-quality rubric. Everything +is additive; 1.0 code compiles and behaves as before. + +### Added + +- `AssessmentRubric`: load options from rubric JSON (string or stream) and write them back; code-defined + definitions plug in by name through `NamedAssessmentDefinition`. +- `TieredAssessment`: ordered tiers of condition, score, finding and recommendation over named measures. +- Conditions (`all`, `any`, `not`, `ref`, `measure`, `textMatches`, `fieldMatches`, `fieldEquals`, `fieldPresent`, + `sectionPresent`, `placeholder`, `assessmentScore`) and measures (`patternMatchCount`, `occurrenceCount`, + `wordCount`, `alphanumericCount`, `length`, `itemCount`, `number`, `criteriaHeadingCount`) over `TextSource`s + that can narrow to a headed section. +- `AppliesWhen` and `NotApplicableExplanation` on assessments, `AppliesWhen` and `Weight` on categories; work that + does not apply has the `NotApplicable` outcome and is left out of the score. +- Named conditions, score caps, grade bands and classifications on the options, and `IAssessmentResult.Overall` + (`AssessmentOverall`: score, uncapped score, grade, applied caps, flags and classifications). +- `Recommendation` on assessments and results; `Outcome` on assessment and category results. +- `AssessmentReport`: a flat, serializable summary of a result, with `ToJson()`. +- `BuiltInRubrics.PanoramicDataTicketQuality`, matching Yoda's `IssueQualityCore.ps1` on its test fixtures and on + 320 real Jira tickets. +- `NormalizedIssue` with `JiraIssueMapper` (REST v2 and v3 / ADF), `GitHubIssueMapper` and + `ServiceNowRecordMapper`. +- `IssueText`: section bodies in Jira wiki and Markdown, criteria headings, placeholder detection, word and + alphanumeric counts. +- Validation of every rubric rule at construction, including undefined and cyclic condition references. + ## [1.0.19] - 2026-10-13 First release. diff --git a/DESIGN.md b/DESIGN.md index 10119a6..252a4a2 100644 --- a/DESIGN.md +++ b/DESIGN.md @@ -316,12 +316,64 @@ used and was removed: it pulled in a vulnerable NCalc version. --- -## Out of scope for 1.0 +## Rubrics (1.1) + +1.1 exists so that two consumers can share one definition of ticket quality: Yoda's PowerShell skills +(`IssueQualityCore.ps1`) and the QA Dashboard, which carried a hand-ported C# copy whose weights had drifted. +Rather than port that engine's code, 1.1 adds the primitives it needed and expresses its rules as data. + +### Why data, not code + +A rubric is a JSON file that `AssessmentRubric.Load` turns into ordinary `AssessmentOptions`. Every rule type is a +record that implements `ICondition` or `IMeasure` and is serialized polymorphically through a `type` discriminator. +This means a non-C# consumer can read and edit the rubric, and a change to it is a reviewable diff. It also means +the rubric is validated as strictly as hand-written options are. + +### Tiered assessments + +Most of the ported rules have the shape "if this, score 100 with this finding; else if that, score 50; else 0". +`TieredAssessment` is exactly that: named measures, then tiers tried in order, the last unconditional. Measures are +evaluated lazily and once per assessment; named conditions are memoised once per issue in the `EvaluationContext`, +which also caches joined text sources, so twenty assessments that all read the description's "steps" section parse +it once. + +### Applicability + +The original engine scored rules that did not apply (a rollback plan on a bug) as neither pass nor fail. 1.1 makes +that explicit: an assessment or category with an `AppliesWhen` that does not hold has the `NotApplicable` outcome, +and its weight is removed from both the achieved and possible score, so the score is renormalised over what applies. +A category whose assessments all fail to apply is itself not applicable and leaves the overall average. + +### Overall verdict + +`AssessmentOverall` is computed after all categories: the weighted average of applicable category scores, then +score caps (the lowest applicable cap wins; caps carry a group so a consumer can report "mandatory" ones), then grade +bands, classifications and the values of every named condition as flags. Caps and classifications can use +`assessmentScore`, which reads finished results, so validation rejects it anywhere else. + +### Parity + +The built-in rubric was checked against the PowerShell engine rather than reasoned about: 74 of its test fixtures +are committed as golden tests, and 320 real Jira tickets (not committed) matched on score, grade, findings in +order, recommendations, flags and classifications, through both that engine's ticket conversion and +`JiraIssueMapper`. Two features exist only because parity needed them: `IssueText.SectionBody` reproduces the +original's section rules for both wiki and Markdown dialects, and placeholders such as `{$.path~pattern}` reproduce +its finding text. + +### Compatibility + +Everything is additive. New members on `IAssessmentOptions` and `IAssessmentResult` are default interface members +returning empty values, existing definitions do not see the `EvaluationContext`, and options without rubric rules +behave exactly as in 1.0 (with an `Overall` that has no grade). + +--- + +## Out of scope - LLM-backed assessments (a `CustomCallbackAssessment` can call one today). - Cross-ticket assessments, such as duplicate or anomaly detection. -- Serialization of results; results are records that `System.Text.Json` can serialize if needed. - A full JSONPath implementation. +- A typed Jira client mapper; `NormalizedIssue` can be filled directly from any client's objects. --- diff --git a/PLAN.md b/PLAN.md index 77d652f..0ff50e2 100644 --- a/PLAN.md +++ b/PLAN.md @@ -89,6 +89,45 @@ Needs, in order: 6. [x] Run the NugetManagement governance assessment and fix what it reports 7. [x] `./Publish.ps1` on a clean, synced `main`; confirm the version on nuget.org -## Phase 9: Consumers (optional) - -- [ ] Decide whether a PanoramicData.Skills skill or a consuming dashboard should use the package +## Phase 9: Consumers + +Decided 2026-10-13: the package replaces both existing ticket-quality engines, QaDashboard's +`TicketQualityService.cs` (MagicSuite) and Yoda's `_common/IssueQualityCore.ps1` (PanoramicData.Skills), which already +disagree (different weights; QaDashboard lacks business case, rollback plan and the mandatory-element cap). + +Decisions: +- Version 1.1: everything below is additive, so existing code keeps compiling. +- The canonical Panoramic Data rubric ships inside this package as JSON, versioned with it. +- Yoda calls the library through a file-based C# skill (`#:package`), emitting the JSON its PowerShell wrappers read. +- Rollout end to end: this package, then PRs in PanoramicData.Skills and MagicSuite. +- Parity spec: Yoda's `Test-IssueQuality*.ps1` tests, plus all engines scored side by side on real MS tickets. + +### 9.1 Scoring core +- [ ] Applicability: `AppliesWhen` conditions on assessments and categories; a not-applicable assessment is left out + of the category's score and bounds, and a not-applicable category out of the overall score +- [ ] Category `Weight` in the overall score +- [ ] `TieredAssessment`: the first matching tier's score (partial credit) +- [ ] Mandatory assessments and an overall score cap when one is unmet +- [ ] Overall result: score in [0, 1], 0–100, grade from configurable bands, and the caps applied + +### 9.2 Findings and recommendations +- [ ] Assessment outcome: Passed, Partial, Failed, NotApplicable, Error +- [ ] `Recommendation` on assessments and tiers, reported when not passed + +### 9.3 Ticket text tools +- [ ] Text from several fields (`AlsoFrom`) +- [ ] Heading sections (Jira wiki and Markdown, label lines), CRLF-safe, with placeholder detection + +### 9.4 Rubrics, results and mappers +- [ ] JSON rubric load and save (polymorphic definitions and conditions; named custom definitions) +- [ ] Serialisable `AssessmentReport` +- [ ] Normalized issue v2 (type, priority, status, project, components, labels, fix versions, attachments) and + mappers for Jira REST (v2 wiki and v3 ADF), GitHub and ServiceNow +- [ ] `PanoramicData.IssueAssessment.Jira`: Jira.Api `Issue` mapper +- [ ] Built-in Panoramic Data ticket-quality rubric, ported from Yoda's engine; parity with its tests + +### 9.5 Adoption +- [ ] Publish 1.1 +- [ ] PanoramicData.Skills: file-based C# skill; IssueQualityCore.ps1 delegates to it; Yoda's tests pass; PR +- [ ] MagicSuite: QaDashboard's `TicketQualityService` uses the package and rubric; PR +- [ ] Side-by-side parity report on real MS tickets diff --git a/PanoramicData.IssueAssessment.Tests/GlobalUsings.cs b/PanoramicData.IssueAssessment.Tests/GlobalUsings.cs index 24e3669..44e0999 100644 --- a/PanoramicData.IssueAssessment.Tests/GlobalUsings.cs +++ b/PanoramicData.IssueAssessment.Tests/GlobalUsings.cs @@ -5,5 +5,6 @@ global using PanoramicData.IssueAssessment.Configuration; global using PanoramicData.IssueAssessment.Core; global using PanoramicData.IssueAssessment.Results; +global using PanoramicData.IssueAssessment.Rules; global using PanoramicData.IssueAssessment.Utilities; global using Xunit; diff --git a/PanoramicData.IssueAssessment.Tests/Integration/CategoryExecutionTests.cs b/PanoramicData.IssueAssessment.Tests/Integration/CategoryExecutionTests.cs index edfe85d..8601467 100644 --- a/PanoramicData.IssueAssessment.Tests/Integration/CategoryExecutionTests.cs +++ b/PanoramicData.IssueAssessment.Tests/Integration/CategoryExecutionTests.cs @@ -23,7 +23,7 @@ public async Task ExecuteAsync_NullInput_ThrowsArgumentNullException() { AssessmentCategory category = CreateSeriesCategory("Category", CreateAssessment("Assessment", new StubDefinition())); - Func act = () => CategoryExecutor.ExecuteAsync(category, null!, null, FailureHandlingStrategy.CollectAll, CancellationToken); + Func act = () => CategoryExecutor.ExecuteAsync(category, (JsonDocument)null!, null, FailureHandlingStrategy.CollectAll, CancellationToken); (await act.Should().ThrowAsync()).WithParameterName("input"); } diff --git a/PanoramicData.IssueAssessment.Tests/Mapping/MapperTests.cs b/PanoramicData.IssueAssessment.Tests/Mapping/MapperTests.cs new file mode 100644 index 0000000..3242b61 --- /dev/null +++ b/PanoramicData.IssueAssessment.Tests/Mapping/MapperTests.cs @@ -0,0 +1,298 @@ +using PanoramicData.IssueAssessment.Mapping; + +namespace PanoramicData.IssueAssessment.Tests.Mapping; + +/// +/// Tests for , , and +/// . +/// +public class MapperTests +{ + private static JsonElement Json(string text) => JsonDocument.Parse(text).RootElement; + + [Fact] + public void Jira_V2Issue_MapsEveryField() + { + NormalizedIssue issue = JiraIssueMapper.FromRestJson( + Json(""" + { + "key": "MS-42", + "fields": { + "summary": "Export fails", + "description": "h2. Steps\r\n# one", + "project": { "key": "MS" }, + "issuetype": { "name": "Bug" }, + "priority": { "name": "Major" }, + "status": { "name": "Open" }, + "assignee": { "displayName": "Alex" }, + "components": [ { "name": "ReportMagic" }, { "id": "no-name" } ], + "labels": [ "ui", 7 ], + "fixVersions": [ { "name": "4.7" } ], + "attachment": [ {}, {} ], + "duedate": "2026-11-20" + } + } + """), + new Uri("https://jira.example.com/")); + + issue.Should().BeEquivalentTo(new NormalizedIssue + { + Key = "MS-42", + Project = "MS", + Title = "Export fails", + Description = "h2. Steps\r\n# one", + IssueType = "Bug", + Priority = "Major", + Status = "Open", + Assignee = "Alex", + Components = ["ReportMagic"], + Labels = ["ui"], + FixVersions = ["4.7"], + AttachmentsCount = 2, + DueDate = "2026-11-20", + Url = "https://jira.example.com/browse/MS-42", + Platform = "Jira", + }); + } + + [Fact] + public void Jira_MinimalIssue_LeavesMissingFieldsUnset() + { + NormalizedIssue issue = JiraIssueMapper.FromRestJson(Json("""{ "fields": { "description": 5, "assignee": null, "components": "x" } }""")); + + issue.Key.Should().BeNull(); + issue.Description.Should().BeNull(); + issue.Assignee.Should().BeNull(); + issue.Components.Should().BeEmpty(); + issue.AttachmentsCount.Should().Be(0); + issue.Url.Should().BeNull("there is no key to link to"); + } + + [Theory] + [InlineData("[]")] + [InlineData("""{ "key": "MS-1" }""")] + [InlineData("""{ "fields": [] }""")] + public void Jira_NotAnIssue_Throws(string json) + { + Action act = () => JiraIssueMapper.FromRestJson(Json(json)); + + act.Should().Throw(); + } + + [Fact] + public void Jira_V3Description_IsFlattenedToWikiText() + { + NormalizedIssue issue = JiraIssueMapper.FromRestJson(Json(""" + { + "fields": { + "description": { + "type": "doc", + "content": [ + { "type": "heading", "attrs": { "level": 3 }, "content": [ { "type": "text", "text": "Steps to reproduce" } ] }, + { "type": "orderedList", "content": [ + { "type": "listItem", "content": [ { "type": "paragraph", "content": [ { "type": "text", "text": "Open" } ] } ] }, + { "type": "listItem", "content": [ + { "type": "paragraph", "content": [ { "type": "text", "text": "Export" } ] }, + { "type": "bulletList", "content": [ { "type": "listItem", "content": [ { "type": "paragraph", "content": [ { "type": "text", "text": "as PDF" } ] } ] } ] } + ] } + ] }, + { "type": "heading", "content": [ { "type": "text", "text": "No level" } ] }, + { "type": "paragraph", "content": [ + { "type": "text", "text": "Ask " }, + { "type": "mention", "attrs": { "text": "@Alex" } }, + { "type": "hardBreak" }, + { "type": "inlineCard", "attrs": { "url": "https://example.invalid/x" } }, + { "type": "emoji", "attrs": { "text": ":)" } }, + { "type": "mention" }, + { "type": "inlineCard" }, + { "type": "strong", "content": [ { "type": "text", "text": " now" } ] } + ] }, + { "type": "codeBlock", "content": [ { "type": "text", "text": "x = 1" } ] }, + { "type": "table", "content": [ { "type": "tableRow", "content": [ { "type": "tableCell", "content": [ { "type": "paragraph", "content": [ { "type": "text", "text": "cell" } ] } ] } ] } ] }, + { "type": "rule" } + ] + } + } + } + """)); + + issue.Description.Should().Be( + "h3. Steps to reproduce\n# Open\n# Export\n#* as PDF\nh2. No level\nAsk @Alex\nhttps://example.invalid/x:)" + " now\n{code}\nx = 1\n{code}\ncell"); + } + + [Fact] + public void Jira_V3Description_ListItemParagraphsAndAttributelessNodes_Flatten() + { + NormalizedIssue issue = JiraIssueMapper.FromRestJson(Json(""" + { + "fields": { + "description": { + "type": "doc", + "content": [ + { "type": "bulletList", "content": [ { "type": "listItem", "content": [ + { "type": "paragraph", "content": [ { "type": "text", "text": "first" } ] }, + { "type": "paragraph", "content": [ { "type": "text" }, { "type": "text", "text": "second" } ] } + ] } ] }, + { "type": "paragraph", "content": [ + { "type": "mention", "attrs": {} }, + { "type": "emoji", "attrs": { "shortName": ":)" } }, + { "type": "inlineCard", "attrs": {} }, + { "type": "text", "text": "end" } + ] } + ] + } + } + } + """)); + + issue.Description.Should().Be("* first\nsecond\nend"); + } + + [Fact] + public void Jira_NoDescription_IsNull() + { + JiraIssueMapper.FromRestJson(Json("""{ "fields": {} }""")).Description.Should().BeNull(); + } + + [Fact] + public void FlattenAdf_NonObjectNode_ReturnsEmpty() + { + JiraIssueMapper.FlattenAdf(Json("[1]")).Should().BeEmpty(); + } + + [Fact] + public void GitHub_Issue_ReadsTypePriorityComponentsAndImagesFromTheBodyAndLabels() + { + NormalizedIssue issue = GitHubIssueMapper.FromRestJson(Json(""" + { + "number": 7, + "title": "Crash", + "body": "![shot](a.png) and https://example.invalid/b.PNG", + "state": "open", + "assignees": [ { "login": "alex" } ], + "labels": [ { "name": "Bug" }, { "name": "priority:HIGH" }, "area:Export", { "name": "component-Export" }, { "id": 1 } ], + "milestone": { "due_on": "2026-12-01T00:00:00Z" }, + "html_url": "https://github.com/o/r/issues/7" + } + """)); + + issue.Key.Should().Be("#7"); + issue.IssueType.Should().Be("Bug"); + issue.Priority.Should().Be("High"); + issue.Components.Should().Equal("Export"); + issue.Labels.Should().Equal("Bug", "priority:HIGH", "area:Export", "component-Export"); + issue.AttachmentsCount.Should().Be(2); + issue.Assignee.Should().Be("alex"); + issue.DueDate.Should().Be("2026-12-01T00:00:00Z"); + issue.SupportsFixVersion.Should().BeFalse(); + issue.Platform.Should().Be("GitHub"); + } + + [Theory] + [InlineData("""[ "enhancement" ]""", "Improvement")] + [InlineData("""[ "user story" ]""", "Improvement")] + [InlineData("""[ "spike" ]""", "Task")] + [InlineData("[]", "Task")] + public void GitHub_IssueType_FollowsLabels(string labels, string expected) + { + GitHubIssueMapper.FromRestJson(Json($$"""{ "labels": {{labels}} }""")).IssueType.Should().Be(expected); + } + + [Fact] + public void GitHub_MinimalIssue_HasDefaults() + { + NormalizedIssue issue = GitHubIssueMapper.FromRestJson(Json("""{ "number": "x", "assignees": [], "milestone": null }""")); + + issue.Key.Should().BeNull(); + issue.Description.Should().BeEmpty(); + issue.Priority.Should().Be("Unknown"); + issue.Assignee.Should().BeNull(); + issue.AttachmentsCount.Should().Be(0); + issue.DueDate.Should().BeNull(); + } + + [Fact] + public void GitHub_NotAnObject_Throws() + { + Action act = () => GitHubIssueMapper.FromRestJson(Json("[]")); + + act.Should().Throw(); + } + + [Fact] + public void ServiceNow_Record_ReadsPlainAndReferenceFields() + { + NormalizedIssue issue = ServiceNowRecordMapper.FromTableJson(Json(""" + { "result": { "number": "INC1", "short_description": "VPN down", "description": " ", "sys_class_name": "incident", + "priority": "2", "state": "1", "assigned_to": { "display_value": "Alex", "link": "x" }, "due_date": "" } } + """)); + + issue.Key.Should().Be("INC1"); + issue.Title.Should().Be("VPN down"); + issue.Description.Should().BeNull("blank values are left unset"); + issue.IssueType.Should().Be("incident"); + issue.Assignee.Should().Be("Alex"); + issue.DueDate.Should().BeNull(); + issue.SupportsComponents.Should().BeFalse(); + issue.Platform.Should().Be("ServiceNow"); + } + + [Fact] + public void ServiceNow_UnwrappedRecordWithOddReference_Maps() + { + NormalizedIssue issue = ServiceNowRecordMapper.FromTableJson(Json("""{ "number": "INC2", "assigned_to": { "value": "abc" }, "state": 3 }""")); + + issue.Key.Should().Be("INC2"); + issue.Assignee.Should().BeNull(); + issue.Status.Should().BeNull(); + } + + [Theory] + [InlineData("[]")] + [InlineData("""{ "result": [] }""")] + public void ServiceNow_NotAnObject_Throws(string json) + { + Action act = () => ServiceNowRecordMapper.FromTableJson(Json(json)); + + act.Should().Throw(); + } + + [Theory] + [InlineData("MS-1234", "MS")] + [InlineData("ops2-1", "ops2")] + [InlineData("#42", null)] + [InlineData("MS-", null)] + [InlineData(null, null)] + public void ProjectFromKey_ReadsTheProjectPrefix(string? key, string? expected) + { + NormalizedIssue.ProjectFromKey(key).Should().Be(expected); + } + + [Fact] + public void ToJsonDocument_WritesSetValuesAndAlwaysTheLists() + { + NormalizedIssue issue = new() + { + Key = "MS-5", + Title = "t", + Labels = ["a"], + SupportsComponents = true, + SupportsFixVersion = false, + AssigneeState = "Assigned", + AcceptanceCriteria = "ac", + ReproductionSteps = "rs", + }; + + using JsonDocument document = issue.ToJsonDocument(); + + document.RootElement.GetRawText().Should().Be( + """{"key":"MS-5","project":"MS","title":"t","assigneeState":"Assigned","components":[],"labels":["a"],"fixVersions":[],"attachmentsCount":0,"supportsFixVersion":false,"supportsComponents":true,"acceptanceCriteria":"ac","reproductionSteps":"rs"}"""); + } + + [Fact] + public void ToJsonObject_ExplicitProject_WinsOverTheKey() + { + new NormalizedIssue { Key = "MS-5", Project = "OPS" }.ToJsonObject()["project"]!.GetValue().Should().Be("OPS"); + new NormalizedIssue().ToJsonObject().ContainsKey("project").Should().BeFalse(); + } +} diff --git a/PanoramicData.IssueAssessment.Tests/PanoramicData.IssueAssessment.Tests.csproj b/PanoramicData.IssueAssessment.Tests/PanoramicData.IssueAssessment.Tests.csproj index 414dc09..b2fefb8 100644 --- a/PanoramicData.IssueAssessment.Tests/PanoramicData.IssueAssessment.Tests.csproj +++ b/PanoramicData.IssueAssessment.Tests/PanoramicData.IssueAssessment.Tests.csproj @@ -14,6 +14,7 @@ + diff --git a/PanoramicData.IssueAssessment.Tests/Results/ReportAndRubricTests.cs b/PanoramicData.IssueAssessment.Tests/Results/ReportAndRubricTests.cs new file mode 100644 index 0000000..223dfb3 --- /dev/null +++ b/PanoramicData.IssueAssessment.Tests/Results/ReportAndRubricTests.cs @@ -0,0 +1,192 @@ +using PanoramicData.IssueAssessment.Serialization; + +namespace PanoramicData.IssueAssessment.Tests.Results; + +/// +/// Tests for , , not-applicable categories and the default +/// members of and . +/// +public class ReportAndRubricTests : TestBase +{ + private const string Rubric = """ + { + "executionStrategy": "series", + "conditions": { "bug": { "type": "fieldEquals", "path": "$.type", "values": [ "Bug" ] } }, + "categories": [ + { + "name": "Bugs", "description": "d", "assessmentExecutionStrategy": "series", + "assessments": [ + { "name": "Steps", "description": "d", "appliesWhen": { "type": "ref", "name": "bug" }, + "definition": { "type": "tiered", "tiers": [ { "score": 0, "finding": "none" } ] } }, + { "name": "Env", "description": "d", "appliesWhen": { "type": "ref", "name": "bug" }, "notApplicableExplanation": "Not a bug: {$.type}.", + "definition": { "type": "tiered", "tiers": [ { "score": 0, "finding": "none" } ] } } + ] + }, + { + "name": "General", "description": "d", "assessmentExecutionStrategy": "parallel", + "assessments": [ + { "name": "Title", "description": "d", "recommendation": "Write a title.", + "definition": { "type": "tiered", "tiers": [ { "when": { "type": "fieldPresent", "path": "$.title" }, "score": 100, "finding": "ok" }, { "score": 0, "finding": "No title." } ] } } + ] + } + ], + "scoreCaps": [ { "name": "No steps", "cap": 0.5, "when": { "type": "assessmentScore", "assessment": "Steps", "lessThan": 50 } } ], + "gradeBands": [ { "name": "GOOD", "minimumScore": 0.5 }, { "name": "POOR", "minimumScore": 0 } ] + } + """; + + [Fact] + public async Task Category_WhoseAssessmentsAllDoNotApply_IsNotApplicable() + { + IssueAssessor assessor = new(AssessmentRubric.Load(Rubric)); + using JsonDocument issue = JsonDocument.Parse("""{ "type": "Task", "title": "t" }"""); + + IAssessmentResult result = await assessor.AssessAsync(issue, CancellationToken); + AssessmentReport report = AssessmentReport.From(result); + + result.Categories[0].Outcome.Should().Be(AssessmentOutcome.NotApplicable); + result.Categories[0].Reason.Should().Be("No assessment in category 'Bugs' applies."); + report.Findings.Should().Equal("Not a bug: Task.", "ok"); + report.Caps.Should().BeEmpty("the capped assessment did not apply"); + report.Score.Should().Be(100); + report.Grade.Should().Be("GOOD"); + report.ToJson().Should().Contain("\"outcome\": \"notApplicable\""); + } + + [Fact] + public async Task Cap_OnAnAssessmentInACategoryThatDoesNotApply_DoesNotApply() + { + AssessmentOptions options = AssessmentRubric.Load(Rubric); + options = options with + { + Categories = [options.Categories[0] with { AppliesWhen = new ConditionReference { Name = "bug" } }, options.Categories[1]], + }; + using JsonDocument issue = JsonDocument.Parse("""{ "type": "Task", "title": "t" }"""); + + IAssessmentResult result = await new IssueAssessor(options).AssessAsync(issue, CancellationToken); + + result.Categories[0].Assessments.Should().BeEmpty(); + result.Overall!.Caps.Should().BeEmpty(); + } + + [Fact] + public async Task Report_RecommendsAndCaps() + { + IssueAssessor assessor = new(AssessmentRubric.Load(Rubric)); + using JsonDocument issue = JsonDocument.Parse("""{ "type": "Bug" }"""); + + AssessmentReport report = AssessmentReport.From(await assessor.AssessAsync(issue, CancellationToken)); + + report.Recommendations.Should().Equal("Write a title."); + report.Caps.Select(cap => cap.Name).Should().Equal("No steps"); + report.Score.Should().Be(0); + } + + [Fact] + public void Report_FromAResultWithoutAnOverall_HasNoScoreAndListsFailures() + { + AssessmentResult failed = new("A", new AssessmentScore(null, null, 0, 1), "boom", AssessmentFailureReason.ThrowingException, new InvalidOperationException("inner")); + PlainResult result = new( + [new CategoryResult("C", new AssessmentScore(null, null, 0, 1), "r", [failed])], + [new AssessmentFailure("C", "A", AssessmentFailureReason.ThrowingException, "boom", new InvalidOperationException("inner")), new AssessmentFailure("C", "B", AssessmentFailureReason.ThrowingException, "b", null)]); + + AssessmentReport report = AssessmentReport.From(result); + + ((IAssessmentResult)result).Overall.Should().BeNull(); + report.Score.Should().BeNull(); + report.Grade.Should().BeNull(); + report.Flags.Should().BeEmpty(); + report.Classifications.Should().BeEmpty(); + report.Failures.Select(failure => failure.Error).Should().Equal("inner", null); + report.Categories[0].Assessments[0].Error.Should().Be("inner"); + } + + [Fact] + public void Overall_WithoutAScore_HasNoScore100() + { + new AssessmentOverall(null, null, null, [], ImmutableDictionary.Empty, ImmutableDictionary.Empty) + .Score100.Should().BeNull(); + } + + [Fact] + public async Task Options_ImplementedWithoutRubricMembers_UseEmptyDefaults() + { + AssessmentOptions loaded = AssessmentRubric.Load(Rubric); + IAssessmentOptions options = new PlainOptions([loaded.Categories[1]]); + using JsonDocument issue = JsonDocument.Parse("""{ "title": "t" }"""); + + IAssessmentResult result = await new IssueAssessor(options).AssessAsync(issue, CancellationToken); + + options.Conditions.Should().BeEmpty(); + options.ScoreCaps.Should().BeEmpty(); + options.GradeBands.Should().BeEmpty(); + options.Classifications.Should().BeEmpty(); + result.Overall!.Grade.Should().BeNull(); + result.Overall.Score100.Should().Be(100); + } + + [Fact] + public void Load_FromAStream_ReadsTheRubric() + { + using MemoryStream stream = new(System.Text.Encoding.UTF8.GetBytes(Rubric)); + + AssessmentRubric.Load(stream).Categories.Should().HaveCount(2); + } + + [Fact] + public void Load_NullJson_Throws() + { + Action act = () => AssessmentRubric.Load("null"); + + act.Should().Throw().WithMessage("The rubric JSON is null."); + } + + [Fact] + public void Load_NamedDefinitions_ReplacesOnlyRegisteredNames() + { + const string json = """ + { + "executionStrategy": "series", + "categories": [ { "name": "C", "description": "d", "assessmentExecutionStrategy": "series", "assessments": [ + { "name": "A", "description": "d", "definition": { "type": "named", "name": "known" } }, + { "name": "B", "description": "d", "definition": { "type": "named", "name": "unknown" } }, + { "name": "C", "description": "d", "definition": { "type": "tiered", "tiers": [ { "score": 0, "finding": "x" } ] } } + ] } ] + } + """; + TieredAssessment known = new() { Tiers = [new Tier { Score = 1, Finding = "k" }] }; + + AssessmentOptions options = AssessmentRubric.Load(json, new Dictionary { ["known"] = known }); + + options.Categories[0].Assessments.Select(assessment => assessment.Definition).Should().SatisfyRespectively( + definition => definition.Should().BeSameAs(known), + definition => definition.Should().BeOfType(), + definition => definition.Should().BeOfType()); + } + + [Fact] + public void ToJson_RoundTrips() + { + AssessmentOptions options = AssessmentRubric.Load(Rubric); + + AssessmentRubric.ToJson(AssessmentRubric.Load(AssessmentRubric.ToJson(options))).Should().Be(AssessmentRubric.ToJson(options)); + } + + private sealed record PlainResult(ImmutableList Categories, ImmutableList Failures) : IAssessmentResult + { + public AssessmentAggregations Aggregations => null!; + + public JsonDocument? VirtualDocument => null; + } + + private sealed record PlainOptions(ImmutableList Categories) : IAssessmentOptions + { + public CategoryExecutionStrategy ExecutionStrategy => CategoryExecutionStrategy.Series; + + public TimeSpan? GlobalTimeout => null; + + public FailureHandlingStrategy FailureHandling => FailureHandlingStrategy.CollectAll; + + public bool CaptureVirtualDocument => false; + } +} diff --git a/PanoramicData.IssueAssessment.Tests/Rubrics/BuiltInRubricParityTests.cs b/PanoramicData.IssueAssessment.Tests/Rubrics/BuiltInRubricParityTests.cs new file mode 100644 index 0000000..c405bf3 --- /dev/null +++ b/PanoramicData.IssueAssessment.Tests/Rubrics/BuiltInRubricParityTests.cs @@ -0,0 +1,92 @@ +using PanoramicData.IssueAssessment.Mapping; +using PanoramicData.IssueAssessment.Rubrics; + +namespace PanoramicData.IssueAssessment.Tests.Rubrics; + +/// +/// The built-in rubric scores every fixture from PanoramicData.Skills' issue-quality tests exactly as Yoda's +/// IssueQualityCore.ps1 did: same score, grade, findings in order, recommendations, flags, automation type and +/// mandatory caps. +/// +/// +/// yoda-fixtures.jsonl was recorded by running Yoda's six Test-IssueQuality*.ps1 scripts against an +/// instrumented copy of its engine on 2026-10-14; each line holds the input ticket in Yoda's shape and Yoda's result. +/// One recorded case was left out because its test temporarily changes two of Yoda's weights. The same comparison +/// passed on 320 real Jira tickets, which are not committed because they are customer data. +/// +public class BuiltInRubricParityTests : TestBase +{ + private static readonly IssueAssessor Assessor = new(BuiltInRubrics.PanoramicDataTicketQuality); + + public static TheoryData Cases => [.. Enumerable.Range(0, File.ReadAllLines(FixturePath).Length)]; + + private static string FixturePath => Path.Combine(AppContext.BaseDirectory, "Rubrics", "yoda-fixtures.jsonl"); + + [Theory] + [MemberData(nameof(Cases))] + public async Task Assess_YodaFixture_MatchesYoda(int index) + { + using JsonDocument record = JsonDocument.Parse(File.ReadAllLines(FixturePath)[index]); + JsonElement issue = record.RootElement.GetProperty("Issue"); + JsonElement yoda = record.RootElement.GetProperty("Result"); + using JsonDocument document = ToNormalized(issue, record.RootElement.GetProperty("Platform").GetString()!).ToJsonDocument(); + + AssessmentReport report = AssessmentReport.From(await Assessor.AssessAsync(document, CancellationToken)); + + List mandatory = [.. report.Caps.Where(cap => cap.Group == "mandatory").Select(cap => cap.Name)]; + List findings = [.. report.Findings]; + if (mandatory.Count > 0) + { + findings.Add($"✗ Mandatory: {string.Join(", ", mandatory)} not met, so EXCELLENT is unreachable (score capped at 79)"); + } + + bool isEpic = report.Flags["isEpic"]; + report.Score.Should().Be((int)yoda.GetProperty("TotalScore").GetDouble()); + report.Grade.Should().Be(yoda.GetProperty("Grade").GetString()); + findings.Should().Equal(Strings(yoda, "Findings")); + report.Recommendations.Should().Equal(Strings(yoda, "Recommendations")); + mandatory.Should().Equal(Strings(yoda, "MandatoryMissing")); + (!isEpic && report.Flags["isBug"]).Should().Be(yoda.GetProperty("IsBug").GetBoolean()); + report.Flags["isInvestigation"].Should().Be(yoda.GetProperty("IsInvestigation").GetBoolean()); + report.Flags["isUiTicket"].Should().Be(yoda.GetProperty("IsUiTicket").GetBoolean()); + report.Classifications["TestAutomationType"].Should().Be(yoda.GetProperty("TestAutomationType").GetString()); + } + + [Fact] + public void BuiltInRubric_Json_RoundTripsThroughTheLoader() + { + AssessmentOptions loaded = Serialization.AssessmentRubric.Load(BuiltInRubrics.PanoramicDataTicketQualityJson); + + loaded.Categories.Select(category => category.Name).Should().Equal("Ticket Quality", "Epic Quality"); + loaded.GradeBands.Select(band => band.Name).Should().Equal("EXCELLENT", "GOOD", "ADEQUATE", "NEEDS IMPROVEMENT", "POOR"); + } + + private static NormalizedIssue ToNormalized(JsonElement issue, string platform) => new() + { + Key = Text(issue, "Key"), + Title = Text(issue, "Summary"), + Description = Text(issue, "Description"), + IssueType = Text(issue, "IssueType"), + Priority = Text(issue, "Priority"), + Status = Text(issue, "Status"), + Components = [.. Strings(issue, "Components")], + Labels = [.. Strings(issue, "Labels")], + FixVersions = [.. Strings(issue, "FixVersions")], + AttachmentsCount = issue.TryGetProperty("AttachmentsCount", out JsonElement count) && count.ValueKind == JsonValueKind.Number ? count.GetInt32() : 0, + SupportsFixVersion = Flag(issue, "SupportsFixVersion"), + SupportsComponents = Flag(issue, "SupportsComponents"), + Platform = platform, + }; + + private static string? Text(JsonElement element, string name) + => element.TryGetProperty(name, out JsonElement value) && value.ValueKind == JsonValueKind.String ? value.GetString() : null; + + private static bool? Flag(JsonElement element, string name) + => element.TryGetProperty(name, out JsonElement value) && value.ValueKind is JsonValueKind.True or JsonValueKind.False ? value.GetBoolean() : null; + + private static List Strings(JsonElement element, string name) + => !element.TryGetProperty(name, out JsonElement value) ? [] + : value.ValueKind == JsonValueKind.Array ? [.. value.EnumerateArray().Select(item => item.GetString()!)] + : value.ValueKind == JsonValueKind.String ? [value.GetString()!] + : []; +} diff --git a/PanoramicData.IssueAssessment.Tests/Rubrics/yoda-fixtures.jsonl b/PanoramicData.IssueAssessment.Tests/Rubrics/yoda-fixtures.jsonl new file mode 100644 index 0000000..d58f45e --- /dev/null +++ b/PanoramicData.IssueAssessment.Tests/Rubrics/yoda-fixtures.jsonl @@ -0,0 +1,74 @@ +{"Platform":"Jira","Issue":{"Key":"MS-25889","Summary":"Export button does nothing with no reports selected","Description":"*Current / Actual Behavior*\r\n\r\nThe export button on the report list silently does nothing when the user has no reports selected.\r\n\r\n*Expected Behavior*\r\n\r\nThe export button should be disabled, or show a toast explaining that at least one report must be\r\nselected, rather than appearing clickable and doing nothing.","IssueType":"Bug","Priority":"Major","Status":"Open","Assignee":"Someone","Url":""},"Result":{"Key":"MS-25889","Summary":"Export button does nothing with no reports selected","IssueType":"Bug","Priority":"Major","Status":"Open","Assignee":"Someone","Url":"","TotalScore":59.0,"Grade":"ADEQUATE","Scores":{"Attachments":0,"RollbackPlan":0,"EnvironmentVersion":100,"CurrentVsExpected":100,"Certifications":100,"StepsToReproduce":0,"Component":0,"BusinessCase":100,"Description":50,"Testability":100,"UIConsistency":100,"Documentation":0},"Findings":["✗ Component: Not identified","⚠ Description: 49 words (brief)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Open)","⚠ Attachments: None (recommended for bugs)","⚠ Documentation: Not referenced","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Add component to issue","Add more context to description","Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","Add screenshot or error log","Update documentation/help text for this change"],"IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Generic","Issue":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","Description":"h2. Background\r\nThe ReportMagic export ignores the instance filter whenever two or more datapoint specs are supplied. Report authors then see every instance, which is wrong and hides the data they asked for. The change is in the export code path and the documentation for it needs updating.\r\n\r\nh2. Proposed change\r\nApply the instance filter once per datapoint spec instead of once per export. Update the ReportMagic documentation page for the export macro to say so.\r\nh2. Acceptance criteria\r\n* The export must return only the filtered instances when two datapoint specs are supplied.\r\n* Verify that a single spec still returns the same rows as before.\r\n* Expected result: the report shows only the requested instances.","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1,"SupportsFixVersion":true,"Url":"https://example.invalid/MS-00000"},"Result":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Url":"https://example.invalid/MS-00000","TotalScore":83.0,"Grade":"EXCELLENT","Scores":{"CurrentVsExpected":100,"EnvironmentVersion":100,"Description":100,"UIConsistency":100,"Testability":100,"RollbackPlan":0,"StepsToReproduce":100,"Attachments":100,"Documentation":70,"Certifications":0,"BusinessCase":0,"Component":100},"Findings":["✓ Component: ReportMagic","✓ Description: 117 words (comprehensive)","✓ Testability: Clear criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Open)","✓ Attachments: 1 item(s)","✓ Documentation: Referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided"],"Recommendations":["Consider adding more documentation updates","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","Description":"h2. Background\r\nThe ReportMagic export ignores the instance filter whenever two or more datapoint specs are supplied. Report authors then see every instance, which is wrong and hides the data they asked for. The change is in the export code path and the documentation for it needs updating.\r\n\r\nh2. Proposed change\r\nApply the instance filter once per datapoint spec instead of once per export. Update the ReportMagic documentation page for the export macro to say so.\r\nh2. Acceptance criteria\r\n* The export must return only the filtered instances when two datapoint specs are supplied.\r\n* Verify that a single spec still returns the same rows as before.\r\n* Expected result: the report shows only the requested instances.\r\nh2. Explicit pass and fail criteria\r\n* Pass: the filtered report matches the instance list.\r\n* Fail: any unrequested instance appears.","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1,"SupportsFixVersion":true,"Url":"https://example.invalid/MS-00000"},"Result":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Url":"https://example.invalid/MS-00000","TotalScore":79.0,"Grade":"GOOD","Scores":{"CurrentVsExpected":100,"EnvironmentVersion":100,"Description":100,"UIConsistency":100,"Testability":100,"RollbackPlan":0,"StepsToReproduce":100,"Attachments":100,"Documentation":70,"Certifications":0,"BusinessCase":0,"Component":100},"Findings":["✓ Component: ReportMagic","✓ Description: 138 words (comprehensive)","✓ Testability: Clear criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Open)","✓ Attachments: 1 item(s)","✓ Documentation: Referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided","✗ Duplication: acceptance criteria stated under 2 headings (Acceptance criteria; Explicit pass and fail criteria), so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Consider adding more documentation updates","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","State the acceptance criteria once, under one heading, and remove the others"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","Description":"h2. Background\r\nThe ReportMagic export ignores the instance filter whenever two or more datapoint specs are supplied. Report authors then see every instance, which is wrong and hides the data they asked for. The change is in the export code path and the documentation for it needs updating.\r\n\r\nh2. Proposed change\r\nApply the instance filter once per datapoint spec instead of once per export. Update the ReportMagic documentation page for the export macro to say so.\r\nh2. Acceptance criteria\r\n* The export must return only the filtered instances when two datapoint specs are supplied.\r\n* Verify that a single spec still returns the same rows as before.\r\n* Expected result: the report shows only the requested instances.\r\nh2. Explicit pass and fail criteria\r\n* Pass: the filtered report matches the instance list.\r\n* Fail: any unrequested instance appears.\r\nh2. Verification evidence to attach on completion\r\n* A screenshot of the filtered report.","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1,"SupportsFixVersion":true,"Url":"https://example.invalid/MS-00000"},"Result":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Url":"https://example.invalid/MS-00000","TotalScore":79.0,"Grade":"GOOD","Scores":{"CurrentVsExpected":100,"EnvironmentVersion":100,"Description":100,"UIConsistency":100,"Testability":100,"RollbackPlan":0,"StepsToReproduce":100,"Attachments":100,"Documentation":70,"Certifications":0,"BusinessCase":0,"Component":100},"Findings":["✓ Component: ReportMagic","✓ Description: 152 words (comprehensive)","✓ Testability: Clear criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Open)","✓ Attachments: 1 item(s)","✓ Documentation: Referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided","✗ Duplication: acceptance criteria stated under 3 headings (Acceptance criteria; Explicit pass and fail criteria; Verification evidence to attach on completion), so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Consider adding more documentation updates","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","State the acceptance criteria once, under one heading, and remove the others"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","Description":"h2. Background\r\nThe ReportMagic export ignores the instance filter whenever two or more datapoint specs are supplied. Report authors then see every instance, which is wrong and hides the data they asked for. The change is in the export code path and the documentation for it needs updating.\r\n\r\nh2. Proposed change\r\nApply the instance filter once per datapoint spec instead of once per export. Update the ReportMagic documentation page for the export macro to say so.\r\nh2. Acceptance criteria\r\n* The export must return only the filtered instances when two datapoint specs are supplied.\r\n* Verify that a single spec still returns the same rows as before.\r\n* Expected result: the report shows only the requested instances.\r\nh2. Verification Evidence\r\n\r\n*Automated coverage of the MUST criteria*\r\n|| Criterion || Test || Location ||\r\n| MUST 1 | Export_TwoDatapointSpecs_ReturnsFilteredInstances | ReportMagic.Test/ExportTests.cs |\r\n\r\n*Reachability - how to make the covered code actually run*\r\nRun any report that uses the export macro with two datapoint specs.\r\n\r\n*Suggested QA route:* B (regression suite)","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1,"SupportsFixVersion":true,"Url":"https://example.invalid/MS-00000"},"Result":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Url":"https://example.invalid/MS-00000","TotalScore":83.0,"Grade":"EXCELLENT","Scores":{"CurrentVsExpected":100,"EnvironmentVersion":100,"Description":100,"UIConsistency":100,"Testability":100,"RollbackPlan":0,"StepsToReproduce":100,"Attachments":100,"Documentation":70,"Certifications":0,"BusinessCase":0,"Component":100},"Findings":["✓ Component: ReportMagic","✓ Description: 169 words (comprehensive)","✓ Testability: Clear criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Open)","✓ Attachments: 1 item(s)","✓ Documentation: Referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided"],"Recommendations":["Consider adding more documentation updates","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","Description":"h2. Background\r\nThe ReportMagic export ignores the instance filter whenever two or more datapoint specs are supplied and the report shows every instance as a result, which is wrong. The documentation needs to be updated as part of the change and the export code path is where it happens.\r\n\r\nh2. Acceptance criteria\r\nThe export is fixed.","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1,"SupportsFixVersion":true,"Url":"https://example.invalid/MS-00000"},"Result":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Url":"https://example.invalid/MS-00000","TotalScore":76.0,"Grade":"GOOD","Scores":{"CurrentVsExpected":100,"EnvironmentVersion":100,"Description":80,"UIConsistency":100,"Testability":60,"RollbackPlan":0,"StepsToReproduce":100,"Attachments":100,"Documentation":70,"Certifications":0,"BusinessCase":0,"Component":100},"Findings":["✓ Component: ReportMagic","✓ Description: 56 words (adequate)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Open)","✓ Attachments: 1 item(s)","✓ Documentation: Referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided"],"Recommendations":["Make the existing acceptance criteria explicit pass/fail checks, rather than adding another section","Consider adding more documentation updates","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","Description":"h2. Background\r\nThe ReportMagic export ignores the instance filter whenever two or more datapoint specs are supplied. Report authors then see every instance, which is wrong and hides the data they asked for. The change is in the export code path and the documentation for it needs updating.\r\n\r\nh2. Proposed change\r\nApply the instance filter once per datapoint spec instead of once per export. Update the ReportMagic documentation page for the export macro to say so.\r\n","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1,"SupportsFixVersion":true,"Url":"https://example.invalid/MS-00000"},"Result":{"Key":"MS-00000","Summary":"ReportMagic export drops the instance filter when two datapoints are supplied","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"Unassigned","Url":"https://example.invalid/MS-00000","TotalScore":64.0,"Grade":"GOOD","Scores":{"CurrentVsExpected":40,"EnvironmentVersion":100,"Description":80,"UIConsistency":100,"Testability":20,"RollbackPlan":0,"StepsToReproduce":100,"Attachments":100,"Documentation":70,"Certifications":0,"BusinessCase":0,"Component":100},"Findings":["✓ Component: ReportMagic","✓ Description: 76 words (adequate)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Open)","✓ Attachments: 1 item(s)","✓ Documentation: Referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided"],"Recommendations":["Define acceptance criteria","Clarify expected outcome","Consider adding more documentation updates","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nCapture the unit configuration with systemctl cat before changing it, to a file held off the\r\nnode; restoring is that file applied back and a daemon reload. The restart itself cannot be\r\nundone, so it is performed one node at a time with the lease-watch test re-run after each, and\r\nthe roll stops on the first node that does not recover within five minutes.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":66.0,"Grade":"GOOD","Scores":{"Description":80,"CurrentVsExpected":40,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","✓ Description: 77 words (adequate)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nTBD","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 14 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nh2. Acceptance criteria\r\n\r\nThe daemon is running.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":50.0,"Grade":"ADEQUATE","Scores":{"Description":50,"CurrentVsExpected":100,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":60},"Findings":["✓ Component: Operations","⚠ Description: 20 words (brief)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Add more context to description","Make the existing acceptance criteria explicit pass/fail checks, rather than adding another section","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nh3. Immediate remediation\r\n\r\nMount the filesystem and restart the container.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":55.0,"Grade":"ADEQUATE","Scores":{"Description":50,"CurrentVsExpected":40,"RollbackPlan":60,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 23 words (brief)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","⚠ Rollback Plan: Present but brief"],"Recommendations":["Add more context to description","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Expand the rollback plan: name what is captured beforehand, and say what is irreversible and what compensates for it"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Risk and mitigations\r\n\r\nReverting is a single-line revert of the ingress mapping followed by a Flux reconcile; the\r\npre-change mapping is captured from the rendered ConfigMap before the change is applied.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":64.0,"Grade":"GOOD","Scores":{"Description":50,"CurrentVsExpected":40,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 42 words (brief)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Add more context to description","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon.\r\n\r\nh2. Rollback plan\r\n\r\nRevert the commit and reconcile.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":53.0,"Grade":"ADEQUATE","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":60,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 14 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","⚠ Rollback Plan: Present but brief"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Expand the rollback plan: name what is captured beforehand, and say what is irreversible and what compensates for it"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nMap the Change Request issue type onto the existing workflow scheme.\r\n\r\nh2. Rollback plan\r\n\r\n*Captured before the change*\r\n\r\nRun get-workflow-scheme --id 10000 > before.json and keep it off the instance.\r\n\r\n*What reverses it*\r\n\r\nremove-workflow-mapping --scheme-id 10000 --issue-type-id 10600, then publish the scheme.\r\n\r\n*What is NOT reversible*\r\n\r\nA resolution already written by the post-function, compensated because it can only set Done.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":66.0,"Grade":"GOOD","Scores":{"Description":80,"CurrentVsExpected":40,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","✓ Description: 62 words (adequate)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Ai","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nMap the Change Request issue type onto the existing workflow scheme.\n\nh2. Rollback plan\n\n*Captured before the change*\n\nRun get-workflow-scheme --id 10000 > before.json and keep it off the instance.\n\n*What reverses it*\n\nremove-workflow-mapping --scheme-id 10000 --issue-type-id 10600, then publish the scheme.\n\n*What is NOT reversible*\n\nA resolution already written by the post-function, compensated because it can only set Done.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":66.0,"Grade":"GOOD","Scores":{"Description":80,"CurrentVsExpected":40,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","✓ Description: 62 words (adequate)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Ai","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nMap the Change Request issue type onto the existing workflow scheme.\r\n\r\nh2. Rollback plan\r\n\r\n*Captured before the change*\r\n\r\nRun get-workflow-scheme --id 10000 > before.json and keep it off the instance.\r\n\r\n*What reverses it*\r\n\r\nremove-workflow-mapping --scheme-id 10000 --issue-type-id 10600, then publish the scheme.\r\n\r\n*What is NOT reversible*\r\n\r\nA resolution already written by the post-function, compensated because it can only set Done.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":66.0,"Grade":"GOOD","Scores":{"Description":80,"CurrentVsExpected":40,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","✓ Description: 62 words (adequate)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Ai","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nEnable DNSSEC on magicsuite.net and reportmagic.net.\r\n\r\nh2. Rollback plan\r\n\r\nh3. What is captured before the change\r\n\r\nFull zone exports taken with Joker.ps1 -Action records -Format zone, attached to the ticket.\r\n\r\nh3. What reverses it\r\n\r\nDisable DNSSEC on the domain's DNS page at Joker.\r\n\r\nh3. What is NOT reversible\r\n\r\nThe DS is cached by validating resolvers for up to the DS TTL; compensated by a two-stage rollout.\r\n\r\nh2. Acceptance criteria\r\n\r\nBoth domains validate from two resolvers.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":75.0,"Grade":"GOOD","Scores":{"Description":80,"CurrentVsExpected":100,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":60},"Findings":["✓ Component: Operations","✓ Description: 77 words (adequate)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Make the existing acceptance criteria explicit pass/fail checks, rather than adding another section","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nEnable DNSSEC on magicsuite.net and reportmagic.net.\n\nh2. Rollback plan\n\nh3. What is captured before the change\n\nFull zone exports taken with Joker.ps1 -Action records -Format zone, attached to the ticket.\n\nh3. What reverses it\n\nDisable DNSSEC on the domain's DNS page at Joker.\n\nh3. What is NOT reversible\n\nThe DS is cached by validating resolvers for up to the DS TTL; compensated by a two-stage rollout.\n\nh2. Acceptance criteria\n\nBoth domains validate from two resolvers.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":75.0,"Grade":"GOOD","Scores":{"Description":80,"CurrentVsExpected":100,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":60},"Findings":["✓ Component: Operations","✓ Description: 77 words (adequate)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Make the existing acceptance criteria explicit pass/fail checks, rather than adding another section","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nEnable DNSSEC on magicsuite.net and reportmagic.net.\r\n\r\nh2. Rollback plan\r\n\r\nh3. What is captured before the change\r\n\r\nFull zone exports taken with Joker.ps1 -Action records -Format zone, attached to the ticket.\r\n\r\nh3. What reverses it\r\n\r\nDisable DNSSEC on the domain's DNS page at Joker.\r\n\r\nh3. What is NOT reversible\r\n\r\nThe DS is cached by validating resolvers for up to the DS TTL; compensated by a two-stage rollout.\r\n\r\nh2. Acceptance criteria\r\n\r\nBoth domains validate from two resolvers.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":75.0,"Grade":"GOOD","Scores":{"Description":80,"CurrentVsExpected":100,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":60},"Findings":["✓ Component: Operations","✓ Description: 77 words (adequate)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Make the existing acceptance criteria explicit pass/fail checks, rather than adding another section","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Rollback plan\n\nh3. Approach\n\nTBD\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 6 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Section too short to describe anything","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Expand the rollback plan to state what reverses the change and what cannot be reversed"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nWork.\n\nh2. Rollback plan\n\nN/A\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 7 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nWork.\n\nh2. Rollback plan\n\nn/a\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 7 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nWork.\n\nh2. Rollback plan\n\nNone\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 7 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nWork.\n\nh2. Rollback plan\n\nTODO\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 7 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nWork.\n\nh2. Rollback plan\n\nTo be confirmed\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 9 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nWork.\n\nh2. Rollback plan\n\n* TBD\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 8 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nWork.\n\nh2. Rollback plan\n\nPending\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 7 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nSomething is broken and should be fixed.\n","IssueType":"Bug","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Bug","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":53.0,"Grade":"ADEQUATE","Scores":{"Description":25,"CurrentVsExpected":0,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":0,"Component":100,"BusinessCase":100,"StepsToReproduce":0,"Certifications":100,"Testability":60},"Findings":["✓ Component: Operations","⚠ Description: 9 words (minimal)","⚠ Testability: Some criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✗ Current vs Expected: Neither documented","○ Fix Version: Not required yet (status: Change Requested)","⚠ Attachments: None (recommended for bugs)","⚠ Documentation: Not referenced","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Add explicit pass/fail criteria","Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","Add current behavior and expected behavior sections","Add screenshot or error log","Update documentation/help text for this change"],"IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nSomething is broken and should be fixed.\n","IssueType":"Task","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Task","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":56.0,"Grade":"ADEQUATE","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":60},"Findings":["✓ Component: Operations","⚠ Description: 9 words (minimal)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided"],"Recommendations":["Expand description with details","Add explicit pass/fail criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nSomething is broken and should be fixed.\n","IssueType":"Improvement","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Improvement","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":56.0,"Grade":"ADEQUATE","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":60},"Findings":["✓ Component: Operations","⚠ Description: 9 words (minimal)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not a UI ticket","✗ Business Case: Not provided"],"Recommendations":["Expand description with details","Add explicit pass/fail criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nSomething is broken and should be fixed.\n","IssueType":"New Feature","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"New Feature","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":56.0,"Grade":"ADEQUATE","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":60},"Findings":["✓ Component: Operations","⚠ Description: 9 words (minimal)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not a UI ticket","✗ Business Case: Not provided"],"Recommendations":["Expand description with details","Add explicit pass/fail criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nSomething is broken and should be fixed.\n","IssueType":"Bug","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Bug","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":53.0,"Grade":"ADEQUATE","Scores":{"Description":25,"CurrentVsExpected":0,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":0,"Component":100,"BusinessCase":100,"StepsToReproduce":0,"Certifications":100,"Testability":60},"Findings":["✓ Component: Operations","⚠ Description: 9 words (minimal)","⚠ Testability: Some criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✗ Current vs Expected: Neither documented","○ Fix Version: Not required yet (status: Change Requested)","⚠ Attachments: None (recommended for bugs)","⚠ Documentation: Not referenced","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Add explicit pass/fail criteria","Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","Add current behavior and expected behavior sections","Add screenshot or error log","Update documentation/help text for this change"],"IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nSomething is broken and should be fixed.\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":43.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":60},"Findings":["✓ Component: Operations","⚠ Description: 9 words (minimal)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: No rollback or mitigation section","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Add explicit pass/fail criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Add a 'Rollback plan' section: what is captured before the change, what reverses it, and what is not reversible"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nCapture the unit configuration with systemctl cat before changing it, to a file held off the\r\nnode; restoring is that file applied back and a daemon reload. The restart itself cannot be\r\nundone, so it is performed one node at a time with the lease-watch test re-run after each, and\r\nthe roll stops on the first node that does not recover within five minutes.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":66.0,"Grade":"GOOD","Scores":{"Description":80,"CurrentVsExpected":40,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","✓ Description: 77 words (adequate)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nTBD","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 14 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nCapture the unit configuration with systemctl cat before changing it, to a file held off the\r\nnode; restoring is that file applied back and a daemon reload. The restart itself cannot be\r\nundone, so it is performed one node at a time with the lease-watch test re-run after each, and\r\nthe roll stops on the first node that does not recover within five minutes.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":66.0,"Grade":"GOOD","Scores":{"Description":80,"CurrentVsExpected":40,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","✓ Description: 77 words (adequate)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon.\r\n\r\nh2. Rollback plan\r\n\r\nRevert the commit and reconcile.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":53.0,"Grade":"ADEQUATE","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":60,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 14 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","⚠ Rollback Plan: Present but brief"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Expand the rollback plan: name what is captured beforehand, and say what is irreversible and what compensates for it"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Risk and mitigations\r\n\r\nReverting is a single-line revert of the ingress mapping followed by a Flux reconcile; the\r\npre-change mapping is captured from the rendered ConfigMap before the change is applied.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":64.0,"Grade":"GOOD","Scores":{"Description":50,"CurrentVsExpected":40,"RollbackPlan":100,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 42 words (brief)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✓ Rollback Plan: Documented"],"Recommendations":["Add more context to description","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nTBD","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 14 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nh2. Acceptance criteria\r\n\r\nThe daemon is running.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":50.0,"Grade":"ADEQUATE","Scores":{"Description":50,"CurrentVsExpected":100,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":60},"Findings":["✓ Component: Operations","⚠ Description: 20 words (brief)","⚠ Testability: Some criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: Heading present but the section is empty or a placeholder","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Add more context to description","Make the existing acceptance criteria explicit pass/fail checks, rather than adding another section","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\r\n\r\nRestart the wedged daemon on the affected node.\r\n\r\nh2. Rollback plan\r\n\r\nh3. Immediate remediation\r\n\r\nMount the filesystem and restart the container.","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":55.0,"Grade":"ADEQUATE","Scores":{"Description":50,"CurrentVsExpected":40,"RollbackPlan":60,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 23 words (brief)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","⚠ Rollback Plan: Present but brief"],"Recommendations":["Add more context to description","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Expand the rollback plan: name what is captured beforehand, and say what is irreversible and what compensates for it"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"OPS-00000","Summary":"Restore service on the affected node","Description":"h2. Summary\n\nWork only.\n","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Components":["Operations"],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"SupportsFixVersion":true,"Url":"https://example.invalid/OPS-00000"},"Result":{"Key":"OPS-00000","Summary":"Restore service on the affected node","IssueType":"Change Request","Priority":"Major","Status":"Change Requested","Assignee":"Unassigned","Url":"https://example.invalid/OPS-00000","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"Description":25,"CurrentVsExpected":40,"RollbackPlan":0,"EnvironmentVersion":100,"UIConsistency":100,"Documentation":0,"Attachments":100,"Component":100,"BusinessCase":0,"StepsToReproduce":100,"Certifications":0,"Testability":20},"Findings":["✓ Component: Operations","⚠ Description: 4 words (minimal)","✗ Testability: No clear criteria","○ Steps: Not required for this type","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Change Requested)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: No rollback or mitigation section","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Add a 'Rollback plan' section: what is captured before the change, what reverses it, and what is not reversible"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 65 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.\n\nh2. Acceptance criteria\n# PDF is non-empty\n# PDF opens in Acrobat\n# Images are rendered","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 81 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section (a numbered list was found, but not under a Steps to Reproduce heading)","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.\n\nPlease investigate why this happens.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 70 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.\n\nh2. Notes\n* Seen twice\n* Not seen on alpha","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 75 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Steps to reproduce\nRun it and look at what comes out.\n\nh2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":40,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 77 words (adequate)","✓ Testability: Clear criteria found","⚠ Steps to Reproduce: Section present but the steps are not numbered","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Rewrite the steps to reproduce as a numbered list","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":0},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":70,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 65 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: Inline images found","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Steps to Reproduce\n# Open a report with images on test2\n# Run ExportPDF\n# Download the file\n\nh2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":98.0,"Grade":"EXCELLENT","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":100,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 84 words (adequate)","✓ Testability: Clear criteria found","✓ Steps to Reproduce: 3 numbered steps","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs"],"Recommendations":null,"IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":[]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.\n\nh2. Acceptance criteria\n# PDF is non-empty\n# PDF opens in Acrobat\n# Images are rendered","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 81 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section (a numbered list was found, but not under a Steps to Reproduce heading)","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.\n\nPlease investigate why this happens.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 70 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Steps to Reproduce\nNot reproducible on demand. Seen three times on test2 in the last week, each time during the nightly schedule run between 02:00 and 03:00, on reports with more than 50 embedded images.\n\nh2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":60,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":100},"Findings":["✓ Component: ReportMagic","✓ Description: 101 words (comprehensive)","✓ Testability: Clear criteria found","⚠ Steps to Reproduce: Stated as not reproducible on demand, with observed conditions","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add numbered steps once a reliable reproduction is found; until then the ticket stays at GOOD","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":["investigation"],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":84.0,"Grade":"EXCELLENT","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":100,"CurrentVsExpected":30,"Testability":50,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 65 words (adequate)","⚠ Investigation Clarity: Missing scope","○ Steps: Not required for investigations","⚠ Investigation Context: Missing context","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs"],"Recommendations":["Define investigation scope and questions to answer","Add background and motivation for why this investigation is needed","Consider adding a timebox for the investigation"],"IsBug":true,"IsInvestigation":true,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":[]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Improvement","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Improvement","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":84.0,"Grade":"EXCELLENT","Scores":{"BusinessCase":0,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":0,"StepsToReproduce":100,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 65 words (adequate)","✓ Testability: Clear criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: No exam question created","○ UI Consistency: Not a UI ticket","✗ Business Case: Not provided"],"Recommendations":["Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":[]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Task","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Task","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":84.0,"Grade":"EXCELLENT","Scores":{"BusinessCase":0,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":0,"StepsToReproduce":100,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 65 words (adequate)","✓ Testability: Clear criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided"],"Recommendations":["Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":[]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"New Feature","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"New Feature","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":84.0,"Grade":"EXCELLENT","Scores":{"BusinessCase":0,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":0,"StepsToReproduce":100,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 65 words (adequate)","✓ Testability: Clear criteria found","○ Steps: Not required for this type","✓ Expected Behavior: Documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: No exam question created","○ UI Consistency: Not a UI ticket","✗ Business Case: Not provided"],"Recommendations":["Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":[]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Defect","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Defect","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 65 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Defect","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 65 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Jira","Issue":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","Description":"h2. Current behaviour\r\nThe ExportPDF macro returns a 0-byte file on the test2 environment when the report contains embedded images. The file cannot be opened.\r\n\r\nh2. Expected behaviour\r\nThe macro should produce a valid, non-empty PDF file. Verify that the file opens. Acceptance criteria: file size greater than zero.\r\n\r\nh2. Environment\r\ntest2, build 4.3.120. See the user guide and the help topic for ExportPDF documentation.","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Components":["ReportMagic"],"Labels":[],"FixVersions":[],"AttachmentsCount":1},"Result":{"Key":"MS-1","Summary":"ReportMagic: The ExportPDF macro should produce a valid PDF file","IssueType":"Bug","Priority":"Major","Status":"Ready for Progress","Assignee":"Unassigned","Url":"","TotalScore":79.0,"Grade":"GOOD","Scores":{"BusinessCase":100,"Component":100,"EnvironmentVersion":100,"RollbackPlan":0,"Documentation":100,"UIConsistency":100,"Certifications":100,"StepsToReproduce":0,"CurrentVsExpected":100,"Testability":100,"Attachments":100,"Description":80},"Findings":["✓ Component: ReportMagic","✓ Description: 65 words (adequate)","✓ Testability: Clear criteria found","✗ Steps to Reproduce: No steps-to-reproduce section","✓ Current vs Expected: Both documented","○ Fix Version: Not required yet (status: Ready for Progress)","✓ Attachments: 1 item(s)","✓ Documentation: Multiple references found","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs","✗ Mandatory: Steps to reproduce not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":"Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')","IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Jira","MandatoryMissing":["Steps to reproduce"]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Bug","Priority":"Major","Status":"Open","Assignee":"a","Components":[],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Bug","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":46.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"BusinessCase":100,"RollbackPlan":0,"CurrentVsExpected":0,"StepsToReproduce":100,"Component":0,"Description":25,"Documentation":0,"Certifications":100,"EnvironmentVersion":100,"Attachments":0,"Testability":20,"UIConsistency":100},"Findings":["✗ Component: Not identified","⚠ Description: 8 words (minimal)","✗ Testability: No clear criteria","✓ Steps to Reproduce: 2 numbered steps","✗ Current vs Expected: Neither documented","○ Fix Version: Not required yet (status: Open)","⚠ Attachments: None (recommended for bugs)","⚠ Documentation: Not referenced","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs"],"Recommendations":["Add component to issue","Expand description with details","Define acceptance criteria","Add current behavior and expected behavior sections","Add screenshot or error log","Update documentation/help text for this change"],"IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Bug","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui"],"Labels":["ui"],"FixVersions":["ui"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Bug","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":56.0,"Grade":"ADEQUATE","Scores":{"BusinessCase":100,"RollbackPlan":0,"CurrentVsExpected":0,"StepsToReproduce":100,"Component":100,"Description":25,"Documentation":0,"Certifications":100,"EnvironmentVersion":70,"Attachments":0,"Testability":20,"UIConsistency":100},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Testability: No clear criteria","✓ Steps to Reproduce: 2 numbered steps","✗ Current vs Expected: Neither documented","✓ Fix Version: ui","⚠ Attachments: None (recommended for bugs)","⚠ Documentation: Not referenced","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs"],"Recommendations":["Expand description with details","Define acceptance criteria","Add current behavior and expected behavior sections","Add screenshot or error log","Update documentation/help text for this change"],"IsBug":true,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Bug","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui","investigation"],"Labels":["ui","investigation"],"FixVersions":["ui","investigation"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Bug","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":58.0,"Grade":"ADEQUATE","Scores":{"BusinessCase":100,"RollbackPlan":0,"CurrentVsExpected":30,"StepsToReproduce":100,"Component":100,"Description":25,"Documentation":0,"Certifications":100,"EnvironmentVersion":70,"Attachments":0,"Testability":20,"UIConsistency":100},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Investigation Clarity: No scope or deliverables","○ Steps: Not required for investigations","⚠ Investigation Context: Missing context","✓ Fix Version: ui","⚠ Attachments: None (recommended for bugs)","⚠ Documentation: Not referenced","○ Certifications: Not applicable for bugs","○ UI Consistency: Not applicable for Bug","○ Business Case: Not required for bugs"],"Recommendations":["Expand description with details","Add investigation scope, questions to answer, and expected deliverables","Add background and motivation for why this investigation is needed","Consider adding a timebox for the investigation","Add screenshot or error log","Update documentation/help text for this change"],"IsBug":true,"IsInvestigation":true,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Change Request","Priority":"Major","Status":"Open","Assignee":"a","Components":[],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Change Request","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":30.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"BusinessCase":0,"RollbackPlan":0,"CurrentVsExpected":40,"StepsToReproduce":100,"Component":0,"Description":25,"Documentation":0,"Certifications":0,"EnvironmentVersion":100,"Attachments":100,"Testability":20,"UIConsistency":100},"Findings":["✗ Component: Not identified","⚠ Description: 8 words (minimal)","✗ Testability: No clear criteria","✓ Steps: Workflow documented","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Open)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: No rollback or mitigation section","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Add component to issue","Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Add a 'Rollback plan' section: what is captured before the change, what reverses it, and what is not reversible"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Change Request","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui"],"Labels":["ui"],"FixVersions":["ui"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Change Request","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":38.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"BusinessCase":0,"RollbackPlan":0,"CurrentVsExpected":40,"StepsToReproduce":100,"Component":100,"Description":25,"Documentation":0,"Certifications":0,"EnvironmentVersion":70,"Attachments":100,"Testability":20,"UIConsistency":100},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Testability: No clear criteria","✓ Steps: Workflow documented","⚠ Expected Behavior: Not explicit","✓ Fix Version: ui","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","✗ Business Case: Not provided","✗ Rollback Plan: No rollback or mitigation section","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work","Add a 'Rollback plan' section: what is captured before the change, what reverses it, and what is not reversible"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Change Request","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui","investigation"],"Labels":["ui","investigation"],"FixVersions":["ui","investigation"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Change Request","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":45.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"BusinessCase":100,"RollbackPlan":0,"CurrentVsExpected":30,"StepsToReproduce":100,"Component":100,"Description":25,"Documentation":0,"Certifications":0,"EnvironmentVersion":70,"Attachments":100,"Testability":20,"UIConsistency":100},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Investigation Clarity: No scope or deliverables","○ Steps: Not required for investigations","⚠ Investigation Context: Missing context","✓ Fix Version: ui","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Change Request","○ Business Case: Not required for investigations","✗ Rollback Plan: No rollback or mitigation section","✗ Mandatory: Rollback plan not met, so EXCELLENT is unreachable (score capped at 79)"],"Recommendations":["Expand description with details","Add investigation scope, questions to answer, and expected deliverables","Add background and motivation for why this investigation is needed","Consider adding a timebox for the investigation","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a 'Rollback plan' section: what is captured before the change, what reverses it, and what is not reversible"],"IsBug":false,"IsInvestigation":true,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":["Rollback plan"]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Improvement","Priority":"Major","Status":"Open","Assignee":"a","Components":[],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Improvement","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"BusinessCase":0,"RollbackPlan":0,"CurrentVsExpected":40,"StepsToReproduce":100,"Component":0,"Description":25,"Documentation":0,"Certifications":0,"EnvironmentVersion":100,"Attachments":100,"Testability":20,"UIConsistency":100},"Findings":["✗ Component: Not identified","⚠ Description: 8 words (minimal)","✗ Testability: No clear criteria","✓ Steps: Workflow documented","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Open)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not a UI ticket","✗ Business Case: Not provided"],"Recommendations":["Add component to issue","Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Improvement","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui"],"Labels":["ui"],"FixVersions":["ui"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Improvement","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":38.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"BusinessCase":0,"RollbackPlan":0,"CurrentVsExpected":40,"StepsToReproduce":100,"Component":100,"Description":25,"Documentation":0,"Certifications":0,"EnvironmentVersion":70,"Attachments":100,"Testability":20,"UIConsistency":0},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Testability: No clear criteria","✓ Steps: Workflow documented","⚠ Expected Behavior: Not explicit","✓ Fix Version: ui","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","✗ UI Consistency: No reference to look & feel or similar features","✗ Business Case: Not provided"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add UI consistency details or explicitly state no similar features exist","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":true,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Improvement","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui","investigation"],"Labels":["ui","investigation"],"FixVersions":["ui","investigation"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Improvement","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":58.0,"Grade":"ADEQUATE","Scores":{"BusinessCase":100,"RollbackPlan":0,"CurrentVsExpected":30,"StepsToReproduce":100,"Component":100,"Description":25,"Documentation":0,"Certifications":0,"EnvironmentVersion":70,"Attachments":100,"Testability":20,"UIConsistency":100},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Investigation Clarity: No scope or deliverables","○ Steps: Not required for investigations","⚠ Investigation Context: Missing context","✓ Fix Version: ui","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Improvement","○ Business Case: Not required for investigations"],"Recommendations":["Expand description with details","Add investigation scope, questions to answer, and expected deliverables","Add background and motivation for why this investigation is needed","Consider adding a timebox for the investigation","Update documentation/help text for this change","Consider creating a certification exam question for this new feature"],"IsBug":false,"IsInvestigation":true,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"a","Components":[],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":39.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"BusinessCase":0,"RollbackPlan":0,"CurrentVsExpected":40,"StepsToReproduce":100,"Component":0,"Description":25,"Documentation":0,"Certifications":0,"EnvironmentVersion":100,"Attachments":100,"Testability":20,"UIConsistency":100},"Findings":["✗ Component: Not identified","⚠ Description: 8 words (minimal)","✗ Testability: No clear criteria","✓ Steps: Workflow documented","⚠ Expected Behavior: Not explicit","○ Fix Version: Not required yet (status: Open)","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided"],"Recommendations":["Add component to issue","Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui"],"Labels":["ui"],"FixVersions":["ui"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":49.0,"Grade":"NEEDS IMPROVEMENT","Scores":{"BusinessCase":0,"RollbackPlan":0,"CurrentVsExpected":40,"StepsToReproduce":100,"Component":100,"Description":25,"Documentation":0,"Certifications":0,"EnvironmentVersion":70,"Attachments":100,"Testability":20,"UIConsistency":100},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Testability: No clear criteria","✓ Steps: Workflow documented","⚠ Expected Behavior: Not explicit","✓ Fix Version: ui","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","✗ Business Case: Not provided"],"Recommendations":["Expand description with details","Define acceptance criteria","Clarify expected outcome","Update documentation/help text for this change","Consider creating a certification exam question for this new feature","Add a business case explaining the value, impact, or cost of not doing this work"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui","investigation"],"Labels":["ui","investigation"],"FixVersions":["ui","investigation"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Task","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":58.0,"Grade":"ADEQUATE","Scores":{"BusinessCase":100,"RollbackPlan":0,"CurrentVsExpected":30,"StepsToReproduce":100,"Component":100,"Description":25,"Documentation":0,"Certifications":0,"EnvironmentVersion":70,"Attachments":100,"Testability":20,"UIConsistency":100},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Investigation Clarity: No scope or deliverables","○ Steps: Not required for investigations","⚠ Investigation Context: Missing context","✓ Fix Version: ui","○ Attachments: None (optional for non-bugs)","⚠ Documentation: Not referenced","○ Certifications: No exam question created","○ UI Consistency: Not applicable for Task","○ Business Case: Not required for investigations"],"Recommendations":["Expand description with details","Add investigation scope, questions to answer, and expected deliverables","Add background and motivation for why this investigation is needed","Consider adding a timebox for the investigation","Update documentation/help text for this change","Consider creating a certification exam question for this new feature"],"IsBug":false,"IsInvestigation":true,"IsUiTicket":false,"TestAutomationType":"Human","Platform":"Generic","MandatoryMissing":[]}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Epic","Priority":"Major","Status":"Open","Assignee":"a","Components":[],"Labels":[],"FixVersions":[],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Epic","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":11.0,"Grade":"POOR","Scores":{"Attachments":30,"Component":0,"Description":25,"Architecture":0,"ScopeDefinition":0,"BusinessCase":0,"Decomposition":0,"AcceptanceCriteria":0,"FixVersion":100},"Findings":["✗ Component: Not set","⚠ Description: 8 words (minimal)","✗ Architecture: No technical design","✗ Decomposition: No work breakdown","✗ Scope: Not defined","✗ Acceptance Criteria: Not defined","○ Fix Version: Not required yet (status: Open)","⚠ Attachments: None (diagrams/mockups recommended for epics)","✗ Business Case: Not provided"],"Recommendations":["Set the component field","Add comprehensive problem statement and solution overview","Add architecture/technical design section","Decompose into child tickets or define implementation phases","Add scope boundaries (in scope, out of scope, goals)","Add high-level acceptance/success criteria and definition of done","Consider adding architecture diagrams or mockups","Add business case explaining why this epic is valuable"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic"}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Epic","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui"],"Labels":["ui"],"FixVersions":["ui"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Epic","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":21.0,"Grade":"POOR","Scores":{"Attachments":30,"Component":100,"Description":25,"Architecture":0,"ScopeDefinition":0,"BusinessCase":0,"Decomposition":0,"AcceptanceCriteria":0,"FixVersion":100},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Architecture: No technical design","✗ Decomposition: No work breakdown","✗ Scope: Not defined","✗ Acceptance Criteria: Not defined","✓ Fix Version: ui","⚠ Attachments: None (diagrams/mockups recommended for epics)","✗ Business Case: Not provided"],"Recommendations":["Add comprehensive problem statement and solution overview","Add architecture/technical design section","Decompose into child tickets or define implementation phases","Add scope boundaries (in scope, out of scope, goals)","Add high-level acceptance/success criteria and definition of done","Consider adding architecture diagrams or mockups","Add business case explaining why this epic is valuable"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic"}} +{"Platform":"Generic","Issue":{"Key":"MS-1","Summary":"x should y","Description":"h2. Steps to Reproduce\n# a\n# b","IssueType":"Epic","Priority":"Major","Status":"Open","Assignee":"a","Components":["ui","investigation"],"Labels":["ui","investigation"],"FixVersions":["ui","investigation"],"AttachmentsCount":0,"Url":""},"Result":{"Key":"MS-1","Summary":"x should y","IssueType":"Epic","Priority":"Major","Status":"Open","Assignee":"a","Url":"","TotalScore":21.0,"Grade":"POOR","Scores":{"Attachments":30,"Component":100,"Description":25,"Architecture":0,"ScopeDefinition":0,"BusinessCase":0,"Decomposition":0,"AcceptanceCriteria":0,"FixVersion":100},"Findings":["✓ Component: ui","⚠ Description: 8 words (minimal)","✗ Architecture: No technical design","✗ Decomposition: No work breakdown","✗ Scope: Not defined","✗ Acceptance Criteria: Not defined","✓ Fix Version: ui","⚠ Attachments: None (diagrams/mockups recommended for epics)","✗ Business Case: Not provided"],"Recommendations":["Add comprehensive problem statement and solution overview","Add architecture/technical design section","Decompose into child tickets or define implementation phases","Add scope boundaries (in scope, out of scope, goals)","Add high-level acceptance/success criteria and definition of done","Consider adding architecture diagrams or mockups","Add business case explaining why this epic is valuable"],"IsBug":false,"IsInvestigation":false,"IsUiTicket":false,"TestAutomationType":"None","Platform":"Generic"}} diff --git a/PanoramicData.IssueAssessment.Tests/Rules/RuleEngineTests.cs b/PanoramicData.IssueAssessment.Tests/Rules/RuleEngineTests.cs new file mode 100644 index 0000000..5b80916 --- /dev/null +++ b/PanoramicData.IssueAssessment.Tests/Rules/RuleEngineTests.cs @@ -0,0 +1,141 @@ +using PanoramicData.IssueAssessment.Serialization; + +namespace PanoramicData.IssueAssessment.Tests.Rules; + +/// +/// Unit tests for , the built-in conditions and measures, and tiered placeholders. +/// +public class RuleEngineTests : TestBase +{ + private static EvaluationContext Context(string json) => new(JsonDocument.Parse(json)); + + private static TextSource Source(string path, string? section = null) => new() { Paths = [path], Section = section }; + + [Fact] + public void Condition_UnknownName_Throws() + { + Action act = () => Context("{}").Condition("nope"); + + act.Should().Throw().WithMessage("*'nope'*"); + } + + [Fact] + public void Measure_UnknownName_Throws() + { + Action act = () => Context("{}").Measure("nope"); + + act.Should().Throw().WithMessage("*'nope'*"); + } + + [Theory] + [InlineData("$.n", new[] { "12.5" })] + [InlineData("$.b", new[] { "true" })] + [InlineData("$.s", new[] { "x" })] + [InlineData("$.a", new[] { "x", "3" })] + [InlineData("$.z", new string[0])] + [InlineData("$.missing", new string[0])] + public void Strings_ReadsScalarsAndArraysAsText(string path, string[] expected) + { + Context("""{ "n": 12.5, "b": true, "s": "x", "a": [ "x", null, 3 ], "z": null }""").Strings(path).Should().Equal(expected); + } + + [Fact] + public void TextMatches_AbsentSection_IsFalse() + { + new TextMatchesCondition { Source = Source("$.d", "Steps"), Patterns = ["."] }.Evaluate(Context("""{ "d": "no heading" }""")) + .Should().BeFalse(); + } + + [Theory] + [InlineData(false, true)] + [InlineData(true, false)] + public void FieldEquals_HonoursCaseSensitivity(bool caseSensitive, bool expected) + { + new FieldEqualsCondition { Path = "$.t", Values = ["BUG"], CaseSensitive = caseSensitive }.Evaluate(Context("""{ "t": "bug" }""")) + .Should().Be(expected); + } + + [Fact] + public void AssessmentScore_OutsideCapsAndClassifications_Throws() + { + Action act = () => new AssessmentScoreCondition { Assessment = "A" }.Evaluate(Context("{}")); + + act.Should().Throw(); + } + + [Fact] + public void Measures_OnAnAbsentSection_MeasureZero() + { + EvaluationContext context = Context("""{ "d": "no heading" }"""); + TextSource absent = Source("$.d", "Steps"); + + new PatternMatchCountMeasure { Source = absent, Patterns = ["^$"] }.Evaluate(context).Should().Be(1, "the absent section reads as empty text"); + new OccurrenceCountMeasure { Source = absent, Pattern = "x" }.Evaluate(context).Should().Be(0); + new LengthMeasure { Source = absent }.Evaluate(context).Should().Be(0); + new LengthMeasure { Source = Source("$.d") }.Evaluate(context).Should().Be(10); + } + + [Theory] + [InlineData(false, 2)] + [InlineData(true, 1)] + public void OccurrenceCount_HonoursCaseSensitivity(bool caseSensitive, double expected) + { + new OccurrenceCountMeasure { Source = Source("$.d"), Pattern = "ab", CaseSensitive = caseSensitive } + .Evaluate(Context("""{ "d": "ab AB" }""")).Should().Be(expected); + } + + [Theory] + [InlineData("""{ "n": "7" }""", 7)] + [InlineData("""{ "n": "seven" }""", 0)] + [InlineData("""{ "n": "1e999" }""", 0)] + [InlineData("""{ "n": [ 1, 2 ] }""", 0)] + public void Number_ReadsOneFiniteNumber(string json, double expected) + { + new NumberMeasure { Path = "$.n" }.Evaluate(Context(json)).Should().Be(expected); + } + + [Fact] + public async Task Tiered_WithoutAContext_EvaluatesAgainstTheIssue() + { + TieredAssessment tiered = new() + { + Measures = ImmutableDictionary.Empty.Add("words", new WordCountMeasure { Source = Source("$.d") }), + Tiers = + [ + new Tier { When = new MeasureCondition { Measure = "words", AtLeast = 3 }, Score = 100, Finding = "{words} words; {$.k~^MS-} {$.k~^OPS-}." }, + new Tier { Score = 0, Finding = "short" }, + ], + }; + using JsonDocument issue = JsonDocument.Parse("""{ "d": "one two three", "k": "MS-1" }"""); + + AssessmentExecutionResult result = await tiered.ExecuteAsync(issue, null, CancellationToken); + + result.RawScore.Should().Be(100); + result.ExplanationText.Should().Be("3 words; MS-1 ."); + } + + [Fact] + public async Task Tiered_WithoutAnIssue_Throws() + { + Func act = () => new TieredAssessment { Tiers = [new Tier { Score = 0, Finding = "x" }] }.ExecuteAsync((JsonDocument)null!, null, CancellationToken); + + await act.Should().ThrowAsync(); + } + + [Fact] + public async Task NamedDefinition_Executed_Throws() + { + using JsonDocument issue = JsonDocument.Parse("{}"); + + Func act = () => new NamedAssessmentDefinition { Name = "custom" }.ExecuteAsync(issue, null, CancellationToken); + + await act.Should().ThrowAsync().WithMessage("*'custom'*"); + } + + [Fact] + public void RuleRegex_CaseSensitive_DoesNotIgnoreCase() + { + new TextMatchesCondition { Source = Source("$.d"), Patterns = ["Bug"], CaseSensitive = true }.Evaluate(Context("""{ "d": "bug" }""")) + .Should().BeFalse(); + } +} diff --git a/PanoramicData.IssueAssessment.Tests/Scoring/ScoreCalculationTests.cs b/PanoramicData.IssueAssessment.Tests/Scoring/ScoreCalculationTests.cs index ddd16b7..4f144ed 100644 --- a/PanoramicData.IssueAssessment.Tests/Scoring/ScoreCalculationTests.cs +++ b/PanoramicData.IssueAssessment.Tests/Scoring/ScoreCalculationTests.cs @@ -63,7 +63,7 @@ public void CalculateCategoryRawRange_NegativeWeight_SwapsBounds() [Fact] public void CalculateCategoryRawRange_NullCategory_Throws() { - Action act = () => ScoreCalculator.CalculateCategoryRawRange(null!); + Action act = () => ScoreCalculator.CalculateCategoryRawRange((AssessmentCategory)null!); act.Should().Throw(); } diff --git a/PanoramicData.IssueAssessment.Tests/Text/IssueTextTests.cs b/PanoramicData.IssueAssessment.Tests/Text/IssueTextTests.cs new file mode 100644 index 0000000..c5883f6 --- /dev/null +++ b/PanoramicData.IssueAssessment.Tests/Text/IssueTextTests.cs @@ -0,0 +1,164 @@ +using PanoramicData.IssueAssessment.Text; + +namespace PanoramicData.IssueAssessment.Tests.Text; + +/// +/// Tests for . +/// +public class IssueTextTests +{ + private const string Steps = @"(?:reproduc|\brepro\b)"; + + public static TheoryData Sections => new() + { + // Wiki heading: closes at the next wiki heading of the same or stronger level; sub-headings stay inside. + { "h2. Steps to reproduce\r\n# one\r\n# two\r\nh3. Detail\r\nmore\r\nh2. Expected\r\nx", "\r\n# one\r\n# two\r\nh3. Detail\r\nmore\r\n" }, + // In wiki text "# item" and a "## item" continuing it are list items, not headings. + { "h2. Steps to reproduce\n# one\n## nested\n# two\nh2. Next", "\n# one\n## nested\n# two\n" }, + // A Markdown "##" heading after wiki text closes the section when it is not a list item. + { "h2. Steps to reproduce\ntext\n\n## Next\nafter", "\ntext\n\n" }, + // ATX heading. + { "## Steps to reproduce\n1. open\n### Sub\nx\n## Next", "\n1. open\n### Sub\nx\n" }, + // Double-asterisk bold heading closes at the next bold line. + { "**Steps to reproduce**\n1. open\n**Expected**\nx", "\n1. open\n" }, + // Single-asterisk (Jira wiki) bold heading. + { "*Steps to reproduce*\n# open\n*Expected*\nx", "\n# open\n" }, + // Label line. + { "Steps to reproduce:\n1. open\nExpected result:\nx", "\n1. open\n" }, + // Heading without a closer runs to the end. + { "Intro\n## Repro\nall the rest", "\nall the rest" }, + // Bold heading without a closer runs to the end. + { "**Steps to reproduce**\n1. open\n2. export", "\n1. open\n2. export" }, + // A Markdown heading on the first line of wiki text has no previous line to continue. + { "## Repro\nbody\nh2. Next\nx", "\nbody\n" }, + { "No heading here", null }, + { " ", null }, + }; + + [Theory] + [MemberData(nameof(Sections))] + public void SectionBody_ReadsTheSectionForEachDialect(string text, string? expected) + { + IssueText.SectionBody(text, Steps).Should().Be(expected); + } + + [Fact] + public void SectionBody_SeveralMatchingHeadings_ReturnsTheLongestBody() + { + const string text = "## Repro\nshort\n## Other\nx\n## Steps to reproduce\nmuch longer body here\n## End"; + + IssueText.SectionBody(text, Steps).Should().Be("\nmuch longer body here\n"); + } + + [Fact] + public void SectionBody_WikiTextHashLines_CannotOpenASection() + { + // "# Repro" is a list item; "## Repro nested" continues it; and "## Repro heading", whose previous non-blank line + // is that "##" item, continues the list too. Yoda's Get-IssueSectionBody returns null for this text. + const string text = "h2. Summary\n# Repro the issue\n## Repro nested\n\n## Repro heading\nbody"; + + IssueText.SectionBody(text, Steps).Should().BeNull(); + } + + [Fact] + public void SectionBody_WikiTextMarkdownHeadingAfterProse_OpensASection() + { + const string text = "h2. Summary\nprose\n\n## Repro heading\nbody"; + + IssueText.SectionBody(text, Steps).Should().Be("\nbody"); + } + + [Theory] + [InlineData(null)] + [InlineData(" ")] + public void SectionBody_BlankPattern_Throws(string? pattern) + { + Action act = () => IssueText.SectionBody("x", pattern!); + + act.Should().Throw(); + } + + [Fact] + public void CriteriaHeadings_SkipsListItemsAndSectionsHoldingAMappingTable() + { + // "Definition of done" runs to the end (bold and h3. lines are weaker), so its section holds the + // criterion-to-test table and is evidence, as is "Verification evidence". Matches Yoda's Get-IssueCriteriaHeadings. + const string text = """ + h2. Acceptance criteria + # a wiki list item about criteria + h2. Definition of done: + done + *Verification evidence* + || Criterion || Test || + | one | test A | + h3. Other + """; + + IssueText.CriteriaHeadings(text).Should().Equal("Acceptance criteria"); + } + + [Fact] + public void CriteriaHeadings_LabelledHeadingsWithoutATable_AreAllCounted() + { + const string text = "h2. Acceptance criteria\na\nh2. Definition of done:\nb\nh2. Verification evidence\nc"; + + IssueText.CriteriaHeadings(text).Should().Equal("Acceptance criteria", "Definition of done", "Verification evidence"); + } + + [Fact] + public void CriteriaHeadings_MarkdownText_CountsAtxAndBoldHeadings() + { + const string text = "## Acceptance Criteria\n- a\n**Success criteria:**\n- b\n# Criteria item"; + + IssueText.CriteriaHeadings(text).Should().Equal("Acceptance Criteria", "Success criteria", "Criteria item"); + } + + [Theory] + [InlineData(null)] + [InlineData("")] + public void CriteriaHeadings_BlankText_IsEmpty(string? text) + { + IssueText.CriteriaHeadings(text).Should().BeEmpty(); + } + + [Theory] + [InlineData(null, true)] + [InlineData(" ", true)] + [InlineData("TBD", true)] + [InlineData("* N/A", true)] + [InlineData("1. To be confirmed.", true)] + [InlineData("- see above", true)] + [InlineData("Restore the backup", false)] + public void IsPlaceholder_RecognisesPlaceholderBodies(string? body, bool expected) + { + IssueText.IsPlaceholder(body).Should().Be(expected); + } + + [Theory] + [InlineData(null, 0)] + [InlineData("", 0)] + [InlineData(" one two\tthree\n", 3)] + public void WordCount_CountsWords(string? text, int expected) + { + IssueText.WordCount(text).Should().Be(expected); + } + + [Theory] + [InlineData(null, 0)] + [InlineData("a1-*é", 2)] + public void AlphanumericCount_CountsAsciiLettersAndDigits(string? text, int expected) + { + IssueText.AlphanumericCount(text).Should().Be(expected); + } + + [Theory] + [InlineData("h3. Title", 3)] + [InlineData(" ## Title", 2)] + [InlineData("**Bold**", 7)] + [InlineData("", 7)] + [InlineData(null, 7)] + public void HeadingLevel_MapsBothDialectsToOneScale(string? line, int expected) + { + IssueText.HeadingLevel(line).Should().Be(expected); + } +} diff --git a/PanoramicData.IssueAssessment.Tests/Validation/RubricValidationTests.cs b/PanoramicData.IssueAssessment.Tests/Validation/RubricValidationTests.cs new file mode 100644 index 0000000..652cdb0 --- /dev/null +++ b/PanoramicData.IssueAssessment.Tests/Validation/RubricValidationTests.cs @@ -0,0 +1,325 @@ +using PanoramicData.IssueAssessment.Serialization; + +namespace PanoramicData.IssueAssessment.Tests.Validation; + +/// +/// Every rubric error (conditions, measures, tiers, placeholders, caps, bands and classifications) is rejected when +/// is constructed, with a message saying what is wrong. +/// +public class RubricValidationTests +{ + private const string Tiered = """{ "type": "tiered", "tiers": [ { "score": 100, "finding": "ok" } ] }"""; + + /// + /// A one-category, one-assessment rubric with the given parts substituted. + /// + private static string Rubric( + string conditions = "{}", + string category = "", + string assessment = "", + string definition = Tiered, + string caps = "[]", + string bands = "[]", + string classifications = "{}") + => $$""" + { + "executionStrategy": "series", + "conditions": {{conditions}}, + "categories": [ { + "name": "C", "description": "C", "assessmentExecutionStrategy": "series" {{category}}, + "assessments": [ { "name": "A", "description": "A", "definition": {{definition}} {{assessment}} } ] + } ], + "scoreCaps": {{caps}}, + "gradeBands": {{bands}}, + "classifications": {{classifications}} + } + """; + + private static Action Construct(string json) => () => _ = new IssueAssessor(AssessmentRubric.Load(json)); + + private static string Tier(string when) => $$"""{ "type": "tiered", "tiers": [ { "when": {{when}}, "score": 0, "finding": "x" }, { "score": 100, "finding": "ok" } ] }"""; + + [Fact] + public void ValidRubric_IsAccepted() + { + Construct(Rubric( + conditions: """{ "bug": { "type": "fieldEquals", "path": "$.type", "values": [ "Bug" ] } }""", + category: """, "weight": 2, "appliesWhen": { "type": "ref", "name": "bug" }""", + assessment: """, "appliesWhen": { "type": "not", "condition": { "type": "ref", "name": "bug" } }, "notApplicableExplanation": "n/a for {$.type}" """, + caps: """[ { "name": "cap", "cap": 0.5, "when": { "type": "assessmentScore", "assessment": "A", "lessThan": 1 } } ]""", + bands: """[ { "name": "GOOD", "minimumScore": 0.5 }, { "name": "BAD", "minimumScore": 0 } ]""", + classifications: """{ "kind": [ { "when": { "type": "ref", "name": "bug" }, "value": "bug" }, { "value": "other" } ] }""")) + .Should().NotThrow(); + } + + public static TheoryData InvalidConditions => new() + { + { """{ "type": "all", "conditions": null }""", "combined condition without conditions" }, + { """{ "type": "any", "conditions": [ { "type": "ref", "name": "missing" } ] }""", "undefined condition 'missing'" }, + { """{ "type": "not", "condition": null }""", "'not' condition without a condition" }, + { """{ "type": "measure", "atLeast": 1 }""", "exactly one of Measure and Of" }, + { """{ "type": "measure", "measure": "m", "of": { "type": "number", "path": "$.a" } }""", "exactly one of Measure and Of" }, + { """{ "type": "measure", "measure": "m" }""", "undefined measure 'm'" }, + { """{ "type": "measure", "of": { "type": "itemCount", "path": "bad" } }""", "invalid JSONPath 'bad'" }, + { """{ "type": "assessmentScore", "assessment": "A" }""", "only score caps and classifications may use" }, + { """{ "type": "textMatches", "source": { "paths": [] }, "patterns": [ "x" ] }""", "text source without paths" }, + { """{ "type": "textMatches", "source": null, "patterns": [ "x" ] }""", "text source without paths" }, + { """{ "type": "textMatches", "source": { "paths": [ "$.a", "bad" ] }, "patterns": [ "x" ] }""", "invalid JSONPath 'bad'" }, + { """{ "type": "textMatches", "source": { "paths": [ "$.a" ], "section": "(" }, "patterns": [ "x" ] }""", "invalid regex pattern '('" }, + { """{ "type": "textMatches", "source": { "paths": [ "$.a" ] }, "patterns": [] }""", "has no patterns" }, + { """{ "type": "textMatches", "source": { "paths": [ "$.a" ] }, "patterns": [ " " ] }""", "has an empty pattern" }, + { """{ "type": "fieldMatches", "path": "bad", "patterns": [ "x" ] }""", "invalid JSONPath 'bad'" }, + { """{ "type": "fieldMatches", "path": "$.a", "patterns": [ "[" ] }""", "invalid regex pattern '['" }, + { """{ "type": "fieldEquals", "path": "$.a", "values": [] }""", "'fieldEquals' condition without values" }, + { """{ "type": "fieldEquals", "path": "bad", "values": [ "x" ] }""", "invalid JSONPath 'bad'" }, + { """{ "type": "fieldPresent", "path": "bad" }""", "invalid JSONPath 'bad'" }, + { """{ "type": "sectionPresent", "source": { "paths": [ "$.a" ] } }""", "whose source has no section" }, + { """{ "type": "sectionPresent", "source": { "paths": [ "bad" ], "section": "x" } }""", "invalid JSONPath 'bad'" }, + { """{ "type": "placeholder", "source": { "paths": [ "bad" ] } }""", "invalid JSONPath 'bad'" }, + }; + + [Theory] + [MemberData(nameof(InvalidConditions))] + public void InvalidCondition_InAppliesWhen_IsRejected(string condition, string message) + { + Construct(Rubric(assessment: $$""", "appliesWhen": {{condition}} """)).Should().Throw() + .WithMessage($"Assessment 'A' in category 'C' *{message}*"); + } + + [Theory] + [MemberData(nameof(InvalidConditions))] + public void InvalidCondition_AsANamedCondition_IsRejected(string condition, string message) + { + Construct(Rubric(conditions: $$"""{ "named": {{condition}} }""")).Should().Throw() + .WithMessage($"Condition 'named' *{message}*"); + } + + public static TheoryData InvalidMeasures => new() + { + { "null", "has a missing measure" }, + { """{ "type": "patternMatchCount", "source": { "paths": [ "$.a" ] }, "patterns": [ "(" ] }""", "invalid regex pattern '('" }, + { """{ "type": "occurrenceCount", "source": { "paths": [ "$.a" ] }, "pattern": "" }""", "has an empty pattern" }, + { """{ "type": "wordCount", "source": { "paths": [ "bad" ] } }""", "invalid JSONPath 'bad'" }, + { """{ "type": "alphanumericCount", "source": { "paths": [ "bad" ] } }""", "invalid JSONPath 'bad'" }, + { """{ "type": "length", "source": { "paths": [ "bad" ] } }""", "invalid JSONPath 'bad'" }, + { """{ "type": "criteriaHeadingCount", "source": { "paths": [ "bad" ] } }""", "invalid JSONPath 'bad'" }, + { """{ "type": "itemCount", "path": "bad" }""", "invalid JSONPath 'bad'" }, + { """{ "type": "number", "path": "bad" }""", "invalid JSONPath 'bad'" }, + }; + + [Theory] + [MemberData(nameof(InvalidMeasures))] + public void InvalidMeasure_InATieredAssessment_IsRejected(string measure, string message) + { + string definition = $$"""{ "type": "tiered", "measures": { "m": {{measure}} }, "tiers": [ { "score": 100, "finding": "ok" } ] }"""; + + Construct(Rubric(definition: definition)).Should().Throw().WithMessage($"*{message}*"); + } + + public static TheoryData InvalidTiers => new() + { + { """{ "type": "tiered", "tiers": [] }""", "at least one tier" }, + { """{ "type": "tiered", "tiers": null }""", "at least one tier" }, + { """{ "type": "tiered", "minimumScore": 5, "maximumScore": 5, "tiers": [ { "score": 5, "finding": "x" } ] }""", "MinimumScore that is not below its MaximumScore" }, + { """{ "type": "tiered", "tiers": [ { "when": { "type": "fieldPresent", "path": "$.a" }, "score": 1, "finding": "x" } ] }""", "must end with a tier that has no condition" }, + { """{ "type": "tiered", "tiers": [ { "score": 1, "finding": "x" }, { "score": 2, "finding": "y" } ] }""", "tier without a condition before its last tier" }, + { """{ "type": "tiered", "tiers": [ { "score": 1, "finding": null } ] }""", "tier without a finding" }, + { """{ "type": "tiered", "tiers": [ { "score": 101, "finding": "x" } ] }""", "tier score of 101 outside its range 0-100" }, + { """{ "type": "tiered", "tiers": [ { "score": -1, "finding": "x" } ] }""", "tier score of -1 outside its range 0-100" }, + { """{ "type": "tiered", "tiers": [ { "score": 1, "finding": "{unknown}" } ] }""", "undefined measure 'unknown' in a placeholder" }, + { """{ "type": "tiered", "tiers": [ { "score": 1, "finding": "x", "recommendation": "{m:text}" } ] }""", "undefined measure 'm' in a placeholder" }, + { """{ "type": "tiered", "tiers": [ { "score": 1, "finding": "{$bad}" } ] }""", "invalid JSONPath '$bad'" }, + { """{ "type": "tiered", "tiers": [ { "score": 1, "finding": "{$.a~(}" } ] }""", "invalid regex pattern '('" }, + { """{ "type": "tiered", "tiers": [ { "score": 1, "finding": "{$bad~x}" } ] }""", "invalid JSONPath '$bad'" }, + { """{ "type": "named", "name": "custom" }""", "custom definition 'custom', which was not registered" }, + }; + + [Theory] + [MemberData(nameof(InvalidTiers))] + public void InvalidTieredDefinition_IsRejected(string definition, string message) + { + Construct(Rubric(definition: definition)).Should().Throw().WithMessage($"Assessment 'A' in category 'C' *{message}*"); + } + + [Fact] + public void TierCondition_UsingAnUndefinedMeasure_IsRejected() + { + Construct(Rubric(definition: Tier("""{ "type": "measure", "measure": "m", "atLeast": 1 }"""))).Should().Throw() + .WithMessage("*undefined measure 'm'*"); + } + + [Fact] + public void TierCondition_UsingAnAssessmentScore_IsRejected() + { + Construct(Rubric(definition: Tier("""{ "type": "assessmentScore", "assessment": "A" }"""))).Should().Throw() + .WithMessage("*only score caps and classifications may use*"); + } + + [Theory] + [InlineData("{done}")] + [InlineData("{$bad}")] + public void NotApplicableExplanation_WithAnInvalidPlaceholder_IsRejected(string explanation) + { + Construct(Rubric(assessment: $$""", "notApplicableExplanation": "{{explanation}}" """)).Should().Throw() + .WithMessage("Assessment 'A' in category 'C' *"); + } + + [Theory] + [InlineData("""{ "a": { "type": "ref", "name": "a" } }""", "Condition 'a' refers to itself.")] + [InlineData("""{ "a": { "type": "not", "condition": { "type": "ref", "name": "b" } }, "b": { "type": "any", "conditions": [ { "type": "all", "conditions": [ { "type": "ref", "name": "a" } ] } ] } }""", "Condition '?' refers to itself.")] + public void CyclicNamedConditions_AreRejected(string conditions, string message) + { + Construct(Rubric(conditions: conditions)).Should().Throw().WithMessage($"{message}*"); + } + + [Fact] + public void SharedButAcyclicNamedConditions_AreAccepted() + { + Construct(Rubric(conditions: """ + { + "a": { "type": "all", "conditions": [ { "type": "ref", "name": "c" }, { "type": "ref", "name": "c" } ] }, + "b": { "type": "ref", "name": "c" }, + "c": { "type": "fieldPresent", "path": "$.x" } + } + """)).Should().NotThrow(); + } + + [Fact] + public void NullNamedCondition_IsRejected() + { + Construct(Rubric(conditions: """{ "a": null }""")).Should().Throw().WithMessage("Condition 'a' is missing.*"); + } + + [Theory] + [InlineData(""", "weight": -1""", "Category 'C' has invalid weight -1")] + [InlineData(""", "appliesWhen": { "type": "ref", "name": "missing" }""", "Category 'C' refers to undefined condition 'missing'")] + public void InvalidCategoryRule_IsRejected(string category, string message) + { + Construct(Rubric(category: category)).Should().Throw().WithMessage($"{message}*"); + } + + [Theory] + [InlineData(double.NaN)] + [InlineData(double.PositiveInfinity)] + public void NonFiniteCategoryWeight_IsRejected(double weight) + { + AssessmentOptions options = AssessmentRubric.Load(Rubric()); + options = options with { Categories = [options.Categories[0] with { Weight = weight }] }; + + Action act = () => _ = new IssueAssessor(options); + + act.Should().Throw().WithMessage("Category 'C' has invalid weight*"); + } + + public static TheoryData InvalidCaps => new() + { + { """[ { "name": " ", "cap": 0.5, "when": { "type": "fieldPresent", "path": "$.a" } } ]""", "A score cap has no name." }, + { """[ { "name": "x", "cap": 1.5, "when": { "type": "fieldPresent", "path": "$.a" } } ]""", "Score cap 'x' must be between 0 and 1." }, + { """[ { "name": "x", "cap": -0.1, "when": { "type": "fieldPresent", "path": "$.a" } } ]""", "Score cap 'x' must be between 0 and 1." }, + { """[ { "name": "x", "cap": 0.5, "when": null } ]""", "Score cap 'x' has no condition." }, + { """[ { "name": "x", "cap": 0.5, "when": { "type": "assessmentScore", "assessment": "Nope" } } ]""", "Score cap 'x' refers to unknown assessment 'Nope'." }, + }; + + [Theory] + [MemberData(nameof(InvalidCaps))] + public void InvalidScoreCap_IsRejected(string caps, string message) + { + Construct(Rubric(caps: caps)).Should().Throw().WithMessage($"{message}*"); + } + + [Fact] + public void NaNScoreCap_IsRejected() + { + AssessmentOptions options = AssessmentRubric.Load(Rubric()) with + { + ScoreCaps = [new ScoreCap { Name = "x", Cap = double.NaN, When = new FieldPresentCondition { Path = "$.a" } }], + }; + + Action act = () => _ = new IssueAssessor(options); + + act.Should().Throw().WithMessage("Score cap 'x' must be between 0 and 1.*"); + } + + public static TheoryData InvalidBands => new() + { + { """[ { "name": "", "minimumScore": 0.5 } ]""", "Grade band '' must have a name" }, + { """[ { "name": "A", "minimumScore": 2 } ]""", "Grade band 'A' must have a name and a minimum score between 0 and 1." }, + { """[ { "name": "A", "minimumScore": -1 } ]""", "Grade band 'A' must have a name and a minimum score between 0 and 1." }, + { """[ { "name": "A", "minimumScore": 0.5 }, { "name": "A", "minimumScore": 0.1 } ]""", "Grade band 'A' is defined more than once." }, + }; + + [Theory] + [MemberData(nameof(InvalidBands))] + public void InvalidGradeBand_IsRejected(string bands, string message) + { + Construct(Rubric(bands: bands)).Should().Throw().WithMessage($"{message}*"); + } + + [Fact] + public void NaNGradeBand_IsRejected() + { + AssessmentOptions options = AssessmentRubric.Load(Rubric()) with { GradeBands = [new GradeBand { Name = "A", MinimumScore = double.NaN }] }; + + Action act = () => _ = new IssueAssessor(options); + + act.Should().Throw().WithMessage("Grade band 'A'*"); + } + + public static TheoryData InvalidClassifications => new() + { + { """{ "k": [] }""", "Classification 'k' has no rules." }, + { """{ "k": null }""", "Classification 'k' has no rules." }, + { """{ "k": [ { "value": " " } ] }""", "Classification 'k' has a rule without a value." }, + { """{ "k": [ { "when": { "type": "ref", "name": "missing" }, "value": "v" } ] }""", "Classification 'k' refers to undefined condition 'missing'." }, + }; + + [Theory] + [MemberData(nameof(InvalidClassifications))] + public void InvalidClassification_IsRejected(string classifications, string message) + { + Construct(Rubric(classifications: classifications)).Should().Throw().WithMessage($"{message}*"); + } + + [Fact] + public void CustomConditionAndMeasureTypes_AreNotInspected() + { + AssessmentOptions options = AssessmentRubric.Load(Rubric()) with + { + Conditions = ImmutableDictionary.Empty.Add("custom", new AlwaysTrue()), + }; + options = options with + { + Categories = + [ + options.Categories[0] with + { + Assessments = + [ + options.Categories[0].Assessments[0] with + { + Definition = new TieredAssessment + { + Measures = ImmutableDictionary.Empty.Add("m", new Seven()), + Tiers = [new Tier { Score = 100, Finding = "{m}" }], + }, + }, + ], + }, + ], + }; + + Action act = () => _ = new IssueAssessor(options); + + act.Should().NotThrow(); + } + + private sealed class AlwaysTrue : ICondition + { + public bool Evaluate(EvaluationContext context) => context is not null; + } + + private sealed class Seven : IMeasure + { + public double Evaluate(EvaluationContext context) => context is null ? 0d : 7d; + } +} diff --git a/PanoramicData.IssueAssessment/Aggregation/AggregationCalculator.cs b/PanoramicData.IssueAssessment/Aggregation/AggregationCalculator.cs index 3a269b1..df6b363 100644 --- a/PanoramicData.IssueAssessment/Aggregation/AggregationCalculator.cs +++ b/PanoramicData.IssueAssessment/Aggregation/AggregationCalculator.cs @@ -45,7 +45,7 @@ public static AssessmentAggregations Calculate(ImmutableList cat (double? weightedAverage, _, double weightSum) = ScoreCalculator.CalculateWeightedAverages( categories - .Select(category => (Score: category.Score.NormalizedScore, Weight: 1.0d)) + .Select(category => (Score: category.Outcome == AssessmentOutcome.NotApplicable ? null : category.Score.NormalizedScore, category.Weight)) .ToList()); WeightedAggregationMetrics weighted = new(weightedAverage, weightSum); diff --git a/PanoramicData.IssueAssessment/Aggregation/OverallCalculator.cs b/PanoramicData.IssueAssessment/Aggregation/OverallCalculator.cs new file mode 100644 index 0000000..10a02b2 --- /dev/null +++ b/PanoramicData.IssueAssessment/Aggregation/OverallCalculator.cs @@ -0,0 +1,65 @@ +namespace PanoramicData.IssueAssessment.Aggregation; + +/// +/// Computes the overall verdict on an issue from its category results and the rubric. +/// +internal static class OverallCalculator +{ + /// + /// Scores are compared with grade bands after a division, so 80/100 can arrive as 0.79999999999999993; a band is + /// reached within this tolerance. + /// + private const double Tolerance = 1e-9; + + public static AssessmentOverall Calculate( + IAssessmentOptions options, + EvaluationContext context, + ImmutableList categories) + { + double? uncapped = WeightedAverage(categories); + EvaluationContext scoring = context.WithAssessmentResults(ResultsByName(categories)); + ImmutableList caps = [.. options.ScoreCaps + .Where(cap => cap.When.Evaluate(scoring)) + .Select(cap => new AppliedScoreCap(cap.Name, cap.Group, cap.Cap))]; + + double? score = uncapped is { } value ? caps.Aggregate(value, (current, cap) => Math.Min(current, cap.Cap)) : null; + return new AssessmentOverall(score, uncapped, Grade(options, score), caps, context.AllConditions(), Classify(options, scoring)); + } + + /// + /// The weighted average of the scored categories, leaving out those that do not apply; null when none is scored. + /// + private static double? WeightedAverage(ImmutableList categories) + { + List scored = [.. categories.Where(category => + category.Outcome != AssessmentOutcome.NotApplicable && category.Score.NormalizedScore.HasValue)]; + double weightSum = scored.Sum(category => category.Weight); + return weightSum > 0d + ? scored.Sum(category => category.Weight * category.Score.NormalizedScore!.Value) / weightSum + : null; + } + + private static Dictionary ResultsByName(ImmutableList categories) + { + Dictionary results = new(StringComparer.Ordinal); + foreach (AssessmentResult result in categories.SelectMany(category => category.Assessments)) + { + results[result.AssessmentName] = result; + } + + return results; + } + + private static string? Grade(IAssessmentOptions options, double? score) + => score is { } graded + ? options.GradeBands + .OrderByDescending(band => band.MinimumScore) + .FirstOrDefault(band => graded >= band.MinimumScore - Tolerance)?.Name + : null; + + private static ImmutableDictionary Classify(IAssessmentOptions options, EvaluationContext scoring) + => options.Classifications + .Select(classification => (classification.Key, Rule: classification.Value.Find(rule => rule.When is null || rule.When.Evaluate(scoring)))) + .Where(pair => pair.Rule is not null) + .ToImmutableDictionary(pair => pair.Key, pair => pair.Rule!.Value, StringComparer.Ordinal); +} diff --git a/PanoramicData.IssueAssessment/Assessments/NamedAssessmentDefinition.cs b/PanoramicData.IssueAssessment/Assessments/NamedAssessmentDefinition.cs new file mode 100644 index 0000000..d702e8c --- /dev/null +++ b/PanoramicData.IssueAssessment/Assessments/NamedAssessmentDefinition.cs @@ -0,0 +1,24 @@ +namespace PanoramicData.IssueAssessment.Assessments; + +/// +/// Stands in, in a JSON rubric, for an assessment definition written in code. Loading the rubric replaces it with the +/// definition registered under . +/// +/// +/// Definitions that hold delegates (, ) +/// and custom implementations cannot be written as JSON, so a rubric names them and +/// the caller supplies them to . +/// An assessor rejects one that was never replaced. +/// +public sealed record NamedAssessmentDefinition : IAssessmentDefinition +{ + /// + /// Gets the name the definition is registered under. + /// + public required string Name { get; init; } + + /// + /// Always: a named definition must be replaced before use. + public Task ExecuteAsync(JsonDocument input, JsonDocument? virtualExtension, CancellationToken cancellationToken) + => throw new InvalidOperationException($"Custom definition '{Name}' was not registered when the rubric was loaded."); +} diff --git a/PanoramicData.IssueAssessment/Assessments/TieredAssessment.cs b/PanoramicData.IssueAssessment/Assessments/TieredAssessment.cs new file mode 100644 index 0000000..cff1bd0 --- /dev/null +++ b/PanoramicData.IssueAssessment/Assessments/TieredAssessment.cs @@ -0,0 +1,108 @@ + +namespace PanoramicData.IssueAssessment.Assessments; + +/// +/// Scores an issue by the first of its whose condition holds: partial credit, with a finding and a +/// recommendation for each level. +/// +/// +/// +/// A tier's condition can use the assessment's named (through ), the +/// rubric's named conditions and any other . The last tier must have no condition, so every +/// issue lands in some tier. +/// +/// +/// Findings and recommendations may contain placeholders: {name} is replaced by the value of a measure, +/// {name:text} by a described measure's text (for example the duplicated criteria headings), +/// {$.path} by the values at a JSONPath, comma-separated, and {$.path~pattern} by the first of them that +/// matches the pattern. A finding may be empty, for a tier that only recommends. +/// +/// +/// +/// Three or more business-case phrases score 100, one or two score 50, none scores 0: +/// +/// new TieredAssessment +/// { +/// Measures = ImmutableDictionary<string, IMeasure>.Empty.Add("matches", new PatternMatchCountMeasure { Source = fullText, Patterns = businessCase }), +/// Tiers = +/// [ +/// new Tier { When = new MeasureCondition { Measure = "matches", AtLeast = 3 }, Score = 100, Finding = "✓ Business case: clear" }, +/// new Tier { When = new MeasureCondition { Measure = "matches", AtLeast = 1 }, Score = 50, Finding = "⚠ Business case: brief", Recommendation = "Expand the business case" }, +/// new Tier { Score = 0, Finding = "✗ Business case: none", Recommendation = "Add a business case" }, +/// ], +/// } +/// +/// +public sealed record TieredAssessment : IAssessmentDefinition, IContextualAssessmentDefinition +{ + /// + /// Gets measures the tiers can refer to by name. + /// + public ImmutableDictionary Measures { get; init; } = ImmutableDictionary.Empty; + + /// + /// Gets the tiers, evaluated in order; the first whose condition holds gives the score. + /// + public required ImmutableList Tiers { get; init; } + + /// + /// Gets the lowest possible score, for normalization. Default 0. + /// + public double MinimumScore { get; init; } + + /// + /// Gets the highest possible score, for normalization. Default 100. + /// + public double MaximumScore { get; init; } = 100d; + + /// + public Task ExecuteAsync( + JsonDocument input, + JsonDocument? virtualExtension, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(input); + return ExecuteAsync(new EvaluationContext(input).WithVirtualExtension(virtualExtension), cancellationToken); + } + + /// + public Task ExecuteAsync(EvaluationContext context, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(context); + cancellationToken.ThrowIfCancellationRequested(); + + EvaluationContext scoped = context.WithMeasures(Measures); + Tier tier = Tiers.Find(candidate => candidate.When is null || candidate.When.Evaluate(scoped)) + ?? throw new InvalidOperationException("No tier matched; the last tier must have no condition."); + + return Task.FromResult(new AssessmentExecutionResult( + RawScore: tier.Score, + ExplanationText: Format(tier.Finding, scoped), + FailureReason: null, + InnerException: null, + VirtualExtensionOutput: null) + { + Recommendation = tier.Recommendation is null ? null : Format(tier.Recommendation, scoped), + }); + } + + private string Format(string text, EvaluationContext context) => PlaceholderText.Format(text, context, Measures); +} + +/// +/// One level of a . +/// +public sealed record Tier +{ + /// Gets the condition for this tier, or null for the final, catch-all tier. + public ICondition? When { get; init; } + + /// Gets the score this tier gives. + public required double Score { get; init; } + + /// Gets the finding reported for this tier; it may contain placeholders. + public required string Finding { get; init; } + + /// Gets what to recommend at this tier, if anything; it may contain placeholders. + public string? Recommendation { get; init; } +} diff --git a/PanoramicData.IssueAssessment/Configuration/AssessmentOptions.cs b/PanoramicData.IssueAssessment/Configuration/AssessmentOptions.cs index 22e1eda..895bac3 100644 --- a/PanoramicData.IssueAssessment/Configuration/AssessmentOptions.cs +++ b/PanoramicData.IssueAssessment/Configuration/AssessmentOptions.cs @@ -28,6 +28,19 @@ public sealed record AssessmentOptions : IAssessmentOptions /// Gets a value indicating whether the final virtual document should be included in results. /// public bool CaptureVirtualDocument { get; init; } + + /// + public ImmutableDictionary Conditions { get; init; } = ImmutableDictionary.Empty; + + /// + public ImmutableList ScoreCaps { get; init; } = []; + + /// + public ImmutableList GradeBands { get; init; } = []; + + /// + public ImmutableDictionary> Classifications { get; init; } + = ImmutableDictionary>.Empty; } /// @@ -70,6 +83,17 @@ public sealed record AssessmentCategory /// Gets an optional gate used when categories execute in series. /// public ISeriesGateCondition? SeriesGateCondition { get; init; } + + /// + /// Gets the category's weight in the overall score. Default 1; must not be negative. + /// + public double Weight { get; init; } = 1d; + + /// + /// Gets an optional condition under which the category applies. When it is false the category is not run, is + /// reported as not applicable, and does not count in the overall score. + /// + public ICondition? AppliesWhen { get; init; } } /// @@ -111,4 +135,22 @@ public sealed record Assessment /// Gets the logical name for the virtual output value. /// public string? VirtualOutputName { get; init; } + + /// + /// Gets an optional condition under which the assessment applies. When it is false the assessment is not run, is + /// reported as not applicable, and is left out of its category's score and score range, so the remaining weights + /// are renormalised. + /// + public ICondition? AppliesWhen { get; init; } + + /// + /// Gets what to recommend when the assessment does not pass. A tier's own recommendation takes precedence. + /// + public string? Recommendation { get; init; } + + /// + /// Gets the explanation reported when is false, for example "○ Component: not used by this + /// project". When not set, a not-applicable assessment reports an empty explanation. + /// + public string? NotApplicableExplanation { get; init; } } diff --git a/PanoramicData.IssueAssessment/Configuration/RubricRules.cs b/PanoramicData.IssueAssessment/Configuration/RubricRules.cs new file mode 100644 index 0000000..7e1bbb6 --- /dev/null +++ b/PanoramicData.IssueAssessment/Configuration/RubricRules.cs @@ -0,0 +1,48 @@ +namespace PanoramicData.IssueAssessment.Configuration; + +/// +/// A cap on the overall score: when holds, the score is held at or below . +/// +/// +/// Use a cap rather than a weight for an element without which an issue must not reach the top grade however good the +/// rest is, such as steps to reproduce on a bug. A weight only costs points; a cap makes the grade unreachable. +/// +public sealed record ScoreCap +{ + /// Gets the cap's name, reported when it applies. + public required string Name { get; init; } + + /// Gets the highest score allowed, in [0, 1]. + public required double Cap { get; init; } + + /// Gets the condition under which the cap applies. It may use . + public required ICondition When { get; init; } + + /// Gets an optional group name, such as "mandatory", for reporting caps together. + public string? Group { get; init; } +} + +/// +/// A grade band: scores of at least earn unless a higher band applies. +/// +public sealed record GradeBand +{ + /// Gets the grade's name, such as "EXCELLENT". + public required string Name { get; init; } + + /// Gets the lowest score, in [0, 1], that earns this grade. + public required double MinimumScore { get; init; } +} + +/// +/// One rule of a classification: when holds (or is not set), the classification takes +/// . Rules are evaluated in order and the first match wins. +/// +public sealed record ClassificationRule +{ + /// Gets the condition, or null for a rule that always matches. It may use . + public ICondition? When { get; init; } + + /// Gets the value assigned when the rule matches. + public required string Value { get; init; } +} diff --git a/PanoramicData.IssueAssessment/Core/Interfaces.cs b/PanoramicData.IssueAssessment/Core/Interfaces.cs index d605712..b949506 100644 --- a/PanoramicData.IssueAssessment/Core/Interfaces.cs +++ b/PanoramicData.IssueAssessment/Core/Interfaces.cs @@ -1,3 +1,6 @@ +using System.Text.Json.Serialization; +using PanoramicData.IssueAssessment.Gates; + namespace PanoramicData.IssueAssessment.Core; /// /// Defines immutable options used to configure issue assessments. @@ -28,11 +31,45 @@ public interface IAssessmentOptions /// Gets a value indicating whether the final virtual document should be captured in results. /// bool CaptureVirtualDocument { get; } + + /// + /// Gets named conditions that rules refer to with a , and that are reported as the + /// result's flags. + /// + ImmutableDictionary Conditions => ImmutableDictionary.Empty; + + /// + /// Gets caps on the overall score, applied when their condition holds: for example, a missing mandatory element + /// holds the score below the top grade. + /// + ImmutableList ScoreCaps => []; + + /// + /// Gets the grade bands the overall score is graded by. + /// + ImmutableList GradeBands => []; + + /// + /// Gets classifications: for each name, rules evaluated in order whose first match gives the value. + /// + ImmutableDictionary> Classifications + => ImmutableDictionary>.Empty; } /// /// Defines an executable assessment implementation. /// +[JsonPolymorphic(TypeDiscriminatorPropertyName = "type")] +[JsonDerivedType(typeof(PresenceAssessment), "presence")] +[JsonDerivedType(typeof(RegexAssessment), "regex")] +[JsonDerivedType(typeof(StringLengthAssessment), "stringLength")] +[JsonDerivedType(typeof(WordCountAssessment), "wordCount")] +[JsonDerivedType(typeof(NumericRangeAssessment), "numericRange")] +[JsonDerivedType(typeof(DateRangeAssessment), "dateRange")] +[JsonDerivedType(typeof(EnumAssessment), "enum")] +[JsonDerivedType(typeof(PatternCountAssessment), "patternCount")] +[JsonDerivedType(typeof(TieredAssessment), "tiered")] +[JsonDerivedType(typeof(NamedAssessmentDefinition), "named")] public interface IAssessmentDefinition { /// @@ -48,6 +85,22 @@ Task ExecuteAsync( CancellationToken cancellationToken); } +/// +/// An assessment definition that can use the : the rubric's named conditions and the +/// cached text of the issue. The assessor calls this overload instead of +/// . +/// +public interface IContextualAssessmentDefinition +{ + /// + /// Executes the assessment against the issue and virtual extension in . + /// + /// The evaluation context. + /// Cancellation token. + /// The execution result. + Task ExecuteAsync(EvaluationContext context, CancellationToken cancellationToken); +} + /// /// Marker interface for assessment definitions that operate on a JSONPath selector. /// @@ -73,6 +126,9 @@ public interface IRegexPatternAssessmentDefinition /// /// Defines a gate condition that controls whether the next series category should execute. /// +[JsonPolymorphic(TypeDiscriminatorPropertyName = "type")] +[JsonDerivedType(typeof(ScoreThresholdGate), "scoreThreshold")] +[JsonDerivedType(typeof(NoFailuresGate), "noFailures")] public interface ISeriesGateCondition { /// @@ -140,6 +196,12 @@ public interface IAssessmentResult /// Gets the final virtual document, if captured. /// JsonDocument? VirtualDocument { get; } + + /// + /// Gets the overall verdict: the weighted score across categories after any score caps, its grade, and the + /// rubric's flags and classifications. Null for results produced before 1.1 or by custom implementations. + /// + AssessmentOverall? Overall => null; } /// diff --git a/PanoramicData.IssueAssessment/Execution/AssessmentExecutor.cs b/PanoramicData.IssueAssessment/Execution/AssessmentExecutor.cs index 331cb25..7e71262 100644 --- a/PanoramicData.IssueAssessment/Execution/AssessmentExecutor.cs +++ b/PanoramicData.IssueAssessment/Execution/AssessmentExecutor.cs @@ -8,66 +8,111 @@ namespace PanoramicData.IssueAssessment.Execution; internal static class AssessmentExecutor { /// - /// Executes an assessment and returns normalized result artifacts. + /// Executes an assessment against with no rubric conditions. /// - public static async Task ExecuteAsync( + public static Task ExecuteAsync( string categoryName, Assessment assessment, JsonDocument input, JsonDocument? virtualExtension, CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(input); + return ExecuteAsync(categoryName, assessment, new EvaluationContext(input), virtualExtension, cancellationToken); + } + + /// + /// Executes an assessment and returns normalized result artifacts. An assessment whose + /// is false is not run and is reported as not applicable. + /// + public static async Task ExecuteAsync( + string categoryName, + Assessment assessment, + EvaluationContext context, + JsonDocument? virtualExtension, + CancellationToken cancellationToken) { ArgumentNullException.ThrowIfNull(categoryName); ArgumentNullException.ThrowIfNull(assessment); + ArgumentNullException.ThrowIfNull(context); - (double minPossibleRaw, double maxPossibleRaw) = ScoreCalculator.CalculateAssessmentRawRange(assessment); - AssessmentExecutionResult executionResult = await RunAsync(assessment, input, virtualExtension, cancellationToken) - .ConfigureAwait(false); + EvaluationContext scoped = context.WithVirtualExtension(virtualExtension); + AssessmentExecutionResult? executionResult = await RunAsync(assessment, scoped, cancellationToken).ConfigureAwait(false); + if (executionResult is null) + { + return NotApplicable(assessment, scoped); + } + + return new AssessmentExecutionItemResult( + AssessmentResult: Result(assessment, executionResult), + Failure: Failure(categoryName, assessment, executionResult), + VirtualDocument: executionResult.VirtualExtensionOutput); + } + /// + /// The scored result, with the recommendation when the assessment did not pass. + /// + private static AssessmentResult Result(Assessment assessment, AssessmentExecutionResult executionResult) + { + (double minPossibleRaw, double maxPossibleRaw) = ScoreCalculator.CalculateAssessmentRawRange(assessment); double? normalizedScore = executionResult.RawScore.HasValue ? ScoreCalculator.Normalize(executionResult.RawScore.Value, minPossibleRaw, maxPossibleRaw) : null; - AssessmentScore score = new( - RawScore: executionResult.RawScore, - NormalizedScore: normalizedScore, - MinPossibleRaw: minPossibleRaw, - MaxPossibleRaw: maxPossibleRaw); - - AssessmentResult assessmentResult = new( + AssessmentResult result = new( AssessmentName: assessment.Name, - Score: score, + Score: new AssessmentScore(executionResult.RawScore, normalizedScore, minPossibleRaw, maxPossibleRaw), ExplanationText: executionResult.ExplanationText ?? string.Empty, FailureReason: executionResult.FailureReason, InnerException: executionResult.InnerException); + return result.Outcome == AssessmentOutcome.Passed + ? result + : result with { Recommendation = executionResult.Recommendation ?? assessment.Recommendation }; + } - AssessmentFailure? failure = executionResult.FailureReason is null - ? null - : new AssessmentFailure( + private static AssessmentFailure? Failure(string categoryName, Assessment assessment, AssessmentExecutionResult executionResult) + => executionResult.FailureReason is { } reason + ? new AssessmentFailure( CategoryName: categoryName, AssessmentName: assessment.Name, - Reason: executionResult.FailureReason.Value, + Reason: reason, Message: executionResult.ExplanationText ?? "Assessment failed.", - InnerException: executionResult.InnerException); + InnerException: executionResult.InnerException) + : null; - return new AssessmentExecutionItemResult( - AssessmentResult: assessmentResult, - Failure: failure, - VirtualDocument: executionResult.VirtualExtensionOutput); - } + private static AssessmentExecutionItemResult NotApplicable(Assessment assessment, EvaluationContext context) + => new( + AssessmentResult: new AssessmentResult( + AssessmentName: assessment.Name, + Score: new AssessmentScore(null, null, 0d, 0d), + ExplanationText: assessment.NotApplicableExplanation is { } explanation + ? PlaceholderText.Format(explanation, context, ImmutableDictionary.Empty) + : string.Empty, + FailureReason: null, + InnerException: null) + { + Outcome = AssessmentOutcome.NotApplicable, + }, + Failure: null, + VirtualDocument: null); /// - /// Runs the definition, turning a cancellation into a timeout result and any other exception into a failure result. + /// Checks applicability, then runs the definition, turning a cancellation into a timeout result and any other + /// exception into a failure result. Returns null when the assessment does not apply. /// - private static async Task RunAsync( + private static async Task RunAsync( Assessment assessment, - JsonDocument input, - JsonDocument? virtualExtension, + EvaluationContext context, CancellationToken cancellationToken) { try { - return await ExecuteDefinitionAsync(assessment, input, virtualExtension, cancellationToken).ConfigureAwait(false); + if (assessment.AppliesWhen is { } appliesWhen && !appliesWhen.Evaluate(context)) + { + return null; + } + + return await ExecuteDefinitionAsync(assessment, context, cancellationToken).ConfigureAwait(false); } catch (OperationCanceledException) when (cancellationToken.IsCancellationRequested) { @@ -91,13 +136,12 @@ private static async Task RunAsync( private static async Task ExecuteDefinitionAsync( Assessment assessment, - JsonDocument input, - JsonDocument? virtualExtension, + EvaluationContext context, CancellationToken cancellationToken) { if (!assessment.AssessmentTimeout.HasValue) { - return await assessment.Definition.ExecuteAsync(input, virtualExtension, cancellationToken).ConfigureAwait(false); + return await InvokeAsync(assessment.Definition, context, cancellationToken).ConfigureAwait(false); } using CancellationTokenSource timeoutCts = new(assessment.AssessmentTimeout.Value); @@ -105,7 +149,7 @@ private static async Task ExecuteDefinitionAsync( try { - return await assessment.Definition.ExecuteAsync(input, virtualExtension, linkedCts.Token).ConfigureAwait(false); + return await InvokeAsync(assessment.Definition, context, linkedCts.Token).ConfigureAwait(false); } catch (OperationCanceledException) when (timeoutCts.IsCancellationRequested && !cancellationToken.IsCancellationRequested) { @@ -117,4 +161,12 @@ private static async Task ExecuteDefinitionAsync( VirtualExtensionOutput: null); } } + + private static Task InvokeAsync( + IAssessmentDefinition definition, + EvaluationContext context, + CancellationToken cancellationToken) + => definition is IContextualAssessmentDefinition contextual + ? contextual.ExecuteAsync(context, cancellationToken) + : definition.ExecuteAsync(context.Issue, context.VirtualExtension, cancellationToken); } diff --git a/PanoramicData.IssueAssessment/Execution/CategoryExecutor.cs b/PanoramicData.IssueAssessment/Execution/CategoryExecutor.cs index 2f7abb8..26adab7 100644 --- a/PanoramicData.IssueAssessment/Execution/CategoryExecutor.cs +++ b/PanoramicData.IssueAssessment/Execution/CategoryExecutor.cs @@ -8,17 +8,37 @@ namespace PanoramicData.IssueAssessment.Execution; internal static class CategoryExecutor { /// - /// Executes category assessments and returns category output. + /// Executes category assessments against with no rubric conditions. /// - public static async Task<(CategoryResult Category, ImmutableList Failures, JsonDocument? VirtualDocument)> ExecuteAsync( + public static Task<(CategoryResult Category, ImmutableList Failures, JsonDocument? VirtualDocument)> ExecuteAsync( AssessmentCategory category, JsonDocument input, JsonDocument? incomingVirtualDocument, FailureHandlingStrategy defaultFailureHandling, CancellationToken cancellationToken) { - ArgumentNullException.ThrowIfNull(category); ArgumentNullException.ThrowIfNull(input); + return ExecuteAsync(category, new EvaluationContext(input), incomingVirtualDocument, defaultFailureHandling, cancellationToken); + } + + /// + /// Executes category assessments and returns category output. A category whose + /// is false is not run and is reported as not applicable. + /// + public static async Task<(CategoryResult Category, ImmutableList Failures, JsonDocument? VirtualDocument)> ExecuteAsync( + AssessmentCategory category, + EvaluationContext context, + JsonDocument? incomingVirtualDocument, + FailureHandlingStrategy defaultFailureHandling, + CancellationToken cancellationToken) + { + ArgumentNullException.ThrowIfNull(category); + ArgumentNullException.ThrowIfNull(context); + + if (category.AppliesWhen is { } appliesWhen && !appliesWhen.Evaluate(context)) + { + return (NotApplicable(category), [], incomingVirtualDocument); + } FailureHandlingStrategy effectiveFailureHandling = category.CategoryFailureHandling ?? defaultFailureHandling; @@ -31,21 +51,28 @@ internal static class CategoryExecutor } return category.AssessmentExecutionStrategy == AssessmentExecutionStrategy.Parallel - ? await ExecuteParallelAsync(category, input, incomingVirtualDocument, effectiveFailureHandling, timeoutSource.Token) + ? await ExecuteParallelAsync(category, context, incomingVirtualDocument, effectiveFailureHandling, timeoutSource.Token) .ConfigureAwait(false) - : await ExecuteSeriesAsync(category, input, incomingVirtualDocument, effectiveFailureHandling, timeoutSource.Token) + : await ExecuteSeriesAsync(category, context, incomingVirtualDocument, effectiveFailureHandling, timeoutSource.Token) .ConfigureAwait(false); } + private static CategoryResult NotApplicable(AssessmentCategory category) + => new(category.Name, new AssessmentScore(null, null, 0d, 0d), $"Category '{category.Name}' does not apply.", []) + { + Outcome = AssessmentOutcome.NotApplicable, + Weight = category.Weight, + }; + private static async Task<(CategoryResult Category, ImmutableList Failures, JsonDocument? VirtualDocument)> ExecuteSeriesAsync( AssessmentCategory category, - JsonDocument input, + EvaluationContext context, JsonDocument? incomingVirtualDocument, FailureHandlingStrategy failureHandling, CancellationToken cancellationToken) { List assessmentResults = new(category.Assessments.Count); - List failures = new(); + List failures = []; JsonDocument? virtualDocument = incomingVirtualDocument; foreach (Assessment assessment in category.Assessments) @@ -53,7 +80,7 @@ internal static class CategoryExecutor AssessmentExecutionItemResult item = await AssessmentExecutor.ExecuteAsync( category.Name, assessment, - input, + context, virtualDocument, cancellationToken).ConfigureAwait(false); @@ -77,51 +104,63 @@ internal static class CategoryExecutor } CategoryResult categoryResult = BuildCategoryResult(category, assessmentResults); - return (categoryResult, failures.ToImmutableList(), virtualDocument); + return (categoryResult, [.. failures], virtualDocument); } private static async Task<(CategoryResult Category, ImmutableList Failures, JsonDocument? VirtualDocument)> ExecuteParallelAsync( AssessmentCategory category, - JsonDocument input, + EvaluationContext context, JsonDocument? incomingVirtualDocument, FailureHandlingStrategy failureHandling, CancellationToken cancellationToken) { - Task[] tasks = category.Assessments - .Select(assessment => AssessmentExecutor.ExecuteAsync(category.Name, assessment, input, incomingVirtualDocument, cancellationToken)) - .ToArray(); + Task[] tasks = [.. category.Assessments + .Select(assessment => AssessmentExecutor.ExecuteAsync(category.Name, assessment, context, incomingVirtualDocument, cancellationToken))]; AssessmentExecutionItemResult[] executed = await Task.WhenAll(tasks).ConfigureAwait(false); - List failures = executed - .Where(item => item.Failure is not null) - .Select(item => item.Failure!) - .ToList(); - - ImmutableList assessmentResults = executed - .Select(item => item.AssessmentResult) - .ToImmutableList(); - + List failures = [.. executed.Where(item => item.Failure is not null).Select(item => item.Failure!)]; if (failureHandling == FailureHandlingStrategy.FailFast && failures.Count > 1) { - failures = new List { failures[0] }; + failures = [failures[0]]; } - CategoryResult categoryResult = BuildCategoryResult(category, assessmentResults); - return (categoryResult, failures.ToImmutableList(), incomingVirtualDocument); + CategoryResult categoryResult = BuildCategoryResult(category, executed.Select(item => item.AssessmentResult)); + return (categoryResult, [.. failures], incomingVirtualDocument); } + /// + /// Scores the category over the assessments that applied: those reported not applicable are left out of both the + /// weighted raw score and its range, so the remaining weights are renormalised. + /// private static CategoryResult BuildCategoryResult(AssessmentCategory category, IEnumerable assessmentResults) { - ImmutableList resultList = assessmentResults.ToImmutableList(); - (double minPossibleRaw, double maxPossibleRaw) = ScoreCalculator.CalculateCategoryRawRange(category); + ImmutableList resultList = [.. assessmentResults]; + HashSet notApplicable = [.. resultList + .Select((result, index) => (result, index)) + .Where(pair => pair.result.Outcome == AssessmentOutcome.NotApplicable) + .Select(pair => pair.index)]; + + if (resultList.Count > 0 && notApplicable.Count == resultList.Count && resultList.Count == category.Assessments.Count) + { + return NotApplicable(category) with + { + Reason = $"No assessment in category '{category.Name}' applies.", + Assessments = resultList, + }; + } + + (double minPossibleRaw, double maxPossibleRaw) = ScoreCalculator.CalculateCategoryRawRange( + category.Assessments.Where((_, index) => !notApplicable.Contains(index))); + + List<(AssessmentResult Result, Assessment Assessment)> applicable = [.. resultList + .Zip(category.Assessments) + .Where(pair => pair.First.Outcome != AssessmentOutcome.NotApplicable)]; double? rawScore = null; - if (resultList.Count > 0 && resultList.All(result => result.Score.RawScore.HasValue)) + if (applicable.Count > 0 && applicable.TrueForAll(pair => pair.Result.Score.RawScore.HasValue)) { - rawScore = resultList - .Zip(category.Assessments, (result, assessment) => result.Score.RawScore!.Value * assessment.Weight) - .Sum(); + rawScore = applicable.Sum(pair => pair.Result.Score.RawScore!.Value * pair.Assessment.Weight); } double? normalizedScore = rawScore.HasValue @@ -135,6 +174,6 @@ private static CategoryResult BuildCategoryResult(AssessmentCategory category, I MaxPossibleRaw: maxPossibleRaw); string reason = $"Executed {resultList.Count} assessment(s) in category '{category.Name}'."; - return new CategoryResult(category.Name, score, reason, resultList); + return new CategoryResult(category.Name, score, reason, resultList) { Weight = category.Weight }; } } diff --git a/PanoramicData.IssueAssessment/Execution/ParallelExecutor.cs b/PanoramicData.IssueAssessment/Execution/ParallelExecutor.cs index e821728..78940d0 100644 --- a/PanoramicData.IssueAssessment/Execution/ParallelExecutor.cs +++ b/PanoramicData.IssueAssessment/Execution/ParallelExecutor.cs @@ -10,14 +10,14 @@ internal static class ParallelExecutor /// public static async Task ExecuteAsync( IAssessmentOptions options, - JsonDocument input, + EvaluationContext context, CancellationToken cancellationToken) { ArgumentNullException.ThrowIfNull(options); - ArgumentNullException.ThrowIfNull(input); + ArgumentNullException.ThrowIfNull(context); Task<(CategoryResult Category, ImmutableList Failures, JsonDocument? VirtualDocument)>[] tasks = options.Categories - .Select(category => CategoryExecutor.ExecuteAsync(category, input, null, options.FailureHandling, cancellationToken)) + .Select(category => CategoryExecutor.ExecuteAsync(category, context, null, options.FailureHandling, cancellationToken)) .ToArray(); (CategoryResult Category, ImmutableList Failures, JsonDocument? VirtualDocument)[] executed = diff --git a/PanoramicData.IssueAssessment/Execution/SeriesExecutor.cs b/PanoramicData.IssueAssessment/Execution/SeriesExecutor.cs index 35cca21..172b7b1 100644 --- a/PanoramicData.IssueAssessment/Execution/SeriesExecutor.cs +++ b/PanoramicData.IssueAssessment/Execution/SeriesExecutor.cs @@ -10,11 +10,11 @@ internal static class SeriesExecutor /// public static async Task ExecuteAsync( IAssessmentOptions options, - JsonDocument input, + EvaluationContext context, CancellationToken cancellationToken) { ArgumentNullException.ThrowIfNull(options); - ArgumentNullException.ThrowIfNull(input); + ArgumentNullException.ThrowIfNull(context); List categories = new(options.Categories.Count); List failures = []; @@ -24,7 +24,7 @@ public static async Task ExecuteAsync( { AssessmentCategory category = options.Categories[index]; (CategoryResult categoryResult, ImmutableList categoryFailures, JsonDocument? categoryVirtualDocument) = - await CategoryExecutor.ExecuteAsync(category, input, virtualDocument, options.FailureHandling, cancellationToken) + await CategoryExecutor.ExecuteAsync(category, context, virtualDocument, options.FailureHandling, cancellationToken) .ConfigureAwait(false); categories.Add(categoryResult); diff --git a/PanoramicData.IssueAssessment/GlobalUsings.cs b/PanoramicData.IssueAssessment/GlobalUsings.cs index e4c33f1..ade2409 100644 --- a/PanoramicData.IssueAssessment/GlobalUsings.cs +++ b/PanoramicData.IssueAssessment/GlobalUsings.cs @@ -5,5 +5,6 @@ global using PanoramicData.IssueAssessment.Configuration; global using PanoramicData.IssueAssessment.Core; global using PanoramicData.IssueAssessment.Results; +global using PanoramicData.IssueAssessment.Rules; global using PanoramicData.IssueAssessment.Utilities; global using PanoramicData.IssueAssessment.Validation; diff --git a/PanoramicData.IssueAssessment/IssueAssessor.cs b/PanoramicData.IssueAssessment/IssueAssessor.cs index 6052c46..5fb00e9 100644 --- a/PanoramicData.IssueAssessment/IssueAssessor.cs +++ b/PanoramicData.IssueAssessment/IssueAssessor.cs @@ -96,9 +96,10 @@ private async Task ExecuteAsync(JsonDocument issue, Cancellat ? CancellationTokenSource.CreateLinkedTokenSource(cancellationToken) : CancellationTokenSource.CreateLinkedTokenSource(cancellationToken, globalTimeoutCts.Token); + EvaluationContext context = new(issue, _options.Conditions); ExecutionResult executionResult = _options.ExecutionStrategy == CategoryExecutionStrategy.Parallel - ? await ParallelExecutor.ExecuteAsync(_options, issue, linkedCts.Token).ConfigureAwait(false) - : await SeriesExecutor.ExecuteAsync(_options, issue, linkedCts.Token).ConfigureAwait(false); + ? await ParallelExecutor.ExecuteAsync(_options, context, linkedCts.Token).ConfigureAwait(false) + : await SeriesExecutor.ExecuteAsync(_options, context, linkedCts.Token).ConfigureAwait(false); // Executors record a cancelled assessment as a timeout failure so that the global timeout yields a partial // result. Cancellation requested by the caller is not a timeout: surface it rather than return that result. @@ -110,6 +111,7 @@ private async Task ExecuteAsync(JsonDocument issue, Cancellat Failures = executionResult.Failures, Aggregations = AggregationCalculator.Calculate(executionResult.Categories), VirtualDocument = _options.CaptureVirtualDocument ? executionResult.VirtualDocument : null, + Overall = OverallCalculator.Calculate(_options, context, executionResult.Categories), }; } } diff --git a/PanoramicData.IssueAssessment/Mapping/GitHubIssueMapper.cs b/PanoramicData.IssueAssessment/Mapping/GitHubIssueMapper.cs new file mode 100644 index 0000000..10156b3 --- /dev/null +++ b/PanoramicData.IssueAssessment/Mapping/GitHubIssueMapper.cs @@ -0,0 +1,114 @@ +using System.Text.RegularExpressions; + +namespace PanoramicData.IssueAssessment.Mapping; + +/// +/// Maps an issue from the GitHub REST API (GET /repos/{owner}/{repo}/issues/{number}) to a +/// . +/// +/// +/// GitHub has no issue type, priority, component or fix version fields, so they are read the way PanoramicData.Skills' +/// GitHub quality check reads them: the type from labels (bug is a Bug; enhancement, feature, improvement or story +/// is an Improvement; anything else a Task), the priority from a priority:high-style label, components from +/// component: or area: labels, and attachments from images in the body. Fix versions are marked +/// unsupported. +/// +public static partial class GitHubIssueMapper +{ + /// + /// Maps a GitHub REST issue. + /// + /// The issue JSON. + /// The normalized issue. + /// Thrown when the JSON is not an object. + public static NormalizedIssue FromRestJson(JsonElement issue) + { + if (issue.ValueKind != JsonValueKind.Object) + { + throw new ArgumentException("A GitHub issue must be an object.", nameof(issue)); + } + + ImmutableList labels = Labels(issue); + string body = Text(issue, "body") ?? string.Empty; + + return new NormalizedIssue + { + Key = Key(issue), + Title = Text(issue, "title"), + Description = body, + IssueType = IssueType(labels), + Priority = Priority(labels), + Status = Text(issue, "state"), + Assignee = FirstAssignee(issue), + Components = Components(labels), + Labels = labels, + AttachmentsCount = MarkdownImage().Count(body) + ImageUrl().Count(body), + SupportsFixVersion = false, + DueDate = issue.TryGetProperty("milestone", out JsonElement milestone) ? Text(milestone, "due_on") : null, + Url = Text(issue, "html_url"), + Platform = "GitHub", + }; + } + + private static ImmutableList Labels(JsonElement issue) + => issue.TryGetProperty("labels", out JsonElement labels) && labels.ValueKind == JsonValueKind.Array + ? [.. labels.EnumerateArray().Select(label => label.ValueKind == JsonValueKind.String ? label.GetString() : Text(label, "name")).OfType()] + : []; + + private static string? Key(JsonElement issue) + => issue.TryGetProperty("number", out JsonElement number) && number.ValueKind == JsonValueKind.Number ? $"#{number.GetRawText()}" : null; + + private static string? FirstAssignee(JsonElement issue) + => issue.TryGetProperty("assignees", out JsonElement assignees) && assignees.ValueKind == JsonValueKind.Array && assignees.GetArrayLength() > 0 + ? Text(assignees[0], "login") + : null; + + private static string IssueType(ImmutableList labels) + { + if (labels.Exists(label => label.Equals("bug", StringComparison.OrdinalIgnoreCase))) + { + return "Bug"; + } + + return labels.Exists(label => FeatureLabel().IsMatch(label)) ? "Improvement" : "Task"; + } + + private static string Priority(ImmutableList labels) + { + Match? match = labels.Select(label => PriorityLabel().Match(label)).FirstOrDefault(found => found.Success); + if (match is null) + { + return "Unknown"; + } + + string value = match.Groups[1].Value.ToLowerInvariant(); + return char.ToUpperInvariant(value[0]) + value[1..]; + } + + private static ImmutableList Components(ImmutableList labels) + => [.. labels + .Select(label => ComponentLabel().Match(label)) + .Where(match => match.Success) + .Select(match => match.Groups[2].Value.Trim()) + .Distinct(StringComparer.Ordinal)]; + + private static string? Text(JsonElement element, string property) + => element.ValueKind == JsonValueKind.Object && element.TryGetProperty(property, out JsonElement value) && value.ValueKind == JsonValueKind.String + ? value.GetString() + : null; + + [GeneratedRegex("enhancement|feature|improvement|story", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, 1000)] + private static partial Regex FeatureLabel(); + + [GeneratedRegex(@"priority[:\-]?(critical|high|major|medium|minor|low|trivial)", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, 1000)] + private static partial Regex PriorityLabel(); + + [GeneratedRegex(@"^(component|area)[:\-](.+)$", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, 1000)] + private static partial Regex ComponentLabel(); + + [GeneratedRegex(@"!\[[^\]]*\]\(([^)]+)\)", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, 1000)] + private static partial Regex MarkdownImage(); + + [GeneratedRegex(@"https?://\S+\.(png|jpg|jpeg|gif|webp|bmp)", RegexOptions.IgnoreCase | RegexOptions.CultureInvariant, 1000)] + private static partial Regex ImageUrl(); +} diff --git a/PanoramicData.IssueAssessment/Mapping/JiraIssueMapper.cs b/PanoramicData.IssueAssessment/Mapping/JiraIssueMapper.cs new file mode 100644 index 0000000..abba341 --- /dev/null +++ b/PanoramicData.IssueAssessment/Mapping/JiraIssueMapper.cs @@ -0,0 +1,196 @@ +using System.Collections.Frozen; +using System.Text; + +namespace PanoramicData.IssueAssessment.Mapping; + +/// +/// Maps an issue from the Jira REST API (GET /rest/api/2/issue/{key} or /rest/api/3/issue/{key}) to a +/// . +/// +/// +/// API v2 returns the description as wiki markup, which is used as it is. API v3 returns Atlassian Document Format, +/// which is flattened to wiki-style text: headings become h2. lines and list items * or # lines, +/// so section rules read both the same way. +/// +public static class JiraIssueMapper +{ + // ADF list types, and the wiki marker each nesting level adds. + private static readonly FrozenDictionary ListMarkers = new Dictionary(StringComparer.Ordinal) + { + ["bulletList"] = "*", + ["orderedList"] = "#", + }.ToFrozenDictionary(StringComparer.Ordinal); + + // ADF nodes written as one line of inline text. + private static readonly FrozenSet LineBlockTypes = FrozenSet.Create( + StringComparer.Ordinal, + "paragraph", "blockquote", "panel", "text", "mention", "inlineCard", "hardBreak"); + + /// + /// Maps a Jira REST issue. + /// + /// The issue JSON: an object with key and fields. + /// The normalized issue. + /// Thrown when the JSON has no fields object. + public static NormalizedIssue FromRestJson(JsonElement issue) + => FromRestJson(issue, null); + + /// + /// Maps a Jira REST issue, with a link built from . + /// + /// The issue JSON: an object with key and fields. + /// The Jira site, such as https://example.atlassian.net, or null for no link. + /// The normalized issue. + /// Thrown when the JSON has no fields object. + public static NormalizedIssue FromRestJson(JsonElement issue, Uri? baseUrl) + { + JsonElement fields = Fields(issue); + string? key = Text(issue, "key"); + return new NormalizedIssue + { + Key = key, + Project = Text(fields, "project", "key"), + Title = Text(fields, "summary"), + Description = Description(fields), + IssueType = Text(fields, "issuetype", "name"), + Priority = Text(fields, "priority", "name"), + Status = Text(fields, "status", "name"), + Assignee = Text(fields, "assignee", "displayName"), + Components = Names(fields, "components"), + Labels = Strings(fields, "labels"), + FixVersions = Names(fields, "fixVersions"), + AttachmentsCount = ArrayLength(fields, "attachment"), + DueDate = Text(fields, "duedate"), + Url = BrowseUrl(baseUrl, key), + Platform = "Jira", + }; + } + + private static JsonElement Fields(JsonElement issue) + { + if (issue.ValueKind == JsonValueKind.Object + && issue.TryGetProperty("fields", out JsonElement fields) + && fields.ValueKind == JsonValueKind.Object) + { + return fields; + } + + throw new ArgumentException("A Jira issue must be an object with a 'fields' object.", nameof(issue)); + } + + private static int ArrayLength(JsonElement element, string name) + => element.TryGetProperty(name, out JsonElement array) && array.ValueKind == JsonValueKind.Array ? array.GetArrayLength() : 0; + + private static string? BrowseUrl(Uri? baseUrl, string? key) + => baseUrl is null || key is null ? null : new Uri(baseUrl, $"browse/{key}").ToString(); + + /// + /// Flattens an Atlassian Document Format node to wiki-style text. + /// + /// An ADF document or node. + /// The text. + public static string FlattenAdf(JsonElement document) + { + StringBuilder text = new(); + AppendBlock(document, text, string.Empty); + return text.ToString().TrimEnd(); + } + + private static string? Description(JsonElement fields) + => fields.TryGetProperty("description", out JsonElement description) ? description.ValueKind switch + { + JsonValueKind.String => description.GetString(), + JsonValueKind.Object => FlattenAdf(description), + _ => null, + } : null; + + private static void AppendBlock(JsonElement node, StringBuilder text, string listPrefix) + { + string type = Text(node, "type") ?? string.Empty; + if (ListMarkers.TryGetValue(type, out string? listMarker)) + { + AppendList(node, text, listPrefix + listMarker); + } + else if (type == "heading") + { + text.Append(CultureInfo.InvariantCulture, $"h{AdfHeadingLevel(node)}. ").Append(Inline(node)).Append('\n'); + } + else if (type == "codeBlock") + { + text.Append("{code}\n").Append(Inline(node)).Append("\n{code}\n"); + } + else if (LineBlockTypes.Contains(type)) + { + text.Append(Inline(node)).Append('\n'); + } + else + { + foreach (JsonElement child in Children(node)) + { + AppendBlock(child, text, listPrefix); + } + } + } + + private static void AppendList(JsonElement list, StringBuilder text, string marker) + { + foreach (JsonElement item in Children(list)) + { + AppendListItem(item, text, marker); + } + } + + private static int AdfHeadingLevel(JsonElement heading) + => heading.TryGetProperty("attrs", out JsonElement attrs) && attrs.TryGetProperty("level", out JsonElement value) && value.TryGetInt32(out int level) + ? level + : 2; + + private static void AppendListItem(JsonElement item, StringBuilder text, string marker) + { + bool first = true; + foreach (JsonElement child in Children(item)) + { + if (Text(child, "type") is "bulletList" or "orderedList") + { + AppendBlock(child, text, marker); + } + else + { + text.Append(first ? marker + " " : string.Empty).Append(Inline(child)).Append('\n'); + first = false; + } + } + } + + private static string Inline(JsonElement node) => Text(node, "type") switch + { + "text" => Text(node, "text") ?? string.Empty, + "hardBreak" => "\n", + "mention" or "emoji" => node.TryGetProperty("attrs", out JsonElement attrs) ? Text(attrs, "text") ?? string.Empty : string.Empty, + "inlineCard" => node.TryGetProperty("attrs", out JsonElement attrs) ? Text(attrs, "url") ?? string.Empty : string.Empty, + _ => string.Concat(Children(node).Select(Inline)), + }; + + private static JsonElement[] Children(JsonElement node) + => node.ValueKind == JsonValueKind.Object && node.TryGetProperty("content", out JsonElement content) && content.ValueKind == JsonValueKind.Array + ? [.. content.EnumerateArray()] + : []; + + private static string? Text(JsonElement element, string property) + => element.ValueKind == JsonValueKind.Object && element.TryGetProperty(property, out JsonElement value) && value.ValueKind == JsonValueKind.String + ? value.GetString() + : null; + + private static string? Text(JsonElement element, string property, string nested) + => element.ValueKind == JsonValueKind.Object && element.TryGetProperty(property, out JsonElement value) ? Text(value, nested) : null; + + private static ImmutableList Names(JsonElement fields, string property) + => fields.TryGetProperty(property, out JsonElement values) && values.ValueKind == JsonValueKind.Array + ? [.. values.EnumerateArray().Select(value => Text(value, "name")).OfType()] + : []; + + private static ImmutableList Strings(JsonElement fields, string property) + => fields.TryGetProperty(property, out JsonElement values) && values.ValueKind == JsonValueKind.Array + ? [.. values.EnumerateArray().Where(value => value.ValueKind == JsonValueKind.String).Select(value => value.GetString()!)] + : []; +} diff --git a/PanoramicData.IssueAssessment/Mapping/NormalizedIssue.cs b/PanoramicData.IssueAssessment/Mapping/NormalizedIssue.cs new file mode 100644 index 0000000..073df6f --- /dev/null +++ b/PanoramicData.IssueAssessment/Mapping/NormalizedIssue.cs @@ -0,0 +1,136 @@ +using System.Text.Json.Nodes; +using System.Text.RegularExpressions; + +namespace PanoramicData.IssueAssessment.Mapping; + +/// +/// A ticket from any system in the shape the built-in categories and rubrics read. Build one with a mapper +/// (, , ) or directly, +/// then assess . +/// +/// +/// JSON property names are camelCase ($.title, $.issueType, $.components). Unset values are left +/// out, so they read as missing; list properties are always written, so an empty list reads as present but empty. +/// +public sealed partial record NormalizedIssue +{ + /// Gets the issue key, such as MS-1234 or #42. + public string? Key { get; init; } + + /// Gets the project key: set explicitly, or taken from a key such as MS-1234. + public string? Project { get; init; } + + /// Gets the one-line summary. + public string? Title { get; init; } + + /// Gets the description, as text (Jira wiki markup or Markdown). + public string? Description { get; init; } + + /// Gets the issue type, such as Bug, Story, Epic or Change Request. + public string? IssueType { get; init; } + + /// Gets the priority name. + public string? Priority { get; init; } + + /// Gets the workflow status name. + public string? Status { get; init; } + + /// Gets the assignee's name or login. + public string? Assignee { get; init; } + + /// Gets the assignment state the built-in ownership category reads (Assigned, InProgress, Owned). + public string? AssigneeState { get; init; } + + /// Gets the component names. + public ImmutableList Components { get; init; } = []; + + /// Gets the labels. + public ImmutableList Labels { get; init; } = []; + + /// Gets the fix version names. + public ImmutableList FixVersions { get; init; } = []; + + /// Gets the number of attachments. + public int AttachmentsCount { get; init; } + + /// Gets whether the system has fix versions at all; null when not known. + public bool? SupportsFixVersion { get; init; } + + /// Gets whether the project uses components; null to let the rubric decide. + public bool? SupportsComponents { get; init; } + + /// Gets the due date, as text. + public string? DueDate { get; init; } + + /// Gets acceptance criteria held in a field of their own, if the system has one. + public string? AcceptanceCriteria { get; init; } + + /// Gets reproduction steps held in a field of their own, if the system has one. + public string? ReproductionSteps { get; init; } + + /// Gets a link to the issue. + public string? Url { get; init; } + + /// Gets the system the issue came from, such as Jira, GitHub or ServiceNow. + public string? Platform { get; init; } + + /// + /// The project key from an issue key such as MS-1234, or null. + /// + public static string? ProjectFromKey(string? key) + => key is null ? null : ProjectKeyPattern().Match(key) is { Success: true } match ? match.Groups[1].Value : null; + + /// + /// Writes the issue as the JSON document assessments read. + /// + /// A new document; the caller disposes it. + public JsonDocument ToJsonDocument() => JsonDocument.Parse(ToJsonObject().ToJsonString()); + + /// + /// Writes the issue as a mutable JSON object, for callers that add fields of their own. + /// + public JsonObject ToJsonObject() + { + JsonObject issue = []; + Add(issue, "key", Key); + Add(issue, "project", Project ?? ProjectFromKey(Key)); + Add(issue, "title", Title); + Add(issue, "description", Description); + Add(issue, "issueType", IssueType); + Add(issue, "priority", Priority); + Add(issue, "status", Status); + Add(issue, "assignee", Assignee); + Add(issue, "assigneeState", AssigneeState); + issue["components"] = new JsonArray([.. Components.Select(value => (JsonNode?)value)]); + issue["labels"] = new JsonArray([.. Labels.Select(value => (JsonNode?)value)]); + issue["fixVersions"] = new JsonArray([.. FixVersions.Select(value => (JsonNode?)value)]); + issue["attachmentsCount"] = AttachmentsCount; + if (SupportsFixVersion is { } supportsFixVersion) + { + issue["supportsFixVersion"] = supportsFixVersion; + } + + if (SupportsComponents is { } supportsComponents) + { + issue["supportsComponents"] = supportsComponents; + } + + Add(issue, "dueDate", DueDate); + Add(issue, "acceptanceCriteria", AcceptanceCriteria); + Add(issue, "reproductionSteps", ReproductionSteps); + Add(issue, "url", Url); + Add(issue, "platform", Platform); + return issue; + } + + private static void Add(JsonObject issue, string name, string? value) + { + if (value is not null) + { + issue[name] = value; + } + } + + [GeneratedRegex(@"^([A-Za-z][A-Za-z0-9]+)-\d+$", RegexOptions.CultureInvariant, 1000)] + private static partial Regex ProjectKeyPattern(); +} diff --git a/PanoramicData.IssueAssessment/Mapping/ServiceNowRecordMapper.cs b/PanoramicData.IssueAssessment/Mapping/ServiceNowRecordMapper.cs new file mode 100644 index 0000000..4dca3eb --- /dev/null +++ b/PanoramicData.IssueAssessment/Mapping/ServiceNowRecordMapper.cs @@ -0,0 +1,66 @@ +namespace PanoramicData.IssueAssessment.Mapping; + +/// +/// Maps a record from the ServiceNow Table API (GET /api/now/table/{table}/{sys_id}) to a +/// . +/// +/// +/// Accepts the response with or without its result wrapper. Reference fields may be plain strings or, with +/// sysparm_display_value=true, objects whose display_value is used. ServiceNow has no fix versions or +/// components, so both are marked unsupported. +/// +public static class ServiceNowRecordMapper +{ + /// + /// Maps a ServiceNow Table API record. + /// + /// The record JSON. + /// The normalized issue. + /// Thrown when the JSON is not an object. + public static NormalizedIssue FromTableJson(JsonElement record) + { + if (record.ValueKind == JsonValueKind.Object && record.TryGetProperty("result", out JsonElement result)) + { + record = result; + } + + if (record.ValueKind != JsonValueKind.Object) + { + throw new ArgumentException("A ServiceNow record must be an object.", nameof(record)); + } + + return new NormalizedIssue + { + Key = Value(record, "number"), + Title = Value(record, "short_description"), + Description = Value(record, "description"), + IssueType = Value(record, "sys_class_name"), + Priority = Value(record, "priority"), + Status = Value(record, "state"), + Assignee = Value(record, "assigned_to"), + SupportsFixVersion = false, + SupportsComponents = false, + DueDate = Value(record, "due_date"), + Platform = "ServiceNow", + }; + } + + /// + /// A field's value: a non-blank string, or a reference object's display value. + /// + private static string? Value(JsonElement record, string property) + { + if (!record.TryGetProperty(property, out JsonElement value)) + { + return null; + } + + string? text = value.ValueKind switch + { + JsonValueKind.String => value.GetString(), + JsonValueKind.Object when value.TryGetProperty("display_value", out JsonElement display) && display.ValueKind == JsonValueKind.String => display.GetString(), + _ => null, + }; + return string.IsNullOrWhiteSpace(text) ? null : text; + } +} diff --git a/PanoramicData.IssueAssessment/PanoramicData.IssueAssessment.csproj b/PanoramicData.IssueAssessment/PanoramicData.IssueAssessment.csproj index d9dc2ef..4744a92 100644 --- a/PanoramicData.IssueAssessment/PanoramicData.IssueAssessment.csproj +++ b/PanoramicData.IssueAssessment/PanoramicData.IssueAssessment.csproj @@ -12,12 +12,12 @@ git MIT Icon.png - Schema-agnostic issue and ticket quality assessment for .NET: score Jira, GitHub, ServiceNow or any JSON ticket with composable, weighted assessments, built-in quality categories, series or parallel execution, aggregation and optional caching. + Schema-agnostic issue and ticket quality assessment for .NET: score Jira, GitHub, ServiceNow or any JSON ticket with composable, weighted assessments, built-in quality categories, JSON rubrics with grades, caps and recommendations, Jira/GitHub/ServiceNow mappers, series or parallel execution, aggregation and optional caching. false true snupkg true - Issue;Ticket;Quality;Assessment;Scoring;Jira;GitHub;ServiceNow;JSON;PanoramicData + Issue;Ticket;Quality;Assessment;Scoring;Rubric;Jira;GitHub;ServiceNow;JSON;PanoramicData README.md @@ -25,6 +25,10 @@ + + + + diff --git a/PanoramicData.IssueAssessment/Results/AssessmentOutcome.cs b/PanoramicData.IssueAssessment/Results/AssessmentOutcome.cs new file mode 100644 index 0000000..2493771 --- /dev/null +++ b/PanoramicData.IssueAssessment/Results/AssessmentOutcome.cs @@ -0,0 +1,55 @@ +namespace PanoramicData.IssueAssessment.Results; + +/// +/// How an assessment or category came out. +/// +public enum AssessmentOutcome +{ + /// Scored its maximum. + Passed, + + /// Scored between its minimum and maximum. + Partial, + + /// Scored its minimum. A missing value is a failure, not an error. + Failed, + + /// Did not apply to this issue, so it was not scored and does not count. + NotApplicable, + + /// Could not be scored: it timed out, threw, or produced no score. + Error, +} + +/// +/// Derives an from a score. +/// +public static class AssessmentOutcomes +{ + private const double Tolerance = 1e-9; + + /// + /// The outcome a score represents: without a score or after a timeout or + /// exception; otherwise passed, partial or failed by the normalized score. + /// + /// The score. + /// The failure reason, if any. is + /// a scored failure, not an error. + /// The outcome. + public static AssessmentOutcome FromScore(AssessmentScore score, AssessmentFailureReason? failureReason) + { + ArgumentNullException.ThrowIfNull(score); + if (failureReason is { } reason && reason != AssessmentFailureReason.JsonPathNotFound) + { + return AssessmentOutcome.Error; + } + + return score.NormalizedScore switch + { + null => AssessmentOutcome.Error, + >= 1d - Tolerance => AssessmentOutcome.Passed, + <= Tolerance => AssessmentOutcome.Failed, + _ => AssessmentOutcome.Partial, + }; + } +} diff --git a/PanoramicData.IssueAssessment/Results/AssessmentOverall.cs b/PanoramicData.IssueAssessment/Results/AssessmentOverall.cs new file mode 100644 index 0000000..1e00772 --- /dev/null +++ b/PanoramicData.IssueAssessment/Results/AssessmentOverall.cs @@ -0,0 +1,33 @@ +namespace PanoramicData.IssueAssessment.Results; + +/// +/// The overall verdict on an issue. +/// +/// The weighted average of the applicable categories' normalized scores, in [0, 1], after score +/// caps; null when no category was scored. +/// The same average before score caps. +/// The name of the highest grade band the score reaches, or null without bands or a score. +/// The score caps that applied, in rubric order. +/// The value of every named condition in the rubric. +/// The value each classification in the rubric assigned. +public sealed record AssessmentOverall( + double? Score, + double? UncappedScore, + string? Grade, + ImmutableList Caps, + ImmutableDictionary Flags, + ImmutableDictionary Classifications) +{ + /// + /// Gets the score on a 0-100 scale, rounded half to even; null without a score. + /// + public int? Score100 => Score is { } score ? (int)Math.Round(score * 100d, MidpointRounding.ToEven) : null; +} + +/// +/// A score cap that applied to an issue. +/// +/// The cap's name, for example the mandatory element that was missing. +/// The cap's group, for example "mandatory", or null. +/// The highest score the cap allows, in [0, 1]. +public sealed record AppliedScoreCap(string Name, string? Group, double Cap); diff --git a/PanoramicData.IssueAssessment/Results/AssessmentReport.cs b/PanoramicData.IssueAssessment/Results/AssessmentReport.cs new file mode 100644 index 0000000..5d70772 --- /dev/null +++ b/PanoramicData.IssueAssessment/Results/AssessmentReport.cs @@ -0,0 +1,155 @@ +using PanoramicData.IssueAssessment.Serialization; + +namespace PanoramicData.IssueAssessment.Results; + +/// +/// A plain, serialisable summary of an : no exceptions or JSON documents, only values, +/// so it can be written as JSON, stored, or passed to another process. +/// +public sealed record AssessmentReport +{ + /// Gets the overall score on a 0-100 scale, after caps; null when nothing was scored. + public int? Score { get; init; } + + /// Gets the overall score in [0, 1], after caps. + public double? NormalizedScore { get; init; } + + /// Gets the grade, if the rubric defines grade bands. + public string? Grade { get; init; } + + /// Gets every non-empty finding, in rubric order. + public required ImmutableList Findings { get; init; } + + /// Gets every recommendation, in rubric order, without duplicates. + public required ImmutableList Recommendations { get; init; } + + /// Gets the score caps that applied. + public required ImmutableList Caps { get; init; } + + /// Gets the value of each named condition. + public required ImmutableDictionary Flags { get; init; } + + /// Gets the value of each classification. + public required ImmutableDictionary Classifications { get; init; } + + /// Gets each category's result. + public required ImmutableList Categories { get; init; } + + /// Gets the failures, with exception messages in place of exceptions. + public required ImmutableList Failures { get; init; } + + /// + /// Builds the report for . + /// + /// The assessment result. + /// The report. + public static AssessmentReport From(IAssessmentResult result) + { + ArgumentNullException.ThrowIfNull(result); + AssessmentOverall overall = result.Overall ?? NoOverall; + List assessments = [.. result.Categories.SelectMany(category => category.Assessments)]; + + return new AssessmentReport + { + Score = overall.Score100, + NormalizedScore = overall.Score, + Grade = overall.Grade, + Findings = [.. assessments.Select(assessment => assessment.ExplanationText).Where(text => !string.IsNullOrWhiteSpace(text))], + Recommendations = [.. assessments.Select(assessment => assessment.Recommendation).OfType().Distinct(StringComparer.Ordinal)], + Caps = overall.Caps, + Flags = overall.Flags, + Classifications = overall.Classifications, + Categories = [.. result.Categories.Select(CategoryReport.From)], + Failures = [.. result.Failures.Select(FailureReport.From)], + }; + } + + /// + /// Stands in for the overall verdict of a result that has none: no score, grade, caps, flags or classifications. + /// + private static readonly AssessmentOverall NoOverall = new( + null, + null, + null, + [], + ImmutableDictionary.Empty, + ImmutableDictionary.Empty); + + /// + /// Writes the report as indented JSON with camelCase names and enum names as strings. + /// + /// The JSON. + public string ToJson() => JsonSerializer.Serialize(this, AssessmentRubric.SerializerOptions); +} + +/// +/// One category in an . +/// +/// The category name. +/// How the category came out. +/// The category's weight in the overall score. +/// The category's normalized score, if scored. +/// The category's assessments. +public sealed record CategoryReport( + string Name, + AssessmentOutcome Outcome, + double Weight, + double? NormalizedScore, + ImmutableList Assessments) +{ + internal static CategoryReport From(CategoryResult category) + => new( + category.CategoryName, + category.Outcome, + category.Weight, + category.Score.NormalizedScore, + [.. category.Assessments.Select(assessment => new AssessmentReportItem( + assessment.AssessmentName, + assessment.Outcome, + assessment.Score.RawScore, + assessment.Score.NormalizedScore, + assessment.ExplanationText, + assessment.Recommendation, + assessment.FailureReason, + assessment.InnerException?.Message))]); +} + +/// +/// One assessment in an . +/// +/// The assessment name. +/// How the assessment came out. +/// The raw score, if scored. +/// The normalized score, if scored. +/// The finding (explanation). +/// The recommendation, if any. +/// The failure reason, if any. +/// The exception message, if one was thrown. +public sealed record AssessmentReportItem( + string Name, + AssessmentOutcome Outcome, + double? RawScore, + double? NormalizedScore, + string Finding, + string? Recommendation, + AssessmentFailureReason? FailureReason, + string? Error); + +/// +/// One failure in an . +/// +/// The category name. +/// The assessment name. +/// The failure reason. +/// The failure message. +/// The exception message, if one was thrown. +public sealed record FailureReport( + string CategoryName, + string AssessmentName, + AssessmentFailureReason Reason, + string Message, + string? Error) +{ + internal static FailureReport From(AssessmentFailure failure) + => new(failure.CategoryName, failure.AssessmentName, failure.Reason, failure.Message, failure.InnerException?.Message); +} diff --git a/PanoramicData.IssueAssessment/Results/AssessmentResult.cs b/PanoramicData.IssueAssessment/Results/AssessmentResult.cs index 58036b1..7aab610 100644 --- a/PanoramicData.IssueAssessment/Results/AssessmentResult.cs +++ b/PanoramicData.IssueAssessment/Results/AssessmentResult.cs @@ -13,7 +13,18 @@ public sealed record AssessmentResult( AssessmentScore Score, string ExplanationText, AssessmentFailureReason? FailureReason, - Exception? InnerException); + Exception? InnerException) +{ + /// + /// Gets how the assessment came out: passed, partly met, failed, not applicable, or an error. + /// + public AssessmentOutcome Outcome { get; init; } = AssessmentOutcomes.FromScore(Score, FailureReason); + + /// + /// Gets what to do to improve the result, when the assessment did not pass and one is configured. + /// + public string? Recommendation { get; init; } +} /// /// Represents the aggregated result for one category. @@ -26,7 +37,19 @@ public sealed record CategoryResult( string CategoryName, AssessmentScore Score, string Reason, - ImmutableList Assessments); + ImmutableList Assessments) +{ + /// + /// Gets how the category came out, from its normalized score; not applicable when the category did not apply or + /// none of its assessments did. + /// + public AssessmentOutcome Outcome { get; init; } = AssessmentOutcomes.FromScore(Score, null); + + /// + /// Gets the category's weight in the overall score. + /// + public double Weight { get; init; } = 1d; +} /// /// Represents score information including normalization bounds. @@ -54,7 +77,13 @@ public sealed record AssessmentExecutionResult( string? ExplanationText, AssessmentFailureReason? FailureReason, Exception? InnerException, - JsonDocument? VirtualExtensionOutput); + JsonDocument? VirtualExtensionOutput) +{ + /// + /// Gets what the definition recommends, overriding . + /// + public string? Recommendation { get; init; } +} /// /// Represents a failure captured during execution. @@ -146,4 +175,7 @@ public sealed record IssueAssessmentResult : IAssessmentResult /// Gets optional captured virtual document. /// public JsonDocument? VirtualDocument { get; init; } + + /// + public AssessmentOverall? Overall { get; init; } } diff --git a/PanoramicData.IssueAssessment/Rubrics/BuiltInRubrics.cs b/PanoramicData.IssueAssessment/Rubrics/BuiltInRubrics.cs new file mode 100644 index 0000000..5ac5d63 --- /dev/null +++ b/PanoramicData.IssueAssessment/Rubrics/BuiltInRubrics.cs @@ -0,0 +1,50 @@ +using PanoramicData.IssueAssessment.Serialization; + +namespace PanoramicData.IssueAssessment.Rubrics; + +/// +/// Rubrics that ship with the package. +/// +public static class BuiltInRubrics +{ + private const string TicketQualityResource = "PanoramicData.IssueAssessment.Rubrics.panoramicdata-ticket-quality.json"; + + /// + /// Gets the Panoramic Data ticket quality rubric: the specification-completeness check Panoramic Data applies to Jira + /// and GitHub tickets before work or testing starts. + /// + /// + /// + /// It reads a . Bugs, stories, tasks and change requests are scored on component, + /// description, testability, steps to reproduce, current versus expected behaviour, fix version and environment, + /// attachments, documentation, certification questions, UI consistency, business case and (change requests only) a + /// rollback plan. Epics have their own rubric: design, breakdown, scope and success criteria. + /// + /// + /// A bug without numbered steps to reproduce, a change request without a rollback plan, or acceptance criteria + /// stated under more than one heading cap the score at 0.79, below EXCELLENT. Grades are EXCELLENT (0.8), GOOD + /// (0.6), ADEQUATE (0.5), NEEDS IMPROVEMENT (0.3) and POOR. Flags report whether the ticket is a bug, an epic, an + /// investigation or a UI ticket, and the TestAutomationType classification says whether it suits AI testing + /// (Ai, Hybrid, Human or None). + /// + /// + /// Each call returns a new copy, so callers can change it with with without affecting others. + /// + /// + public static AssessmentOptions PanoramicDataTicketQuality => Load(TicketQualityResource); + + /// + /// Gets the JSON of the Panoramic Data ticket quality rubric, for inspection or as a starting point for another. + /// + public static string PanoramicDataTicketQualityJson => Read(TicketQualityResource); + + private static AssessmentOptions Load(string resource) => AssessmentRubric.Load(Read(resource)); + + private static string Read(string resource) + { + using Stream stream = typeof(BuiltInRubrics).Assembly.GetManifestResourceStream(resource) + ?? throw new InvalidOperationException($"The built-in rubric '{resource}' is missing from the assembly."); + using StreamReader reader = new(stream); + return reader.ReadToEnd(); + } +} diff --git a/PanoramicData.IssueAssessment/Rubrics/panoramicdata-ticket-quality.json b/PanoramicData.IssueAssessment/Rubrics/panoramicdata-ticket-quality.json new file mode 100644 index 0000000..457b8df --- /dev/null +++ b/PanoramicData.IssueAssessment/Rubrics/panoramicdata-ticket-quality.json @@ -0,0 +1,560 @@ +{ + // Panoramic Data ticket quality rubric. + // Ported from PanoramicData.Skills .github/skills/_common/IssueQualityCore.ps1 (Yoda), which defined it first: + // factors, weights, patterns, findings and recommendations are transcribed from there, so both score a ticket the + // same. Epics use their own rubric; Change Requests add a rollback plan; Bugs need steps to reproduce. + "executionStrategy": "series", + "conditions": { + "isBug": { "type": "fieldMatches", "path": "$.issueType", "patterns": [ "Bug|Defect" ] }, + "isEpic": { "type": "fieldEquals", "path": "$.issueType", "values": [ "Epic" ] }, + "isChangeRequest": { "type": "fieldEquals", "path": "$.issueType", "values": [ "Change Request" ] }, + "isInvestigation": { + "type": "all", + "conditions": [ + { "type": "not", "condition": { "type": "ref", "name": "isEpic" } }, + { + "type": "any", + "conditions": [ + { + "type": "all", + "conditions": [ + { "type": "not", "condition": { "type": "ref", "name": "isBug" } }, + { + "type": "textMatches", + "source": { "paths": [ "$.title", "$.description" ] }, + "patterns": [ "\\binvestigate\\b", "\\bresearch\\b", "\\bspike\\b", "\\banalysis\\b", "\\bexplore\\b", "\\bfeasibility\\b", "\\bproof\\s+of\\s+concept\\b", "\\bpoc\\b", "\\bstudy\\b", "\\bevaluate\\b", "\\bassess\\b" ] + } + ] + }, + { "type": "fieldEquals", "path": "$.labels", "values": [ "investigation", "research", "spike" ] } + ] + } + ] + }, + "supportsComponents": { + "type": "any", + "conditions": [ + { "type": "fieldEquals", "path": "$.supportsComponents", "values": [ "true" ] }, + { + "type": "all", + "conditions": [ + { "type": "not", "condition": { "type": "fieldPresent", "path": "$.supportsComponents" } }, + { "type": "not", "condition": { "type": "fieldEquals", "path": "$.project", "values": [ "PS" ] } } + ] + } + ] + }, + "supportsFixVersion": { "type": "not", "condition": { "type": "fieldEquals", "path": "$.supportsFixVersion", "values": [ "false" ] } }, + "shouldHaveFixVersion": { "type": "fieldEquals", "path": "$.status", "values": [ "Ready for Test", "In Test", "Ready for Release", "Closed", "Done", "Resolved" ] }, + "hasFixVersion": { "type": "fieldPresent", "path": "$.fixVersions" }, + "hasEnvironmentMention": { "type": "textMatches", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "test2|alpha|beta|staging|production|environment" ] }, + "nonUiComponentsOnly": { + "type": "fieldMatches", + "path": "$.components", + "requireAll": true, + "patterns": [ "API", "API\\ Client", "Backend", "Database", "Data", "Connections", "Workers", "Worker", "Scheduler", "Schedules", "Infrastructure", "Security", "Authentication", "Migration", "Unit\\ tests", "Performance", "Configuration", "Logging" ] + }, + "uiConsistencyApplies": { + "type": "all", + "conditions": [ + { "type": "not", "condition": { "type": "ref", "name": "isBug" } }, + { "type": "not", "condition": { "type": "ref", "name": "isInvestigation" } }, + { "type": "fieldMatches", "path": "$.issueType", "patterns": [ "Feature|Improvement|Story|Enhancement" ] } + ] + }, + "isUiTicket": { + "type": "all", + "conditions": [ + { "type": "ref", "name": "uiConsistencyApplies" }, + { + "type": "any", + "conditions": [ + { + "type": "all", + "conditions": [ + { "type": "not", "condition": { "type": "ref", "name": "nonUiComponentsOnly" } }, + { "type": "fieldMatches", "path": "$.components", "patterns": [ "\\bUI\\b", "\\bFrontend\\b", "\\bFront-End\\b", "\\bWeb\\b", "\\bPortal\\b", "\\bDashboard\\b" ] } + ] + }, + { "type": "fieldMatches", "path": "$.labels", "patterns": [ "^ui$", "^frontend$", "^front-end$", "^web-ui$", "^ux$", "^visual$", "^design$" ] }, + { + "type": "all", + "conditions": [ + { "type": "not", "condition": { "type": "ref", "name": "nonUiComponentsOnly" } }, + { + "type": "measure", + "atLeast": 2, + "of": { + "type": "patternMatchCount", + "source": { "paths": [ "$.title", "$.description" ] }, + "patterns": [ "\\bUI\\b", "\\buser\\s+interface\\b", "\\bscreen\\b", "\\bdialog\\b", "\\bmodal\\b", "\\bform\\b", "\\bbutton\\b", "\\bpage\\b", "\\bpanel\\b", "\\bdashboard\\b", "\\bwidget\\b", "\\bview\\b", "\\blayout\\b", "\\btoolbar\\b", "\\bmenu\\b", "\\bdropdown\\b", "\\btab\\b", "\\bnavigation\\b", "\\bfront[- ]?end\\b", "\\bstyle\\b", "\\bCSS\\b", "\\btheme\\b", "\\bicon\\b", "\\btooltip\\b", "\\bpopup\\b", "\\bpopover\\b", "\\bgrid\\b", "\\btable\\b", "\\bcolumn\\b", "\\bheader\\b", "\\bfooter\\b", "\\bsidebar\\b" ] + } + } + ] + } + ] + } + ] + }, + "isCliRelated": { "type": "textMatches", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "magicsuite\\s+cli", "ms\\s+cli", "command[\\s-]?line", "msmagic", "cli\\s+command", "--\\w+", "invoke[\\s-]command" ] }, + "hasCliLabel": { "type": "fieldMatches", "path": "$.labels", "patterns": [ "CLI|Command" ] }, + "hasCliComponent": { "type": "fieldMatches", "path": "$.components", "patterns": [ "CLI" ] }, + "hasUiIndicators": { "type": "textMatches", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "button", "click", "dialog", "modal", "dropdown", "menu", "screen", "page", "form", "input\\s+field", "checkbox", "radio", "tab", "panel", "dashboard", "report\\s*studio", "ui\\s+", "user\\s+interface", "browser", "blazor" ] }, + "hasUiLabel": { "type": "fieldMatches", "path": "$.labels", "patterns": [ "^(UI|Blazor|ReportStudio)$" ] } + }, + "categories": [ + { + "name": "Ticket Quality", + "description": "Specification completeness of a bug, story, task or change request.", + "assessmentExecutionStrategy": "series", + "appliesWhen": { "type": "not", "condition": { "type": "ref", "name": "isEpic" } }, + "assessments": [ + { + "name": "Component", + "description": "The component the ticket concerns is identified.", + "weight": 14, + "appliesWhen": { "type": "ref", "name": "supportsComponents" }, + "notApplicableExplanation": "○ Component: Not applicable for the {$.project} project", + "definition": { + "type": "tiered", + "measures": { "components": { "type": "itemCount", "path": "$.components" } }, + "tiers": [ + { "when": { "type": "measure", "measure": "components", "atLeast": 1 }, "score": 100, "finding": "✓ Component: {$.components[0]}" }, + { "when": { "type": "fieldMatches", "path": "$.labels", "patterns": [ "DataMagic|ReportMagic|AlertMagic|Admin|Connect|CLI|Docs" ] }, "score": 70, "finding": "⚠ Component: Via label ({$.labels~DataMagic|ReportMagic|AlertMagic|Admin|Connect|CLI|Docs})", "recommendation": "Set component field" }, + { "when": { "type": "textMatches", "source": { "paths": [ "$.title" ] }, "patterns": [ "DataMagic|ReportMagic|AlertMagic|Admin|Connect|CLI|Docs|Report\\s*Studio" ] }, "score": 50, "finding": "⚠ Component: Mentioned in summary only", "recommendation": "Set component field explicitly" }, + { "score": 0, "finding": "✗ Component: Not identified", "recommendation": "Add component to issue" } + ] + } + }, + { + "name": "Description", + "description": "The description says enough.", + "weight": 14, + "definition": { + "type": "tiered", + "measures": { "words": { "type": "wordCount", "source": { "paths": [ "$.description" ] } } }, + "tiers": [ + { "when": { "type": "measure", "measure": "words", "atLeast": 100 }, "score": 100, "finding": "✓ Description: {words} words (comprehensive)" }, + { "when": { "type": "measure", "measure": "words", "atLeast": 50 }, "score": 80, "finding": "✓ Description: {words} words (adequate)" }, + { "when": { "type": "measure", "measure": "words", "atLeast": 20 }, "score": 50, "finding": "⚠ Description: {words} words (brief)", "recommendation": "Add more context to description" }, + { "when": { "type": "measure", "measure": "words", "atLeast": 1 }, "score": 25, "finding": "⚠ Description: {words} words (minimal)", "recommendation": "Expand description with details" }, + { "score": 0, "finding": "✗ Description: Empty", "recommendation": "Add description" } + ] + } + }, + { + "name": "Testability", + "description": "Pass/fail criteria, or for an investigation its scope and deliverables, are stated.", + "weight": 16, + "definition": { + "type": "tiered", + "measures": { + "scope": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "scope:", "h[1-6]\\.[^\\r\\n]*scope", "\\*\\*[^\\r\\n]*scope", "questions?\\s+to\\s+(answer|investigate)", "investigate\\s+(whether|if|how|why)", "determine\\s+(whether|if|how|why)", "find\\s+out", "understand\\s+(how|why|what)" ] }, + "deliverables": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "deliverable", "findings", "recommendation", "report", "document", "conclusion", "outcome", "result\\s+of\\s+(investigation|research|analysis)" ] }, + "criteria": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "should\\s+(be|display|show|return|have|not)", "must\\s+(be|display|show|return|have|not)", "verify\\s+that", "acceptance\\s+criteria", "h[23]\\.\\s*acceptance\\s+criteria", "expected\\s+(result|behavior|behaviour|outcome)", "pass\\s+criteria", "success\\s+criteria" ] }, + "criteriaHeadings": { "type": "criteriaHeadingCount", "source": { "paths": [ "$.description" ] } } + }, + "tiers": [ + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isInvestigation" }, { "type": "measure", "measure": "scope", "atLeast": 2 }, { "type": "measure", "measure": "deliverables", "atLeast": 1 } ] }, "score": 100, "finding": "✓ Investigation Clarity: Clear scope and deliverables" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isInvestigation" }, { "type": "measure", "measure": "scope", "atLeast": 1 }, { "type": "measure", "measure": "deliverables", "atLeast": 1 } ] }, "score": 80, "finding": "✓ Investigation Clarity: Scope and deliverables defined" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isInvestigation" }, { "type": "measure", "measure": "scope", "atLeast": 1 } ] }, "score": 50, "finding": "⚠ Investigation Clarity: Missing deliverables", "recommendation": "Define investigation deliverables (findings document, recommendation, report)" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isInvestigation" }, { "type": "measure", "measure": "deliverables", "atLeast": 1 } ] }, "score": 50, "finding": "⚠ Investigation Clarity: Missing scope", "recommendation": "Define investigation scope and questions to answer" }, + { "when": { "type": "ref", "name": "isInvestigation" }, "score": 20, "finding": "✗ Investigation Clarity: No scope or deliverables", "recommendation": "Add investigation scope, questions to answer, and expected deliverables" }, + { "when": { "type": "measure", "measure": "criteria", "atLeast": 3 }, "score": 100, "finding": "✓ Testability: Clear criteria found" }, + { "when": { "type": "all", "conditions": [ { "type": "measure", "measure": "criteria", "atLeast": 1 }, { "type": "measure", "measure": "criteriaHeadings", "lessThan": 1 } ] }, "score": 60, "finding": "⚠ Testability: Some criteria found", "recommendation": "Add explicit pass/fail criteria" }, + { "when": { "type": "measure", "measure": "criteria", "atLeast": 1 }, "score": 60, "finding": "⚠ Testability: Some criteria found", "recommendation": "Make the existing acceptance criteria explicit pass/fail checks, rather than adding another section" }, + { "score": 20, "finding": "✗ Testability: No clear criteria", "recommendation": "Define acceptance criteria" } + ] + } + }, + { + "name": "StepsToReproduce", + "description": "A bug has numbered steps to reproduce under their own heading.", + "weight": 12, + "definition": { + "type": "tiered", + "measures": { + "steps": { "type": "occurrenceCount", "source": { "paths": [ "$.description" ], "section": "(?:reproduc|\\brepro\\b)" }, "pattern": "(?m)^[ \\t]*(?:#(?!#)|\\d+[.)])[ \\t]+\\S" }, + "listedElsewhere": { "type": "occurrenceCount", "source": { "paths": [ "$.description" ] }, "pattern": "(?m)^[ \\t]*(?:#(?!#)|\\d+[.)])[ \\t]+\\S" }, + "sectionText": { "type": "alphanumericCount", "source": { "paths": [ "$.description" ], "section": "(?:reproduc|\\brepro\\b)" } } + }, + "tiers": [ + { "when": { "type": "ref", "name": "isInvestigation" }, "score": 100, "finding": "○ Steps: Not required for investigations" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "not", "condition": { "type": "sectionPresent", "source": { "paths": [ "$.description" ], "section": "(?:reproduc|\\brepro\\b)" } } }, { "type": "measure", "measure": "listedElsewhere", "atLeast": 2 } ] }, "score": 0, "finding": "✗ Steps to Reproduce: No steps-to-reproduce section (a numbered list was found, but not under a Steps to Reproduce heading)", "recommendation": "Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "not", "condition": { "type": "sectionPresent", "source": { "paths": [ "$.description" ], "section": "(?:reproduc|\\brepro\\b)" } } } ] }, "score": 0, "finding": "✗ Steps to Reproduce: No steps-to-reproduce section", "recommendation": "Add an 'h2. Steps to Reproduce' section with numbered steps (Jira '# step' or '1. step')" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "measure", "measure": "steps", "atLeast": 2 } ] }, "score": 100, "finding": "✓ Steps to Reproduce: {steps} numbered steps" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "textMatches", "source": { "paths": [ "$.description" ], "section": "(?:reproduc|\\brepro\\b)" }, "patterns": [ "(?i)\\b(?:not|never)\\s+(?:been\\s+)?(?:reliably\\s+|consistently\\s+)?reproduc|\\b(?:cannot|can't|could\\s+not|couldn't|unable\\s+to)\\s+(?:be\\s+)?(?:reliably\\s+|consistently\\s+)?reproduc|\\bintermittent" ] }, { "type": "measure", "measure": "sectionText", "atLeast": 60 } ] }, "score": 60, "finding": "⚠ Steps to Reproduce: Stated as not reproducible on demand, with observed conditions", "recommendation": "Add numbered steps once a reliable reproduction is found; until then the ticket stays at GOOD" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "textMatches", "source": { "paths": [ "$.description" ], "section": "(?:reproduc|\\brepro\\b)" }, "patterns": [ "(?i)\\b(?:not|never)\\s+(?:been\\s+)?(?:reliably\\s+|consistently\\s+)?reproduc|\\b(?:cannot|can't|could\\s+not|couldn't|unable\\s+to)\\s+(?:be\\s+)?(?:reliably\\s+|consistently\\s+)?reproduc|\\bintermittent" ] } ] }, "score": 30, "finding": "⚠ Steps to Reproduce: Stated as not reproducible, with no observed conditions", "recommendation": "Describe when it was seen: environment, frequency, time, data and what was being done" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "measure", "measure": "steps", "atLeast": 1 } ] }, "score": 60, "finding": "⚠ Steps to Reproduce: Only one numbered step", "recommendation": "Break the reproduction into numbered steps from a known starting state" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "any", "conditions": [ { "type": "placeholder", "source": { "paths": [ "$.description" ], "section": "(?:reproduc|\\brepro\\b)" } }, { "type": "measure", "measure": "sectionText", "lessThan": 15 } ] } ] }, "score": 0, "finding": "✗ Steps to Reproduce: Heading present but the section is empty or a placeholder", "recommendation": "Fill in the steps to reproduce - a heading over 'TBD' does not say how to see the bug" }, + { "when": { "type": "ref", "name": "isBug" }, "score": 40, "finding": "⚠ Steps to Reproduce: Section present but the steps are not numbered", "recommendation": "Rewrite the steps to reproduce as a numbered list" }, + { "when": { "type": "textMatches", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "steps?\\s*(to\\s+reproduce|:)", "(?m)^\\s*\\d+\\.\\s+", "(?m)^\\s*#\\s+", "(?m)^\\s*-\\s+", "h[23]\\.\\s*steps?\\s*(to\\s+reproduce)?", "how\\s+to\\s+reproduce", "reproduction\\s+steps", "repro\\s+steps" ] }, "score": 100, "finding": "✓ Steps: Workflow documented" }, + { "score": 100, "finding": "○ Steps: Not required for this type" } + ] + } + }, + { + "name": "CurrentVsExpected", + "description": "Current and expected behaviour are documented, or for an investigation its background.", + "weight": 12, + "definition": { + "type": "tiered", + "measures": { + "context": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "background:", "h[1-6]\\.[^\\r\\n]*background", "context:", "h[1-6]\\.[^\\r\\n]*context", "\\*\\*[^\\r\\n]*(background|context)", "motivation:", "why\\s+it\\s+matters", "why\\s+(we|this)", "reason\\s+for", "because" ] }, + "current": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "current(ly)?(\\s+behavior|\\s+behaviour)?:", "h[23]\\.\\s*current(\\s*/\\s*actual)?\\s*(behavior|behaviour)", "\\*\\*current\\s*(behavior|behaviour)\\*\\*", "\\*current(\\s*/\\s*actual)?\\s*(behavior|behaviour)?\\*", "\\*actual\\s*(behavior|behaviour)?\\*", "actual(\\s+behavior|\\s+behaviour)?:", "what\\s+happens:", "observed:", "bug:", "issue:", "problem:" ] }, + "expected": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "expected(\\s+behavior|\\s+behaviour)?:", "h[23]\\.\\s*expected\\s*(behavior|behaviour)", "\\*\\*expected\\s*(behavior|behaviour)\\*\\*", "\\*expected\\s*(behavior|behaviour)?\\*", "should\\s+be:", "desired:", "correct\\s+behavior:", "fix:", "solution:", "h2\\.\\s*acceptance\\s+criteria", "h3\\.\\s*acceptance\\s+criteria", "acceptance\\s+criteria:", "\\*\\*acceptance\\s+criteria\\*\\*" ] } + }, + "tiers": [ + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isInvestigation" }, { "type": "measure", "measure": "context", "atLeast": 2 } ] }, "score": 100, "finding": "✓ Investigation Context: Well documented" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isInvestigation" }, { "type": "measure", "measure": "context", "atLeast": 1 } ] }, "score": 70, "finding": "⚠ Investigation Context: Partial context", "recommendation": "Add more background/motivation for the investigation" }, + { "when": { "type": "ref", "name": "isInvestigation" }, "score": 30, "finding": "⚠ Investigation Context: Missing context", "recommendation": "Add background and motivation for why this investigation is needed" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "measure", "measure": "current", "atLeast": 1 }, { "type": "measure", "measure": "expected", "atLeast": 1 } ] }, "score": 100, "finding": "✓ Current vs Expected: Both documented" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "measure", "measure": "current", "atLeast": 1 } ] }, "score": 50, "finding": "⚠ Current vs Expected: Missing expected behavior", "recommendation": "Document both current and expected behavior" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "isBug" }, { "type": "measure", "measure": "expected", "atLeast": 1 } ] }, "score": 50, "finding": "⚠ Current vs Expected: Missing current behavior", "recommendation": "Document both current and expected behavior" }, + { "when": { "type": "ref", "name": "isBug" }, "score": 0, "finding": "✗ Current vs Expected: Neither documented", "recommendation": "Add current behavior and expected behavior sections" }, + { "when": { "type": "measure", "measure": "expected", "atLeast": 1 }, "score": 100, "finding": "✓ Expected Behavior: Documented" }, + { "score": 40, "finding": "⚠ Expected Behavior: Not explicit", "recommendation": "Clarify expected outcome" } + ] + } + }, + { + "name": "InvestigationTimebox", + "description": "An investigation states how long it may take. Recommendation only; it carries no weight.", + "weight": 0, + "appliesWhen": { "type": "ref", "name": "isInvestigation" }, + "definition": { + "type": "tiered", + "tiers": [ + { "when": { "type": "textMatches", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "timebox", "time-?box", "\\d+\\s*(hour|day|week)s?", "maximum\\s+time", "time\\s+limit" ] }, "score": 100, "finding": "" }, + { "score": 0, "finding": "", "recommendation": "Consider adding a timebox for the investigation" } + ] + } + }, + { + "name": "EnvironmentVersion", + "description": "The fix version and the environment are stated.", + "weight": 7, + "definition": { + "type": "tiered", + "tiers": [ + { "when": { "type": "not", "condition": { "type": "ref", "name": "supportsFixVersion" } }, "score": 100, "finding": "○ Fix Version: Not applicable on {$.platform}" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "hasFixVersion" }, { "type": "ref", "name": "hasEnvironmentMention" } ] }, "score": 100, "finding": "✓ Fix Version: {$.fixVersions[0]}" }, + { "when": { "type": "ref", "name": "hasFixVersion" }, "score": 70, "finding": "✓ Fix Version: {$.fixVersions[0]}" }, + { "when": { "type": "all", "conditions": [ { "type": "ref", "name": "shouldHaveFixVersion" }, { "type": "ref", "name": "hasEnvironmentMention" } ] }, "score": 30, "finding": "⚠ Fix Version: Not specified (expected for '{$.status}' status)", "recommendation": "Add fix version" }, + { "when": { "type": "ref", "name": "shouldHaveFixVersion" }, "score": 0, "finding": "⚠ Fix Version: Not specified (expected for '{$.status}' status)", "recommendation": "Add fix version" }, + { "score": 100, "finding": "○ Fix Version: Not required yet (status: {$.status})" } + ] + } + }, + { + "name": "Attachments", + "description": "A bug carries a screenshot or log.", + "weight": 3, + "definition": { + "type": "tiered", + "measures": { "attachments": { "type": "number", "path": "$.attachmentsCount" } }, + "tiers": [ + { "when": { "type": "measure", "measure": "attachments", "atLeast": 1 }, "score": 100, "finding": "✓ Attachments: {attachments} item(s)" }, + { "when": { "type": "textMatches", "source": { "paths": [ "$.description" ] }, "patterns": [ "!\\[|screenshot|image|http.*\\.(png|jpg|gif)" ] }, "score": 70, "finding": "✓ Attachments: Inline images found" }, + { "when": { "type": "ref", "name": "isBug" }, "score": 0, "finding": "⚠ Attachments: None (recommended for bugs)", "recommendation": "Add screenshot or error log" }, + { "score": 100, "finding": "○ Attachments: None (optional for non-bugs)" } + ] + } + }, + { + "name": "Documentation", + "description": "Documentation or help updates are referenced.", + "weight": 13, + "definition": { + "type": "tiered", + "measures": { "references": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "documentation|docs?", "help\\s+(file|topic|section)", "user\\s+guide", "knowledge\\s+base|kb\\s+article", "wiki|confluence", "readme|readme\\.md", "api\\s+(reference|documentation|docs)", "manual", "guide|tutorial" ] } }, + "tiers": [ + { "when": { "type": "measure", "measure": "references", "atLeast": 2 }, "score": 100, "finding": "✓ Documentation: Multiple references found" }, + { "when": { "type": "measure", "measure": "references", "atLeast": 1 }, "score": 70, "finding": "✓ Documentation: Referenced", "recommendation": "Consider adding more documentation updates" }, + { "score": 0, "finding": "⚠ Documentation: Not referenced", "recommendation": "Update documentation/help text for this change" } + ] + } + }, + { + "name": "Certifications", + "description": "A new feature has a certification exam question.", + "weight": 4, + "definition": { + "type": "tiered", + "tiers": [ + { "when": { "type": "ref", "name": "isBug" }, "score": 100, "finding": "○ Certifications: Not applicable for bugs" }, + { "when": { "type": "textMatches", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "certifica?tion|exam|certification\\s+question", "cert\\s+question", "assessment\\s+question", "test\\s+question" ] }, "score": 100, "finding": "✓ Certifications: Exam question(s) created" }, + { "score": 0, "finding": "○ Certifications: No exam question created", "recommendation": "Consider creating a certification exam question for this new feature" } + ] + } + }, + { + "name": "UIConsistency", + "description": "A UI change references the look and feel of similar features, or says there are none.", + "weight": 13, + "definition": { + "type": "tiered", + "measures": { "references": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "consistent\\s+(with|look|feel|style|design|behaviour|behavior)", "look\\s+and\\s+feel", "match(es|ing)?\\s+(the|existing|current|other|similar)", "similar\\s+(to|feature|screen|page|dialog|panel|view|component)", "same\\s+(style|approach|pattern|design|way)\\s+(as|used)", "in\\s+line\\s+with", "align(ed)?\\s+with", "follows?\\s+(the|existing|current)\\s+(pattern|convention|style|design)", "precedent", "no\\s+similar\\s+(feature|screen|page|component|view)", "no\\s+existing\\s+(feature|screen|page|component|view)", "first\\s+of\\s+(its|this)\\s+kind", "new\\s+pattern", "no\\s+precedent", "h[23]\\.\\s*UI\\s+Consistency", "\\*\\*UI\\s+Consistency\\*\\*", "UI\\s+consistency:" ] } }, + "tiers": [ + { "when": { "type": "not", "condition": { "type": "ref", "name": "uiConsistencyApplies" } }, "score": 100, "finding": "○ UI Consistency: Not applicable for {$.issueType}" }, + { "when": { "type": "not", "condition": { "type": "ref", "name": "isUiTicket" } }, "score": 100, "finding": "○ UI Consistency: Not a UI ticket" }, + { "when": { "type": "measure", "measure": "references", "atLeast": 2 }, "score": 100, "finding": "✓ UI Consistency: Look & feel references with similar features" }, + { "when": { "type": "measure", "measure": "references", "atLeast": 1 }, "score": 70, "finding": "⚠ UI Consistency: Partial reference to similar features", "recommendation": "Expand UI consistency details - reference specific similar features or state there are none" }, + { "score": 0, "finding": "✗ UI Consistency: No reference to look & feel or similar features", "recommendation": "Add UI consistency details or explicitly state no similar features exist" } + ] + } + }, + { + "name": "BusinessCase", + "description": "The value, impact or cost of not doing the work is stated.", + "weight": 12, + "definition": { + "type": "tiered", + "measures": { "justification": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "business\\s+case", "business\\s+value", "business\\s+justification", "business\\s+impact", "business\\s+benefit", "cost\\s+of\\s+(not\\s+doing|inaction|delay)", "return\\s+on\\s+investment", "\\bROI\\b", "value\\s+proposition", "revenue\\s+impact", "customer\\s+impact", "user\\s+impact", "compliance\\s+requirement", "regulatory\\s+requirement", "risk\\s+(reduction|mitigation)", "operational\\s+(benefit|efficiency|saving)", "h[23]\\.\\s*(Business\\s+Case|Business\\s+Value|Business\\s+Justification|Motivation|Rationale)", "\\*\\*(Business\\s+Case|Business\\s+Value|Justification|Motivation|Rationale)\\*\\*", "why\\s+(this|we)\\s+(is|are|need|should|must)", "justif(y|ication|ied)", "rationale" ] } }, + "tiers": [ + { "when": { "type": "ref", "name": "isBug" }, "score": 100, "finding": "○ Business Case: Not required for bugs" }, + { "when": { "type": "ref", "name": "isInvestigation" }, "score": 100, "finding": "○ Business Case: Not required for investigations" }, + { "when": { "type": "measure", "measure": "justification", "atLeast": 3 }, "score": 100, "finding": "✓ Business Case: Clear justification provided" }, + { "when": { "type": "measure", "measure": "justification", "atLeast": 2 }, "score": 80, "finding": "✓ Business Case: Justification referenced" }, + { "when": { "type": "measure", "measure": "justification", "atLeast": 1 }, "score": 50, "finding": "⚠ Business Case: Brief justification", "recommendation": "Expand business case with impact, value, or cost of inaction" }, + { "score": 0, "finding": "✗ Business Case: Not provided", "recommendation": "Add a business case explaining the value, impact, or cost of not doing this work" } + ] + } + }, + { + "name": "RollbackPlan", + "description": "A change request documents how the change is undone.", + "weight": 34, + "appliesWhen": { "type": "ref", "name": "isChangeRequest" }, + "definition": { + "type": "tiered", + "measures": { "planText": { "type": "alphanumericCount", "source": { "paths": [ "$.description" ], "section": "(?:rollback|mitigation)" } } }, + "tiers": [ + { "when": { "type": "not", "condition": { "type": "sectionPresent", "source": { "paths": [ "$.description" ], "section": "(?:rollback|mitigation)" } } }, "score": 0, "finding": "✗ Rollback Plan: No rollback or mitigation section", "recommendation": "Add a 'Rollback plan' section: what is captured before the change, what reverses it, and what is not reversible" }, + { "when": { "type": "placeholder", "source": { "paths": [ "$.description" ], "section": "(?:rollback|mitigation)" } }, "score": 0, "finding": "✗ Rollback Plan: Heading present but the section is empty or a placeholder", "recommendation": "Fill in the rollback plan - a heading over 'TBD' does not describe how the change is undone" }, + { "when": { "type": "measure", "measure": "planText", "lessThan": 15 }, "score": 0, "finding": "✗ Rollback Plan: Section too short to describe anything", "recommendation": "Expand the rollback plan to state what reverses the change and what cannot be reversed" }, + { "when": { "type": "measure", "measure": "planText", "lessThan": 120 }, "score": 60, "finding": "⚠ Rollback Plan: Present but brief", "recommendation": "Expand the rollback plan: name what is captured beforehand, and say what is irreversible and what compensates for it" }, + { "score": 100, "finding": "✓ Rollback Plan: Documented" } + ] + } + }, + { + "name": "CriteriaDuplication", + "description": "Acceptance criteria are stated under one heading only. It carries no weight; duplication caps the score instead.", + "weight": 0, + "definition": { + "type": "tiered", + "measures": { "headings": { "type": "criteriaHeadingCount", "source": { "paths": [ "$.description" ] } } }, + "tiers": [ + { "when": { "type": "measure", "measure": "headings", "atLeast": 2 }, "score": 0, "finding": "✗ Duplication: acceptance criteria stated under {headings} headings ({headings:text}), so EXCELLENT is unreachable (score capped at 79)", "recommendation": "State the acceptance criteria once, under one heading, and remove the others" }, + { "score": 100, "finding": "" } + ] + } + } + ] + }, + { + "name": "Epic Quality", + "description": "Specification completeness of an epic: design, breakdown, scope and success criteria.", + "assessmentExecutionStrategy": "series", + "appliesWhen": { "type": "ref", "name": "isEpic" }, + "assessments": [ + { + "name": "EpicComponent", + "description": "The component the epic concerns is set.", + "weight": 10, + "appliesWhen": { "type": "ref", "name": "supportsComponents" }, + "notApplicableExplanation": "○ Component: Not applicable for the {$.project} project", + "definition": { + "type": "tiered", + "measures": { "components": { "type": "itemCount", "path": "$.components" } }, + "tiers": [ + { "when": { "type": "measure", "measure": "components", "atLeast": 1 }, "score": 100, "finding": "✓ Component: {$.components[0]}" }, + { "score": 0, "finding": "✗ Component: Not set", "recommendation": "Set the component field" } + ] + } + }, + { + "name": "EpicDescription", + "description": "The epic's description states the problem and the solution.", + "weight": 18, + "definition": { + "type": "tiered", + "measures": { "words": { "type": "wordCount", "source": { "paths": [ "$.description" ] } } }, + "tiers": [ + { "when": { "type": "measure", "measure": "words", "atLeast": 200 }, "score": 100, "finding": "✓ Description: {words} words (comprehensive)" }, + { "when": { "type": "measure", "measure": "words", "atLeast": 100 }, "score": 80, "finding": "✓ Description: {words} words (good)" }, + { "when": { "type": "measure", "measure": "words", "atLeast": 50 }, "score": 50, "finding": "⚠ Description: {words} words (brief for an epic)", "recommendation": "Expand description with problem statement and solution overview" }, + { "when": { "type": "measure", "measure": "words", "atLeast": 1 }, "score": 25, "finding": "⚠ Description: {words} words (minimal)", "recommendation": "Add comprehensive problem statement and solution overview" }, + { "score": 0, "finding": "✗ Description: Empty", "recommendation": "Add description with problem statement and solution overview" } + ] + } + }, + { + "name": "Architecture", + "description": "The technical design is described.", + "weight": 13, + "definition": { + "type": "tiered", + "measures": { "design": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "architect(ure|ural)", "design\\s+(overview|decision|approach)", "technical\\s+(design|approach|solution|overview)", "flow\\s+diagram", "sequence\\s+diagram", "data\\s+flow", "system\\s+design", "component\\s+diagram", "integration\\s+point", "API\\s+(design|contract|endpoint)", "schema\\s+(design|change)", "database\\s+(design|change|migration)", "h[23]\\.\\s*(Technical|Architecture|Design)", "\\*\\*(Technical|Architecture|Design)\\s+(Design|Overview|Approach)\\*\\*" ] } }, + "tiers": [ + { "when": { "type": "measure", "measure": "design", "atLeast": 3 }, "score": 100, "finding": "✓ Architecture: Comprehensive technical design" }, + { "when": { "type": "measure", "measure": "design", "atLeast": 2 }, "score": 80, "finding": "✓ Architecture: Technical approach documented" }, + { "when": { "type": "measure", "measure": "design", "atLeast": 1 }, "score": 50, "finding": "⚠ Architecture: Brief technical mention", "recommendation": "Add technical design details (approach, diagrams, integration points)" }, + { "score": 0, "finding": "✗ Architecture: No technical design", "recommendation": "Add architecture/technical design section" } + ] + } + }, + { + "name": "Decomposition", + "description": "The work is broken down into child tickets or phases.", + "weight": 13, + "definition": { + "type": "tiered", + "measures": { "breakdown": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "child\\s+(ticket|issue|task|story)", "sub-task", "breakdown", "decompos(e|ed|ition)", "phase\\s*\\d", "step\\s*\\d", "work\\s*(stream|package|item)", "MS-\\d{4,5}", "implementation\\s+order", "dependencies", "h[23]\\.\\s*(Breakdown|Decomposition|Child|Sub-tasks|Implementation\\s+Order|Work\\s+Items)", "\\*\\*(Breakdown|Decomposition|Child|Sub-tasks|Implementation\\s+Order)\\*\\*" ] } }, + "tiers": [ + { "when": { "type": "measure", "measure": "breakdown", "atLeast": 3 }, "score": 100, "finding": "✓ Decomposition: Clear work breakdown" }, + { "when": { "type": "measure", "measure": "breakdown", "atLeast": 2 }, "score": 80, "finding": "✓ Decomposition: Work items referenced" }, + { "when": { "type": "measure", "measure": "breakdown", "atLeast": 1 }, "score": 50, "finding": "⚠ Decomposition: Partial breakdown", "recommendation": "Add clear work breakdown with child tickets or phases" }, + { "score": 0, "finding": "✗ Decomposition: No work breakdown", "recommendation": "Decompose into child tickets or define implementation phases" } + ] + } + }, + { + "name": "ScopeDefinition", + "description": "What is in and out of scope is defined.", + "weight": 13, + "definition": { + "type": "tiered", + "measures": { "scope": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "in\\s+scope", "out\\s+of\\s+scope", "scope\\s+(of|definition|boundary|boundaries)", "not\\s+included", "excluded", "boundaries", "goal(s)?", "objective(s)?", "non-goal", "h[23]\\.\\s*(Scope|Goal|Objective|Boundaries|Non-Goal|Out of Scope)", "\\*\\*(Scope|Goal|Objective|In Scope|Out of Scope|Non-Goal)\\*\\*" ] } }, + "tiers": [ + { "when": { "type": "measure", "measure": "scope", "atLeast": 3 }, "score": 100, "finding": "✓ Scope: Well-defined boundaries" }, + { "when": { "type": "measure", "measure": "scope", "atLeast": 2 }, "score": 80, "finding": "✓ Scope: Goals documented" }, + { "when": { "type": "measure", "measure": "scope", "atLeast": 1 }, "score": 50, "finding": "⚠ Scope: Partially defined", "recommendation": "Clarify what is in scope and out of scope" }, + { "score": 0, "finding": "✗ Scope: Not defined", "recommendation": "Add scope boundaries (in scope, out of scope, goals)" } + ] + } + }, + { + "name": "AcceptanceCriteria", + "description": "The epic's success criteria and definition of done are stated.", + "weight": 13, + "definition": { + "type": "tiered", + "measures": { "criteria": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "acceptance\\s+criteria", "success\\s+criteria", "done\\s+when", "definition\\s+of\\s+done", "complete\\s+when", "considered\\s+complete", "epic\\s+(is\\s+)?complete", "milestone", "deliverable(s)?", "h[23]\\.\\s*(Acceptance|Success|Done|Definition of Done|Deliverables)", "\\*\\*(Acceptance|Success)\\s+Criteria\\*\\*" ] } }, + "tiers": [ + { "when": { "type": "measure", "measure": "criteria", "atLeast": 2 }, "score": 100, "finding": "✓ Acceptance Criteria: High-level success criteria defined" }, + { "when": { "type": "measure", "measure": "criteria", "atLeast": 1 }, "score": 70, "finding": "⚠ Acceptance Criteria: Partially defined", "recommendation": "Add clearer definition of done for this epic" }, + { "score": 0, "finding": "✗ Acceptance Criteria: Not defined", "recommendation": "Add high-level acceptance/success criteria and definition of done" } + ] + } + }, + { + "name": "FixVersion", + "description": "The epic's fix version is set once it is expected.", + "weight": 5, + "definition": { + "type": "tiered", + "tiers": [ + { "when": { "type": "not", "condition": { "type": "ref", "name": "supportsFixVersion" } }, "score": 100, "finding": "○ Fix Version: Not applicable on {$.platform}" }, + { "when": { "type": "ref", "name": "hasFixVersion" }, "score": 100, "finding": "✓ Fix Version: {$.fixVersions[0]}" }, + { "when": { "type": "ref", "name": "shouldHaveFixVersion" }, "score": 0, "finding": "⚠ Fix Version: Not set (expected for '{$.status}' status)", "recommendation": "Add fix version" }, + { "score": 100, "finding": "○ Fix Version: Not required yet (status: {$.status})" } + ] + } + }, + { + "name": "EpicAttachments", + "description": "The epic has diagrams or mockups.", + "weight": 5, + "definition": { + "type": "tiered", + "measures": { "attachments": { "type": "number", "path": "$.attachmentsCount" } }, + "tiers": [ + { "when": { "type": "measure", "measure": "attachments", "atLeast": 1 }, "score": 100, "finding": "✓ Attachments: {attachments} item(s)" }, + { "when": { "type": "textMatches", "source": { "paths": [ "$.description" ] }, "patterns": [ "!\\[|screenshot|image|diagram|http.*\\.(png|jpg|gif)" ] }, "score": 70, "finding": "✓ Attachments: Inline images/diagrams found" }, + { "score": 30, "finding": "⚠ Attachments: None (diagrams/mockups recommended for epics)", "recommendation": "Consider adding architecture diagrams or mockups" } + ] + } + }, + { + "name": "EpicBusinessCase", + "description": "Why the epic is valuable is stated.", + "weight": 10, + "definition": { + "type": "tiered", + "measures": { "justification": { "type": "patternMatchCount", "source": { "paths": [ "$.title", "$.description" ] }, "patterns": [ "business\\s+case", "business\\s+value", "business\\s+justification", "business\\s+impact", "cost\\s+of\\s+(not\\s+doing|inaction|delay)", "return\\s+on\\s+investment", "\\bROI\\b", "value\\s+proposition", "strategic\\s+(value|benefit|alignment)", "revenue\\s+impact", "customer\\s+impact", "compliance\\s+requirement", "risk\\s+(reduction|mitigation)", "operational\\s+(benefit|efficiency|saving)", "h[23]\\.\\s*(Business\\s+Case|Business\\s+Value|Business\\s+Justification|Motivation|Rationale)", "\\*\\*(Business\\s+Case|Business\\s+Value|Justification|Motivation|Rationale)\\*\\*", "justif(y|ication|ied)", "rationale", "why\\s+(this|we)\\s+(is|are|need|should|must)" ] } }, + "tiers": [ + { "when": { "type": "measure", "measure": "justification", "atLeast": 3 }, "score": 100, "finding": "✓ Business Case: Clear justification provided" }, + { "when": { "type": "measure", "measure": "justification", "atLeast": 2 }, "score": 80, "finding": "✓ Business Case: Justification referenced" }, + { "when": { "type": "measure", "measure": "justification", "atLeast": 1 }, "score": 50, "finding": "⚠ Business Case: Brief justification", "recommendation": "Expand business case with impact, value, or cost of inaction" }, + { "score": 0, "finding": "✗ Business Case: Not provided", "recommendation": "Add business case explaining why this epic is valuable" } + ] + } + } + ] + } + ], + "scoreCaps": [ + { + "name": "Criteria duplication", + "group": "duplication", + "cap": 0.79, + "when": { "type": "assessmentScore", "assessment": "CriteriaDuplication", "lessThan": 100 } + }, + { + "name": "Steps to reproduce", + "group": "mandatory", + "cap": 0.79, + "when": { + "type": "all", + "conditions": [ + { "type": "ref", "name": "isBug" }, + { "type": "not", "condition": { "type": "ref", "name": "isInvestigation" } }, + { "type": "assessmentScore", "assessment": "StepsToReproduce", "lessThan": 100 } + ] + } + }, + { + "name": "Rollback plan", + "group": "mandatory", + "cap": 0.79, + "when": { "type": "assessmentScore", "assessment": "RollbackPlan", "lessThan": 1 } + } + ], + "gradeBands": [ + { "name": "EXCELLENT", "minimumScore": 0.8 }, + { "name": "GOOD", "minimumScore": 0.6 }, + { "name": "ADEQUATE", "minimumScore": 0.5 }, + { "name": "NEEDS IMPROVEMENT", "minimumScore": 0.3 }, + { "name": "POOR", "minimumScore": 0 } + ], + "classifications": { + "TestAutomationType": [ + { "when": { "type": "ref", "name": "isEpic" }, "value": "None" }, + { + "when": { + "type": "all", + "conditions": [ + { "type": "any", "conditions": [ { "type": "ref", "name": "isCliRelated" }, { "type": "ref", "name": "hasCliLabel" }, { "type": "ref", "name": "hasCliComponent" } ] }, + { "type": "not", "condition": { "type": "ref", "name": "hasUiIndicators" } }, + { "type": "not", "condition": { "type": "ref", "name": "hasUiLabel" } } + ] + }, + "value": "Ai" + }, + { + "when": { + "type": "all", + "conditions": [ + { "type": "any", "conditions": [ { "type": "ref", "name": "hasUiIndicators" }, { "type": "ref", "name": "hasUiLabel" } ] }, + { "type": "any", "conditions": [ { "type": "ref", "name": "isCliRelated" }, { "type": "ref", "name": "hasCliLabel" } ] } + ] + }, + "value": "Hybrid" + }, + { "when": { "type": "any", "conditions": [ { "type": "ref", "name": "hasUiIndicators" }, { "type": "ref", "name": "hasUiLabel" } ] }, "value": "Human" }, + { "value": "None" } + ] + } +} diff --git a/PanoramicData.IssueAssessment/Rules/Conditions.cs b/PanoramicData.IssueAssessment/Rules/Conditions.cs new file mode 100644 index 0000000..7a51b11 --- /dev/null +++ b/PanoramicData.IssueAssessment/Rules/Conditions.cs @@ -0,0 +1,277 @@ +using System.Text.Json.Serialization; +using PanoramicData.IssueAssessment.Text; + +namespace PanoramicData.IssueAssessment.Rules; + +/// +/// A true-or-false question about an issue, such as "is it a bug?" or "does its description have a steps section?". +/// Conditions decide which tier an assessment scores, whether an assessment or category applies at all, which score +/// caps apply and how an issue is classified. +/// +[JsonPolymorphic(TypeDiscriminatorPropertyName = "type")] +[JsonDerivedType(typeof(AllCondition), "all")] +[JsonDerivedType(typeof(AnyCondition), "any")] +[JsonDerivedType(typeof(NotCondition), "not")] +[JsonDerivedType(typeof(ConditionReference), "ref")] +[JsonDerivedType(typeof(MeasureCondition), "measure")] +[JsonDerivedType(typeof(TextMatchesCondition), "textMatches")] +[JsonDerivedType(typeof(FieldMatchesCondition), "fieldMatches")] +[JsonDerivedType(typeof(FieldEqualsCondition), "fieldEquals")] +[JsonDerivedType(typeof(FieldPresentCondition), "fieldPresent")] +[JsonDerivedType(typeof(SectionPresentCondition), "sectionPresent")] +[JsonDerivedType(typeof(PlaceholderCondition), "placeholder")] +[JsonDerivedType(typeof(AssessmentScoreCondition), "assessmentScore")] +public interface ICondition +{ + /// + /// Evaluates the condition for the issue in . + /// + bool Evaluate(EvaluationContext context); +} + +/// True when every one of is true (and when there are none). +public sealed record AllCondition : ICondition +{ + /// Gets the conditions that must all hold. + public required ImmutableList Conditions { get; init; } + + /// + public bool Evaluate(EvaluationContext context) => Conditions.TrueForAll(condition => condition.Evaluate(context)); +} + +/// True when at least one of is true. +public sealed record AnyCondition : ICondition +{ + /// Gets the conditions of which one must hold. + public required ImmutableList Conditions { get; init; } + + /// + public bool Evaluate(EvaluationContext context) => Conditions.Exists(condition => condition.Evaluate(context)); +} + +/// True when is false. +public sealed record NotCondition : ICondition +{ + /// Gets the condition to negate. + public required ICondition Condition { get; init; } + + /// + public bool Evaluate(EvaluationContext context) => !Condition.Evaluate(context); +} + +/// The value of a named condition defined in . +public sealed record ConditionReference : ICondition +{ + /// Gets the name of the condition. + public required string Name { get; init; } + + /// + public bool Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return context.Condition(Name); + } +} + +/// +/// Compares a measure with bounds: true when it is at least and less than +/// , ignoring a bound that is not set. +/// +/// Set exactly one of (a named measure of the assessment) or . +public sealed record MeasureCondition : ICondition +{ + /// Gets the name of one of the assessment's measures. + public string? Measure { get; init; } + + /// Gets a measure defined in place, for use outside a tiered assessment. + public IMeasure? Of { get; init; } + + /// Gets the inclusive lower bound. + public double? AtLeast { get; init; } + + /// Gets the exclusive upper bound. + public double? LessThan { get; init; } + + /// + public bool Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + double value = Of?.Evaluate(context) ?? context.Measure(Measure!); + return Bounds.Contain(value, AtLeast, LessThan); + } +} + +/// +/// The optional bounds conditions compare a value against. +/// +internal static class Bounds +{ + /// + /// True when is at least and less than + /// ; an unset bound always holds. + /// + public static bool Contain(double value, double? atLeast, double? lessThan) + => (atLeast is not { } low || value >= low) && (lessThan is not { } high || value < high); +} + +/// +/// True when the text matches any of , case-insensitively unless is +/// set. An absent section matches nothing. +/// +public sealed record TextMatchesCondition : ICondition +{ + /// Gets the text to search. + public required TextSource Source { get; init; } + + /// Gets the regular expressions, any of which may match. + public required ImmutableList Patterns { get; init; } + + /// Gets a value indicating whether matching is case-sensitive. Default false. + public bool CaseSensitive { get; init; } + + /// + public bool Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return context.Text(Source) is { } text && Patterns.Exists(pattern => RuleRegex.Get(pattern, CaseSensitive).IsMatch(text)); + } +} + +/// +/// Tests the values at a JSONPath (a scalar, or each element of an array) against , +/// case-insensitively unless is set. +/// +/// +/// With false (the default), true when any value matches any pattern. With it true, true when +/// there is at least one value and every value matches some pattern: for example, "every component is a back-end one". +/// +public sealed record FieldMatchesCondition : ICondition +{ + /// Gets the JSONPath to read. + public required string Path { get; init; } + + /// Gets the regular expressions to test values against. + public required ImmutableList Patterns { get; init; } + + /// Gets a value indicating whether every value, rather than any, must match. Default false. + public bool RequireAll { get; init; } + + /// Gets a value indicating whether matching is case-sensitive. Default false. + public bool CaseSensitive { get; init; } + + /// + public bool Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + IReadOnlyList values = context.Strings(Path); + bool Matches(string value) => Patterns.Exists(pattern => RuleRegex.Get(pattern, CaseSensitive).IsMatch(value)); + return RequireAll ? values.Count > 0 && values.All(Matches) : values.Any(Matches); + } +} + +/// +/// True when any value at a JSONPath (a scalar, or any element of an array) equals one of , +/// ignoring case unless is set. Booleans and numbers compare by their JSON text +/// (true, 42). +/// +public sealed record FieldEqualsCondition : ICondition +{ + /// Gets the JSONPath to read. + public required string Path { get; init; } + + /// Gets the accepted values. + public required ImmutableList Values { get; init; } + + /// Gets a value indicating whether comparison is case-sensitive. Default false. + public bool CaseSensitive { get; init; } + + /// + public bool Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + StringComparer comparer = CaseSensitive ? StringComparer.Ordinal : StringComparer.OrdinalIgnoreCase; + return context.Strings(Path).Any(value => Values.Contains(value, comparer)); + } +} + +/// +/// True when the JSONPath has a value: not missing, not null, not a blank string and not an empty array. +/// +public sealed record FieldPresentCondition : ICondition +{ + /// Gets the JSONPath to read. + public required string Path { get; init; } + + /// + public bool Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return context.Strings(Path).Any(value => !string.IsNullOrWhiteSpace(value)); + } +} + +/// +/// True when the source's section heading is present, whatever is under it. The source must have a +/// . +/// +public sealed record SectionPresentCondition : ICondition +{ + /// Gets the section to look for. + public required TextSource Source { get; init; } + + /// + public bool Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return context.Text(Source) is not null; + } +} + +/// +/// True when the text is blank or only a placeholder such as TBD, N/A, none or to do (see +/// ). An absent section is a placeholder. +/// +public sealed record PlaceholderCondition : ICondition +{ + /// Gets the text to test. + public required TextSource Source { get; init; } + + /// + public bool Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return IssueText.IsPlaceholder(context.Text(Source)); + } +} + +/// +/// Compares another assessment's raw score with bounds, once scoring is complete. Only score caps and classifications +/// can use it. False when that assessment did not apply or was not scored. +/// +public sealed record AssessmentScoreCondition : ICondition +{ + /// Gets the name of the assessment. + public required string Assessment { get; init; } + + /// Gets the inclusive lower bound on its raw score. + public double? AtLeast { get; init; } + + /// Gets the exclusive upper bound on its raw score. + public double? LessThan { get; init; } + + /// + /// Thrown when used before scoring is complete. + public bool Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + if (context.AssessmentResults is null) + { + throw new InvalidOperationException("An assessment score condition can only be used by score caps and classifications."); + } + + return context.AssessmentResults.TryGetValue(Assessment, out AssessmentResult? result) + && result.Outcome != AssessmentOutcome.NotApplicable + && result.Score.RawScore is { } score + && Bounds.Contain(score, AtLeast, LessThan); + } +} diff --git a/PanoramicData.IssueAssessment/Rules/EvaluationContext.cs b/PanoramicData.IssueAssessment/Rules/EvaluationContext.cs new file mode 100644 index 0000000..b567d04 --- /dev/null +++ b/PanoramicData.IssueAssessment/Rules/EvaluationContext.cs @@ -0,0 +1,195 @@ +using System.Collections.Concurrent; +using PanoramicData.IssueAssessment.Text; + +namespace PanoramicData.IssueAssessment.Rules; + +/// +/// What rules ( and ) can see while one issue is assessed: the issue, the +/// virtual extension, the rubric's named conditions, the measures of the assessment being scored and, once scoring is +/// done, the assessment results. +/// +/// +/// Text read from a and the value of each named condition are computed once per issue and +/// shared, so rules that scan the same text do not each rebuild it. Named conditions are always evaluated against the +/// issue itself, never a virtual extension, so their value does not depend on the order assessments run in. +/// +public sealed class EvaluationContext +{ + private readonly Shared _shared; + private readonly ImmutableDictionary _measures; + private readonly ConcurrentDictionary _measureValues = new(StringComparer.Ordinal); + private readonly ConcurrentDictionary _texts; + + /// + /// Initializes a context for with no named conditions. + /// + /// The issue being assessed. + public EvaluationContext(JsonDocument issue) + : this(issue, ImmutableDictionary.Empty) + { + } + + /// + /// Initializes a context for with the rubric's named conditions. + /// + /// The issue being assessed. + /// Conditions that rules refer to by name. + /// Thrown when an argument is null. + public EvaluationContext(JsonDocument issue, IReadOnlyDictionary namedConditions) + { + ArgumentNullException.ThrowIfNull(issue); + ArgumentNullException.ThrowIfNull(namedConditions); + _shared = new Shared(issue, namedConditions); + _measures = ImmutableDictionary.Empty; + _texts = _shared.IssueTexts; + _shared.Root = this; + } + + private EvaluationContext( + Shared shared, + JsonDocument? virtualExtension, + ImmutableDictionary measures, + IReadOnlyDictionary? assessmentResults) + { + _shared = shared; + VirtualExtension = virtualExtension; + _measures = measures; + AssessmentResults = assessmentResults; + _texts = virtualExtension is null ? shared.IssueTexts : new(StringComparer.Ordinal); + } + + /// + /// Gets the issue being assessed. + /// + public JsonDocument Issue => _shared.Issue; + + /// + /// Gets the virtual extension produced by earlier assessments in a series, if any. + /// + public JsonDocument? VirtualExtension { get; } + + /// + /// Gets the results of every scored assessment, by name, once scoring is complete; null while assessments run. + /// Only score caps and classifications see them. + /// + public IReadOnlyDictionary? AssessmentResults { get; } + + /// + /// Evaluates a named condition, once per issue. + /// + /// The condition's name in the rubric. + /// The condition's value. + /// Thrown when the rubric defines no condition with that name. + public bool Condition(string name) + { + if (!_shared.NamedConditions.TryGetValue(name, out ICondition? condition)) + { + throw new KeyNotFoundException($"No condition named '{name}' is defined."); + } + + return _shared.ConditionValues.GetOrAdd(name, _ => condition.Evaluate(_shared.Root)); + } + + /// + /// Evaluates a measure of the assessment being scored, once per assessment. + /// + /// The measure's name in the assessment. + /// The measured value. + /// Thrown when the assessment defines no measure with that name. + public double Measure(string name) + { + if (!_measures.TryGetValue(name, out IMeasure? measure)) + { + throw new KeyNotFoundException($"No measure named '{name}' is defined."); + } + + return _measureValues.GetOrAdd(name, _ => measure.Evaluate(this)); + } + + /// + /// Reads the text a describes: its fields joined by line breaks, narrowed to a section when + /// one is given. + /// + /// The source to read. + /// The text; null only when a section was asked for and no heading matches. + public string? Text(TextSource source) + { + ArgumentNullException.ThrowIfNull(source); + return _texts.GetOrAdd(source.Key, _ => + { + string joined = string.Join('\n', source.Paths.Select(path => string.Join('\n', Strings(path)))); + return source.Section is null ? joined : IssueText.SectionBody(joined, source.Section); + }); + } + + /// + /// The string values at : one for a scalar, one per element for an array, none when the + /// value is missing or null. Numbers and booleans are returned as their JSON text. + /// + /// The JSONPath to read. + /// The values, in order. + public IReadOnlyList Strings(string jsonPath) + { + if (!TrySelect(jsonPath, out JsonElement value)) + { + return []; + } + + return value.ValueKind == JsonValueKind.Array + ? [.. value.EnumerateArray().Select(ToText).OfType()] + : ToText(value) is { } text ? [text] : []; + } + + /// + /// Selects the value at from the issue, or from the virtual extension when the issue + /// does not contain it. + /// + public bool TrySelect(string jsonPath, out JsonElement value) + => JsonPathHelper.TrySelect(Issue, VirtualExtension, jsonPath, out value); + + /// + /// A context that reads the given virtual extension and shares this one's issue, named conditions and caches. + /// + internal EvaluationContext WithVirtualExtension(JsonDocument? virtualExtension) + => virtualExtension is null && AssessmentResults is null && _measures.IsEmpty + ? _shared.Root + : new(_shared, virtualExtension, ImmutableDictionary.Empty, null); + + /// + /// A context in which can be referred to by name. + /// + internal EvaluationContext WithMeasures(ImmutableDictionary measures) + => new(_shared, VirtualExtension, measures, AssessmentResults); + + /// + /// A context that can see the finished assessment results, for score caps and classifications. + /// + internal EvaluationContext WithAssessmentResults(IReadOnlyDictionary results) + => new(_shared, null, ImmutableDictionary.Empty, results); + + /// + /// The value of every named condition, for reporting. + /// + internal ImmutableDictionary AllConditions() + => _shared.NamedConditions.Keys.ToImmutableDictionary(name => name, Condition, StringComparer.Ordinal); + + private static string? ToText(JsonElement element) => element.ValueKind switch + { + JsonValueKind.String => element.GetString(), + JsonValueKind.Null or JsonValueKind.Undefined => null, + _ => element.GetRawText(), + }; + + private sealed class Shared(JsonDocument issue, IReadOnlyDictionary namedConditions) + { + public JsonDocument Issue { get; } = issue; + + public IReadOnlyDictionary NamedConditions { get; } = namedConditions; + + public ConcurrentDictionary ConditionValues { get; } = new(StringComparer.Ordinal); + + public ConcurrentDictionary IssueTexts { get; } = new(StringComparer.Ordinal); + + public EvaluationContext Root { get; set; } = null!; + } +} diff --git a/PanoramicData.IssueAssessment/Rules/Measures.cs b/PanoramicData.IssueAssessment/Rules/Measures.cs new file mode 100644 index 0000000..567a82d --- /dev/null +++ b/PanoramicData.IssueAssessment/Rules/Measures.cs @@ -0,0 +1,191 @@ +using System.Text.Json.Serialization; +using PanoramicData.IssueAssessment.Text; + +namespace PanoramicData.IssueAssessment.Rules; + +/// +/// A number measured from an issue, such as how many of a list of patterns its text matches. Measures feed +/// s and the placeholders in tier findings. +/// +[JsonPolymorphic(TypeDiscriminatorPropertyName = "type")] +[JsonDerivedType(typeof(PatternMatchCountMeasure), "patternMatchCount")] +[JsonDerivedType(typeof(OccurrenceCountMeasure), "occurrenceCount")] +[JsonDerivedType(typeof(WordCountMeasure), "wordCount")] +[JsonDerivedType(typeof(AlphanumericCountMeasure), "alphanumericCount")] +[JsonDerivedType(typeof(LengthMeasure), "length")] +[JsonDerivedType(typeof(ItemCountMeasure), "itemCount")] +[JsonDerivedType(typeof(NumberMeasure), "number")] +[JsonDerivedType(typeof(CriteriaHeadingCountMeasure), "criteriaHeadingCount")] +public interface IMeasure +{ + /// + /// Measures the issue in . + /// + double Evaluate(EvaluationContext context); +} + +/// +/// How many of match the text at least once, case-insensitively unless +/// is set. Each pattern counts at most once, however often it occurs. +/// +public sealed record PatternMatchCountMeasure : IMeasure +{ + /// Gets the text to search. + public required TextSource Source { get; init; } + + /// Gets the regular expressions to look for. + public required ImmutableList Patterns { get; init; } + + /// Gets a value indicating whether matching is case-sensitive. Default false. + public bool CaseSensitive { get; init; } + + /// + public double Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + string text = context.Text(Source) ?? string.Empty; + return Patterns.Count(pattern => RuleRegex.Get(pattern, CaseSensitive).IsMatch(text)); + } +} + +/// +/// How many times occurs in the text, case-insensitively unless is set. +/// +public sealed record OccurrenceCountMeasure : IMeasure +{ + /// Gets the text to search; an absent section counts zero. + public required TextSource Source { get; init; } + + /// Gets the regular expression to count. + public required string Pattern { get; init; } + + /// Gets a value indicating whether matching is case-sensitive. Default false. + public bool CaseSensitive { get; init; } + + /// + public double Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return RuleRegex.Get(Pattern, CaseSensitive).Count(context.Text(Source) ?? string.Empty); + } +} + +/// +/// The number of whitespace-separated words in the text. +/// +public sealed record WordCountMeasure : IMeasure +{ + /// Gets the text to count; an absent section counts zero. + public required TextSource Source { get; init; } + + /// + public double Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return IssueText.WordCount(context.Text(Source)); + } +} + +/// +/// The number of ASCII letters and digits in the text: how much it says, ignoring markup. +/// +public sealed record AlphanumericCountMeasure : IMeasure +{ + /// Gets the text to count; an absent section counts zero. + public required TextSource Source { get; init; } + + /// + public double Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return IssueText.AlphanumericCount(context.Text(Source)); + } +} + +/// +/// The length of the text in characters. +/// +public sealed record LengthMeasure : IMeasure +{ + /// Gets the text to measure; an absent section measures zero. + public required TextSource Source { get; init; } + + /// + public double Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return context.Text(Source)?.Length ?? 0; + } +} + +/// +/// How many values are at a JSONPath: an array's length, 1 for a present scalar, 0 when missing or null. +/// +public sealed record ItemCountMeasure : IMeasure +{ + /// Gets the JSONPath to count. + public required string Path { get; init; } + + /// + public double Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return context.Strings(Path).Count; + } +} + +/// +/// The number at a JSONPath (a JSON number or an invariant numeric string), or 0 when there is none. +/// +public sealed record NumberMeasure : IMeasure +{ + /// Gets the JSONPath to read. + public required string Path { get; init; } + + /// + public double Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return context.Strings(Path) is [string text] + && double.TryParse(text, NumberStyles.Float, CultureInfo.InvariantCulture, out double value) + && double.IsFinite(value) + ? value + : 0d; + } +} + +/// +/// How many headings in the text open an acceptance-criteria section (see +/// ). More than one means the criteria are stated twice. Described as the +/// headings' text, separated by semicolons. +/// +public sealed record CriteriaHeadingCountMeasure : IMeasure, IDescribedMeasure +{ + /// Gets the text to search. + public required TextSource Source { get; init; } + + /// + public double Evaluate(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return IssueText.CriteriaHeadings(context.Text(Source)).Count; + } + + /// + public string Describe(EvaluationContext context) + { + ArgumentNullException.ThrowIfNull(context); + return string.Join("; ", IssueText.CriteriaHeadings(context.Text(Source))); + } +} + +/// +/// A measure that can also describe what it measured, for {name:text} placeholders in tier findings. +/// +public interface IDescribedMeasure +{ + /// + /// Describes what was measured for the issue in . + /// + string Describe(EvaluationContext context); +} diff --git a/PanoramicData.IssueAssessment/Rules/PlaceholderText.cs b/PanoramicData.IssueAssessment/Rules/PlaceholderText.cs new file mode 100644 index 0000000..910268f --- /dev/null +++ b/PanoramicData.IssueAssessment/Rules/PlaceholderText.cs @@ -0,0 +1,46 @@ +using System.Text.RegularExpressions; + +namespace PanoramicData.IssueAssessment.Rules; + +/// +/// Fills the placeholders in findings, recommendations and not-applicable explanations. +/// +/// +/// {$.path} becomes the values at a JSONPath, comma-separated; {$.path~pattern} the first of them that +/// matches the pattern; {name} a measure's value; and {name:text} a described measure's text. +/// +internal static partial class PlaceholderText +{ + /// + /// The keys of the placeholders in . + /// + public static IEnumerable Keys(string? text) + => text is null ? [] : Pattern().Matches(text).Select(match => match.Groups[1].Value); + + /// + /// Fills the placeholders in , resolving measure names through + /// . + /// + public static string Format(string text, EvaluationContext context, ImmutableDictionary measures) + => Pattern().Replace(text, match => Resolve(match.Groups[1].Value, context, measures)); + + private static string Resolve(string key, EvaluationContext context, ImmutableDictionary measures) + { + if (key.StartsWith('$')) + { + int tilde = key.IndexOf('~', StringComparison.Ordinal); + return tilde < 0 + ? string.Join(", ", context.Strings(key)) + : context.Strings(key[..tilde]).FirstOrDefault(value => RuleRegex.Get(key[(tilde + 1)..], false).IsMatch(value)) ?? string.Empty; + } + + return key.EndsWith(":text", StringComparison.Ordinal) + && measures.TryGetValue(key[..^":text".Length], out IMeasure? measure) + && measure is IDescribedMeasure described + ? described.Describe(context) + : context.Measure(key).ToString(CultureInfo.InvariantCulture); + } + + [GeneratedRegex(@"\{([^{}\s]+)\}", RegexOptions.CultureInvariant, 1000)] + private static partial Regex Pattern(); +} diff --git a/PanoramicData.IssueAssessment/Rules/RuleRegex.cs b/PanoramicData.IssueAssessment/Rules/RuleRegex.cs new file mode 100644 index 0000000..ccef21e --- /dev/null +++ b/PanoramicData.IssueAssessment/Rules/RuleRegex.cs @@ -0,0 +1,15 @@ +using System.Text.RegularExpressions; + +namespace PanoramicData.IssueAssessment.Rules; + +/// +/// Compiled, time-limited regular expressions for rules, case-insensitive unless asked otherwise. +/// +internal static class RuleRegex +{ + /// + /// The regex for , from , so it has the same match timeout. + /// + public static Regex Get(string pattern, bool caseSensitive) + => RegexCache.GetOrAdd(caseSensitive ? pattern : "(?i)" + pattern); +} diff --git a/PanoramicData.IssueAssessment/Rules/TextSource.cs b/PanoramicData.IssueAssessment/Rules/TextSource.cs new file mode 100644 index 0000000..c0ee90b --- /dev/null +++ b/PanoramicData.IssueAssessment/Rules/TextSource.cs @@ -0,0 +1,30 @@ +namespace PanoramicData.IssueAssessment.Rules; + +/// +/// Where a rule reads text from: one or more fields joined by line breaks, optionally narrowed to the body under a +/// heading. +/// +/// +/// The summary and description together: new TextSource { Paths = ["$.title", "$.description"] }. +/// The steps-to-reproduce section of the description: +/// new TextSource { Paths = ["$.description"], Section = @"(?:reproduc|\brepro\b)" }. +/// +public sealed record TextSource +{ + /// + /// Gets the JSONPaths whose values are joined, in order, with a line break. A missing value contributes an empty + /// line; an array contributes its elements, one per line. + /// + public required ImmutableList Paths { get; init; } + + /// + /// Gets an optional regular expression fragment matched against heading text. When set, the text is the body under + /// the best matching heading (see ), or absent when no heading matches. + /// + public string? Section { get; init; } + + /// + /// A key identifying this source's text within one evaluation, so equal sources share one read. + /// + internal string Key => string.Join('\u001f', Paths) + '\u001e' + Section; +} diff --git a/PanoramicData.IssueAssessment/Scoring/ScoreCalculator.cs b/PanoramicData.IssueAssessment/Scoring/ScoreCalculator.cs index 262867f..3ed4bbf 100644 --- a/PanoramicData.IssueAssessment/Scoring/ScoreCalculator.cs +++ b/PanoramicData.IssueAssessment/Scoring/ScoreCalculator.cs @@ -22,6 +22,7 @@ public static (double MinPossibleRaw, double MaxPossibleRaw) CalculateAssessment PresenceAssessment presence => ToRange(presence.PresentScore, presence.AbsentScore), ValueAssessment value => ToRange(value.SuccessScore, value.FailureScore), JsonPathSelectorAssessment selector => ToRange(selector.SuccessScore, selector.FailureScore), + TieredAssessment tiered => (tiered.MinimumScore, tiered.MaximumScore), // Callbacks and custom definitions declare no bounds, so their raw scores are taken to lie in [0, 1]. _ => (DefaultUnknownMinimum, DefaultUnknownMaximum), }; @@ -35,11 +36,23 @@ public static (double MinPossibleRaw, double MaxPossibleRaw) CalculateAssessment public static (double MinPossibleRaw, double MaxPossibleRaw) CalculateCategoryRawRange(AssessmentCategory category) { ArgumentNullException.ThrowIfNull(category); + return CalculateCategoryRawRange(category.Assessments); + } + + /// + /// Calculates the weighted minimum and maximum raw score bounds over , for example + /// only those of a category that applied. + /// + /// The assessments. + /// The weighted raw score bounds. + public static (double MinPossibleRaw, double MaxPossibleRaw) CalculateCategoryRawRange(IEnumerable assessments) + { + ArgumentNullException.ThrowIfNull(assessments); double min = 0d; double max = 0d; - foreach (Assessment assessment in category.Assessments) + foreach (Assessment assessment in assessments) { (double assessmentMin, double assessmentMax) = CalculateAssessmentRawRange(assessment); double weight = assessment.Weight; diff --git a/PanoramicData.IssueAssessment/Serialization/AssessmentRubric.cs b/PanoramicData.IssueAssessment/Serialization/AssessmentRubric.cs new file mode 100644 index 0000000..0c766e0 --- /dev/null +++ b/PanoramicData.IssueAssessment/Serialization/AssessmentRubric.cs @@ -0,0 +1,111 @@ +using System.Text.Json.Serialization; +using System.Text.Json.Serialization.Metadata; + +namespace PanoramicData.IssueAssessment.Serialization; + +/// +/// Reads and writes as JSON, so a rubric can live in a file shared by every consumer +/// rather than in each one's code. +/// +/// +/// +/// Definitions, conditions, measures and gates carry a "type" discriminator, for example +/// { "type": "tiered", ... } or { "type": "fieldEquals", "path": "$.issueType", "values": ["Bug"] }. +/// Property names are camelCase, enums are camelCase strings, and comments and trailing commas are allowed. +/// +/// +/// Definitions that hold delegates, and custom definition types, cannot be written as JSON. A rubric refers to one as +/// { "type": "named", "name": "..." } and the caller supplies it when loading. +/// +/// +/// Loading does not validate: constructing an does, with messages naming the category and +/// assessment. +/// +/// +public static class AssessmentRubric +{ + /// + /// Gets the serializer options rubrics are read and written with. + /// + public static JsonSerializerOptions SerializerOptions { get; } = CreateSerializerOptions(); + + /// + /// Reads a rubric from JSON. + /// + /// The rubric JSON. + /// The options the rubric describes. + /// Thrown when the JSON is not a valid rubric. + public static AssessmentOptions Load(string json) + => Load(json, ImmutableDictionary.Empty); + + /// + /// Reads a rubric from JSON, replacing each with the definition registered + /// under its name. + /// + /// The rubric JSON. + /// Definitions written in code, by name. + /// The options the rubric describes. + /// Thrown when the JSON is not a valid rubric. + public static AssessmentOptions Load(string json, IReadOnlyDictionary namedDefinitions) + { + ArgumentNullException.ThrowIfNull(json); + ArgumentNullException.ThrowIfNull(namedDefinitions); + AssessmentOptions options = JsonSerializer.Deserialize(json, SerializerOptions) + ?? throw new JsonException("The rubric JSON is null."); + return Resolve(options, namedDefinitions); + } + + /// + /// Reads a rubric from a UTF-8 JSON stream. + /// + /// The rubric JSON. + /// The options the rubric describes. + /// Thrown when the JSON is not a valid rubric. + public static AssessmentOptions Load(Stream utf8Json) + { + ArgumentNullException.ThrowIfNull(utf8Json); + using StreamReader reader = new(utf8Json); + return Load(reader.ReadToEnd()); + } + + /// + /// Writes options as rubric JSON. + /// + /// The options to write. + /// Indented rubric JSON. + /// Thrown when the options hold a definition that cannot be written as JSON; + /// use a in its place. + public static string ToJson(AssessmentOptions options) + { + ArgumentNullException.ThrowIfNull(options); + return JsonSerializer.Serialize(options, SerializerOptions); + } + + private static AssessmentOptions Resolve(AssessmentOptions options, IReadOnlyDictionary namedDefinitions) + => options with + { + Categories = [.. options.Categories.Select(category => category with + { + Assessments = [.. category.Assessments.Select(assessment => + assessment.Definition is NamedAssessmentDefinition named + && namedDefinitions.TryGetValue(named.Name, out IAssessmentDefinition? definition) + ? assessment with { Definition = definition } + : assessment)], + })], + }; + + private static JsonSerializerOptions CreateSerializerOptions() + { + JsonSerializerOptions options = new(JsonSerializerDefaults.Web) + { + WriteIndented = true, + DefaultIgnoreCondition = JsonIgnoreCondition.WhenWritingNull, + ReadCommentHandling = JsonCommentHandling.Skip, + AllowTrailingCommas = true, + TypeInfoResolver = new DefaultJsonTypeInfoResolver(), + }; + options.Converters.Add(new JsonStringEnumConverter(JsonNamingPolicy.CamelCase)); + options.MakeReadOnly(); + return options; + } +} diff --git a/PanoramicData.IssueAssessment/Text/IssueText.cs b/PanoramicData.IssueAssessment/Text/IssueText.cs new file mode 100644 index 0000000..d02a10d --- /dev/null +++ b/PanoramicData.IssueAssessment/Text/IssueText.cs @@ -0,0 +1,286 @@ +using System.Text.RegularExpressions; + +namespace PanoramicData.IssueAssessment.Text; + +/// +/// Reads the structure of ticket text written in Jira wiki markup or Markdown: the body under a heading, the headings +/// that state acceptance criteria, and whether a section is only a placeholder. +/// +/// +/// A port of the hardened readers in PanoramicData.Skills' IssueQualityCore.ps1, so that a rubric scored by this +/// library reads a ticket exactly as Yoda does. Jira returns descriptions with CRLF line endings, and every pattern +/// here accepts both CRLF and LF. +/// +public static class IssueText +{ + private const RegexOptions Options = RegexOptions.CultureInvariant; + private static readonly TimeSpan Timeout = RegexCache.MatchTimeout; + + private static readonly Regex WikiHeadingAnywhere = new(@"(?im)^[ \t]*h[1-6]\.", Options, Timeout); + private static readonly Regex WikiHeadingLevel = new(@"(?im)^[ \t]*h([1-6])\.", Options, Timeout); + private static readonly Regex AtxHeadingLevel = new(@"(?im)^[ \t]*(#{1,6})[ \t]", Options, Timeout); + private static readonly Regex StartsWithHash = new(@"^[ \t]*#", Options | RegexOptions.IgnoreCase, Timeout); + private static readonly Regex StartsWithSingleHashItem = new(@"^[ \t]*#[ \t]", Options | RegexOptions.IgnoreCase, Timeout); + private static readonly Regex HashListLine = new(@"^[ \t]*#+[ \t]", Options | RegexOptions.IgnoreCase, Timeout); + private static readonly Regex StartsWithWiki = new(@"^[ \t]*h[1-6]\.", Options | RegexOptions.IgnoreCase, Timeout); + private static readonly Regex StartsWithDoubleAsterisk = new(@"^[ \t]*\*\*", Options | RegexOptions.IgnoreCase, Timeout); + private static readonly Regex StartsWithAsterisk = new(@"^[ \t]*\*", Options | RegexOptions.IgnoreCase, Timeout); + private static readonly Regex LineBreak = new(@"\r?\n", Options, Timeout); + + private static readonly Regex WikiNext = new(@"(?im)^[ \t]*(?:h[1-6]\.|#{2,6}[ \t])", Options, Timeout); + private static readonly Regex AtxNext = new(@"(?im)^[ \t]*(?:h[1-6]\.|#{1,6}[ \t])", Options, Timeout); + private static readonly Regex Bold2Next = new(@"(?im)^[ \t]*(?:h[1-6]\.|#{1,6}[ \t]|\*{1,2}[^\r\n*]+\*{1,2}[ \t]*\r?$)", Options, Timeout); + private static readonly Regex Bold1Next = new(@"(?im)^[ \t]*(?:h[1-6]\.|\*{1,2}[^\r\n*]+\*{1,2}[ \t]*\r?$)", Options, Timeout); + private static readonly Regex LabelNext = new(@"(?im)^[ \t]*(?:h[1-6]\.|\*{1,2}[^\r\n*]+\*{1,2}[ \t]*\r?$|(?=[A-Za-z])[^\r\n:#*]{1,60}:[ \t]*\r?$)", Options, Timeout); + + private static readonly Regex CriteriaHeadingLine = new(@"^[ \t]*(?:h([1-6])\.|(#{1,6})[ \t]|\*{1,2}(?=[^*\r\n]+\*{1,2}[ \t]*:?[ \t]*$))(.*)$", Options, Timeout); + private static readonly Regex CriteriaHeadingText = new(@"(?i)criteria|definition\s+of\s+done|verification\s+evidence", Options, Timeout); + private static readonly Regex CriteriaMappingTable = new(@"(?im)^[ \t]*\|.*criteri.*\|.*\btests?\b", Options, Timeout); + + private static readonly Regex PlaceholderListMarker = new(@"(?m)^[ \t]*(?:[-*#>]+|\d+\.)[ \t]*", Options, Timeout); + private static readonly Regex PlaceholderPunctuation = new(@"[\*_`{}\[\]\(\)\|:;,\.\-–—/\\]", Options, Timeout); + private static readonly Regex PlaceholderWords = new(@"^\s*(tbd|to\s*be\s*(done|decided|determined|confirmed|advised)|n\s*/?\s*a|none|nil|todo|to\s*do|pending|unknown|see\s+above)\s*$", Options | RegexOptions.IgnoreCase, Timeout); + + private static readonly Regex Whitespace = new(@"\s+", Options, Timeout); + private static readonly Regex Alphanumeric = new("[A-Za-z0-9]", Options, Timeout); + + /// + /// Returns the body text under the best heading whose text matches , or + /// null when there is no such heading. + /// + /// The ticket text. + /// A regular expression fragment matched against the heading text. + /// + /// The section body, which may be empty when a heading has nothing under it; null when no heading matches. When + /// several headings match, the longest body wins. + /// + /// + /// + /// Recognises Jira wiki headings (h1. to h6.), Markdown ATX headings (# to ######), + /// single- or double-asterisk bold used as a heading, and a label line that is only the label and a colon + /// (Steps to reproduce:). + /// + /// + /// What closes a section depends on what opened it. A wiki or ATX heading closes at a heading of the same level or + /// stronger, so sub-headings stay inside. In wiki text a line starting # is a numbered list item, never a + /// heading, and a ## line continuing such a list is a nested item. Bold and label headings close at the + /// next heading-like line. + /// + /// + public static string? SectionBody(string? text, string headingKeywordPattern) + { + ArgumentException.ThrowIfNullOrWhiteSpace(headingKeywordPattern); + if (string.IsNullOrWhiteSpace(text)) + { + return null; + } + + Regex heading = RegexCache.GetOrAdd(@"(?im)^[ \t]*(?:h[1-6]\.|#{1,6}[ \t]|\*{1,2}(?=[^\r\n]*\*{1,2}[ \t]*\r?$))[^\r\n]*" + headingKeywordPattern + @"[^\r\n]*"); + Regex label = RegexCache.GetOrAdd(@"(?im)^[ \t]*(?![hH][1-6]\.)(?=[A-Za-z])[^\r\n:#*]{0,60}?" + headingKeywordPattern + @"[^\r\n:]{0,60}:[ \t]*\r?$"); + + IEnumerable openings = heading.Matches(text).Concat(label.Matches(text)); + + // In wiki text a "#" line is a list item, so it cannot open a section; only a "##"-or-deeper line that does not + // continue a "#" list may. + if (WikiHeadingAnywhere.IsMatch(text)) + { + openings = openings.Where(opening => IsWikiTextOpening(text, opening)); + } + + string? best = null; + foreach (Match opening in openings) + { + string body = SectionAfter(opening.Value, text[(opening.Index + opening.Length)..]); + if (best is null || body.Length > best.Length) + { + best = body; + } + } + + return best; + } + + /// + /// Returns the text of every heading that opens an acceptance-criteria section: a heading whose text mentions + /// criteria, a definition of done or verification evidence. + /// + /// The ticket text. + /// The heading texts, in order, without markup. + /// + /// Only headings count, never body text. A section holding a criterion-to-test mapping table is evidence rather than + /// restated criteria, so its heading is not counted. In wiki text a single # line is a list item. + /// + public static IReadOnlyList CriteriaHeadings(string? text) + { + if (string.IsNullOrWhiteSpace(text)) + { + return []; + } + + string[] lines = LineBreak.Split(text); + List<(int Index, int Level, string Text)> headings = Headings(lines, WikiHeadingAnywhere.IsMatch(text)); + List found = []; + for (int current = 0; current < headings.Count; current++) + { + string headingText = headings[current].Text; + if (CriteriaHeadingText.IsMatch(headingText) && !IsMappingTableSection(lines, headings, current)) + { + found.Add(headingText); + } + } + + return found; + } + + /// + /// True when a section body is a placeholder (TBD, N/A, none, to do, pending, ...) or blank, rather than content. + /// + /// The section body. + /// for a blank or placeholder body. + public static bool IsPlaceholder(string? body) + { + if (string.IsNullOrWhiteSpace(body)) + { + return true; + } + + // Remove list bullets, numbering and emphasis so "* TBD" reads as "TBD". + string stripped = PlaceholderListMarker.Replace(body, string.Empty); + stripped = PlaceholderPunctuation.Replace(stripped, " ").Trim(); + return PlaceholderWords.IsMatch(stripped); + } + + /// + /// The number of whitespace-separated words in . + /// + public static int WordCount(string? text) + => string.IsNullOrEmpty(text) ? 0 : Whitespace.Split(text).Count(word => word.Length > 0); + + /// + /// The number of ASCII letters and digits in : a measure of how much a section actually says, + /// ignoring markup and punctuation. + /// + public static int AlphanumericCount(string? text) + => string.IsNullOrEmpty(text) ? 0 : Alphanumeric.Count(text); + + /// + /// The level of a heading line: 1 for h1. or #, up to 6; 7 for a line that is not a structural + /// heading, which is weaker than every real level and so never closes a section by accident. + /// + public static int HeadingLevel(string? headingLine) + { + if (string.IsNullOrWhiteSpace(headingLine)) + { + return 7; + } + + Match wiki = WikiHeadingLevel.Match(headingLine); + if (wiki.Success) + { + return int.Parse(wiki.Groups[1].Value, CultureInfo.InvariantCulture); + } + + Match atx = AtxHeadingLevel.Match(headingLine); + return atx.Success ? atx.Groups[1].Value.Length : 7; + } + + /// + /// Every heading line, with its index and level; in wiki text a single # line is a list item, not a heading. + /// + private static List<(int Index, int Level, string Text)> Headings(string[] lines, bool isWiki) + { + List<(int Index, int Level, string Text)> headings = []; + for (int index = 0; index < lines.Length; index++) + { + Match match = CriteriaHeadingLine.Match(lines[index]); + if (match.Success && !(isWiki && StartsWithSingleHashItem.IsMatch(lines[index]))) + { + headings.Add((index, HeadingLevel(match), match.Groups[3].Value.Trim().Trim('*').Trim().TrimEnd(':').Trim())); + } + } + + return headings; + } + + private static int HeadingLevel(Match criteriaHeading) + { + if (criteriaHeading.Groups[1].Success) + { + return int.Parse(criteriaHeading.Groups[1].Value, CultureInfo.InvariantCulture); + } + + return criteriaHeading.Groups[2].Success ? criteriaHeading.Groups[2].Value.Length : 7; + } + + private static bool IsWikiTextOpening(string text, Match opening) + { + if (!StartsWithHash.IsMatch(opening.Value)) + { + return true; + } + + return !StartsWithSingleHashItem.IsMatch(opening.Value) && !PreviousLineIsHashList(text[..opening.Index]); + } + + /// + /// Cuts at the line that closes the section opened. + /// + private static string SectionAfter(string opener, string rest) + { + bool openedByWiki = StartsWithWiki.IsMatch(opener); + if (openedByWiki || StartsWithHash.IsMatch(opener)) + { + return LevelledSectionAfter(opener, rest, openedByWiki); + } + + Regex closer = StartsWithDoubleAsterisk.IsMatch(opener) ? Bold2Next + : StartsWithAsterisk.IsMatch(opener) ? Bold1Next + : LabelNext; + Match next = closer.Match(rest); + return next.Success ? rest[..next.Index] : rest; + } + + /// + /// Cuts at the next heading of the same level as or stronger, never + /// at a weaker one. + /// + private static string LevelledSectionAfter(string opener, string rest, bool openedByWiki) + { + int openLevel = HeadingLevel(opener); + foreach (Match candidate in (openedByWiki ? WikiNext : AtxNext).Matches(rest)) + { + // A "## " line continuing a "#" list is a nested wiki list item, not a Markdown heading. + bool continuesWikiList = openedByWiki && StartsWithHash.IsMatch(candidate.Value) && PreviousLineIsHashList(rest[..candidate.Index]); + if (!continuesWikiList && HeadingLevel(candidate.Value) <= openLevel) + { + return rest[..candidate.Index]; + } + } + + return rest; + } + + private static bool PreviousLineIsHashList(string before) + { + string? previous = LineBreak.Split(before).LastOrDefault(line => line.Trim().Length > 0); + return previous is not null && HashListLine.IsMatch(previous); + } + + private static bool IsMappingTableSection(string[] lines, List<(int Index, int Level, string Text)> headings, int current) + { + (int index, int level, _) = headings[current]; + + // The section runs to the next heading of the same or a higher level. + int end = lines.Length; + for (int next = current + 1; next < headings.Count; next++) + { + if (headings[next].Level <= level) + { + end = headings[next].Index; + break; + } + } + + return end > index + 1 && CriteriaMappingTable.IsMatch(string.Join('\n', lines[(index + 1)..end])); + } +} diff --git a/PanoramicData.IssueAssessment/Validation/AssessmentValidator.cs b/PanoramicData.IssueAssessment/Validation/AssessmentValidator.cs index c85a5a4..90fd301 100644 --- a/PanoramicData.IssueAssessment/Validation/AssessmentValidator.cs +++ b/PanoramicData.IssueAssessment/Validation/AssessmentValidator.cs @@ -9,7 +9,7 @@ internal static class AssessmentValidator private const double MinimumWeight = -100d; private const double MaximumWeight = 100d; - internal static void Validate(AssessmentCategory category, Assessment assessment) + internal static void Validate(AssessmentCategory category, Assessment assessment, RuleScope scope) { ArgumentNullException.ThrowIfNull(assessment); @@ -18,15 +18,7 @@ internal static void Validate(AssessmentCategory category, Assessment assessment throw new ArgumentException($"Category '{category.Name}' contains an assessment with no name.", nameof(assessment)); } - Exception? innerException = null; - string? error = AssessmentError(assessment); - if (error is null) - { - error = SelectorError(assessment.Definition, out innerException) - ?? DefinitionError(assessment.Definition) - ?? SettingsError(assessment); - } - + string? error = Error(assessment, scope, out Exception? innerException); if (error is not null) { throw new ArgumentException( @@ -36,6 +28,35 @@ internal static void Validate(AssessmentCategory category, Assessment assessment } } + /// + /// The first problem with the assessment, checked in order: identity, selectors, definition settings, assessment + /// settings, then rules. + /// + private static string? Error(Assessment assessment, RuleScope scope, out Exception? innerException) + { + innerException = null; + return AssessmentError(assessment) + ?? SelectorError(assessment.Definition, out innerException) + ?? DefinitionError(assessment.Definition) + ?? SettingsError(assessment) + ?? RuleError(assessment, scope); + } + + /// + /// Validates the assessment's applicability condition and, for a tiered or named definition, its rules. + /// + private static string? RuleError(Assessment assessment, RuleScope scope) + => RuleValidator.ConditionError(assessment.AppliesWhen, scope) + ?? PlaceholderText.Keys(assessment.NotApplicableExplanation) + .Select(key => key.StartsWith('$') ? RuleValidator.PathError(key) : $"has a not-applicable explanation placeholder '{{{key}}}' that is not a JSONPath.") + .FirstOrDefault(error => error is not null) + ?? assessment.Definition switch + { + TieredAssessment tiered => TieredValidator.Error(tiered, scope), + NamedAssessmentDefinition named => $"refers to custom definition '{named.Name}', which was not registered when the rubric was loaded.", + _ => null, + }; + private static string? AssessmentError(Assessment assessment) { if (string.IsNullOrWhiteSpace(assessment.Description)) diff --git a/PanoramicData.IssueAssessment/Validation/OptionsValidator.cs b/PanoramicData.IssueAssessment/Validation/OptionsValidator.cs index 21c2fbc..48d0d93 100644 --- a/PanoramicData.IssueAssessment/Validation/OptionsValidator.cs +++ b/PanoramicData.IssueAssessment/Validation/OptionsValidator.cs @@ -26,12 +26,14 @@ public static void Validate(IAssessmentOptions options) throw new ArgumentException("GlobalTimeout must be greater than zero.", nameof(options)); } + RuleScope scope = RubricValidator.ValidateConditions(options); foreach (AssessmentCategory category in options.Categories) { - ValidateCategory(category); + ValidateCategory(category, scope); } ValidateUniqueNames(options); + RubricValidator.ValidateRubricRules(options, scope); } /// @@ -59,11 +61,11 @@ private static void ValidateUniqueNames(IAssessmentOptions options) private static string? FirstDuplicate(IEnumerable names) => names.GroupBy(name => name, StringComparer.Ordinal).FirstOrDefault(group => group.Count() > 1)?.Key; - private static void ValidateCategory(AssessmentCategory category) + private static void ValidateCategory(AssessmentCategory category, RuleScope scope) { ArgumentNullException.ThrowIfNull(category); - string? error = CategoryError(category); + string? error = CategoryError(category) ?? RubricValidator.CategoryRuleError(category, scope); if (error is not null) { throw new ArgumentException(error, nameof(category)); @@ -80,7 +82,7 @@ private static void ValidateCategory(AssessmentCategory category) foreach (Assessment assessment in category.Assessments) { - AssessmentValidator.Validate(category, assessment); + AssessmentValidator.Validate(category, assessment, scope); } } diff --git a/PanoramicData.IssueAssessment/Validation/RubricValidator.cs b/PanoramicData.IssueAssessment/Validation/RubricValidator.cs new file mode 100644 index 0000000..d03bbd1 --- /dev/null +++ b/PanoramicData.IssueAssessment/Validation/RubricValidator.cs @@ -0,0 +1,106 @@ +namespace PanoramicData.IssueAssessment.Validation; + +/// +/// Validates the rubric-level parts of the options: named conditions, category rules, score caps, grade bands and +/// classifications. +/// +internal static class RubricValidator +{ + /// + /// Validates the named conditions and returns the scope rules elsewhere may use. + /// + public static RuleScope ValidateConditions(IAssessmentOptions options) + { + RuleScope scope = new([.. options.Conditions.Keys], [], null); + foreach ((string name, ICondition condition) in options.Conditions) + { + string? error = condition is null ? "is missing." : RuleValidator.ConditionError(condition, scope); + if (error is not null) + { + throw new ArgumentException($"Condition '{name}' {error}", nameof(options)); + } + } + + string? cycle = RuleValidator.CycleError(options.Conditions); + return cycle is null ? scope : throw new ArgumentException(cycle, nameof(options)); + } + + public static string? CategoryRuleError(AssessmentCategory category, RuleScope scope) + { + if (!double.IsFinite(category.Weight) || category.Weight < 0d) + { + return string.Create(CultureInfo.InvariantCulture, $"Category '{category.Name}' has invalid weight {category.Weight}; it must be zero or more."); + } + + return RuleValidator.ConditionError(category.AppliesWhen, scope) is { } error + ? $"Category '{category.Name}' {error}" + : null; + } + + /// + /// Validates score caps, grade bands and classifications, which may also use assessment scores. + /// + public static void ValidateRubricRules(IAssessmentOptions options, RuleScope scope) + { + RuleScope scoring = scope with + { + AssessmentNames = [.. options.Categories.SelectMany(category => category.Assessments).Select(assessment => assessment.Name)], + }; + + string? error = options.ScoreCaps.Select(cap => CapError(cap, scoring)).FirstOrDefault(found => found is not null) + ?? BandsError(options.GradeBands) + ?? options.Classifications.Select(pair => ClassificationError(pair.Key, pair.Value, scoring)).FirstOrDefault(found => found is not null); + if (error is not null) + { + throw new ArgumentException(error, nameof(options)); + } + } + + private static string? CapError(ScoreCap cap, RuleScope scope) + { + if (string.IsNullOrWhiteSpace(cap.Name)) + { + return "A score cap has no name."; + } + + if (cap.Cap is < 0d or > 1d || double.IsNaN(cap.Cap)) + { + return $"Score cap '{cap.Name}' must be between 0 and 1."; + } + + return cap.When is null + ? $"Score cap '{cap.Name}' has no condition." + : RuleValidator.ConditionError(cap.When, scope) is { } error ? $"Score cap '{cap.Name}' {error}" : null; + } + + private static string? BandsError(ImmutableList bands) + { + GradeBand? invalid = bands.Find(band => string.IsNullOrWhiteSpace(band.Name) || band.MinimumScore is < 0d or > 1d || double.IsNaN(band.MinimumScore)); + if (invalid is not null) + { + return $"Grade band '{invalid.Name}' must have a name and a minimum score between 0 and 1."; + } + + string? duplicate = bands.GroupBy(band => band.Name, StringComparer.Ordinal).FirstOrDefault(group => group.Count() > 1)?.Key; + return duplicate is null ? null : $"Grade band '{duplicate}' is defined more than once."; + } + + private static string? ClassificationError(string name, ImmutableList rules, RuleScope scope) + { + if (rules is not { Count: > 0 }) + { + return $"Classification '{name}' has no rules."; + } + + if (rules.Exists(rule => string.IsNullOrWhiteSpace(rule.Value))) + { + return $"Classification '{name}' has a rule without a value."; + } + + return rules + .Select(rule => RuleValidator.ConditionError(rule.When, scope)) + .FirstOrDefault(error => error is not null) is { } found + ? $"Classification '{name}' {found}" + : null; + } +} diff --git a/PanoramicData.IssueAssessment/Validation/RuleValidator.cs b/PanoramicData.IssueAssessment/Validation/RuleValidator.cs new file mode 100644 index 0000000..1a42b8e --- /dev/null +++ b/PanoramicData.IssueAssessment/Validation/RuleValidator.cs @@ -0,0 +1,181 @@ +namespace PanoramicData.IssueAssessment.Validation; + +/// +/// What a rule may refer to where it is used. +/// +/// The rubric's named conditions. +/// The measures of the tiered assessment being validated; empty elsewhere. +/// Every assessment's name, where assessment scores are visible (score caps and +/// classifications); null elsewhere. +internal sealed record RuleScope( + ImmutableHashSet ConditionNames, + ImmutableHashSet MeasureNames, + ImmutableHashSet? AssessmentNames); + +/// +/// Validates conditions, measures and text sources. Custom and +/// implementations are not inspected. +/// +internal static class RuleValidator +{ + public static string? ConditionError(ICondition? condition, RuleScope scope) => condition switch + { + null => null, + AllCondition all => ConditionsError(all.Conditions, scope), + AnyCondition any => ConditionsError(any.Conditions, scope), + NotCondition not => not.Condition is null ? "has a 'not' condition without a condition." : ConditionError(not.Condition, scope), + ConditionReference reference => scope.ConditionNames.Contains(reference.Name) + ? null + : $"refers to undefined condition '{reference.Name}'.", + MeasureCondition measure => MeasureConditionError(measure, scope), + AssessmentScoreCondition score => AssessmentScoreError(score, scope), + _ => FieldConditionError(condition), + }; + + public static string? MeasureError(IMeasure? measure) => measure switch + { + null => "has a missing measure.", + PatternMatchCountMeasure value => SourceError(value.Source) ?? PatternsError(value.Patterns), + OccurrenceCountMeasure value => SourceError(value.Source) ?? PatternError(value.Pattern), + WordCountMeasure value => SourceError(value.Source), + AlphanumericCountMeasure value => SourceError(value.Source), + LengthMeasure value => SourceError(value.Source), + CriteriaHeadingCountMeasure value => SourceError(value.Source), + ItemCountMeasure value => PathError(value.Path), + NumberMeasure value => PathError(value.Path), + _ => null, + }; + + /// + /// The first named condition that refers to itself, directly or through others. + /// + public static string? CycleError(IReadOnlyDictionary conditions) + { + foreach (string name in conditions.Keys) + { + if (ReachesItself(name, conditions)) + { + return $"Condition '{name}' refers to itself."; + } + } + + return null; + } + + public static string? PathError(string? jsonPath) + { + try + { + JsonPathHelper.Validate(jsonPath!); + return null; + } + catch (InvalidJsonPathException) + { + return $"has invalid JSONPath '{jsonPath}'."; + } + } + + public static string? PatternError(string? pattern) + { + if (string.IsNullOrWhiteSpace(pattern)) + { + return "has an empty pattern."; + } + + try + { + _ = RegexCache.GetOrAdd(pattern); + return null; + } + catch (ArgumentException) + { + return $"has invalid regex pattern '{pattern}'."; + } + } + + private static string? FieldConditionError(ICondition condition) => condition switch + { + TextMatchesCondition value => SourceError(value.Source) ?? PatternsError(value.Patterns), + FieldMatchesCondition value => PathError(value.Path) ?? PatternsError(value.Patterns), + FieldEqualsCondition value => PathError(value.Path) ?? (value.Values is not { Count: > 0 } ? "has a 'fieldEquals' condition without values." : null), + FieldPresentCondition value => PathError(value.Path), + SectionPresentCondition value => SourceError(value.Source) ?? (value.Source.Section is null ? "has a 'sectionPresent' condition whose source has no section." : null), + PlaceholderCondition value => SourceError(value.Source), + _ => null, + }; + + private static string? ConditionsError(ImmutableList? conditions, RuleScope scope) + => conditions is null + ? "has a combined condition without conditions." + : conditions.Select(condition => ConditionError(condition, scope)).FirstOrDefault(error => error is not null); + + private static string? MeasureConditionError(MeasureCondition condition, RuleScope scope) + { + if ((condition.Measure is null) == (condition.Of is null)) + { + return "has a 'measure' condition that must set exactly one of Measure and Of."; + } + + return condition.Of is not null + ? MeasureError(condition.Of) + : scope.MeasureNames.Contains(condition.Measure!) ? null : $"refers to undefined measure '{condition.Measure}'."; + } + + private static string? AssessmentScoreError(AssessmentScoreCondition condition, RuleScope scope) + { + if (scope.AssessmentNames is null) + { + return "uses an 'assessmentScore' condition, which only score caps and classifications may use."; + } + + return scope.AssessmentNames.Contains(condition.Assessment) ? null : $"refers to unknown assessment '{condition.Assessment}'."; + } + + private static string? SourceError(TextSource? source) + { + if (source?.Paths is not { Count: > 0 }) + { + return "has a text source without paths."; + } + + return source.Paths.Select(PathError).FirstOrDefault(error => error is not null) + ?? (source.Section is null ? null : PatternError(source.Section)); + } + + private static string? PatternsError(ImmutableList? patterns) + => patterns is not { Count: > 0 } + ? "has no patterns." + : patterns.Select(PatternError).FirstOrDefault(error => error is not null); + + private static bool ReachesItself(string start, IReadOnlyDictionary conditions) + { + Stack pending = new(References(conditions[start])); + HashSet seen = []; + while (pending.TryPop(out string? name)) + { + if (name == start) + { + return true; + } + + if (seen.Add(name) && conditions.TryGetValue(name, out ICondition? next)) + { + foreach (string reference in References(next)) + { + pending.Push(reference); + } + } + } + + return false; + } + + private static IEnumerable References(ICondition? condition) => condition switch + { + ConditionReference reference => [reference.Name], + AllCondition all => all.Conditions.SelectMany(References), + AnyCondition any => any.Conditions.SelectMany(References), + NotCondition not => References(not.Condition), + _ => [], + }; +} diff --git a/PanoramicData.IssueAssessment/Validation/TieredValidator.cs b/PanoramicData.IssueAssessment/Validation/TieredValidator.cs new file mode 100644 index 0000000..5ef197f --- /dev/null +++ b/PanoramicData.IssueAssessment/Validation/TieredValidator.cs @@ -0,0 +1,71 @@ +namespace PanoramicData.IssueAssessment.Validation; + +/// +/// Validates a : its score range, its tiers and the measures and placeholders they use. +/// +internal static class TieredValidator +{ + public static string? Error(TieredAssessment tiered, RuleScope scope) + { + if (tiered.Tiers is not { Count: > 0 }) + { + return "must define at least one tier."; + } + + if (tiered.MinimumScore >= tiered.MaximumScore) + { + return "has a MinimumScore that is not below its MaximumScore."; + } + + RuleScope tierScope = scope with { MeasureNames = [.. tiered.Measures.Keys] }; + return StructureError(tiered) + ?? tiered.Measures.Values.Select(RuleValidator.MeasureError).FirstOrDefault(error => error is not null) + ?? tiered.Tiers.Select(tier => TierError(tiered, tier, tierScope)).FirstOrDefault(error => error is not null); + } + + private static string? StructureError(TieredAssessment tiered) + { + if (tiered.Tiers[^1].When is not null) + { + return "must end with a tier that has no condition, so every issue lands in a tier."; + } + + return tiered.Tiers.Take(tiered.Tiers.Count - 1).Any(tier => tier.When is null) + ? "has a tier without a condition before its last tier, so the tiers after it can never apply." + : null; + } + + private static string? TierError(TieredAssessment tiered, Tier tier, RuleScope scope) + { + if (tier.Finding is null) + { + return "has a tier without a finding; use an empty finding for a tier that only recommends."; + } + + if (tier.Score < tiered.MinimumScore || tier.Score > tiered.MaximumScore) + { + return string.Create( + CultureInfo.InvariantCulture, + $"has a tier score of {tier.Score} outside its range {tiered.MinimumScore}-{tiered.MaximumScore}."); + } + + return RuleValidator.ConditionError(tier.When, scope) + ?? PlaceholderError(tier.Finding, scope) + ?? PlaceholderError(tier.Recommendation, scope); + } + + private static string? PlaceholderError(string? text, RuleScope scope) + => PlaceholderText.Keys(text).Select(key => PlaceholderKeyError(key, scope)).FirstOrDefault(error => error is not null); + + private static string? PlaceholderKeyError(string key, RuleScope scope) + { + if (key.StartsWith('$')) + { + int tilde = key.IndexOf('~', StringComparison.Ordinal); + return tilde < 0 ? RuleValidator.PathError(key) : RuleValidator.PathError(key[..tilde]) ?? RuleValidator.PatternError(key[(tilde + 1)..]); + } + + string name = key.EndsWith(":text", StringComparison.Ordinal) ? key[..^":text".Length] : key; + return scope.MeasureNames.Contains(name) ? null : $"refers to undefined measure '{name}' in a placeholder."; + } +} diff --git a/README.md b/README.md index da8ec14..1aedc23 100644 --- a/README.md +++ b/README.md @@ -81,9 +81,23 @@ The built-in categories read a small normalized shape: | `assigneeState` | `Assigned`, `InProgress` or `Owned` when someone has taken it | | `dueDate` | Any date or date-time string (`2026-11-20`, `2026-11-20T17:00:00Z`) | -Map your platform's JSON to that shape before assessing. The -[Examples project](PanoramicData.IssueAssessment.Examples) has complete, runnable mappers for the Jira REST API v2, -the GitHub REST API and the ServiceNow Table API, including two details that matter in practice: +Map your platform's JSON to that shape before assessing. Since 1.1 the package includes mappers to +`NormalizedIssue`, which also carries issue type, priority, status, project, components, labels, fix versions and +an attachment count: + +```csharp +using PanoramicData.IssueAssessment.Mapping; + +NormalizedIssue issue = JiraIssueMapper.FromRestJson(jiraIssueJson, new Uri("https://jira.example.com/")); +// or GitHubIssueMapper.FromRestJson(githubIssueJson), ServiceNowRecordMapper.FromTableJson(recordJson) +using JsonDocument ticket = issue.ToJsonDocument(); +``` + +`JiraIssueMapper` reads REST API v2 (wiki text) and v3 (Atlassian Document Format, flattened to wiki text). If you +already hold typed objects, such as a Jira client's `Issue`, set the `NormalizedIssue` properties directly. + +The [Examples project](PanoramicData.IssueAssessment.Examples) has complete, runnable mappers for the Jira REST +API v2, the GitHub REST API and the ServiceNow Table API, including two details that matter in practice: - GitHub has no fields for reproduction steps or acceptance criteria, so they are read from the headed sections that issue templates produce. @@ -96,6 +110,94 @@ Your own assessments can of course read any JSONPath in the raw ticket directly. dotnet run --project PanoramicData.IssueAssessment.Examples ``` +## Rubrics + +A rubric is a whole configuration written as JSON: categories, assessments, named conditions, score caps, grade +bands and classifications. It suits teams that share one definition of a good ticket between several tools. + +### The built-in ticket-quality rubric + +`BuiltInRubrics.PanoramicDataTicketQuality` is the rubric Panoramic Data uses for Jira and GitHub tickets. It +scores components, description, testability, steps to reproduce, current versus expected behaviour, environment, +attachments, documentation, certifications, UI consistency, business case and rollback plan for ordinary tickets, +and a separate set of criteria for epics. It grades the result EXCELLENT, GOOD, ADEQUATE, NEEDS IMPROVEMENT or +POOR, and caps a ticket below EXCELLENT when a mandatory element (steps to reproduce on a bug, a rollback plan on a +change request) is missing. + +```csharp +using PanoramicData.IssueAssessment.Rubrics; + +var assessor = new IssueAssessor(BuiltInRubrics.PanoramicDataTicketQuality); +AssessmentReport report = AssessmentReport.From(await assessor.AssessAsync(ticket, cancellationToken)); + +Console.WriteLine($"{report.Score}/100 {report.Grade}"); +report.Findings.ForEach(Console.WriteLine); // "✓ Steps to Reproduce: 3 numbered steps", ... +report.Recommendations.ForEach(Console.WriteLine); // what to add to raise the score +Console.WriteLine(report.ToJson()); +``` + +`BuiltInRubrics.PanoramicDataTicketQualityJson` returns its JSON, a good starting point for your own. + +### Writing a rubric + +```jsonc +{ + "executionStrategy": "series", + "conditions": { + "isBug": { "type": "fieldEquals", "path": "$.issueType", "values": [ "Bug", "Defect" ] } + }, + "categories": [ { + "name": "Ticket Quality", "description": "Is the ticket ready to work on?", "assessmentExecutionStrategy": "series", + "assessments": [ { + "name": "Steps to reproduce", "description": "Bugs say how to reproduce them", + "appliesWhen": { "type": "ref", "name": "isBug" }, + "recommendation": "Add numbered steps to reproduce.", + "definition": { + "type": "tiered", + "measures": { + "steps": { "type": "occurrenceCount", "source": { "paths": [ "$.description" ], "section": "reproduc" }, "pattern": "(?m)^\\s*\\d+\\." } + }, + "tiers": [ + { "when": { "type": "measure", "measure": "steps", "atLeast": 3 }, "score": 100, "finding": "✓ {steps} steps to reproduce" }, + { "when": { "type": "sectionPresent", "source": { "paths": [ "$.description" ], "section": "reproduc" } }, "score": 50, "finding": "⚠ Steps are not a numbered list" }, + { "score": 0, "finding": "✗ No steps to reproduce" } + ] + } + } ] + } ], + "scoreCaps": [ { "name": "Steps to reproduce", "cap": 0.79, "group": "mandatory", + "when": { "type": "assessmentScore", "assessment": "Steps to reproduce", "lessThan": 100 } } ], + "gradeBands": [ { "name": "GOOD", "minimumScore": 0.6 }, { "name": "POOR", "minimumScore": 0 } ] +} +``` + +```csharp +AssessmentOptions options = AssessmentRubric.Load(File.ReadAllText("rubric.json")); +``` + +The building blocks: + +- **Tiered assessments** (`"type": "tiered"`): tiers are tried in order and the first whose `when` holds gives the + score (0–100 by default), the finding and an optional recommendation. The last tier has no condition. +- **Measures** count things: `patternMatchCount`, `occurrenceCount`, `wordCount`, `alphanumericCount`, `length`, + `itemCount`, `number` and `criteriaHeadingCount`. +- **Conditions**: `all`, `any`, `not`, `ref` (a named condition), `measure`, `textMatches`, `fieldMatches`, + `fieldEquals`, `fieldPresent`, `sectionPresent`, `placeholder` (a section that only says "TBD") and, for caps and + classifications, `assessmentScore`. +- **Text sources** join one or more fields and can narrow to a section by heading, in Jira wiki or Markdown. +- **Placeholders** in findings and recommendations: `{measure}`, `{measure:text}`, `{$.path}` and + `{$.path~pattern}` (the first value matching the pattern). +- **Applicability**: `appliesWhen` on an assessment or a category. An assessment that does not apply is reported as + `NotApplicable` (with its `notApplicableExplanation`) and is left out of the score entirely. +- **Category weights**, **score caps** (the lowest applicable cap wins), **grade bands** and **classifications** + (first matching rule gives the value). Named conditions are reported back as flags. + +Every rule is validated when `IssueAssessor` is constructed, as below. Definitions written in code can sit in a +rubric as `{ "type": "named", "name": "..." }` and be supplied to `AssessmentRubric.Load(json, namedDefinitions)`. + +The text tools the rubric uses are public too: `IssueText.SectionBody`, `CriteriaHeadings`, `IsPlaceholder`, +`WordCount` and `AlphanumericCount`. + ## Concepts ### Assessments @@ -265,8 +367,9 @@ thread): median 0.22 ms, 95th percentile 0.49 ms, 99th percentile 0.95 ms per ti ## Quality -- 545 unit tests; 100% line coverage of the library (the one unmeasured branch is the compiler-generated null check - in `await foreach`). +- 815 unit tests; 100% line coverage of the library. +- The built-in rubric reproduces the scores, grades, findings and recommendations of the PowerShell engine it + replaced on all 74 of that engine's test fixtures and on 320 real Jira tickets. - Builds with zero warnings under `TreatWarningsAsErrors` and the `latest-recommended` analyzers. - CodeQL and Codacy analysis on every push. diff --git a/coverage.config b/coverage.config index 92cb2e3..6731cee 100644 --- a/coverage.config +++ b/coverage.config @@ -7,9 +7,16 @@ .*PanoramicData\.IssueAssessment\.dll$ + + + .*[\\/]obj[\\/].* + + ^System\.Diagnostics\.CodeAnalysis\.ExcludeFromCodeCoverageAttribute$ + + ^System\.CodeDom\.Compiler\.GeneratedCodeAttribute$ diff --git a/version.json b/version.json index 6f98fcd..28d4cab 100644 --- a/version.json +++ b/version.json @@ -1,6 +1,6 @@ { "$schema": "https://raw.githubusercontent.com/AArnott/Nerdbank.GitVersioning/master/src/NerdBank.GitVersioning/version.schema.json", - "version": "1.0", + "version": "1.1", "publicReleaseRefSpec": [ "^refs/heads/main$", "^refs/tags/\\d+\\.\\d+\\.\\d+$"