Skip to content

About

Browser extension and FastAPI service that scans GitHub commits with Semgrep and uses a CodeBERT classifier to filter false-positive alerts, using the diff, surrounding code and commit message as context. Bachelor diploma project, AITU 2026–2027.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Latest commit

 

History

52 Commits

Folders and files

NameName
Last commit message
Last commit date
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Commit Guard

Context-aware false positive reduction for security analysis of Git commits

Status Client Backend ML Tracker

Bachelor Diploma Project (2026–2027)

Topic: Development of an intelligent system for contextual analysis of Git commits to identify vulnerabilities and reduce false positives using machine learning methods

Institution: Astana IT University (AITU) Student group: SE-2423 Supervisor: Tulebayev Yersultan Bakhytovich Milestone: Pre-Defense 1 (October 2026)


Project Overview

Commit Guard is a browser extension for GitHub backed by a REST API. When a developer opens a commit or pull request page and clicks Analyze, the system checks the code changed by the commit and shows which security findings are likely to be real, instead of a long list of alerts to review one by one

The platform combines three ideas:

  • Static analysis as a source of candidates. Semgrep scans the files changed by the commit and produces candidate alerts
  • Context-aware classification. A CodeBERT-based classifier decides whether each alert is a true or a false positive, using the context of the change: the code diff, the surrounding code and the commit message
  • Feedback loop. Users can mark a result as a false alarm; the feedback is stored for retraining
flowchart LR
    U[Developer on a GitHub commit page] --> E[Browser extension]
    E -->|repo + SHA| API[FastAPI backend]
    API <-->|diff, message, metadata| GH[GitHub REST API]
    API --> S[Semgrep]
    S -->|candidate alerts| C[Context builder]
    C --> M[CodeBERT classifier]
    M -->|true / false positive| R[Risk level and explanations]
    R --> E
    API <--> DB[(PostgreSQL)]
Loading

Aim: to reduce false positives of static analysis alerts while preserving vulnerability recall

Problem Statement, Target Users & Outcomes

The Problem

  • Alert noise. SAST tools produce many alerts that are not real problems. In our interviews with six practitioners, the respondents who gave estimates (4 of 6) considered only about 10–30% of automated alerts important
  • Time lost on triage. Reviewing alerts takes 10–30 minutes per alert or 1–2 hours per week for the developer on duty
  • Weakened protection. Because of the noise, teams raise severity thresholds, ignore notifications and disable rules, so real vulnerabilities can slip through
  • No change context. Scanners look at code without the context of the change: what the commit modified and why (diff and commit message)

Target Users

  • Software developers: fast, accurate security feedback on their commits without noise
  • DevSecOps and cybersecurity specialists: automated commit-level checks and less manual audit work
  • QA and code quality engineers: a security check before code is merged into production branches

Expected Outcomes

  • A browser extension (Chrome / Edge) for on-demand analysis of GitHub commit and pull request pages
  • A context-aware ML pipeline that filters Semgrep alerts using diffs and commit messages
  • A findings view with the overall risk level (Low / Medium / High), highlighted lines, explanations and confidence scores
  • A backend REST API (FastAPI) with PostgreSQL storage
  • An evaluation report comparing the proposed system with baselines on precision, recall, F1, FPR, MCC, PR-AUC and latency

Team Members, Roles & Supervision

Full Name Student Group Engineering Role Responsibilities GitHub Account
Aiymzhan Abilgazy SE-2423 Frontend Developer & Data Engineer Data collection and preprocessing, database design, UI/UX and browser extension @aiymzhanabilgazy
Aibar Tolegenov SE-2423 Machine Learning Engineer False positive reduction model, training and evaluation, baselines and metrics @oiibar
Marzhan Tulebayeva SE-2423 System Architect & Backend Engineer System architecture, FastAPI backend, GitHub integration, system integration and testing @Marzhan-b

Academic Supervisor: Tulebayev Yersultan Bakhytovich, Astana IT University (AITU)

Scope Boundaries & Development Status

Scope Boundaries

In scope for the diploma MVP:

  • GitHub commit and pull request pages, analyzed on demand from the extension
  • Fetching the commit diff and metadata through the GitHub REST API; private repositories with a user-provided token.
  • Semgrep on the changed files; only alerts on lines changed by the commit are kept
  • Context extraction: changed lines, surrounding code (tree-sitter), commit message
  • Alert classification (true / false positive) with CodeBERT, compared with baselines B1–B3
  • Risk level, highlighted findings with explanations, "False alarm" feedback stored in PostgreSQL
  • Evaluation on the CVEfixes dataset with a project-based or time-based split

Out of scope (future work):

  • Automatic scanning through webhooks or CI
  • GitLab and Bitbucket support
  • A team dashboard
  • Dependency (SCA) alerts

Pre-Defense 1 Components

# Component Document Status
1 Project proposal & scope Project Proposal ✅ Done
2 Evidence of relevance (six practitioner interviews) Presentation; summary 🟡 Separate document planned
3 Literature review (20 sources) Literature Review ✅ Done
4 Existing system comparison (5 systems, 8 criteria) Existing Systems Comparison ✅ Done
5 Technology comparison (5 decisions) Technology Stack & Comparison ✅ Done
6 Requirements (10 FR with MoSCoW priorities, 7 NFR) Requirements, Frontend Requirements ✅ Done
7 Architecture diagrams: component decomposition, use case, system context Chrome Extension Architecture, docs/architecture/ ✅ Done
8 UX prototype: 11 screens and Figma prototype/ ✅ Done
9 Verification & evaluation plan Verification & Evaluation Plan ✅ Done
10 Project management & risks This README, YouTrack ✅ Done
11 Supervisor feedback log Supervisor Feedback ✅ Done

Implementation Progress

Component Location Status
GitHub client: commit diff and metadata src/backend/fetch_diff.py ✅ Done
Diff parser: line numbers added by a commit src/backend/diff_parser.py ✅ Done
Unit tests for the diff parser tests/test_diff_parser.py ✅ Done (3 tests)
FastAPI server with /health src/backend/main.py ✅ Done
ML data preparation and labeling (CVEfixes) src/ml/ 🟡 In progress
/analyze endpoint, Semgrep runner, context extraction src/backend/ ⬜ Planned
Baselines B1–B3 and CodeBERT classifier src/ml/ ⬜ Planned
PostgreSQL storage and feedback saving src/backend/ ⬜ Planned
Browser extension src/extension/ ⬜ Planned (UX prototype ready)

No evaluation results have been measured yet. All numbers in the requirements and the evaluation plan are targets

Evidence of Relevance (Summary)

We held informal, anonymous interviews with six practitioners (five software engineers and one support engineer). The sample is small and the figures are subjective estimates, because the teams do not keep statistics

  • Of the respondents who gave estimates (4 of 6), only about 10–30% of automated alerts were considered important
  • Triage takes 10–30 minutes per alert or 1–2 hours per week for the developer on duty
  • Noise leads to relaxed settings: higher severity thresholds, ignored notifications, disabled rules
  • Secrets leaked or almost leaked into repositories in several teams, often noticed by chance
  • Code-level (SAST) evidence comes from R2 and R6 (R3 secondhand); dependency alerts (R1, R5) are out of scope of the current version

Project Proposal

Problem. Traditional SAST tools generate numerous false-positive alerts and may not consider the context of individual code changes (Git diffs, commit messages). Developers spend significant time on irrelevant warnings, and the noise makes genuine issues harder to prioritize. Commit Guard combines static analysis with contextual machine learning to separate relevant findings from likely false positives. The effectiveness of this approach will be assessed experimentally rather than assumed in advance.

Objectives

  1. Analysis and review: Study scientific publications and existing methods for detecting vulnerabilities in commits and reducing false-positive alerts.
  2. Data work: Collect, label, clean, and prepare a dataset of Git commits, including code changes and available metadata.
  3. Architecture and backend: Design the database and implement a server-side REST API with Git integration.
  4. Machine-learning models: Train and evaluate models that classify security alerts using commit context.
  5. Integration and testing: Combine the browser extension, backend, static analyzer, and ML component; compare results with baseline approaches such as Semgrep alone, rule-based filtering, and XGBoost where feasible.

Proposed Solution

Commit Guard is planned as a browser extension for GitHub commit and pull-request pages, supported by a backend analysis service. The proposed workflow is:

  1. A user opens a supported GitHub commit or pull-request page and requests an analysis.
  2. The extension obtains the relevant commit identifier or page context and sends an analysis request to the backend.
  3. The backend retrieves and parses Git changes and related metadata.
  4. Semgrep produces candidate static-analysis findings.
  5. A contextual ML component assesses the findings using available code-change context.
  6. The interface displays prioritized findings, their locations, risk indicators, and explanations.

Evaluation Approach

The main evaluation question is:

Can a contextual ML filter reduce false-positive Semgrep alerts on Git commit changes without substantially reducing recall for real vulnerabilities?

The planned comparison includes Semgrep, rule-based filtering, and an XGBoost baseline, where a suitable labeled dataset allows fair comparison, against the proposed Semgrep + contextual ML pipeline.

The evaluation will report precision, recall, F1-score, false-positive measures, and analysis latency. The team will document the dataset, labeling procedure, train/test separation, and relevant limitations.

Literature Review

Source: docs/articles/Commit_Guard_Literature_Synthesis_Review.pdf (20 sources, November 2026). Sources fall into six topics: (A) learning to triage static-analysis alerts, (B) commit context as a security signal, (C) datasets and label quality, (D) learning-based vulnerability detectors, (E) static analysis in practice, (F) foundations: CodeBERT, baselines, metrics.

What the Literature Shows

  • Learned filters remove much alert noise. Keeping only CppCheck alerts inside functions that a fine-tuned CodeBERT judged vulnerable raised precision on Big-Vul from 5.9% to 96.2%, while recall fell from 46.9% to 40.7% (Apostolidis et al., 2025). A commit-metadata classifier gave over 99% fewer false positives than Flawfinder at the same recall (Perl et al., 2015).
  • Context helps. Method-level context improved warning triage over the warning line alone (Ge et al., 2025). Commit messages and diffs carry security signal that a code snapshot lacks (Perl et al., 2015; Zhou & Sharma, 2017).
  • A filter can only remove alerts. Its recall can never exceed the analyser's own, which is the ceiling Commit Guard has with Semgrep.

Where Results Do Not Hold

  • Random splits overstate performance. F1 fell from 49% to 9.4% on unseen projects (Chen et al., 2023); cross-project triage is worse than within-project (Ge et al., 2025); on time-ordered real projects no detector exceeded F1 23.3% (Zhang et al., 2023).
  • Labels are noisy. Fix-commit heuristics gave 20-71% inaccurate vulnerable labels, and correcting them cut measured precision by 29-80% (Croft et al., 2023). Only 51.7% of sampled CVEfixes labels were correct (Chen et al., 2023). The negative class is also unconfirmed: latent and silent fixes outnumber reported ones by more than 3:1 (Croft et al., 2022).
  • Duplicates inflate metrics. Removing cross-set duplicates dropped MCC on D2A from 0.774 to 0.141.
  • Narrow evaluation. Studies focus on C/C++ and Java analysers; none uses Semgrep. In triage studies commits only create labels, and MCC, PR-AUC and latency are rarely reported together.

Research Gap

Previous studies show that learned models can reduce static-analysis false alarms, but most use code-only inputs, C/C++ or Java analysers and random or pooled splits. It is insufficiently investigated whether commit context (fix diff and commit message) improves per-alert true/false-positive classification of Semgrep findings on real-world fix commits, under project-held-out evaluation, at matched recall and practical latency.

Implications for Commit Guard

Design element Evidence Action
Data split Random splits inflate results Project-based split as primary (forks in the same split); time-based split as secondary check
TP labelling rule SAST warnings rarely hit fix lines (line-level recall <= 0.016 in C/C++); fixes often add checks next to the vulnerable line Measure Semgrep hit rate on fix lines early; test a small line window around inserted hunks; ignore whitespace- and comment-only hunks
Label noise 20-71% wrong positive labels; negatives unconfirmed Audit a sample of TP/FP labels with two raters (Cohen's kappa); treat FP as "not confirmed", precision as a lower bound
Duplicates Cross-set duplicates inflate MCC Deduplicate (rule, normalised snippet) across splits; drop conflicting duplicates
Model input CodeBERT: [CLS] text [SEP] code, 512 tokens; truncation loses lines in long functions Text: rule ID, message, commit message. Code: window centred on the alert line plus diff hunk
Baselines Tree ensembles are strong and cheap Keep B1-B3; give XGBoost commit features (additions, hunks, author experience) and SAST features (warning density, severity)
Class imbalance Class-weighted loss was the best remedy on unseen projects Class-weighted loss; choose threshold on validation set at target recall
Metrics F1 ignores true negatives; MCC and PR-AUC are better under imbalance Keep metric set, add inspection ratio; accept-all MCC is undefined, set to 0
Recall ceiling Filtering cannot add recall Report Semgrep's own recall on fix lines and the share the model retains
Latency 0.02-0.5 s per sample for encoder models on GPU Measure median end-to-end per-commit time on CPU and GPU
Backbone CodeT5/NatGen and UniXcoder beat CodeBERT in some studies Keep CodeBERT; optionally run one alternative encoder as sensitivity check

The literature supports the central design: filter Semgrep instead of replacing it, use a pre-trained encoder with commit context, split by project, report MCC and false-alarm reduction at matched recall. The main risk is in labelling: the TP/FP rule inherits fix-commit noise, and strict line matching may give a small positive class.

Existing Systems Comparison

We compared five existing systems that reduce noise in security checks or analyze commits, and added Commit Guard as the sixth column. Two of the systems are commercial platforms with free tiers, one is a commercial platform without one, and two are research tools

Evaluation Criteria Semgrep Assistant GitGuardian GitHub Advanced Security ZeroFalse VulCurator Commit Guard (proposed)
System Type Commercial platform built on top of open-source Semgrep CE Commercial platform Commercial platform Research prototype (arXiv preprint 2510.02534) Research tool (ESEC/FSE 2022, demo track) Research prototype (diploma project)
Analyzed Objects Semgrep findings: code, dependencies, and secrets Secrets in commits, pull requests, and repositories Code via CodeQL, secrets via secret scanning, dependencies via Dependabot Static analysis findings Commits: whether a commit fixes a vulnerability Semgrep findings in a GitHub commit or pull request
Noise Reduction Mechanism ML classification of findings as true or false positives with confidence scoring FP Remover ML model (restricted to generic secrets) Focus on detection and automated fixes (Autofix); no dedicated AI noise triage found in reviewed sources LLM-based finding evaluation Not a noise filter: classifies commits CodeBERT classifies alerts as true or false positives using commit context
Context Awareness Stores developer feedback as project context Partial Partial Data flow traces, code context, and CWE knowledge Commit messages, code diffs, and task/issue descriptions Code diff, surrounding code, commit message
Decision Explanation Natural language explanations Risk score (0–100) with explanation Not specified in open sources Not specified in open sources Not specified (classifier output) Highlighted lines, risk level, confidence score, suggested fix
Claimed Efficiency ~60% triage automation; >96% decision agreement rate (vendor data) Up to 50% false positives removed, 0.3% true positives missed (vendor data) No data found Fewer false positives while maintaining coverage (authors) Up to 16.1% better F1 than baselines (authors) Target: ≥ 30% lower FPR, recall drop ≤ 5 pp (to be measured)
Availability / Pricing Free for teams up to 10 contributors; paid plans start at $30/month per contributor Free plan for up to 25 developers; custom enterprise pricing Paid service (~$19 to $30/month per active committer) Not specified in open sources Open-source on GitHub Not decided yet
Trigger Point / Stage After scan in CI/CD Commit and pull request checks Push protection blocks pushes with secrets; code scanning on pull requests After analyzer run (post-processing) After the fact (inspects existing commits) On demand on GitHub commit and PR pages

Data from vendor documentation and publications; pricing checked in October 2026. Efficiency figures are claims of vendors or authors. The values for Commit Guard are design goals, not measured results

Key takeaways

  • Noise filtering exists, but each tool covers its own scope: Semgrep Assistant works on Semgrep findings, and FP Remover only on generic secrets
  • VulCurator classifies whether a commit fixes a vulnerability, but it does not filter static analysis alerts
  • Commit Guard filters Semgrep alerts using the context of the commit and shows the result on demand on the GitHub commit page
  • Price is not the difference: Semgrep and GitGuardian have free plans for small teams. The difference to be tested is using the commit context as a signal for alert filtering

Requirements

Priorities use MoSCoW: Must (the project fails without it), Should (important, but a working product could exist without it), Could (only if time remains).

Functional Requirements

ID Requirement Priority Acceptance criteria
FR-1 The extension detects a GitHub commit or pull request page and extracts the repository and commit SHA. Must Works on 10 test commit URLs without errors.
FR-2 The extension sends an analysis request to the REST API backend. Must The request reliably reaches the backend.
FR-3 The backend fetches the commit diff and metadata via the GitHub REST API. Must The diff matches the real GitHub diff for 10 test commits.
FR-4 Context extraction: code changes, surrounding code and commit message. Must Context is extracted for at least 90% of changed functions in supported languages.
FR-5 The Semgrep static analyzer is run to obtain the initial alerts. Must The analyzer runs and returns alerts for all supported languages.
FR-6 The ML model classifies each alert as true positive or false positive. Must Meets the ML evaluation criteria (see the Verification & Evaluation Plan).
FR-7 The system outputs an overall commit risk level (Low / Medium / High). Must A risk level is returned for 100% of analyzed commits.
FR-8 Vulnerable code lines are highlighted with an explanation, confidence score and suggested fix. Should Each finding shows the vulnerability type, a brief description, the confidence score and a suggested fix.
FR-9 User feedback collection (Correct / False alarm). Should Feedback is saved to the database for model retraining.
FR-10 Export of analysis results (JSON / CSV). Could The user can download a report from the extension UI.

Non-Functional Requirements

ID Quality attribute Requirement Priority Acceptance criteria
NFR-1 Performance Fast processing for typical commits (up to 500 changed lines). Must Median latency up to 10 s; 95th percentile up to 30 s.
NFR-2 Security Secure data transfer over HTTPS; no API tokens in logs. Must Passes the security review checklist.
NFR-3 Accuracy Significant reduction of the false positive rate (FPR). Must FPR reduced by at least 30% versus the raw analyzer at similar recall.
NFR-4 Compatibility Cross-browser support for the extension. Must Extension tests pass on the latest Chrome and Edge.
NFR-5 Usability Intuitive first-time user experience. Should At least 80% task success rate; SUS score at least 68.
NFR-6 Maintainability Modular architecture and API documentation. Should Unit test coverage at least 60%.
NFR-7 Reproducibility Reproducible ML models and deployment. Should Fixed random seeds, versioned data, Dockerized setup.

Frontend Requirements

YouTrack task DIPLOMA-28. Status: proposed specification; implementation and acceptance tests pending. Scope: Chrome/Edge extension UI and its integration boundaries.

This document extracts frontend responsibilities from the Commit Guard system-level requirements presented at the first pre-defense. It retains the original FR identifiers, priorities, and acceptance criteria to ensure traceability. It does not claim that the requirements have been implemented or tested.

The extension is intended to operate on supported GitHub commit and pull-request pages, initiate analysis via the backend REST API, and present analysis results. The backend is responsible for retrieving diffs and metadata, extracting code context, running Semgrep, and performing ML classification.

The following functional and non-functional requirements are derived from the system-level Requirements Specification documented in Requirements. Only requirements relevant to the frontend are included, while the original requirement IDs and priorities are preserved for consistency and traceability.

Frontend Functional Requirements

Original ID Requirement Priority Acceptance criteria from project specification Frontend responsibility
FR-1 Extension detects GitHub commit/PR page and extracts repo/SHA. Must Works on 10/10 test commit URLs without errors. Detect supported pages; extract repository and commit identifier; handle unsupported URLs
FR-2 Extension sends an analysis request to the REST API backend. Must Request reliably reaches the server. Initiate the request and handle its loading, success, and failure outcomes.
FR-7 Output overall commit risk level (Low / Medium / High). Must Risk score returned for 100% of analyzed commits. Display the risk level returned by the backend; backend owns score generation.
FR-8 Highlight vulnerable code lines with explanation, confidence score and suggested fix. Should Display vulnerability type and brief description. Render finding locations and the available explanation, confidence and fix information.
FR-9 User feedback collection (Correct / False Alert). Should Feedback saved to DB for model retraining. DB for model retraining. Capture and submit the user's selection; backend owns persistence and retraining processes.
FR-10 Export analysis results (JSON / CSV). Could User can download report via extension UI. Offer download controls for supported result formats.

System Requirements the Frontend Depends On

Original ID Requirement Priority Original acceptance criteria Frontend dependency
FR-3 Backend fetches commit diff and metadata via GitHub REST API. Must Diff matches real GitHub diff for 10 test commits. The UI supplies a valid repository/commit reference.
FR-4 Context extraction (code changes, surrounding code, commit message). Must Context extracted for ≥ 90% of functions in supported languages. Context is processed by backend services.
FR-5 Execution of Semgrep static analyzer to fetch initial alerts. Must Analyzer outputs alerts for all supported languages. The UI presents returned findings.
FR-6 ML model classifies alerts as True Positive or False Positive. Must Meets ML evaluation criteria. The UI displays available classification output.

Non-Functional Requirements Relevant to the Frontend

The source slides assign NFRs by quality attribute rather than by numbered ID. The IDs below are local document identifiers, not replacements for the system specification.

Local ID Quality attribute System-level requirement Priority Acceptance criteria from project specification Frontend relevance
FE-NFR-01 Performance Fast processing for typical commits (≤ 500 changed lines). Must Median latency ≤ 10s; 95th percentile ≤ 30s. Keep the interface responsive and show progress while the backend processes the request; latency is measured end-to-end.
FE-NFR-02 Security Secure data transfer over HTTPS; no API tokens in logs. Must Pass Security Review Checklist. Use secure API communication; avoid logging tokens or sensitive content.
FE-NFR-03 Accuracy Significant reduction in False Positive Rate (FPR). Must FPR reduced by ≥ 30% vs. raw analyzer at similar recall. Display results accurately; the ML/backend evaluation owns the FPR target.
FE-NFR-04 Compatibility Cross-browser support for extension. Must Extension tests pass on latest Chrome & Edge. Test extension functionality on both browsers.
FE-NFR-05 Usability Intuitive first-time user experience. Should ≥ 80% task success rate; SUS score ≥ 68. Provide understandable controls, statuses, and findings; participate in usability testing.
FE-NFR-06 Maintainability Modular architecture & API documentation. Should Unit test code coverage ≥ 60%. Use separable UI components and documented API contracts; track frontend test coverage.
FE-NFR-07 Reproducibility Reproducible ML models and deployment. Should Fixed random seeds, versioned data, Dockerized setup Primarily an ML/deployment requirement; frontend should document its build and configuration during implementation.

Main User Interaction Flow

  1. The user opens a supported GitHub commit or pull-request page.
  2. The extension identifies the repository and relevant commit reference (FR-1).
  3. The user starts an analysis and the extension submits a backend request (FR-2).
  4. The interface waits for the response while remaining usable (FE-NFR-01).
  5. The interface displays the overall risk level (FR-7) and, when available, detailed findings (FR-8).
  6. Optionally, the user submits feedback (FR-9) or exports results (FR-10). Open design decision: The method for selecting a commit on pull-request pages, the exact API schema, and the location of code highlighting will be defined in DIPLOMA-29 and DIPLOMA-30. They are not finalized by these requirements.

Acceptance and Verification Checklist

  • FR-1 is verified against 10 supported GitHub commit URLs.
  • FR-2 is verified with a backend test endpoint and error handling scenarios.
  • FR-7 displays the backend risk level for valid analysis results.
  • FR-8 displays vulnerability type and description; extended fields are checked if supported by the API.
  • FR-9 feedback is persisted by the backend when the feature is implemented.
  • FR-10 downloads valid JSON and CSV files when the feature is implemented.
  • Extension compatibility tests pass on the latest Chrome and Edge.
  • Security review verifies HTTPS use and absence of token logging.
  • Usability testing checks task success and SUS against the specified targets.
  • Frontend test coverage is measured against the project's maintainability target.

Traceability and Boundaries

  • Source: Commit Guard pre-defense presentation, Requirements Specification slides (FR-1 through FR-10 and Non-Functional Requirements).
  • Related project documentation: Requirements (system-wide requirements).
  • Next design tasks: DIPLOMA-29 (Chrome extension architecture) and DIPLOMA-30 (frontend system design).
  • Out of scope for this task: Implementing the extension, backend GitHub fetching, Semgrep execution, ML model training, and proving the system-wide FPR target. This file documents requirements, not completed functionality. Numerical thresholds and priorities reflect the supplied project presentation and should be revised only through team/supervisor agreement.

Verification & Evaluation Plan

Software Testing

Level What is tested
Unit Parser, context extractor, risk scorer
Integration Extension, API, GitHub API, analyzer, ML model
End-to-end Full flow on real commit pages
Usability 5 real users complete four scenarios; task success rate and SUS questionnaire
Non-functional Response time (median and 95th percentile), HTTPS and no tokens in logs, Chrome and Edge compatibility

ML Evaluation

  • Data: CVEfixes (vulnerabilities and their fixes from open-source software). Duplicates and noisy labels are removed.
  • Labeling: Semgrep is run on the pre-fix version of each CVEfixes commit. Alerts on lines changed by the fix are labeled true positives, the other alerts false positives.
  • Split: project-based or time-based, not random, to prevent data leakage.
  • Limitations: labels come from a heuristic and are approximate; recall is measured only over alerts that Semgrep raises.
Metric Formula Meaning
Recall TP / (TP + FN) Of all real vulnerabilities, how many are found
Precision TP / (TP + FP) Of the alerts shown, how many are real
F1 2 x P x R / (P + R) Balance of precision and recall
FPR FP / (FP + TN) Of all false alarms, how many are still shown
MCC (TP x TN - FP x FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)) Reliable on imbalanced data (-1 to +1)
PR-AUC Area under the precision-recall curve Trade-off across thresholds
Latency Time to analyze one commit Median and 95th percentile

Baselines

ID Description
B1 Accepts all Semgrep alerts
B2 Semgrep plus a simple rule (for example ignore alerts in test files)
B3 XGBoost on hand-made features
Proposed Semgrep plus CodeBERT with commit context

Measurement Conditions

Same test set and Semgrep rules for every system; fixed random seeds; 3-5 runs reported as mean and standard deviation; same hardware for latency; runs with and without commit context.

Success Criteria

Criterion Target
False positive reduction At least 30% lower FPR than B1, recall drop at most 5 percentage points
Latency Median up to 10s; 95th percentile up to 30s
Usability SUS >= 68
Functional compliance 100% of Must requirements pass their acceptance criteria
NFRs HTTPS, no tokens in logs; Chrome and Edge tests; unit test coverage at least 60%; Docker and fixed seeds

Technology Stack & Comparison

Each decision below is linked to our requirements.

Selected Stack

Layer Technology
Browser extension TypeScript, HTML/JS, Manifest V3 (Chrome, Edge)
Backend & API Python 3, FastAPI, Uvicorn
Source data GitHub REST API
Static analysis Semgrep
Code parsing tree-sitter
Machine learning CodeBERT (fine-tuned), XGBoost baseline
Training data CVEfixes
Database PostgreSQL
Testing pytest
Design Figma
Project tracking YouTrack, Git & GitHub

Decisions

Decision Alternatives considered Selected Why (linked to requirements)
Backend framework FastAPI, Django REST Framework, Flask FastAPI Same Python ecosystem as the ML pipeline and Semgrep; async I/O for GitHub API calls (NFR Performance); automatic OpenAPI docs (NFR Maintainability). Django adds admin/ORM overhead we do not need; Flask needs extra libraries for validation
Database PostgreSQL, SQLite, MongoDB PostgreSQL Relational data (users, commits, findings, feedback) with ACID guarantees; JSONB for raw analyzer output; handles concurrent writes for feedback storage (FR-9). SQLite has limited concurrency; flexible schema of MongoDB is not needed
Component communication REST over HTTPS, WebSocket, gRPC REST over HTTPS (JSON) Natively supported by browser extensions via fetch; stateless and easy to test (NFR Security, Maintainability). Analysis takes about 10 s, so status polling is enough. WebSocket adds connection management; gRPC needs a proxy (gRPC-Web) in browsers
Static analyzer Semgrep, CodeQL, Bandit Semgrep Multi-language, rule-based, runs on changed files (NFR latency ≤ 10 s). CodeQL builds a code database first, which is heavier; Bandit supports Python only
ML model CodeBERT (fine-tuned), XGBoost on hand-made features, LLM via API CodeBERT with commit context Pre-trained on code, fine-tuned on CVEfixes, runs on our server so code does not leave it. XGBoost is fast but needs manual features (kept as baseline B3); LLM API is costly, slower and harder to reproduce (NFR Reproducibility)

Key Takeaways

  • One ecosystem: Python for backend and ML (FastAPI, Semgrep, CodeBERT), so the model and the API share code and tooling
  • Simple by default: REST over HTTPS and PostgreSQL are chosen to fit the latency and security requirements without extra infrastructure
  • Privacy and reproducibility: a local CodeBERT instead of an external LLM API keeps code on our server and allows fixed-seed experiments
  • Trade-offs accepted: status polling instead of streaming; an on-demand extension instead of automatic webhooks (future work)

Chrome Extension Architecture

Purpose and Scope

This document describes the proposed architecture of the Commit Guard Chrome/Edge extension. The extension identifies supported GitHub commit or pull-request pages, lets users request an analysis, and presents security findings returned by the backend. Vulnerability scanning and ML classification are performed by backend services, not in the browser.

Architecture Overview

Component Responsibilities

Component Responsibility Boundary
GitHub page Displays the selected commit or pull request External website; untrusted page content
Content script Detect supported URLs, extract repository identity and commit SHA where available, observe navigation changes No vulnerability analysis or secret handling
Popup UI Offer Analyze action, show loading/errors, risk level and finding details Presentation and user interactions
Background service worker Validate messages, coordinate extension components, send API requests, handle failures No ML inference or static analysis
FastAPI backend Retrieve Git diffs and metadata, run analysis pipeline, return structured findings Owned by backend/ML team

Proposed User and Data Flow

  1. The user opens a supported GitHub commit or pull-request page.
  2. The content script detects the page type and extracts repository and revision context.
  3. The popup shows the current context and enables the Analyze action when valid.
  4. The user requests analysis. The popup sends a message to the background service worker.
  5. The service worker validates the request and communicates with the FastAPI backend over HTTPS.
  6. The backend retrieves the diff and metadata, runs Semgrep and the ML filtering pipeline, and returns a result or job status.
  7. The extension displays the overall risk level and available findings, explanations, and statuses.
  8. On failure, the extension shows a recoverable error and offers retry where appropriate.

Interface Contracts (Proposed)

Before writing the code, it is necessary to approve the final endpoint paths and data structure with the backend developer. Conceptually, the request can transmit the repository owner and name (repository_owner, repository_name), the commit hash (commit_sha), or the PR identifier (if pull request analysis is supported). The API response may include analysis_id, status, risk_level, and a list of found issues (findings) containing the location, criticality, description, and confidence level of the model.

Manifest V3 and Permissions

  • Use Manifest V3 with a background service worker.
  • Restrict content scripts to required GitHub URL patterns; validate repository and revision identifiers.
  • Prefer the least-privilege permission set. Candidate permissions may include activeTab, scripting, and/or storage only if implementation needs them.
  • Limit host_permissions to the required GitHub pages and the approved backend origin; avoid <all_urls>.
  • Define a Content Security Policy compatible with extension restrictions; do not execute remote scripts.
  • Do not embed API tokens or other secrets in client-side source code. The final manifest.json permissions will be determined during implementation and security review.

Security and Error Handling

  • Treat GitHub page text, commit messages, and backend responses as untrusted data; render as text rather than injecting raw HTML.
  • Use HTTPS for backend requests outside local development.
  • Validate extension messages and API payloads; avoid leaking repository data or credentials in logs.
  • Handle unsupported GitHub URLs, missing SHA, rate limits, network failures, timeouts, and unavailable backend services.
  • Do not store raw diffs or sensitive analysis content persistently unless there is a documented requirement and retention policy.
  • A false-positive filter must not silently hide potentially critical findings without an agreed UX and evaluation policy.

Requirement Traceability

System requirement Extension architecture responsibility
FR-1 Content script detects supported GitHub commit/PR pages and extracts repository/revision context
FR-2 Popup and service worker initiate a request to the backend
FR-7 Popup presents overall risk level supplied by the backend
FR-8 UI presents finding locations, explanations and available confidence/fix guidance
FR-9 Optional feedback UI and submission flow; backend persistence is a dependency
FR-10 Optional JSON/CSV export initiated from the UI
FR-3–FR-6 Backend and ML dependencies; not extension-owned

Priorities and acceptance criteria remain defined in the system-level requirements document. FR-8 and FR-9 are Should, and FR-10 is Could, so they are not all mandatory MVP deliverables.

Verification Plan

Check Proposed verification
GitHub URL detection Test at least 10 supported commit URLs as required by FR-1; separately test PR pages
Message passing Confirm context and analysis actions reach the intended extension components
API communication Use a mock API, then the integrated FastAPI service, including error cases
Results rendering Verify risk levels and findings against known mock responses
Compatibility Test current Chrome and Edge versions
Security Review permissions, HTTPS, secret handling, and unsafe HTML rendering
Recovery Test unsupported URLs, malformed payloads, network failures, and backend timeouts

Open Design Decisions

  • Does the MVP analyze a single commit, a whole PR, or both?
  • Will the popup own the full findings view, or will an extension side panel be used?
  • Is analysis synchronous, or does the backend return a job ID for polling?
  • Which authentication model and backend deployment origin will be used?
  • What exact finding fields and confidence indicators will the backend return?

These decisions should be confirmed with the backend engineer and the UX prototype before frontend implementation.

Supervisor Feedback

# Feedback Action Status
1 Remove the contents slide; merge the title slide with the first proposal slide Applied in the updated presentation In progress
2 Literature table: remove the DOI column, add "Application to our project" Done Completed
3 Add "Project Management and Risks" with a September 2026 - May 2027 timeline Done Completed
4 Add "Supervisor Feedback and Improvements" Done Completed

Project Management & Risks

Timeline (September 2026 – May 2027)

Stage Period
Research and requirements Sep – Oct 2026
System design and UX Oct 2026
Data preparation Nov – Dec 2026
Backend and ML development Oct 2026 – Feb 2027
System integration Feb – Mar 2027
Testing and evaluation Jan – Apr 2027
Finalization Apr – May 2027

Dependencies: data preparation starts after research and requirements; ML development needs a cleaned and labeled dataset; system integration depends on the backend, the extension and the ML components

Risks

Risk Probability Impact Mitigation
Noisy or insufficient training data High High Validate CVE-linked commits; review uncertain labels manually
High false positive rate High High Compare Semgrep-only and ML-filtered results; tune decision thresholds
CodeBERT performance limitations Medium High Limit input size, optimize inference, measure latency
Integration delays Medium Medium Agree on API contracts early; integrate and test incrementally

Repository Layout

commit-guard/
├── README.md                     # Project overview (this file)
├── docs/
│   ├── architecture/             # Chrome extension architecture and sequence diagrams (PNG)
│   └── ...                       # Paper drafts, reports and article notes
├── prototype/
│   ├── README.md                 # Screen descriptions and Figma link
│   └── designs/                  # 11 UI screens (PNG)
├── src/
│   ├── backend/                  # FastAPI app, GitHub client, diff parser
│   ├── extension/                # Browser extension (planned)
│   └── ml/                       # Data preparation, labeling, configs, metrics, notebooks
└── tests/                        # pytest unit tests

Project Tracking & External Links

Local Setup & Testing Instructions

1. Prerequisites

  • Python 3.10 or newer
  • Git

2. Environment Setup

# Clone the repository
git clone https://github.com/oiibar/commit-guard.git
cd commit-guard

# Create and activate a virtual environment
python -m venv .venv

# On Linux / macOS:
source .venv/bin/activate
# On Windows (PowerShell):
.venv\Scripts\Activate.ps1

# Install backend dependencies
pip install -r src/backend/requirements.txt

3. Running Automated Tests

python -m pytest

Expected result: 3 passed.

4. Fetching the Changes of a Commit

python src/backend/fetch_diff.py <owner> <repo> <sha>
# example:
python src/backend/fetch_diff.py oiibar commit-guard 610db04

The script prints the commit message, the changed files with their status and the patch of each file

5. Running the API Server

cd src/backend
uvicorn main:app --reload

Then open http://127.0.0.1:8000/health (expected response: {"status": "ok"}). Interactive API documentation is available at http://127.0.0.1:8000/docs

Requests to the GitHub API without a token are limited to about 60 per hour. Never commit tokens or passwords: keep them in a local .env file, which is ignored by Git

Setup instructions for the ML part and the extension will be added when they are implemented

Git Workflow & Peer Review Policy

  • Direct pushes to main are not allowed. Work is done in dedicated branches: feat/... for code, docs/... for documentation, test/... for tests.
  • One YouTrack issue is one branch and one pull request.
  • Every pull request needs an approval from another team member before merging.
  • Commit and pull request titles follow the format DIPLOMA-<id> <type>(<scope>): <summary>, for example DIPLOMA-33 feat(backend): parse added line numbers from patch.
  • A YouTrack issue is moved to Done only after its pull request is merged.

Limitations

  • Commit Guard filters alerts produced by Semgrep. It does not search for new vulnerabilities on its own, so the recall of the system cannot exceed the recall of Semgrep.
  • Labels derived automatically from CVEfixes are a heuristic and can be noisy; uncertain labels are reviewed manually.
  • The prototype screens use Python code; the final list of supported languages will be fixed during implementation
  • Findings are review hints, not a security verdict. A developer should check the code before making a decision

Astana IT University · 2026

About

Browser extension and FastAPI service that scans GitHub commits with Semgrep and uses a CodeBERT classifier to filter false-positive alerts, using the diff, surrounding code and commit message as context. Bachelor diploma project, AITU 2026–2027.

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages