Bachelor Diploma Project (2026–2027)
Topic: Development of an intelligent system for contextual analysis of Git commits to identify vulnerabilities and reduce false positives using machine learning methods
Institution: Astana IT University (AITU) Student group: SE-2423 Supervisor: Tulebayev Yersultan Bakhytovich Milestone: Pre-Defense 1 (October 2026)
Commit Guard is a browser extension for GitHub backed by a REST API. When a developer opens a commit or pull request page and clicks Analyze, the system checks the code changed by the commit and shows which security findings are likely to be real, instead of a long list of alerts to review one by one
The platform combines three ideas:
- Static analysis as a source of candidates. Semgrep scans the files changed by the commit and produces candidate alerts
- Context-aware classification. A CodeBERT-based classifier decides whether each alert is a true or a false positive, using the context of the change: the code diff, the surrounding code and the commit message
- Feedback loop. Users can mark a result as a false alarm; the feedback is stored for retraining
flowchart LR
U[Developer on a GitHub commit page] --> E[Browser extension]
E -->|repo + SHA| API[FastAPI backend]
API <-->|diff, message, metadata| GH[GitHub REST API]
API --> S[Semgrep]
S -->|candidate alerts| C[Context builder]
C --> M[CodeBERT classifier]
M -->|true / false positive| R[Risk level and explanations]
R --> E
API <--> DB[(PostgreSQL)]
Aim: to reduce false positives of static analysis alerts while preserving vulnerability recall
- Alert noise. SAST tools produce many alerts that are not real problems. In our interviews with six practitioners, the respondents who gave estimates (4 of 6) considered only about 10–30% of automated alerts important
- Time lost on triage. Reviewing alerts takes 10–30 minutes per alert or 1–2 hours per week for the developer on duty
- Weakened protection. Because of the noise, teams raise severity thresholds, ignore notifications and disable rules, so real vulnerabilities can slip through
- No change context. Scanners look at code without the context of the change: what the commit modified and why (diff and commit message)
- Software developers: fast, accurate security feedback on their commits without noise
- DevSecOps and cybersecurity specialists: automated commit-level checks and less manual audit work
- QA and code quality engineers: a security check before code is merged into production branches
- A browser extension (Chrome / Edge) for on-demand analysis of GitHub commit and pull request pages
- A context-aware ML pipeline that filters Semgrep alerts using diffs and commit messages
- A findings view with the overall risk level (Low / Medium / High), highlighted lines, explanations and confidence scores
- A backend REST API (FastAPI) with PostgreSQL storage
- An evaluation report comparing the proposed system with baselines on precision, recall, F1, FPR, MCC, PR-AUC and latency
| Full Name | Student Group | Engineering Role | Responsibilities | GitHub Account |
|---|---|---|---|---|
| Aiymzhan Abilgazy | SE-2423 | Frontend Developer & Data Engineer | Data collection and preprocessing, database design, UI/UX and browser extension | @aiymzhanabilgazy |
| Aibar Tolegenov | SE-2423 | Machine Learning Engineer | False positive reduction model, training and evaluation, baselines and metrics | @oiibar |
| Marzhan Tulebayeva | SE-2423 | System Architect & Backend Engineer | System architecture, FastAPI backend, GitHub integration, system integration and testing | @Marzhan-b |
Academic Supervisor: Tulebayev Yersultan Bakhytovich, Astana IT University (AITU)
In scope for the diploma MVP:
- GitHub commit and pull request pages, analyzed on demand from the extension
- Fetching the commit diff and metadata through the GitHub REST API; private repositories with a user-provided token.
- Semgrep on the changed files; only alerts on lines changed by the commit are kept
- Context extraction: changed lines, surrounding code (tree-sitter), commit message
- Alert classification (true / false positive) with CodeBERT, compared with baselines B1–B3
- Risk level, highlighted findings with explanations, "False alarm" feedback stored in PostgreSQL
- Evaluation on the CVEfixes dataset with a project-based or time-based split
Out of scope (future work):
- Automatic scanning through webhooks or CI
- GitLab and Bitbucket support
- A team dashboard
- Dependency (SCA) alerts
| # | Component | Document | Status |
|---|---|---|---|
| 1 | Project proposal & scope | Project Proposal | ✅ Done |
| 2 | Evidence of relevance (six practitioner interviews) | Presentation; summary | 🟡 Separate document planned |
| 3 | Literature review (20 sources) | Literature Review | ✅ Done |
| 4 | Existing system comparison (5 systems, 8 criteria) | Existing Systems Comparison | ✅ Done |
| 5 | Technology comparison (5 decisions) | Technology Stack & Comparison | ✅ Done |
| 6 | Requirements (10 FR with MoSCoW priorities, 7 NFR) | Requirements, Frontend Requirements | ✅ Done |
| 7 | Architecture diagrams: component decomposition, use case, system context | Chrome Extension Architecture, docs/architecture/ |
✅ Done |
| 8 | UX prototype: 11 screens and Figma | prototype/ |
✅ Done |
| 9 | Verification & evaluation plan | Verification & Evaluation Plan | ✅ Done |
| 10 | Project management & risks | This README, YouTrack | ✅ Done |
| 11 | Supervisor feedback log | Supervisor Feedback | ✅ Done |
| Component | Location | Status |
|---|---|---|
| GitHub client: commit diff and metadata | src/backend/fetch_diff.py |
✅ Done |
| Diff parser: line numbers added by a commit | src/backend/diff_parser.py |
✅ Done |
| Unit tests for the diff parser | tests/test_diff_parser.py |
✅ Done (3 tests) |
FastAPI server with /health |
src/backend/main.py |
✅ Done |
| ML data preparation and labeling (CVEfixes) | src/ml/ |
🟡 In progress |
/analyze endpoint, Semgrep runner, context extraction |
src/backend/ |
⬜ Planned |
| Baselines B1–B3 and CodeBERT classifier | src/ml/ |
⬜ Planned |
| PostgreSQL storage and feedback saving | src/backend/ |
⬜ Planned |
| Browser extension | src/extension/ |
⬜ Planned (UX prototype ready) |
No evaluation results have been measured yet. All numbers in the requirements and the evaluation plan are targets
We held informal, anonymous interviews with six practitioners (five software engineers and one support engineer). The sample is small and the figures are subjective estimates, because the teams do not keep statistics
- Of the respondents who gave estimates (4 of 6), only about 10–30% of automated alerts were considered important
- Triage takes 10–30 minutes per alert or 1–2 hours per week for the developer on duty
- Noise leads to relaxed settings: higher severity thresholds, ignored notifications, disabled rules
- Secrets leaked or almost leaked into repositories in several teams, often noticed by chance
- Code-level (SAST) evidence comes from R2 and R6 (R3 secondhand); dependency alerts (R1, R5) are out of scope of the current version
Problem. Traditional SAST tools generate numerous false-positive alerts and may not consider the context of individual code changes (Git diffs, commit messages). Developers spend significant time on irrelevant warnings, and the noise makes genuine issues harder to prioritize. Commit Guard combines static analysis with contextual machine learning to separate relevant findings from likely false positives. The effectiveness of this approach will be assessed experimentally rather than assumed in advance.
- Analysis and review: Study scientific publications and existing methods for detecting vulnerabilities in commits and reducing false-positive alerts.
- Data work: Collect, label, clean, and prepare a dataset of Git commits, including code changes and available metadata.
- Architecture and backend: Design the database and implement a server-side REST API with Git integration.
- Machine-learning models: Train and evaluate models that classify security alerts using commit context.
- Integration and testing: Combine the browser extension, backend, static analyzer, and ML component; compare results with baseline approaches such as Semgrep alone, rule-based filtering, and XGBoost where feasible.
Commit Guard is planned as a browser extension for GitHub commit and pull-request pages, supported by a backend analysis service. The proposed workflow is:
- A user opens a supported GitHub commit or pull-request page and requests an analysis.
- The extension obtains the relevant commit identifier or page context and sends an analysis request to the backend.
- The backend retrieves and parses Git changes and related metadata.
- Semgrep produces candidate static-analysis findings.
- A contextual ML component assesses the findings using available code-change context.
- The interface displays prioritized findings, their locations, risk indicators, and explanations.
The main evaluation question is:
Can a contextual ML filter reduce false-positive Semgrep alerts on Git commit changes without substantially reducing recall for real vulnerabilities?
The planned comparison includes Semgrep, rule-based filtering, and an XGBoost baseline, where a suitable labeled dataset allows fair comparison, against the proposed Semgrep + contextual ML pipeline.
The evaluation will report precision, recall, F1-score, false-positive measures, and analysis latency. The team will document the dataset, labeling procedure, train/test separation, and relevant limitations.
Source: docs/articles/Commit_Guard_Literature_Synthesis_Review.pdf (20 sources, November 2026). Sources fall into six topics: (A) learning to triage static-analysis alerts, (B) commit context as a security signal, (C) datasets and label quality, (D) learning-based vulnerability detectors, (E) static analysis in practice, (F) foundations: CodeBERT, baselines, metrics.
- Learned filters remove much alert noise. Keeping only CppCheck alerts inside functions that a fine-tuned CodeBERT judged vulnerable raised precision on Big-Vul from 5.9% to 96.2%, while recall fell from 46.9% to 40.7% (Apostolidis et al., 2025). A commit-metadata classifier gave over 99% fewer false positives than Flawfinder at the same recall (Perl et al., 2015).
- Context helps. Method-level context improved warning triage over the warning line alone (Ge et al., 2025). Commit messages and diffs carry security signal that a code snapshot lacks (Perl et al., 2015; Zhou & Sharma, 2017).
- A filter can only remove alerts. Its recall can never exceed the analyser's own, which is the ceiling Commit Guard has with Semgrep.
- Random splits overstate performance. F1 fell from 49% to 9.4% on unseen projects (Chen et al., 2023); cross-project triage is worse than within-project (Ge et al., 2025); on time-ordered real projects no detector exceeded F1 23.3% (Zhang et al., 2023).
- Labels are noisy. Fix-commit heuristics gave 20-71% inaccurate vulnerable labels, and correcting them cut measured precision by 29-80% (Croft et al., 2023). Only 51.7% of sampled CVEfixes labels were correct (Chen et al., 2023). The negative class is also unconfirmed: latent and silent fixes outnumber reported ones by more than 3:1 (Croft et al., 2022).
- Duplicates inflate metrics. Removing cross-set duplicates dropped MCC on D2A from 0.774 to 0.141.
- Narrow evaluation. Studies focus on C/C++ and Java analysers; none uses Semgrep. In triage studies commits only create labels, and MCC, PR-AUC and latency are rarely reported together.
Previous studies show that learned models can reduce static-analysis false alarms, but most use code-only inputs, C/C++ or Java analysers and random or pooled splits. It is insufficiently investigated whether commit context (fix diff and commit message) improves per-alert true/false-positive classification of Semgrep findings on real-world fix commits, under project-held-out evaluation, at matched recall and practical latency.
| Design element | Evidence | Action |
|---|---|---|
| Data split | Random splits inflate results | Project-based split as primary (forks in the same split); time-based split as secondary check |
| TP labelling rule | SAST warnings rarely hit fix lines (line-level recall <= 0.016 in C/C++); fixes often add checks next to the vulnerable line | Measure Semgrep hit rate on fix lines early; test a small line window around inserted hunks; ignore whitespace- and comment-only hunks |
| Label noise | 20-71% wrong positive labels; negatives unconfirmed | Audit a sample of TP/FP labels with two raters (Cohen's kappa); treat FP as "not confirmed", precision as a lower bound |
| Duplicates | Cross-set duplicates inflate MCC | Deduplicate (rule, normalised snippet) across splits; drop conflicting duplicates |
| Model input | CodeBERT: [CLS] text [SEP] code, 512 tokens; truncation loses lines in long functions |
Text: rule ID, message, commit message. Code: window centred on the alert line plus diff hunk |
| Baselines | Tree ensembles are strong and cheap | Keep B1-B3; give XGBoost commit features (additions, hunks, author experience) and SAST features (warning density, severity) |
| Class imbalance | Class-weighted loss was the best remedy on unseen projects | Class-weighted loss; choose threshold on validation set at target recall |
| Metrics | F1 ignores true negatives; MCC and PR-AUC are better under imbalance | Keep metric set, add inspection ratio; accept-all MCC is undefined, set to 0 |
| Recall ceiling | Filtering cannot add recall | Report Semgrep's own recall on fix lines and the share the model retains |
| Latency | 0.02-0.5 s per sample for encoder models on GPU | Measure median end-to-end per-commit time on CPU and GPU |
| Backbone | CodeT5/NatGen and UniXcoder beat CodeBERT in some studies | Keep CodeBERT; optionally run one alternative encoder as sensitivity check |
The literature supports the central design: filter Semgrep instead of replacing it, use a pre-trained encoder with commit context, split by project, report MCC and false-alarm reduction at matched recall. The main risk is in labelling: the TP/FP rule inherits fix-commit noise, and strict line matching may give a small positive class.
We compared five existing systems that reduce noise in security checks or analyze commits, and added Commit Guard as the sixth column. Two of the systems are commercial platforms with free tiers, one is a commercial platform without one, and two are research tools
| Evaluation Criteria | Semgrep Assistant | GitGuardian | GitHub Advanced Security | ZeroFalse | VulCurator | Commit Guard (proposed) |
|---|---|---|---|---|---|---|
| System Type | Commercial platform built on top of open-source Semgrep CE | Commercial platform | Commercial platform | Research prototype (arXiv preprint 2510.02534) | Research tool (ESEC/FSE 2022, demo track) | Research prototype (diploma project) |
| Analyzed Objects | Semgrep findings: code, dependencies, and secrets | Secrets in commits, pull requests, and repositories | Code via CodeQL, secrets via secret scanning, dependencies via Dependabot | Static analysis findings | Commits: whether a commit fixes a vulnerability | Semgrep findings in a GitHub commit or pull request |
| Noise Reduction Mechanism | ML classification of findings as true or false positives with confidence scoring | FP Remover ML model (restricted to generic secrets) | Focus on detection and automated fixes (Autofix); no dedicated AI noise triage found in reviewed sources | LLM-based finding evaluation | Not a noise filter: classifies commits | CodeBERT classifies alerts as true or false positives using commit context |
| Context Awareness | Stores developer feedback as project context | Partial | Partial | Data flow traces, code context, and CWE knowledge | Commit messages, code diffs, and task/issue descriptions | Code diff, surrounding code, commit message |
| Decision Explanation | Natural language explanations | Risk score (0–100) with explanation | Not specified in open sources | Not specified in open sources | Not specified (classifier output) | Highlighted lines, risk level, confidence score, suggested fix |
| Claimed Efficiency | ~60% triage automation; >96% decision agreement rate (vendor data) | Up to 50% false positives removed, 0.3% true positives missed (vendor data) | No data found | Fewer false positives while maintaining coverage (authors) | Up to 16.1% better F1 than baselines (authors) | Target: ≥ 30% lower FPR, recall drop ≤ 5 pp (to be measured) |
| Availability / Pricing | Free for teams up to 10 contributors; paid plans start at $30/month per contributor | Free plan for up to 25 developers; custom enterprise pricing | Paid service (~$19 to $30/month per active committer) | Not specified in open sources | Open-source on GitHub | Not decided yet |
| Trigger Point / Stage | After scan in CI/CD | Commit and pull request checks | Push protection blocks pushes with secrets; code scanning on pull requests | After analyzer run (post-processing) | After the fact (inspects existing commits) | On demand on GitHub commit and PR pages |
Data from vendor documentation and publications; pricing checked in October 2026. Efficiency figures are claims of vendors or authors. The values for Commit Guard are design goals, not measured results
- Noise filtering exists, but each tool covers its own scope: Semgrep Assistant works on Semgrep findings, and FP Remover only on generic secrets
- VulCurator classifies whether a commit fixes a vulnerability, but it does not filter static analysis alerts
- Commit Guard filters Semgrep alerts using the context of the commit and shows the result on demand on the GitHub commit page
- Price is not the difference: Semgrep and GitGuardian have free plans for small teams. The difference to be tested is using the commit context as a signal for alert filtering
Priorities use MoSCoW: Must (the project fails without it), Should (important, but a working product could exist without it), Could (only if time remains).
| ID | Requirement | Priority | Acceptance criteria |
|---|---|---|---|
| FR-1 | The extension detects a GitHub commit or pull request page and extracts the repository and commit SHA. | Must | Works on 10 test commit URLs without errors. |
| FR-2 | The extension sends an analysis request to the REST API backend. | Must | The request reliably reaches the backend. |
| FR-3 | The backend fetches the commit diff and metadata via the GitHub REST API. | Must | The diff matches the real GitHub diff for 10 test commits. |
| FR-4 | Context extraction: code changes, surrounding code and commit message. | Must | Context is extracted for at least 90% of changed functions in supported languages. |
| FR-5 | The Semgrep static analyzer is run to obtain the initial alerts. | Must | The analyzer runs and returns alerts for all supported languages. |
| FR-6 | The ML model classifies each alert as true positive or false positive. | Must | Meets the ML evaluation criteria (see the Verification & Evaluation Plan). |
| FR-7 | The system outputs an overall commit risk level (Low / Medium / High). | Must | A risk level is returned for 100% of analyzed commits. |
| FR-8 | Vulnerable code lines are highlighted with an explanation, confidence score and suggested fix. | Should | Each finding shows the vulnerability type, a brief description, the confidence score and a suggested fix. |
| FR-9 | User feedback collection (Correct / False alarm). | Should | Feedback is saved to the database for model retraining. |
| FR-10 | Export of analysis results (JSON / CSV). | Could | The user can download a report from the extension UI. |
| ID | Quality attribute | Requirement | Priority | Acceptance criteria |
|---|---|---|---|---|
| NFR-1 | Performance | Fast processing for typical commits (up to 500 changed lines). | Must | Median latency up to 10 s; 95th percentile up to 30 s. |
| NFR-2 | Security | Secure data transfer over HTTPS; no API tokens in logs. | Must | Passes the security review checklist. |
| NFR-3 | Accuracy | Significant reduction of the false positive rate (FPR). | Must | FPR reduced by at least 30% versus the raw analyzer at similar recall. |
| NFR-4 | Compatibility | Cross-browser support for the extension. | Must | Extension tests pass on the latest Chrome and Edge. |
| NFR-5 | Usability | Intuitive first-time user experience. | Should | At least 80% task success rate; SUS score at least 68. |
| NFR-6 | Maintainability | Modular architecture and API documentation. | Should | Unit test coverage at least 60%. |
| NFR-7 | Reproducibility | Reproducible ML models and deployment. | Should | Fixed random seeds, versioned data, Dockerized setup. |
YouTrack task DIPLOMA-28. Status: proposed specification; implementation and acceptance tests pending. Scope: Chrome/Edge extension UI and its integration boundaries.
This document extracts frontend responsibilities from the Commit Guard system-level requirements presented at the first pre-defense. It retains the original FR identifiers, priorities, and acceptance criteria to ensure traceability. It does not claim that the requirements have been implemented or tested.
The extension is intended to operate on supported GitHub commit and pull-request pages, initiate analysis via the backend REST API, and present analysis results. The backend is responsible for retrieving diffs and metadata, extracting code context, running Semgrep, and performing ML classification.
The following functional and non-functional requirements are derived from the system-level Requirements Specification documented in Requirements. Only requirements relevant to the frontend are included, while the original requirement IDs and priorities are preserved for consistency and traceability.
| Original ID | Requirement | Priority | Acceptance criteria from project specification | Frontend responsibility |
|---|---|---|---|---|
| FR-1 | Extension detects GitHub commit/PR page and extracts repo/SHA. | Must | Works on 10/10 test commit URLs without errors. | Detect supported pages; extract repository and commit identifier; handle unsupported URLs |
| FR-2 | Extension sends an analysis request to the REST API backend. | Must | Request reliably reaches the server. | Initiate the request and handle its loading, success, and failure outcomes. |
| FR-7 | Output overall commit risk level (Low / Medium / High). | Must | Risk score returned for 100% of analyzed commits. | Display the risk level returned by the backend; backend owns score generation. |
| FR-8 | Highlight vulnerable code lines with explanation, confidence score and suggested fix. | Should | Display vulnerability type and brief description. | Render finding locations and the available explanation, confidence and fix information. |
| FR-9 | User feedback collection (Correct / False Alert). | Should | Feedback saved to DB for model retraining. | DB for model retraining. Capture and submit the user's selection; backend owns persistence and retraining processes. |
| FR-10 | Export analysis results (JSON / CSV). | Could | User can download report via extension UI. | Offer download controls for supported result formats. |
| Original ID | Requirement | Priority | Original acceptance criteria | Frontend dependency |
|---|---|---|---|---|
| FR-3 | Backend fetches commit diff and metadata via GitHub REST API. | Must | Diff matches real GitHub diff for 10 test commits. | The UI supplies a valid repository/commit reference. |
| FR-4 | Context extraction (code changes, surrounding code, commit message). | Must | Context extracted for ≥ 90% of functions in supported languages. | Context is processed by backend services. |
| FR-5 | Execution of Semgrep static analyzer to fetch initial alerts. | Must | Analyzer outputs alerts for all supported languages. | The UI presents returned findings. |
| FR-6 | ML model classifies alerts as True Positive or False Positive. | Must | Meets ML evaluation criteria. | The UI displays available classification output. |
The source slides assign NFRs by quality attribute rather than by numbered ID. The IDs below are local document identifiers, not replacements for the system specification.
| Local ID | Quality attribute | System-level requirement | Priority | Acceptance criteria from project specification | Frontend relevance |
|---|---|---|---|---|---|
| FE-NFR-01 | Performance | Fast processing for typical commits (≤ 500 changed lines). | Must | Median latency ≤ 10s; 95th percentile ≤ 30s. | Keep the interface responsive and show progress while the backend processes the request; latency is measured end-to-end. |
| FE-NFR-02 | Security | Secure data transfer over HTTPS; no API tokens in logs. | Must | Pass Security Review Checklist. | Use secure API communication; avoid logging tokens or sensitive content. |
| FE-NFR-03 | Accuracy | Significant reduction in False Positive Rate (FPR). | Must | FPR reduced by ≥ 30% vs. raw analyzer at similar recall. | Display results accurately; the ML/backend evaluation owns the FPR target. |
| FE-NFR-04 | Compatibility | Cross-browser support for extension. | Must | Extension tests pass on latest Chrome & Edge. | Test extension functionality on both browsers. |
| FE-NFR-05 | Usability | Intuitive first-time user experience. | Should | ≥ 80% task success rate; SUS score ≥ 68. | Provide understandable controls, statuses, and findings; participate in usability testing. |
| FE-NFR-06 | Maintainability | Modular architecture & API documentation. | Should | Unit test code coverage ≥ 60%. | Use separable UI components and documented API contracts; track frontend test coverage. |
| FE-NFR-07 | Reproducibility | Reproducible ML models and deployment. | Should | Fixed random seeds, versioned data, Dockerized setup | Primarily an ML/deployment requirement; frontend should document its build and configuration during implementation. |
- The user opens a supported GitHub commit or pull-request page.
- The extension identifies the repository and relevant commit reference (FR-1).
- The user starts an analysis and the extension submits a backend request (FR-2).
- The interface waits for the response while remaining usable (FE-NFR-01).
- The interface displays the overall risk level (FR-7) and, when available, detailed findings (FR-8).
- Optionally, the user submits feedback (FR-9) or exports results (FR-10). Open design decision: The method for selecting a commit on pull-request pages, the exact API schema, and the location of code highlighting will be defined in DIPLOMA-29 and DIPLOMA-30. They are not finalized by these requirements.
- FR-1 is verified against 10 supported GitHub commit URLs.
- FR-2 is verified with a backend test endpoint and error handling scenarios.
- FR-7 displays the backend risk level for valid analysis results.
- FR-8 displays vulnerability type and description; extended fields are checked if supported by the API.
- FR-9 feedback is persisted by the backend when the feature is implemented.
- FR-10 downloads valid JSON and CSV files when the feature is implemented.
- Extension compatibility tests pass on the latest Chrome and Edge.
- Security review verifies HTTPS use and absence of token logging.
- Usability testing checks task success and SUS against the specified targets.
- Frontend test coverage is measured against the project's maintainability target.
- Source: Commit Guard pre-defense presentation, Requirements Specification slides (FR-1 through FR-10 and Non-Functional Requirements).
- Related project documentation: Requirements (system-wide requirements).
- Next design tasks: DIPLOMA-29 (Chrome extension architecture) and DIPLOMA-30 (frontend system design).
- Out of scope for this task: Implementing the extension, backend GitHub fetching, Semgrep execution, ML model training, and proving the system-wide FPR target. This file documents requirements, not completed functionality. Numerical thresholds and priorities reflect the supplied project presentation and should be revised only through team/supervisor agreement.
| Level | What is tested |
|---|---|
| Unit | Parser, context extractor, risk scorer |
| Integration | Extension, API, GitHub API, analyzer, ML model |
| End-to-end | Full flow on real commit pages |
| Usability | 5 real users complete four scenarios; task success rate and SUS questionnaire |
| Non-functional | Response time (median and 95th percentile), HTTPS and no tokens in logs, Chrome and Edge compatibility |
- Data: CVEfixes (vulnerabilities and their fixes from open-source software). Duplicates and noisy labels are removed.
- Labeling: Semgrep is run on the pre-fix version of each CVEfixes commit. Alerts on lines changed by the fix are labeled true positives, the other alerts false positives.
- Split: project-based or time-based, not random, to prevent data leakage.
- Limitations: labels come from a heuristic and are approximate; recall is measured only over alerts that Semgrep raises.
| Metric | Formula | Meaning |
|---|---|---|
| Recall | TP / (TP + FN) | Of all real vulnerabilities, how many are found |
| Precision | TP / (TP + FP) | Of the alerts shown, how many are real |
| F1 | 2 x P x R / (P + R) | Balance of precision and recall |
| FPR | FP / (FP + TN) | Of all false alarms, how many are still shown |
| MCC | (TP x TN - FP x FN) / sqrt((TP+FP)(TP+FN)(TN+FP)(TN+FN)) | Reliable on imbalanced data (-1 to +1) |
| PR-AUC | Area under the precision-recall curve | Trade-off across thresholds |
| Latency | Time to analyze one commit | Median and 95th percentile |
| ID | Description |
|---|---|
| B1 | Accepts all Semgrep alerts |
| B2 | Semgrep plus a simple rule (for example ignore alerts in test files) |
| B3 | XGBoost on hand-made features |
| Proposed | Semgrep plus CodeBERT with commit context |
Same test set and Semgrep rules for every system; fixed random seeds; 3-5 runs reported as mean and standard deviation; same hardware for latency; runs with and without commit context.
| Criterion | Target |
|---|---|
| False positive reduction | At least 30% lower FPR than B1, recall drop at most 5 percentage points |
| Latency | Median up to 10s; 95th percentile up to 30s |
| Usability | SUS >= 68 |
| Functional compliance | 100% of Must requirements pass their acceptance criteria |
| NFRs | HTTPS, no tokens in logs; Chrome and Edge tests; unit test coverage at least 60%; Docker and fixed seeds |
Each decision below is linked to our requirements.
| Layer | Technology |
|---|---|
| Browser extension | TypeScript, HTML/JS, Manifest V3 (Chrome, Edge) |
| Backend & API | Python 3, FastAPI, Uvicorn |
| Source data | GitHub REST API |
| Static analysis | Semgrep |
| Code parsing | tree-sitter |
| Machine learning | CodeBERT (fine-tuned), XGBoost baseline |
| Training data | CVEfixes |
| Database | PostgreSQL |
| Testing | pytest |
| Design | Figma |
| Project tracking | YouTrack, Git & GitHub |
| Decision | Alternatives considered | Selected | Why (linked to requirements) |
|---|---|---|---|
| Backend framework | FastAPI, Django REST Framework, Flask | FastAPI | Same Python ecosystem as the ML pipeline and Semgrep; async I/O for GitHub API calls (NFR Performance); automatic OpenAPI docs (NFR Maintainability). Django adds admin/ORM overhead we do not need; Flask needs extra libraries for validation |
| Database | PostgreSQL, SQLite, MongoDB | PostgreSQL | Relational data (users, commits, findings, feedback) with ACID guarantees; JSONB for raw analyzer output; handles concurrent writes for feedback storage (FR-9). SQLite has limited concurrency; flexible schema of MongoDB is not needed |
| Component communication | REST over HTTPS, WebSocket, gRPC | REST over HTTPS (JSON) | Natively supported by browser extensions via fetch; stateless and easy to test (NFR Security, Maintainability). Analysis takes about 10 s, so status polling is enough. WebSocket adds connection management; gRPC needs a proxy (gRPC-Web) in browsers |
| Static analyzer | Semgrep, CodeQL, Bandit | Semgrep | Multi-language, rule-based, runs on changed files (NFR latency ≤ 10 s). CodeQL builds a code database first, which is heavier; Bandit supports Python only |
| ML model | CodeBERT (fine-tuned), XGBoost on hand-made features, LLM via API | CodeBERT with commit context | Pre-trained on code, fine-tuned on CVEfixes, runs on our server so code does not leave it. XGBoost is fast but needs manual features (kept as baseline B3); LLM API is costly, slower and harder to reproduce (NFR Reproducibility) |
- One ecosystem: Python for backend and ML (FastAPI, Semgrep, CodeBERT), so the model and the API share code and tooling
- Simple by default: REST over HTTPS and PostgreSQL are chosen to fit the latency and security requirements without extra infrastructure
- Privacy and reproducibility: a local CodeBERT instead of an external LLM API keeps code on our server and allows fixed-seed experiments
- Trade-offs accepted: status polling instead of streaming; an on-demand extension instead of automatic webhooks (future work)
This document describes the proposed architecture of the Commit Guard Chrome/Edge extension. The extension identifies supported GitHub commit or pull-request pages, lets users request an analysis, and presents security findings returned by the backend. Vulnerability scanning and ML classification are performed by backend services, not in the browser.
| Component | Responsibility | Boundary |
|---|---|---|
| GitHub page | Displays the selected commit or pull request | External website; untrusted page content |
| Content script | Detect supported URLs, extract repository identity and commit SHA where available, observe navigation changes | No vulnerability analysis or secret handling |
| Popup UI | Offer Analyze action, show loading/errors, risk level and finding details | Presentation and user interactions |
| Background service worker | Validate messages, coordinate extension components, send API requests, handle failures | No ML inference or static analysis |
| FastAPI backend | Retrieve Git diffs and metadata, run analysis pipeline, return structured findings | Owned by backend/ML team |
- The user opens a supported GitHub commit or pull-request page.
- The content script detects the page type and extracts repository and revision context.
- The popup shows the current context and enables the Analyze action when valid.
- The user requests analysis. The popup sends a message to the background service worker.
- The service worker validates the request and communicates with the FastAPI backend over HTTPS.
- The backend retrieves the diff and metadata, runs Semgrep and the ML filtering pipeline, and returns a result or job status.
- The extension displays the overall risk level and available findings, explanations, and statuses.
- On failure, the extension shows a recoverable error and offers retry where appropriate.
Before writing the code, it is necessary to approve the final endpoint paths and data structure with the backend developer. Conceptually, the request can transmit the repository owner and name (repository_owner, repository_name), the commit hash (commit_sha), or the PR identifier (if pull request analysis is supported). The API response may include analysis_id, status, risk_level, and a list of found issues (findings) containing the location, criticality, description, and confidence level of the model.
- Use Manifest V3 with a background service worker.
- Restrict content scripts to required GitHub URL patterns; validate repository and revision identifiers.
- Prefer the least-privilege permission set. Candidate permissions may include activeTab, scripting, and/or storage only if implementation needs them.
- Limit host_permissions to the required GitHub pages and the approved backend origin; avoid <all_urls>.
- Define a Content Security Policy compatible with extension restrictions; do not execute remote scripts.
- Do not embed API tokens or other secrets in client-side source code. The final manifest.json permissions will be determined during implementation and security review.
- Treat GitHub page text, commit messages, and backend responses as untrusted data; render as text rather than injecting raw HTML.
- Use HTTPS for backend requests outside local development.
- Validate extension messages and API payloads; avoid leaking repository data or credentials in logs.
- Handle unsupported GitHub URLs, missing SHA, rate limits, network failures, timeouts, and unavailable backend services.
- Do not store raw diffs or sensitive analysis content persistently unless there is a documented requirement and retention policy.
- A false-positive filter must not silently hide potentially critical findings without an agreed UX and evaluation policy.
| System requirement | Extension architecture responsibility |
|---|---|
| FR-1 | Content script detects supported GitHub commit/PR pages and extracts repository/revision context |
| FR-2 | Popup and service worker initiate a request to the backend |
| FR-7 | Popup presents overall risk level supplied by the backend |
| FR-8 | UI presents finding locations, explanations and available confidence/fix guidance |
| FR-9 | Optional feedback UI and submission flow; backend persistence is a dependency |
| FR-10 | Optional JSON/CSV export initiated from the UI |
| FR-3–FR-6 | Backend and ML dependencies; not extension-owned |
Priorities and acceptance criteria remain defined in the system-level requirements document. FR-8 and FR-9 are Should, and FR-10 is Could, so they are not all mandatory MVP deliverables.
| Check | Proposed verification |
|---|---|
| GitHub URL detection | Test at least 10 supported commit URLs as required by FR-1; separately test PR pages |
| Message passing | Confirm context and analysis actions reach the intended extension components |
| API communication | Use a mock API, then the integrated FastAPI service, including error cases |
| Results rendering | Verify risk levels and findings against known mock responses |
| Compatibility | Test current Chrome and Edge versions |
| Security | Review permissions, HTTPS, secret handling, and unsafe HTML rendering |
| Recovery | Test unsupported URLs, malformed payloads, network failures, and backend timeouts |
- Does the MVP analyze a single commit, a whole PR, or both?
- Will the popup own the full findings view, or will an extension side panel be used?
- Is analysis synchronous, or does the backend return a job ID for polling?
- Which authentication model and backend deployment origin will be used?
- What exact finding fields and confidence indicators will the backend return?
These decisions should be confirmed with the backend engineer and the UX prototype before frontend implementation.
| # | Feedback | Action | Status |
|---|---|---|---|
| 1 | Remove the contents slide; merge the title slide with the first proposal slide | Applied in the updated presentation | In progress |
| 2 | Literature table: remove the DOI column, add "Application to our project" | Done | Completed |
| 3 | Add "Project Management and Risks" with a September 2026 - May 2027 timeline | Done | Completed |
| 4 | Add "Supervisor Feedback and Improvements" | Done | Completed |
| Stage | Period |
|---|---|
| Research and requirements | Sep – Oct 2026 |
| System design and UX | Oct 2026 |
| Data preparation | Nov – Dec 2026 |
| Backend and ML development | Oct 2026 – Feb 2027 |
| System integration | Feb – Mar 2027 |
| Testing and evaluation | Jan – Apr 2027 |
| Finalization | Apr – May 2027 |
Dependencies: data preparation starts after research and requirements; ML development needs a cleaned and labeled dataset; system integration depends on the backend, the extension and the ML components
| Risk | Probability | Impact | Mitigation |
|---|---|---|---|
| Noisy or insufficient training data | High | High | Validate CVE-linked commits; review uncertain labels manually |
| High false positive rate | High | High | Compare Semgrep-only and ML-filtered results; tune decision thresholds |
| CodeBERT performance limitations | Medium | High | Limit input size, optimize inference, measure latency |
| Integration delays | Medium | Medium | Agree on API contracts early; integrate and test incrementally |
commit-guard/
├── README.md # Project overview (this file)
├── docs/
│ ├── architecture/ # Chrome extension architecture and sequence diagrams (PNG)
│ └── ... # Paper drafts, reports and article notes
├── prototype/
│ ├── README.md # Screen descriptions and Figma link
│ └── designs/ # 11 UI screens (PNG)
├── src/
│ ├── backend/ # FastAPI app, GitHub client, diff parser
│ ├── extension/ # Browser extension (planned)
│ └── ml/ # Data preparation, labeling, configs, metrics, notebooks
└── tests/ # pytest unit tests
- GitHub repository: https://github.com/oiibar/commit-guard
- YouTrack board: https://commit-guard.youtrack.cloud/projects/DIPLOMA (project key:
DIPLOMA) - Figma prototype: https://www.figma.com/design/VoR6uSJm0GcrpxgoARPnIx/Commit-Guard?node-id=0-1&t=CxOapwv2GMFGicDK-1
- Python 3.10 or newer
- Git
# Clone the repository
git clone https://github.com/oiibar/commit-guard.git
cd commit-guard
# Create and activate a virtual environment
python -m venv .venv
# On Linux / macOS:
source .venv/bin/activate
# On Windows (PowerShell):
.venv\Scripts\Activate.ps1
# Install backend dependencies
pip install -r src/backend/requirements.txtpython -m pytestExpected result: 3 passed.
python src/backend/fetch_diff.py <owner> <repo> <sha>
# example:
python src/backend/fetch_diff.py oiibar commit-guard 610db04The script prints the commit message, the changed files with their status and the patch of each file
cd src/backend
uvicorn main:app --reloadThen open http://127.0.0.1:8000/health (expected response: {"status": "ok"}). Interactive API documentation is available at http://127.0.0.1:8000/docs
Requests to the GitHub API without a token are limited to about 60 per hour. Never commit tokens or passwords: keep them in a local
.envfile, which is ignored by Git
Setup instructions for the ML part and the extension will be added when they are implemented
- Direct pushes to
mainare not allowed. Work is done in dedicated branches:feat/...for code,docs/...for documentation,test/...for tests. - One YouTrack issue is one branch and one pull request.
- Every pull request needs an approval from another team member before merging.
- Commit and pull request titles follow the format
DIPLOMA-<id> <type>(<scope>): <summary>, for exampleDIPLOMA-33 feat(backend): parse added line numbers from patch. - A YouTrack issue is moved to Done only after its pull request is merged.
- Commit Guard filters alerts produced by Semgrep. It does not search for new vulnerabilities on its own, so the recall of the system cannot exceed the recall of Semgrep.
- Labels derived automatically from CVEfixes are a heuristic and can be noisy; uncertain labels are reviewed manually.
- The prototype screens use Python code; the final list of supported languages will be fixed during implementation
- Findings are review hints, not a security verdict. A developer should check the code before making a decision
Astana IT University · 2026

