feat(mirror): incremental LFS pointer discovery in mirrorSync by rremer · Pull Request #38781 · go-gitea/gitea

rremer · 2026-08-04T23:04:21Z

When lfs server is enabled for a mirror repository, each mirror sync attempt will run cat-file over every object in the mirror. This is very expensive for large repositories.

This branch introduces a significant performance improvement for mirrorSync times for larger repositories by:

only running the cat-file check for discovering lfs files over the delta of refs synced during that iteration, as opposed to the whole repository every sync
appending discovered lfs pointers to a new lfs_mirror_pending table to track missing oids across subsequent runs (for instance when a client failed to git lfs push, we want to retry on subsequent syncs)
adds a prometheus metric labelled per repository which counts the number of missing lfs files

I chose to make this a new table instead of adding a column to lfs_meta_object (which currently stores the lfs pointer metadata) for two reasons:

rollback: the behavior this way is that you could roll back to a prior gitea version, and simply inherit the prior behavior where lfs pointer discovery scanned the whole repo every mirrorSync. If it was a column on the existing table, rollback would require reverting the database change
there are quite a few apis for serving lfs files which treat any row/entry in lfs_meta_object as authoritative that the lfs file existed. I would have had to sprinkle conditionals all throughout services/lfs/server.go, repo/view.go,, repo/download.go, api/v1/repo/file.go, probably elsewhere. This change was already big enough.

Using #38153 to track sync times for various steps, this took the 'lfs' step for one of our very large repositories from ~40 minutes on average down to ~20 seconds. This is not round-trip time to the upstream lfs server, this is purely the time taken to cat-file all the objects in the repository with the fastest disk available in our cloud-provider and plenty of compute headroom.

I could see potentially extending this functionality in the future with some configurable backoff algorithm for missing lfs files. At my company, we tend to have branches that stick around for a long time and will basically never get their missing lfs files ever pushed, retrying them every mirrorSync is rude to our lfs upstream. We could default to something like exponential backoff and store the 'last attempted' time in this new table. Again, this change was big enough, so I didn't include that this first round.

Copilot

Pull request overview

This PR optimizes pull-mirror LFS synchronization by switching LFS pointer discovery from a full-repo scan to an incremental scan across newly-updated refs, persisting missing objects for retries, and exposing new Prometheus metrics to observe mirror/LFS state.

Changes:

Add incremental LFS pointer scanning for mirrors (scan only new objects since last successful sync) and persist missing LFS objects in a new lfs_mirror_pending table for retries.
Track mirror ref “watermark” (lfs_last_refs) to support incremental scans across sync iterations.
Add Prometheus metrics for mirror sync step timings, sync status, and per-repository pending LFS object counts.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
services/repository/lfs.go	Extends LFS GC to clean up orphaned `lfs_mirror_pending` rows.
services/mirror/mirror_pull.go	Switches mirror LFS sync to incremental mode and records pending LFS metrics.
services/mirror/mirror_pull_metrics.go	Introduces Prometheus collectors and helpers for mirror sync metrics.
services/mirror/mirror_pull_metrics_test.go	Adds unit tests for the new mirror metrics helpers.
modules/setting/metrics.go	Adds `EnabledMirrorSyncDuration` metrics toggle.
modules/repository/repo.go	Adds incremental mirror LFS sync flow, pending retry logic, and ref watermarking.
modules/repository/lfs_mirror_test.go	Adds tests for new LFS mirror download/pending behaviors.
modules/lfs/pointer_scanner_nogogit.go	Adds `SearchPointerBlobsInRange` (rev-list range scan) for non-gogit builds.
modules/lfs/pointer_scanner_gogit.go	Adds `SearchPointerBlobsInRange` fallback (full scan) for gogit builds.
models/repo/mirror.go	Adds `LFSLastRefs` field to persist last successful ref tips.
models/git/lfs_mirror_pending.go	Adds `lfs_mirror_pending` model + helpers to track pending mirror LFS objects.
models/git/lfs_mirror_pending_test.go	Adds tests for pending-table uniqueness/idempotency behavior.
modelmigration/v1_28/v347.go	Adds migration for `lfs_mirror_pending` table and `lfs_last_refs` column.
modelmigration/migrations.go	Registers the new migration task.
go.mod	Adds/promotes dependencies needed for metrics tests.

Suppressed comments (2)

modules/repository/lfs_mirror_test.go:100

This test mutates the global setting.LFS.StartServer but does not restore it, which can leak state into other tests in this package. Save the original value and defer restoring it.

	unittest.PrepareTestEnv(t)
	setting.LFS.StartServer = true
	require.NoError(t, storage.Init())

modules/repository/lfs_mirror_test.go:128

This test mutates the global setting.LFS.StartServer but does not restore it, which can leak state into other tests in this package. Save the original value and defer restoring it.

	unittest.PrepareTestEnv(t)
	setting.LFS.StartServer = true
	require.NoError(t, storage.Init())

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+			pending, pendingErr := git_model.GetLFSMirrorPendingByRepoID(ctx, m.Repo.ID)
+			if pendingErr != nil {
+				log.Error("SyncMirrors [repo: %-v]: failed to get pending LFS count: %v", m.Repo, pendingErr)
+			} else {
+				recordMirrorLFSPending(m.Repo.OwnerName, m.Repo.Name, len(pending))


+var mirrorSyncStatus = prometheus.NewHistogramVec(prometheus.HistogramOpts{
+	Namespace: "gitea",
+	Subsystem: "mirror",
+	Name:      "sync_status",
+	Help:      "Count of mirror pull sync completions, labeled by success/failure.",


+// Rows for pointers whose git blobs have been garbage-collected (e.g. after a
+// branch deletion) are eventually removed by the gc_lfs cron task, which checks
+// blob reachability. If gc_lfs is disabled, these rows persist and retry
+// indefinitely — a no-op network cost per sync but not harmful.


rremer · 2026-08-05T21:08:57Z

+	unittest.PrepareTestEnv(t)
+	setting.LFS.StartServer = true
+	require.NoError(t, storage.Init())


The callout is legitimate, but the solution is still not great as it's still global state which could affect other tests. Instead I just create a new temporary ContentStore to pass around and avoid this config entirely.

rremer · 2026-08-05T18:43:18Z

+		for _, p := range pending {
+			pointer := lfs.Pointer{Oid: p.Oid, Size: p.Size}
+			pointerSha := git.ComputeBlobHash(objectFormat, []byte(pointer.StringContent()))
+			if !gitRepo.IsObjectExist(ctx, pointerSha.String()) {


I don't agree with this concern, but re-reading this generated code I think it's unnecessary. I could just store the blob's hash in the db table I have during pointer discovery and avoid recomputing the hash.

rremer · 2026-08-05T20:50:15Z

+// SearchPointerBlobsInRange scans objects reachable from headRefs but not
+// excludeRefs for LFS pointer files. For the gogit build this falls back to a
+// full scan since go-git does not easily support rev-list range queries.
+func SearchPointerBlobsInRange(ctx context.Context, repo *git.Repository, headRefs, excludeRefs []string, pointerChan chan<- PointerBlob, errChan chan<- error) {


I do not have a deployment built from gogit to test this functionality in my environments, but this should be a drop-in replacement of the old behavior just with the new interface.

Claude and I couldn't find an equivalent to rev-list ^excludeRef in gogit, which I rely on to find the reachable objects between refs. This incremental lfs mirroring could be implemented for gogit builds in a future pr, but I'd probably start with a feature on go-git to support similar functionality first.

rremer · 2026-08-05T20:54:37Z

 ;; Enable issue by repository metrics; default is false
 ;ENABLED_ISSUE_BY_REPOSITORY = false
+;; Enable mirror sync duration metrics (high-cardinality: owner+repo+step); default is false
+;ENABLED_MIRROR_SYNC_DURATION = false


so this and the EnabledMirrorSyncDuration in metrics.go below may seem like an odd naming for this specific feature of incremental lfs 'pending objects' metrics, but it's because mirror_pull_metrics.go and this config are per-mirror-repo and the same type is on #38153. That PR adds metrics for every step in a mirror pull, and either of these could be merged in either order after a rebase.

rremer · 2026-08-05T22:18:59Z

This is ready for review after addressing all the agent comments.

rremer · 2026-08-11T00:09:14Z

Just keeping this rebased, the migration files keep getting taken in main, so each rebase is incrementing the model filename.

* new repositories still perform full sync, cat-file to discover all LFS pointers and attempt download * subsequent syncs only cat-file what is in the delta of refs * LFS blobs which failed to download are added to a new lfs_last_refs table, which shows the first git ref which mentioned an lfs pointer * new gitea_mirror_lfs_pending_objects guage shows pending/missing LFS blobs by owner/repo Assisted-by: Claude Opus 4.6 Signed-off-by: Royce Remer <royceremer@gmail.com>

GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Aug 4, 2026

github-actions Bot added the type/feature Completely new functionality. Can only be merged if feature freeze is not active. label Aug 4, 2026

rremer force-pushed the mirror-incremental-lfs branch from c048d0d to 04e8caf Compare August 5, 2026 00:16

lunny added performance/speed performance issues with slow downs and removed type/feature Completely new functionality. Can only be merged if feature freeze is not active. labels Aug 5, 2026

lunny requested a lite review from Copilot August 5, 2026 06:47

Copilot started reviewing on behalf of lunny August 5, 2026 06:48 View session

Copilot AI reviewed Aug 5, 2026

View reviewed changes

rremer force-pushed the mirror-incremental-lfs branch from 04e8caf to 35a1235 Compare August 5, 2026 20:48

github-actions Bot added docs-update-needed The document needs to be updated synchronously type/feature Completely new functionality. Can only be merged if feature freeze is not active. labels Aug 5, 2026

rremer commented Aug 5, 2026

View reviewed changes

rremer force-pushed the mirror-incremental-lfs branch 3 times, most recently from 2e784f6 to 89d2a9f Compare August 5, 2026 21:57

silverwind changed the title ~~feat(mirror): incremental LFS pointer discovery in mirrorSync:~~ feat(mirror): incremental LFS pointer discovery in mirrorSync Aug 8, 2026

rremer force-pushed the mirror-incremental-lfs branch 2 times, most recently from 1220fa6 to b83f963 Compare August 11, 2026 00:07

rremer force-pushed the mirror-incremental-lfs branch from b83f963 to dba18b9 Compare August 11, 2026 17:20

Navigation Menu

Uh oh!

Uh oh!

feat(mirror): incremental LFS pointer discovery in mirrorSync - #38781

feat(mirror): incremental LFS pointer discovery in mirrorSync#38781
rremer wants to merge 1 commit into
go-gitea:mainfrom
rremer:mirror-incremental-lfs

rremer commented Aug 4, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

rremer Aug 5, 2026 •

edited

Loading

Uh oh!

rremer Aug 5, 2026

Uh oh!

rremer Aug 5, 2026 •

edited

Loading

Uh oh!

rremer Aug 5, 2026

Uh oh!

rremer commented Aug 5, 2026

Uh oh!

rremer commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

rremer commented Aug 4, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

rremer Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rremer Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

rremer Aug 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rremer Aug 5, 2026

Choose a reason for hiding this comment

Uh oh!

rremer commented Aug 5, 2026

Uh oh!

rremer commented Aug 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

rremer Aug 5, 2026 •

edited

Loading

rremer Aug 5, 2026 •

edited

Loading