Repository navigation
Safer incremental backups #244
Description
Status quo
At the moment, incremental backups always use the last update of the backup as a since date, even if the last backup failed. This is documented as a known issue.
Proposed feature
The behavior could be improved by storing the timestamps of the last successful updates in a file, e.g. /.successful-timestamps.json:
{
"some-repo/issues": 1707118232,
"some-repo/pulls": 1707118232,
// ...
}Then failed backups would not break future incremental backups.
Migration
The implementation would need to be backwards-compatible with existing backup stores that don't have this file yet or where the file is incomplete. It could work like this:
- If the file exists and contains the key for the backup source in question, use the timestamp as the
sincetime. - If the file doesn't exist or does not contain the relevant key but an incremental backup was requested, use the old behavior.
- In any case (full or incremental backup), create or update the file with the new timestamps for the next run(s).
Reacted by Preston Van Loon
Activity
josegonzalez commentedon Feb 5, 2024
This project is considered feature complete for the primary maintainer @josegonzalez. If you would like a bugfix or enhancement, pull requests are welcome. Feel free to contact the maintainer for consulting estimates if you'd like to sponsor the work instead.
hkmaly commentedon Jan 16, 2025
I'm not sure why bother with generating separate file. There is already information of when was any specific file updated, in form of filesystem's modification time of the file. And yes, I will try to make the pull request.
Iamrodos commentedon Apr 30, 2026
@josegonzalez I think this can be closed.
This is effectively addressed by #504 (commit b3a8241) in v0.62.0. Each resource now writes its last_update checkpoint after backup_issues/backup_pulls returns, so a failed run leaves the prior timestamp intact and the next incremental run re-fetches from the last successful point.