Repository navigation
Conversation
… stop Each run numbered an hour's cans from 0 and PUT them unconditionally, and took any hour that was over even if the fastpath was still writing to it. When measurements reach the spool after their hour was uploaded (a fastpath backlog, or the reuploader re-sending old measurements, which keep their uid and so their hour), the next run wrote the same keys again and replaced the earlier cans, losing their measurements from the current view and leaving their jsonl rows pointing at other measurements. Scanning every version under raw/ in ooni-data-eu-fra for 2026 found 35,131 measurements that exist only in replaced versions (Feb 23, Mar 11, Apr 10, Apr 11, Apr 23, Apr 25, Jul 14). - Number a run's cans after the highest index already on S3 for the hour, cc, test and collector, and skip any index whose key exists. - Upload an hour's directory only once nothing has been written to it for quiet_minutes (default 5, in api-uploader.conf). - Delete the POSTs only after the cans are uploaded and their jsonl rows inserted, so a failed upload is retried by the next run instead of losing them. - Keep measurements that arrive during a run for the next one, delete only stale .tmp files, and remove an hour's directory only after it has been empty for a day: removing it right after an upload could land between the fastpath's mkdir and its write. - Lock the spool so a run started by hand cannot overlap the timer's.
Runs ooni_api_uploader.py against the fastpath image production runs, a versioned S3 (MinIO) and ClickHouse 24.8, in the uploader's production runtime (Ubuntu 22.04 packages). It submits real measurements the way the ooniprobe service does and creates the situations that replaced cans in ooni-data-eu-fra: measurements written after their hour was uploaded, a full fastpath queue failing over to the failed-reports bucket, the reuploader (master and production's version) re-sending into uploaded hours, S3 unreachable during an upload, and overlapping runs. Every measurement is then looked up as oonimeasurements does (jsonl -> s3path, linenum) and its body compared; the run fails on any missing or wrong measurement, replaced S3 object, or leftover in the spool or failed bucket. The uploader before this branch fails it (120 of 270 measurements found, 4 objects replaced); this branch passes.
The workflow reads production's reuploader from ooni/backend at the
commit in its image tag (8b35a38f). That commit is only on #1219's
branch, which was squash-merged, so the checkout did not have it
("fatal: invalid object name '8b35a38f'"). Fetch refs/pull/*/head too.
When the fastpath uploader restarted its can index at 0 for an hour it had already uploaded, it replaced earlier .tar.gz/.jsonl.gz cans in ooni-data-eu-fra. Bucket versioning kept the old objects, but the jsonl rows for the measurements in them point at the wrong file, or at a line holding another measurement. scripts/repair-overwritten-cans/repair_cans.py walks the object versions and, step by step (each step is a dry run unless --apply is given): - inventory: lists every version under the affected prefixes - restore: publishes the measurements found only in noncurrent versions as new .n9. cans, never writing to an existing key - fix-jsonl: points those measurements' jsonl rows at the restored cans - verify: checks every measurement is served with its own body - prune: deletes noncurrent versions whose measurements are all served from a current object, only after a successful verify test/test_repair.sh damages a versioned MinIO bucket with the uploader as it was before the fix, runs the repair, and checks that unaffected objects and jsonl rows are left untouched. A workflow runs it on changes to the script.
…reupload In the 2026-10-13/14 outage ooniprobe saved submissions to the failed-reports bucket after the load balancer had answered 504, so probes submitted again. submit_measurement gives each submission a new report_id and measurement_uid, so reuploading those postcans would add measurements fastpath already has under other ids. dedupe_postcans.py identifies a measurement by its content without report_id. plan classifies every postcan: already_ingested (its own uid is in fastpath), retried (fastpath has it under another uid, confirmed against the body its jsonl row points to), repeat_postcan (another postcan holds it), unconfirmed (columns match, no jsonl row yet), reupload, unreadable. apply moves the first three to dedup/ and the unconfirmed ones to hold/ in a separate bucket, copying and checking each before deleting the original. verify checks after the reupload that each reuploaded measurement is in fastpath once, with a jsonl row pointing at its own body. The dedup bucket has to be another bucket: the reuploader reads S3_PREFIX but walks the whole bucket, and deletes whatever fastpath accepts. test/test_dedupe.sh builds every case in the uploader e2e environment and runs plan, apply, the reuploader, the uploader and verify; it also runs in the repair workflow.
aagbsn
marked this pull request as draft
October 20, 2026 15:44
Contributor
Author
|
this is a bit of a monster (generated) script. it has several phrase, a plan, which determines what type of action to perform for each postcan, apply which performs the action, deduplicating and skipping postcans that already have jsonl rows for their uid, and verify, to confirm that each reuploaded measurement appears only once. It needs a careful review to ensure it does what it says on the tin. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
first pass at a repair script first for deduplicating postcans that were reuploaded multiple times.