feat!: create the cache deletion job instead of the seeding job (MAPCO-11265) by almog8k · Pull Request #119 · MapColonies/overseer

almog8k · 2026-08-12T16:28:15Z

Question	Answer
Bug fix	✖
New feature	✔
Breaking change	✔
Deprecations	✔
Documentation	✖
Tests added	✔
Chore	✖

Related issues: MAPCO-11265 (epic MAPCO-11261)

Further information:

Replaces the MapProxy tile-cache seeding job with a cache deletion job. On ingestion finalize, overseer now tells the cleaner worker which Redis tile keys to drop instead of driving mapproxy-seed; tiles repopulate lazily on cache miss.

Cleaner resolves its strategy from a combined job+task token, so the job type selects the behaviour while the task type stays tiles-deletion:

flow	job type	task params
update	`Update_Delete_Cache`	tile ranges over the ingested footprint, z0..maxZoom
swap-update	`Swap_Delete_Cache`	whole-prefix wipe + `delaySeconds`

The Redis key prefix is composed as ${cacheName}_${grid} (mapproxy loader.py's rule) because mapproxy-api reports neither a prefix nor a grid. Verified against a production key and pinned as a regression test. GEODETIC_GRIDS therefore gates construction: footprintToTileRanges only implements the geodetic grid, and against any other it would return plausible ranges for the wrong tiles.

Not deployable on its own. job-tracker has no handler or config entry for the two new job types — getJobHandler throws BadRequestError for anything outside its switch — so it would reject every task event for these jobs. Ticket pending. It also completes a job on completedTasks === taskCount, which needs a decision since tasks are streamed after the job is created. Cleaner work is MAPCO-11263/11264; both dependency pins are CI tarballs to be swapped for published versions before production (MAPCO-11267).

BREAKING CHANGE: overseer no longer creates seeding jobs. Ingestion finalize now creates an Update_Delete_Cache or Swap_Delete_Cache job, and the Ingestion_Seed job type is no longer produced.

Adds the Update_Delete_Cache / Swap_Delete_Cache job types and the cacheDeletion task config, wired through to the chart in the same change. Pins raster-shared to the build exporting GEODETIC_GRIDS.

Composes the redis key prefix from the mapproxy cache name and the configured grid, emits a prefix-wipe task for swap-update and streamed range tasks for update, and fails a partially-populated job explicitly.

…O-11265) Both ingestion finalize handlers now create an Update_Delete_Cache or Swap_Delete_Cache job instead of a seeding job, and SeedingJobCreator with its SeedMode and Seed* param types is removed. BREAKING CHANGE: overseer no longer creates seeding jobs.

…1265) The previous toMatchObject assertion used an empty parameters object, which matches anything, so ingestionJobId and ingestionJobType were unasserted. Verified the new assertion fails when they are removed.

razbroc · 2026-08-23T08:19:48Z

        },
+        "cacheDeletion": {
+          "type": "CACHE_DELETION_TASK_TYPE",
+          "grid": "CACHE_DELETION_GRID",


Discussed with Shlomi: to prevent maintaining multiple sources of truth, we shouldn't force a static grid on cache deletion.

We've updated the mapproxy-api endpoint (GET /layer/{layerName}/{cacheType}) to return the grid configuration. Since this service already calls that endpoint for the layer details, please consume the grid from that response instead.

(Check the latest mapproxy-api PR for the updated contract).

razbroc · 2026-08-23T08:38:46Z

+          "maxRangesPerTask": { "__name": "CACHE_DELETION_MAX_RANGES_PER_TASK", "__format": "number" },
+          "taskBatchSize": { "__name": "CACHE_DELETION_TASK_BATCH_SIZE", "__format": "number" },
+          "gracefulReloadMaxSeconds": { "__name": "CACHE_DELETION_GRACEFUL_RELOAD_MAX_SECONDS", "__format": "number" },
+          "reloadWindowMarginSeconds": { "__name": "CACHE_DELETION_RELOAD_WINDOW_MARGIN_SECONDS", "__format": "number" }


I reviewed the need for this variable/mechanism and it seems redundant. Let me know if I'm missing an edge case.

According to the in-code comments, this handles a scenario where a pod hasn’t reloaded the new mapproxy-api config yet and is still serving the previous layer version. However, for cache invalidation/deletion during a swap, this shouldn't be an issue since both the layer name and grid remain identical to the previous version.

Unless there’s another scenario this covers, we can safely remove it.

You are right about the layer name and grid- they both remain identical, but the path to the tiles(display path is changed)- so the maproxinator reloads all the pods of mapproxy in result to this.

razbroc · 2026-08-23T08:40:25Z

-    super(msg);
-    this.name = SeedJobCreationError.name;
-    this.stack = err.stack;
+export class UnsupportedGridError extends Error {


Referencing the previous comment, this mechanism won't be needed if we get rid of the grid configuration.

The mechanism is needed. Explained in previous comment.

razbroc · 2026-08-23T08:43:00Z

@@ -60,6 +62,10 @@ export interface JobConfig {

 export interface IngestionJobsConfig {
  seed: JobConfig | undefined;


Will be removed in separated pr: https://mapcolonies.atlassian.net/browse/MAPCO-11266

razbroc · 2026-08-23T08:43:13Z


 export interface IngestionJobsConfig {
  seed: JobConfig | undefined;
+  /** job type for the update flow's cache deletion; selects cleaner's range-deletion strategy */


I don't like the comments

Comments removed.

razbroc · 2026-08-23T10:33:14Z

+    });
+  }
+
+  /**


This is self-explanatory

Remove redundant comment.

razbroc · 2026-08-23T10:58:10Z

+    const cacheName = await this.mapproxyClient.getRedisCacheName({ layerName, cacheType: LayerCacheType.REDIS });
+    const prefix = `${cacheName}_${this.taskConfig.grid}`;
+
+    logger.info({ msg: 'Composed redis key prefix', cacheName, grid: this.taskConfig.grid, prefix });


razbroc · 2026-08-23T10:58:43Z

+  }
+
+  /**
+   * Tasks are streamed to avoid memory overhead. The job is created with its first batch


razbroc · 2026-08-23T11:09:00Z

+  }
+}
+
+/**


self explanatory

Actually this comment is relevant.

razbroc · 2026-08-23T11:09:31Z

i didn't examine tests. waiting on the mapproxy api refactor.

…m mapproxy

almog8k added 6 commits August 12, 2026 12:20

feat: add range-capped tile batch generator (MAPCO-11265)

ee96837

feat: add cache-deletion job and task configuration (MAPCO-11265)

7447389

Adds the Update_Delete_Cache / Swap_Delete_Cache job types and the cacheDeletion task config, wired through to the chart in the same change. Pins raster-shared to the build exporting GEODETIC_GRIDS.

feat: add cache deletion job creator (MAPCO-11265)

ab2e16c

Composes the redis key prefix from the mapproxy cache name and the configured grid, emits a prefix-wipe task for swap-update and streamed range tasks for update, and fails a partially-populated job explicitly.

refactor: remove redundant comments in CacheDeletionJobCreator tests

9adaa8d

mapcolonies-devops assigned almog8k Aug 12, 2026

razbroc requested changes Aug 23, 2026

View reviewed changes

almog8k added 5 commits September 3, 2026 15:37

feat: remove grid configuration and update related logic to fetch fro…

3dcb29d

…m mapproxy

chore: remove redundant comments

d20c15f

chore: remove redundant comments in CacheDeletionJobCreator

64508f6

feat: update mc-utils and raster-shared dependencies to latest versions

dc09d08

refactor: use strategy pattern for task creation

5a5e3f5

		@@ -60,6 +62,10 @@ export interface JobConfig {

		export interface IngestionJobsConfig {
		seed: JobConfig \| undefined;

+                  });
+                }
+                /**

+                }
+              }
+              /**

Uh oh!

Conversation

almog8k commented Aug 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

almog8k commented Aug 12, 2026 •

edited

Loading