-
Notifications
You must be signed in to change notification settings - Fork 728
feat: integration builder POC [CM-1372] #4463
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
mbani01
wants to merge
44
commits into
main
Choose a base branch
from
feat/integration-builder-poc
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+4,570
−137
Open
Changes from all commits
Commits
Show all changes
44 commits
Select commit
Hold shift + click to select a range
e00a0cb
feat: add integration.sync_units control table
mbani01 f5dafd7
feat: add sync units data access layer
mbani01 f41df42
feat: add integration-builder lib with connector registry
mbani01 5062058
fix: skip sync units of soft-deleted integrations when claiming
mbani01 327d6d0
feat: add getCredential facade
mbani01 4daecd3
fix: export integration builder DAL from package root
mbani01 31d697f
style: fix prettier formatting in credentials facade
mbani01 94901f8
refactor: rename integration-builder to connectors
mbani01 d681c11
feat: add connectors worker with dispatcher workflow
mbani01 8b22542
feat: add sync-run workflow with dummy connector
mbani01 ff1248f
feat: add connectors worker docker setup
mbani01 ab10d0c
fix: rethrow sync run errors so temporal reflects failures
mbani01 9de976f
feat: add connectors http core with error taxonomy
mbani01 f76efcc
fix: guard heartbeat callback and add connectors worker dockerignore
mbani01 8a1391b
fix: restrict rate limit detection to unambiguous signals
mbani01 5e4117a
fix: always apply pooled token in http client auth
mbani01 b582876
fix: add default http timeout and narrow claimed unit payload
mbani01 fac2c62
feat: add redis token pool for connectors
mbani01 032d4cd
feat: add token budgets with dispatcher admission
mbani01 1f5324b
feat: add emit with zod validation reusing legacy results publishing
mbani01 5a88aed
chore: sync pnpm lockfile with main
mbani01 2bac076
fix: publish schema-parsed record instead of raw input in emit
mbani01 ff4a063
Merge remote-tracking branch 'origin/main' into feat/integration-buil…
mbani01 a0dafee
fix: address http client and emit schema review findings
mbani01 12f2cdd
fix: align node 24 targets, migration ordering and applier classifica…
mbani01 47f6e53
fix: park tokens on secondary rate limits and ignore stale reset headers
mbani01 2644746
fix: rotate to a fresh token after quarantining on auth failure
mbani01 57fc8af
feat: wire http client and emit into sync runs
mbani01 89eee85
fix: register dummy connector only in dev environments
mbani01 871980b
fix: persist partial progress when a sync run parks on rate limit
mbani01 c19e860
feat: add per-platform manifest hooks for tokens, budgets and respons…
mbani01 84a0801
fix: admit units with no healthy tokens so seeding can revive the pool
mbani01 8c78a77
feat: add github app auth, graphql client and connector skeleton
mbani01 8935e49
fix: address review findings on timeouts, partial-run recency and sin…
mbani01 bbf15c5
feat: add github sync-unit seeding script to connectors worker
mbani01 1219e42
chore: add connectors worker builder definition
mbani01 266690b
feat: resolve github app installation id with env and flag overrides
mbani01 6e406ae
feat: add github connector syncs with full pagination and nango paylo…
mbani01 bd43a59
fix: park github syncs on body-level and secondary rate limits with e…
mbani01 1187b0e
fix: fetch pr children per item to avoid github batch node drops
mbani01 3b69675
fix: align github member payloads and event skips with nango implemen…
mbani01 7dcd42d
perf: probe token budgets concurrently in dispatcher admission
mbani01 ed1c940
feat: add github discussions sync with nested reply traversal
mbani01 e918730
feat: add github stars sync and register discussions and stars in man…
mbani01 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
25 changes: 25 additions & 0 deletions
25
backend/src/database/migrations/V1787829533__createSyncUnitsTable.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| CREATE TABLE IF NOT EXISTS integration.sync_units ( | ||
| id UUID PRIMARY KEY DEFAULT uuid_generate_v4(), | ||
| "integrationId" UUID NOT NULL REFERENCES public.integrations(id), | ||
| platform TEXT NOT NULL, | ||
| "channelId" TEXT NOT NULL, | ||
| "channelName" TEXT NOT NULL, | ||
| "syncName" TEXT NOT NULL, | ||
| status TEXT NOT NULL DEFAULT 'active' | ||
| CHECK (status IN ('active','paused','dead_letter','decommissioned')), | ||
| "nextRunAt" TIMESTAMPTZ NOT NULL DEFAULT now(), | ||
| "lockedAt" TIMESTAMPTZ, | ||
| "lastRunAt" TIMESTAMPTZ, | ||
| "lastSuccessAt" TIMESTAMPTZ, | ||
| "consecutiveFailures" INT NOT NULL DEFAULT 0, | ||
| "lastErrorClass" TEXT, | ||
| watermark JSONB, | ||
| "emittedCount" INT, | ||
| "createdAt" TIMESTAMPTZ NOT NULL DEFAULT now(), | ||
| "updatedAt" TIMESTAMPTZ NOT NULL DEFAULT now(), | ||
| UNIQUE ("integrationId", "channelId", "syncName") | ||
| ); | ||
|
|
||
| CREATE INDEX IF NOT EXISTS "ix_sync_units_due" | ||
| ON integration.sync_units ("nextRunAt") | ||
| WHERE status = 'active'; | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.