Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,15 @@ volumes:
cache-data:
```

Direct downloads (`ENABLE_DIRECT_DOWNLOADS=true`) hand the runner a signed storage URL for merged entries instead of proxying the bytes.
They require token validation: the server refuses to start with `SKIP_TOKEN_VALIDATION=true`, because unverified callers could then obtain signed URLs for any repository's cache.

| Variable | Default | Effect |
| -------------------------------------------------------- | ------- | ------------------------------------------------------------------------------------------------- |
| `DIRECT_DOWNLOAD_URL_TTL_SECONDS` | `600` | Lifetime of a signed download URL, `1` to `3600`. Lower it to limit replay of a leaked URL. |
| `ACTIONS_TOKEN_AUDIENCE` | unset | Comma-separated `aud` values a token must carry; unset skips the check, an empty list is refused. |
| `UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION` | `false` | Allows direct downloads with `SKIP_TOKEN_VALIDATION=true`. Local testing only. |

## Documentation

👉 <https://gha-cache-server.falcondev.io/getting-started> 👈
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Storage Reader Leases protect active downloads

Concurrent downloads hold durable, expiring Storage Reader Leases, including direct downloads whose lease lasts no longer than the signed URL. A renewable, fenced Merge Lease grants a single worker authority to complete the Merge; Part Reader Leases protect concurrent readers of Parts. Both expiring lease types renew every 30 seconds while active and expire after two minutes, except direct-download leases, which expire with their signed URLs. Parts are deleted only after the merge completes and no active Part Reader Leases remain; a Storage Location is deleted only after no Storage Reader Leases remain. A reader acquires its lease in the same locked transaction that reads the Storage Location's merge state, so its scope is chosen atomically: a download arriving before the merge takes a Part Reader Lease that blocks parts deletion, and one arriving after takes a Storage Reader Lease over the merged representation. There is therefore no window in which a reader can fail to acquire a Part Reader Lease against parts that are being deleted.
Concurrent downloads hold durable, expiring Storage Reader Leases, including direct downloads, whose lease outlives the signed URL by a fixed margin. A renewable, fenced Merge Lease grants a single worker authority to complete the Merge; Part Reader Leases protect concurrent readers of Parts. Both expiring lease types renew every 30 seconds while active and expire after two minutes, except direct-download leases, which expire a fixed margin after their signed URLs: storage authenticates a GET when it starts, so a transfer begun just before the URL expires keeps reading after it. Parts are deleted only after the merge completes and no active Part Reader Leases remain; a Storage Location is deleted only after no Storage Reader Leases remain. A reader acquires its lease in the same locked transaction that reads the Storage Location's merge state, so its scope is chosen atomically: a download arriving before the merge takes a Part Reader Lease that blocks parts deletion, and one arriving after takes a Storage Reader Lease over the merged representation. There is therefore no window in which a reader can fail to acquire a Part Reader Lease against parts that are being deleted.
7 changes: 7 additions & 0 deletions docs/adr/0010-direct-downloads-require-verified-tokens.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
# Direct downloads require verified tokens

A direct-download URL is a bearer credential: whoever holds it can read the merged object from storage until it expires, without asking the server again. The only thing standing between a caller and that URL is `getCacheScope`, which authorizes a lookup from the token's `repository_id` and `ac` claims. With `SKIP_TOKEN_VALIDATION=true` those claims are decoded but not verified, so any client that can reach the server can name any repository and scope and walk away with a signed URL for its cache. The server therefore refuses to start with `ENABLE_DIRECT_DOWNLOADS=true` and `SKIP_TOKEN_VALIDATION=true` together. `UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION=true` overrides the check for local testing only; it is named so that it cannot be set by accident. This extends ADR-0006: skipping verification stays a dev/test escape hatch, and the one feature that turns an unverified claim into a storage credential is now unreachable with it.

A signed URL lives `DIRECT_DOWNLOAD_URL_TTL_SECONDS` (default 600, the lifetime it had before the setting existed; at most 3600). Clients request the URL immediately before fetching it, and storage authenticates a GET when it starts rather than for its whole duration, so the lifetime only has to cover the start of the transfer and the client's retries, not a large restore. A shorter lifetime limits replay of a URL that leaks into job logs; with temporary AWS credentials, 300 or less also stays inside the window in which the AWS SDK refreshes them (5 minutes before they expire), so a URL is never signed by credentials that lapse before it does. Clients that cache a URL and fetch it later, such as BuildKit's `type=gha` cache importing a layer late in a build, need a longer lifetime. The Storage Reader Lease outlives the URL by a fixed margin (ADR-0002), so cleanup cannot delete an object while a transfer begun just before expiry is still reading it.

Token verification requires an `exp` claim, because jose only checks expiry when the claim is present. The optional `ACTIONS_TOKEN_AUDIENCE` pins the expected `aud`, restricting the server to tokens issued for one installation; a value that lists no audience (blank, or only commas) stops startup rather than silently disabling the check. Token rejections return and log the jose error code and failing claim name, never the token or its decoded claims, which jose attaches to its errors. Cache scope permissions are bit flags (read = 1, write = 2): lookups use only scopes with the read bit, and uploads only scopes with the write bit.
13 changes: 12 additions & 1 deletion lib/env.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,10 @@
import type { envDbDriverSchema, envStorageDriverSchema } from './schemas'
import arkenv from 'arkenv'
import { envSchema } from './schemas'
import {
assertDirectDownloadsRequireTokenValidation,
envSchema,
parseActionsTokenAudience,
} from './schemas'

export const env = arkenv(envSchema, {
env: Object.assign(
Expand All @@ -13,3 +17,10 @@ export const env = arkenv(envSchema, {
process.env,
),
})

assertDirectDownloadsRequireTokenValidation(env)

// GitHub-issued runtime tokens carry a deployment-specific `aud`, so pinning it
// restricts the server to one installation's tokens. Parsed here so a bad value
// stops startup.
export const actionsTokenAudience = parseActionsTokenAudience(env.ACTIONS_TOKEN_AUDIENCE)
48 changes: 48 additions & 0 deletions lib/schemas.ts
Original file line number Diff line number Diff line change
Expand Up @@ -76,13 +76,16 @@ export const envBaseSchema = type({
"string.url = 'https://results-receiver.actions.githubusercontent.com'",
'ACTIONS_TOKEN_ISSUER': "string.url = 'https://token.actions.githubusercontent.com'",
'ACTIONS_TOKEN_JWKS_URL?': 'string.url',
'ACTIONS_TOKEN_AUDIENCE?': 'string',
'CACHE_CLEANUP_OLDER_THAN_DAYS': 'number = 90',
'CACHE_MAX_SIZE_BYTES?': 'number.integer > 0',
'CACHE_FILESYSTEM_MAX_USAGE_PERCENT': 'number > 0 & number <= 100 = 90',
'ORPHANED_STORAGE_GRACE_PERIOD_HOURS': 'number.integer >= 1 = 24',
'DISABLE_CLEANUP_JOBS?': 'boolean',
'DEBUG?': 'unknown',
'ENABLE_DIRECT_DOWNLOADS': 'boolean = false',
'DIRECT_DOWNLOAD_URL_TTL_SECONDS': '1 <= number.integer <= 3600 = 600',
'UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION': 'boolean = false',
'EAGER_MERGE': 'boolean = false',
'BENCHMARK': 'boolean = false',
'SKIP_TOKEN_VALIDATION': 'boolean = false',
Expand All @@ -91,3 +94,48 @@ export const envBaseSchema = type({

export const envSchema = envBaseSchema.and(envStorageDriverSchema).and(envDbDriverSchema)
export type Env = typeof envSchema.infer

/**
* A direct-download URL is a bearer credential for the stored object. Handing
* one out to a caller whose token was never verified lets anyone who can reach
* the server read any repository's cache straight from storage, so the two
* settings are refused together unless explicitly overridden for local testing.
*/
export function assertDirectDownloadsRequireTokenValidation(
env: Pick<
Env,
| 'ENABLE_DIRECT_DOWNLOADS'
| 'SKIP_TOKEN_VALIDATION'
| 'UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION'
>,
) {
if (
env.ENABLE_DIRECT_DOWNLOADS &&
env.SKIP_TOKEN_VALIDATION &&
!env.UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION
)
throw new Error(
'ENABLE_DIRECT_DOWNLOADS=true requires token validation, but SKIP_TOKEN_VALIDATION=true. ' +
'Unverified callers would receive signed storage URLs for any cache entry. ' +
'Unset SKIP_TOKEN_VALIDATION, or set UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION=true for local testing only.',
)
}

/**
* The `aud` values a token must carry. Unset means no audience check; set but
* listing no audience (blank, or only commas) is refused, so a typo cannot
* silently switch the check off.
*/
export function parseActionsTokenAudience(value: string | undefined) {
if (value === undefined) return
const audience = value
.split(',')
.map((entry) => entry.trim())
.filter(Boolean)
if (audience.length === 0)
throw new Error(
'ACTIONS_TOKEN_AUDIENCE is set but lists no audience. ' +
'Set one or more comma-separated audiences, or unset it to skip the audience check.',
)
return audience
}
57 changes: 42 additions & 15 deletions lib/scope.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,8 @@
import type { H3Event } from 'h3'
import * as jose from 'jose'
import { hasAtLeast } from 'remeda'
import { env } from './env'
import { z } from 'zod'
import { actionsTokenAudience, env } from './env'
import { logger } from './logger'

const issuer = env.ACTIONS_TOKEN_ISSUER.replace(/\/$/, '')
Expand Down Expand Up @@ -64,20 +65,50 @@ async function verifyGitHubActionsToken(token: string) {
return jose.decodeJwt(token)
}

const { payload } = await jose.jwtVerify(token, await getJwks(), { issuer })
const { payload } = await jose.jwtVerify(token, await getJwks(), {
issuer,
audience: actionsTokenAudience,
// jose checks `exp` and `nbf` only when present; a token without `exp`
// would otherwise never expire.
requiredClaims: ['exp'],
})
return payload
}

/**
* The reason a token was rejected, safe to log and return: never the token or
* its claims. jose errors carry the decoded payload, so they must not become
* the `cause` of a logged error.
*/
function describeTokenError(err: unknown) {
if (err instanceof jose.errors.JWTClaimValidationFailed || err instanceof jose.errors.JWTExpired)
return `${err.code} (${err.claim})`
if (err instanceof jose.errors.JOSEError) return err.code
return err instanceof Error ? err.name : 'unknown error'
}

export const CACHE_PERMISSION_READ = 1
export const CACHE_PERMISSION_WRITE = 2

const cacheScopesSchema = z.array(
z.object({
Scope: z.string().min(1),
Permission: z.number().int().nonnegative(),
}),
)

function parseJsonScopes(json: string) {
let value: unknown
try {
return JSON.parse(json)
} catch (err) {
throw createError({
statusCode: 401,
message: 'Invalid JSON in cache scopes',
cause: err,
})
value = JSON.parse(json)
} catch {
// no `cause`: the parser's message quotes the claim
throw createError({ statusCode: 401, message: 'Invalid JSON in cache scopes' })
}

const parsed = cacheScopesSchema.safeParse(value)
if (!parsed.success) throw createError({ statusCode: 401, message: 'Invalid cache scopes' })
return parsed.data
}

export async function getCacheScope(event: H3Event) {
Expand All @@ -86,18 +117,14 @@ export async function getCacheScope(event: H3Event) {
throw createError({ statusCode: 401, message: 'Authorization header missing or malformed' })

const decoded = await verifyGitHubActionsToken(token).catch((err) => {
throw createError({
statusCode: 401,
message: 'Invalid token',
cause: err,
})
throw createError({ statusCode: 401, message: `Invalid token: ${describeTokenError(err)}` })
})

const scopesJson = decoded.ac
if (!scopesJson || typeof scopesJson !== 'string')
throw createError({ statusCode: 401, message: 'Token does not contain cache scopes' })

const scopes = parseJsonScopes(scopesJson) as Array<{ Scope: string; Permission: number }>
const scopes = parseJsonScopes(scopesJson)
if (!hasAtLeast(scopes, 1))
throw createError({ statusCode: 401, message: 'Token does not contain any cache scopes' })

Expand Down
7 changes: 6 additions & 1 deletion lib/storage-leases.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,12 @@ import { env } from './env'

export const LEASE_DURATION_MS = 2 * 60 * 1000
export const LEASE_RENEWAL_MS = 30 * 1000
export const DIRECT_DOWNLOAD_LEASE_DURATION_MS = 10 * 60 * 1000
/**
* How long a direct-download reader lease outlives its signed URL. A download
* that starts just before the URL expires keeps streaming, so the lease must
* cover that transfer too.
*/
export const DIRECT_DOWNLOAD_LEASE_MARGIN_MS = 10 * 60 * 1000

export async function acquireMergeLease(
db: Kysely<Database>,
Expand Down
18 changes: 9 additions & 9 deletions lib/storage.ts
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ import { logger } from './logger'
import {
acquireMergeLease,
createReaderLease,
DIRECT_DOWNLOAD_LEASE_DURATION_MS,
DIRECT_DOWNLOAD_LEASE_MARGIN_MS,
LEASE_RENEWAL_MS,
releaseMergeLease,
releaseReaderLease,
Expand Down Expand Up @@ -774,9 +774,12 @@ export class Storage {
cacheEntry: cacheEntry.match,
}

// Merged entry with direct downloads enabled: take a reader lease bounded
// by the signed URL lifetime, then sign.
const directDownloadExpiresAt = Date.now() + DIRECT_DOWNLOAD_LEASE_DURATION_MS
// Merged entry with direct downloads enabled: take a reader lease that
// outlives the signed URL, then sign. Storage authenticates a GET when it
// starts, so a download begun just before the URL expires keeps reading
// after it; the margin keeps cleanup from deleting the object under it.
const urlExpiresAt = Date.now() + env.DIRECT_DOWNLOAD_URL_TTL_SECONDS * 1000
const leaseExpiresAt = urlExpiresAt + DIRECT_DOWNLOAD_LEASE_MARGIN_MS
const leased = await this.db.transaction().execute(async (tx) => {
let query = tx
.selectFrom('storage_locations')
Expand All @@ -785,7 +788,7 @@ export class Storage {
if (env.DB_DRIVER !== 'sqlite') query = query.forUpdate()
const current = await query.executeTakeFirst()
if (!current?.mergedAt) return current
await createReaderLease(tx, current.id, 'storage', directDownloadExpiresAt)
await createReaderLease(tx, current.id, 'storage', leaseExpiresAt)
await tx
.updateTable('storage_locations')
.set({ lastDownloadedAt: Date.now() })
Expand All @@ -797,10 +800,7 @@ export class Storage {

// The merge could have been undone between validation and the lease read.
const downloadUrl = leased.mergedAt
? await this.adapter.createDownloadUrl(
`${leased.folderName}/merged`,
directDownloadExpiresAt,
)
? await this.adapter.createDownloadUrl(`${leased.folderName}/merged`, urlExpiresAt)
: defaultUrl

return {
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { z } from 'zod'
import { env } from '~/lib/env'
import { getCacheScope } from '~/lib/scope'
import { CACHE_PERMISSION_WRITE, getCacheScope } from '~/lib/scope'
import { AlreadyExistsError, getStorage } from '~/lib/storage'
import { readTwirpRequest, sendTwirpResponse, TwirpMessage } from '~/lib/twirp'

Expand All @@ -19,7 +19,7 @@ export default defineEventHandler(async (event) => {
)

const storage = await getStorage()
const writeScope = scopes.find((s) => s.Permission >= 2)
const writeScope = scopes.find((s) => (s.Permission & CACHE_PERMISSION_WRITE) !== 0)
if (!writeScope)
throw createError({ statusCode: 403, message: 'No scope with write permission found' })

Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
import { z } from 'zod'
import { cacheUploadsTotal } from '~/lib/metrics'
import { getCacheScope } from '~/lib/scope'
import { CACHE_PERMISSION_WRITE, getCacheScope } from '~/lib/scope'
import { getStorage } from '~/lib/storage'
import { readTwirpRequest, sendTwirpResponse, TwirpMessage } from '~/lib/twirp'

Expand All @@ -19,7 +19,7 @@ export default defineEventHandler(async (event) => {
)

const storage = await getStorage()
const writeScope = scopes.find((s) => s.Permission >= 2)
const writeScope = scopes.find((s) => (s.Permission & CACHE_PERMISSION_WRITE) !== 0)
if (!writeScope)
throw createError({ statusCode: 403, message: 'No scope with write permission found' })

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
import { map, pipe, prop, sortBy } from 'remeda'
import { filter, map, pipe, prop, sortBy } from 'remeda'
import { z } from 'zod'
import { cacheRequestsTotal } from '~/lib/metrics'
import { getCacheScope } from '~/lib/scope'
import { CACHE_PERMISSION_READ, getCacheScope } from '~/lib/scope'
import { getStorage } from '~/lib/storage'
import { readTwirpRequest, sendTwirpResponse, TwirpMessage } from '~/lib/twirp'

Expand All @@ -13,6 +13,15 @@ const bodySchema = z.object({

export default defineEventHandler(async (event) => {
const { scopes, repoId } = await getCacheScope(event)
// Scopes are bit flags; a write-only scope must not be read from.
const readScopes = pipe(
scopes,
filter((s) => (s.Permission & CACHE_PERMISSION_READ) !== 0),
sortBy([prop('Permission'), 'desc']),
map(prop('Scope')),
)
if (readScopes.length === 0)
throw createError({ statusCode: 403, message: 'No scope with read permission found' })

const { key, restore_keys, version } = await readTwirpRequest(
event,
Expand All @@ -24,7 +33,7 @@ export default defineEventHandler(async (event) => {
const match = await storage.getCacheEntryWithDownloadUrl({
keys: [key, ...(restore_keys ?? [])],
version,
scopes: pipe(scopes, sortBy([prop('Permission'), 'desc']), map(prop('Scope'))),
scopes: readScopes,
repoId,
})

Expand Down
8 changes: 6 additions & 2 deletions tests/cleanup-lifecycle.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ import { afterAll, beforeAll, describe, expect, test, vi } from 'vitest'
import { getDatabase } from '~/lib/db'
import { env } from '~/lib/env'
import { Storage } from '~/lib/storage'
import { DIRECT_DOWNLOAD_LEASE_MARGIN_MS } from '~/lib/storage-leases'

describe('cleanup lifecycle', () => {
beforeAll(() => {
Expand Down Expand Up @@ -327,7 +328,7 @@ describe('cleanup lifecycle', () => {
}, 30_000)

test.skipIf(process.env.VITEST_STORAGE_DRIVER !== 's3')(
'a direct-download reader lease covers the signed URL lifetime',
'a direct-download reader lease outlives the signed URL, which honors the TTL',
async () => {
const db = await getDatabase()
const storage = await Storage.fromEnv()
Expand Down Expand Up @@ -387,7 +388,10 @@ describe('cleanup lifecycle', () => {
)

expect(signedLifetimeSeconds).toBeGreaterThan(0)
expect(lease.expiresAt - startedAt).toBeGreaterThanOrEqual(signedLifetimeSeconds * 1000)
expect(signedLifetimeSeconds).toBeLessThanOrEqual(env.DIRECT_DOWNLOAD_URL_TTL_SECONDS)
expect(lease.expiresAt - startedAt).toBeGreaterThanOrEqual(
signedLifetimeSeconds * 1000 + DIRECT_DOWNLOAD_LEASE_MARGIN_MS,
)
expect(accessed.lastDownloadedAt).toBeGreaterThanOrEqual(startedAt)
} finally {
env.ENABLE_DIRECT_DOWNLOADS = directDownloadsEnabled
Expand Down
Loading
Loading