Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
# Storage Reader Leases protect active downloads

Concurrent downloads hold durable, expiring Storage Reader Leases, including direct downloads whose lease lasts no longer than the signed URL. A renewable, fenced Merge Lease grants a single worker authority to complete the Merge; Part Reader Leases protect concurrent readers of Parts. Both expiring lease types renew every 30 seconds while active and expire after two minutes, except direct-download leases, which expire with their signed URLs. Parts are deleted only after the merge completes and no active Part Reader Leases remain; a Storage Location is deleted only after no Storage Reader Leases remain. A reader acquires its lease in the same locked transaction that reads the Storage Location's merge state, so its scope is chosen atomically: a download arriving before the merge takes a Part Reader Lease that blocks parts deletion, and one arriving after takes a Storage Reader Lease over the merged representation. There is therefore no window in which a reader can fail to acquire a Part Reader Lease against parts that are being deleted.
Concurrent downloads hold durable, expiring Storage Reader Leases, including direct downloads, whose lease outlives the signed URL by a fixed margin. A renewable, fenced Merge Lease grants a single worker authority to complete the Merge; Part Reader Leases protect concurrent readers of Parts. Both expiring lease types renew every 30 seconds while active and expire after two minutes, except direct-download leases, which expire ten minutes after their signed URLs: storage authenticates a GET when it starts, so a transfer begun just before the URL expires keeps reading after it. Parts are deleted only after the merge completes and no active Part Reader Leases remain; a Storage Location is deleted only after no Storage Reader Leases remain. A reader acquires its lease in the same locked transaction that reads the Storage Location's merge state, so its scope is chosen atomically: a download arriving before the merge takes a Part Reader Lease that blocks parts deletion, and one arriving after takes a Storage Reader Lease over the merged representation. There is therefore no window in which a reader can fail to acquire a Part Reader Lease against parts that are being deleted.
11 changes: 11 additions & 0 deletions docs/adr/0013-direct-downloads-require-verified-tokens.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
---
status: accepted
---

# Direct downloads require verified tokens

A direct-download URL is a bearer credential: whoever holds it reads the object from storage until it expires, without asking the server again. The runtime token's `repository_id` and `ac` claims are all that stand between a caller and that URL, and with `SKIP_TOKEN_VALIDATION=true` they are decoded but not verified, so any client that reaches the server could name any repository and walk away with a signed URL for its cache. The server therefore refuses to start with `ENABLE_DIRECT_DOWNLOADS=true` and `SKIP_TOKEN_VALIDATION=true` together, unless `UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION=true` overrides it for local testing. This extends ADR-0006: skipping verification stays a dev/test escape hatch, and the one feature that turns an unverified claim into a storage credential is out of its reach. Setups that ran both flags stop at startup after upgrading.

A signed URL lives `DIRECT_DOWNLOAD_URL_TTL_SECONDS`, 600 by default (the fixed lifetime before the setting existed) and at most 3600. Clients request the URL right before fetching it, so the lifetime only has to cover the start of the transfer and retries; a shorter one limits replay of a URL that leaks into job logs, while clients that hold a URL and fetch it later, such as BuildKit's `type=gha` cache, need a longer one. The Storage Reader Lease outlives the URL by a fixed margin (ADR-0002).

Verified tokens must carry `exp`; GitHub's always do, and a token without one would never expire. The optional `ACTIONS_TOKEN_AUDIENCE` lists accepted `aud` values and then requires one of them, restricting the server to tokens issued for one installation; a value that lists none (only commas) stops startup rather than silently disabling the check. Scope permissions are bit flags (read 1, write 2): lookups use only scopes with the read bit and saves only scopes with the write bit, and a scope without a name or with a malformed permission rejects the token. Rejections log the error kind and claim name, never the token or claim values.
Original file line number Diff line number Diff line change
Expand Up @@ -125,6 +125,8 @@ Generate environment variables from config values.
value: {{ default (printf "http://%s.%s.svc.cluster.local:%v" (include "github-actions-cache-server.fullname" .) .Release.Namespace .Values.service.port) .Values.config.apiBaseUrl | quote }}
- name: ENABLE_DIRECT_DOWNLOADS
value: {{ .Values.config.enableDirectDownloads | quote }}
- name: DIRECT_DOWNLOAD_URL_TTL_SECONDS
value: {{ .Values.config.directDownloadUrlTtlSeconds | quote }}
- name: EAGER_MERGE
value: {{ .Values.config.eagerMerge | quote }}
- name: MAX_CONCURRENT_MERGES
Expand Down
4 changes: 4 additions & 0 deletions install/kubernetes/github-actions-cache-server/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,10 @@ config:
# The runner must be able to reach the storage provider directly.
enableDirectDownloads: false

# -- Seconds a signed direct-download URL stays valid, 1 to 3600. Lower limits
# replay of a leaked URL; clients that fetch a URL late (BuildKit) need more.
directDownloadUrlTtlSeconds: 600

# -- Merge cache parts right after upload instead of on first download, so
# the first restore can be a direct download. On object storage the parts
# stream through the server; on the filesystem the merge is a kernel copy.
Expand Down
55 changes: 55 additions & 0 deletions src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,8 @@ pub(crate) struct Config {
/// OIDC issuer without trailing slash.
pub(crate) actions_token_issuer: String,
pub(crate) actions_token_jwks_url: Option<Url>,
/// `aud` values a token must carry one of; never empty, `None` skips the check.
pub(crate) actions_token_audience: Option<Vec<String>>,
/// How long unused entries are kept; `None` disables age-based cleanup.
pub(crate) cache_retention: Option<Duration>,
pub(crate) cache_max_size_bytes: Option<NonZeroU64>,
Expand All @@ -44,6 +46,8 @@ pub(crate) struct Config {
pub(crate) disable_cleanup_jobs: bool,
pub(crate) debug: bool,
pub(crate) enable_direct_downloads: bool,
/// Lifetime of a signed direct-download URL, 1 second to 1 hour.
pub(crate) direct_download_url_ttl: Duration,
pub(crate) eager_merge: bool,
/// Merges this process runs at once, eager or lazy; at least 1.
pub(crate) max_concurrent_merges: usize,
Expand Down Expand Up @@ -168,6 +172,13 @@ fn port(raw: &str) -> Result<NonZeroU16, String> {
NonZeroU16::new(integer(raw)?).ok_or_else(|| "0 is not a port".to_owned())
}

fn url_ttl(raw: &str) -> Result<Duration, String> {
Some(integer::<u64>(raw)?)
.filter(|seconds| (1..=3600).contains(seconds))
.map(Duration::from_secs)
.ok_or_else(|| format!("{raw} is not between 1 and 3600"))
}

fn number(raw: &str) -> Result<f64, String> {
raw.trim()
.parse::<f64>()
Expand Down Expand Up @@ -246,8 +257,14 @@ struct Vars {
/// [default: discovered from the issuer]
#[arg(long, env)]
actions_token_jwks_url: Option<Url>,
/// Comma-separated `aud` values a token must carry one of [default: not checked]
#[arg(long, env)]
actions_token_audience: Option<String>,
#[arg(long, env, action = ArgAction::Set, default_value_t = false)]
skip_token_validation: bool,
/// Allows direct downloads with `SKIP_TOKEN_VALIDATION`; for local testing only
#[arg(long, env, action = ArgAction::Set, default_value_t = false)]
unsafe_allow_direct_downloads_without_token_validation: bool,
/// Days an unused entry is kept; 0 keeps it forever
#[arg(long, env, default_value = "90", value_parser = number)]
cache_cleanup_older_than_days: f64,
Expand All @@ -264,6 +281,9 @@ struct Vars {
debug: Option<String>,
#[arg(long, env, action = ArgAction::Set, default_value_t = false)]
enable_direct_downloads: bool,
/// Seconds a signed direct-download URL stays valid, 1 to 3600
#[arg(long, env, default_value = "600", value_parser = url_ttl)]
direct_download_url_ttl_seconds: Duration,
#[arg(long, env, action = ArgAction::Set, default_value_t = false)]
eager_merge: bool,
#[arg(long, env, default_value = "16", value_parser = integer::<NonZeroU64>)]
Expand Down Expand Up @@ -522,6 +542,39 @@ impl Config {
.chain(args),
)?;
let mut given = Vars::from_arg_matches(&matches)?;
// A signed URL is a storage credential; unverified claims must not obtain one (ADR-0013).
if given.enable_direct_downloads
&& given.skip_token_validation
&& !given.unsafe_allow_direct_downloads_without_token_validation
{
return Err(ConfigError::Invalid(
"ENABLE_DIRECT_DOWNLOADS=true requires token validation, but \
SKIP_TOKEN_VALIDATION=true would hand signed storage URLs to unverified callers. \
Unset SKIP_TOKEN_VALIDATION, or set \
UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION=true for local testing only"
.to_owned(),
));
}
let actions_token_audience = given
.actions_token_audience
.take()
.map(|raw| {
let audience: Vec<String> = raw
.split(',')
.map(str::trim)
.filter(|entry| !entry.is_empty())
.map(str::to_owned)
.collect();
// A typo must not silently switch the check off.
if audience.is_empty() {
return Err(ConfigError::Invalid(
"ACTIONS_TOKEN_AUDIENCE lists no audience; unset it to skip the check"
.to_owned(),
));
}
Ok(audience)
})
.transpose()?;
let storage = given.storage()?;
let db = given.db()?;
let api_base_url = given.api_base_url.trim_end_matches('/').to_owned();
Expand All @@ -533,6 +586,7 @@ impl Config {
api_base_url,
default_actions_results_url: given.default_actions_results_url,
actions_token_jwks_url: given.actions_token_jwks_url,
actions_token_audience,
cache_retention: days(given.cache_cleanup_older_than_days)?,
cache_max_size_bytes: given.cache_max_size_bytes,
cache_filesystem_max_usage: given.cache_filesystem_max_usage_percent,
Expand All @@ -542,6 +596,7 @@ impl Config {
disable_cleanup_jobs: given.disable_cleanup_jobs,
debug: given.debug.is_some(),
enable_direct_downloads: given.enable_direct_downloads,
direct_download_url_ttl: given.direct_download_url_ttl_seconds,
eager_merge: given.eager_merge,
max_concurrent_merges: usize::try_from(given.max_concurrent_merges.get())
.unwrap_or(usize::MAX)
Expand Down
48 changes: 48 additions & 0 deletions src/config/tests.rs
Original file line number Diff line number Diff line change
Expand Up @@ -158,3 +158,51 @@ fn ignores_variables_of_other_drivers() {
"the selected driver does"
);
}

#[test]
fn direct_downloads_require_token_validation() {
let unverified = [
("ENABLE_DIRECT_DOWNLOADS", "true"),
("SKIP_TOKEN_VALIDATION", "true"),
];
assert!(config(&unverified).is_err(), "refused without the override");
let overridden = [
unverified.as_slice(),
&[(
"UNSAFE_ALLOW_DIRECT_DOWNLOADS_WITHOUT_TOKEN_VALIDATION",
"true",
)],
]
.concat();
assert!(config(&overridden).is_ok(), "allowed with the override");
}

#[test]
fn bounds_the_direct_download_url_lifetime() {
assert_eq!(
config(&[]).unwrap().direct_download_url_ttl,
Duration::from_mins(10),
"ten minutes by default"
);
for ttl in ["0", "3601", "1.5"] {
assert!(
config(&[("DIRECT_DOWNLOAD_URL_TTL_SECONDS", ttl)]).is_err(),
"{ttl} is out of range"
);
}
}

#[test]
fn parses_the_token_audience() {
assert_eq!(
config(&[("ACTIONS_TOKEN_AUDIENCE", " a, ,b ")])
.unwrap()
.actions_token_audience,
Some(vec!["a".to_owned(), "b".to_owned()]),
"trimmed, blanks dropped"
);
assert!(
config(&[("ACTIONS_TOKEN_AUDIENCE", " , ")]).is_err(),
"an audience list without audiences is refused"
);
}
59 changes: 49 additions & 10 deletions src/scope.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ use std::time::{Duration, Instant};

use axum::http::HeaderMap;
use axum::http::header::AUTHORIZATION;
use jsonwebtoken::errors::ErrorKind;
use jsonwebtoken::jwk::{
AlgorithmParameters, EllipticCurve, Jwk, JwkSet, KeyOperations, PublicKeyUse,
};
Expand All @@ -15,16 +16,18 @@ use tokio::sync::{Mutex, OnceCell};
use crate::config::Config;
use crate::error::{Failure, Result};

/// Permission bit that allows restoring from a scope.
const READ_PERMISSION: u32 = 1;
/// Permission bit that allows saving to a scope.
const WRITE_PERMISSION: i64 = 2;
const WRITE_PERMISSION: u32 = 2;

/// A cache scope the token grants, with its permission bits.
#[derive(Debug, Clone, PartialEq, Eq, Deserialize)]
pub(crate) struct ScopeGrant {
#[serde(rename = "Scope")]
pub(crate) scope: String,
#[serde(rename = "Permission")]
pub(crate) permission: i64,
pub(crate) permission: u32,
}

/// Scopes and repository a request may access.
Expand All @@ -39,12 +42,17 @@ impl CacheScope {
pub(crate) fn write_scope(&self) -> Option<&ScopeGrant> {
self.scopes
.iter()
.find(|grant| grant.permission >= WRITE_PERMISSION)
.find(|grant| grant.permission & WRITE_PERMISSION != 0)
}

/// Scope names by descending permission, the order restores search them in.
/// Readable scope names by descending permission, the order restores search them in.
pub(crate) fn read_scopes(&self) -> Vec<String> {
let mut grants = self.scopes.clone();
let mut grants: Vec<ScopeGrant> = self
.scopes
.iter()
.filter(|grant| grant.permission & READ_PERMISSION != 0)
.cloned()
.collect();
grants.sort_by_key(|grant| std::cmp::Reverse(grant.permission));
grants.into_iter().map(|grant| grant.scope).collect()
}
Expand Down Expand Up @@ -152,6 +160,7 @@ fn selects(jwk: &Jwk, header: &Header) -> bool {
#[derive(Debug)]
pub(crate) struct TokenVerifier {
issuer: String,
audience: Option<Vec<String>>,
skip_validation: bool,
http: reqwest::Client,
override_jwks: Option<Arc<RemoteJwks>>,
Expand All @@ -173,6 +182,7 @@ impl TokenVerifier {
.map(|url| RemoteJwks::new(url.to_string())),
discovered_jwks: OnceCell::new(),
skip_validation: config.skip_token_validation,
audience: config.actions_token_audience.clone(),
issuer,
http: reqwest::Client::builder().timeout(FETCH_TIMEOUT).build()?,
})
Expand Down Expand Up @@ -230,14 +240,19 @@ impl TokenVerifier {
}
let header = jsonwebtoken::decode_header(token)?;
let keys = self.jwks().await.keys(&self.http, &header).await?;
// Like the original server's `jose`: the issuer is required, `exp` and `nbf` are checked
// when present without leeway, and the audience is not checked.
// The issuer and `exp` are required, `nbf` is checked when present, all without leeway.
// The audience is checked only when configured, and then required.
let mut validation = Validation::new(header.alg);
validation.set_issuer(&[&self.issuer]);
validation.set_required_spec_claims(&["iss"]);
validation.validate_nbf = true;
validation.validate_aud = false;
validation.leeway = 0;
if let Some(audience) = &self.audience {
validation.set_audience(audience);
validation.set_required_spec_claims(&["iss", "exp", "aud"]);
} else {
validation.validate_aud = false;
validation.set_required_spec_claims(&["iss", "exp"]);
}
// Without a `kid`, several keys can match; like `jose`, the first that verifies wins.
let mut verified = Err(Failure::msg("no matching key in JWKS"));
for jwk in &keys {
Expand All @@ -263,7 +278,7 @@ impl TokenVerifier {
.and_then(|value| value.strip_prefix("Bearer "))
.ok_or(Unauthorized("Authorization header missing or malformed"))?;
let claims = self.claims(token).await.map_err(|error| {
tracing::debug!(%error, "invalid token");
tracing::debug!(reason = rejection_reason(&error), "invalid token");
Unauthorized("Invalid token")
})?;
let Some(scopes) = claims.ac.filter(|ac| !ac.is_empty()) else {
Expand All @@ -274,12 +289,36 @@ impl TokenVerifier {
if scopes.is_empty() {
return Err(Unauthorized("Token does not contain any cache scopes"));
}
if scopes.iter().any(|grant| grant.scope.is_empty()) {
return Err(Unauthorized("Invalid cache scopes"));
}
let Some(repo_id) = claims.repository_id.filter(|id| !id.is_empty()) else {
return Err(Unauthorized("Token does not contain repository id"));
};
Ok(CacheScope { scopes, repo_id })
}
}

/// Why a token was rejected, without the token or its claims: JSON errors quote the values they
/// failed on.
fn rejection_reason(error: &Failure) -> String {
let Failure::Token(error) = error else {
return error.to_string();
};
if let ErrorKind::MissingRequiredClaim(claim) = error.kind() {
return format!("missing claim `{claim}`");
}
if let ErrorKind::InvalidClaimFormat(claim) = error.kind() {
return format!("invalid claim `{claim}`");
}
if matches!(
error.kind(),
ErrorKind::Json(_) | ErrorKind::Base64(_) | ErrorKind::Utf8(_)
) {
return "malformed token".to_owned();
}
error.to_string()
}

#[cfg(test)]
mod tests;
Loading
Loading