Repository navigation
fix(thin): reduce log noise from not-yet-bound PVC in ShouldUpdateUFS - #6233
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #6233 +/- ##
==========================================
+ Coverage 65.75% 65.78% +0.02%
==========================================
Files 486 486
Lines 34459 34470 +11
==========================================
+ Hits 22660 22675 +15
+ Misses 10044 10041 -3
+ Partials 1755 1754 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
There was a problem hiding this comment.
🟡 Changes recommended
Lost PVC bindings are misclassified as transient, suppressing error-level visibility of storage failures, and the new test does not verify verbosity level 1 for pending-PVC logs.
2 open issues
What changed in this PR
This PR follows up on #6187 to reduce ThinRuntime log noise while PVC mounts await binding, addressing #6203 without changing error propagation.
Changes:
- Adds
ErrPVCNotBoundand wraps it when a PVC is not bound. - Logs matching errors at V(1), retaining Error-level logging for other errors.
- Adds tests for sentinel propagation and pending-PVC versus unexpected-error logging.
| File | Description |
|---|---|
pkg/ddc/thin/ufs.go |
Selects log level using the PVC-not-bound sentinel. |
pkg/ddc/thin/ufs_test.go |
Tests sentinel propagation through config updates. |
pkg/ddc/thin/ufs_ginkgo_test.go |
Adds log capture and logging behavior tests. |
pkg/ddc/thin/transform_config.go |
Defines and wraps the PVC-not-bound sentinel. |
pkg/ddc/thin/transform_config_test.go |
Checks sentinel identity during volume extraction and config transformation. |
Review effort: Balanced
|
|
||
| if len(pvc.Spec.VolumeName) == 0 || pvc.Status.Phase != corev1.ClaimBound { | ||
| err = fmt.Errorf("persistent volume claim %s not bounded yet", pvcName) | ||
| err = fmt.Errorf("persistent volume claim %s not bounded yet: %w", pvcName, ErrPVCNotBound) |
| func (s *testCaptureLogSink) Info(level int, msg string, keysAndValues ...interface{}) { | ||
| s.infoMsgs = append(s.infoMsgs, msg) | ||
| } |
cheyang
left a comment
There was a problem hiding this comment.
The premise behind this PR checks out end to end: on the base branch, a pending pvc:// mount makes ShouldUpdateUFS log Failed to update fuse config at Error level every sync period (~5s), and on this head the same harnesses record zero Error logs and one V(1) notice. Two independent verification harnesses agree — fake-client unit tests on both, plus envtest against a real API server (k8s 1.34.1) — and the errors.Is(err, ErrPVCNotBound) chain holds through the full pkg/errors Wrapf, unexpected errors keep Error-level visibility, and the sync loop's nil-guard is untouched. The whole pkg/ddc/thin suite is green at head and every CI check passes (23/23; only the tide merge bot is pending). No blocking findings — this is a small, well-scoped, well-tested change.
Next round
[MUST] None — merge is not gated on anything from this review.
[SHOULD] Decide the ClaimLost disposition: the not-yet-bound demotion also swallows ClaimLost (a lost binding is a storage failure, not transient provisioning) into V(1) on the sync path. Either narrow the wrap predicate to ClaimPending || VolumeName == "", or record that demoting it is intended.
[SHOULD] Make the new demotion spec level-aware: the capture sink discards the verbosity level, so a V(1)→V(0) regression would keep the test green — record the level and assert it is >= 1.
[SHOULD] Deduplicate the wrapped message text (... not bounded yet: persistent volume claim not bounded yet); it also shows up in setup-path reconcile errors.
[Filed] None.
The one place the two reviews diverged was framing, not substance: whether the missing-PVC Error cadence belongs to #6203's scope (it is pre-existing and unchanged by this PR, so it stays a maintainer question rather than a fix request). Copilot's advisory review flagged the same two items the debate confirmed (ClaimLost over-demotion, level-blind test sink) and raised nothing the debaters could not corroborate. Harnesses: verify/thin-pvc-notbound-claude and verify/pr6233-pvcnotbound-log-codex on the reviewer fork, re-runnable via each branch's re-verify.sh.
Non-blocking observations (5) — not part of this round's ask; fix now, follow up, or file an issue as you see fit
- pkg/ddc/thin/transform_config.go:165 [minor] — ClaimLost PVCs are folded into the transient not-yet-bound demotion
F2 (consensus after cross-critique; also flagged by Copilot as C4507914753): the wrap at transform_config.go:165-166 fires for any non-Bound phase —len(pvc.Spec.VolumeName) == 0 || pvc.Status.Phase != corev1.ClaimBound— which includesClaimLost(a bound PV lost underneath the claim; VolumeName still set). A lost binding is a storage failure, not the self-resolving provisioning state issue #6203 describes, yet on the periodic sync path it is now demoted to V(1) and invisible at default verbosity. Proven PR-introduced by two independent base/head canaries: on base a01c038 a ClaimLost PVC produces one Error record ('Failed to update fuse config'); on head 952e37e it produces none (V(1) notice only). Impact is bounded — the update is skipped either way, and the setup path (transform_fuse.go:125) still surfaces the error as a reconcile failure — hence minor rather than major. Suggested fix: wrap the sentinel only whenpvc.Status.Phase == corev1.ClaimPending || len(pvc.Spec.VolumeName) == 0so ClaimLost keeps Error-level visibility, or record explicitly that demoting it is intended. - pkg/ddc/thin/ufs.go:71 [minor] — pre-existing, not introduced by this PR — PVC-not-found still logs at Error with the same ~5s cadence (pre-existing)
F4 (question, pre-existing — facts confirmed by both reviewers): apvc://mount whose PVC does not exist at all still logs 'Failed to update fuse config' at Error level once per sync period (~5s) — the same alert-noise shape as issue #6203, one state earlier. Issue #6203 scopes the noise to a PVC that exists but is not yet Bound, so demotingapierrors.IsNotFoundhere would be a scope expansion rather than an in-scope fix; but in dynamic-mount flows (#3432) the dataset can briefly precede its PVC, and operators wired to controller error rates still see a sustained Error stream. Maintainer call: demote IsNotFound too, or document the Error level as intended. - pkg/ddc/thin/ufs_ginkgo_test.go:412 [minor] — New demotion test cannot catch a V(1)→V(0) regression
F3 (consensus after cross-critique; also flagged by Copilot as C4507914788): the new spec 'demotes log to V(1) and returns nil when a pvc mount is not yet bound' cannot detect a V(1)→V(0) regression. testCaptureLogSink (ufs_ginkgo_test.go:411-414) returns true from Enabled for every level and discards the level parameter in Info, so the assertion only checks message containment in infoMsgs — replacingt.Log.V(1).Infowith plaint.Log.Infowould keep the spec green while restoring per-sync spam at default verbosity, the exact failure mode this PR exists to remove. The head code is correct today (level-recording sinks from both reviewers record the notice at level 1), so this is a test-hardening gap, not a code bug. Fix: record the level in the sink and assert it is >= 1 (see verifyPR6233LogSink in the verification harness for a working example). - pkg/ddc/thin/transform_config.go:166 [nit] — Sentinel message duplicates the wrapper text
F1 (consensus — independently found by both reviewers): the sentinelErrPVCNotBound= errors.New("persistent volume claim not bounded yet") is wrapped byfmt.Errorf("persistent volume claim %s not bounded yet: %w", pvcName, ErrPVCNotBound), so the rendered chain reads 'failed to extract volume info from PersistentVolumeClaim "my-pvc": persistent volume claim my-pvc not bounded yet: persistent volume claim not bounded yet'. Cosmetic on the demoted V(1) sync path, but the same text surfaces on the setup path (transform_fuse.go:125 → reconcile error/events) where operators read it. Either shorten the sentinel (e.g. errors.New("pvc not bound")) or drop the duplicated phrase from the wrapper; the existing substring-style tests keep passing either way. - pkg/ddc/thin/transform_fuse.go:250 [nit] — pre-existing, not introduced by this PR — Pre-existing gofmt drift in transform_fuse.go (file untouched by this PR)
F5 (pre-existing; confirmed by both reviewers):gofmt -l pkg/ddc/thin/flags transform_fuse.go (misindented createVolumeAndMount around line 250). Reproduces on both base a01c038 and head 952e37e; the PR diff does not touch the file (it touches only transform_config.go, ufs.go and their tests), and nothing in CI gates on it (all checks SUCCESS at head). Worth a one-offgofmt -wcleanup commit at some point so agofmt -l pkg/ddc/thin/verify recipe stays clean.
Verified review delta
- F2: open (initial-pr)
- F4: open (preexisting)
- F3: open (initial-pr)
- F1: open (initial-pr)
- F5: open (preexisting)
Code: clear. CI: success; attribution: baseline. Required CI checks still apply.
Introduce typed sentinel ErrPVCNotBound in extractVolumeInfo and check for it in ShouldUpdateUFS to demote transient not-yet-bound PVC errors to V(1) info level instead of Error level. Signed-off-by: Siddhanth Sadashiv Raikar <raikarsiddhanth@gmail.com>
|
@cheyang Thanks for the review — addressed all three
Re-verified with |
952e37e to
68b6a5d
Compare
|
cheyang
left a comment
There was a problem hiding this comment.
Re-checked the delta since the 11-08 review (952e37e -> 68b6a5d); all three SHOULDs are in. ClaimLost and any other phase that is neither Pending nor Bound now return a plain error and keep Error-level visibility on the sync path, while only Pending or an unset volumeName map to ErrPVCNotBound. The demotion spec records the logr level and asserts V(1), and the wrapped message no longer repeats itself. CI is green on this head, and pkg/ddc/thin passes locally on top of current master (together with #6234). Thanks @frenemy17!





Ⅰ. Describe what this PR does
Follow-up to #6187 and fixes #6203.
When
updateFuseConfigOnChangewas updated to propagate the error fromtransformFuseConfig(#6187), the callerShouldUpdateUFSbegan unconditionally logging all errors atErrorlevel:When a Dataset specifies a
pvc://mount whose PVC is not yet bound (ClaimPendingor emptyVolumeName),extractVolumeInforeturns an error. This is a normal, transient self-resolving state during provisioning, butShouldUpdateUFSemitted anErrorlog roughly once per sync period (default 5s) until the PVC bound, causing sustained log noise and triggering alerts wired to controller error rates.Following approach 2 outlined in #6203:
var ErrPVCNotBound = errors.New("persistent volume claim not bounded yet")inpkg/ddc/thin.ErrPVCNotBoundinextractVolumeInfowhen checkinglen(pvc.Spec.VolumeName) == 0 || pvc.Status.Phase != corev1.ClaimBound.ShouldUpdateUFS, checkif errors.Is(err, ErrPVCNotBound):t.Log.V(1).Info("PVC is not bounded yet, will update fuse config once bounded", "error", err).t.Log.Error(err, "Failed to update fuse config")for all unexpected errors.Ⅱ. Does this pull request fix one issue?
fixes #6203
Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.
pkg/ddc/thin/transform_config_test.go:ThinEngine extractVolumeInfoverifyingErrPVCNotBoundis returned when a PVC is pending.errors.Is(err, ErrPVCNotBound)assertion to the existing pending PVC test inThinEngine transformFuseConfig.pkg/ddc/thin/ufs_ginkgo_test.go:demotes log to V(1) and returns nil when a pvc mount is not yet boundverifying zero Error logs and the V(1) info message.logs an error and returns nil when updateFuseConfigOnChange fails with an unexpected errorverifying unexpected errors still log at Error level.pkg/ddc/thin/ufs_test.go:"transform fuse config failed due to pending pvc"toTestThinEngine_updateFuseConfigOnChangeassertingwantErrIs: ErrPVCNotBound.Ⅳ. Describe how to verify it
Ⅴ. Special notes for reviews
None.