Fix/senkin match original - #68
Open
VladimirShitov wants to merge 5 commits into
Open
Conversation
The 2026-08-14 run had senkin_tmp_train hit the 8h walltime on the CITE datasets and OOM/crash on the rest. Two causes: 1. LightGBM set no num_threads, so it defaulted to one thread per core the container *sees* (the whole node) while the job is cgroup-throttled to meta["cpus"] -- the threads oversubscribe and thrash. The original solution ran on Kaggle with dedicated cores so never hit this. Pin num_threads to meta["cpus"] (results-preserving; same class of fix as guanlab in #59). 2. senkin has no applicability guard, so it ran on all 8 datasets including Multiome and the ADT->GEX swap, where it wastes hours and OOMs. Add an exit_non_applicable guard (via src/utils/exit_codes.py) so it only runs on the GEX->ADT CITE direction, and point the component test at bmmc_cite/normal. Verified: viash test passes (2/2) on the normal (GEX->ADT) direction; the guard correctly skips the swap direction. Note: pinning threads removes the oversubscription, but senkin runs 4 LightGBM models (134 targets x 5 folds) + 2 NNs in one job, which the original spread across separate multi-hour Kaggle sessions - it may still need a walltime above 8h or a lighter config on the full data.
…agnostic Uses the fixed senkin-tmp-cite-pred (fix/match-original-pipeline) and restores the original feature blocks: log1p(CPM) inputs, CLR-TSVD 200, correlated genes selected on the log-normalized RNA per batch (plus genes whose name matches a protein), TSVD-100 and PCA-64 of the sqrt/z-score/ batch-median normalization, and the four LightGBM prediction blocks, all z-scored per cell before the two neural networks. The batch column is used as is (or a single batch when absent) instead of parsing day/donor out of it, which only made sense for the NeurIPS 2022 naming. Predictions are rescaled to the target scale with one global affine transform fitted on the out-of-fold training predictions, so RMSE/MAE are meaningful while every per-cell and per-protein correlation is unchanged. The predict step checks that the test cells match the trained bundle and uses light resource labels. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Describe your changes
Fix the implementation to match the competition 2022 results
Checklist before requesting a review
I have performed a self-review of my code
Check the correct box. Does this PR contain:
Proposed changes are described in the CHANGELOG.md
CI Tests succeed and look good!