Repository navigation
M-BATCH2: batch resources and GPU flags, complete opt results, frequency pre-opt + saddle retry, lazy imports, GPU Hessian - #141
Merged
Conversation
…(B2.1) Generated batch scripts asked for --ntasks=N, while the worker is one process with OpenMP threads. With --ntasks Slurm sets no SLURM_CPUS_PER_TASK, so a value inherited from a submitting allocation set OMP_NUM_THREADS. Scripts now ask for --ntasks=1 --cpus-per-task=N. The worker always ran under `apptainer exec --nv`, yet no script ever requested a GPU (GOTCHAS 2026-09-23). New optional settings QUANTUI_SLURM_GRES / _QOS / _ACCOUNT add #SBATCH lines (validated before the job folder is written); --nv is used only when the GRES asks for a GPU. Inside the image the worker now runs plain `python` from the image's PATH: the app may run in the CPU image, whose sys.executable does not exist in the GPU batch image. The quantui-batch launcher's port follows; its parity test now covers the GPU lines and normalizes the one intended difference (the launcher keeps the image Python path recorded at install time). Contributions: - Claude (Opus 5.5): code edits, review, and conceptual discussion - Jonathan Schultz: overall vision, planning, review, and orchestration Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…iken (B2.3) optimization_result_payload wrote energy, steps and a trajectory pointer only, though OptimizationResult already carries the final-SCF dipole and Mulliken charges. CHEM-3200 Lab 3 needed an extra single-point preset just to read the dipole, and batch_chain had to dig the geometry out of trajectory.json. The payload now adds final_molecule (atoms, coords, charge, multiplicity) and the same ground-state fields the frequency and TD-DFT payloads use; ingest already reads them. batch_chain prefers final_molecule and falls back to the trajectory for older results. Checked with a real worker run (H2O RHF/STO-3G: 1.709 D, charges recorded). Contributions: - Claude (Opus 5.5): code edits, review, and conceptual discussion - Jonathan Schultz: overall vision, planning, review, and orchestration Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The QM pre-optimization before a frequency calculation used the plain Geometry Opt threshold (0.05 eV/Å). Harmonic frequencies need a stationary point; the residual forces left soft modes wrong, and CHEM-3200 Lab 4's MoO2(acac)2 came back with two imaginary methyl-rotor modes (112i, 48i cm-1). New optimizer.FREQ_PREOPT_FMAX = 0.01 eV/Å (about 2e-4 Ha/bohr, between Gaussian's default and "tight" force criteria), used by the app's Frequency pre-opt and the batch worker's frequency pre-opt, and logged. Other calc types keep 0.05. Cloud check, RHF/STO-3G, MMFF start, lowest modes 0.05 -> 0.01: acetone 104.9 -> 119.1 cm-1 (+3 BFGS steps); propane 222.0 -> 228.6 cm-1 (+10 steps). No imaginary modes at either threshold for these two; the shift in the rotor modes is the effect. Contributions: - Claude (Opus 5.5): code edits, review, and conceptual discussion - Jonathan Schultz: overall vision, planning, review, and orchestration Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ncy run (B2.5) A tighter pre-opt threshold alone can land exactly on a rotor saddle: toluene RHF/STO-3G went from 49.7 cm-1 (0.05 eV/Å, not a stationary point) to 39.4i (0.01 eV/Å, the eclipsed methyl saddle), and Lab 4's 112i mode is too large to be convergence noise. New freq_calc.retry_frequency_off_saddle: for modes below -20 cm-1 it displaces along each (perturb_along_mode, 0.3 Å scale), re-optimizes at FREQ_PREOPT_FMAX and reruns the frequencies, once. Used by the app and the batch worker only when the run had its own pre-opt (a user-supplied geometry may be a saddle on purpose). The retry is recorded in FreqResult.imaginary_mode_retry, result.json spectra.ir and the result card; anything still imaginary is reported. --from chaining off a frequency job now uses the geometry its frequencies used. Real cloud run, toluene RHF/STO-3G: 39.4i -> 10.1 cm-1 after a 7-step re-optimization, no imaginary modes left (energy change 5e-8 Ha: a nearly free rotor). 467 s for pre-opt + two frequency runs. Contributions: - Claude (Opus 5.5): code edits, review, and conceptual discussion - Jonathan Schultz: overall vision, planning, review, and orchestration Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
M-JOBDIRS (PR #127) gave every run of submit.slurm its own attempt folder and made it the worker's staging dir. The CL2.8/CHK.4 checkpoint (<staging>/.checkpoint) and the cached pre-opt geometry moved with it, so a resubmitted job, which starts in a new attempt folder, never saw them: killed geometry optimizations, PES scans and frequency runs have restarted from scratch since 2026-09-26. Found while reviewing B2.5. _job_state_dir() returns the job folder for an attempt folder (the legacy single staging dir is unchanged). New test: a second attempt resumes the first attempt's checkpoint; it fails on the old code. Contributions: - Claude (Opus 5.5): code edits, review, and conceptual discussion - Jonathan Schultz: overall vision, planning, review, and orchestration Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
From a /code-review of the M-BATCH2 branch: - A failed retry (exception, or a second frequency run without frequencies) no longer replaces the complete first result; it is kept with its imaginary modes and the log says so. - The retry runs only after a pre-opt that succeeded. A failed pre-opt falls back to the user's geometry, which may be a saddle on purpose. - Batch resume: the retry's start (displaced geometry) and its finished re-optimization are saved in the job folder, the re-optimization and the second frequency run are checkpointed, so a job killed during the retry resumes it instead of redoing the first frequency run. (The old comment claimed otherwise.) - preopt_trajectory.json and the app's pre-opt trajectory now end at the geometry the frequencies used (optimizer.join_optimizations). - The History card shows the retry too (shared saddle_retry_rows), with any modes still imaginary. - QUANTUI_SLURM_NV=1/0 forces --nv on/off for a GPU GRES not named "gpu" (e.g. shard:1); launcher port and parity test included. - worker_payload reuses molecule_to_dict; imaginary_mode_retry moved to the end of FreqResult. Not changed: the first pass still computes IR/Raman intensities before the imaginary-mode check; skipping them would need run_freq_calc to stop after the harmonic analysis, and the retry only fires on saddles. Contributions: - Claude (Opus 5.5): code edits, review, and conceptual discussion - Jonathan Schultz: overall vision, planning, review, and orchestration Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…B2.6) quantui/__init__.py re-exported ~90 names eagerly, so `import quantui` and every submodule import (quantui.cli, quantui.backends.*, the batch worker's startup) pulled in numpy, RDKit, ASE, Plotly/plotlymol3d, IPython, py3Dmol and requests. On the NCShare login node that took ~10 s and numpy could fail against the per-user thread limit, which is why the quantui-batch launcher re-implements the prepare step. All public names now resolve through the module __getattr__ (PEP 562), extending the existing lazy QuantUIApp/StepProgress/help entries; availability flags are computed on first access. Names are not cached, so patches on the defining submodule are seen. A name whose optional module cannot import raises AttributeError, as before. Cloud measurement (warm cache): import quantui 0.44 s -> 0.01 s; quantui, quantui.cli, quantui.backends.slurm/batch_input/worker now load none of the heavy packages (new subprocess test guards this). slurm_ingest imports save_result from results_storage for typing. Contributions: - Claude (Opus 5.5): code edits, review, and conceptual discussion - Jonathan Schultz: overall vision, planning, review, and orchestration Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…2026-09-26) On the 0.9.0 GPU image a frequency job used the GPU for its pre-opt and the IR displacement SCFs only; the reference SCF and the analytical Hessian ran on CPU (NCShare job 742762: 7900 s CPU / 662 s wall on 12 cores, GPU idle). run_freq_calc now calls gpu_offload.try_to_gpu after density fitting and PCM, as session_calc and the optimizer (PR #125) do, logs the "GPU offload active" marker, and computes the Hessian on the offloaded object (_analytical_hessian). If the GPU Hessian raises (ECP, dispersion or solvent terms gpu4pyscf may not cover; memory), it is recomputed on CPU from mf.to_cpu(). Right after the Hessian, mf goes back to CPU, so the IR/Raman displacement SCFs (which offload themselves), their dm0 and the thermochemistry see the same CPU object as before. HOMO-LUMO arrays go through _host_array. FreqResult and the batch payload record gpu_used/gpu_name. Cloud tests use a stand-in GPU object around a real water RHF/STO-3G run: offload attempted, Hessian on the offloaded object, to_cpu after it, CPU fallback when the GPU Hessian raises, CPU path unchanged. Hardware confirmation (H200, incl. the Mo ECP case) is owed. Contributions: - Claude (Opus 5.5): code edits, review, and conceptual discussion - Jonathan Schultz: overall vision, planning, review, and orchestration Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The worker writes live.log as UTF-8, but the tests read it with the
platform default encoding. On the Windows runner (cp1252) the new log
lines ("eV/Å", "cm⁻¹") failed to decode or to match, breaking three tests
in test_freq_saddle_retry and test_backends_worker. Every live.log read
in the tests now passes encoding="utf-8". Reproduced and verified
locally with a non-UTF-8 locale.
Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration
Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cluster-workflow follow-ups from M-BATCH2 (B2.1, B2.3, B2.5, B2.6), a fix for batch checkpoint resume, and GPU offload for frequency jobs.
Changes
SLURM batch scripts (B2.1)
--ntasks=1 --cpus-per-task=N(one multi-threaded worker) instead of N tasks. An inheritedSLURM_CPUS_PER_TASKno longer sets the thread count.QUANTUI_SLURM_GRES,QUANTUI_SLURM_QOSandQUANTUI_SLURM_ACCOUNTadd#SBATCHlines. They are validated before the job folder is written.apptainer exec --nvis used only when the GRES asks for a GPU.QUANTUI_SLURM_NV=1/0overrides that choice.pythonfrom the image's PATH, not the submitting process's interpreter path.quantui-batchlauncher port and its parity tests follow the same rules.Batch geometry optimization results (B2.3)
result.jsonnow recordsfinal_molecule, the dipole moment and Mulliken charges.--fromchaining uses the recorded geometry.Pre-optimization before Frequency (B2.5)
spectra.ir.imaginary_mode_retryrecord what was done.Batch checkpoint resume (fix)
Lazy
import quantui(B2.6)import quantui,quantui.cliandquantui.backends.*no longer load numpy, RDKit, ASE, Plotly, PySCF, IPython or requests.import quantuiwent from 0.44 s to 0.01 s here.Frequency jobs on the GPU
Testing
pytest -m "not network": 3810 passed, 19 skipped, 0 failed.--nvdecision, including parity with the launcher;Not checked yet (needs cluster hardware):
quantui submit --helptiming on the login node.Authorship