Skip to content

M-BATCH2: batch resources and GPU flags, complete opt results, frequency pre-opt + saddle retry, lazy imports, GPU Hessian - #141

Merged
NCCU-Schultz-Lab merged 9 commits into
mainfrom
claude/m-batch2-b21-b23-b25
Oct 10, 2026
Merged

NCCU-Schultz-Lab merged 9 commits into
mainfrom
claude/m-batch2-b21-b23-b25

Conversation

@NCCU-Schultz-Lab

@NCCU-Schultz-Lab NCCU-Schultz-Lab commented Oct 10, 2026 •

Copy link
Copy Markdown
Collaborator

Cluster-workflow follow-ups from M-BATCH2 (B2.1, B2.3, B2.5, B2.6), a fix for batch checkpoint resume, and GPU offload for frequency jobs.

Changes

SLURM batch scripts (B2.1)

  • Scripts ask for --ntasks=1 --cpus-per-task=N (one multi-threaded worker) instead of N tasks. An inherited SLURM_CPUS_PER_TASK no longer sets the thread count.
  • New optional settings QUANTUI_SLURM_GRES, QUANTUI_SLURM_QOS and QUANTUI_SLURM_ACCOUNT add #SBATCH lines. They are validated before the job folder is written.
  • apptainer exec --nv is used only when the GRES asks for a GPU. QUANTUI_SLURM_NV=1/0 overrides that choice.
  • Inside the image, the worker runs plain python from the image's PATH, not the submitting process's interpreter path.
  • The quantui-batch launcher port and its parity tests follow the same rules.

Batch geometry optimization results (B2.3)

  • result.json now records final_molecule, the dipole moment and Mulliken charges.
  • --from chaining uses the recorded geometry.

Pre-optimization before Frequency (B2.5)

  • It now converges to 0.01 eV/Å instead of 0.05.
  • If the frequencies still show an imaginary mode below 20i cm⁻¹ (typically a methyl rotor at its eclipsed saddle point), QuantUI displaces along the mode, re-optimizes and repeats the frequencies once. This applies in the app and in batch jobs.
  • The retry only runs after a pre-optimization that succeeded. If it fails, the first result is kept.
  • In batch, the retry resumes after a resubmission.
  • The result card, the History card and spectra.ir.imaginary_mode_retry record what was done.

Batch checkpoint resume (fix)

  • Since job folders got one attempt folder per run, the worker's checkpoint and cached pre-opt geometry were kept inside the attempt folder.
  • A resubmitted job never resumed: killed optimizations, PES scans and frequency runs started over.
  • Both now live in the job folder.

Lazy import quantui (B2.6)

  • Every public name is resolved on first use.
  • import quantui, quantui.cli and quantui.backends.* no longer load numpy, RDKit, ASE, Plotly, PySCF, IPython or requests.
  • import quantui went from 0.44 s to 0.01 s here.

Frequency jobs on the GPU

  • The reference SCF and the analytical Hessian are now offloaded like single points and optimizations.
  • If the GPU Hessian fails, it is recomputed on CPU.
  • After the Hessian, the IR/Raman steps and thermochemistry use the CPU object as before.

Testing

  • pytest -m "not network": 3810 passed, 19 skipped, 0 failed.
  • ruff and black: clean.
  • mypy, run as in CI: clean.
  • New tests cover:
    • script directives and the --nv decision, including parity with the launcher;
    • the geometry-opt payload;
    • the frequency pre-opt threshold;
    • the saddle retry, including its failure and resume paths;
    • checkpoint resume across attempt folders (this test fails on the old code);
    • the lazy-import guarantee;
    • frequency GPU offload with a stand-in GPU object.
  • Real worker run, toluene RHF/STO-3G with pre-opt: the first pass gave 39.4i cm⁻¹; after the retry the lowest mode is 10.1 cm⁻¹ and no imaginary modes remain.

Not checked yet (needs cluster hardware):

  • GPU and CPU jobs on NCShare with the new directives.
  • The Lab 4 MoO2(acac)2 frequency job on an H200, including whether the Mo ECP Hessian runs on the GPU.
  • Resume after a resubmission.
  • quantui submit --help timing on the login node.

Authorship

  • Claude (Opus 5.5): code edits, review, and conceptual discussion
  • Jonathan Schultz: overall vision, planning, review, and orchestration

NCCU-Schultz-Lab and others added 9 commits October 10, 2026 13:49
…(B2.1)

Generated batch scripts asked for --ntasks=N, while the worker is one
process with OpenMP threads. With --ntasks Slurm sets no
SLURM_CPUS_PER_TASK, so a value inherited from a submitting allocation
set OMP_NUM_THREADS. Scripts now ask for --ntasks=1 --cpus-per-task=N.

The worker always ran under `apptainer exec --nv`, yet no script ever
requested a GPU (GOTCHAS 2026-09-23). New optional settings
QUANTUI_SLURM_GRES / _QOS / _ACCOUNT add #SBATCH lines (validated
before the job folder is written); --nv is used only when the GRES asks
for a GPU.

Inside the image the worker now runs plain `python` from the image's
PATH: the app may run in the CPU image, whose sys.executable does not
exist in the GPU batch image.

The quantui-batch launcher's port follows; its parity test now covers
the GPU lines and normalizes the one intended difference (the launcher
keeps the image Python path recorded at install time).

Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration

Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…iken (B2.3)

optimization_result_payload wrote energy, steps and a trajectory pointer
only, though OptimizationResult already carries the final-SCF dipole and
Mulliken charges. CHEM-3200 Lab 3 needed an extra single-point preset
just to read the dipole, and batch_chain had to dig the geometry out of
trajectory.json.

The payload now adds final_molecule (atoms, coords, charge,
multiplicity) and the same ground-state fields the frequency and TD-DFT
payloads use; ingest already reads them. batch_chain prefers
final_molecule and falls back to the trajectory for older results.
Checked with a real worker run (H2O RHF/STO-3G: 1.709 D, charges
recorded).

Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration

Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The QM pre-optimization before a frequency calculation used the plain
Geometry Opt threshold (0.05 eV/Å). Harmonic frequencies need a
stationary point; the residual forces left soft modes wrong, and
CHEM-3200 Lab 4's MoO2(acac)2 came back with two imaginary methyl-rotor
modes (112i, 48i cm-1).

New optimizer.FREQ_PREOPT_FMAX = 0.01 eV/Å (about 2e-4 Ha/bohr, between
Gaussian's default and "tight" force criteria), used by the app's
Frequency pre-opt and the batch worker's frequency pre-opt, and logged.
Other calc types keep 0.05.

Cloud check, RHF/STO-3G, MMFF start, lowest modes 0.05 -> 0.01:
acetone 104.9 -> 119.1 cm-1 (+3 BFGS steps); propane 222.0 -> 228.6
cm-1 (+10 steps). No imaginary modes at either threshold for these two;
the shift in the rotor modes is the effect.

Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration

Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…ncy run (B2.5)

A tighter pre-opt threshold alone can land exactly on a rotor saddle:
toluene RHF/STO-3G went from 49.7 cm-1 (0.05 eV/Å, not a stationary
point) to 39.4i (0.01 eV/Å, the eclipsed methyl saddle), and Lab 4's
112i mode is too large to be convergence noise.

New freq_calc.retry_frequency_off_saddle: for modes below -20 cm-1 it
displaces along each (perturb_along_mode, 0.3 Å scale), re-optimizes at
FREQ_PREOPT_FMAX and reruns the frequencies, once. Used by the app and
the batch worker only when the run had its own pre-opt (a user-supplied
geometry may be a saddle on purpose). The retry is recorded in
FreqResult.imaginary_mode_retry, result.json spectra.ir and the result
card; anything still imaginary is reported. --from chaining off a
frequency job now uses the geometry its frequencies used.

Real cloud run, toluene RHF/STO-3G: 39.4i -> 10.1 cm-1 after a 7-step
re-optimization, no imaginary modes left (energy change 5e-8 Ha: a
nearly free rotor). 467 s for pre-opt + two frequency runs.

Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration

Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
M-JOBDIRS (PR #127) gave every run of submit.slurm its own attempt
folder and made it the worker's staging dir. The CL2.8/CHK.4 checkpoint
(<staging>/.checkpoint) and the cached pre-opt geometry moved with it,
so a resubmitted job, which starts in a new attempt folder, never saw
them: killed geometry optimizations, PES scans and frequency runs have
restarted from scratch since 2026-09-26. Found while reviewing B2.5.

_job_state_dir() returns the job folder for an attempt folder (the
legacy single staging dir is unchanged). New test: a second attempt
resumes the first attempt's checkpoint; it fails on the old code.

Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration

Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
From a /code-review of the M-BATCH2 branch:

- A failed retry (exception, or a second frequency run without
  frequencies) no longer replaces the complete first result; it is kept
  with its imaginary modes and the log says so.
- The retry runs only after a pre-opt that succeeded. A failed pre-opt
  falls back to the user's geometry, which may be a saddle on purpose.
- Batch resume: the retry's start (displaced geometry) and its finished
  re-optimization are saved in the job folder, the re-optimization and
  the second frequency run are checkpointed, so a job killed during the
  retry resumes it instead of redoing the first frequency run. (The old
  comment claimed otherwise.)
- preopt_trajectory.json and the app's pre-opt trajectory now end at the
  geometry the frequencies used (optimizer.join_optimizations).
- The History card shows the retry too (shared saddle_retry_rows), with
  any modes still imaginary.
- QUANTUI_SLURM_NV=1/0 forces --nv on/off for a GPU GRES not named
  "gpu" (e.g. shard:1); launcher port and parity test included.
- worker_payload reuses molecule_to_dict; imaginary_mode_retry moved to
  the end of FreqResult.

Not changed: the first pass still computes IR/Raman intensities before
the imaginary-mode check; skipping them would need run_freq_calc to stop
after the harmonic analysis, and the retry only fires on saddles.

Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration

Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…B2.6)

quantui/__init__.py re-exported ~90 names eagerly, so `import quantui`
and every submodule import (quantui.cli, quantui.backends.*, the batch
worker's startup) pulled in numpy, RDKit, ASE, Plotly/plotlymol3d,
IPython, py3Dmol and requests. On the NCShare login node that took ~10 s
and numpy could fail against the per-user thread limit, which is why the
quantui-batch launcher re-implements the prepare step.

All public names now resolve through the module __getattr__ (PEP 562),
extending the existing lazy QuantUIApp/StepProgress/help entries;
availability flags are computed on first access. Names are not cached,
so patches on the defining submodule are seen. A name whose optional
module cannot import raises AttributeError, as before.

Cloud measurement (warm cache): import quantui 0.44 s -> 0.01 s; quantui,
quantui.cli, quantui.backends.slurm/batch_input/worker now load none of
the heavy packages (new subprocess test guards this). slurm_ingest
imports save_result from results_storage for typing.

Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration

Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
…2026-09-26)

On the 0.9.0 GPU image a frequency job used the GPU for its pre-opt and
the IR displacement SCFs only; the reference SCF and the analytical
Hessian ran on CPU (NCShare job 742762: 7900 s CPU / 662 s wall on 12
cores, GPU idle).

run_freq_calc now calls gpu_offload.try_to_gpu after density fitting and
PCM, as session_calc and the optimizer (PR #125) do, logs the "GPU
offload active" marker, and computes the Hessian on the offloaded
object (_analytical_hessian). If the GPU Hessian raises (ECP, dispersion
or solvent terms gpu4pyscf may not cover; memory), it is recomputed on
CPU from mf.to_cpu(). Right after the Hessian, mf goes back to CPU, so
the IR/Raman displacement SCFs (which offload themselves), their dm0 and
the thermochemistry see the same CPU object as before. HOMO-LUMO arrays
go through _host_array. FreqResult and the batch payload record
gpu_used/gpu_name.

Cloud tests use a stand-in GPU object around a real water RHF/STO-3G
run: offload attempted, Hessian on the offloaded object, to_cpu after
it, CPU fallback when the GPU Hessian raises, CPU path unchanged.
Hardware confirmation (H200, incl. the Mo ECP case) is owed.

Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration

Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The worker writes live.log as UTF-8, but the tests read it with the
platform default encoding. On the Windows runner (cp1252) the new log
lines ("eV/Å", "cm⁻¹") failed to decode or to match, breaking three tests
in test_freq_saddle_retry and test_backends_worker. Every live.log read
in the tests now passes encoding="utf-8". Reproduced and verified
locally with a non-UTF-8 locale.

Contributions:
- Claude (Opus 5.5): code edits, review, and conceptual discussion
- Jonathan Schultz: overall vision, planning, review, and orchestration

Co-authored-by: Jonathan Schultz <nccu-schultz-lab@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
@NCCU-Schultz-Lab
NCCU-Schultz-Lab merged commit 3bc2976 into main Oct 10, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant