Repository navigation
Conversation
… restart when the job log rotates
…ead of ending the run
github-actions
Bot
added
Module: Scheduler
Module: Species
Module: Local
Module: trsh
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| stop_heartbeat.set() | ||
| try: | ||
| os.remove(heartbeat_path) | ||
| except FileNotFoundError: |
This was referenced Nov 2, 2026
kfir4444
changed the title
| write_json_atomic(os.path.join(dirs['done'], f'{ticket}.json'), content, dirs['tmp']) | ||
| try: | ||
| os.remove(os.path.join(dirs['running'], f'{ticket}.json')) | ||
| except FileNotFoundError: |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## mlip_run_fixes #1073 +/- ##
==================================================
+ Coverage 67.12% 67.15% +0.02%
==================================================
Files 123 123
Lines 43706 43772 +66
Branches 11161 11175 +14
==================================================
+ Hits 29338 29393 +55
+ Misses 11224 11223 -1
- Partials 3144 3156 +12
Flags with carried forward coverage won't be shown. Click here to find out more. 🚀 New features to boost your workflow:
|
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Critical worker-ticket lifecycle defects can hang runs, trigger concurrent or duplicate execution, and cause stale results to be accepted.
10 open issues
What changed in this PR
Adds persistent ASE workers so ARC can reuse loaded machine-learned potentials across jobs instead of reloading models for every calculation. This provides the worker layer for the follow-on ASE enhancements.
Changes:
- Introduces a file-spool worker with model caching, heartbeats, ticket tracking, and direct-execution fallback.
- Adds worker lifecycle tests and isolates finite-difference vibration caches by job.
- Uses insertion order for optimization troubleshooting and makes local queue-deletion failures nonfatal.
| File | Description |
|---|---|
arc/settings/settings.py |
Adds worker spool configuration through ASE_WORKERS. |
arc/scheduler.py |
Tracks worker tickets and changes optimization-attempt selection. |
arc/job/local.py |
Logs unresolved deletion failures instead of raising. |
arc/job/adapters/scripts/ase_script.py |
Implements persistent workers, reusable model caches, and job-scoped vibration caches. |
arc/job/adapters/ase_test.py |
Tests ticket handling, worker execution, fallback, cache isolation, and IRC endpoints. |
arc/job/adapters/ase_adapter.py |
Adds asynchronous worker submission and ticket-status handling. |
Review effort: Max
| self.script_path = os.path.join(os.path.dirname(__file__), 'scripts', 'ase_script.py') | ||
| # A live persistent worker runs this calculator's jobs on this machine. They are handed to it as | ||
| # local queue jobs: submitted without waiting, and tracked by their ticket IDs (see execute_queue()). | ||
| self.worker_spool = self.determine_worker_spool() |
| self.script_path = os.path.join(os.path.dirname(__file__), 'scripts', 'ase_script.py') | ||
| # A live persistent worker runs this calculator's jobs on this machine. They are handed to it as | ||
| # local queue jobs: submitted without waiting, and tracked by their ticket IDs (see execute_queue()). | ||
| self.worker_spool = self.determine_worker_spool() |
| reactions=reactions, | ||
| rotor_index=rotor_index, | ||
| server=server, | ||
| server='local' if self.worker_spool else server, |
Comment on lines
+392
to
+393
| self.job_id = submit_ticket(self.worker_spool, os.path.abspath(os.path.join(self.local_path, 'input.yml')), | ||
| label=self.job_name) |
| if content.get('started') and content.get('finished'): | ||
| self.initial_time = datetime.datetime.fromtimestamp(content['started']) | ||
| self.final_time = datetime.datetime.fromtimestamp(content['finished']) | ||
| return 'done' |
| if state == 'queued' and not live_workers(spool) and withdraw_ticket(spool, ticket): | ||
| logger.warning(f'No live ASE worker on {spool}; running {self.job_name} directly.') | ||
| return self.run_instead_of_worker() | ||
| if state == 'running' and content.get('worker') not in live_workers(spool): |
Comment on lines
+448
to
+451
| self.initial_time = datetime.datetime.now() | ||
| self.run_directly() | ||
| self.final_time = datetime.datetime.now() | ||
| return 'done' |
| self.server_job_ids.extend(check_running_jobs_ids()) | ||
| if specific_server in (None, 'local'): | ||
| # Jobs handed to a persistent ASE worker run on this machine, tracked by their ticket IDs. | ||
| self.server_job_ids.extend(active_worker_tickets()) |
Comment on lines
+170
to
+173
| # E.g., PBS refuses to delete a job that is already exiting, and qstat still lists it. The job ends on its | ||
| # own and the caller has dropped it from its running jobs, so this must not end the ARC run. | ||
| logger.error(f'Job {job_id} was scheduled for deletion, but the deletion command has appeared to errored. ' | ||
| f'The job is still running.') | ||
| raise RuntimeError(f'Could not delete job {job_id}') | ||
| f'The job is still running; leaving it to finish on its own.') |
Comment on lines
+4880
to
+4882
| opt_jobs = list(self.job_dict[label]['opt'].values()) | ||
| job = opt_jobs[-1] if opt_jobs else None | ||
| previous_job = opt_jobs[-2] if len(opt_jobs) > 1 else None |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
A persistent ASE worker that keeps machine-learned potentials (MLIPs, here UMA) loaded between ARC jobs, and two scheduler/job fixes found while running ARC on MLIPs at scale on a PBS cluster.
Stack: builds on #1072 (
mlip_run_fixes). It is the first of four PRs: worker (this one) ← Hessians (#1075) ← NEB (#1076) ← TS scans (#1077).Changes
ase_script.py --serve) keeps models loaded and takes jobs from a file spool (claims by atomicrename, heartbeats). The adapter submits a ticket to a live worker and falls back to running the job directly when no worker is left or its worker dies. A smoke test ran 1.64 times faster in wall time. The!Fcommit moves Fix failures of ARC runs on machine-learned potentials (ASE) found in a benchmark #1072's IRC test to the worker'srun_job().initiated_jobs.csv, because job numbers restart there. It now uses the order in which the jobs were spawned.qstatstill lists it, sodelete_job()raised. It now logs the error and leaves the job to finish.Checks
ase_test.py,scheduler_test.py,trsh_test.py,species_test.pyandchecks/ts_test.pypass, except two tests that already fail onmainin this environment because they run Arkane (test_check_rxn_e0,test_compute_rxn_e0).Tests
test_serve_runs_tickets_and_survives_a_failed_job,test_a_claimed_ticket_cannot_be_claimed_or_withdrawn,test_live_workers_ignores_a_stale_heartbeat,test_execute_incore_through_a_live_worker,test_execute_through_worker_withdraws_when_no_worker_is_left,test_execute_through_worker_falls_back_when_its_worker_dies,test_a_live_worker_takes_jobs_without_blocking,test_a_worker_job_ends_when_its_ticket_does,test_a_ticket_no_worker_is_left_to_take_runs_directly,test_a_job_whose_worker_died_runs_directly,test_a_queue_job_id_is_not_a_ticket,test_no_live_worker_leaves_the_execution_type_alone,test_run_vibrational_analysis_ignores_a_stale_cache, andtest_an_irc_job_ends_at_the_end_of_its_path(moved)