Repository navigation
Save sources over SSH/SFTP - #33
Closed
DionJones615 wants to merge 10 commits into
Closed
DionJones615 wants to merge 10 commits into
DionJones615 wants to merge 10 commits into
Conversation
The desktop shell can browse and load the .sav from the machine that runs the game using what that machine already has -- its OpenSSH server -- with a mounted SMB/NFS share as the second transport. Nothing listens on the game machine and no port is opened for it: `ssh`, `sftp` and `rsync` are run as subprocesses, so host keys, keys, ssh-agent, `~/.ssh/config` and ProxyJump stay OpenSSH's business and the app ships no SSH stack to keep current. (Upstream had an SFTP sync panel in the Flask era, with the password in a config file; it went with the Python runtime. This brings the capability back client-side, key/agent only.) One dialog, Save File -> From another machine: - Validate connection answers what the transport can do before a path is typed: SSH login, whether the SFTP subsystem answers (browsing and change detection depend on it), whether the far side has rsync, the platform and home, and how many .sav files are in the folder. A partial answer stays partial -- no SFTP subsystem is reported as such, not as a failed login. - Save folder lists the remote directory and walks it (folders first, .sav marked, Up/parent, keyboard reachable). Remote names are untrusted data and are rendered as text. Without SFTP the listing is disabled and the path is typed by hand. - Watch it and reload polls name/size/timestamp -- one listing, not a download -- and reloads only when the fetched bytes hash differently, so a coarse timestamp can cost a transfer but never a wrong reload. Paused while the editor holds pending edits, while a parse runs, and while the window is hidden; it resumes on its own. Rust: rust_parser/tauri/src/ssh_source.rs (listing parse, ssh config, the three transports, hashing) plus five commands -- ssh_hosts, source_validate, source_browse, source_stat, source_fetch. Every remote path crosses into an SFTP batch script, so control characters and quotes are rejected at the boundary, and a host/user/port that could be read as an option is rejected too. A fetched save lands in the app data dir under remote-saves/, written to a .part and renamed only when complete, capped at 8 GiB. Checks: 9 unit tests over real `sftp -b -` output (including OpenSSH-for-Windows rows with ? owner/group and full-path names), a live run against a local sshd covering validate/browse/stat/fetch over sftp and rsync, names with spaces, change detection and every refusal path, the module plus the command bodies type-checked against real serde/sha2, and the dialog driven in a browser with a stubbed bridge (both transports, watch ticks, capability gating, pending-edit pause, no markup injection).
The dialog is Tauri-gated, and the Tauri tree needs WebKitGTK/GTK development packages an immutable distro does not carry -- so on the machine reviewing this, there was no way to look at the thing being reviewed. `tools/serve_save_source_demo.py` serves map/static/map/ with a stubbed desktop bridge instead: the three commands answer from a real folder, so the listing, its sizes and timestamps, the "newest .sav" pick and the watch's change check are all real (drop a file in the folder and the next tick notices). Only the SSH transport is canned, and loading stops at the parser -- sav_core is the desktop build, dist/pkg the wasm build -- so the status line says that rather than pretending. docs/remote-save-ssh.md gains the three ways to run this: the demo server, the distrobox recipe for the real app on Linux, and the Windows build plus the sshd setup. It also records the one local trap worth knowing: an ostree install can leave /etc/ssh/ssh_config.d/* owned by an unmapped uid, which OpenSSH treats as fatal, so every ssh invocation dies before it connects and the dialog shows the client's own "Bad owner or permissions" line until it is fixed.
The dialog put every control on one screen: two protocol tabs, seven fields, a four-row connection report, a folder listing, three checkboxes and two blocks of prose, all competing for the same panel. It now walks the three decisions in order -- Protocol, Connection, Save to load -- with only the current step laid out, a stepper whose numbers go back, Back/Next in the foot, and focus landing on the first field of each step. - Protocol is two explained choices instead of two tabs. - Connection is that protocol's questions, one Validate button, and -- when it fails -- the ssh client's own words plus the one thing to try next (a refused key, an unknown host key, a dead port, a machine whose own ssh is broken, a server with no SFTP subsystem). The old four-row report was mostly a restatement of the error; a passing validate shows three lines, a failing one shows the error and no rows. - Save to load is the folder, the file and the freshness toggles, with the path guessed only when the platform is one the game's folders are known for: it used to append the Proton path to any non-Windows home, which is a wrong path rather than no path. - A remembered source reopens at the file it was left on, and the dialog no longer greets the user with the previous session's last watch tick. Wording: "Save file" reads as a verb phrase. The third step is "Save to load" and the sidebar panel it opens from is "Your save" -- a determiner, so the noun reading is the only reading. The docs and the demo server follow. Also: tools/serve_save_source_demo.py reports platform "demo", which is what makes the no-guess path above visible, and it serves the failure path when the host contains "fail" so the debug screen can be looked at without a game machine.
…llback A change is now pushed to the map instead of waited for. `inotifywait -m` runs on the machine that holds the folder -- locally, or over the ssh connection -- and each name it prints wakes the same tick that the interval used to: re-list, compare, hash, reload only if the bytes moved. On Windows the equivalent is PowerShell's FileSystemWatcher, sent as a base64 `-EncodedCommand` (UTF-16LE), which is the only quoting that survives cmd.exe, the default shell of Windows OpenSSH; the encoder is checked against coreutils/iconv byte for byte. The interval stays, but only where events are impossible, and the dialog says which one is in use: - no inotifywait on the machine holding the saves; - a mounted share on Linux, where inotify can never fire: it only reports writes made through this kernel's own VFS, so the game machine writing into an SMB or NFS share is invisible -- silent rather than late, which is worse. The filesystem type is checked with findmnt before choosing, so this is decided rather than hoped. Windows is the exception: FileSystemWatcher does receive SMB notifications, so a UNC path or mapped drive there still gets events. An event is a nudge, not a verdict: the burst a save write produces is debounced (1.5s), then the listing comparison and the byte hash decide. A missed or spurious event therefore cannot reload the map wrongly, and a watcher that dies at once -- no inotifywait, no such folder, a refused key -- reports the tool's own words and the frontend falls back to the interval with that reason in the status line. Verified against a real sshd with an inotifywait test double: the local watcher (a write produced an event), the remote one (a write on the far side arrived over the connection), mode selection with and without the tool, and the dead-watcher error path. The browser run covers arming with events (zero intervals scheduled), a debounced event causing one check, non-.sav events ignored, turning it off, and the fallback when start fails.
Checked against the real machine (bazzite@172.27.30.50, Bazzite, 1ms away): validate, browse, fetch over sftp and rsync, and the change check all work on 166 real saves; rsync is there, so re-fetches are deltas. Two things came out of it. The game's layout is <SaveGames>/<steamid64>/*.sav, and the level above it holds that one folder plus the dedicated server's 116-byte ServerManager_V2.sav -- so "newest .sav" there picked a file that is not a save. When the single folder beside the saves is the most recently written thing, the browser now walks into it once: the dialog opens on the saves instead of one click above them. Up still goes back, and it does not re-descend. And a correction: the "Bad owner or permissions on /etc/ssh/ssh_config.d/..." note added earlier blamed an ostree ownership bug on the user's install. It is not that -- those files are root-owned, and inside a user namespace root maps to an unmapped uid, so they look like nfsnobody to the ssh client. The docs now say what it actually is and what to do (run outside the namespace, keep your uid, or `ssh -F /dev/null`), instead of telling people to chown a healthy system file. tools/serve_save_source_demo.py answers a listing for the folder the dialog asks for, inside the folder it was pointed at, so browsing a real tree -- down, up, and the walk-into-the-only-folder above -- can be seen locally.
The watch was reachable only through the SSH/mounted-share dialog, so a save opened from disk -- the file dialog, a drag-drop -- sat there going stale while the game autosaved. It is the same machinery now, for any loaded save with a path: the folder is watched, the filesystem decides events or interval, and the bytes decide whether the map reloads. - A "Keep it fresh" switch appears under the loaded save in the sidebar (desktop only -- a browser File object has no path to watch), and disappears when the loaded save is remote or nothing is loaded. save_source.js owns the watch, so it owns the switch: data.js only says what was loaded. - `source_watch_start` picks the mechanism itself when the caller passes no mode, which is what a local path needs: it never went through a validate, so nothing ever probed that machine for inotifywait or asked findmnt what filesystem the folder is on. The platform is an optional argument now. - A picked file means that file -- following "newest" would swap the map onto a save nobody opened -- so when a strictly newer save lands in the folder the status line names it and the dialog's "Newest .sav" toggle is one click away. Same-minute saves are not called newer than each other. Verified in the browser: adopting a path shows the switch, ticking it arms the watcher with zero intervals scheduled, an event on that file reloads through the same path the dialog uses, the dialog reflects the same state, unloading hides the switch, and a strictly newer save in the folder is named in the status line. The Rust side still passes its 11 unit tests, the live sshd run (local and remote watchers, both transports, every refusal) and the real-host run.
Two failures were wearing the same mask: the browser's own "Failed to fetch",
which says nothing about whether ssh answered.
- The demo server now reports its own death ("the demo server is gone --
restart tools/serve_save_source_demo.py and reload this page") instead of
letting the browser's message read like a connection failure.
- The dialog recognises a transport-level failure and says what it is: the
app's command channel, not the remote host. On Linux the desktop shell carries
its commands over a local fetch, so a backend that is gone, or a frontend
rebuilt against an older binary, arrives as exactly this string -- the hint
now says restart/rebuild rather than leaving the user to guess at ssh.
- docs: the case is listed with the other troubleshooting entries.
Reproduced first: with the stub's fetch rejecting, the dialog shows
"Failed to fetch" with "Not connected." and no check rows, byte for byte what a
dead demo server produces.
Reported as "it connects, but there is no way to navigate the paths on the ssh
host", and reproduced: after a successful validate, step 3 had an empty path and
a completely blank list -- no rows, no text, nothing to click. The browser only
knew where to start when the platform was Windows or Linux/macOS (the game's
usual folder); anything else, including the demo bridge, left it with nothing,
and a blank list says nothing about what to do about it.
- suggestDir falls back to the remote home directory, which validate always
reports: the game's folder when we know it, somewhere real to walk from
otherwise. On the real machine that lands on the saves directly (the layout
fix from the previous commit walks into the <steamid64> folder).
- A guessed folder that does not exist -- a second Steam library, a host whose
layout we do not know -- falls back to the home directory once and says so,
instead of dead-ending on "Can't ls: not found". Every Open or Up is its own
attempt, so a second try is not swallowed by the first fallback.
- Step 3 with no folder at all explains what to do ("go back and press Validate
connection, or type a path and press Open") rather than rendering nothing.
- A failure that cannot be recovered goes to the status line as well as the
list, so the line never sits on a stale "Listing /..." .
The demo's listing is strict now too: a path outside the folder it was pointed
at, or one that is not there, is refused the way the real backend refuses it,
instead of silently answering with a different folder.
Save sources over SSH/SFTP (opposing take on #2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Provenance
Same change as DionJones615/satisfactory-live-map#4, which was merged into that fork; its branch was deleted after the merge, so this re-creates it on top of upstream
main(the fork'smainis exactly upstreammain+ this). Nothing here depends on the fork.What
Instead of asking the gaming PC to host its save folder over HTTP, this loads the
.savoff that machine with what it already has: its OpenSSH server. A mounted SMB/NFS share is the second transport.The pitch is what is not added: no listener on the game machine, no port opened for this app, no service to keep running, no password stored.
ssh,sftpandrsyncrun as subprocesses, so host keys, keys,ssh-agent,~/.ssh/configandProxyJumpare all OpenSSH's, and the app ships no SSH stack to keep current.Upstream had an SFTP sync panel in the Flask era (
sftp_config.json: host, password,remote_path,sync_interval_seconds); it was deleted with the Python runtime in f0ea245. This brings the capability back client-side, key/agent only —BatchMode=yesmeans no password is ever asked for or stored.One dialog, three steps: Your save → From another machine
.savmarked, click to descend,Upfor the parent, or type a path and pressOpen), then the file. It opens at the game's usual folder when the platform is known and otherwise at the remote home directory, so there is always something to walk from; a guess that turns out not to exist falls back there and says so.Only the current step is laid out, the stepper's numbers go back, focus lands on the first field of each step, and a remembered source reopens at the file it was left on. Remote names are rendered as text, never as markup.
Naming: "Save file" reads as a verb phrase, so the step is Save to load and the sidebar panel it opens from is Your save — a determiner, so the noun reading is the only reading.
Changes are pushed, not polled, whenever the machine can push them
inotifywait -mruns on the machine that holds the folder — locally or over the ssh connection — and every name it prints wakes the same check; on Windows it is PowerShell'sFileSystemWatcher, sent as a base64-EncodedCommand(UTF-16LE), the only quoting that survivescmd.exe. Nothing is polled while events are live.The interval stays for the two cases where events are impossible, and the dialog says which one you got:
inotifywaiton that machine (Fedora/Bazzite:rpm-ostree install inotify-tools; Debian:apt install inotify-tools). Windows needs nothing — the watcher is part of .NET.inotifycan never fire: it only reports writes made through this kernel's own VFS, so the game machine writing into an SMB or NFS share is invisible — silent rather than late, which is worse. The filesystem type is checked withfindmntbefore choosing. Windows is the exception:FileSystemWatcherdoes receive SMB notifications.An event is a nudge, not a verdict: the burst a save write produces is debounced, then the folder is re-listed and the fetched bytes are hashed, so a missed or spurious event cannot reload the map wrongly. A watcher that dies at once — no
inotifywait, no such folder, a refused key — reports the tool's own words and falls back to the interval with that reason on screen. Reloading is refused while the editor holds pending edits, while a parse runs, and while the window is hidden.A save opened from this machine
The same watch applies to a
.savopened from disk — the file dialog, a drag-drop, or the dialog's mounted-share transport. A Keep it fresh switch appears under the loaded save in the sidebar (desktop only: a browserFileobject has no path to watch), driving the same machinery;source_watch_startpicks the mechanism itself when the caller doesn't know it, which is what a local path needs. A file you picked means that file, so when a strictly newer save lands in the folder the status line names it and the dialog's Newest.savtoggle is one click away.Shape
Three commands for the dialog's three jobs —
source_validate,source_browse,source_fetch— plussource_watch_start/source_watch_stopfor the event watcher (a long-lived child process, local or over ssh, kept in Tauri state because a webview cannot hold one). The watch probe is asource_browse: a listing is what says both what is in the folder and whether it moved, so there is no separate stat command.main.rsgrew ~130 lines; the SSH work lives in one module (ssh_source.rs) with its tests.Trust boundaries kept: every remote path crosses into an SFTP batch script, so control characters and quotes are rejected at the boundary; a host, user or port that could be read as an option is rejected; a fetched save is written to a
.partfile and renamed only when complete, capped at 8 GiB.Tested against the real machine
bazzite@172.27.30.50(Bazzite, 1 ms away), which is where the game actually runs — the app built from this branch was pointed at it and works:/var/home/bazzite, 166 saves, newestNo.42_autosave_1.savinotifywaitis not installed there, so the probe chose polling and the dialog showsinotifywait: command not foundbefore falling backThe real layout is
<SaveGames>/<steamid64>/*.savwith the dedicated server's 116-byteServerManager_V2.savbeside it, so the browser walks into the only folder when that folder is the most recently written thing — otherwise "newest.sav" picks a file that is not a save.Other checks: 11 unit tests over verbatim
sftp -b -output (the batch-mode echo, the?link count OpenSSH never sends, full-path names,Jan 3 2025double-spaced dates) and the Windows-shaped rows; a live run against a localsshdcovering validate/browse/stat/fetch over sftp and rsync, names with spaces, change detection and every refusal path, plus both watchers (a write produced an event locally, and one on the far side arrived over the connection); the module and the command bodies type-checked against real serde/sha2; the base64/UTF-16LE encoder checked againsticonv | base64byte for byte, with the decoded PowerShell read back; and a handler audit (27 registered, none missing — the same audit findsstartup_urlundefined in #2).Trying it
docs/remote-save-ssh.mdhas the distrobox recipe for the Linux-develpackages, the Windows build, the sshd setup for the game machine, and a note for hosts where-develpackages cannot be installed at all.tools/serve_save_source_demo.pyserves the dialog with a stubbed bridge for anyone who wants to look at its states without a build.Known gaps
sshdwas not exercised — the Windows-shaped listing rows are fixtures and theFileSystemWatcherscript is verified by decoding, not against a real Windows host.sshchild can outlive it until the remote closes the channel..savURL is shareable and launchable from a shortcut, and HTTP needs no sshd on the game machine. The two can coexist — separate commands, separate dialogs.