Possibly no safety check when staged file is corrupted with Cntrl-C Interrupt

[manishvenu]

Hello,

I'm not sure if this is a question or a feature request, but I recently tried to rimport a new runoff dataset. Halfway through the first file, I interrupted it with Cntrl-C, and it looks like the file was corrupted in the staging area (about 8GB too small).

If I had permissions for relinking (which thankfully I didn't have) I think the original file would have be overwritten when I ran rimport again.

Is there anyway to have rimport be able to abort the copy operation cleanly? Or maybe have a check with checksums or at least file sizes to ensure consistency before relinking?

Also, Claude had me do sudo -u cesmdata rm to get rid of the corrupted file, which worked (I hope?)

Thanks,
Manish V.

[samsrabin]

Great idea! Unfortunately I'm not sure we can implement it here; it all depends on whether we can reliably determine whether the file has been made available publicly by GDEX. For that, I need to know more about the technical details behind their publication process. I'll look into it.

[samsrabin]

Sorry, I didn't think that through fully—yes, we definitely should add

1. Something in the relink step that prevents (or at least warns about) deletion of the Campaign-space version if the GDEX-space version doesn't match it exactly, and
2. Something in the copy step that immediately deletes the GDEX-space version if the copy didn't complete perfectly. (rsync has this option built in.)

[samsrabin]

@manishvenu Are you still down to implement these changes? I think the "whether we can reliably determine whether the file has been made available publicly by GDEX" question is more or less irrelevant here, as it's pretty improbable that the GDEX uploader would start processing the file before our scripts even finish running.

[manishvenu]

@manishvenu Are you still down to implement these changes? I think the "whether we can reliably determine whether the file has been made available publicly by GDEX" question is more or less irrelevant here, as it's pretty improbable that the GDEX uploader would start processing the file before our scripts even finish running.

Sure! I'm assuming both of them?

[samsrabin]

Yes please! Ideally in separate PRs, or at least one PR with a clear and useful commit history.