Add tests to ensure we don't leak detailed errors

[StanFromIreland]

We don't have coverage for these cases (to ensure we don't leak any other output), but that can be a follow up.

Originally posted by @StanFromIreland in #61