Repository navigation
Conversation
When <ssl> has extra_conf, in_prometheus serves the metrics with WEBrick and loads private_key_path with OpenSSL::PKey.read. That call was given the path instead of the file content since bc7c2ba, so it always raised "Could not parse PKey" and the plugin failed to start. Read the file before parsing it, as is done for certificate_path. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the fix and the clear reproduction.
With the file-path configuration in the PR description, I confirmed that master fails to start with Could not parse PKey, while this branch serves the configured certificate. The new spec fails on master and passes here. Reading the file before OpenSSL::PKey.read also matches how fluentd's cert_option loads the key.
I left one MUST comment. It is about a config that puts the PEM text itself in private_key_path. With this change, that config writes the whole private key to the log, even when the config dump is turned off.
|
|
||
| if @ssl['private_key_path'] | ||
| key = OpenSSL::PKey.read(@ssl['private_key_path']) | ||
| key = OpenSSL::PKey.read(File.read(@ssl['private_key_path'])) |
There was a problem hiding this comment.
MUST: With this change, a config that works on master writes the whole private key to the log at startup, even with --suppress-config-dump or -qq.
On master, private_key_path can hold the PEM text itself instead of a path. OpenSSL::PKey.read parses that string, so the plugin starts. This only worked because of the bug. I am fine with it no longer working, since the parameter is documented as a path.
With this change, File.read treats the PEM text as a file name and raises a file system error, such as Errno::ENOENT. The exception message contains the whole key, and fluentd writes it to its log. On master, the key is not logged when the config dump is turned off (--suppress-config-dump) or the log level is error (-qq). With this change, it is.
Example:
<ssl>
enable true
certificate_path /path/to/cert.pem
private_key_path "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n"
extra_conf {"SSLVerifyClient": 0}
</ssl>
Could you make a failure to read the file raise an error that does not include the value, and add a spec for it? The original exception should not be kept as the cause either, because its message has the value. Any path that this PR can read now should keep working. One way:
| key = OpenSSL::PKey.read(File.read(@ssl['private_key_path'])) | |
| begin | |
| key_contents = File.read(@ssl['private_key_path']) | |
| rescue SystemCallError => e | |
| raise Fluent::ConfigError, "failed to read private_key_path (#{e.class})", cause: nil | |
| end | |
| key = OpenSSL::PKey.read(key_contents) |
This spec, placed inside the context you added, fails on the current branch and passes with the change above:
context 'when private_key_path holds the key itself instead of a path' do
let(:private_key_path) { %["#{key.to_pem.gsub("\n", '\n')}"] }
it 'raises an error that does not include the key' do
expect { driver.run(timeout: 1) }.to raise_error(Fluent::ConfigError, /\Afailed to read private_key_path \(Errno::\w+\)\z/) { |e| expect(e.cause).to be_nil }
end
end…read When private_key_path holds the PEM text itself, File.read raised an error whose message had the whole key, and fluentd logged it. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
|
Thanks, good catch. Pushed a015416 with your suggestion: a I added your spec inside the new context. On the previous commit it fails with
|
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for applying the change so quickly. The file-read error in the WEBrick path no longer includes the private key, and its cause is cleared, so the issue in my previous comment is fixed: #276 (comment)
However, my suggestion there caused a new problem, and I am sorry about that. A Fluent::ConfigError raised in start makes fluentd stop entirely instead of restarting the worker. I left a MUST comment with a fix, and a matching change to the spec.
| begin | ||
| key_contents = File.read(@ssl['private_key_path']) | ||
| rescue SystemCallError => e | ||
| raise Fluent::ConfigError, "failed to read private_key_path (#{e.class})", cause: nil |
There was a problem hiding this comment.
MUST: My previous suggestion was wrong here. A Fluent::ConfigError raised in start makes the worker exit with status 2. The supervisor treats that as unrecoverable and stops all workers.
This stops a worker that keeps processing logs on master. For example:
<system>
workers 2
</system>
<worker 0>
<source>
@type sample
tag sample
</source>
<match sample>
@type stdout
</match>
</worker>
<worker 1>
<source>
@type prometheus
<ssl>
enable true
certificate_path /path/to/cert.pem
private_key_path /path/to/missing.pem
extra_conf {"SSLVerifyClient": 0}
</ssl>
</source>
</worker>
On master, worker 1 keeps restarting, and worker 0 keeps processing events. With this commit, fluentd exits with status 2 after about two seconds, so worker 0 stops too. The same happens when private_key_path holds the PEM text itself.
It also removes recovery when the key file appears after startup. With the previous commit, the worker restarted and served metrics once the file was there. With this commit, fluentd stays stopped.
Could you keep the original SystemCallError subclass, so the worker restarts as before? Please keep cause: nil. Only the message needs to change. Readable paths, including symlinks and FIFOs, should still go through the same File.read call. With the change below, the message for Errno::ENOENT becomes No such file or directory - private_key_path.
| raise Fluent::ConfigError, "failed to read private_key_path (#{e.class})", cause: nil | |
| raise e.class, 'private_key_path', cause: nil |
| let(:private_key_path) { %["#{key.to_pem.gsub("\n", '\n')}"] } | ||
|
|
||
| it 'raises an error that does not include the key' do | ||
| expect { driver.run(timeout: 1) }.to raise_error(Fluent::ConfigError, /\Afailed to read private_key_path \(Errno::\w+\)\z/) { |e| expect(e.cause).to be_nil } |
There was a problem hiding this comment.
MUST: This is the spec change for my comment on in_prometheus.rb line 184. The error is no longer a Fluent::ConfigError.
Comparing with a new exception of the same class checks that the message has only the errno text and the parameter name. It works with any errno, such as Errno::ENAMETOOLONG, because the PEM text can be too long for a file name. The cause must stay nil.
| expect { driver.run(timeout: 1) }.to raise_error(Fluent::ConfigError, /\Afailed to read private_key_path \(Errno::\w+\)\z/) { |e| expect(e.cause).to be_nil } | |
| expect { driver.run(timeout: 1) }.to raise_error(SystemCallError) { |e| | |
| expect(e.message).to eq(e.class.new('private_key_path').message) | |
| expect(e.cause).to be_nil | |
| } |
in_prometheusdoes not start when the deprecated<ssl>section hasextra_conftogether withcertificate_pathandprivate_key_path.With
extra_conf, the plugin serves the metrics with WEBrick (start_webrick), which loads the key withOpenSSL::PKey.read(@ssl['private_key_path']). That parses the path string, not the file, so it always raises. It has been like this since bc7c2ba ("Use OpenSSL::PKey.read"), which replacedOpenSSL::PKey::RSA.new(File.read(path))and dropped theFile.read.certificate_pathright above it is read correctly.The fix reads the file before parsing it.
Reproduction
master:
this branch:
Tests
serves the given certificateunder "when extra_conf is used with certificate_path and private_key_path" inspec/fluent/plugin/in_prometheus_spec.rb: writes a generated key and self-signed certificate to a tmpdir, requests/metricsover https and checks the peer certificate is the configured one. On master it fails withOpenSSL::PKey::PKeyError: Could not parse PKey.bundle exec rake: 185 examples, 0 failures (Ruby 4.0.7, fluentd 1.20.1).