Skip to content

Read the private key file when <ssl> is served by WEBrick - #276

Open
lbarrosby wants to merge 2 commits into
fluent:masterfrom
lbarrosby:fix-webrick-ssl-private-key
Open

lbarrosby wants to merge 2 commits into
fluent:masterfrom
lbarrosby:fix-webrick-ssl-private-key

Conversation

@lbarrosby

Copy link
Copy Markdown
Contributor

in_prometheus does not start when the deprecated <ssl> section has extra_conf together with certificate_path and private_key_path.

With extra_conf, the plugin serves the metrics with WEBrick (start_webrick), which loads the key with OpenSSL::PKey.read(@ssl['private_key_path']). That parses the path string, not the file, so it always raises. It has been like this since bc7c2ba ("Use OpenSSL::PKey.read"), which replaced OpenSSL::PKey::RSA.new(File.read(path)) and dropped the File.read. certificate_path right above it is read correctly.

The fix reads the file before parsing it.

Reproduction

<source>
  @type prometheus
  port 24252
  <ssl>
    enable true
    certificate_path /path/to/cert.pem
    private_key_path /path/to/key.pem
    extra_conf {"SSLVerifyClient": 0}
  </ssl>
</source>

master:

[error]: #0 unexpected error error_class=OpenSSL::PKey::PKeyError error="Could not parse PKey"
  [error]: #0 lib/fluent/plugin/in_prometheus.rb:181:in 'OpenSSL::PKey.read'
  [error]: #0 lib/fluent/plugin/in_prometheus.rb:181:in 'Fluent::Plugin::PrometheusInput#start_webrick'
  [error]: #0 lib/fluent/plugin/in_prometheus.rb:95:in 'Fluent::Plugin::PrometheusInput#start'

this branch:

$ curl -sk -o /dev/null -w 'HTTP %{http_code}\n' https://localhost:24252/metrics
HTTP 200
$ echo | openssl s_client -connect localhost:24252 2>/dev/null | grep subject=
subject=CN=localhost

Tests

  • New example serves the given certificate under "when extra_conf is used with certificate_path and private_key_path" in spec/fluent/plugin/in_prometheus_spec.rb: writes a generated key and self-signed certificate to a tmpdir, requests /metrics over https and checks the peer certificate is the configured one. On master it fails with OpenSSL::PKey::PKeyError: Could not parse PKey.
  • bundle exec rake: 185 examples, 0 failures (Ruby 4.0.7, fluentd 1.20.1).

When <ssl> has extra_conf, in_prometheus serves the metrics with
WEBrick and loads private_key_path with OpenSSL::PKey.read. That call
was given the path instead of the file content since bc7c2ba, so it
always raised "Could not parse PKey" and the plugin failed to start.

Read the file before parsing it, as is done for certificate_path.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix and the clear reproduction.

With the file-path configuration in the PR description, I confirmed that master fails to start with Could not parse PKey, while this branch serves the configured certificate. The new spec fails on master and passes here. Reading the file before OpenSSL::PKey.read also matches how fluentd's cert_option loads the key.

I left one MUST comment. It is about a config that puts the PEM text itself in private_key_path. With this change, that config writes the whole private key to the log, even when the config dump is turned off.

Comment thread lib/fluent/plugin/in_prometheus.rb Outdated

if @ssl['private_key_path']
key = OpenSSL::PKey.read(@ssl['private_key_path'])
key = OpenSSL::PKey.read(File.read(@ssl['private_key_path']))

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: With this change, a config that works on master writes the whole private key to the log at startup, even with --suppress-config-dump or -qq.

On master, private_key_path can hold the PEM text itself instead of a path. OpenSSL::PKey.read parses that string, so the plugin starts. This only worked because of the bug. I am fine with it no longer working, since the parameter is documented as a path.

With this change, File.read treats the PEM text as a file name and raises a file system error, such as Errno::ENOENT. The exception message contains the whole key, and fluentd writes it to its log. On master, the key is not logged when the config dump is turned off (--suppress-config-dump) or the log level is error (-qq). With this change, it is.

Example:

<ssl>
  enable true
  certificate_path /path/to/cert.pem
  private_key_path "-----BEGIN PRIVATE KEY-----\nMIIE...\n-----END PRIVATE KEY-----\n"
  extra_conf {"SSLVerifyClient": 0}
</ssl>

Could you make a failure to read the file raise an error that does not include the value, and add a spec for it? The original exception should not be kept as the cause either, because its message has the value. Any path that this PR can read now should keep working. One way:

Suggested change
key = OpenSSL::PKey.read(File.read(@ssl['private_key_path']))
begin
key_contents = File.read(@ssl['private_key_path'])
rescue SystemCallError => e
raise Fluent::ConfigError, "failed to read private_key_path (#{e.class})", cause: nil
end
key = OpenSSL::PKey.read(key_contents)

This spec, placed inside the context you added, fails on the current branch and passes with the change above:

context 'when private_key_path holds the key itself instead of a path' do
  let(:private_key_path) { %["#{key.to_pem.gsub("\n", '\n')}"] }

  it 'raises an error that does not include the key' do
    expect { driver.run(timeout: 1) }.to raise_error(Fluent::ConfigError, /\Afailed to read private_key_path \(Errno::\w+\)\z/) { |e| expect(e.cause).to be_nil }
  end
end

…read

When private_key_path holds the PEM text itself, File.read raised an
error whose message had the whole key, and fluentd logged it.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@lbarrosby

Copy link
Copy Markdown
Contributor Author

Thanks, good catch. Pushed a015416 with your suggestion: a SystemCallError from reading private_key_path now raises Fluent::ConfigError with failed to read private_key_path (Errno::...) and cause: nil, so the value is not in the message or the cause. A readable path goes through the same OpenSSL::PKey.read as before.

I added your spec inside the new context. On the previous commit it fails with Errno::ENAMETOOLONG and the whole key in the message. It passes now.

bundle exec rake spec: 186 examples, 0 failures.

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for applying the change so quickly. The file-read error in the WEBrick path no longer includes the private key, and its cause is cleared, so the issue in my previous comment is fixed: #276 (comment)

However, my suggestion there caused a new problem, and I am sorry about that. A Fluent::ConfigError raised in start makes fluentd stop entirely instead of restarting the worker. I left a MUST comment with a fix, and a matching change to the spec.

begin
key_contents = File.read(@ssl['private_key_path'])
rescue SystemCallError => e
raise Fluent::ConfigError, "failed to read private_key_path (#{e.class})", cause: nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: My previous suggestion was wrong here. A Fluent::ConfigError raised in start makes the worker exit with status 2. The supervisor treats that as unrecoverable and stops all workers.

This stops a worker that keeps processing logs on master. For example:

<system>
  workers 2
</system>
<worker 0>
  <source>
    @type sample
    tag sample
  </source>
  <match sample>
    @type stdout
  </match>
</worker>
<worker 1>
  <source>
    @type prometheus
    <ssl>
      enable true
      certificate_path /path/to/cert.pem
      private_key_path /path/to/missing.pem
      extra_conf {"SSLVerifyClient": 0}
    </ssl>
  </source>
</worker>

On master, worker 1 keeps restarting, and worker 0 keeps processing events. With this commit, fluentd exits with status 2 after about two seconds, so worker 0 stops too. The same happens when private_key_path holds the PEM text itself.

It also removes recovery when the key file appears after startup. With the previous commit, the worker restarted and served metrics once the file was there. With this commit, fluentd stays stopped.

Could you keep the original SystemCallError subclass, so the worker restarts as before? Please keep cause: nil. Only the message needs to change. Readable paths, including symlinks and FIFOs, should still go through the same File.read call. With the change below, the message for Errno::ENOENT becomes No such file or directory - private_key_path.

Suggested change
raise Fluent::ConfigError, "failed to read private_key_path (#{e.class})", cause: nil
raise e.class, 'private_key_path', cause: nil

let(:private_key_path) { %["#{key.to_pem.gsub("\n", '\n')}"] }

it 'raises an error that does not include the key' do
expect { driver.run(timeout: 1) }.to raise_error(Fluent::ConfigError, /\Afailed to read private_key_path \(Errno::\w+\)\z/) { |e| expect(e.cause).to be_nil }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: This is the spec change for my comment on in_prometheus.rb line 184. The error is no longer a Fluent::ConfigError.

Comparing with a new exception of the same class checks that the message has only the errno text and the parameter name. It works with any errno, such as Errno::ENAMETOOLONG, because the PEM text can be too long for a file name. The cause must stay nil.

Suggested change
expect { driver.run(timeout: 1) }.to raise_error(Fluent::ConfigError, /\Afailed to read private_key_path \(Errno::\w+\)\z/) { |e| expect(e.cause).to be_nil }
expect { driver.run(timeout: 1) }.to raise_error(SystemCallError) { |e|
expect(e.message).to eq(e.class.new('private_key_path').message)
expect(e.cause).to be_nil
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants