Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion lib/fluent/plugin/in_prometheus.rb
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,12 @@ def start_webrick
end

if @ssl['private_key_path']
key = OpenSSL::PKey.read(@ssl['private_key_path'])
begin
key_contents = File.read(@ssl['private_key_path'])
rescue SystemCallError => e
raise Fluent::ConfigError, "failed to read private_key_path (#{e.class})", cause: nil

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: My previous suggestion was wrong here. A Fluent::ConfigError raised in start makes the worker exit with status 2. The supervisor treats that as unrecoverable and stops all workers.

This stops a worker that keeps processing logs on master. For example:

<system>
  workers 2
</system>
<worker 0>
  <source>
    @type sample
    tag sample
  </source>
  <match sample>
    @type stdout
  </match>
</worker>
<worker 1>
  <source>
    @type prometheus
    <ssl>
      enable true
      certificate_path /path/to/cert.pem
      private_key_path /path/to/missing.pem
      extra_conf {"SSLVerifyClient": 0}
    </ssl>
  </source>
</worker>

On master, worker 1 keeps restarting, and worker 0 keeps processing events. With this commit, fluentd exits with status 2 after about two seconds, so worker 0 stops too. The same happens when private_key_path holds the PEM text itself.

It also removes recovery when the key file appears after startup. With the previous commit, the worker restarted and served metrics once the file was there. With this commit, fluentd stays stopped.

Could you keep the original SystemCallError subclass, so the worker restarts as before? Please keep cause: nil. Only the message needs to change. Readable paths, including symlinks and FIFOs, should still go through the same File.read call. With the change below, the message for Errno::ENOENT becomes No such file or directory - private_key_path.

Suggested change
raise Fluent::ConfigError, "failed to read private_key_path (#{e.class})", cause: nil
raise e.class, 'private_key_path', cause: nil

end
key = OpenSSL::PKey.read(key_contents)
ssl_config[:SSLPrivateKey] = key
end

Expand Down
52 changes: 52 additions & 0 deletions spec/fluent/plugin/in_prometheus_spec.rb
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,8 @@

require 'net/http'
require 'zlib'
require 'tmpdir'
require 'fileutils'

describe Fluent::Plugin::PrometheusInput do
CONFIG = %[
Expand Down Expand Up @@ -157,6 +159,56 @@
end
end

context 'when extra_conf is used with certificate_path and private_key_path' do
let(:tmpdir) { Dir.mktmpdir }
let(:key) { OpenSSL::PKey::RSA.new(2048) }
let(:cert) do
cert = OpenSSL::X509::Certificate.new
cert.version = 2
cert.serial = 1
cert.subject = cert.issuer = OpenSSL::X509::Name.parse('/CN=localhost')
cert.public_key = key.public_key
cert.not_before = Time.now - 60
cert.not_after = Time.now + 3600
cert.sign(key, OpenSSL::Digest::SHA256.new)
cert
end
let(:cert_path) { File.join(tmpdir, 'cert.pem').tap { |path| File.write(path, cert.to_pem) } }
let(:private_key_path) { File.join(tmpdir, 'key.pem').tap { |path| File.write(path, key.to_pem) } }
let(:config) do
%[
@type prometheus
bind 127.0.0.1
<ssl>
enable true
certificate_path #{cert_path}
private_key_path #{private_key_path}
extra_conf { "SSLVerifyClient": 0 }
</ssl>
]
end

after { FileUtils.rm_rf(tmpdir) }

it 'serves the given certificate' do
driver.run(timeout: 1) do
Net::HTTP.start('127.0.0.1', port, verify_mode: OpenSSL::SSL::VERIFY_NONE, use_ssl: true) do |http|
res = http.request(Net::HTTP::Get.new('/metrics'))
expect(res.code).to eq('200')
expect(http.peer_cert.to_der).to eq(cert.to_der)
end
end
end

context 'when private_key_path holds the key itself instead of a path' do
let(:private_key_path) { %["#{key.to_pem.gsub("\n", '\n')}"] }

it 'raises an error that does not include the key' do
expect { driver.run(timeout: 1) }.to raise_error(Fluent::ConfigError, /\Afailed to read private_key_path \(Errno::\w+\)\z/) { |e| expect(e.cause).to be_nil }

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MUST: This is the spec change for my comment on in_prometheus.rb line 184. The error is no longer a Fluent::ConfigError.

Comparing with a new exception of the same class checks that the message has only the errno text and the parameter name. It works with any errno, such as Errno::ENAMETOOLONG, because the PEM text can be too long for a file name. The cause must stay nil.

Suggested change
expect { driver.run(timeout: 1) }.to raise_error(Fluent::ConfigError, /\Afailed to read private_key_path \(Errno::\w+\)\z/) { |e| expect(e.cause).to be_nil }
expect { driver.run(timeout: 1) }.to raise_error(SystemCallError) { |e|
expect(e.message).to eq(e.class.new('private_key_path').message)
expect(e.cause).to be_nil
}

end
end
end

context 'cert_path and private_key_path combination' do
let(:config) do
%[
Expand Down
Loading