diff --git a/CHANGELOG.md b/CHANGELOG.md index 6cd9df1..08a0e5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,8 @@ ## Changes +* Add `SSLConfig#cacert_file=` to select a CA file, including `OpenSSL::X509::DEFAULT_CERT_FILE` for opting in to the future MRI default. The MRI default remains the bundled CA certificates until HTTPClient 3.0.0; JRuby continues to use runtime defaults ([#480](https://github.com/nahi/httpclient/issues/480)). +* Warn once per process when the bundled CA certificates (`SSLConfig::BUNDLED_CACERT_FILE`) are loaded. Starting with HTTPClient 3.0.0, MRI will use `OpenSSL::X509::DEFAULT_CERT_FILE` by default. + ## Change Log ### Changes in 2.10.0 diff --git a/README.md b/README.md index 80e695b..6bb0cbc 100644 --- a/README.md +++ b/README.md @@ -72,6 +72,43 @@ You can install httpclient via rubygems: `gem install httpclient` See [HTTPClient](http://www.rubydoc.info/gems/httpclient/frames) for documentation. You can also check sample/howto.rb how to use APIs. +### TLS trust store + + + +Before HTTPClient 3.0.0, MRI uses the bundled Mozilla CA certificates +(`HTTPClient::SSLConfig::BUNDLED_CACERT_FILE`) by default. Loading them, +implicitly or explicitly, emits a deprecation warning once per process. Merely +creating a client does not emit the warning. + +**Starting with HTTPClient 3.0.0, MRI will use +`OpenSSL::X509::DEFAULT_CERT_FILE` by default.** To opt in and test your +application before upgrading: + +```ruby +client = HTTPClient.new +client.ssl_config.cacert_file = OpenSSL::X509::DEFAULT_CERT_FILE +``` + +You can also select a custom CA file with `cacert_file=`. Assigning it loads +that file into a new certificate store, replacing the current store and +resetting existing sessions. You can assign it after a connection has been +made. If loading fails, the current store and selected file are unchanged. +`load_trust_ca` adds the selected file to the current store and resets existing +sessions. Use `client.ssl_config.set_default_paths` on MRI to load OpenSSL's +configured default certificate file and directory instead of a single file. +The trusted CA certificates can differ between the bundled file and your +OpenSSL installation; test connections to your application's HTTPS endpoints +before switching. + +On JRuby, CA selection continues to default to the JVM trust store and can be +overridden with `SSL_CERT_FILE`; the MRI default change does not apply. +Assigning `cacert_file` explicitly selects a certificate file instead of the +runtime defaults. Calling `load_trust_ca` adds the selected file (the bundled +CA certificates unless changed explicitly) to the current store, preserving +the runtime's default trust anchors unless the store was replaced or cleared. +Using the JVM defaults does not emit the bundled CA warning. + ### TLS certificate restrictions Blocked public keys are identified by their SPKI SHA-256 hashes in @@ -91,11 +128,10 @@ context. With peer verification enabled, its return value controls certificate verification. The caller is responsible for the custom callback's validation policy. Hostname verification is unchanged. -Note: MRI uses HTTPClient's bundled `cacert.pem` by default, while JRuby uses -the JVM's CA certificates and certificate validation. Secure operation on JRuby -depends on keeping the JVM and its CA certificates up to date. HTTPClient's -additional check may not detect blocked intermediates found only in a local -trust store or obtained through JVM issuer retrieval (such as AIA when enabled). +On JRuby, HTTPClient uses the JVM's CA certificates and validation by default. +Keep the JVM and its CA certificates up to date. The additional blocked-key +check may not detect intermediates found only in a local trust store or obtained +through JVM issuer retrieval (such as AIA when enabled). Remove unused certificates containing blocked public keys from the server's certificate chain configuration. If a certificate containing a blocked public key diff --git a/lib/httpclient/ssl_config.rb b/lib/httpclient/ssl_config.rb index 2994442..affec9d 100644 --- a/lib/httpclient/ssl_config.rb +++ b/lib/httpclient/ssl_config.rb @@ -20,23 +20,23 @@ class HTTPClient # # == Trust Anchor Control # - # SSLConfig loads 'httpclient/cacert.pem' as a trust anchor - # (trusted certificate(s)) with add_trust_ca in initialization time. - # This means that HTTPClient instance trusts some CA certificates by default, - # like Web browsers. 'httpclient/cacert.pem' is downloaded from curl web - # site by the author and included in released package. + # Before HTTPClient 3.0.0, MRI uses the bundled Mozilla CA certificates + # (BUNDLED_CACERT_FILE) by default. Loading them emits a deprecation warning + # once per process. Starting with HTTPClient 3.0.0, MRI will use + # OpenSSL::X509::DEFAULT_CERT_FILE by default. To opt in now: # - # On JRuby, HTTPClient uses Java runtime's trusted CA certificates, not - # cacert.pem by default. You can load cacert.pem by calling - # SSLConfig#load_trust_ca manually like: + # client = HTTPClient.new + # client.ssl_config.cacert_file = OpenSSL::X509::DEFAULT_CERT_FILE # - # HTTPClient.new { self.ssl_config.load_trust_ca }.get("https://...") + # Assigning cacert_file loads the selected file into a new certificate store, + # replacing the current store and resetting existing sessions. It can be + # changed after connections have been made. load_trust_ca loads the selected + # file into the current store, adding to its existing certificates. # - # Secure operation on JRuby depends on keeping the JVM and its CA certificates - # up to date. - # - # You may want to change trust anchor by yourself. Call clear_cert_store - # then add_trust_ca for that purpose. + # On JRuby, CA selection defaults to the JVM's trust store and can be + # overridden with SSL_CERT_FILE. Assign cacert_file to select a file directly, + # or call load_trust_ca to add the selected file (the bundled CA certificates + # unless changed explicitly) to the runtime's default trust anchors. # # == TLS certificate restrictions # @@ -116,6 +116,8 @@ def attr_config(symbol) CIPHERS_DEFAULT = "ALL:!aNULL:!eNULL:!SSLv2" # OpenSSL >1.0.0 default + # Bundled Mozilla CA certificates, used by default on MRI before HTTPClient 3.0.0. + BUNDLED_CACERT_FILE = File.join(File.dirname(__FILE__), 'cacert.pem').freeze # Public keys whose SPKI SHA-256 hashes are listed below are considered blocked. # The list covers eight eMudhra intermediate CA public keys whose corresponding @@ -197,9 +199,12 @@ def self.blocked_public_key?(public_key_der) # See ciphers(1) man in OpenSSL for more detail. attr_config :ciphers - # OpenSSL::X509::X509::Store used for verification. You can reset the - # store with clear_cert_store and set the new store with cert_store=. + # OpenSSL::X509::Store used for verification. You can reset the store with + # clear_cert_store and set the new store with cert_store=. attr_reader :cert_store # don't use if you don't know what it is. + # Selected CA certificate file. Defaults to BUNDLED_CACERT_FILE. On JRuby, + # the selected file is only loaded explicitly, leaving runtime defaults intact. + attr_reader :cacert_file # For server side configuration. Ignore this. attr_config :client_ca # :nodoc: @@ -214,6 +219,9 @@ def cert_store_items; @cert_store._httpclient_cert_store_items; end def initialize(client) return unless SSLEnabled @client = client + # TODO(3.0.0): On MRI, the default becomes OpenSSL::X509::DEFAULT_CERT_FILE. + # When it changes, also update the notice below and every TODO(3.0.0) in the tests and README. + @cacert_file = BUNDLED_CACERT_FILE @cert_store = X509::Store.new @cert_store_crl_items = [] @client_cert = @client_key = @client_key_pass = @client_ca = nil @@ -309,16 +317,35 @@ def add_trust_ca(trust_ca_file_or_hashed_dir) end alias set_trust_ca add_trust_ca + # Selects a CA certificate file. Replaces the current certificate store + # with a new store loaded from file and resets existing sessions. If loading + # fails, the current store and selected file are unchanged. + def cacert_file=(file) + store = X509::Store.new + store._httpclient_cert_store_items.clear if defined?(JRUBY_VERSION) + add_trust_ca_to_store(store, file) + @cacert_file = file + self.cert_store = store + end + def add_trust_ca_to_store(cert_store, trust_ca_file_or_hashed_dir) - if FileTest.directory?(trust_ca_file_or_hashed_dir) + result = if FileTest.directory?(trust_ca_file_or_hashed_dir) cert_store.add_path(trust_ca_file_or_hashed_dir) else cert_store.add_file(trust_ca_file_or_hashed_dir) end + if File.expand_path(trust_ca_file_or_hashed_dir) == File.expand_path(BUNDLED_CACERT_FILE) + # TODO(3.0.0): Remove or rewrite this notice with the default change, and update its text. + warning('HTTPClient: the bundled CA certificates are deprecated and will be replaced ' \ + 'with OpenSSL::X509::DEFAULT_CERT_FILE as the default CA file on MRI in HTTPClient 3.0.0. ' \ + 'To switch now, use client.ssl_config.cacert_file = OpenSSL::X509::DEFAULT_CERT_FILE.') + end + result end - # Loads default trust anchors. - # Calling this method resets all existing sessions. + # Loads the selected cacert_file into the current certificate store. + # Existing certificates in the store are kept. Calling this method resets + # all existing sessions. def load_trust_ca load_cacerts(@cert_store) change_notify @@ -521,10 +548,8 @@ def change_notify nil end - # Use 2048 bit certs trust anchor def load_cacerts(cert_store) - file = File.join(File.dirname(__FILE__), 'cacert.pem') - add_trust_ca_to_store(cert_store, file) + add_trust_ca_to_store(cert_store, @cacert_file) end end diff --git a/test/test_ssl.rb b/test/test_ssl.rb index c837274..825e445 100644 --- a/test/test_ssl.rb +++ b/test/test_ssl.rb @@ -31,6 +31,17 @@ def read_path(filename) File.read(path(filename)) end + def test_cacert_file_replaces_store_after_connection + @client.ssl_config.cacert_file = path('ca-chain.pem') + assert_equal('hello', @client.get_content(@url)) + + @client.ssl_config.cacert_file = path('fixtures/verify.foo.cert') + assert_raise(OpenSSL::SSL::SSLError) { @client.get(@url) } + + @client.ssl_config.cacert_file = path('ca-chain.pem') + assert_equal('hello', @client.get_content(@url)) + end + def test_proxy_ssl setup_proxyserver escape_noproxy do @@ -106,6 +117,8 @@ def test_blocked_peer_certificate_before_request def test_blocked_intermediate_with_default_paths escape_env do + # Nix's OpenSSL can prioritize this CA bundle path over SSL_CERT_FILE. + ENV.delete('NIX_SSL_CERT_FILE') ENV['SSL_CERT_FILE'] = path('ca-chain.pem') @client.ssl_config.set_default_paths @server.ssl_context.extra_chain_cert = [cert('subca.cert')] @@ -373,6 +386,8 @@ def test_set_default_paths @client.get(@url) end escape_env do + # Nix's OpenSSL can prioritize this CA bundle path over SSL_CERT_FILE. + ENV.delete('NIX_SSL_CERT_FILE') ENV['SSL_CERT_FILE'] = File.join(DIR, 'ca-chain.pem') @client.ssl_config.set_default_paths @client.get(@url) diff --git a/test/test_ssl_config.rb b/test/test_ssl_config.rb index 67b82d2..c1d96fc 100644 --- a/test/test_ssl_config.rb +++ b/test/test_ssl_config.rb @@ -1,5 +1,7 @@ require 'test/unit' require 'httpclient' +require 'open3' +require 'rbconfig' class TestSSLConfig < Test::Unit::TestCase # Older ruby-openssl returns generic PKey objects without DER export methods. @@ -49,3 +51,207 @@ def test_blocked_cross_signed_keys_without_der_export end end end + +class TestCACertFile < Test::Unit::TestCase + def setup + @client = HTTPClient.new + @config = @client.ssl_config + @ca_file = File.join(__dir__, 'ca.cert') + end + + def teardown + @client.reset_all + end + + # TODO(3.0.0): These tests describe the 2.x MRI default. Rewrite them with the default change. + def test_bundled_ca_is_selected_by_default + assert_equal(HTTPClient::SSLConfig::BUNDLED_CACERT_FILE, @config.cacert_file) + end + + if defined?(JRUBY_VERSION) + def test_jruby_keeps_runtime_default_trust_store + default_items = [ENV['SSL_CERT_FILE'] || :default] + HTTPClient::JRubySSLSocket.allocate.create_ssl_context(@config) + + assert_equal(default_items, @config.cert_store_items) + end + + def test_load_trust_ca_adds_bundled_ca_to_jruby_defaults + default_items = @config.cert_store_items.dup + @config.load_trust_ca + + assert_equal(default_items + [HTTPClient::SSLConfig::BUNDLED_CACERT_FILE], + @config.cert_store_items) + end + end + + def test_cacert_file_replaces_store + @config.add_trust_ca(File.join(__dir__, 'ca-chain.pem')) + original_store = @config.cert_store + @config.cacert_file = @ca_file + + assert_not_same(original_store, @config.cert_store) + assert_equal(@ca_file, @config.cacert_file) + assert_equal([@ca_file], @config.cert_store_items) if defined?(JRUBY_VERSION) + end + + def test_cacert_file_failure_preserves_current_configuration + @config.cacert_file = @ca_file + original_store = @config.cert_store + + assert_raise(OpenSSL::X509::StoreError) do + @config.cacert_file = File.join(__dir__, 'missing-ca-file.pem') + end + + assert_same(original_store, @config.cert_store) + assert_equal(@ca_file, @config.cacert_file) + end + + def test_load_trust_ca_uses_selected_file + @config.cacert_file = @ca_file + @config.clear_cert_store + @config.load_trust_ca + + certificate = OpenSSL::X509::Certificate.new(File.read(@ca_file)) + @config.cert_store.time = certificate.not_before + 1 + assert(@config.cert_store.verify(certificate), @config.cert_store.error_string) + end +end + +# JRuby uses its JVM trust manager instead of SSLConfig#set_context. +unless defined?(JRUBY_VERSION) + class TestOpenSSLTrustStore < Test::Unit::TestCase + def setup + @client = HTTPClient.new + @config = @client.ssl_config + @loaded_files = [] + loaded_files = @loaded_files + @config.define_singleton_method(:add_trust_ca_to_store) do |_store, file| + loaded_files << file + end + end + + def teardown + @client.reset_all if @client + end + + # TODO(3.0.0): These tests describe the 2.x MRI default. Rewrite them with the default change. + def test_bundled_ca_is_loaded_once_by_default + configure_context + configure_context + assert_equal([bundled_ca_file], @loaded_files) + end + + def test_openssl_default_ca_can_be_selected_with_cacert_file + @config.cacert_file = OpenSSL::X509::DEFAULT_CERT_FILE + configure_context + configure_context + assert_equal([OpenSSL::X509::DEFAULT_CERT_FILE], @loaded_files) + end + + private + + def configure_context + context = OpenSSL::SSL::SSLContext.new + @config.set_context(context) + assert_same(@config.cert_store, context.cert_store) + end + + def bundled_ca_file + HTTPClient::SSLConfig::BUNDLED_CACERT_FILE + end + end +end + +# Deprecation notice for the bundled CA certificates. +# TODO(3.0.0): These tests describe the 2.x MRI default. Rewrite them with the default change. +class TestBundledCACertNotice < Test::Unit::TestCase + LIB_DIR = File.expand_path('../lib', __dir__) + NOTICE_PATTERN = /bundled CA certificates are deprecated/ + + def setup + @client = HTTPClient.new + @config = @client.ssl_config + @notices = [] + notices = @notices + # Records messages before Util#warning's once-per-process filter applies. + # The filter itself is checked in a subprocess below. + @config.define_singleton_method(:warning) { |message| notices << message } + end + + def teardown + @client.reset_all + end + + def test_selecting_bundled_ca_file_emits_notice + @config.cacert_file = HTTPClient::SSLConfig::BUNDLED_CACERT_FILE + assert_equal(1, @notices.size) + end + + def test_loading_default_trust_anchors_emits_notice + @config.load_trust_ca + assert_equal(1, @notices.size) + end + + def test_notice_names_the_3_0_0_default_change_and_opt_in + @config.load_trust_ca + message = @notices.first + assert_match(/deprecated/, message) + assert_match(/HTTPClient 3\.0\.0/, message) + assert_match(/OpenSSL::X509::DEFAULT_CERT_FILE/, message) + assert_match(/cacert_file = OpenSSL::X509::DEFAULT_CERT_FILE/, message) + end + + def test_custom_ca_file_does_not_emit_notice + @config.cacert_file = File.join(__dir__, 'ca.cert') + @config.load_trust_ca + assert_empty(@notices) + end + + if defined?(JRUBY_VERSION) + # JRuby uses the JVM trust store unless the bundled file is loaded explicitly. + def test_jruby_default_trust_store_does_not_emit_notice + HTTPClient::JRubySSLSocket.allocate.create_ssl_context(@config) + assert_empty(@notices) + end + else + def test_default_ssl_context_emits_notice + @config.set_context(OpenSSL::SSL::SSLContext.new) + assert_equal(1, @notices.size) + end + + # On JRuby, DEFAULT_CERT_FILE is a JDK keystore that OpenSSL cannot load as PEM. + def test_openssl_default_ca_file_does_not_emit_notice + omit('OpenSSL default CA file is missing') unless File.exist?(OpenSSL::X509::DEFAULT_CERT_FILE) + @config.cacert_file = OpenSSL::X509::DEFAULT_CERT_FILE + @config.load_trust_ca + assert_empty(@notices) + end + end + + # Subprocess checks: process-wide once-only output and no output on creation. + # JRuby may print its own startup warnings to stderr, so only notice lines count. + + def test_creating_client_writes_no_notice + assert_empty(notice_lines_from('HTTPClient.new.ssl_config')) + end + + def test_repeated_loads_write_notice_once + lines = notice_lines_from(<<~RUBY) + config = HTTPClient.new.ssl_config + config.load_trust_ca + config.cacert_file = HTTPClient::SSLConfig::BUNDLED_CACERT_FILE + HTTPClient.new.ssl_config.load_trust_ca + RUBY + assert_equal(1, lines.size) + end + + private + + def notice_lines_from(script) + _stdout, stderr, status = Open3.capture3(RbConfig.ruby, '-I', LIB_DIR, + '-e', "require 'httpclient'\n#{script}") + assert(status.success?, stderr) + stderr.lines.grep(NOTICE_PATTERN) + end +end