Repository navigation
Fix /aggregated_metrics failing with content_encoding gzip - #275
Conversation
/aggregated_metrics fetches /metrics of every worker and merges the text. With content_encoding gzip the workers answer with a gzip body, and the async-http client (used whenever the async gem is available, which fluentd depends on) does not decode it, unlike Net::HTTP. The aggregator then parsed the compressed bytes, raised NoMethodError and the endpoint returned 500. Decode a gzip body in AsyncHttpWrapper#get, so it returns the text as the Net::HTTP path does. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the fix. I reproduced the 500 on master with 2 workers and content_encoding gzip, and this branch returns 200 with a gzip body. The new gzip spec fails on master and passes here.
One request inline: decode with Zlib.gunzip, so the decoded body does not depend on the process locale.
Not for this PR: with the WEBrick server (used with an IPv6 bind or with <ssl> and extra_conf), /aggregated_metrics still returns 500 with gzip. The WEBrick handlers in start_webrick copy only Content-Type, so the worker responses have no Content-Encoding header and Net::HTTP does not decode them. /metrics on that path also sends a gzip body without the header. This is a pre-existing issue in the WEBrick handlers. No WEBrick changes are requested in this PR. It can be handled separately.
GzipReader#read returns a string in Encoding.default_external, so a non-ASCII label value made /aggregated_metrics fail when that is US-ASCII. Zlib.gunzip returns a binary string, like the identity path. Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
|
Thanks. Pushed ac3643d with your suggestion: the gzip body is decoded with I also added a spec under
I will leave the WEBrick |
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the update. Zlib.gunzip fixes the locale issue. I checked ac3643d in two setups: LC_ALL=C with --no-supervisor, and RUBYOPT=-EUS-ASCII:US-ASCII with 2 workers. In both, /aggregated_metrics returns 200 with the non-ASCII label value. The new spec fails with the previous GzipReader code and passes here. The full suite passes (187 examples, 0 failures).
LGTM.
With
content_encoding gzip,/aggregated_metricsalways answers 500.all_workers_metricsfetches/metricsfrom every worker and feeds the bodies toPromMetricsAggregator. The workers compress their answer, and sincefluentddepends onasync-http,do_requestcomes fromAsyncWrapper, whose client does not decode the body (Net::HTTP does it by itself). The aggregator then splits the gzip bytes into "lines", the first one is not a#comment, and@metrics['']is nil:This decodes a gzip body in
AsyncHttpWrapper#get, so it hands the text to the aggregator like the Net::HTTP path does. The response of/aggregated_metricsis still gzip-encoded byresponseas configured. This fixes gzip aggregation throughAsyncWrapper. The WEBrick path still omitsContent-Encodingand needs a separate fix.Reproduction
master:
this branch:
With
content_encoding identityboth give 200.Tests
#run /aggregated_metricsexamples inspec/fluent/plugin/in_prometheus_spec.rb, foridentityandgzip: request/aggregated_metrics, check the status, theContent-Encodingand that the decoded body has the registered metric. Thegzipone fails on master (expected "200", got "500").gzipexample that setsEncoding.default_externalto US-ASCII and aggregates a non-ASCII label value. It fails with an earlier version of this change that usedZlib::GzipReader#read(500).bundle exec rake: 187 examples, 0 failures (Ruby 4.0.7, fluentd 1.20.1, async-http installed).