Skip to content

Fix /aggregated_metrics failing with content_encoding gzip - #275

Merged
Watson1978 merged 2 commits into
fluent:masterfrom
lbarrosby:fix-aggregated-metrics-gzip
Nov 15, 2026
Merged

Watson1978 merged 2 commits into
fluent:masterfrom
lbarrosby:fix-aggregated-metrics-gzip

Conversation

@lbarrosby

@lbarrosby lbarrosby commented Nov 12, 2026 •

Copy link
Copy Markdown
Contributor

With content_encoding gzip, /aggregated_metrics always answers 500.

all_workers_metrics fetches /metrics from every worker and feeds the bodies to PromMetricsAggregator. The workers compress their answer, and since fluentd depends on async-http, do_request comes from AsyncWrapper, whose client does not decode the body (Net::HTTP does it by itself). The aggregator then splits the gzip bytes into "lines", the first one is not a # comment, and @metrics[''] is nil:

[error]: #0 in_prometheus: failed to render workers metrics error_class=NoMethodError error=#<NoMethodError: undefined method 'add_metric_value' for nil>

This decodes a gzip body in AsyncHttpWrapper#get, so it hands the text to the aggregator like the Net::HTTP path does. The response of /aggregated_metrics is still gzip-encoded by response as configured. This fixes gzip aggregation through AsyncWrapper. The WEBrick path still omits Content-Encoding and needs a separate fix.

Reproduction

<system>
  workers 2
</system>
<source>
  @type prometheus
  port 24251
  content_encoding gzip
</source>
<source>
  @type sample
  tag app
  rate 1
</source>
<match app>
  @type prometheus
  <metric>
    name app_records_total
    type counter
    desc Records seen by each worker.
  </metric>
</match>
curl -s -o agg.bin -w 'HTTP %{http_code}\n' localhost:24251/aggregated_metrics; gunzip -c agg.bin | grep app_records

master:

HTTP 500

this branch:

HTTP 200
# TYPE app_records_total counter
# HELP app_records_total Records seen by each worker.
app_records_total{worker_id="0"} 9.0
app_records_total{worker_id="1"} 9.0

With content_encoding identity both give 200.

Tests

  • New #run /aggregated_metrics examples in spec/fluent/plugin/in_prometheus_spec.rb, for identity and gzip: request /aggregated_metrics, check the status, the Content-Encoding and that the decoded body has the registered metric. The gzip one fails on master (expected "200", got "500").
  • A gzip example that sets Encoding.default_external to US-ASCII and aggregates a non-ASCII label value. It fails with an earlier version of this change that used Zlib::GzipReader#read (500).
  • bundle exec rake: 187 examples, 0 failures (Ruby 4.0.7, fluentd 1.20.1, async-http installed).

/aggregated_metrics fetches /metrics of every worker and merges the
text. With content_encoding gzip the workers answer with a gzip body,
and the async-http client (used whenever the async gem is available,
which fluentd depends on) does not decode it, unlike Net::HTTP. The
aggregator then parsed the compressed bytes, raised NoMethodError and
the endpoint returned 500.

Decode a gzip body in AsyncHttpWrapper#get, so it returns the text as
the Net::HTTP path does.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix. I reproduced the 500 on master with 2 workers and content_encoding gzip, and this branch returns 200 with a gzip body. The new gzip spec fails on master and passes here.

One request inline: decode with Zlib.gunzip, so the decoded body does not depend on the process locale.

Not for this PR: with the WEBrick server (used with an IPv6 bind or with <ssl> and extra_conf), /aggregated_metrics still returns 500 with gzip. The WEBrick handlers in start_webrick copy only Content-Type, so the worker responses have no Content-Encoding header and Net::HTTP does not decode them. /metrics on that path also sends a gzip body without the header. This is a pre-existing issue in the WEBrick handlers. No WEBrick changes are requested in this PR. It can be handled separately.

GzipReader#read returns a string in Encoding.default_external, so a
non-ASCII label value made /aggregated_metrics fail when that is
US-ASCII. Zlib.gunzip returns a binary string, like the identity path.

Signed-off-by: Lucas Barros <lucasbarros.kam@gmail.com>
@lbarrosby

Copy link
Copy Markdown
Contributor Author

Thanks. Pushed ac3643d with your suggestion: the gzip body is decoded with Zlib.gunzip(body).

I also added a spec under content_encoding gzip that increments a counter with a 日本語 label value, sets Encoding.default_external to US-ASCII and requests /aggregated_metrics. With GzipReader#read it gets 500. With Zlib.gunzip it gets 200 and the label in the body.

bundle exec rake spec: 187 examples, 0 failures.

I will leave the WEBrick Content-Encoding issue for a separate PR.

@Watson1978 Watson1978 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update. Zlib.gunzip fixes the locale issue. I checked ac3643d in two setups: LC_ALL=C with --no-supervisor, and RUBYOPT=-EUS-ASCII:US-ASCII with 2 workers. In both, /aggregated_metrics returns 200 with the non-ASCII label value. The new spec fails with the previous GzipReader code and passes here. The full suite passes (187 examples, 0 failures).

LGTM.

@Watson1978
Watson1978 merged commit 2077c98 into fluent:master Nov 15, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants