Repository navigation
Parse histogram buckets such as .005 and 1e-3 as numbers - #274
Conversation
The buckets parameter turned any value which did not look like "digits.digits" into an integer with String#to_i. ".005, .01, .025" became [0, 0, 0], so every observation fell into the +Inf bucket, and "1e-3" became 1. A value which is not a number became 0 instead of being refused. Parse a bucket with Integer or Float, and raise ConfigError for a value which is not a number. Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thank you for the fix. .005 and 1e-3 now parse as 0.005 and 0.001, and a value that is not a number, such as abc, raises a ConfigError.
One problem remains. This change also changes the le label of some integer buckets that work on master. Please see the inline comments.
| # keep an integer as is, so that its `le` label stays "1", not "1.0" | ||
| e.match?(/\A\d+\z/) ? Integer(e, 10) : Float(e) |
There was a problem hiding this comment.
MUST: This changes the le label of integer buckets that work on master.
The old regexp did not escape ., so it also matched integers of three or more digits, and they became floats. Signed integers did not match it, so they stayed integers.
buckets |
le on master |
le with this PR |
|---|---|---|
10, 100, 1000 |
"10", "100.0", "1000.0" |
"10", "100", "1000" |
-5, 0, +5 |
"-5", "0", "5" |
"-5.0", "0", "5.0" |
Prometheus 3 normalizes le on ingestion, so the old and new labels become the same there. But Prometheus 2 keeps the labels of the text format as is, so it gets new series. Then queries, recording rules and alerts that use le="100.0" stop matching.
Please keep the master values and labels in the table above, and keep the fixes for .005, 1e-3 and abc. One way is to parse each value with Float(e), and use the old result when it is equal to that value. Keep the old regexp as is, because escaping its . changes the labels again. I checked this suggestion with the existing specs, the cases in this PR and the cases above:
| # keep an integer as is, so that its `le` label stays "1", not "1.0" | |
| e.match?(/\A\d+\z/) ? Integer(e, 10) : Float(e) | |
| # keep the old result if it is right, so that existing `le` labels such as "100.0" stay the same | |
| value = Float(e) | |
| legacy = e[/\A\d+.\d+\Z/] ? e.to_f : e.to_i | |
| legacy == value ? legacy : value |
There was a problem hiding this comment.
Thanks, applied in be8b6c5. 10, 100, 1000 and -5, 0, +5 keep the le labels from master, and .005, 1e-3 and abc keep the fix.
| it 'keeps integer and decimal buckets' do | ||
| expect(buckets_of('0.1, 1, 5, 10')).to eq([0.1, 1, 5, 10]) | ||
| end |
There was a problem hiding this comment.
SHOULD: eq cannot catch the le change in lib/fluent/plugin/prometheus.rb, because [1.0] == [1] is true in Ruby. Please use eql and add the cases from that comment. With this PR as is, the two new examples fail:
| it 'keeps integer and decimal buckets' do | |
| expect(buckets_of('0.1, 1, 5, 10')).to eq([0.1, 1, 5, 10]) | |
| end | |
| it 'keeps integer and decimal buckets' do | |
| expect(buckets_of('0.1, 1, 5, 10')).to eql([0.1, 1, 5, 10]) | |
| end | |
| it 'keeps buckets of three or more digits as floats, as before' do | |
| expect(buckets_of('10, 100, 1000')).to eql([10, 100.0, 1000.0]) | |
| end | |
| it 'keeps signed integer buckets as integers' do | |
| expect(buckets_of('-5, 0, +5')).to eql([-5, 0, 5]) | |
| end |
There was a problem hiding this comment.
Done in be8b6c5. The examples use eql, and the two new ones fail on the previous commit and pass now. bundle exec rspec passes (184 examples).
The previous change parsed every all-digit bucket as an Integer. On master, buckets of three or more digits such as 100 matched the unescaped "digits.digits" regexp and became floats, and signed integers such as -5 stayed integers. Their le labels changed from "100.0" to "100" and from "-5" to "-5.0", which creates new series on Prometheus 2. Keep the old result when it equals the parsed Float value, and use eql in the specs so that a change between 1 and 1.0 fails them. Signed-off-by: Raphael Fakhri <153192858+RaphaelFakhri@users.noreply.github.com>
Watson1978
left a comment
There was a problem hiding this comment.
Thank you for the update. I checked be8b6c5. 10, 100, 1000 and -5, 0, +5 keep the bucket values and le labels from master. .005 and 1e-3 still parse correctly, and abc still raises a ConfigError. The specs now use eql, so they catch a change between Integer and Float. I also updated the PR description to match be8b6c5. LGTM.
The
bucketsparameter of a histogram<metric>converted any value that does not matchdigits.digitswithString#to_i:buckets .005, .01, .025became[0, 0, 0], so every observation fell into the+Infbucket without an error.buckets 1e-3, 1e-2, 0.1became[1, 1, 0.1], and startup failed withUnsorted buckets, typo?.abc, became0.This change parses each bucket with
Float, and raisesConfigErrorfor a value that is not a number. When the old conversion gives the same number, the old result is kept. This keeps the existinglelabels. For example,1stays"1",100stays"100.0"and-5stays"-5". The old regexp did not escape., so integers of three or more digits were floats on master.Tests
spec/fluent/plugin/prometheus/histogram_spec.rbcovers integer and decimal buckets, buckets that keep the same values and types as master (10, 100, 1000and-5, 0, +5), decimals without a leading zero, exponent notation and an invalid value. The examples useeql, so they catch a change between Integer and Float. Without the fix, 3 of its 6 examples fail. With the fix, the full suite passes (184 examples, 0 failures).Fixes #273