Support partial write on PRW2.0

[SungJin1212]

Implements partial write for Prometheus Remote Write 2.0 requests, following the Prometheus receiver behavior.
Previously a single invalid series aborted the whole conversion, so the entire batch was rejected with a 400 and the X-Prometheus-Remote-Write-*-Written response headers were not set at all; now invalid series are skipped and reported together in the response, the valid ones are written, and the written stats headers are always set.

This also makes Cortex accept a v2 write request that contains an exemplar-only series.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]
• docs/configuration/v1-guarantees.md updated if this PR introduces experimental flags

[yeya24]

Previously a single invalid series aborted the whole conversion, so the entire batch was rejected with a 400 and the X-Prometheus-Remote-Write-*-Written response headers were not set at all; now invalid series are skipped and reported together in the response, the valid ones are written, and the written stats headers are always set.

Interesting, is this what Prometheus does today? If that's Prometheus' behavior I am ok with it. But it is a behavior change in Cortex regardless. Do you think we can put this behind a FF?

[SungJin1212]

Interesting, is this what Prometheus does today?

Yes, Prometheus' v2 receiver has addressed the partial write since v2.54.0.

Do you think we can put this behind a FF?

I'd rather not add another flag here: the v2 path is already opt-in and experimental. Currently, Prometheus can send exemplar-only series (prometheus/prometheus#17857), and Cortex drops the whole request, so partial write is what makes exemplars usable over RW2.0 at all.
Or how about a flag that defaults to true?

[friedrichg]

Thanks for this. The partial write looks right to me and matches the spec, which requires a non-2xx on partial writes and requires the X-Prometheus-Remote-Write-*-Written headers on failures too, so not setting them on a 400 was a spec violation before.

Could we split the exemplar-only acceptance into a follow-up PR? It is independent of partial write, and it carries an open question that partial write does not:

• It is deliberately more permissive than the Prometheus receiver, which still rejects exemplar-only TimeSeries.
• The upstream direction is unresolved. prometheus/prometheus#17857 is open with two competing approaches.
• It only works when the series already exists in the ingester. app.AppendExemplar does not create the series, so for a new series the ingester soft fails with errExemplarRef, returns 400, and drops the exemplar. Making it work properly needs an ingester change, which is its own discussion about series creation and limits.

[friedrichg]

What happens if UnitRef is out of range and enableTypeAndUnitLabels is on? unit is "" here, so shouldAttachTypeAndUnitLabels below decides from the type alone and we store the series without the unit label the sender sent, under a different identity.

[SungJin1212]

Should we drop the series when UnitRef is out of range and enableTypeAndUnitLabels is on?

[SungJin1212]

It only works when the series already exists in the ingester. app.AppendExemplar does not create the series, so for a new series the ingester soft fails with errExemplarRef, returns 400, and drops the exemplar. Making it work properly needs an ingester change, which is its own discussion about series creation and limits.

I never thought about it. I changed to drop the exemplar-only series and updated it to the latest commit.

[friedrichg]

Yes, exactly this. With the flag on, unit feeds unit, which is part of the series identity, so keeping the series stores it under a label set the sender never sent. Dropping it and reporting the error is what current Prometheus does for any bad metadata ref.