Support for delta temporality in the OTLP ingest

[irisgve]

What is the problem you are trying to solve?

It looks like Prometheus now has support for delta ingestion in OTLP, as in prometheus/prometheus#15165. Raising this to request corresponding support for delta temporality as well in Mimir.

Which solution do you envision (roughly)?

Expect delta temprality metrics to be ingested in Mimir as in the spec: https://opentelemetry.io/docs/specs/otel/metrics/data-model/#sums-delta-to-cumulative

Have you considered any alternatives?

The alternative is to run the deltatocumulative processor ourselves. It would be ideal if the metrics backend could support this similar to Prometheus.

[aknuds1]

@jesusvazquez you are more familiar with delta temporality support in Prometheus than I am. What's your take on supporting it in Mimir too?

[jesusvazquez]

I believe @fionaliao made an analysis on the Prometheus implementation and possible issues of running it in Mimir so I'll her ler explain why it may not work.

[fionaliao]

I'm investigating possible options for delta support in Mimir, but they'd likely take a fair bit of effort to implement. Unfortunately we can't just use the Prometheus implementation linked in the PR (which does the delta to cumulative conversion as part of the translation from OTLP to Prometheus) due to Mimir's distributed nature.

Currently Mimir's OTLP to Prometheus translation is done in the distributor. We can't guarantee all samples for the same series arrive on the same distributor, which is needed to generate the correct cumulative result. Additionally, distributors are designed to be stateless but the deltatocumulative conversion is stateful so it's not the right place to do the conversion anyway. In general, there's not a great place in Mimir to keep track of that state at the moment.