• [BUGFIX] Scraping: allow multiple samples on same series, with explicit timestamps. #14685
• [BUGFIX] Docker SD: fix crash in match_first_network mode when container is reconnected to a new network. #14654
• [BUGFIX] PromQL: fix experimental native histogram counter reset detection on stale samples. #14514
• [BUGFIX] PromQL: fix experimental native histograms getting corrupted due to vector selector bug in range queries. #14538
• [BUGFIX] PromQL: fix experimental native histogram memory corruption when using histogram_count or histogram_sum. #14605

Full Changelog: v2.54.0...v2.54.1

Release 2.54 brings a release candidate of a major new version of Remote Write: 2.0.
This is experimental at this time and may still change.
Remote-write v2 is enabled by default, but can be disabled via feature-flag web.remote-write-receiver.accepted-protobuf-messages.

• [CHANGE] Remote-Write: highest_timestamp_in_seconds and queue_highest_sent_timestamp_seconds metrics now initialized to 0. #14437
• [CHANGE] API: Split warnings from info annotations in API response. #14327
• [FEATURE] Remote-Write: Version 2.0 experimental, plus metadata in WAL via feature flag metadata-wal-records (defaults on). #14395,#14427,#14444
• [FEATURE] PromQL: add limitk() and limit_ratio() aggregation operators. #12503
• [ENHANCEMENT] PromQL: Accept underscores in literal numbers, e.g. 1_000_000 for 1 million. #12821
• [ENHANCEMENT] PromQL: float literal numbers and durations are now interchangeable (experimental). Example: time() - my_timestamp > 10m. #9138
• [ENHANCEMENT] PromQL: use Kahan summation for sum(). #14074,#14362
• [ENHANCEMENT] PromQL (experimental native histograms): Optimize histogram_count and histogram_sum functions. #14097
• [ENHANCEMENT] TSDB: Better support for out-of-order experimental native histogram samples. #14438
• [ENHANCEMENT] TSDB: Optimise seek within index. #14393
• [ENHANCEMENT] TSDB: Optimise deletion of stale series. #14307
• [ENHANCEMENT] TSDB: Reduce locking to optimise adding and removing series. #13286,#14286
• [ENHANCEMENT] TSDB: Small optimisation: streamline special handling for out-of-order data. #14396,#14584
• [ENHANCEMENT] Regexps: Optimize patterns with multiple prefixes. #13843,#14368
• [ENHANCEMENT] Regexps: Optimize patterns containing multiple literal strings. #14173
• [ENHANCEMENT] AWS SD: expose Primary IPv6 addresses as __meta_ec2_primary_ipv6_addresses. #14156
• [ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks. #10490
• [ENHANCEMENT] OpenStack SD: Use flavor.original_name if available. #14312
• [ENHANCEMENT] UI (experimental native histograms): more accurate representation. #13680,#14430
• [ENHANCEMENT] Agent: out_of_order_time_window config option now applies to agent. #14094
• [ENHANCEMENT] Notifier: Send any outstanding Alertmanager notifications when shutting down. #14290
• [ENHANCEMENT] Rules: Add label-matcher support to Rules API. #10194
• [ENHANCEMENT] HTTP API: Add url to message logged on error while sending response. #14209
• [BUGFIX] CLI: escape | characters when generating docs. #14420
• [BUGFIX] PromQL (experimental native histograms): Fix some binary operators between native histogram values. #14454
• [BUGFIX] TSDB: LabelNames API could fail during compaction. #14279
• [BUGFIX] TSDB: Fix rare issue where pending OOO read can be left dangling if creating querier fails. #14341
• [BUGFIX] TSDB: fix check for context cancellation in LabelNamesFor. #14302
• [BUGFIX] Rules: Fix rare panic on reload. #14366
• [BUGFIX] Config: In YAML marshalling, do not output a regexp field if it was never set. #14004
• [BUGFIX] Remote-Write: reject samples with future timestamps. #14304
• [BUGFIX] Remote-Write: Fix data corruption in remote write if max_sample_age is applied. #14078
• [BUGFIX] Notifier: Fix Alertmanager discovery not updating under heavy load. #14174
• [BUGFIX] Regexes: some Unicode characters were not matched by case-insensitive comparison. #14170,#14299
• [BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515

Many thanks to the Prometheus Team and contributors:
@zenador
@jjo
@rexagod
@darshanime
@charleskorn
@fpetkovski
@carrieedwards
@colega
@pracucci
@akunszt
@DrAuYueng
@paulojmdias
@Maniktherana
@rabenhorst
@saswatamcode
@B1F030
@yeya24
@rapphil
@liam-howe-maersk
@jkroepke
@FUSAKLA
@Ranveer777

Fix a bug where Prometheus would crash with a segmentation fault if a remote-read
request accessed a block on disk at about the same time as TSDB created a new block.

[BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515,#14523

RC1 includes a fix to a bug introduced in RC0:

• [BUGFIX] TSDB: Exclude OOO chunks mapped after compaction starts (introduced by #14396). #14584

Special release candidate build that incorporates performance improvements using
the dedupelabels Go tag. The new labels data structure deduplicates label names
and values via symbol or intern tables, trading off ~10% higher CPU usage for lower
memory usage.

docker pull prom/prometheus:v2.54.0-rc.0-dedupelabels

Release 2.54 brings a release candidate of a major new version of Remote Write: 2.0.
This is experimental at this time and may still change.
Remote-write v2 is enabled by default, but can be disabled via feature-flag web.remote-write-receiver.accepted-protobuf-messages.

• [CHANGE] Remote-Write: highest_timestamp_in_seconds and queue_highest_sent_timestamp_seconds metrics now initialized to 0. #14437
• [CHANGE] API: Split warnings from info annotations in API response. #14327
• [FEATURE] Remote-Write: Version 2.0 experimental, plus metadata in WAL via feature flag metadata-wal-records (defaults on). #14395,#14427,#14444
• [FEATURE] PromQL: add limitk() and limit_ratio() aggregation operators. #12503
• [ENHANCEMENT] PromQL: Accept underscores in literal numbers, e.g. 1_000_000 for 1 million. #12821
• [ENHANCEMENT] PromQL: float literal numbers and durations are now interchangeable (experimental). Example: time() - my_timestamp > 10m. #9138
• [ENHANCEMENT] PromQL: use Kahan summation for sum(). #14074,#14362
• [ENHANCEMENT] PromQL (experimental native histograms): Optimize histogram_count and histogram_sum functions. #14097
• [ENHANCEMENT] TSDB: Better support for out-of-order experimental native histogram samples. #14438
• [ENHANCEMENT] TSDB: Optimise seek within index. #14393
• [ENHANCEMENT] TSDB: Optimise deletion of stale series. #14307
• [ENHANCEMENT] TSDB: Reduce locking to optimise adding and removing series. #13286,#14286
• [ENHANCEMENT] Regexps: Optimize patterns with multiple prefixes. #13843,#14368
• [ENHANCEMENT] Regexps: Optimize patterns containing multiple literal strings. #14173
• [ENHANCEMENT] AWS SD: expose Primary IPv6 addresses as __meta_ec2_primary_ipv6_addresses. #14156
• [ENHANCEMENT] Docker SD: add MatchFirstNetwork for containers with multiple networks. #10490
• [ENHANCEMENT] OpenStack SD: Use flavor.original_name if available. #14312
• [ENHANCEMENT] UI (experimental native histograms): more accurate representation. #13680,#14430
• [ENHANCEMENT] Agent: out_of_order_time_window config option now applies to agent. #14094
• [ENHANCEMENT] Notifier: Send any outstanding Alertmanager notifications when shutting down. #14290
• [ENHANCEMENT] Rules: Add label-matcher support to Rules API. #10194
• [ENHANCEMENT] HTTP API: Add url to message logged on error while sending response. #14209
• [BUGFIX] CLI: escape | characters when generating docs. #14420
• [BUGFIX] PromQL (experimental native histograms): Fix some binary operators between native histogram values. #14454
• [BUGFIX] TSDB: LabelNames API could fail during compaction. #14279
• [BUGFIX] TSDB: Fix rare issue where pending OOO read can be left dangling if creating querier fails. #14341
• [BUGFIX] TSDB: fix check for context cancellation in LabelNamesFor. #14302
• [BUGFIX] Rules: Fix rare panic on reload. #14366
• [BUGFIX] Config: In YAML marshalling, do not output a regexp field if it was never set. #14004
• [BUGFIX] Remote-Write: reject samples with future timestamps. #14304
• [BUGFIX] Remote-Write: Fix data corruption in remote write if max_sample_age is applied. #14078
• [BUGFIX] Notifier: Fix Alertmanager discovery not updating under heavy load. #14174
• [BUGFIX] Regexes: some Unicode characters were not matched by case-insensitive comparison. #14170,#14299
• [BUGFIX] Remote-Read: Resolve occasional segmentation fault on query. #14515

This is a bug-fix release, reverting a change introduced in v2.51.0.

The bug was that remote-write would drop samples if the sending flow stalled for longer than it takes to write one "WAL segment". How long this takes depends on the data rate of your Prometheus; as a rough guide with 10 million series scraping once per minute it could be about 5 minutes. The issue is #14087.

• [BUGFIX] Remote-write: stop dropping samples in catch-up #14446

As usual, container images are available at https://quay.io/repository/prometheus/prometheus?tab=tags and https://hub.docker.com/r/prom/prometheus/tags

This release contains security fixes in dependencies and has been built with go1.21.11. #14309.

This release changes the default for GOGC, the Go runtime control for the trade-off between excess memory use and CPU usage. We have found that Prometheus operates with minimal additional CPU usage, but greatly reduced memory by adjusting the upstream Go default from 100 to 75.

• [CHANGE] Rules: Execute 1 query instead of N (where N is the number of alerts within alert rule) when restoring alerts. #13980 #14048
• [CHANGE] Runtime: Change GOGC threshold from 100 to 75 #14176 #14285
• [FEATURE] Rules: Add new option query_offset for each rule group via rule group configuration file and rule_query_offset as part of the global configuration to have more resilience for remote write delays. #14061 #14216 #14273
• [ENHANCEMENT] Rules: Add rule_group_last_restore_duration_seconds metric to measure the time it takes to restore a rule group. #13974
• [ENHANCEMENT] OTLP: Improve remote write format translation performance by using label set hashes for metric identifiers instead of string based ones. #14006 #13991
• [ENHANCEMENT] TSDB: Optimize querying with regexp matchers. #13620
• [BUGFIX] OTLP: Don't generate target_info unless there are metrics and at least one identifying label is defined. #13991
• [BUGFIX] Scrape: Do no try to ingest native histograms when the native histograms feature is turned off. This happened when protobuf scrape was enabled by for example the created time feature. #13987
• [BUGFIX] Scaleway SD: Use the instance's public IP if no private IP is available as the __address__ meta label. #13941
• [BUGFIX] Query logger: Do not leak file descriptors on error. #13948
• [BUGFIX] TSDB: Let queries with heavy regex matches be cancelled and not use up the CPU. #14096 #14103 #14118 #14199
• [BUGFIX] API: Do not warn if result count is equal to the limit, only when exceeding the limit for the series, label-names and label-values APIs. #14116
• [BUGFIX] TSDB: Fix head stats and hooks when replaying a corrupted snapshot. #14079

This release changes the default for GOGC, the Go runtime control for the trade-off between excess memory use and CPU usage. We have found that Prometheus operates with minimal additional CPU usage, but greatly reduced memory by adjusting the upstream Go default from 100 to 75.

• [CHANGE] Runtime: Change GOGC threshold from 50 to 75 #14285
• [BUGFIX] Rules: Fix Group.Equals() to take in account the new queryOffset too. Followup to #14061. #14273