Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

"Label names with matchers" call returns "not found" during compaction #14278

Closed
colega opened this issue Jun 7, 2024 · 0 comments · Fixed by #14279
Closed

"Label names with matchers" call returns "not found" during compaction #14278

colega opened this issue Jun 7, 2024 · 0 comments · Fixed by #14279

Comments

@colega
Copy link
Contributor

colega commented Jun 7, 2024

What did you do?

I called the "label names with matchers" during TSDB compaction.

What did you expect to see?

A successful response.

What did you see instead? Under which circumstances?

A "not found" error.

When it calls the headIndexReader.LabelNames(...):

prometheus/tsdb/head_read.go

Lines 90 to 104 in edd5588

// LabelNames returns all the unique label names present in the head
// that are within the time range mint to maxt.
func (h *headIndexReader) LabelNames(ctx context.Context, matchers ...*labels.Matcher) ([]string, error) {
if h.maxt < h.head.MinTime() || h.mint > h.head.MaxTime() {
return []string{}, nil
}
if len(matchers) == 0 {
labelNames := h.head.postings.LabelNames()
slices.Sort(labelNames)
return labelNames, nil
}
return labelNamesWithMatchers(ctx, h, matchers...)
}

That calls the labelNamesWithMatcher(...):

prometheus/tsdb/querier.go

Lines 445 to 460 in edd5588

func labelNamesWithMatchers(ctx context.Context, r IndexReader, matchers ...*labels.Matcher) ([]string, error) {
p, err := PostingsForMatchers(ctx, r, matchers...)
if err != nil {
return nil, err
}
var postings []storage.SeriesRef
for p.Next() {
postings = append(postings, p.At())
}
if err := p.Err(); err != nil {
return nil, fmt.Errorf("postings for label names with matchers: %w", err)
}
return r.LabelNamesFor(ctx, postings...)
}

Imagine that a compaction happens between p, err := PostingsForMatchers(ctx, r, matchers...) and return r.LabelNamesFor(ctx, postings...) calls in that method, then postings... contains references to series that were garbage-collected.

However, headIndexReader.LabelNamesFor fails in that case:

prometheus/tsdb/head_read.go

Lines 278 to 281 in edd5588

memSeries := h.head.series.getByID(chunks.HeadSeriesRef(id))
if memSeries == nil {
return nil, storage.ErrNotFound
}

This error breaks the loop and is not handled.

IMO, it shouldn't break the loop, this is a common case during compaction.

System information

No response

Prometheus version

No response

Prometheus configuration file

No response

Alertmanager version

No response

Alertmanager configuration file

No response

Logs

No response
colega added a commit to colega/prometheus that referenced this issue Jun 7, 2024
@colega
headIndexReader.LabelNamesFor: skip not found series
d0d361d 
It's quite common during the compaction cycle to hold series IDs for
series that aren't in the TSDB head anymore.

We shouldn't fail if that happens, as the caller has no way to figure
out which one of the IDs doesn't exist.

Fixes prometheus#14278

Signed-off-by: Oleg Zaytsev <[email protected]>
@colega colega mentioned this issue Jun 7, 2024
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

headIndexReader.LabelNamesFor: skip not found series colega/prometheus
1 participant
@colega