docid deltas while indexing #2249
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
storing deltas is especially helpful for repetitive data like logs. In those cases, recording a doc on a term costed 4 bytes instead of 1 byte now. HDFS Indexing 1.1GB Total memory consumption: Before: 760 MB Now: 590 MB
PSeitz
force-pushed
the
index_delta_docids
branch
from
c11ddec to
f6512d4
Compare
src/postings/recorder.rs
Outdated
| VInt32Reader::new(&buffer[..]) | ||
| .map(|old_doc_id| doc_id_map.get_new_doc_id(old_doc_id)), | ||
| ); | ||
| doc_ids.extend(VInt32Reader::new(&buffer[..]).map(|delta_doc_id| { |
There was a problem hiding this comment.
Usage of prev_doc here looks like scan instead of map could make this more obvious.
fulmicoton
reviewed
Nov 13, 2023
src/postings/recorder.rs
Outdated
| VInt32Reader::new(&buffer[..]) | ||
| .map(|old_doc_id| doc_id_map.get_new_doc_id(old_doc_id)), | ||
| ); | ||
| doc_ids.extend(VInt32Reader::new(&buffer[..]).map(|delta_doc_id| { |
There was a problem hiding this comment.
- 1 in favor of scan of a loop.
Having prev_doc outside of the if-statement is not a good idea. Having it in both statement makes the code easier to (proof)read.
There was a problem hiding this comment.
switched to scan, but we can't apply the same technique in the other cases, because they contain a mix of delta and non delta encoded values
fulmicoton
approved these changes
Nov 13, 2023
PSeitz
added a commit
that referenced
this pull request
Apr 10, 2024
* docid deltas while indexing storing deltas is especially helpful for repetitive data like logs. In those cases, recording a doc on a term costed 4 bytes instead of 1 byte now. HDFS Indexing 1.1GB Total memory consumption: Before: 760 MB Now: 590 MB * use scan for delta decoding
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
storing deltas is especially helpful for repetitive data like logs.
In those cases, recording a doc on a term costed 4 bytes instead of 1
byte now.
HDFS Indexing 1.1GB Total memory consumption:
Before: 760 MB
Now: 590 MB