Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

indexer: fix bulk indexing #1202

Merged
merged 1 commit into from
Sep 29, 2020
Merged

indexer: fix bulk indexing #1202

merged 1 commit into from
Sep 29, 2020

Conversation

rerowep
Copy link
Contributor

@rerowep rerowep commented Sep 8, 2020

  • Fixes race condition during parallel indexing of documents with
    persons creation.

Co-Authored-by: Peter Weber [email protected]

Why are you opening this PR?

  • Which task/US does it implement?
  • Which issue does it fix?

Dependencies

My PR depends on the following rero-ils-ui's PR(s):

  • rero/rero-ils-ui#

How to test?

  • What command should I have to run to test your PR?
  • What should I test through the UI?

Code review check list

  • Commit message template compliance.
  • Commit message without typos.
  • File names.
  • Functions names.
  • Functions docstrings.
  • Unnecessary commited files?

@rerowep rerowep self-assigned this Sep 8, 2020
@rerowep rerowep marked this pull request as draft September 8, 2020 11:12
@rerowep rerowep force-pushed the wep-fix-bulk-index branch 6 times, most recently from e1d3b9e to 6c381a4 Compare September 8, 2020 12:14
@iGormilhit iGormilhit added this to the v0.12.0 milestone Sep 9, 2020
@iGormilhit iGormilhit removed this from the v0.12.0 milestone Sep 9, 2020
@rerowep rerowep force-pushed the wep-fix-bulk-index branch 9 times, most recently from 948c602 to 03e7104 Compare September 11, 2020 15:14
@rerowep rerowep marked this pull request as ready for review September 12, 2020 06:12
@rerowep rerowep added this to the v0.13.0 milestone Sep 12, 2020
@rerowep rerowep force-pushed the wep-fix-bulk-index branch 3 times, most recently from 4e63581 to 6890af5 Compare September 15, 2020 07:05
rero_ils/modules/persons/api.py Show resolved Hide resolved
rero_ils/modules/persons/tasks.py Outdated Show resolved Hide resolved
* Fixes race condition during parallel indexing of documents with
persons creation.
* Adds cli to post create persons from documents file.
* Adds online update function for persons.

without person     : poetry run setup  503,53s user 35,71s system 62% cpu 14:19,69 total
with person        : poetry run setup  508,56s user 34,93s system 62% cpu 14:32,93 total
with person enqueue: poetry run setup -k  495,98s user 34,25s system 62% cpu 14:12,54 total

Co-Authored-by: Peter Weber <[email protected]>
@rerowep rerowep merged commit 3b0f108 into rero:dev Sep 29, 2020
@rerowep rerowep deleted the wep-fix-bulk-index branch October 1, 2020 06:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants