Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Slowness when uploading large amount of files #3199

Open
jimmybreeze opened this issue Nov 20, 2024 · 13 comments
Open

Slowness when uploading large amount of files #3199

jimmybreeze opened this issue Nov 20, 2024 · 13 comments
Labels
bug

Comments

@jimmybreeze
Copy link

Nicotine version: 3.3.6
Operating System/Distribution: Windows 11

Describe the bug

I have a user downloading a large amount of files, around 900GB . It seems that the client is very slow and freezes constantly when trying to navigate around the app. It also seems to make the entire system slow at times. I'm not sure this is to be expected when someone is downloading that much, but wanted to just report this.

Specs of server shouldn't be a concern:
i3-13100
32GB RAM
Files and software on NVME SSD, plenty of space

Expected behavior

The program to not freeze and cause the system to hang

Steps to reproduce the bug

Have someone download a lot of files and see if it freezes

Additional context

Let me know if I need to get any logs.
@mathiascode
Copy link
Member

Thanks for opening an issue. A couple of questions:

  • How many files are listed in the Uploads tab?
  • How many users are you uploading to (mentioned in the status bar, next to the total upload speed)?
  • What does CPU, memory and disk usage for Nicotine look like in Windows' Task Manager?
  • Does disconnecting from the server (Menu -> Disconnect) stop the slowness?

@jimmybreeze
Copy link
Author

Number of files is 44,989.
Currently 3, one single user is the one with 900GB in uploads.
CPU is hovering around 0-0.5% usage, RAM at 84MB, disk topping out at about 8MB/s
After disconnecting, clicking around the different tabs was fine. Connected again and first click, it got hung up.

@mathiascode
Copy link
Member

How is the CPU usage while Nicotine hangs? Does it hang no matter what part of the application you use? Does limiting the upload speed in Nicotine help?

@mathiascode
Copy link
Member

Another thing I forgot to ask: do you have any plugins installed/activated? I remember someone mentioning that the more-upload-stats plugin was causing slowdown issues.

@jimmybreeze
Copy link
Author

Cpu for nicotine goes up to 20-29%, but overall cpu usage is still below 40%. Freezing seems to be random, happens in search, uploads, downloads.

Speed for that user isn't anything crazy, have had other sessions with multiple users downloading much greater speed. I run nicotine through my VPN, getting about 47Mbps while my actual connection is 1000Mbps.

0 plugins installed

@slook
Copy link
Member

slook commented Nov 22, 2024 •
edited
Loading

N process only uses a single core, the i3-13100 is 4-core and Windows Task Manager shows aggregated figures, so 25% CPU usage is maxing out the available resources which is when freezing will happen, probably during iterating through 44,989 items while updating the transferring list item.

However this should only occur while the Uploads tab is active, since the list shouldn't update when another tab is active.

So, the freezing must be happening somewhere in the core, such as here:

nicotine-plus/pynicotine/uploads.py

Lines 204 to 205 in 8e12c76

len(queued_uploads)
for username, queued_uploads in self.queued_users.items() if self.is_privileged(username)

nicotine-plus/pynicotine/search.py

Line 758 in 8e12c76

inqueue=core.uploads.get_upload_queue_size(username),

@jimmybreeze Could you try turning Off the following option and let us know if the problem disappears or not?...
Preferences > Searches > "Respond to search requests from other users" > Off

@jimmybreeze
Copy link
Author

jimmybreeze commented Nov 22, 2024 via email

@jimmybreeze
Copy link
Author

jimmybreeze commented Nov 22, 2024 via email

@jimmybreeze
Copy link
Author

After disabling that, it did freeze a few times still. Currently down to 27k files and has seemed to stop freezing at this point. Can close if need be, unless any ideas on how to improve performance on huge upload lists can be thought of.

@slook
Copy link
Member

slook commented Nov 26, 2024

Leave this issue open. At some point we'll do some profiling to identify where the sticking points are.

My theory for one potential hang up is that when the client serves search results to a privileged user, it is needed to iterate through all the items of all the users that are in the uploads list to calculate their Queue position for each returned result item, because this value isn't readily available and isn't cached like it is for the ordinary (non-privileged) Queue (which is simply the total number of queued uploads).

@mathiascode
Copy link
Member

mathiascode commented Nov 27, 2024 •
edited
Loading

My theory for one potential hang up is that when the client serves search results to a privileged user, it is needed to iterate through all the items of all the users that are in the uploads list to calculate their Queue position for each returned result item, because this value isn't readily available and isn't cached like it is for the ordinary (non-privileged) Queue (which is simply the total number of queued uploads).

I think it's more likely the queue position request for each queued upload is causing problems. 45k requests at once from a single user is quite a lot of work.

Relevant code:

nicotine-plus/pynicotine/uploads.py

Lines 1158 to 1203 in 892e7c4

def _place_in_queue_request(self, msg):
"""Peer code 51."""
username = msg.username
virtual_path = msg.file
upload = self.queued_users.get(username, {}).get(virtual_path)
if upload is None:
return
is_fifo_queue = config.sections["transfers"]["fifoqueue"]
is_privileged_queue = self.is_privileged(username)
privileged_queued_users = {k: v for k, v in self.queued_users.items() if self.is_privileged(k)}
queue_position = 0
if is_fifo_queue:
num_non_privileged = 0
for position, i_upload in enumerate(self.queued_transfers, start=1):
if is_privileged_queue and i_upload.username not in privileged_queued_users:
num_non_privileged = 1
if i_upload == upload:
queue_position = position - num_non_privileged
break
else:
for position, i_upload in enumerate(self.queued_users.get(username, {}).values(), start=1):
if i_upload == upload:
if is_privileged_queue:
num_queued_users = len(privileged_queued_users)
else:
# Cycling through privileged users first
queue_position = sum(
len(queued_uploads) for queued_uploads in privileged_queued_users.values())
num_queued_users = len(self.queued_users)
queue_position = position * num_queued_users
break
if queue_position > 0:
core.send_message_to_peer(
username, PlaceInQueueResponse(virtual_path, queue_position))
# Update queue position in our list of uploads
upload.queue_position = queue_position
self._update_transfer(upload, update_parent=False)

@slook
Copy link
Member

slook commented Nov 27, 2024

@jimmybreeze What is your "Upload queue type" preference set to, is it "Round robin" or "First In, First Out"?

@jimmybreeze
Copy link
Author

jimmybreeze commented Nov 27, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mathiascode @jimmybreeze @slook