Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS-S3 uploads failing in certain versions of windows 10 #1699

Open
calvindn opened this issue Jun 28, 2019 · 18 comments
Open

AWS-S3 uploads failing in certain versions of windows 10 #1699

calvindn opened this issue Jun 28, 2019 · 18 comments
Assignees
Labels
AWS S3 Plugin that handles uploads to Amazon AWS S3 Stale Old issues that haven't had activity recently

Comments

@calvindn
Copy link

calvindn commented Jun 28, 2019

Quick thank you to everyone who works on this project!

This is an odd issue I'm running into. I'm using the aws-s3 plugin, status bar, presigned url and we are not using companion. On some versions of Windows 10, large (~900mb) uploads fail across all browsers (Edge, Firefox, Chrome). It works great on my Windows 8 VM and one Windows 10 laptop I'm testing with. I'm trying to gather more information and a smaller example of the issue but I thought I'd see if there is any information out there already or if any of the maintainers have a suspicion as to what is happening.

The upload will progress very quickly, sit at 100% for 30 seconds and then Uppy will throw its timeout error. No upload-success or upload-complete event is thrown. The size of the upload reported by Uppy in upload-progress is consistent with the other operating systems I have tested.

I feel like I have to be missing something obvious... Appreciate any help.

@goto-bus-stop goto-bus-stop added the AWS S3 Plugin that handles uploads to Amazon AWS S3 label Jun 29, 2019
@goto-bus-stop goto-bus-stop self-assigned this Jun 29, 2019
@goto-bus-stop
Copy link
Contributor

I can't see anything obvious tbh! I'll also give this a shot with a Win10 VM on Monday.

@calvindn
Copy link
Author

calvindn commented Jul 2, 2019

Thank you, @goto-bus-stop.

I was wrong about upload-complete not being thrown. It's thrown once the XHRUpload times out and the video marked as failed. XHRUpload doesn't report timer done the way it does with uploads that complete correctly (debug on).

I tested a Lenovo laptop (version 1903, os build 18362.207, not working), Surface (version 1803, os build 17134.829, not working), Dell Laptop (version 1803, os build 17134.407, working) and a VM (version 1809, os build 17763.529, working).

For the time being we updated the timeout to be 5 minutes and that at least allows the uploads to complete. Sorry for the lack of information, it's such a weird issue. Hopefully soon I'll have some time to look into it a bit more closely.

@calvindn
Copy link
Author

This is really old we haven't had any issues come up in production, I think this should be closed.

@Velora
Copy link

Velora commented Jun 30, 2020

Hey @calvindn you've still never experienced this in production? We're running into this issue with some users and it is exactly as you describe: Some versions of Windows 10, large (~900mb) uploads fail across all browsers (Edge, Firefox, Chrome) with S3 uploads.

Are you running the adjusted timeout in production? We have an older version of uppy running in this instance, we are going to test the latest release to see if that solves it for us, but we are experiencing this in production.

@Velora
Copy link

Velora commented Jul 1, 2020

Following up on this because it is still an issue for us @goto-bus-stop.

We've updated Uppy to use the latest version, and we still have some Windows 10 users experiencing this issue in production on all browsers.

We're trying to get a HAR file from them to see more details. On the older release of Uppy we experienced exactly as calvindn described: "The upload will progress very quickly, sit at 100% for 30 seconds and then Uppy will throw its timeout error."

On the current release (v1.16.1), the progress will very quickly go up to 100%, but it will now never show an error. Two users that we have who are experiencing this issue said that the progress just sat at 100% for 20-30 minutes, at which point they decided to close the page.

It looks like the files are getting to us, but for some reason the user is not getting the upload success events. We've asked the users to make sure they don't have any browser extensions while attempting the upload so we are certain that browser extensions aren't causing the issue. Users experienced the same results in Chrome, Firefox, Edge, and Opera.

We've been unable to reproduce this on our Windows 10 machines, and for 99% of our users on Windows there are no issues.

One of our users with the issue mentioned using the following Windows 10 version:
Windows 10 Home
Version 1909
OS Build 18363.900

Any ideas of what the issue might be, or steps we can take to resolve it?

Thank you!

@Velora
Copy link

Velora commented Jul 1, 2020

Update, maybe related to #2336

We just took a look at the HAR file from one of our users with this issue.

We can see that the file was successfully uploaded to our S3 cache, but the success event never triggered for our user.

Relevant portion of the request/response from the HAR file below:

Resource update
The method used can update, delete or change data in the server

POST request data incomplete
The post data in the request entry is incomplete

Headers:
Accept */*
Accept-Encoding gzip, deflate, br
Accept-Language en-US,en;q=0.9,en-GB;q=0.8
Connection keep-alive
Content-Length 66635316
Content-Type audio/mpeg
Host heights.s3.amazonaws.com
Origin https://site.com
Referer https://site.com
Sec-Fetch-Dest empty
Sec-Fetch-Mode cors
Sec-Fetch-Site cross-site
User-Agent Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
0 Missing status code HTTP/1.1
Missing status code
The HAR file does not contain any status code for this response. This might be because the server never answered, or because the HTTP capture recording was stopped too soon, or because a network error happened.

Missing response headers
The headers are missing from the response data

More info about HTTP status codes in Wikipedia .
Chrome error
net::ERR_ABORTED
Headers:
No headers!?
Cache:
Cache data missing
The cache information is missing from the entry

Size:
Headers: (computed) 44 bytesBody: (computed) 0 bytesTotal: (computed) 44 bytes

@calvindn
Copy link
Author

calvindn commented Jul 2, 2020

@Velora Sorry for the slow reply, we had a holiday up here in Canada.

you've still never experienced this in production?

It would be more accurate to say we haven't had any complaints in production. I can see we've definitely hit the timeout (unfortunately we do have the large timeout running in prod) but I don't know the cause. I was planning on updating to the latest version to see if there was any impact on the number of timeouts being hit. Unfortunately, I don't have access to the devices that I could reproduce the issue on, for the time being. The only version I've tested is 1.0.2.

@Velora
Copy link

Velora commented Jul 7, 2020

Thanks @calvindn

@goto-bus-stop or @arturi do you have any ideas about this? Our users that are affected by this issue are able to reproduce it 100% of the time. Unfortunately, for these users it means that they will not be able to use our service as they cannot upload files no matter which browser they use.

Thank you!

@dreig
Copy link

dreig commented Jul 9, 2020

Hi,
We also have the same issue, pretty much as described by @calvindn and @Velora
Here's a screenshot of the network tab when I upload a 717MB file on the latest Chrome (83.0.4103.116) in windows:
Windows 10 Pro
Version 1903
Build 18362.900

uppy-aws-s3-upload-717MB(windows-chrome)
As you can see, the upload progresses very quickly, and then the browser waits 1.2min for the response from S3. And the default timeout or 30 seconds would have been too short for the request.
In fact, the upload progress is much too fast for my upload speed. at ~90Mbit/s I would expect 700MB to upload in roughly ~60sec.

For comparison, here is a screenshot of uploading the same file in Ubuntu (where the upload succeeds without extending the default timeout):
uppy-aws-s3-upload-717MB(ubuntu-chromium)

I've also found the following relevant issue ladjs/superagent#1236

It really seems to be a Windows10/XMLHttpRequest issue.

@goto-bus-stop, @arturi did you encounter anything similar before? How would you go about solving this?
A hacky way might be to have a special, really long timeout just for the case when the progress reaches 100%?

@goto-bus-stop goto-bus-stop reopened this Jul 9, 2020
@goto-bus-stop
Copy link
Contributor

I have a windows machine now so I'll see if I can repro this too.

@dreig
Copy link

dreig commented Jul 9, 2020

And another thing, if I go into Chrome settings and throttle my upload speed to 100Mbits/s (which is higher than what my ISP offers me: ~90Mbits/s) then the progress moves slower (and more accurately) and the upload finishes without triggering the timeout.

@Velora
Copy link

Velora commented Jul 10, 2020

Thanks for the note about the throttling @dreig.

We had one of our users who is experiencing uploading issues try various throttling settings, and they were able to see the progress bar move more accurately, but the largest file they were able to upload was about 70mb. They still ran into the timeout when trying to upload a video file. Progress would appear to go up accurately, but then would get stuck at 100% and timeout.

@goto-bus-stop if it is helpful, we have a screen recording with voiceover and a HAR file from this user when they tried the throttling yesterday. We can email it to you if you believe it would help, but I cannot post it here as it has their personal data.

Thanks!

@dreig
Copy link

dreig commented Jul 10, 2020

Did you try to implement direct uploads with any other library and let the affected users try to upload files to S3 using that instead?
Because we implemented direct uploads using ActiveStorage.js from rails (https://www.npmjs.com/package/@rails/activestorage) and it's almost the same story: the upload progress quickly, reaches 100% and then the user waits a few minutes for the upload to finish. The difference is that ActiveStorage.js doesn't implement any timeout, so the request is allowed to continue indefinitely, until it eventually succeeds.
This is why I think this issue is not related to any file format, but arises from a combination of Windows 10, AwsS3 and (relatively) high upload speed.

BTW, in my examples above, the timeout was set to 5 minutes.

@Velora
Copy link

Velora commented Jul 10, 2020

I agree the issue is definitely related to Windows 10, and not a particular file format, but it seems to happen with large files only (smallest reported issue of 66mb and average reported issue of 700mb ). In the case of the user that is working with us to diagnose the issue, they are at the extreme end where files above 70mb cannot upload (they have a 10mbps upload speed). Our other user reported the same issue only gets it with files that are around 800mb plus (still waiting on a HAR file and other details from them). We've already had them try turning off all browser extensions, disabling firewall, ensure the internet connection isn't dropping, etc.

We do have thousands of Windows 10 users uploading larger files (500mb-2GB) using Uppy XHRUpload to S3 daily, who don't have this issue. I haven't been able to reproduce this on a Windows 10 machine myself yet. I wonder if it could be a router or internet configuration issue with Windows where the upload speed reported isn't accurate?

@stale
Copy link

stale bot commented Jul 10, 2021

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the Stale Old issues that haven't had activity recently label Jul 10, 2021
@stale stale bot closed this as completed Jul 25, 2021
@danielvouch
Copy link

Popping this back up from the past but we seem to be experiencing this same issue, did anyone manage to solve this in the end?

@larswww
Copy link

larswww commented Sep 6, 2023

Same, in mobile Safari.

@arturi arturi assigned aduh95 and unassigned goto-bus-stop Sep 7, 2023
@arturi arturi reopened this Sep 7, 2023
@arturi
Copy link
Contributor

arturi commented Sep 7, 2023

@danielvouch @larswww could you give us some steps to reproduce, which type/size file to use, which OS/browser, type of connection?

@aduh95 aduh95 added Stale Old issues that haven't had activity recently and removed Stale Old issues that haven't had activity recently labels Jan 18, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
AWS S3 Plugin that handles uploads to Amazon AWS S3 Stale Old issues that haven't had activity recently
Projects
None yet
Development

No branches or pull requests

8 participants