I just tried this on dev.nceas.ucsb,.edu, first with a single 1Gb file, and then with two 1Gb files. They all uploaded fine, no errors from my home wifi with a cable modem connection rated at 300Mbps down/10 Mbps up. I did this on Mac with FF 112.0.2. Could be browser differences. I think figuring out the problem is going to take some sleuthing. @vchendrix are there any errors or warnings in the browser javascript console that give additional info we could explore?

Using Safari (MacOS 13.2.1 )

Firefox. Not really any interesting messages

All of those console log errors indicate connection lost or connect reset types of messages. I think tracking down why the connection was reset will be critical to figuring out the cause. The app failure is a side effect of the connection reset.

All of those console log errors indicate connection lost or connect reset types of messages. I think tracking down why the connection was reset will be critical to figuring out the cause. The app failure is a side effect of the connection reset.

Agreed. I am trying to see if there is something in my OS network settings. One thing to note is that my WIFI is the only problem. A wired connecion on my router works fine.

@vchendrix Did you determine if this issue and issue #2135 are local network or VPN problems? It looks like MetacatUI gave an appropriate error message. How would you envision the Metadata Editor handling these issues more gracefully?

@robyngit We discussed this on the ess dive meeting, let"s chat tomorrow on our dev call on issues to consider.

During today"s dev call, Matt provided more details on this issue, including ways in which we could handle network instability better.

In order to lessen the likelihood of getting web connection error in the first place, a longer-term solution could involve adopting newer technologies, like as HTTP/2 (allows multiple data transfers to occur simultaneously on a single connection) or web sockets (allows a more reliable connection between the user and the server). However, these improvements would require changes to both MetacatUI and our server-side systems.

At a minimum, we should improve MetacatUI"s ability to handle interrupted file transfers more gracefully. The objective for this issue is to retain the state of the user"s data package in the event of a network interruption during upload. Here"s what that might look like:

• Even if the upload fails, a user should still be able to save the metadata entered thus far, along with any files that have been successfully uploaded.
• Failed uploads should be removed from the resource map, and the resource map should not get lost or become corrupted/invalid. Permissions should not be changed.
• The user should have the option to retry failed uploads. (or perhaps MetacatUI should automatically retry failed uploads?)

@robyngit your proposed bulleted list seems like a good first pass to prevent data package corruption. I think we can pass on the longer-term solution, and simply work from your list of error-recovery fixes that would keep package editing working correctly. So, I repeat them here as a list to check off as we work through them. I think the first and second would need to be implemented together. I think the third item should be fine as is because the user can always add new files to a dataset -- but we should test that it works properly.

• Even if the upload fails, a user should still be able to save the metadata entered thus far, along with the resource map entries for any files that have been successfully uploaded.
• Failed uploads should be removed from the resource map, and the resource map should not get lost or become corrupted/invalid. Permissions should not be changed.
• TEST: The user should have the option to retry failed uploads. (or perhaps MetacatUI should automatically retry failed uploads?)

Could you work on prioritizing this in your bug fixes please? We can discuss whether this might be a good item for @rushirajnenuji when he is done his folder work.

@rushirajnenuji @robyngit Looking at the Apache logs, I see the following for one of the files in error

2024-01-31 08:57:55.774 PST [2024-01-31 16:57:55.774713] [R:YXyYx3iJ8NI] Request 0 on C:1rzEx3iJ8NI pid:15 tid:134847804729024
2024-01-31 08:57:55.774 PST [2024-01-31 16:57:55.774721] [R:YXyYx3iJ8NI] UA:"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
2024-01-31 08:57:55.774 PST [2024-01-31 16:57:55.774726] [R:YXyYx3iJ8NI] Referer:"https://portal-demo.wfsi-data.org/submit/wfsi 20240131T143232469-15b3e9a99ea5230"
2024-01-31 08:57:55.775 PST [2024-01-31 16:57:55.774729] [proxy_ajp:error] [R:YXyYx3iJ8NI] (70008)Partial results are valid but processing is incomplete: AH02822: dialog with client 35.191.26.153:57200 failed
2024-01-31 08:57:55.776 PST 35.191.26.153 - - [31/Jan/2024:16:57:52 +0000] [R:YXyYx3iJ8NI] "POST /catalog/d1/mn/v2/object/ HTTP/1.1" 400 226 - "https://portal-demo.wfsi-data.org/submit/wfsi-20240131T143232469-15b3e9a99ea5230" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"

From Slack:

We got three datasets today where PIs made updates through the website and the data files were removed. One of them got stuck on the spinning "Submitting..." button, and one knows that she had an internet connectivity issue while trying to submit. Justin and I haven"t been able to recreate these types of problems so we don"t really have more details, but it does seems to be happening at an increasing rate
It also seems more common for data packages with lots of data files. But maybe that"s just because there"s a higher likelihood that they"ll run into connectivity issues since they take longer to upload