Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Corruption when zipping fast #504

Closed
DebianArch64 opened this issue Apr 10, 2024 · 6 comments
Closed

Corruption when zipping fast #504

DebianArch64 opened this issue Apr 10, 2024 · 6 comments

Comments

@DebianArch64
Copy link

Probably something on my end, but for some reason Zip files are more corrupt when having no delay between adding files. So, there are two issues but hoping to have them both fixed on my side

The Output
Output without delay:

Archive:  /Users/debianarch/Downloads/install (11).ipa
  inflating: Payload/Taurine.app/essential_0-4_iphoneos-arm.deb  
  inflating: Payload/Taurine.app/bootstrap.tar.gz  
  error:  invalid compressed data to inflate
file #3:  bad zipfile offset (local header sig):  12863496
file #4:  bad zipfile offset (local header sig):  12603319
file #5:  bad zipfile offset (local header sig):  12617048
file #6:  bad zipfile offset (local header sig):  15534336
file #7:  bad zipfile offset (local header sig):  15535204
file #8:  bad zipfile offset (local header sig):  15535920
file #9:  bad zipfile offset (local header sig):  15536177
file #10:  bad zipfile offset (local header sig):  15776964
file #11:  bad zipfile offset (local header sig):  15777490
file #12:  bad zipfile offset (local header sig):  15779564
file #13:  bad zipfile offset (local header sig):  15780306
file #14:  bad zipfile offset (local header sig):  15797892
file #15:  bad zipfile offset (local header sig):  15799206
file #16:  bad zipfile offset (local header sig):  15799489
file #17:  bad zipfile offset (local header sig):  28942874
file #18:  bad zipfile offset (local header sig):  28961940
file #19:  bad zipfile offset (local header sig):  28968084
  inflating: Payload/Taurine.app/signcert.p12  
  inflating: Payload/Taurine.app/Info.plist  
  inflating: Payload/Taurine.app/PkgInfo  
  inflating: Payload/Taurine.app/_CodeSignature/CodeResources  
  inflating: Payload/Taurine.app/Taurine

Output with delay:

Archive:  /Users/debianarch/Downloads/install (13).ipa
  inflating: Payload/Taurine.app/essential_0-4_iphoneos-arm.deb  
  inflating: Payload/Taurine.app/bootstrap.tar.gz  
  error:  invalid compressed data to inflate
  inflating: Payload/Taurine.app/uicache.gz  
  inflating: Payload/Taurine.app/[email protected]  
  inflating: Payload/Taurine.app/basebinaries.tar  
  error:  invalid compressed data to inflate
  inflating: Payload/Taurine.app/Base.lproj/LaunchScreen.storyboardc/01J-lp-oVM-view-Ze5-6b-2t3.nib  
  inflating: Payload/Taurine.app/Base.lproj/LaunchScreen.storyboardc/UIViewController-01J-lp-oVM.nib  
  inflating: Payload/Taurine.app/Base.lproj/LaunchScreen.storyboardc/Info.plist  
  inflating: Payload/Taurine.app/tar.gz  
  inflating: Payload/Taurine.app/BuildConfigs.xcconfig  
  inflating: Payload/Taurine.app/Main.storyboardc/DKI-Yp-loo-view-cPN-qX-cZW.nib  
  inflating: Payload/Taurine.app/Main.storyboardc/UIViewController-DKI-Yp-loo.nib  
  inflating: Payload/Taurine.app/Main.storyboardc/BYZ-38-t0r-view-8bC-Xf-vdC.nib  
  inflating: Payload/Taurine.app/Main.storyboardc/UINavigationController-veQ-yo-6fF.nib  
  inflating: Payload/Taurine.app/Main.storyboardc/Info.plist  
  inflating: Payload/Taurine.app/Assets.car  
  error:  invalid compressed data to inflate
  inflating: Payload/Taurine.app/AppIcon76x76@2x~ipad.png  
  inflating: Payload/Taurine.app/ctbypass.gz  
  inflating: Payload/Taurine.app/org.coolstar.sileo_2.3_iphoneos-arm.deb  
  inflating: Payload/Taurine.app/signcert.p12  
  inflating: Payload/Taurine.app/Info.plist  
  inflating: Payload/Taurine.app/PkgInfo  
  inflating: Payload/Taurine.app/_CodeSignature/CodeResources  
  inflating: Payload/Taurine.app/Taurine

The Code
Pretty much, I have an emscripten WASM directory that I'm zipping up using zip.js. To be speedy I am uploading the chunks to a websocket as it's being zipped up:

const writer =
  new zip.ZipWriterStream({
    dataDescriptor: true, // this is required for streams ??? https://github.com/gildas-lormeau/zip.js/issues/274#issuecomment-923032335
    extendedTimestamp: false,

    keepOrder: true,
  });
  
  -----
 writer.readable.pipeTo(
  new WritableStream({
    write: (chunk) => {
       // here I send the chunks to a websocket
    },
    close: () => {
      console.error("closing socket");
      socket.close();
    },
    abort: () => {
      console.error("aborting socket");
      socket.close();
    },
  }),
  { preventClose: true }
);

----

const test = new Blob([
      FS.readFile(path, { flags: "r", encoding: "binary" }) as Uint8Array,
    ]);
    // await delay(1000); // this is the delay i need to add for things to be less broken
    await writer.zipWriter.add(inferedPath, new zip.BlobReader(test));
@gildas-lormeau
Copy link
Owner

gildas-lormeau commented Apr 10, 2024 •
edited
Loading

Did you implement anything to ensure that websocket messages are ordered or does your environment guarantee it? See https://stackoverflow.com/questions/11804721/can-websocket-messages-arrive-out-of-order

@DebianArch64
Copy link
Author

DebianArch64 commented Apr 10, 2024 •
edited
Loading

Did you implement anything to ensure that websocket messages are ordered or does your environment guarantee it? See https://stackoverflow.com/questions/11804721/can-websocket-messages-arrive-out-of-order

The websocket isn't an issue, it's receiving the same amount of bytes and in the right order (I printed out size of bytes received in my erlang backend and size of bytes written from JavaScript) so the zip I'm making using .add function is what's wonky, it's really a few of the exact same entries that aren't writing correctly (the failed to inflate message for the same files each time)

@gildas-lormeau
Copy link
Owner

Can you reproduce the issue if you replace new WritableStream({ ... }) with a writableproperty from a TransformStream object? In that case, is the readable property of the TransformStream returning a corrupted stream?

@DebianArch64
Copy link
Author

Same thing happens

const {readable, writable} = new TransformStream({flush: () => { writer.close() }})
  const reader = readable.getReader()
  const yes = async () => {
  while (true) {
    const { done, value } = await reader.read();
    if (done) {
      break;
    }
    
    // send chunk
  }
  yes()
  writer.readable.pipeTo(writable, { preventClose: true })

Output without delay:

Archive:  /Users/debianarch/Downloads/install (18).ipa
inflating: Payload/Taurine.app/essential_0-4_iphoneos-arm.deb  
inflating: Payload/Taurine.app/bootstrap.tar.gz  
error:  invalid compressed data to inflate
file #3:  bad zipfile offset (local header sig):  12863496
file #4:  bad zipfile offset (local header sig):  12603319
file #5:  bad zipfile offset (local header sig):  12617048
file #6:  bad zipfile offset (local header sig):  15534336
file #7:  bad zipfile offset (local header sig):  15535204
file #8:  bad zipfile offset (local header sig):  15535920
file #9:  bad zipfile offset (local header sig):  15536177
file #10:  bad zipfile offset (local header sig):  15776964
file #11:  bad zipfile offset (local header sig):  15777490
file #12:  bad zipfile offset (local header sig):  15779564
file #13:  bad zipfile offset (local header sig):  15780306
file #14:  bad zipfile offset (local header sig):  15797892
file #15:  bad zipfile offset (local header sig):  15799206
file #16:  bad zipfile offset (local header sig):  15799489
file #17:  bad zipfile offset (local header sig):  28942874
file #18:  bad zipfile offset (local header sig):  28961940
file #19:  bad zipfile offset (local header sig):  28968084
inflating: Payload/Taurine.app/signcert.p12  
inflating: Payload/Taurine.app/Info.plist  
inflating: Payload/Taurine.app/PkgInfo  
inflating: Payload/Taurine.app/_CodeSignature/CodeResources  
inflating: Payload/Taurine.app/Taurine

@DebianArch64
Copy link
Author

DebianArch64 commented Apr 10, 2024 •
edited
Loading

My bad found the issue, i'm embarrassed sorry for wasting your time

@gildas-lormeau
Copy link
Owner

gildas-lormeau commented Apr 10, 2024 •
edited
Loading

No problem, I'm glad to hear that the issue is fixed :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@gildas-lormeau @DebianArch64