Hi,
Thanks for creating this issue. I just tried this real quick in our Unity storage quickstart example and could not reproduce it.
Here are the steps I tried,

• Unity 2020.1.7f1 and FirebaseStorage 7.1.0
• Uploaded "test.txt", "test.zip" and "test.gz" (created with gzip -c test.txt > test.gz) to storage bucket via the Firebase console web interface.
• Downloaded bytes for each of them in Editor and on my Android 8.0 device and I didn't see any crashes.

Could you verify if our quickstart example crashes for you on Android too? If yes, could you try enabling debug logging and running once again,

Firebase.LogLevel logLevel = Firebase.LogLevel.Debug;

Thank you for the reply! Did you also set the content encoding metadata to gzip for the gz stream? To be clear, it is the (correct) metadata that causes SDK to crash, there is no problem per se in storing gzipped streams without encoding metadata. I'm not sure if that can be done via the console web interface, hence I used gsutil cp -Z, which is the documented way of uploading gzip-encoded content to cloud storage. I will check the quickstart next week. Thanks again!

I see. I didn't try setting the content encoding metadata though the file type is identified as "application/x-gzip
" in the console. It will be quicker for us to debug if we can reproduce this easily in our quickstarts.

Hi @vimanyu ,

Instructions for reproducing the crash with the quickstart-unity project. You do not need to make any changes.

• Upload file to storage with gzip encoding. One way to do this is to use gsutil cp -Z <local-file> <remote-file>.
• You can verify that the file opens from Firebase console. In this instance the compressed file is automatically decompressed.
• Launch quickstart-unity storage sample
• Enter <remote-file> to the quickstart "Storage Location" field.
• Press "Download Bytes"
• Expected result: get the gzip-encoded bytestream

This works as expected in Editor, but produces a crash as shown on the top of this thread on Android. Tested with trunk on this repo and the linked Firebase SDK 7.1.0 for XCode 12 and dotnet4. Unity 2019.4.14f1. gsutil is from https://cloud.google.com/storage/docs/gsutil .

If you omit -Z from the gsutil upload command, the file is not encoded or compressed, and the setup works on both platforms. Gzip encoding is mandatory for us because it reduces some storage file sizes by a factor of 50:1, but because of this bug we cannot use the correct encoding metadata, and have to communicate encoding out-of-band.

Hi @vnorilo, I followed your repro steps, use gsutil with -Z to upload a file, and download bytes with no issue. Is there any other steps that I should pay attention to? How big is your file?

Hi Cynthia! Thanks for picking this up. Perhaps it results from a combination of Firebase SDK and Android system. As I speculated on top of this thread, Google Cloud will automatically decompress bytestreams when the http request doesn't have accepts: gzip . If Firebase SDK allocates the buffer based on file size but receives an uncompressed stream, that would explain this behavior.

At this point all I can really do is to try to make sure all the details are correct. Here I've got two files, gs://musopia-donotuse/readme.md and gs://musopia-donotuse/readme-identity.md. The former is uploaded with -Z, the latter not. The files are in a public-readable sandbox bucket, so you could even try these urls on your end.

Metadata as per gsutil

gs://musopia-donotuse/readme.md:
    Creation time:          Wed, 17 Mar 2021 13:34:30 GMT
    Update time:            Wed, 17 Mar 2021 13:34:43 GMT
    Storage class:          REGIONAL
    Cache-Control:          no-transform
    Content-Encoding:       gzip
    Content-Length:         251
    Content-Type:           application/octet-stream
    Metadata:               
        firebaseStorageDownloadTokens:defce827-c865-4ad4-974d-5cacb0c1134d
    Hash (crc32c):          vMMJYw==
    Hash (md5):             8qD9Edh2x6aA9aC AV6/Fg==
    ETag:                   CIaejNe4t 8CEAI=
    Generation:             1615988070223622
    Metageneration:         2

gs://musopia-donotuse/readme-identity.md:
    Creation time:          Wed, 17 Mar 2021 13:35:28 GMT
    Update time:            Wed, 17 Mar 2021 13:35:53 GMT
    Storage class:          REGIONAL
    Content-Length:         374
    Content-Type:           application/octet-stream
    Metadata:               
        firebaseStorageDownloadTokens:fea51e87-425a-45f9-8ab1-8762a6aa6d29
    Hash (crc32c):          F4i 2w==
    Hash (md5):             3tXAmpXvDTWnVfcuy6AM2g==
    ETag:                   CPGc5/K4t 8CEAI=
    Generation:             1615988128337521
    Metageneration:         2

I've attached screenshots of the quickstart app on the crashing device and the device system info screen.

Crash downloading readme.md

Success downloading readme-identity.md

(after correcting typoed url :P )

Device

Thanks again!

Hi @vnorilo, I managed to repro the issue both with our unity testapp and android testapp, our team will be looking into solve the issue.

Update: in the next release of firebase-storage:19.2.2 will contain a potential fix for the issue. Let's wait for the release to get out and check whether it's fixed. Thanks for being patient.

This might not need a fix in the Android Storage SDK, but we need to fix our documentation to state that -1 is a valid return value: firebase/firebase-android-sdk#2618

@schmidt-sebastian

Unity StorageReference.GetBytesAsync() API is calling into StorageReference.getStream(StreamDownloadTask.StreamProcessor processor) through C API

C code did not check StreamDownloadTask.getTotalByteCount() at all to throw any exception.

Judge by the user's callstack, this seem to be thrown from native Android SDK.

2021/03/09 10:16:11.359 6904 7586 Error StorageException java.lang.IndexOutOfBoundsException: The maximum allowed buffer size was exceeded.
2021/03/09 10:16:11.359 6904 7586 Error StorageException 	at com.google.firebase.storage.internal.cpp.CppByteDownloader.doInBackground(CppByteDownloader.java:55)
2021/03/09 10:16:11.359 6904 7586 Error StorageException 	at com.google.firebase.storage.StreamDownloadTask.run(com.google.firebase:firebase-storage@@19.1.1:179)
2021/03/09 10:16:11.359 6904 7586 Error StorageException 	at com.google.firebase.storage.StorageTask.lambda$getRunnable$7(com.google.firebase:firebase-storage@@19.1.1:1072)
2021/03/09 10:16:11.359 6904 7586 Error StorageException 	at com.google.firebase.storage.StorageTask$$Lambda$12.run(Unknown Source:2)
2021/03/09 10:16:11.359 6904 7586 Error StorageException 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1162)
2021/03/09 10:16:11.359 6904 7586 Error StorageException 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:636)
2021/03/09 10:16:11.359 6904 7586 Error StorageException 	at java.lang.Thread.run(Thread.java:784)

Perhaps it works with StorageReference.getBytes() and StorageReference.getStream() but not StorageReference.getStream(StreamDownloadTask.StreamProcessor processor) ?

Further dig:

The exception is thrown from this code, which is indeed part of C /Unity SDK. However, judged by the code, it does not seem to be an issue in C /Unity SDK. Here is the reason:

The user specified the maxDownloadSizeBytes is 1,000,000 byts = 1Mb. That is, this.cppBufferSize is 1,000,000.

FirebaseStorage.DefaultInstance.GetReference(<remote-file>).GetBytesAsync(1000000)

The gzip file provided by the user is 251 bytes, which is much less than 1,000,000.

This implies that stream.read(bytes, 0, bytes.length) still return non--1 value after the file is downloaded, and cause the total downloaded byte size to exceed maxDownloadSizeBytes specified by the user. This implies the issue is in StreamProcessor, a Firebase Android API.

This does not seem to be the same issue to firebase/firebase-android-sdk#2618

@schmidt-sebastian also mentioned that the backend does not provide the file size of gzip encoded file to the client API. Could that be the issue?

Update:

Tried Android testapp and it does not seems like anything odd about the value in bytesDownloaded. getTotalByteCount() returns -1 as @schmidt-sebastian mentioned in firebase/firebase-android-sdk#2618. But that should not cause the exception to be thrown in C SDK

The seems to plague both FirebaseStorage.GetBytesAsync() and FirebaseStorage.GetStreamAsync() but not FirebaseStorage.GetFileAsync().

The only workaround for this probably is to download the file to Android device using FirebaseStorage.GetFileAsync(), then read the file afterward.

Will need to further investigate into the issue.