Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

100 Mo limitation ? #19

Open
stonebig opened this issue Sep 4, 2023 · 9 comments
Open

100 Mo limitation ? #19

stonebig opened this issue Sep 4, 2023 · 9 comments
Assignees
Labels
enhancement New feature or request question Further information is requested

Comments

@stonebig
Copy link

stonebig commented Sep 4, 2023

I'm reading on the nice Minda treacy Cheatsheet that there is currently a limitation on Dataset to 100MB https://twitter.com/KirkDBorne/status/1696723401514369341

Question:

  • is it indeed the case ? is it 100Mo of compressed data ?
  • What could be done to make it less impacting / or a less small limit ?

It would be nice to feel only constrained by what can sanely fit in a WASM32.

@keyur32 keyur32 self-assigned this Sep 6, 2023
@keyur32 keyur32 added the question Further information is requested label Sep 6, 2023
@keyur32
Copy link
Collaborator

keyur32 commented Sep 6, 2023

Hi, yes, we have a limit of 100 MB of data that the built-in xl() function is able to reference either through the grid or through a PowerQuery connection. You can split this up into smaller chunks to workaround or use PowerQuery to summarize the data before loading in Excel.

@keyur32 keyur32 added the enhancement New feature or request label Aug 6, 2024
@keyur32
Copy link
Collaborator

keyur32 commented Sep 11, 2024

Hey @stonebig, is there a sense of data size you're looking for? It's been a while but this enhancement is still one we want to address.

@stonebig
Copy link
Author

100mo was looking an arbitrary low limit for no reason.

If can handle more data in jupyterlite free web than in excel paid Microsoft, I will feel punished for no reason, especially given the price difference

That would be bad, so that may be the measure of what you shall allow

@stonebig
Copy link
Author

I'm a big fan of pivotable, till 100 000 records, and don't like tricky cubes, when it's not sales. Maybe I'm biased

@stonebig
Copy link
Author

stonebig commented Sep 11, 2024

Also the datashader packages distorted my vision of "too much data", a while ago, no to mention duckdb and polars recently

@stonebig
Copy link
Author

imagine if soon gemini or whatever deliver a superior upload and pre-analysis power, and the result in a editable web notebook app.

@keyur32
Copy link
Collaborator

keyur32 commented Sep 18, 2024

Thanks for that note! We do plan to investigate increasing the limit. Will keep this open to track when it becomes available (no date yet).

@stonebig
Copy link
Author

My use is big etl one step, on azure data, in an easier to maintain way than M or vba

@stonebig
Copy link
Author

stonebig commented Oct 11, 2024

The actual problem is that I don't see where it's written about limitations and slow downs when you can't be python premium

It's a bit painful to have not it written in plain details.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants