-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Allow numba functions within zip files to be cached #9630
base: main
Are you sure you want to change the base?
Conversation
333b4b2
to
7bc0c2a
Compare
Thanks for the PR! I'm just marking it as "2 - In progress" while we wait for CI.
Thanks for the comments, I do think they're helpful. It doesn't use pytest because there's a lot of homegrown addons to WRT to code formatting, we only check for flake8 compliance (with a large number of files excluded as they were historically never checked and they never got fixed up yet). I think there's also more that could be done to make things a smoother experience here, but it's been difficult to move forward with relative to other items. |
From a quick skim it looks like the tests aren't platform-agnostic WRT the way paths are formed (the root, and separators) - I haven't looked deeper, but perhaps |
Thanks, fixed |
Yes totally re homegrown addons... I would say that pytest is much much nicer, both from the perspective of maintainers being able to design better test suites, and the perspective of contributors who are running tests — for example I couldn't seem to enter pdb on a test failure with the current setup? Transitioning probably isn't trivial but also probably not herculean — pytest can collect & run standard unittests, and running two test commands for a while could be a way to get moving.
OK, you might want to check out The pytest transition probably requires some expertise but if you wanted a hand with linting & formatting, happy to help. |
This is ready for review! |
Oops, sorry, I forgot to check back when I was waiting for CI the other day. |
Hi team — the gentlest of pings here. I know things can get backlogged. I do think this is a fairly small unambiguous improvement, with some decent comments and a great test. If there's anything I can do to reduce the reviewing burden, please lmk. |
Sorry! I'll review it today |
6f9423e
to
69086f6
Compare
Sorry, I wrongly pushed to your branch an empty commit |
fe6a867
to
69086f6
Compare
No prob re empty commit, push away. Docs error seems unrelated though?
|
Yes! I guess something must have changed on conda-forge. |
I think that fixing the docs issue requires merging |
2650027
to
75221d5
Compare
(I moved the dev tools discussions to other issues. This PR is ready for a second review — thanks!) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Code looks good to me. Just wait for @sklam review and we can merge this one. Thanks for your contribution.
@sklam, are you ok with the current state or would like to review this one as well? If not, we can merge it. |
@max-sixty, can you add a news fragment for this PR? It will then be part of the release note. See https://github.com/numba/numba/tree/main/docs/upcoming_changes for README and examples. |
Great, added! |
/azp run |
Azure Pipelines successfully started running 1 pipeline(s). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just some capitalization fixes.
Co-authored-by: Emergency Self-Construct <[email protected]>
Co-authored-by: Emergency Self-Construct <[email protected]>
Co-authored-by: Emergency Self-Construct <[email protected]>
Co-authored-by: Emergency Self-Construct <[email protected]>
Co-authored-by: Emergency Self-Construct <[email protected]>
Done! |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM!
No urgency but it states still Waiting on Author — I think it's now ready for you to merge — lmk if that's not correct |
Numba doesn't currently allow
.zip
archives to contain numba functions when caching is enabled. This is all but required when using Spark.This change enables that, by recognizing a zip file and using a user-wide cache directory for the cache. It closes (most of?) #4908 (comment)
I'm the maintainer of numbagg, but haven't contributed directly to numba, so please let me know if the contribution needs any work, happy to make changes.
FWIW — hoping to be helpful — I found it not super easy to use the repo — I notice it doesn't use pytest / code formatters / etc. That does make contributions a bit harder for newcomers, though I'm sure it's for good reason...