-
-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
RQ Tasks Failed Upon Importing Native MacOS modules #2058
Comments
You might want to cross-post this on the |
After further investigation, it was revealed that the issue could be reproduced by modules other than |
I'd welcome a PR that changes |
@selwin rq is pretty key to lots of things I do, and I work on modern mac machines mostly (and linux for deployment) I'd be up for trying to take this on? Any further guidance? |
I think starting with changing RQ to use os.spawn() in MacOS would be a good start. |
We could also try using subprocess instead of os.fork(). Ideally whichever method we use, we should allow the user to choose which forking method to use. |
On MacOS, If you attempt to import pyarrow (or importing
pandas
when thepyarrow
library is installed as well) [edit: orrequests
library orurllib.request
from standard library], within the module where you implement your task as illustrated below:This will result in an error message produced in the
rq worker
output, which looks something like this:I tried the same thing on Ubuntu, and the error wasn't reproducible. Therefore, I assume it only happens within MacOS.
One workaround for this issue is to import pyarrow in your worker script prior to the worker forking its process:
Another workaround I found was to run the
rq worker
as follows:Edit:
The issue turns out not to be specific to
pyarrow
. For instance, on MacOS if you importurllib.request
from the standard library on top of the module that implements your RQ jobs, you will encounter the exact same problem.After further investigation, it turns out that
urllib.request
internally imports_scproxy
—adarwin
platform-specific code. Therefore, the root cause of this issue is likely the process forking issue in MacOS.While this issue could presumably be resolved by using an alternative forking method on the
darwin
platform in the worker implementation, it would also be beneficial to document it. Thus, current users of RQ on MacOS could be warned, as they may encounter this problem when utilizing a library that depends on native MacOS libraries. This could even be the case when using some parts of the Python standard library.The text was updated successfully, but these errors were encountered: