Numexpr on dask #8098
Numexpr on dask
#8098
-
Numexpr uses multi-threading to achieve better performance. Will this resource contention on used with dask dataframe to parallelise workflows? |
Beta Was this translation helpful? Give feedback.
Answered by
douglasdavis
Aug 30, 2021
Replies: 1 comment 1 reply
-
Since pandas will sometimes use numexpr if it's available, Dask may compete with numexpr for thread usage with some dask.dataframe workloads. The dask.dataframe documentation suggests limiting numexpr to a single thread; see e.g. dask.dataframe.DataFrame.query |
Beta Was this translation helpful? Give feedback.
1 reply
Answer selected by
marcelned
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Since pandas will sometimes use numexpr if it's available, Dask may compete with numexpr for thread usage with some dask.dataframe workloads. The dask.dataframe documentation suggests limiting numexpr to a single thread; see e.g. dask.dataframe.DataFrame.query