Importing mixed Rust/Python project fails when structured as recommended #490

ArniDagur · 2021-04-07T15:36:07Z

The README of this project says the following:

Mixed rust/python projects

To create a mixed rust/python project, create a folder with your module name (i.e. lib.name in Cargo.toml) next to your Cargo.toml and add your python sources there:
my-project
├── Cargo.toml
├── my_project
│   ├── __init__.py
│   └── bar.py
├── Readme.md
└── src
    └── lib.rs
maturin will add the native extension as a module in your python folder. When using develop, maturin will copy the native library and for cffi also the glue code to your python folder. You should add those files to your gitignore.

With cffi you can do from .my_project import lib and then use lib.my_native_function, with pyo3/rust-cpython you can directly from .my_project import my_native_function.

Example layout with pyo3 after maturin develop:
my-project
├── Cargo.toml
├── my_project
│   ├── __init__.py
│   ├── bar.py
│   └── my_project.cpython-36m-x86_64-linux-gnu.so
├── Readme.md
└── src
    └── lib.rs

This is the structure I'm using for python-adblock. The project is written entirely in Rust, but I'm using a mixed Rust/Python project in order to include Python type stubs. I have an __init__.py which imports the neccesary symbols from the native module.

However, some of my users have reported that the imports inside __init__.py fail as follows:

>>> import adblock
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/adblock/__init__.py", line 1, in <module>
    from .adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'

or

>>> import adblock
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/app/adblock/__init__.py", line 1, in <module>
    from adblock.adblock import __version__, Engine, FilterSet, BlockerResult, UrlSpecificResources
ModuleNotFoundError: No module named 'adblock.adblock'

See ArniDagur/python-adblock#17. As you can see, the result is the same whether I use from .adblock import ... or from adblock.adblock import .... I have only managed to reproduce this when the package is installed system wide (pip3 install .) and the current working directory is the root of my project (git repository).

Does anyone know what is going on? Am I doing something different from what is described in the README.md, or is the advice in the README.md wrong in some way?

The text was updated successfully, but these errors were encountered:

messense · 2021-05-13T15:20:56Z

pip isn't used, but I've added python3 -m sysconfig for these archs.

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391397

https://gitlab.alpinelinux.org/omni/aports/-/jobs/391398

armv7: Platform: "linux-armv8l"
armhf: Platform: "linux-armv8l"

Interesting. I think we are expecting Platform: "linux-armv7l"

I wonder what's is Rust target triple on these platforms.

See also randombit/botan#1543 (comment)

Signed-off-by: Andy Lok <[email protected]>

stusmall · 2022-03-04T15:03:48Z

I'm running into this same issue, but I might have a shorter path to reproducing it. I'm including as much detail as possible because I might be doing something weird in here. I'm not an experienced python dev and I'm kinda winging it as I go.

Check out maturin @ revision d531e49 (head of main when I pulled)
Install with cargo install -f --path .
Goto the cffi-mixed test crate with cd test-crates/cffi-mixed
Create and activate a venv. python3 -m venv venv; source venv/bin/activate
Install tox pip install tox
Run tox tox --skip-missing-interpreters (I'm just using the system interpreter to simply things):

(venv) ➜  cffi-mixed git:(main) tox --skip-missing-interpreters
.package create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.package
.package installdeps: maturin>=0.12,<0.13
py36 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py36
SKIPPED: InterpreterNotFound: python3.6
py37 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py37
SKIPPED: InterpreterNotFound: python3.7
py38 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py38
py38 installdeps: pytest
py38 inst: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz
py38 installed: attrs==21.4.0,cffi==1.15.0,cffi-mixed @ file:///home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pycparser==2.21,pyparsing==3.0.7,pytest==7.0.1,tomli==2.0.1
py38 run-test-pre: PYTHONHASHSEED='1058416854'
py38 run-test: commands[0] | pytest
================================= test session starts ==================================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
cachedir: test-crates/cffi-mixed/.tox/py38/.pytest_cache
rootdir: /home/stusmall/Workspace/maturin
collected 0 items / 1 error

======================================== ERRORS ========================================
______________ ERROR collecting test-crates/cffi-mixed/test_cffi_mixed.py ______________
ImportError while importing test module '/home/stusmall/Workspace/maturin/test-crates/cffi-mixed/test_cffi_mixed.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../../.pyenv/versions/3.8.12/lib/python3.8/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_cffi_mixed.py:3: in <module>
    import cffi_mixed
cffi_mixed/__init__.py:1: in <module>
    from .cffi_mixed import ffi, lib
E   ModuleNotFoundError: No module named 'cffi_mixed.cffi_mixed'
=============================== short test summary info ================================
ERROR test_cffi_mixed.py
!!!!!!!!!!!!!!!!!!!!!!!! Interrupted: 1 error during collection !!!!!!!!!!!!!!!!!!!!!!!!
=================================== 1 error in 0.08s ===================================
ERROR: InvocationError for command /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py38/bin/pytest (exited with code 2)
_______________________________________ summary ________________________________________
SKIPPED:  py36: InterpreterNotFound: python3.6
SKIPPED:  py37: InterpreterNotFound: python3.7
ERROR:   py38: commands failed

For some extra version info just in case it helps for reproducing the issue:

(venv) ➜  cffi-mixed git:(main) python3 --version
Python 3.8.12
(venv) ➜  cffi-mixed git:(main) pip --version
pip 21.1.1 from /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/lib/python3.8/site-packages/pip (python 3.8)
(venv) ➜  cffi-mixed git:(main) tox --version
3.24.5 imported from /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/lib/python3.8/site-packages/tox/__init__.py

What is interesting is that if I run maturin develop before running tox --skip-missing-interpreters then the test works.

env) ➜  cffi-mixed git:(main) maturin develop
🍹 Building a mixed python/rust project
🐍 Using CPython 3.8 at /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/bin/python to generate the cffi bindings
📡 Using build options bindings from pyproject.toml
Collecting cffi
  Using cached cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (446 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Installing collected packages: pycparser, cffi
Successfully installed cffi-1.15.0 pycparser-2.21
WARNING: You are using pip version 21.1.1; however, version 22.0.3 is available.
You should consider upgrading via the '/home/stusmall/Workspace/maturin/test-crates/cffi-mixed/venv/bin/python -m pip install --upgrade pip' command.
   Compiling cffi-mixed v0.1.0 (/home/stusmall/Workspace/maturin/test-crates/cffi-mixed)
    Finished dev [unoptimized   debuginfo] target(s) in 0.55s
📦 Built wheel to /tmp/.tmpsizYQW/cffi_mixed-0.1.0-py3-none-linux_x86_64.whl
🛠  Installed cffi-mixed-0.1.0
(venv) ➜  cffi-mixed git:(main) tox --skip-missing-interpreters
py36 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py36
SKIPPED: InterpreterNotFound: python3.6
py37 create: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/py37
SKIPPED: InterpreterNotFound: python3.7
py38 inst-nodeps: /home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz
py38 installed: attrs==21.4.0,cffi==1.15.0,cffi-mixed @ file:///home/stusmall/Workspace/maturin/test-crates/cffi-mixed/.tox/.tmp/package/1/cffi_mixed-0.1.0.tar.gz,iniconfig==1.1.1,packaging==21.3,pluggy==1.0.0,py==1.11.0,pycparser==2.21,pyparsing==3.0.7,pytest==7.0.1,tomli==2.0.1
py38 run-test-pre: PYTHONHASHSEED='2429669310'
py38 run-test: commands[0] | pytest
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0
cachedir: test-crates/cffi-mixed/.tox/py38/.pytest_cache
rootdir: /home/stusmall/Workspace/maturin
collected 3 items

test_cffi_mixed.py ...                                                                                                                                                                                        [100%]

================================================================================================= 3 passed in 0.02s =================================================================================================
______________________________________________________________________________________________________ summary ______________________________________________________________________________________________________
SKIPPED:  py36: InterpreterNotFound: python3.6
SKIPPED:  py37: InterpreterNotFound: python3.7
  py38: commands succeeded
  congratulations :)

So before I go further, I want to repeat, I'm not a python dev. So this might be missing the point of the initial bug or misunderstanding how modules work. But it looks to me that when from .cffi_mixed import ffi, lib is run instead of importing from cffi_mixed the wheel it is opening the directory on disk cff_mixed. When this happens before maturin develop is run, it doesn't have native.so in place, but it does afterwards and so the tests pass

messense · 2022-06-15T06:36:49Z

@stusmall Sorry for the late reply, I think your hypothesis is correct. We've changed the default project template to include a python folder in #855 to avoid such issue.

It is needed to prevent import errors as the repo root sometimes gets into the PYTHONPATH, e.g. when python interactive is used. In that case the native extension may not be loadable. See also PyO3/maturin#490

- replace poetry by maturin - introduce Makefile for simplify life of devs - add a maturin action - refactor the project to the recommended structure - redefine dependencies as pure pyproject (see PyO3/maturin#490 for details) On branch 18-maturin_build Changes to be committed: new file: .github/workflows/release.yaml modified: .gitignore renamed: h2o-data-rust/Cargo.lock -> Cargo.lock renamed: h2o-data-rust/Cargo.toml -> Cargo.toml new file: Makefile deleted: h2o-data-rust/README.md modified: poetry.lock modified: pyproject.toml renamed: farsante/__init__.py -> python/farsante/__init__.py renamed: farsante/h2o_dataset_create.py -> python/farsante/h2o_dataset_create.py renamed: farsante/h2o_dataset_create_all.py -> python/farsante/h2o_dataset_create_all.py renamed: farsante/pandas_dfs.py -> python/farsante/pandas_dfs.py renamed: farsante/pandas_generators.py -> python/farsante/pandas_generators.py renamed: farsante/pyspark_dfs.py -> python/farsante/pyspark_dfs.py renamed: farsante/pyspark_generators.py -> python/farsante/pyspark_generators.py renamed: h2o-data-rust/src/generators.rs -> src/generators.rs renamed: h2o-data-rust/src/helpers.rs -> src/helpers.rs renamed: h2o-data-rust/src/lib.rs -> src/lib.rs renamed: h2o-data-rust/src/main.rs -> src/main.rs

* Building system of farsante - replace poetry by maturin - introduce Makefile for simplify life of devs - add a maturin action - refactor the project to the recommended structure - redefine dependencies as pure pyproject (see PyO3/maturin#490 for details) On branch 18-maturin_build Changes to be committed: new file: .github/workflows/release.yaml modified: .gitignore renamed: h2o-data-rust/Cargo.lock -> Cargo.lock renamed: h2o-data-rust/Cargo.toml -> Cargo.toml new file: Makefile deleted: h2o-data-rust/README.md modified: poetry.lock modified: pyproject.toml renamed: farsante/__init__.py -> python/farsante/__init__.py renamed: farsante/h2o_dataset_create.py -> python/farsante/h2o_dataset_create.py renamed: farsante/h2o_dataset_create_all.py -> python/farsante/h2o_dataset_create_all.py renamed: farsante/pandas_dfs.py -> python/farsante/pandas_dfs.py renamed: farsante/pandas_generators.py -> python/farsante/pandas_generators.py renamed: farsante/pyspark_dfs.py -> python/farsante/pyspark_dfs.py renamed: farsante/pyspark_generators.py -> python/farsante/pyspark_generators.py renamed: h2o-data-rust/src/generators.rs -> src/generators.rs renamed: h2o-data-rust/src/helpers.rs -> src/helpers.rs renamed: h2o-data-rust/src/lib.rs -> src/lib.rs renamed: h2o-data-rust/src/main.rs -> src/main.rs * Drop poetry.lock By switching to maturing-builds we do not need poetry.lock anymore On branch 18-maturin_build Your branch is up to date with 'origin/18-maturin_build'. Changes to be committed: deleted: poetry.lock

… projects The previous layout leads to an import error when installing with `maturin build` and `pip install .`. This error was common enough that `maturin` changed the recommended project layout to what this commit does. A prior PR attempted to solve this by altering `lib.name` in Cargo.toml, but that did not work for me. - [Prior PR](apache#694) - [maturin ImportError issue](PyO3/maturin#490) - [maturin changes recommended project structure](PyO3/maturin#855)

#695) * chore: update to maturin's recommended project layout for rust/python projects The previous layout leads to an import error when installing with `maturin build` and `pip install .`. This error was common enough that `maturin` changed the recommended project layout to what this commit does. A prior PR attempted to solve this by altering `lib.name` in Cargo.toml, but that did not work for me. - [Prior PR](#694) - [maturin ImportError issue](PyO3/maturin#490) - [maturin changes recommended project structure](PyO3/maturin#855) * ci: update `ruff check` for nested python directory

Per PyO3/maturin#490 and PyO3/maturin#855, this avoids problems when importing from the project dir.

This comment was marked as resolved.

Sign in to view

messense mentioned this issue Apr 12, 2021

Always use EXT_SUFFIX from sysconfig #498

Merged

This comment was marked as resolved.

Sign in to view

messense referenced this issue in tikv/client-py Sep 1, 2021

no i686

cf4b559

Signed-off-by: Andy Lok <[email protected]>

messense mentioned this issue Dec 4, 2021

Out of Memory error when building cryptography wheel on armv7 and armv6 pyca/cryptography#6673

Closed

messense mentioned this issue Mar 15, 2022

Change mixed rust/python template project layout #855

Merged

messense mentioned this issue Jun 15, 2022

rename internal package adriangb/graphlib2#11

Merged

messense closed this as completed Jun 15, 2022

furiosamg mentioned this issue Aug 19, 2022

PyO3과련 기존 레이아웃 변경 Rollback 논의: python/furiosa/models --> furiosa/models furiosa-ai/furiosa-models#25

Closed

messense mentioned this issue Nov 23, 2022

Generate correct wheel name for armv8l #1289

Closed

davidhewitt mentioned this issue Aug 4, 2023

move python source in examples under python/ subdir PyO3/setuptools-rust#350

Merged

kylebarron mentioned this issue Aug 16, 2023

Fix maturin bundling for rust-python code stac-utils/pgstac#200

Merged

frank1010111 mentioned this issue Oct 5, 2023

🐛 fix import error pitfall for maturin frank1010111/pywaterflood#62

Merged

6 tasks

davidhewitt mentioned this issue Dec 4, 2023

try to debug codspeed issue PyO3/pyo3#3627

Merged

SemyonSinchenko mentioned this issue Mar 10, 2024

[Feat][CI] Building system of farsante MrPowers/farsante#19

Merged

Michael-J-Ward mentioned this issue May 14, 2024

fix lib.name in Cargo.toml apache/datafusion-python#694

Closed

Michael-J-Ward mentioned this issue May 14, 2024

chore: update to maturin's recommended project layout for rust/python… apache/datafusion-python#695

Merged

hohav added a commit to hohav/peppi-py that referenced this issue Jul 6, 2024

Nest Python sources under python/

a989bd1

Per PyO3/maturin#490 and PyO3/maturin#855, this avoids problems when importing from the project dir.

hohav added a commit to hohav/peppi-py that referenced this issue Jul 6, 2024

Nest Python sources under python/

7536b19

Per PyO3/maturin#490 and PyO3/maturin#855, this avoids problems when importing from the project dir.

hohav added a commit to hohav/peppi-py that referenced this issue Jul 6, 2024

Nest Python sources under python/

9823d9c

Per PyO3/maturin#490 and PyO3/maturin#855, this avoids problems when importing from the project dir.

hohav added a commit to hohav/peppi-py that referenced this issue Jul 8, 2024

Nest Python sources under python/

ea2d6cf

Per PyO3/maturin#490 and PyO3/maturin#855, this avoids problems when importing from the project dir.

hohav added a commit to hohav/peppi-py that referenced this issue Jul 15, 2024

Nest Python sources under python/

1e18831

Per PyO3/maturin#490 and PyO3/maturin#855, this avoids problems when importing from the project dir.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Importing mixed Rust/Python project fails when structured as recommended #490

Importing mixed Rust/Python project fails when structured as recommended #490

ArniDagur commented Apr 7, 2021

Mixed rust/python projects

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

messense commented May 13, 2021 •

edited

Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

stusmall commented Mar 4, 2022 •

edited

Loading

messense commented Jun 15, 2022

Importing mixed Rust/Python project fails when structured as recommended #490

Importing mixed Rust/Python project fails when structured as recommended #490

Comments

ArniDagur commented Apr 7, 2021

Mixed rust/python projects

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

messense commented May 13, 2021 • edited Loading

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

This comment was marked as resolved.

stusmall commented Mar 4, 2022 • edited Loading

messense commented Jun 15, 2022

messense commented May 13, 2021 •

edited

Loading

stusmall commented Mar 4, 2022 •

edited

Loading