Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implementation of chown, lchown and fchown for the os module #1984

Merged
merged 8 commits into from
Jul 25, 2020

Conversation

LeBoucEtMistere
Copy link
Contributor

Hello all,

I am trying to implement the chown function for the os module (https://docs.python.org/3/library/os.html#os.chown), because of issue #1175 .
I am a new contributor and I"d like to have feedback on this first take at the implementation. It is not finished yet but I have a few questions and I wanted to be sure that I am going in the good direction.

As of now, the chown function works properly given that you invoke rustpython with root permissions. I can successfully change uid and gid of a file, or use -1 to preserve them. I am having a few issue with the path to file, it does not seem yet to accept unix shorthands like ~ (FileNotFound error, might come from nix lib, need to investigate).
I have not tested dir_fd and _follow_symlink yet since I am not sure how to handle these arguments with defaults values.
More particularly, in line 1988, I am not sure to understand how to initialise a new SupportFunc, what it actually is and why it takes Options on bool rather than bool. I took example on other functions and used false in the Options but I think it should be true since this function supports use of file descriptors and following symlinks, unless I did not understand properly what the bool means here.

I am planning on implementing lchown and fchown after this by using to this first chown function.

I am planning to write proper unit tests but I have no clue on how to write them and what you are expecting as proper tests.

Finally, in the python doc, it says that these functions raises AuditingEvents. I am not familiar with this, is it supported in rustpython yet ? Should I care about this ?

Thanks for your feedback

@LeBoucEtMistere
Copy link
Contributor Author

Up ? Is any seasoned contributor available to answer my questions and help me get this PR in a mergeable state ? :)

Copy link
Member

@coolreader18 coolreader18 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I missed this :) Nice work!

Comment on lines 1059 to 1081
let flag = match follow_symlinks.follow_symlinks {
true => nix::unistd::FchownatFlags::FollowSymlink,
false => nix::unistd::FchownatFlags::NoFollowSymlink,
};
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should fix the error in CI.

Suggested change
let flag = match follow_symlinks.follow_symlinks {
true => nix::unistd::FchownatFlags::FollowSymlink,
false => nix::unistd::FchownatFlags::NoFollowSymlink,
};
let flag = if follow_symlinks.follow_symlinks {
nix::unistd::FchownatFlags::FollowSymlink
} else {
nix::unistd::FchownatFlags::NoFollowSymlink
};

@coolreader18
Copy link
Member

SupportFuncs just determine what functions are in the supports_{fd,dir_fd,follow_symlinks} sets. For chown, it looks like it supports dir_fd and follow_symlinks, so you can put None, Some(true), Some(true). I think the Options are because some functions are supposed to support one of them but they don"t yet, so we put None for that as a sort of TODO. I think.

@coolreader18
Copy link
Member

Oh, and yeah, you can ignore the auditing events stuff.

@LeBoucEtMistere
Copy link
Contributor Author

I fixed the clippy lint.

SupportFuncs just determine what functions are in the supports_{fd,dir_fd,follow_symlinks} sets. For chown, it looks like it supports dir_fd and follow_symlinks, so you can put None, Some(true), Some(true). I think the Options are because some functions are supposed to support one of them but they don"t yet, so we put None for that as a sort of TODO. I think.

I also fixed this, thanks. Do you think it could be usefull to refactor this weird Option into a cleaner enum then ? Something like

enum SupportStatus {
    SUPPORTED,
    TODO,
    NOT_SUPPORTED,
}

I"ll gladly open a PR for this if you think this is useful.

@LeBoucEtMistere
Copy link
Contributor Author

Finally, I"d feel more confident if I wrote at least unit tests for this piece of code. I can write some rust tests on the rust function but I think that it misses the point and I"d rather write python tests where I actually call the function os.chown from python code (liek integration testing)
There was nothing in the CONTRIBUTING.md file regarding the tests, what do you expect and where are these kind of tests located in the codebase ? I feel like it"s important to write them as we add more functions to the python library to make sure there are no regressions whatsoever.

@coolreader18
Copy link
Member

Yeah, I think the 3-enum rather than the Option<bool> would definitely make it more clear.

re: unit testing; there is a test_os module in CPython that we could copy over, but it"s ~4000 lines and with just a naive copy over there"s 95 failing tests. You could write a few tests in tests/snippets/stdlib_os.py, though, maybe take inspiration from the chown tests in CPython?

@LeBoucEtMistere
Copy link
Contributor Author

Ok the CPython tests are a nice resource indeed ! I"ll try to draw inspiration from these for sure, and maybe add a few more test cases.

@LeBoucEtMistere
Copy link
Contributor Author

Regarding the issue I mentioned in my original comment regarding the tilde expansions, it seems that neither Rust nor Python do it, so I won"t add support for it.

@coolreader18
Copy link
Member

Ah, yeah, I think you have to manually opt in to that with os.path.expandpath(p)

@LeBoucEtMistere LeBoucEtMistere force-pushed the add_os_chown branch 2 times, most recently from 09a693b to 3071b00 Compare July 16, 2020 21:06
@LeBoucEtMistere
Copy link
Contributor Author

Just wrote a first few tests (mostly copied from CPython test suite) and I have a few things to say:

  • Overall the way testing is done for snippets (Lib) seems super messy and counterintuitive to me. I rather prefer the CPython test suite organisation with clear separation between function being tested, separation of tests cases, and so on. The strange custom runner you"re using is not making a lot of sense to me atm so I"ll gladly take any clarifications on this.
  • Partly because of the first point, I have not a lot of trust in the tests I wrote since there is very little observability on what is running or not inside the tests files. Mainly on the fact that for chown the test suite needs to be started with and without sudo permissions to test both behaviours. Again, if you have any idea or clarification on that, I"ll gladly hear them.

Regarding the CI failing, it seems that with rust stable bumping to 1.45 today, clippy is now more restrictive on certain things and code that was previously considered correct is no longer. It should eventually be fixed in another PR.

Regarding the sister functions of chown (lchown, fchown, ...), I will add them so please don"t close this PR now.

Regarding the adding of an enum for SupportFuncs, I will move this to another PR to keep things clean.

As I am a newcomer on this project and that PR got me to take a look around a few parts of it (namely the standard lib and the integration tests), I will write down my take on this "experience" and on the entry barrier to new contributors as I perceived it. The architecture of the Lib reimplementation could be improved by creating rust module for each library module and creating a single file for each function in this module. That way, code would be cleaner, it would reduce conflicts, it would allow us to write clean rust unit tests in each file to test only a function, without cluttering an already too large file. (eg the os file is already more than 2300 loc and we are far from having everything implemented)
My second point refers back to the test suites and more particularly the snippets. I feel like everything is over-complex and sub-performant. The test files are a mess with no clear organisation and separation. It"s like a gigantic plate of spaghettis.
Of course, this is an open source project with limited contributing workforce and time and I get these points are not a priority. But as a newcomer, I faced them almost instantly and I understand it can discourage people from making polished and well-tested contributions. I would then happily discuss these points with maintainers and open the needed issues to try and improve the codebase if this makes sense as well as the developer"s guidelines. I believe that properly testing the Lib as we add pieces to it is the best way to go, that it is fun to do, and that it is a good way to bring in new contributors with a myriad of self-contained PRs that would encourage best coding practices, mix rust and python and quickly obtain a nice and satisfactory finished new feature.
(All the paragraph above is my personal view on the subject, with the eyes of a newcomer, it might be totally wrong and subject to change)

@coolreader18
Copy link
Member

Yeah, the test snippets aren"t very well organized, partly because we don"t need them to be; the idea is that we just have a bunch of snippets that assert some basic functionality, and eventually we"ll try to rely more on the CPython test suite to ensure that we have parity. I definitely do understand that it might be confusing for new contributors, though, we should explain that more in tests/README.md. Just know that we know that they"re a mess, but our main focus w.r.t testing is getting the Lib/test suite to work.

@coolreader18
Copy link
Member

Also, the clippy lints should be fixed on master now.

@LeBoucEtMistere
Copy link
Contributor Author

I just added support for fd and sister functions fchown and lchown, I think that the PR is mergeable now. Let me know if there is anything to alter still.

Copy link
Member

@youknowone youknowone left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree to SupportStatus. For the snippets, it was historically important before we could run CPython unittests but not anymore like before. These days, we just need to cover a bit more there which is not covered by CPython unittests. I feel like adding note to related documentation is enough and there is not much reason to spend a lot of resources for the snippets test.

@youknowone youknowone merged commit 546ac88 into RustPython:master Jul 25, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants