Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve perfomance even more #2

Open
mre opened this issue Aug 1, 2018 · 5 comments
Open

Improve perfomance even more #2

mre opened this issue Aug 1, 2018 · 5 comments

Comments

@mre
Copy link
Owner

mre commented Aug 1, 2018

Suggestions here: https://www.reddit.com/r/rust/comments/93fbrj/fascat_a_faster_cat_implementation_using_splice/e3dl0f3/

@kishaningithub
Copy link

kishaningithub commented Aug 6, 2018

Came across this comment for your post in HN. This can get rid of the intermediary pipe (using copy_file_range syscall) and directly copy from one file descriptor to another inside kernel space.

@mre
Copy link
Owner Author

mre commented Aug 6, 2018

That's one option. We just need to keep in mind that this only supports Linux Kernels 4.5 and up. So that would be the March 2016 release and later.
I guess that should be fine, but just wanted to raise that concern.
Also just casually looking at the manpage, it seems like we can't specify the buffer size there (?).
Probably I'm missing something or it's not that important.

@mre
Copy link
Owner Author

mre commented Aug 9, 2018

As a first step, shall we increase the buffer size to 65536 bytes as suggested by the comment on Reddit @vbrandl ?
Don't see any downside here. 🤔

@vbrandl
Copy link
Collaborator

vbrandl commented Aug 9, 2018

I'm not too deep into low level systems programming but I think the optimal buffer size depends on the OS pages size, the architecture (32 or 64 bit) and maybe other factors.
Maybe there is a dynamic way to determine the optimal buffer size?

@mre
Copy link
Owner Author

mre commented Aug 9, 2018

The comment over on Reddit mentioned, that a buffer larger than PIPE_BUF would not work to prevent deadlocks, so I assume that is the upper bound. Was looking around for a constant somewhere in nix that we could use to be compliant with the architecture, and I found this:

/// Maximum number of bytes that is guaranteed to be atomic when writing to
/// a pipe.
PIPE_BUF = libc::_PC_PIPE_BUF,

Looks like, that's not public, though. It leads to those constants in libc, though but they have really small values: https://github.com/rust-lang/libc/search?q=PIPE_BUF&type=Code.

In the end, I don't really know what to do other than testing a bigger buffer and measuring the performance. Then take the buffer size with the best performance and wait until somebody screams. 😉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants