Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

allocation failure installing rustc #2229

Closed
rillian opened this issue Feb 13, 2020 · 7 comments
Closed

allocation failure installing rustc #2229

rillian opened this issue Feb 13, 2020 · 7 comments
Labels
bug

Comments

@rillian
Copy link
Contributor

rillian commented Feb 13, 2020

Problem

Running rustup update to upgrade an install to the 1.41.0 stable release today, I have a repeatable failure:

  4.3 MiB /   4.3 MiB (100 %)   1.9 MiB/s in  2s ETA:  0s
info: installing component 'rust-std'
 15.5 MiB /  15.5 MiB (100 %)   2.6 MiB/s in  5s ETA:  0s
info: installing component 'rustc'
 48.6 MiB /  65.5 MiB ( 74 %)   2.0 MiB/s in 26s ETA:  8smemory allocation of 152166216 bytes failedAborted

This is on a mips host with 512 MiB of physical ram. I'd expect this to work on a larger machine, but it also doesn't seem correct that the installer is asking for twice the size of the file it's unpacking.

Steps

  1. Install rustup and rust 1.40.0.
  2. Limit host to half a GB of memory.
  3. Invoke rustup update.

Possible Solution(s)

Since the previous releases all worked, it's possible the rustc package just got big enough to trigger this in the 1.41.0 release.

Notes

I ran rustup self update to upgrade the tool itself to 1.21.1, but that didn't resolve the issue. I can also reproduce with rustup 1.20.2 re-installed by an older rustup-init binary, so the bug doesn't seem to be particularly new.

Note that this left the host without a working toolchain, since the installer removes the old release before unpacking the new one. I'm also unable to downgrade since rustup seems to now require clippy which wasn't available for this plafform in previous releases. Is there a way around that?

Command-line tar xvf works fine to unpack the archive.

Output of rustup --version:

rustup 1.21.1 (7832b2ebe 2019-12-20)

Output of rustup show:

Default host: mipsel-unknown-linux-gnu
rustup home:  /home/giles/.rustup

stable-mipsel-unknown-linux-gnu (default)
(rustc does not exist)
@rillian rillian added the bug label Feb 13, 2020
@rbtcollins
Copy link
Contributor

Please try setting RUSTUP_UNPACK_RAM to a lower figure than 400M, or provide more VM to rustup.

@rillian
Copy link
Contributor Author

rillian commented Feb 13, 2020

Yes RUSTUP_UNPACK_RAM=200000000 works around the issue. Thanks!

Any suggestions on how this could be handled by default? The budget could be clamped to some fraction of available ram, or the package unpack could be split across multiple tasks.

@rbtcollins
Copy link
Contributor

Yes, detecting usable RAM and subtracting an appropriate overhead amount would be a decent way to autotune this.

@kinnison
Copy link
Contributor

My only concern with such an approach would be if we were then unable to unpack due to a large file not fitting in RAM, yet it'd be mmap()d and so not need to fit when used. I think we'd probably need to support chunking up the extraction if we wanted to go down this route.

@rbtcollins
Copy link
Contributor

I think we some different sorts of issues all cropping up.
We have machines with oodles of RAM and artificial limits - ulimits or similar.
We have machines with very real RAM physical constraints and no swap where failures are hard.

In the former case rustc will fail to run, as RAM footprint is not being measured, assuming I recall my basics correctly.
In the latter case, rustc would run correctly, because its virtual address space is permitted, and as you note paging can take place for the binary image portion of the work.

I'm not sure that the two situations make a lot of difference to rustup, other than that we want the min() of the detected resources for autotuning. Autotuning the parameter would be strictly better than the current hardcoded default because we can dynamically cope with these smaller setups. It won't on its own solve problems with very small environments.

Solving very small environments - for that I think we need to fall back to streaming/chunking and optimising for minimising resource usage rather than minimising wall clock time. Autotuning can help here though - if unpack ram is under (say) 200MB, we switch to a different IO strategy.

Putting it altogether, it would look something like:

let unpack_ram=env.get('RUSTUP_UNPACK_RAM').or_else(||{min(magic_detect_ram(), 512MB)})?;
let io_strategy=unpack_ram > 200MB ? parallel : serial;

@kinnison
Copy link
Contributor

I think that makes a lot of sense. So the question is how we do the magic_detect_ram() work. Hmm... https://docs.rs/sys-info/0.5.9/sys_info/fn.mem_info.html looks plausible, plus we could use libc::ulimit() though I don't know what Windows equivalent would be.

@rbtcollins
Copy link
Contributor

mem_info returns avail for Windows, so thats fine; and Windows doesn't do ulimits last I heard.

rbtcollins added a commit to rbtcollins/rustup.rs that referenced this issue Feb 22, 2020
@rbtcollins
Fix rust-lang#2229 detect available memory
716bcf8 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
rbtcollins added a commit to rbtcollins/rustup.rs that referenced this issue Feb 22, 2020
@rbtcollins
Fix rust-lang#2229 detect available memory
81bdfca 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
rbtcollins added a commit to rbtcollins/rustup.rs that referenced this issue Feb 22, 2020
@rbtcollins
Fix rust-lang#2229 detect available memory
53497af 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
rbtcollins added a commit to rbtcollins/rustup.rs that referenced this issue Feb 24, 2020
@rbtcollins
Fix rust-lang#2229 detect available memory
5d88e2c 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
rbtcollins added a commit to rbtcollins/rustup.rs that referenced this issue Feb 26, 2020
@rbtcollins
Fix rust-lang#2229 detect available memory
a3a63c0 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
rbtcollins added a commit to rbtcollins/rustup.rs that referenced this issue Feb 26, 2020
@rbtcollins
Fix rust-lang#2229 detect available memory
719ace8 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
rbtcollins added a commit to rbtcollins/rustup.rs that referenced this issue Feb 26, 2020
@rbtcollins
Fix rust-lang#2229 detect available memory
839a76f 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
rbtcollins added a commit to rbtcollins/rustup.rs that referenced this issue Feb 27, 2020
@rbtcollins
Fix rust-lang#2229 detect available memory
f6d5cce 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
rbtcollins added a commit to rbtcollins/rustup.rs that referenced this issue Feb 27, 2020
@rbtcollins
Fix rust-lang#2229 detect available memory
f1ff29a 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
kinnison added a commit that referenced this issue Feb 27, 2020
@kinnison
Merge pull request #2236 from rbtcollins/2229
ed6f719 
Fix #2229 detect available memory
BeniCheni pushed a commit to BeniCheni/rustup.rs that referenced this issue May 10, 2020
@rbtcollins @BeniCheni
Fix rust-lang#2229 detect available memory
b782b47 
This doesn't implement streaming IO for low memory situations - we still
have the situation that low footprint situations will fail to install,
but while it is the case that rustc's memory footprint is lower than our
unpack footprint this is probably not urgent to fix, though I will get
around to it.

Being less aggressive about unpack buffer size though should reduce the
number of support tickets from folk in these cases, I hope.

We may end up getting tickets from folk with broken ulimit syscalls
though, who knows.
abernix added a commit to apollographql/federation that referenced this issue May 1, 2021
@abernix
infra(ci): Adjust Windows CI job parameters
e25349b 
To overcome an error which was occurring during the "Rustup install" step
that was producing a "memory allocation of 16777216 bytes failed" error that
was ALSO NOT causing the _setup_ step to fail, this commit:

- Use the 64-bit installer rather than the 32-bit!  I would have thought
  that this on its own could have been the major root of the problem, but it
  seems like it's also paired best with the next bullet.

- Limits the amount of RAM for the _installation of Rust_ (and nothing else),
  overriding the auto-detection which seems un-reliable in this virtual env.
  rust-lang/rustup#2229 (comment)

- Uses a larger resource size, which actually makes the tests run faster
  anyhow.  This will use more credits but use less minutes.  I'll look at
  the numbers in a month to see what I think, but I'm not worried right now.
  (note to self: 76203c/1900m, right now, jesse)

- Prevent missed failure catch in installation via .exe file, instead opting
  for using LASTEXITCODE which seems to be a recommended way to do this when
  using EXE files.  Having the "echo" there certainly seems like it could
  have been hurting things too (since echo returns 0, but that didn't seem to
  fix it on its own).
abernix added a commit to apollographql/federation that referenced this issue May 3, 2021
@abernix
infra(ci): Adjust Windows CI job parameters
0d5b9e3 
To overcome an error which was occurring during the "Rustup install" step
that was producing a "memory allocation of 16777216 bytes failed" error that
was ALSO NOT causing the _setup_ step to fail, this commit:

- Use the 64-bit installer rather than the 32-bit!  I would have thought
  that this on its own could have been the major root of the problem, but it
  seems like it's also paired best with the next bullet.

- Limits the amount of RAM for the _installation of Rust_ (and nothing else),
  overriding the auto-detection which seems un-reliable in this virtual env.
  rust-lang/rustup#2229 (comment)

- Uses a larger resource size, which actually makes the tests run faster
  anyhow.  This will use more credits but use less minutes.  I'll look at
  the numbers in a month to see what I think, but I'm not worried right now.
  (note to self: 76203c/1900m, right now, jesse)

- Prevent missed failure catch in installation via .exe file, instead opting
  for using LASTEXITCODE which seems to be a recommended way to do this when
  using EXE files.  Having the "echo" there certainly seems like it could
  have been hurting things too (since echo returns 0, but that didn't seem to
  fix it on its own).
@abernix abernix mentioned this issue May 3, 2021
Merged
abernix added a commit to apollographql/federation that referenced this issue May 3, 2021
@abernix
infra(ci): Adjust Windows CI job parameters (#729)
7f7e13d 
To overcome an error which was occurring during the "Rustup install" step
that was producing a "memory allocation of 16777216 bytes failed" error that
was ALSO NOT causing the _setup_ step to fail, this commit:

- Use the 64-bit installer rather than the 32-bit!  I would have thought
  that this on its own could have been the major root of the problem, but it
  seems like it's also paired best with the next bullet.

- Limits the amount of RAM for the _installation of Rust_ (and nothing else),
  overriding the auto-detection which seems un-reliable in this virtual env.
  rust-lang/rustup#2229 (comment)

- Uses a larger resource size, which actually makes the tests run faster
  anyhow.  This will use more credits but use less minutes.  I'll look at
  the numbers in a month to see what I think, but I'm not worried right now.
  (note to self: 76203c/1900m, right now, jesse)

- Prevent missed failure catch in installation via .exe file, instead opting
  for using LASTEXITCODE which seems to be a recommended way to do this when
  using EXE files.  Having the "echo" there certainly seems like it could
  have been hurting things too (since echo returns 0, but that didn't seem to
  fix it on its own).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@rillian @rbtcollins @kinnison