Building requires >1GB memory #6658

TheBlueMatt · 2015-09-10T22:43:19Z

CXX test/test_test_bitcoin-script_tests.o
g : internal compiler error: Killed (program cc1plus)

script_tests requires more than 1GB memory to build, killing a lot of VMs.

theuni · 2015-09-11T04:24:08Z

I don't think an ICE is really our concern. I assume clang manages ok?

TheBlueMatt · 2015-09-11T04:26:39Z

Hmm? I'd say it being impossible to build Bitcoin Core by default on many VPSes is our concern.

On September 11, 2015 12:24:16 AM EDT, Cory Fields [email protected] wrote:

I don't think an ICE is really our concern. I assume clang manages ok?

Reply to this email directly or view it on GitHub:
#6658 (comment)

sipa · 2015-09-11T04:29:27Z

Internal Compiler Errot

luke-jr · 2015-09-11T04:31:28Z

@TheBlueMatt Try using CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768"

TheBlueMatt · 2015-09-11T04:32:10Z

The ICE in question is OOM killer eating g .

On September 11, 2015 12:29:35 AM EDT, Pieter Wuille [email protected] wrote:

Internal Compiler Errot

Reply to this email directly or view it on GitHub:
#6658 (comment)

TheBlueMatt · 2015-09-11T04:33:17Z

@luke-jr: I can get it to build fine (--disable-tests), I just care that we fix the actual issue.

On September 11, 2015 12:31:36 AM EDT, Luke-Jr [email protected] wrote:

@TheBlueMatt Try using CXXFLAGS="--param ggc-min-expand=1 --param
ggc-min-heapsize=32768"

Reply to this email directly or view it on GitHub:
#6658 (comment)

theuni · 2015-09-11T04:34:49Z

I'd be concerned if no compiler could build on that VPS. However, I'd be willing to bet that it's just a matter of an inefficient gcc version. If clang dies the same way, I'll buy you a beer this weekend :)

luke-jr · 2015-09-11T04:35:20Z

@TheBlueMatt Memory use is not an issue, if it correctly builds with the appropriate CXXFLAGS... IMO

TheBlueMatt · 2015-09-11T04:38:39Z

This is a default gcc on a relatively recent Debian. I really don't think the response to "I can't build Bitcoin Core on a vps with 1GB of memory with default parameters" should be "oh, but if you set these flags you can!"

On September 11, 2015 12:34:57 AM EDT, Cory Fields [email protected] wrote:

I'd be concerned if no compiler could build on that VPS. However, I'd
be willing to bet that it's just a matter of an inefficient gcc
version. If clang dies the same way, I'll buy you a beer this weekend
:)

Reply to this email directly or view it on GitHub:
#6658 (comment)

unsystemizer · 2015-09-11T12:23:40Z

0.11 does require more RAM and in my experience --disable-tests doesn't help.
0.10 could be built on systems with 512MB of RAM (with and without --disable-tests).

jgarzik · 2015-09-11T12:44:04Z

Any instrumentation on why script_test takes so much memory? Would prefer to discern "why" before taking any action.

In general we should be able to build Bitcoin Core on a 1GB VPS.

gmaxwell · 2015-09-11T12:44:54Z

Y'all being overly pedantic. Matt identified a usability problem: User obtains and attempts to compile the software with normal instructions on a common and reasonable platform type, and it crashes.

If you want to suggest that the docs should have instructions with low mem cflags or whatnot, OKAY, but without that it doesn't stop being an issue because just us experts can work around it. Our memory usage is kind of obnoxious during compile time IMO, and if we could do some minor tweaks to reduce the peak that would be nice too-- but at the very least we should provide some instructions.

laanwj · 2015-09-18T00:47:02Z

This is actually known and documented: https://github.com/bitcoin/bitcoin/blob/master/doc/build-unix.md#system-requirements (feel free to add the recommendation for low-memory build flags tho!)

I would have expected that no longer using boost::spirit (for JSON) and boost::asio (for RPC) would reduce compilation memory usage - unfortunate to hear that it became worse.

Anyhow: cross compilation and pseudo-static compilation is extremely easy with the depends system, I'd recommend using that.

laanwj · 2015-09-18T00:56:26Z

Oh wait, the compilation of that file uses more than 1GB of memory?
Ok, reopening, that is definitly not good.

TheBlueMatt · 2015-09-18T19:46:48Z

Yes, the VPS in question has a total of 1.5GB memory (1GB ram .5 swap), and nothing else of note was running at the time.

laanwj · 2015-09-29T17:04:11Z

I measured using gcc 4.8.4 (64 bit): Top 10 memory usage during build:

nr	source	elapsed	maxresident (kB)
1.	main.cpp	0:36.58	`1248524`
2.	init.cpp	0:24.24	964136
3.	wallet/wallet.cpp	0:22.11	896096
4.	test/script_tests.cpp	0:19.44	859708
5.	net.cpp	0:18.85	813836
6.	wallet/rpcwallet.cpp	0:14.16	727956
7.	wallet/walletdb.cpp	0:13.48	719388
8.	wallet/rpcdump.cpp	0:12.42	697660
9.	qt/walletmodel.cpp	0:11.26	680500
10.	rpcrawtransaction.cpp	0:11.44	666548

Will try with clang some other time.

laanwj · 2015-09-30T10:05:37Z

Same information, but for clang 3.6.0 (also 64-bit):

nr	source	elapsed	maxresident
1	main.cpp	0:28.34	723960
2	wallet/wallet.cpp	0:15.93	522068
3	init.cpp	0:19.06	517080
4	net.cpp	0:14.36	428924
5	wallet/rpcwallet.cpp	0:11.53	357224
6	wallet/walletdb.cpp	0:10.16	356792
7	test/script_tests.cpp	0:12.73	342376
8	qt/walletmodel.cpp	0:08.79	333660
9	test/test_bitcoin.cpp	0:08.21	326100
10	wallet/rpcdump.cpp	0:09.19	322376

The contenders are the (mostly) same, although it is faster and uses significantly less memory in all cases. So we should add a recommendation to use clang on lower-memory systems.

laanwj · 2015-09-30T11:57:41Z

I'm not sure why it is this "bad". main and init have huge complex implementation files, but of script_tests you'd say it isn't that bad - just 1016 lines.

Possibly it has to do with embedding the test data as includes:

410344  test/data/script_invalid.json.h
530534  test/data/script_valid.json.h

theuni · 2015-09-30T17:53:46Z

@laanwj I suspect you're right about the test data. Maybe older gcc re-reads that each time it's needed and balloons in size as a result.

@TheBlueMatt Could you test with theuni@1a107da and see if that helps? That moves the test data into cpps instead.
Edit: Whoops. Here's one that actually works: theuni@4b69de1

TheBlueMatt · 2015-10-01T00:16:59Z

Hmm, the VPS has since been upgraded to have more memory (yay pruning means I get more memory and less disk space...) and doesnt kill script_tests anymore :( What may have been happening is the host may have been retroactively killing processes after some other process had casued the VM to hit some high memory bar. In any case, it is just Debian Jessie, so gcc 4.9.2-10.

TheBlueMatt · 2015-10-01T00:18:00Z

Still, main max resident of 1.2GB will probably not build on most VPSes today, but sadly thats a harder one to fix, and probably a very long-term thing.

gmaxwell · 2015-10-01T00:20:05Z

Rather than recommending clang (which would send more hosts into a less tested and potentially consensus inconsistent config) we could recommend compiler flags that reduce memory usage.

unsystemizer · 2015-10-01T04:22:18Z

I ran a comparison test yesterday measuring the peak memory consumption of the Bash shell session in which I was working. gcc - 835 MB, clang - 823 MB (32-bit Raspbian OS). Not a big saving. (I don't know how @laanwj measured RAM consumption, otherwise I would have used his approach).

People low on RAM could use binary builds if they don't have enough RAM to build, or build without tests. Not everyone should be able to build from source on their system (or, in most hosting environments, they can upsize their RAM, build, and then downsize?).

laanwj · 2015-10-01T07:32:30Z

@gmaxwell Possibly both - we're using clang for OSX builds already, so it's not that much of an unknown. I've switched to clang for my run-of-the-mill builds on Linux to save time.

@cfields The solution in https://stackoverflow.com/questions/5479691/is-there-any-standard-way-of-embedding-resources-into-linux-executable-image to use the .incbin directive is also quite nice - create a file blob.S:

    .global blob
    .global blob_size
    .section .rodata
blob:
    .incbin "blob.bin"
1:
blob_size:
    .int 1b - blob

which can just be compiled with gcc along with the rest of the source files. This avoids any kind of conversion.

@unsystemizer see https://gist.github.com/laanwj/108877a28ec03836568a

In general I recommend to use cross-compiling to build for resource-constrained systems. This a) avoids having a compiler toolchain on the target b) is much faster c) Bitcoin Core's depends system makes this easy.

jarret · 2016-01-25T04:33:06Z

After tripping up on this initially, I succeeded in applying Luke-jr's
workaround by giving the CXXFLAGS at configure time:

$ ./configure --disable-wallet CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768"

On a Raspberry Pi 2 building with a single core:

$ time make
(success)
real    159m41.895s
user    154m55.090s
sys     3m1.710s

The memory use seemed to stay below 500 MB throughout, but I didn't measure
that precisely.

Sjors · 2018-06-05T18:23:04Z

I'm looking at Armian builds, since I might as well just build the rest of the OS - and sync the chain - along with cross compiling bitcoind :-)

maflcko · 2018-06-21T19:21:24Z

Considering that clang uses a lot less memory compared to gcc, why don't we mention clang as an alternative in https://github.com/bitcoin/bitcoin/blob/000abbb6b07410357a928768d7d56465ba0d3bac/doc/build-unix.md#memory-requirements

Should I create a pull for this?

dufferzafar · 2018-09-01T17:19:30Z

@MarcoFalke:

why don't we mention clang as an alternative

@gmaxwell said above:

Rather than recommending clang (which would send more hosts into a less tested and potentially consensus inconsistent config) we could recommend compiler flags that reduce memory usage.

laanwj · 2018-09-01T17:41:38Z

well if clang, a compiler many, many people use, generates consensus incompatible code, bitcoin has a huge problem.

(to be clear, no, I don't think this is an issue worrying about here - let's focus on finding configurations that reduce memory usage)

dufferzafar · 2018-09-01T17:49:25Z

To be honest, I myself use clang (along with ccache) but that's only because all my work involves running in regtest, so I don't have to worry about consensus compatibility.

laanwj · 2018-09-01T17:51:56Z

I mean, it might have been a good argument in 2012, but at this point, many platforms are clang-only (MacOSX, FreeBSD, OpenBSD, at least). There's nothing that binds bitcoin to gcc specifically.

meeDamian · 2018-09-11T15:43:46Z

Just wanted to leave an update here, that building Bitcoin Core v0.16.2 is possible¹ on Raspberry Pi Zero W, which has 512MB RAM default of 100MB SWAP.

./autogen.sh
./configure CXXFLAGS="--param ggc-min-expand=1 --param ggc-min-heapsize=32768" --enable-cxx --without-gui --disable-shared --with-pic
--enable-upnp-default --disable-wallet --disable-tests
make
sudo make install

¹ It did take a very long time.

laanwj · 2018-09-11T16:20:38Z

@meeDamian hahaha nice, that's quite a thing, building with less than a GB RAM SWAP

practicalswift · 2019-06-27T08:38:27Z

As noted in this issue we are currently using significantly more compile-time memory than we technically have to.

I plan to address this and you can see steps being taken in #16273 and #16278 (the latter is a subset of the former limited to tests only).

People who care about compile-time memory usage might be interested in reviewing those PR:s :-)

practicalswift · 2019-06-27T09:39:06Z

These are the 25 worst offenders in terms of compile-time memory usage:

#	File	Max memory usage (RSS)
1	`test/test_test_bitcoin-script_tests.o`	1416 MB
2	`wallet/libbitcoin_wallet_a-wallet.o`	1102 MB
3	`wallet/libbitcoin_wallet_a-rpcwallet.o`	1059 MB
4	`libbitcoin_server_a-validation.o`	953 MB
5	`test/test_test_bitcoin-util_tests.o`	905 MB
6	`wallet/test/test_test_bitcoin-wallet_tests.o`	905 MB
7	`wallet/test/test_test_bitcoin-coinselector_tests.o`	897 MB
8	`libbitcoin_server_a-init.o`	892 MB
9	`qt/test/qt_test_test_bitcoin_qt-wallettests.o`	890 MB
10	`wallet/test/test_test_bitcoin-ismine_tests.o`	874 MB
11	`qt/test/qt_test_test_bitcoin_qt-addressbooktests.o`	863 MB
12	`libbitcoin_server_a-net_processing.o`	857 MB
13	`wallet/test/test_test_bitcoin-psbt_wallet_tests.o`	837 MB
14	`test/test_test_bitcoin-transaction_tests.o`	813 MB
15	`qt/qt_libbitcoinqt_a-sendcoinsdialog.o`	809 MB
16	`wallet/libbitcoin_wallet_a-rpcdump.o`	800 MB
17	`test/test_test_bitcoin-rpc_tests.o`	792 MB
18	`libbitcoin_server_a-validationinterface.o`	770 MB
19	`test/test_bitcoin`	752 MB
20	`rpc/libbitcoin_server_a-blockchain.o`	745 MB
21	`qt/qt_libbitcoinqt_a-rpcconsole.o`	741 MB
22	`test/test_test_bitcoin-sighash_tests.o`	739 MB
23	`rpc/libbitcoin_server_a-rawtransaction.o`	721 MB
24	`qt/qt_libbitcoinqt_a-walletmodel.o`	721 MB
25	`wallet/libbitcoin_wallet_a-walletdb.o`	720 MB
...	...	...
449	`support/libbitcoin_util_a-cleanse.o`	24 MB
450	`crypto/sha256_sse4.cpp`	24 MB
451	`compat/libbitcoin_util_a-glibc_sanity.o`	24 MB
452	`script/script_error.cpp`	23 MB
453	`script/libbitcoin_consensus_a-script_error.o`	23 MB
454	`compat/libbitcoin_util_a-strnlen.o`	18 MB

Measured using gcc (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0.

practicalswift · 2019-06-29T14:21:22Z

People interested in enabling builds on low memory machines might be interested in reviewing and testing #16312:

tests: Reduce compile-time memory usage by 0.5 GB when compiling script_tests.cpp (from 1.4 GB to 0.9 GB)

practicalswift · 2019-08-10T06:59:58Z

Chasing {concept,approach} ACK/NACKs for #16331 which tackles this issue.

Let me know if this issue is no longer relevant :-)

laanwj · 2019-08-12T09:51:08Z

It doesn't seem that proposed solutions to this attract much, or any attention. Most notably, no one building on 1GB menory systems seems to be helping testing. So my conclusion would be it's either not relevant anymore or very very low priority, and will only become less so as systems with so little menory become rarer. Sorry for the wasted effort :/

…

On Sat, Aug 10, 2019, 09:00 practicalswift ***@***.***> wrote: Chasing concept ACK or NACKs for #16331 <#16331> which tackles this issue. Let me know if this issue is no longer relevant :-) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#6658?email_source=notifications&email_token=AAA65NSZXEEQVXG6PXRHVK3QDZRPLA5CNFSM4BPNRUDKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4AHRHA#issuecomment-520124572>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAA65NVIG2C2TOST5EDXBCTQDZRPLANCNFSM4BPNRUDA> .

meeDamian · 2019-08-12T10:02:32Z

I'm not sure if I'm not missing something here, but bitcoind was already building on RBP zero (512MB ram with just 100MB swap). Plus, I've just built v0.18.1 on RBP3 (1GB ram, no swap) a few days ago without problems even before applying any PRs. It does take its time, however 😅. So I for once, am not sure what else/how to test?

Sjors · 2019-08-12T11:45:27Z

@meeDamian does the new PR work without those extra config flags?

maflcko · 2019-08-12T12:59:53Z

It is a bit concerning that the unit test have to be disabled to compile on low memory machines

meeDamian · 2019-08-13T13:19:54Z

On a fresh Raspbian on RBP3, running (with or without --enable-lowmem flag):

sudo swapoff --all

sudo apt install git build-essential libtool autotools-dev automake pkg-config bsdmainutils python3 libssl-dev libevent-dev libboost-system-dev libboost-filesystem-dev libboost-chrono-dev libboost-test-dev libboost-thread-dev libminiupnpc-dev libzmq3-dev

git clone  -b fix-absurd-memory-usage-when-compiling-script_build  https://github.com/practicalswift/bitcoin.git bitcoin-swift
cd bitcoin-swift/

./contrib/install_db4.sh $(pwd)
export BDB_PREFIX=$(pwd)/db4
./autogen.sh

./configure BDB_LIBS="-L${BDB_PREFIX}/lib -ldb_cxx-4.8" BDB_CFLAGS="-I${BDB_PREFIX}/include" --enable-lowmem --without-gui

Crashes with:

…
checking for upnpDiscover in -lminiupnpc... yes
checking miniupnpc/miniupnpc.h usability... yes
checking miniupnpc/miniupnpc.h presence... yes
checking for miniupnpc/miniupnpc.h... yes
checking for upnpDiscover in -lminiupnpc... (cached) yes
checking miniupnpc/upnpcommands.h usability... yes
checking miniupnpc/upnpcommands.h presence... yes
checking for miniupnpc/upnpcommands.h... yes
checking for upnpDiscover in -lminiupnpc... (cached) yes
checking miniupnpc/upnperrors.h usability... yes
checking miniupnpc/upnperrors.h presence... yes
checking for miniupnpc/upnperrors.h... yes
checking for upnpDiscover in -lminiupnpc... (cached) yes
checking whether to build Bitcoin Core GUI... no (Qt5)
checking for boostlib >= 1.47.0 (104700)... yes
checking whether the Boost::System library is available... yes
configure: error: Could not find a version of the library!

Running the same on bitcoin v0.18.1 does not crash.

meeDamian · 2019-08-13T13:24:37Z

Actually… bitcoin master also crashes the same way, so likely unrelated to this. Happy to try again if someone advises how to fix/proceed.

Sjors · 2019-08-13T15:24:13Z

@meeDamian did you try with practicalswift/low-mem? #16331

meeDamian · 2019-08-13T16:54:48Z

I'm afraid practicalswift/low-mem crashes the same way as master, and fix-absurd-memory-usage-when-compiling-script_build. Any idea what library version might be missing there, or how to fix it?

fa7789f doc: Mention other ways to conserve memory on compilation (MarcoFalke) Pull request description: Related to: * Building requires >1GB memory #6658 ACKs for top commit: instagibbs: ACK fa7789f kristapsk: ACK fa7789f Tree-SHA512: 828593de9cfa9f9027c6c8e97abe95e3fad9f2ff50e6512808a8950de4a1f9ea901e724edfb96d7119224c3e38a136e60cb798b34ef682c585a7951e02124a3a

maflcko · 2019-10-12T13:27:52Z

The docs are updated and mention that clang can be used, which uses less than 1 GB of memory. Closing for now

https://github.com/bitcoin/bitcoin/blob/561a7d30478b82f5d46dcf0f16e864a9608004f4/doc/build-unix.md#memory-requirements

laanwj closed this as completed Sep 18, 2015

laanwj reopened this Sep 18, 2015

laanwj added the Build system label Sep 18, 2015

jarret mentioned this issue Jan 27, 2016

doc: add example for building with constrained resources #7423

Merged

bitcoin deleted a comment from Rosswestwood Jun 11, 2018

maflcko mentioned this issue May 21, 2019

test: move-only: Split large tests into smaller compile units #16069

Closed

maflcko mentioned this issue Jun 6, 2019

refactor: Remove unused includes #16129

Merged

practicalswift mentioned this issue Jun 29, 2019

tests: Reduce memory usage by 0.5 GB when compiling script_tests.cpp #16312

Closed

practicalswift mentioned this issue Jul 3, 2019

build: Add configure option --enable-lowmem to allow for building in low-memory environments #16331

Closed

maflcko mentioned this issue Aug 12, 2019

doc: Mention other ways to conserve memory on compilation #16586

Merged

maflcko closed this as completed Oct 12, 2019

bitcoin locked as resolved and limited conversation to collaborators Dec 16, 2021

Building requires >1GB memory #6658

Building requires >1GB memory #6658

Comments

TheBlueMatt commented Sep 10, 2015

theuni commented Sep 11, 2015

TheBlueMatt commented Sep 11, 2015

sipa commented Sep 11, 2015 via email

luke-jr commented Sep 11, 2015

TheBlueMatt commented Sep 11, 2015

TheBlueMatt commented Sep 11, 2015

theuni commented Sep 11, 2015

luke-jr commented Sep 11, 2015

TheBlueMatt commented Sep 11, 2015

unsystemizer commented Sep 11, 2015

jgarzik commented Sep 11, 2015

gmaxwell commented Sep 11, 2015

laanwj commented Sep 18, 2015

laanwj commented Sep 18, 2015

TheBlueMatt commented Sep 18, 2015

laanwj commented Sep 29, 2015

laanwj commented Sep 30, 2015

laanwj commented Sep 30, 2015

theuni commented Sep 30, 2015

TheBlueMatt commented Oct 1, 2015

TheBlueMatt commented Oct 1, 2015

gmaxwell commented Oct 1, 2015

unsystemizer commented Oct 1, 2015

laanwj commented Oct 1, 2015

jarret commented Jan 25, 2016

Sjors commented Jun 5, 2018

maflcko commented Jun 21, 2018

dufferzafar commented Sep 1, 2018

laanwj commented Sep 1, 2018 • edited Loading

dufferzafar commented Sep 1, 2018

laanwj commented Sep 1, 2018

meeDamian commented Sep 11, 2018

laanwj commented Sep 11, 2018

practicalswift commented Jun 27, 2019 • edited Loading

practicalswift commented Jun 27, 2019 • edited Loading

practicalswift commented Jun 29, 2019

practicalswift commented Aug 10, 2019 • edited Loading

laanwj commented Aug 12, 2019 via email

meeDamian commented Aug 12, 2019

Sjors commented Aug 12, 2019

maflcko commented Aug 12, 2019

meeDamian commented Aug 13, 2019

meeDamian commented Aug 13, 2019

Sjors commented Aug 13, 2019

meeDamian commented Aug 13, 2019 • edited Loading

maflcko commented Oct 12, 2019

laanwj commented Sep 1, 2018 •

edited

Loading

practicalswift commented Jun 27, 2019 •

edited

Loading

practicalswift commented Jun 27, 2019 •

edited

Loading

practicalswift commented Aug 10, 2019 •

edited

Loading

meeDamian commented Aug 13, 2019 •

edited

Loading