Skip to content

Gitian Build Guide

Jump to bottom
Edward Iskra edited this page Jul 29, 2020 · 7 revisions

Prepare a clean OS

  1. Set up a VM or native Ubuntu 18.04 (other version may work but we will focus on it in this guide).
    • Recommended VM: VMWare Player 15 (free for non-business purpose).
    • RAM: At least 4G. Recommended more than 8G. The more the better.
    • CPU cores: Recommended 2 cores. The more the better.
    • Disk space: 20G or more.
  2. Optionally: set up SSH for remote access. Ignore it if you don't need remote access.

Set up PGP identity (first time)

Expand

This step is only needed if you want to help Gitian build and sign the build result.

In this section we will create a new PGP identity for you. Please keep your private keys safely for future use. You don't need to do this twice.

This simplified guide is based on Ubuntu's GnuPrivacyGuardHowto

  1. Install GnuPG

    sudo apt install gnupg

  2. Generate a key pair for signing

    gpg --gen-key

  3. You should get outputs like below. In this example, the key id is 5014A7638E17E1CD4EFC4A7BE995133B7EFCE689 and the short id is E995133B7EFCE689.

    gpg: /home/ubuntu/.gnupg/trustdb.gpg: trustdb created
gpg: key E995133B7EFCE689 marked as ultimately trusted
gpg: directory '/home/ubuntu/.gnupg/openpgp-revocs.d' created
gpg: revocation certificate stored as '/home/ubuntu/.gnupg/openpgp-revocs.d/5014A7638E17E1CD4EFC4A7BE995133B7EFCE689.rev'
public and secret key created and signed.

pub   rsa3072 2019-04-13 [SC] [expires: 2021-04-12]
      5014A7638E17E1CD4EFC4A7BE995133B7EFCE689
uid                      h4x3rotab <[email protected]>
sub   rsa3072 2019-04-13 [E] [expires: 2021-04-12]

  4. Upload your public key to the public server so that people can download your public key by id. Replace E995133B7EFCE689 with your key id or short id.

    gpg --keyserver hkp://hkps.pool.sks-keyservers.net --send-keys  E995133B7EFCE689

    You can confirm upload success by searching on the name or email address used at http://pool.sks-keyservers.net/. If search by KeyID, you need to include "0x" as a prefix in the search box.

  5. Make a Pull Request to BTG's repo with your PGP key id added to the Gitian Keys file

Prepare Gitian workspace (first time)

Expand

This step is a simplified version of BTG Release Process. If you follow this guide, the basic setup should be:

  • Build root directory: $HOME/ (Usually /home/<your user name>/)

  1. Install dependencies

    sudo apt install python3 git curl

  2. Clone related repositories

    git clone https://github.com/BTCGPU/bitcoingold-detached-sigs.git
git clone https://github.com/devrandom/gitian-builder.git
git clone https://github.com/BTCGPU/BTCGPU.git

  3. Clone Gitian signature repo

    1. If you just want to build the binaries

      git clone https://github.com/BTCGPU/gitian.sigs.git

    2. Or, if you also want to help sign the build results

      1. Fork https://github.com/BTCGPU/gitian.sigs into your own account

      2. Clone your own fork instead of the upstream repo

        git clone [email protected]:<your name>/gitian.sigs.git

        Notice that we are cloning with SSH instead of HTTPS to make the rest build process smoother. If you don't have experience woking with git over SSH, please follow the two guides by GitHub carefully.

        Don't worry. It's easy and just an one-shot effort.

      3. Set up git if this is the first time you run git on your build environment:

        git config --global user.email "<[email protected]>"
git config --global user.name "<Your Name>"

        (Optional) Even better, tell git to sign your commits with your PGP key:

        git config --global user.signingkey <pgp key>

  4. Set up Gitian with default config

    ./BTCGPU/contrib/gitian-build.py -S <pgp signer> 0.17.3
    • -S: Tell the script to setup Gitian
    • <pgp signer>: The PGP key to use when sign the build (e.g. h4x3rotab or E995133B7EFCE689). Not used but still needed when setting up Gitian.
    • 0.17.3: The version to build. Not used but still needed in this case.
    • After a successful execution, you need to reboot to continue.

  5. Download the macOS SDK and put it into ./gitian-builder/inputs

    # Assume we are in $HOME
cd ./gitian-builder
mkdir inputs
cd inputs
wget https://bitcoincore.org/depends-sources/sdks/MacOSX10.11.sdk.tar.gz

    Validate the SHA256 checksum:

    sha256sum MacOSX10.11.sdk.tar.gz
# Should match: bec9d089ebf2e2dd59b1a811a38ec78ebd5da18cbbcd6ab39d1e59f64ac5033f

Prepare Gitian workspace (update only)

Update the local code & signature git repo:

cd ~/bitcoingold-detached-sigs
git pull
cd ~/BTCGPU
git pull

Run Gitian build

A full Gitian build consists of three passes performed by different roles. Below is a list explaining the outcome of each pass. Each pass relies on the previous pass finished and committed in the public repository.

Pass Participants Outcome
1 Everyone Final Linux binaries, intermediate Windows and macOS binaries without code-sign
2 Release engineers only Detached code-sign for producing the final Windows and macOS binaries
3 Everyone Final Windows and macOS binaries with code-sign; signed gitian build receipts

As shown in the table, you may want to participate 1 & 2 if you want to help the full gitian build. But if you just want to reproduce the build for Linux binaries, you may just want to run the first pass. In short, usually you need to participate different passes according to your purpose.

I. Build binaries without code-sign

./BTCGPU/contrib/gitian-build.py -u "$HOME/BTCGPU" -b -o lwm -j 16 -m 8096 <pgp signer> 0.17.3
  • -b: run gitian-build
  • -u $HOME/BTCGPU: uses the local git repo, instead of clone from github
  • -o lwm: builds for macOS (m for macOS, w for Windows and l for Linux. lwm for all)
  • -j 16: utilizes 16 cpu threads
  • -m 8096: max ram usage is 8096MB
  • <pgp signer>: sign by this PGP key (e.g. h4x3rotab or E995133B7EFCE689); can be anything if you don't want to sign it
  • 0.17.: the version to build

A successful build will produce binaries in ./bitcoin-binaries/<version> and commit signed Gitian build receipts in ./gitian.sig.

If you want to help with the Gitian release process, now you should submit your receipts to the official repo. You can check the commit of your receipts with git log:

# In ./gitian.sigs
git log
# The latest commit should look like "Add 0.17.3 unsigned sigs for <pgp signer>"

When confirmed, push the receipts to your own fork of gitian.sigs repo:

# In ./gitian.sigs
git push

Then open the repo in your browser and create a Pull Request to the official repo.

II. Produce detached code-sign signatures

Please omit this step because it is done by release engineers.

  1. Copy the following files to a separate directory: 
    bitcoin-gold-win-unsigned.tar.gz
bitcoin-gold-osx-unsigned.tar.gz
  2. Sign the Windows binaries:
    1. Untar bitcoin-gold-win-unsigned.tar.gz in ./win
    2. Copy codesign.key to ./win
    3. ./detached-sig-create.sh -key codesign.key
  3. Sig the macOS binaries:
    1. Transfer bitcoin-gold-osx-unsigned.tar.gz to a macOS computer
    2. Untar it in ./mac
    3. Replace tar by gtar in detached-sig-create.sh
    4. ./detached-sig-create.sh -s <key name>
  4. Untar and submit the detached sigs to the versioned branch, tagged by the version

III. Assemble code-signed binaries

./BTCGPU/contrib/gitian-build.py -u "$HOME/BTCGPU" -s -o lwm -j 16 -m 8096 <pgp signer> 0.17.3

This pass is relatively fast. You will also need to sign the build receipts with your pgp key. When it's done, it will add a commit to your gitian.sigs with the final signature. You will see the final binary and the macOS DMG file with code signature inside bitcoin-binary directory as follows:

~/gitian.sigs$ ls -lh ~/bitcoin-binaries/0.17.3/
total 1.3G
-rw-rw-r-- 1 ubuntu ubuntu 157M May  5 03:11 bitcoin-gold-0.17.3-aarch64-linux-gnu-debug.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu  27M May  5 03:09 bitcoin-gold-0.17.3-aarch64-linux-gnu.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu 156M May  5 03:06 bitcoin-gold-0.17.3-arm-linux-gnueabihf-debug.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu  24M May  5 03:05 bitcoin-gold-0.17.3-arm-linux-gnueabihf.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu  13M May  6 03:45 bitcoin-gold-0.17.3-osx.dmg
-rw-rw-r-- 1 ubuntu ubuntu 156M May  5 02:56 bitcoin-gold-0.17.3-i686-pc-linux-gnu-debug.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu  30M May  5 02:55 bitcoin-gold-0.17.3-i686-pc-linux-gnu.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu  21M May  5 05:10 bitcoin-gold-0.17.3-osx64.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu  13M May  5 05:10 bitcoin-gold-0.17.3-osx-unsigned.dmg
-rw-rw-r-- 1 ubuntu ubuntu 6.0M May  5 02:52 bitcoin-gold-0.17.3.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu 184M May  5 04:46 bitcoin-gold-0.17.3-win32-debug.zip
-rw-rw-r-- 1 ubuntu ubuntu  15M May  6 03:45 bitcoin-gold-0.17.3-win32-setup.exe
-rw-rw-r-- 1 ubuntu ubuntu  15M May  5 04:45 bitcoin-gold-0.17.3-win32-setup-unsigned.exe
-rw-rw-r-- 1 ubuntu ubuntu  27M May  5 04:45 bitcoin-gold-0.17.3-win32.zip
-rw-rw-r-- 1 ubuntu ubuntu 189M May  5 04:52 bitcoin-gold-0.17.3-win64-debug.zip
-rw-rw-r-- 1 ubuntu ubuntu  15M May  6 03:45 bitcoin-gold-0.17.3-win64-setup.exe
-rw-rw-r-- 1 ubuntu ubuntu  15M May  5 04:51 bitcoin-gold-0.17.3-win64-setup-unsigned.exe
-rw-rw-r-- 1 ubuntu ubuntu  27M May  5 04:51 bitcoin-gold-0.17.3-win64.zip
-rw-rw-r-- 1 ubuntu ubuntu 160M May  5 03:01 bitcoin-gold-0.17.3-x86_64-linux-gnu-debug.tar.gz
-rw-rw-r-- 1 ubuntu ubuntu  28M May  5 03:00 bitcoin-gold-0.17.3-x86_64-linux-gnu.tar.gz

There should be the updated gitian build receipts in the gitian.sig repo. As in step I, you should check the new git commit, push it to your own fork of gitian.sigs repo (git push), and create a pull request to the main repo.

Congratulations! Now you have completed the full gitian build process.

Troubleshooting

Utilities

  • View log while building
    • tail -f gitian-builder/var/install.log
    • tail -f gitian-builder/var/build.log
  • Check CPU usage
    • top
    • Or better: htop (sudo apt install htop)

Error: lxc_find_gateway_addresses

gitian: network.c: lxc_find_gateway_addresses: 2070 Failed to automatically find ipv4 gateway address from link interface "br0"

A reboot of the VM fixed the problem.

Error: "gpg failed to sign the data"

git "error: gpg failed to sign the data"

Try to set the gpg program for git manually:

git config --global gpg.program gpg

Error: "GPG pinentry timeout"

git "error: gpg: signing failed: No pinentry"

The signing process timed out waiting for you to enter your gpg pin. On many systems, setting the timeout to 0 actually uses a default timeout value. If you can't attend the build, you can increase the timeout as follows:

  • find the file pgp-agent.conf in your ~/.gnupg folder (or create it)
  • add the line: pinentry-timeout 86400 to give yourself 24 hours to respond

Troubleshooting on Debian

Thanks to @tolak for the contribution!

Error: missing dirmngr

failed to start the dirmngr

Install dirmngr with command apt install dirmngr

Error: missing script / RuntimeError

No such script: /usr/share/debootstrap/scripts/bionic bin/gbuild:23:in `system!': failed to run make-clean-vm --suite bionic --arch amd64 (RuntimeError)

Both two errors above can be solved by this way. Make sure lxc was installed (you may need to download lxc source code and compile by yourself) and "lxc-exectue --version" >= 2.1.1

Error: Gitian failed to find gateway addresses

gitian: start.c: lxc_spawn: 1637 Failed to find gateway addresses

Follow this link to config lxc: https://github.com/bitcoin-core/docs/blob/master/gitian-building/gitian-building-setup-gitian-debian.md#setting-up-debian-for-gitian-building

https://github.com/BTCGPU/BTCGPU.git

Clone this wiki locally