Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

unreproducible builds from go_normal_opts.mk #1623

Closed
bmwiedemann opened this issue Aug 21, 2023 · 4 comments · Fixed by #2699
Closed

unreproducible builds from go_normal_opts.mk #1623

bmwiedemann opened this issue Aug 21, 2023 · 4 comments · Fixed by #2699
Milestone
1.4.0

Comments

@bmwiedemann
Copy link
Contributor

While working on reproducible builds for openSUSE, I found that
our apptainer varies between builds in its build-id from mlocal/frags/go_normal_opts.mk use of urandom. Dropping that line, makes builds reproducible.

36f05c8 does not have much details on how this relates to stripping of symbol, so maybe you can share some thoughts.

Version of Apptainer

1.2.2

Expected behavior

It should be possible to build identical binaries twice.

Actual behavior

mlocal/frags/go_normal_opts.mk adds randomness into binaries

Steps to reproduce this behavior

How can others reproduce this issue/problem?

build twice

What OS/distro are you running

openSUSE-Tumbleweed 20230819

How did you install Apptainer

N/A
@DrDaveD
Copy link
Contributor

DrDaveD commented Aug 21, 2023

That commit was inherited from the https://github.com/apptainer/singularity repository. If you look at the same commit there it leads to apptainer/singularity#2202 which has a little more context but not much. It does so happen that a commit that references that PR (shown at the end of that PR) has a comment that is a little more helpful:

Remove `-P release-stripped` from the `.spec.in`. PR apptainer#2202 adds
a Build ID which should make stripping unneeded.

Maybe instead of adding a random number it should use a hash of the source tarball. How does that sound?

@bmwiedemann
Copy link
Contributor Author

If you use the source-tarball-hash, you would give the same ID to all different binaries.

It seems, that the tools already auto-assign deterministic build-ids based on input hashes:

for section in .note.go.buildid .note.gnu.build-id ; do
   objdump -s -j $section RPMS/usr/libexec/apptainer/bin/starter
done

RPMS/usr/libexec/apptainer/bin/starter:     file format elf64-x86-64

Contents of section .note.go.buildid:
 03d4 04000000 53000000 04000000 476f0000  ....S.......Go..
 03e4 456d7853 63755a67 73485043 784d3169  EmxScuZgsHPCxM1i
 03f4 6d7a3731 2f6a6355 75583435 59397851  mz71/jcUuX45Y9xQ
 0404 4279374c 4c39746c 612f4d65 4f746757  By7LL9tla/MeOtgW
 0414 6e46546d 534e5a34 4f36486f 714a2f44  nFTmSNZ4O6HoqJ/D
 0424 33345754 36673647 7242695a 4936734f  34WT6g6GrBiZI6sO
 0434 79453400                             yE4.            

RPMS/usr/libexec/apptainer/bin/starter:     file format elf64-x86-64

Contents of section .note.gnu.build-id:
 0390 04000000 14000000 03000000 474e5500  ............GNU.
 03a0 dfc15b49 8ba2ccde 9e73a2ca 5b42a423  ..[I.....s..[B.#
 03b0 53e6f495                             S...

So I"d just drop go_normal_opts.mk.

@DrDaveD DrDaveD added this to the 1.2.3 milestone Aug 23, 2023
@DrDaveD
Copy link
Contributor

DrDaveD commented Aug 23, 2023

Please make a PR to see if it works.

bmwiedemann added a commit to bmwiedemann/apptainer that referenced this issue Aug 24, 2023
@bmwiedemann
Stop using random build-ids
a6a5f4f 
to use the default deterministic build-ids instead

Fixes: apptainer#1623

This patch was done while working on reproducible builds for openSUSE.

Signed-off-by: Bernhard M. Wiedemann <[email protected]>
@bmwiedemann bmwiedemann mentioned this issue Aug 24, 2023
Closed
bmwiedemann added a commit to bmwiedemann/apptainer that referenced this issue Aug 24, 2023
@bmwiedemann
Stop using random build-ids
3f92718 
to use the default deterministic build-ids instead

Fixes: apptainer#1623

This patch was done while working on reproducible builds for openSUSE.

Signed-off-by: Bernhard M. Wiedemann <[email protected]>
@DrDaveD DrDaveD modified the milestones: 1.2.3, 1.2.4 Sep 11, 2023
bmwiedemann added a commit to bmwiedemann/apptainer that referenced this issue Dec 5, 2023
@bmwiedemann
Stop using random build-ids
e89aa32 
to use the default deterministic build-ids instead

Fixes: apptainer#1623

This patch was done while working on reproducible builds for openSUSE.

Signed-off-by: Bernhard M. Wiedemann <[email protected]>
bmwiedemann added a commit to bmwiedemann/apptainer that referenced this issue Dec 5, 2023
@bmwiedemann
Stop using random build-ids
cd4f5e7 
to use the default deterministic build-ids instead

Fixes: apptainer#1623

This patch was done while working on reproducible builds for openSUSE.

Signed-off-by: Bernhard M. Wiedemann <[email protected]>
bmwiedemann added a commit to bmwiedemann/apptainer that referenced this issue Dec 5, 2023
@bmwiedemann
Stop using random build-ids
d7e1635 
to use the default deterministic build-ids instead

Fixes: apptainer#1623

This patch was done while working on reproducible builds for openSUSE.

Signed-off-by: Bernhard M. Wiedemann <[email protected]>
bmwiedemann added a commit to bmwiedemann/apptainer that referenced this issue Dec 5, 2023
@bmwiedemann
Stop using random build-ids
0fd0d92 
to use the default deterministic build-ids instead

Fixes: apptainer#1623

This patch was done while working on reproducible builds for openSUSE.

Signed-off-by: Bernhard M. Wiedemann <[email protected]>
e4t added a commit to e4t/apptainer that referenced this issue Jan 6, 2025
@e4t
Derive the GNU build ID from the Go one
4e9efb8 
This is achieved using the linker option `-B gobuildid` See:
https://pkg.go.dev/cmd/link.
This way binaries will get unique build IDs and the build will be
reproducible as binaries built from identical sources will have
the same build ID.
This fixes issue apptainer#1623.

Signed-off-by: Egbert Eich <[email protected]>
@e4t e4t mentioned this issue Jan 6, 2025
Merged
6 tasks
@e4t
Copy link
Contributor

e4t commented Jan 6, 2025

I"ve replaced the randomly generated GNU build ID by one derived from the Go build ID in our build:

GO_LDFLAGS += -ldflags="-B gobuildid"

See: https://pkg.go.dev/cmd/link
The build succeeds and debuginfo packages are generated. I"ve just created a pull request.

@DrDaveD DrDaveD modified the milestones: 1.2.4, 1.4.0 Jan 7, 2025
e4t added a commit to e4t/apptainer that referenced this issue Jan 8, 2025
@e4t
Derive the GNU build ID from the Go one
e2207ca 
This is achieved using the linker option `-B gobuildid` See:
https://pkg.go.dev/cmd/link.
This way binaries will get unique build IDs and the build will be
reproducible as binaries built from identical sources will have
the same build ID.
This fixes issue apptainer#1623.

Signed-off-by: Egbert Eich <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
1.4.0
3 participants
@bmwiedemann @DrDaveD @e4t