Report forwarded
to debian-bugs-dist@lists.debian.org, reproducible-builds@lists.alioth.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Wed, 06 Jun 2018 18:00:05 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
New Bug report received and forwarded. Copy sent to reproducible-builds@lists.alioth.debian.org, Debian Install System Team <debian-boot@lists.debian.org>.
(Wed, 06 Jun 2018 18:00:05 GMT) (full text, mbox, link).
Subject: debian-installer: Please make the generated images reproducible
Date: Wed, 06 Jun 2018 18:58:24 +0100
Package: debian-installer
Severity: wishlist
Tags: patch
User: reproducible-builds@lists.alioth.debian.org
Usertags: toolchain
X-Debbugs-Cc: reproducible-builds@lists.alioth.debian.org
Hi,
Whilst working on the Reproducible Builds effort [0], we noticed
that debian-installer's images were not reproducible.
I have created merge request #3 [1] on salsa with a patch series
to address this.
[0] https://reproducible-builds.org/
[1] https://salsa.debian.org/installer-team/debian-installer/merge_requests/3
Best wishes,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org / chris-lamb.co.uk
`-
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Sun, 10 Jun 2018 19:24:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Vagrant Cascadian <vagrant@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Sun, 10 Jun 2018 19:24:03 GMT) (full text, mbox, link).
Added blocking bug(s) of 900918: 900409 and 900410
Request was from Vagrant Cascadian <vagrant@debian.org>
to 900918-submit@bugs.debian.org.
(Sun, 10 Jun 2018 19:24:03 GMT) (full text, mbox, link).
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Mon, 11 Jun 2018 21:18:05 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Mon, 11 Jun 2018 21:18:05 GMT) (full text, mbox, link).
To: 900409@bugs.debian.org, 900410@bugs.debian.org,
Anibal Monsalve Salazar <anibal@debian.org>
Cc: 900918@bugs.debian.org
Subject: Re: debian-installer: Please make the generated images reproducible
Date: Mon, 11 Jun 2018 22:15:13 +0100
Dear Anibal,
> mtools: please make the output reproducible (uninitialised memory)
> mtools: please make the output reproducible (directory timestamps)
I'd dearly love to ensure there are no blockers to making the Debian
Installer images to be reproducible well in advance of the «buster»
release. :)
May I gently prod you to review and hopefully apply these patches?
Best wishes,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org / chris-lamb.co.uk
`-
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Thu, 12 Jul 2018 10:03:11 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Thu, 12 Jul 2018 10:03:11 GMT) (full text, mbox, link).
Subject: Re: debian-installer: Please make the generated images reproducible
Date: Thu, 12 Jul 2018 10:52:09 +0100
On 11th June 2018, Chris Lamb wrote:
> > mtools: please make the output reproducible (uninitialised memory)
> > mtools: please make the output reproducible (directory timestamps)
>
> I'd dearly love to ensure there are no blockers to making the Debian
> Installer images to be reproducible well in advance of the «buster»
> release. :)
>
> May I gently prod you to review and hopefully apply these patches?
Another gentle ping on this?
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org / chris-lamb.co.uk
`-
Removed blocking bug(s) of 900918: 900409
Request was from Chris Lamb <lamby@debian.org>
to control@bugs.debian.org.
(Wed, 25 Jul 2018 09:42:08 GMT) (full text, mbox, link).
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Sat, 12 Jan 2019 18:48:04 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Sat, 12 Jan 2019 18:48:04 GMT) (full text, mbox, link).
Subject: Re: debian-installer: Please make the generated images reproducible
Date: Sat, 12 Jan 2019 16:45:25 +0000
Chris Lamb wrote:
> Whilst working on the Reproducible Builds effort [0], we noticed
> that debian-installer's images were not reproducible.
>
> I have created merge request #3 [1] on salsa with a patch series
> to address this.
>
> [0] https://reproducible-builds.org/
> [1] https://salsa.debian.org/installer-team/debian-installer/merge_requests/3
Since working on this in June 2018 I have:
* NMU'd src:mtools to incorporate the required reproducibility
patches in 4.0.18-2.1.
* Subsequently salvaged the mtools package and packaged the latest
upstream version (4.0.23-1).
* Rebased the corresponding merge request since the recent secure
boot / EFI changes by Steve.
I would dearly love to have reproducible installer images for
buster and indeed this would be a good thing to provide and
offer, let alone announce in the release notes, etc.
Now that mtools has been sorted out, could this bug & corresponding
merge request get a closer look? Thanks in advance.
Best wishes,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org / chris-lamb.co.uk
`-
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Fri, 18 Jan 2019 23:33:05 GMT) (full text, mbox, link).
Acknowledgement sent
to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Fri, 18 Jan 2019 23:33:05 GMT) (full text, mbox, link).
Hello Chris,
First off: Thanks a lot for your work, and for your patience.
Chris Lamb <lamby@debian.org> (2019-01-12):
> I would dearly love to have reproducible installer images for buster
> and indeed this would be a good thing to provide and offer, let alone
> announce in the release notes, etc.
>
> Now that mtools has been sorted out, could this bug & corresponding
> merge request get a closer look? Thanks in advance.
There's a new alpha coming up and I've just spent some time cleaning up
things in debian-installer.git (debhelper compat, standards-version,
etc.) right after the upload; I've almost merged your patch series as
is, but I'd like to raise a few points first. I'm fine with amending the
patches myself, just wanted to check with you first…
* 0c19a29645a6a3136f3a51da5d5cf1cfcec5fdfb mentions file system
ordering only, but that's just the sort part AFAIUI; you're also
resetting some timestamps with the find | xargs touch loop aren't
you? I'd be happy to see this documented in the commit message as
well.
[ basically the clamp_mtimes function in build/Makefile in later
commits. ]
* 71391967763708055a65ed68999db8f4ea6fc6e6 sets “deb00001” as the FAT
volume ID; have you checked with people like debian-cd@ whether
another constant might make more sense? (cc-edd)
[ also: “determinstic” in commit message. ]
[ post-scriptum: ok, now I see we already had that in place
elsewhere; keeping the cc anyway. ]
* 7c533fa721c3ae89ca81d1336b5928a80ed0d531 thanks for the clarity, much
appreciated.
[ also: “becuase” in commit message. ]
* c35b8688696b1b4563a45d0feeabc3a0c0f2eccb “determinstic” in commit
message.
* ea1a896181daa3b82c5a62ae31839b457a0dbe0b modifies BUILD_DATE, adding
a few characters (“:SS”); that ends up in various help screens
(hopefully not an issue) but also on some files: version.info,
disk.lbl, etc.
Also, coming to think about BUILD_DATE, it's defined with ?= in
build/config/common but we also set it from debian/rules… I guess
I'll have to check what happens.
* f181b4fe90b9030f515c7e6129239b96131b3926 oh more en_GB, yay.
Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Sat, 19 Jan 2019 12:39:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Sat, 19 Jan 2019 12:39:03 GMT) (full text, mbox, link).
Subject: Re: Bug#900918: debian-installer: Please make the generated images
reproducible
Date: Sat, 19 Jan 2019 12:34:20 +0000
[Keeping Kibi in CC as requested off-list, dropping mtools@p.d.o]
Dear Kibi,
First, thank you so much for your review. I believe I have resolve
all of your issues on the corresponding merge request, as well as
rebasing it against the current master:
https://salsa.debian.org/installer-team/debian-installer/merge_requests/3
§
> * 0c19a29645a6a3136f3a51da5d5cf1cfcec5fdfb mentions file system
> ordering only, but that's just the sort part
I've split this commit and it is much clearer now.
> * 71391967763708055a65ed68999db8f4ea6fc6e6 sets “deb00001” as the FAT
> volume ID; have you checked with people like debian-cd@ whether
> another constant might make more sense?
Clarified where this comes from in the commmit and in the code
itself.
> * 7c533fa721c3ae89ca81d1336b5928a80ed0d531 thanks for the clarity
> [ also: “becuase” in commit message. ]
Fixed.
> * c35b8688696b1b4563a45d0feeabc3a0c0f2eccb “determinstic”
Fixed.
> * ea1a896181daa3b82c5a62ae31839b457a0dbe0b modifies BUILD_DATE, adding
> a few characters (“:SS”); that ends up in various help screens
In my tests, this did not break any visual text wrapping, etc.
> * f181b4fe90b9030f515c7e6129239b96131b3926 oh more en_GB, yay.
Fixed (use 'results' instead).
§
Thank you again for your review. Let me know if you would require
any further modifications before merging. :-)
Best wishes,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org 🍥 chris-lamb.co.uk
`-
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Sat, 19 Jan 2019 21:09:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Sat, 19 Jan 2019 21:09:03 GMT) (full text, mbox, link).
Hi Chris,
Chris Lamb <lamby@debian.org> (2019-01-19):
> > * 0c19a29645a6a3136f3a51da5d5cf1cfcec5fdfb mentions file system
> > ordering only, but that's just the sort part
>
> I've split this commit and it is much clearer now.
Perfect, thanks.
> > * 71391967763708055a65ed68999db8f4ea6fc6e6 sets “deb00001” as the
> > FAT volume ID; have you checked with people like debian-cd@
> > whether another constant might make more sense?
>
> Clarified where this comes from in the commmit and in the code
> itself.
Thanks.
> > * ea1a896181daa3b82c5a62ae31839b457a0dbe0b modifies BUILD_DATE,
> > adding a few characters (“:SS”); that ends up in various help
> > screens
>
> In my tests, this did not break any visual text wrapping, etc.
Perfect.
> > * f181b4fe90b9030f515c7e6129239b96131b3926 oh more en_GB, yay.
>
> Fixed (use 'results' instead).
(I was genuinely happy to learn about the en_GB version; always
struggling to remember how to spell it in French vs. (American) English,
was happy to discover the British English version. :))
> §
>
> Thank you again for your review. Let me know if you would require any
> further modifications before merging. :-)
That's looking good but I'm seeing new warnings because of gzip's being
unhappy about the GZIP environment variable. This seems to have been
deprecated in gzip 1.8:
gzip: warning: GZIP environment variable is deprecated; use an alias or script
I'm attaching the patch I've deployed locally, and it seems to me that
[1] might want an update.
1. https://wiki.debian.org/ReproducibleBuilds/TimestampsInGzipHeaders
An other solution would be to use tar | gzip instead of passing -I to
tar, but I thought I'd learn about that option in the process. :)
Back to d-i results: I'm seeing small differences in the generated
-images tarball, that I'll try to investigate. I'll probably push the
series anyway, as these patches are helping anyway. :)
Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Sat, 19 Jan 2019 22:09:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Sat, 19 Jan 2019 22:09:03 GMT) (full text, mbox, link).
Cyril Brulebois <kibi@debian.org> (2019-01-19):
> Back to d-i results: I'm seeing small differences in the generated
> -images tarball, that I'll try to investigate. I'll probably push the
> series anyway, as these patches are helping anyway. :)
Here are the differences I'm seeing by building multiple times in the
same sid devel chroot:
installer-amd64/20190119/images/cdrom/gtk/initrd.gz
installer-amd64/20190119/images/hd-media/boot.img.gz
installer-amd64/20190119/images/hd-media/gtk/initrd.gz
installer-amd64/20190119/images/MD5SUMS
installer-amd64/20190119/images/netboot/gtk/debian-installer/amd64/initrd.gz
installer-amd64/20190119/images/netboot/gtk/mini.iso
installer-amd64/20190119/images/netboot/gtk/netboot.tar.gz
installer-amd64/20190119/images/netboot/mini.iso
installer-amd64/20190119/images/SHA256SUMS
*SUMS are no-brainers due to changes in other files.
All gtk files have fontconfig-related cache/uuid changes…
Excluding those, remaining files are:
installer-amd64/20190119/images/hd-media/boot.img.gz
installer-amd64/20190119/images/netboot/mini.iso
The boot.img includes two initrds: initrd.gz and initrdg.gz; the latter
is the one with graphical support, which explains why it's also hit by
the fontconfig cache/uuid problem.
The mini.iso has apparently other changes… I'm attaching the diffoscope
output. Could this be because of missing tweaks to the xorriso calls in
build/config/x86.cfg? (In which case build/config/arm.cfg might need an
update as well.) Checking all MINIISO occurrences might also make sense
I suppose?
Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Sat, 19 Jan 2019 23:12:02 GMT) (full text, mbox, link).
Acknowledgement sent
to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Sat, 19 Jan 2019 23:12:03 GMT) (full text, mbox, link).
Control: tag -1 pending
Cyril Brulebois <kibi@debian.org> (2019-01-19):
> The mini.iso has apparently other changes… I'm attaching the diffoscope
> output. Could this be because of missing tweaks to the xorriso calls in
> build/config/x86.cfg? (In which case build/config/arm.cfg might need an
> update as well.) Checking all MINIISO occurrences might also make sense
> I suppose?
FWIW, dropping all fontconfig-related bits (see attached patch) makes it
possible to confirm only mini.iso (regular and gtk ones) are showing
differences now:
installer-amd64/20190119/images/MD5SUMS
installer-amd64/20190119/images/netboot/gtk/mini.iso
installer-amd64/20190119/images/netboot/mini.iso
installer-amd64/20190119/images/SHA256SUMS
I had purged the pigz package during my experiments, just to be sure it
wouldn't interfere:
build/Makefile:gzip = $(shell which pigz >/dev/null 2>&1 && echo "pigz -n -T" || echo "gzip -n")
(Including lintian runtime, using pigz on a 8-way machine cuts real time
from 8m8s to 4m23s.)
Checking what happens, and forgetting about the aforementioned mini.iso
images temporarily, it seems successive builds with pigz lead to the
same results. But those aren't the same as the results generated with
gzip. I don't suppose this is going to be a particularly huge problem
though?
Anyway, summarizing: likely more work to be done on the xorriso front,
(on the debian-installer side) for the mini.iso images produced for
netbooting; and fontconfig needs to get fixed in some way at some point
(but I know it's been a long run as well…); but all the rest looks good.
Pushing once I have updated the changelog; marking as pending, thanks!
Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Subject: Bug #900918 in debian-installer marked as pending
Date: Sat, 19 Jan 2019 23:47:22 +0000
Control: tag -1 pending
Hello,
Bug #900918 in debian-installer reported by you has been fixed in the
Git repository and is awaiting an upload. You can see the commit
message below and you can check the diff of the fix at:
https://salsa.debian.org/installer-team/debian-installer/commit/f153c29eb65324cf9b76f7dc903c56b7f3db90f8
------------------------------------------------------------------------
Document Chris Lamb's patches for reproducible builds (Closes: #900918).
------------------------------------------------------------------------
(this message was generated automatically)
--
Greetings
https://bugs.debian.org/900918
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Mon, 21 Jan 2019 12:27:06 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Mon, 21 Jan 2019 12:27:06 GMT) (full text, mbox, link).
Subject: Re: Bug#900918: debian-installer: Please make the generated images
reproducible
Date: Mon, 21 Jan 2019 12:25:31 +0000
Dear Cyril,
Thank you for your review and timely merge.
> That's looking good but I'm seeing new warnings because of gzip's being
> unhappy about the GZIP environment variable.
Interesting. However when you say "new" warnings I don't believe my
patch set actually added/changed this; indeed, it has not changed
since:
https://salsa.debian.org/lamby/debian-installer/commit/28b863340cc5fd73fbaac85a3fb89e72e842b15c
… so I'm just checking what you are requesting to be done here.
§
> All gtk files have fontconfig-related cache/uuid changes…
[…]
> FWIW, dropping all fontconfig-related bits (see attached patch)
This is #864082 in src:fontconfig — I've been playing whack-a-mole
with fontconfig over the past 18 months or so and this was a
fairly recent regression.
Are you planning on applying this patch to debian-installer.git?
Naturally, I would prefer if #864082 was applied and in buster, or
otherwise closed again.
§
> The mini.iso has apparently other changes… I'm attaching the diffoscope
> output. Could this be because of missing tweaks to the xorriso calls in
> build/config/x86.cfg?
Possibly. Let me try and reproduce and reload all of that into my
brain and get back to you on this; IIRC there was some pushback
against making it change behaviour only on SOURFE_DATE_EPOCH being
present so — as you imply — a command-line change might be required.
§
> (Including lintian runtime, using pigz on a 8-way machine cuts real time
> from 8m8s to 4m23s.)
TIL pigz; thanks.
> Checking what happens [it] seems successive builds with pigz lead to the
> same results. But those aren't the same as the results generated with
> gzip. I don't suppose this is going to be a particularly huge problem
> though?
Alas, it is a problem in thatfrom the outside nobody will know
whether one built with pigz or gzip and thus it will be unclear how
to reproduce the bit-for-bit identical binary. In other words, there
is currently no ".buildinfo" equivalent here that specifies "I used
Arch^Wpigz, btw" and got a SHA of $foo.
One easy solution would be if that, if SOURCE_DATE_EPOCH is present,
then we force the use of one tool. Although that would regrettably
mean the lowest common denominator (ie. gzip) which probably isn't
the ideal due to the aforementioned performance gain for using pigz.
Alternatively, we could make pigz a strict build requirement but
that sounds a little antisocial.
Perhaps we need to record the environment after all; again, I will
reload all of this into my head anyway due to mini.iso (^) so this
will be top-of-stack again.
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org 🍥 chris-lamb.co.uk
`-
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Tue, 22 Jan 2019 06:12:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Cyril Brulebois <kibi@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Tue, 22 Jan 2019 06:12:03 GMT) (full text, mbox, link).
Hi Chris,
Chris Lamb <lamby@debian.org> (2019-01-21):
> > That's looking good but I'm seeing new warnings because of gzip's
> > being unhappy about the GZIP environment variable.
>
> Interesting. However when you say "new" warnings I don't believe my
> patch set actually added/changed this; indeed, it has not changed
> since:
>
> https://salsa.debian.org/lamby/debian-installer/commit/28b863340cc5fd73fbaac85a3fb89e72e842b15c
I think I got stuff mixed up (hard to tell now as I deleted the 10+ GB
of build logs and artefacts I accumulated while testing various things
with your patch series and a couple of extra patches on top of it),
maybe because those warnings were moving around by a line or two in the
logs… Maybe that's why I thought they were new.
Putting the confusion aside, I pushed that to avoid the warnings:
https://salsa.debian.org/installer-team/debian-installer/commit/0b025d7f485ecf7ed8969068f98e49d3141d77fd
> … so I'm just checking what you are requesting to be done here.
All in all, I'm not requesting you to do anything more on the d-i level
(there's been quite a lot of work already); just a mere suggestion to
maybe include an example of direct “gzip -n” specification for tar
invokations on the wiki, so that others don't have to look around how to
do that.
> > All gtk files have fontconfig-related cache/uuid changes…
> […]
> > FWIW, dropping all fontconfig-related bits (see attached patch)
>
> This is #864082 in src:fontconfig — I've been playing whack-a-mole
> with fontconfig over the past 18 months or so and this was a fairly
> recent regression.
>
> Are you planning on applying this patch to debian-installer.git?
> Naturally, I would prefer if #864082 was applied and in buster, or
> otherwise closed again.
I don't plan to apply that patch; it was only meaning to serve as a
basis to double check there were no other reproducibility issues in the
gtk images.
> > The mini.iso has apparently other changes… I'm attaching the
> > diffoscope output. Could this be because of missing tweaks to the
> > xorriso calls in build/config/x86.cfg?
>
> Possibly. Let me try and reproduce and reload all of that into my
> brain and get back to you on this; IIRC there was some pushback
> against making it change behaviour only on SOURFE_DATE_EPOCH being
> present so — as you imply — a command-line change might be required.
OK. I think I'd prefer closing this very bug report whenever what's in
git gets uploaded (it's been a long run and thread already), and see
other issues discussed in a fresh bug report; would that work for you?
> > (Including lintian runtime, using pigz on a 8-way machine cuts real
> > time from 8m8s to 4m23s.)
Now less than 4m, after an extra commit:
https://salsa.debian.org/installer-team/debian-installer/commit/234058c033ef05c9aab3ced7a7c8cd4917daff9b
> TIL pigz; thanks.
>
> > Checking what happens [it] seems successive builds with pigz lead to
> > the same results. But those aren't the same as the results generated
> > with gzip. I don't suppose this is going to be a particularly huge
> > problem though?
>
> Alas, it is a problem in thatfrom the outside nobody will know whether
> one built with pigz or gzip and thus it will be unclear how to
> reproduce the bit-for-bit identical binary. In other words, there is
> currently no ".buildinfo" equivalent here that specifies "I used
> Arch^Wpigz, btw" and got a SHA of $foo.
Ah right, pigz isn't in Build-Depends, so it's not included in the
.buildinfo file…
> One easy solution would be if that, if SOURCE_DATE_EPOCH is present,
> then we force the use of one tool. Although that would regrettably
> mean the lowest common denominator (ie. gzip) which probably isn't
> the ideal due to the aforementioned performance gain for using pigz.
Right, for a development build (meaning without the generation of the
big debian-installer-images tarball), my gut feeling without specific
clocking data is that half the build time is spent waiting for gzip
to finish its work on a single core. Reproducibility is very nice but
I would definitely hate to lose this huge speed-up.
> Alternatively, we could make pigz a strict build requirement but
> that sounds a little antisocial.
Right.
> Perhaps we need to record the environment after all; again, I will
> reload all of this into my head anyway due to mini.iso (^) so this
> will be top-of-stack again.
Not having looked into the format or tool(s) generating it, would it
be reasonable to have an opt-in mechanism so that a package could
declare “please record the state of this package that might or might not
be installed”. So that pigz could be registered in some way, akin to:
“not-really-in-build-depends-but-oh-look-it-was-present-at-version-X”
Thanks again for your valuable feedback.
Cheers,
--
Cyril Brulebois (kibi@debian.org) <https://debamax.com/>
D-I release manager -- Release team member -- Freelance Consultant
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Tue, 22 Jan 2019 09:57:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Philipp Kern <pkern@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Tue, 22 Jan 2019 09:57:03 GMT) (full text, mbox, link).
Subject: Re: Bug#900918: debian-installer: Please make the generated images
reproducible
Date: Tue, 22 Jan 2019 10:53:44 +0100
On 1/22/2019 7:08 AM, Cyril Brulebois wrote:
>> Alternatively, we could make pigz a strict build requirement but
>> that sounds a little antisocial.
> Right.
In what way do you consider this antisocial and what's speaking against
doing that? If it's about CPU time, then maybe it should obey the
parallelization setting of the build process. But then it's already a
build process and if you want that to not be detrimental to your desktop
performance, you nice it. Apart from that thing I really struggle to
find something "antisocial" in that build-dependency.
Kind regards and thanks
Philipp Kern
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Tue, 22 Jan 2019 12:03:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Tue, 22 Jan 2019 12:03:03 GMT) (full text, mbox, link).
Subject: Re: Bug#900918: debian-installer: Please make the generated images
reproducible
Date: Tue, 22 Jan 2019 12:00:06 +0000
Philipp,
> > > Alternatively, we could make pigz a strict build requirement but
> > > that sounds a little antisocial.
> >
> > Right.
>
> In what way do you consider this antisocial and what's speaking against
> doing that?
I was using this word in a somewhat romantic sense; fewer
dependencies are "better" in some aesthetic or poetic sense.
Please do not read too much into it. :)
(I had also slightly assumed that pigz was only on some archs, but
I am clearly mistaken there...)
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org 🍥 chris-lamb.co.uk
`-
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Tue, 22 Jan 2019 12:03:04 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Tue, 22 Jan 2019 12:03:04 GMT) (full text, mbox, link).
Subject: Re: Bug#900918: debian-installer: Please make the generated images
reproducible
Date: Tue, 22 Jan 2019 12:01:04 +0000
Hi Cyril,
> I think I'd prefer closing this very bug report whenever what's in
> git gets uploaded
That would definitely work for me. Indeed, it would probably help
separate discussions/issues correctly as any followups are likely =
to be much more specific.
> half the build time is spent waiting for gzip to finish its work on
> a single core. Reproducibility is very nice but I would definitely
> hate to lose this huge speed-up.
Absolutely, hence my regret at suggesting a fallback to gzip. *nod*
> > Perhaps we need to record the environment after all
[…]
> Ah right, pigz isn't in Build-Depends, so it's not included in the
> .buildinfo file…
"the .buildinfo"? As I understood it we are not recording any
explicit debian-installer build attestation document that records
the environment at the moment. Again, will reload this all into my
head soon and come up with a suggested solution if required in
another bug report.
Thanks again. :)
Best wishes,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org 🍥 chris-lamb.co.uk
`-
Information forwarded
to debian-bugs-dist@lists.debian.org, Debian Install System Team <debian-boot@lists.debian.org>: Bug#900918; Package debian-installer.
(Tue, 22 Jan 2019 13:54:03 GMT) (full text, mbox, link).
Acknowledgement sent
to Ben Hutchings <ben@decadent.org.uk>:
Extra info received and forwarded to list. Copy sent to Debian Install System Team <debian-boot@lists.debian.org>.
(Tue, 22 Jan 2019 13:54:03 GMT) (full text, mbox, link).
On Tue, 2019-01-22 at 10:53 +0100, Philipp Kern wrote:
> On 1/22/2019 7:08 AM, Cyril Brulebois wrote:
> > > Alternatively, we could make pigz a strict build requirement but
> > > that sounds a little antisocial.
> > Right.
> In what way do you consider this antisocial and what's speaking
> against
> doing that? If it's about CPU time, then maybe it should obey the
> parallelization setting of the build process.
[...]
The general way to communicate this is by wrapping the build in
"taskset", and pigz will surely follow that.
Ben.
--
Ben Hutchings
Power corrupts. Absolute power is kind of neat. - John Lehman
Subject: Bug#900918: fixed in debian-installer 20190410
Date: Wed, 10 Apr 2019 20:33:55 +0000
Source: debian-installer
Source-Version: 20190410
We believe that the bug you reported is fixed in the latest version of
debian-installer, which is due to be installed in the Debian FTP archive.
A summary of the changes between this version and the previous one is
attached.
Thank you for reporting the bug, which will now be closed. If you
have further comments please address them to 900918@bugs.debian.org,
and the maintainer will reopen the bug report if appropriate.
Debian distribution maintenance software
pp.
Cyril Brulebois <kibi@debian.org> (supplier of updated debian-installer package)
(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing ftpmaster@ftp-master.debian.org)
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Format: 1.8
Date: Wed, 10 Apr 2019 20:55:32 +0200
Source: debian-installer
Binary: debian-installer
Architecture: source
Version: 20190410
Distribution: unstable
Urgency: medium
Maintainer: Debian Install System Team <debian-boot@lists.debian.org>
Changed-By: Cyril Brulebois <kibi@debian.org>
Description:
debian-installer - Debian Installer documentation
Closes: 820911900918920353920631920676921951
Changes:
debian-installer (20190410) unstable; urgency=medium
.
[ Cyril Brulebois ]
* Bump debhelper build-dep and compat from 7 to 11. Finally!
* Fix “build-depends-on-1-revision build-depends u-boot-rockchip”,
thanks to lintian.
* Set dpkg source format explicitly as both dpkg-source and lintian
are being excessively noisy.
* Drop source.lintian-overrides: the only override was about the
e2fsprogs build-dependency when it was still “Essential: yes”; as
anticipated it's now only “Priority: required”.
* Bump Standards-Version from 3.9.5 to 4.3.0; no changes required, with
the usual exception regarding network access.
* Review and merge Chris Lamb's work making d-i build reproducibly
(Closes: #900918), with many thanks! Remaining issues seem related to
fontconfig's cache and xorriso calls.
* gen-tarball: specify the complete gzip command through a tar option
instead of using the now-deprecated GZIP environment variable.
* gen-tarball: use pigz if available, reducing build time.
* Bump Linux kernel ABI from 4.19.0-1 to 4.19.0-4
* Update translation-status for the release.
.
[ Chris Lamb ]
* Set BUILD_DATE based on SOURCE_DATE_EPOCH, rather than the other way
around.
* Ensure the GRUB memdisk images do not vary on filesystem ordering.
* Ensure the GRUB memdisk images do not vary on filesystem timestamps.
* Ensure that generated EFI images have a determinstic FAT volume ID.
* When generating CPIO archives (initramfs, etc), ensure they are
reproducible.
* build/Makefile: Add a clamp_mtimes utility to avoid DRY violations.
* 'clamp' the mtimes of various files before they are added to archives
or media.
* Specify --invariant to mkfs.fat(8) to ensure deterministic filesystem
timestamps.
* Increase the BUILD_DATE resolution to include the seconds component.
* Clamp mtimes of the final build results in the destination directory.
* Ensure that running the 'release' target does not alter the mtimes
under dest/.
* debian/control: Document that #900409 and #900410 are required for a
reproducible build.
* Ensure build is reproducible regardless of the user's
umask(2). (Closes: #920631)
* Ensure build is reproducible regardless of the underlying filesystem
ordering. (Closes: #920676)
.
[ John Paul Adrian Glaubitz ]
* config: Set DEBIAN_RELEASE to unstable for all architectures
in Debian Ports. (Closes: #920353)
* pkg-lists: Add debian-ports-archive-keyring-udeb package for all
architectures in Debian Ports for cdrom and netboot images.
.
[ Holger Wansing ]
* Add dark theme shortcut (Closes: #820911)
.
[ Samuel Thibault ]
* Rework dark theme shortcut.
* Introduce dark menu theme.
* Fix loading background image on netboot grub.
.
[ Ben Hutchings ]
* armel: Delete unused package list for ixp4xx (Closes: #921951)
.
[ Vagrant Cascadian ]
* [armhf] Add images for Novena.
* Update versioned Build-Depends on u-boot-imx.
* [armhf] Add images for BananaPiM2Berry. Thanks to Lucas Nussbaum.
* Update versioned Build-Depends on u-boot-sunxi.
.
[ Steve McIntyre ]
* Switch to using buster artwork for the splash screen
Checksums-Sha1:
5a159c724cc20717d81828774bc1930983deb8d3 3724 debian-installer_20190410.dsc
7c5cff20f36c1bdd729aa21ad6e01cffd9337c29 1390608 debian-installer_20190410.tar.gz
404016f403c419425e8a2597d1c18f5d3ae8aac0 10977 debian-installer_20190410_source.buildinfo
Checksums-Sha256:
70caf5d7354a4168036eb7069c7a89613c8f4e712d5aaef0f03dd7f0dfaa2e31 3724 debian-installer_20190410.dsc
3fdfc093aa16fa2badd47d3c26f97d8a7f687528dee0e3b04a601f39b8e7c615 1390608 debian-installer_20190410.tar.gz
d493bb82137c793ddc170819375e1b20b4f8755c22ce3d56c6988f4a6415df9a 10977 debian-installer_20190410_source.buildinfo
Files:
1e50c39a7a13837568dcbb12b2a6a0ae 3724 devel optional debian-installer_20190410.dsc
12dfff029664dff64dac3f695055cf7e 1390608 devel optional debian-installer_20190410.tar.gz
4fdb0a438924e87bbf2292ca09ce2938 10977 devel optional debian-installer_20190410_source.buildinfo
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEtg6/KYRFPHDXTPR4/5FK8MKzVSAFAlyuSL0ACgkQ/5FK8MKz
VSAsKQ//UFlDyFDKVv7/WlFDfXqBu/1PS93puO8wKUyFE1yNXcnSeb6DG2NE+CRG
zJokPYR+CQwJqoWoQ9PIEGO3vI0pramWRig5oj0mHBOGbcq5vX2Ls/LEnIMTP4b9
z5FYsMiWwlmw7ANeqQStdAsK26K/DjePVgvDJ219ud0UJf01DbSkJdhSdyktDRRs
+NSXPjoWQJKcokUzKQY9XeG/KuJbgJrrl6CThTPWnELnW6mcIsVIbKG25Cq4QKkB
gIpMjWRyaStVnHq3yRjFVZnjOWiEOtjr4AmHo1zkEQ+b4IOkOlu61duhTXnPLhsm
TqkGs9BawQG8zLAv90GfUug+neSh9rKU9r/ooqCz5vIPCtWIGWCqtSVG/OKJLOmD
iN0SXp/P+841pP5yr0LYukcmbnn5gyCNJCAMl0+WiitX2sGdGVLZJBoSn9oxH8QW
BHKrKnhrADd4H3E8Sf+gXN5Q53L9J7SLgn5RsDvBUEA7vsmG2IKHfP+MAq6A68/i
g+72gxkqIu0/Yp6fTcK1EiIdsp83KaM4ceS18M7QMoIa3PLtES5ZZpFujHzNWuEQ
6ElnNPZJRRm+ZJ8xS5GXNSy6Ap8U61T1Xizm/E/isli41uNrQYQvVlu3EmGKdQIO
f7a+tBkiWmjykbr+xEp/Z2FPTQaotvOt6XjLbS5J4oAmpYi+o3I=
=S4VB
-----END PGP SIGNATURE-----
Bug archived.
Request was from Debbugs Internal Request <owner@bugs.debian.org>
to internal_control@bugs.debian.org.
(Sun, 09 Jun 2019 07:27:01 GMT) (full text, mbox, link).
Debbugs is free software and licensed under the terms of the GNU General
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.