Report forwarded
to debian-bugs-dist@lists.debian.org, reproducible-builds@lists.alioth.debian.org, APT Development Team <deity@lists.debian.org>: Bug#863622; Package apt.
(Mon, 29 May 2017 11:27:06 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
New Bug report received and forwarded. Copy sent to reproducible-builds@lists.alioth.debian.org, APT Development Team <deity@lists.debian.org>.
(Mon, 29 May 2017 11:27:06 GMT) (full text, mbox, link).
Package: apt
Severity: wishlist
X-Debbugs-CC: reproducible-builds@lists.alioth.debian.org
Hi,
APT should (eventually) warn when installing packages that are not
reproducible.
Clearly, all the bits to make this work today are not in dak, APT, the
mirrors, etc. However, I thought it was best to experiment early with
the potential user interface.
This would ensure that we know exactly what data we need and we don't
make a big mistake and miss something.
To this end, I've attached a proof of concept patch. Example output:
$ apt install python-pywt-doc
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
python-pywt-doc
0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
Need to get 102 kB of archives.
After this operation, 978 kB of additional disk space will be used.
WARNING: The following packages are not reproducible!
python-pywt-doc
Install these packages anyway? [y/N]
$ echo $?
130
It takes an expected "--allow-unreproducible" argument, as well as an
"-o Debug::pkgAcquire::Reproducible=true" if you want to debug it. I
might play with it more at https://github.com/lamby/apt on the
reproducible-ui branch:
https://github.com/lamby/apt/tree/lamby/wip/reproducible-ui
Just to be clear, the patch is obviously an digusting hack and you
should not use it, hence the lack of a "patch" tag (!).
(We would also — later please! — need to agree on what "reproducible"
really means in terms of multiple builders.)
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org / chris-lamb.co.uk
`-
Hi Chris;
It's an interesting idea. My first thought is that it might be more
appropriate for apt-listbugs. At least as a user I'm more likely to be
worried about RC bugs than reproducibility, so it feels strange to warn
me about reproducibility by default, but not e.g. known security holes
and data loss.
not an apt maintainer,
d
Information forwarded
to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>: Bug#863622; Package apt.
(Mon, 29 May 2017 12:39:02 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>.
(Mon, 29 May 2017 12:39:02 GMT) (full text, mbox, link).
To: David Bremner <david@tethera.net>, 863622@bugs.debian.org
Subject: Re: Bug#863622: what about other (RC) bugs?
Date: Mon, 29 May 2017 13:36:07 +0100
Hi David,
First, thanks for your interest :)
> feels strange to warn me about reproducibility by default
Oh, who knows whether this would be on by default at this point *g* We
can bikeshed that much later Ifear — it would depend on a lot of currently
unknown factors (archive coverage, etc.).
> It's an interesting idea. My first thought is that it might be more
> appropriate for apt-listbugs.
I had actually not considered this approach. Still, as it stands I am
really trying to work out any kinks in the interface; where the code
actually lives (apt/dpkg/apt-listbugs) isn't really that big a thing.
Hope that helps and thanks again for your input.
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org / chris-lamb.co.uk
`-
Information forwarded
to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>: Bug#863622; Package apt.
(Mon, 29 May 2017 13:30:02 GMT) (full text, mbox, link).
Acknowledgement sent
to Julian Andres Klode <jak@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>.
(Mon, 29 May 2017 13:30:02 GMT) (full text, mbox, link).
To: Chris Lamb <lamby@debian.org>, 863622@bugs.debian.org
Subject: Re: Bug#863622: apt: warn when installing packages that are not
reproducible
Date: Mon, 29 May 2017 15:28:20 +0200
On Mon, May 29, 2017 at 12:24:29PM +0100, Chris Lamb wrote:
> Package: apt
> Severity: wishlist
> X-Debbugs-CC: reproducible-builds@lists.alioth.debian.org
>
> Hi,
>
> APT should (eventually) warn when installing packages that are not
> reproducible.
>
> Clearly, all the bits to make this work today are not in dak, APT, the
> mirrors, etc. However, I thought it was best to experiment early with
> the potential user interface.
>
> This would ensure that we know exactly what data we need and we don't
> make a big mistake and miss something.
>
> To this end, I've attached a proof of concept patch. Example output:
>
> $ apt install python-pywt-doc
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> The following NEW packages will be installed:
> python-pywt-doc
> 0 upgraded, 1 newly installed, 0 to remove and 4 not upgraded.
> Need to get 102 kB of archives.
> After this operation, 978 kB of additional disk space will be used.
> WARNING: The following packages are not reproducible!
> python-pywt-doc
> Install these packages anyway? [y/N]
>
> $ echo $?
> 130
Seems OK.
>
>
> It takes an expected "--allow-unreproducible" argument, as well as an
> "-o Debug::pkgAcquire::Reproducible=true" if you want to debug it. I
> might play with it more at https://github.com/lamby/apt on the
> reproducible-ui branch:
>
> https://github.com/lamby/apt/tree/lamby/wip/reproducible-ui
>
> Just to be clear, the patch is obviously an digusting hack and you
> should not use it, hence the lack of a "patch" tag (!).
Now, we are missing some stuff:
- Checking if a package is reproducible by simply asking for
package name (and version maybe) is not really "safe". You
could be installing a completely different package that happens
to share those attributes. In essence, that's more like asking
a bug tracker. We'd need some hash or something to add to the
lookup (unless the information comes from the same repo).
- For integration in APT, we need:
* sources.list entries
* InRelease files
* Some new kind of index referenced in the InRelease files
(Reproducibility ?) that we can parse - in deb822 format,
with fields Package, Architecture, Version, and
Reproducible: yes. And well, some kind of ID field maybe,
I don't know. Could also be part of the Packages
file (in case of multiple repro sources, if any did not
build reproducible, we can still mark it as not reprod.).
Because as I said before, we don't want to download
anything we can't verify with GPG. We likely also want
to restrict GPG keys to not work for normal repositories,
not sure how that would work.
There's also a reason for requiring reproducibility info
to be signed: If it ever becomes the default (or people use
it as a default), you could MITM a repro server and respond
that all packages are not reproducible, preventing (automatic)
upgrades.
--
Debian Developer - deb.li/jak | jak-linux.org - free software dev
| Ubuntu Core Developer |
When replying, only quote what is necessary, and write each reply
directly below the part(s) it pertains to ('inline'). Thank you.
Information forwarded
to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>: Bug#863622; Package apt.
(Mon, 29 May 2017 13:39:04 GMT) (full text, mbox, link).
Acknowledgement sent
to Chris Lamb <lamby@debian.org>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>.
(Mon, 29 May 2017 13:39:04 GMT) (full text, mbox, link).
To: Julian Andres Klode <jak@debian.org>, 863622@bugs.debian.org
Subject: Re: Bug#863622: apt: warn when installing packages that are not
reproducible
Date: Mon, 29 May 2017 14:34:36 +0100
Julian Andres Klode wrote:
> We'd need some hash or something to add to the lookup (unless the
> information comes from the same repo).
Oh, absolutely :) In fact, my previous version of the patch did exactly
this, querying buildinfo.debian.net by hash of the file. This is actually
a lot cleaner in some respects than the current situation in that it
"solves" the issue you raised but we additionally don't have to work out
the source package name which is actually buggy in my proof-of-concept...
As it happens, and somewhat hilariously, the only hash I could find in
the Item class was MD5… *g*
> There's also a reason for requiring reproducibility info
> to be signed
Indeed.
Regards,
--
,''`.
: :' : Chris Lamb
`. `'` lamby@debian.org / chris-lamb.co.uk
`-
Information forwarded
to debian-bugs-dist@lists.debian.org, APT Development Team <deity@lists.debian.org>: Bug#863622; Package apt.
(Mon, 29 May 2017 15:18:02 GMT) (full text, mbox, link).
Acknowledgement sent
to David Kalnischkies <david@kalnischkies.de>:
Extra info received and forwarded to list. Copy sent to APT Development Team <deity@lists.debian.org>.
(Mon, 29 May 2017 15:18:02 GMT) (full text, mbox, link).
On Mon, May 29, 2017 at 12:24:29PM +0100, Chris Lamb wrote:
> Just to be clear, the patch is obviously an digusting hack and you
> should not use it, hence the lack of a "patch" tag (!).
Oh, no worries, we can manage. I mean, how hacky can it really be………
OH MY GOD, YOU ARE CALLING CURL!!111elf
(lets start with some implementation criticism even if that wasn't your
primary goal – Julian said most of it but lets recap for completeness as
I will reference it later)
As a cleanup I would suggest making the location of your JSON file an
apt repository (aka: add an Release file mentioning the uncompressed
and compressed versions of those files. Ensure the Release file has an
"Architectures" field mentioning all supported architectures so apt
isn't complaining about missing Packages files – as it will assume they
are missing as they are empty, not because the repository doesn't
support this architecture).
You can then add that repository in your sources.list and tell apt to
get it via acquire-additional-files.txt (See appstream, apt-file and
"apt-config dump Acquire::IndexTargets" for examples – that can also
deal with un/recompressing your file) That is good training as we will
need that configuration bit even if the file would eventually end up
alongside the others in the Debian repositories.
JSON is a problem in Debian context, but lets ignore that for the
moment. From a codepoint of view it will be most interesting to get
a mapping from downloaded file to the relevant stanza about a source
package. I would say hashes should work here given that they should be
the same, so if we can find a download source and a source claiming
reproducible for a package (version) with that hash we should be good
– that would also avoid the extra step of mapping to source package.
(mostly the end of implementation critic)
From a user point of view I am not sure what to think here. It is
probably going to be a hard sell to teach every repository owner to
really be honest about reproducibility if I can save a lot of support
time from users seeing scary warnings if I just declare all my packages
as reproducible all the time – so perhaps its actually a good idea to
never merge the data in the repository itself but have it from
"independent" sources (similar to what I described above) [which makes
the mapping all the more interesting code wise] showing all packages we
don't have good info for as "unknown reproducible state" and allowing
certain repositories (and packages) to be declared as good in
sources.list (and an apt_preferences lookalike).
It will also be interesting to figure out what "reproducible" means in
super laymen terms as I am not sure users will understand it. We have
trained people to understand "reproducible" in terms of bugs, but what
that means in the context of installing binary packages… installing
things is pretty reproducible, isn't it?
Best regards
David Kalnischkies
Debbugs is free software and licensed under the terms of the GNU General
Public License version 2. The current version can be obtained
from https://bugs.debian.org/debbugs-source/.