update 3rd party libraries, part2 #3707

georgeliao · 2024-09-30T16:23:08Z

Original repo
1. vcpkg, updated to 2024.09.30.

vcpkg:
1. grpc, 1.52.1-> 1.60.0, ongoing.

There are a lot hassles in grpc update:, so it is worth to talk about it. grpc has two layers of patches in the multipass use case, the first layer is the commits on the top of the standard grpc release.

Layer 1:
Orignally, there are 5 commits on the top of 1.52.1 release tag, that is the branch. They are around two things.

The first 3 commits are about adding the grpc_ssl_server_certificate_request_type options to the client side and disable pem_root_certs check along the way, see this section in the spec for details.
The 4th and 5th commits are simply just using getsockname to get local server address instead of getpeername to get the remote address.
Besides the orginal 5 commits, the 6th commit added and it is about removing newly added config_.pem_root_certs == nullptr checks based on the workflow first 3 commits setup, meaning that config_.pem_root_certs is supposed to be nullptr because we do not want to verify the server certificate. Because of that, the SslCredentialsOptions::pem_root_certs field which is the trust store of server certificates is set nulltptr when we create a channel on the client side.

Layer 2:
The second layer patches is the new patches applied and the changes made (the diff between the vcpkg tempate file and multipass cutom file) in the vcpkg grpc portfile.cmake file. The goal of the custom portfile.cmake is to cut off the unneeded library build and cut down the library size, which will reduce the build time of grpc, multipass and the binary size of them.

snap:

1. sshfs, will be done it in a different PR.

Besides these patches, there are some api calls changes on multipass side. Those are made for some grpc' changes in this duration which broke the backward compatibility. The particular commit is this one. There is also another github issue which has relevant discussions about this. This change affects grpc version 1.57.0 and onwards. The fix we adopt is the one mentioned in the github issue, where they call grpc.insecure_channel(srv_endpoint, options=(('grpc.default_authority', 'localhost'),)) on python client to overwrite the default authority, the C equivalent to this would be changing grpc::CreateChannel calls to grpc::CreateCustomChannel calls with channel arguments, see below code snippet for details

    grpc::ChannelArguments channel_args;
    channel_args.SetString(GRPC_ARG_DEFAULT_AUTHORITY, "localhost");

    // Use CreateCustomChannel to apply ChannelArguments
    auto channel = grpc::CreateCustomChannel(
        server_address, grpc::InsecureChannelCredentials(), channel_args);

Be aware that this change affects all types of address resolvers, that probably is fine in the use cases of multipass.

codecov · 2024-10-07T09:18:56Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 88.94%. Comparing base (f01004e) to head (114fe3f).

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #3707    /-   ##
=======================================
  Coverage   88.94%   88.94%           
=======================================
  Files         256      256           
  Lines       14584    14587     3     
=======================================
  Hits        12972    12975     3     
  Misses       1612     1612

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

georgeliao · 2024-10-09T09:56:38Z

vcpkg.json

@@ -1,5  1,5 @@
 {
-  "builtin-baseline": "ca7b1b15f548c25c766360593a2c732d56ed0133",
+  "builtin-baseline": "c82f74667287d3dc386bce81e44964370c91a289",


builtin-baseline field defines the versions of the dependencies of our vcpkg packages. In this particular case, grpc needs this update to get his dependency versions right because of his own update. The value of builtin-baseline field is simply the commit hash of the vcpkg repo.

georgeliao · 2024-11-06T17:14:17Z

Hi @Saviq @townsend2010

Sorry to bother you guys. Recently, multipass team has been dealing with rebasing the patches (code commits on the top of the release tag) of grpc after updating grpc to newer version. It has been a painful process due to the size of the patches and conflicts with the grpc refactors and changes. This really got us thinking what is the exact reason to keep these patches. Why multipass has a unique use case from the regular ones?

By looking at commit messages and the code, my superficial understanding is that. Multipass wants to support skipping verifying server certificate. Because of that, the grpc_ssl_server_certificate_request_type enum is added and all the corresponding changes are made. But why exactly do we need that is still not clear to me, meaning why does multipass uniquely need this skip the server certificate verification option.

Any information would be appreciated in case you guys know about it, thanks a lot.

Saviq · 2024-11-06T20:25:40Z

why does multipass uniquely need this skip the server certificate verification option

Basically to support SSH-style authorization. Both sides generate their encryption keys in isolation, without a CA between them. All communication is encrypted, and the key fingerprints are what determines whether they want to talk to one other on a higher level, not whether the certificates are valid or not.

georgeliao · 2024-11-12T10:05:49Z

Basically to support SSH-style authorization. Both sides generate their encryption keys in isolation, without a CA between them. All communication is encrypted, and the key fingerprints are what determines whether they want to talk to one other on a higher level, not whether the certificates are valid or not.

Thanks for the reply. But why exactly does multipass have to use the SSH-style authorization? The standard TLS/mTLS always requires the server certificate and verifies it. As a result, grpc api only provides GRPC_SSL_REQUEST_SERVER_CERTIFICATE_AND_VERIFY but not GRPC_SSL_REQUEST_SERVER_CERTIFICATE_BUT_DONT_VERIFY and it seems to make sense. The orthodox way based on the documentation of doing TLS/mTLS grpc is to fill in the pem_root_certs field of grpc::SslCredentialsOptions object on the client side, and later it will be used to verify the server cert and key pair internally.

My wondering here is whether this orthodox way has some inadequacies we are not aware of or it somehow does not fit in the multipass use case. Any info will be appreciated, thanks.

3rd-party/vcpkg-ports/grpc/vcpkg-cmake-wrapper.cmake

townsend2010 · 2024-11-18T15:29:33Z

My wondering here is whether this orthodox way has some inadequacies we are not aware of or it somehow does not fit in the multipass use case. Any info will be appreciated, thanks.

Hey @georgeliao,

As @Saviq alluded to, it was done this way to avoid having to use a CA and generate valid SSL certificates. In the early days when this was implemented, we found it to be easier to just modify gRPC and carry the patches than to deal with a CA. If you all want to deal with a CA and generate valid SSL certs in order to use what gRPC already supports, then that's a technical decision the Multipass team will need to make. Maybe it's easier now than in 2018. 🤷‍♂️ Care will also need to be taken with the whole authorization plumbing, but I think that should still work.

Saviq · 2024-11-18T15:54:41Z

This was the only way to have clients other than the first/default one to get access to the daemon simply by providing a shared passphrase. If you go for the CA approach, you have to generate a key pair and a certificate signing request and ask someone with access to the CA (admin) to sign your CSR and provide you back with the signed certificate. There's also no path to accepting the first client like we do today on Linux and macOS, you'd need to somehow get through the above dance in the installation phase.

Using a passphrase is just a more user-friendly way to achieving the same level of security, is all (one missing bit in the current scheme is managing a list of accepted daemon certificates, .ssh/known_hosts-style).

georgeliao · 2024-11-19T09:52:25Z

@townsend2010
Thanks a lot for the information. Now we understand the context much better.

…me mismatch.

…ld time and lib size.

…, which is non-intrusiveserver address resolver from intrusive change to

Co-authored-by: Ricardo Abreu <6698114 [email protected]> Signed-off-by: George Liao <[email protected]>

georgeliao marked this pull request as draft September 30, 2024 16:23

georgeliao mentioned this pull request Sep 30, 2024

Update 3rd party libs, part1 #3616

Merged

georgeliao force-pushed the update_3rd_lib_version branch from ca31a3a to 25a88f3 Compare October 2, 2024 10:49

georgeliao force-pushed the update_3rd_lib_part2 branch from f7603a6 to 6481118 Compare October 2, 2024 13:39

georgeliao force-pushed the update_3rd_lib_version branch from b254d90 to 2a93369 Compare October 2, 2024 15:53

georgeliao force-pushed the update_3rd_lib_part2 branch from 6481118 to f39584d Compare October 3, 2024 08:45

georgeliao force-pushed the update_3rd_lib_version branch from 2a93369 to d79d2c9 Compare October 3, 2024 09:36

georgeliao force-pushed the update_3rd_lib_part2 branch 2 times, most recently from a45865d to 5bb9771 Compare October 3, 2024 09:50

Base automatically changed from update_3rd_lib_version to main October 3, 2024 13:50

georgeliao force-pushed the update_3rd_lib_part2 branch 2 times, most recently from 84e7715 to 1766a7b Compare October 4, 2024 08:46

ricab added this to the 1.15.0 milestone Oct 7, 2024

georgeliao commented Oct 9, 2024

View reviewed changes

ricab reviewed Nov 15, 2024

View reviewed changes

3rd-party/vcpkg-ports/grpc/vcpkg-cmake-wrapper.cmake Outdated Show resolved Hide resolved

georgeliao marked this pull request as ready for review November 18, 2024 11:34

georgeliao marked this pull request as draft November 18, 2024 11:35

ricab modified the milestones: 1.15.0, 1.15.1 Nov 20, 2024

georgeliao added 2 commits December 4, 2024 10:21

[3rd-party] added vcpkg and grpc update

37ca716

[3rd-party][grpc] move the commit hash of grpc quick fix on server na…

623169e

…me mismatch.

georgeliao and others added 5 commits December 4, 2024 10:21

[3rd party][grpc] remove unneeded plugins build.

7b3be5c

[vcpkg][grpc] added remove unwanted libs patch to reduce the grpc bui…

a5c55b6

…ld time and lib size.

[3rd-party] updated the submodule_info.md file.

2c8ace7

[grpc][3rd lib] changed the fix of the grpc::CreateCustomChannel call…

2fc7432

…, which is non-intrusiveserver address resolver from intrusive change to

Update 3rd-party/vcpkg-ports/grpc/vcpkg-cmake-wrapper.cmake

114fe3f

Co-authored-by: Ricardo Abreu <6698114 [email protected]> Signed-off-by: George Liao <[email protected]>

georgeliao force-pushed the update_3rd_lib_part2 branch from 816e265 to 114fe3f Compare December 4, 2024 09:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

update 3rd party libraries, part2 #3707

update 3rd party libraries, part2 #3707

georgeliao commented Sep 30, 2024 •

edited

Loading

codecov bot commented Oct 7, 2024 •

edited

Loading

georgeliao Oct 9, 2024

georgeliao commented Nov 6, 2024 •

edited

Loading

Saviq commented Nov 6, 2024

georgeliao commented Nov 12, 2024 •

edited

Loading

townsend2010 commented Nov 18, 2024

Saviq commented Nov 18, 2024

georgeliao commented Nov 19, 2024

update 3rd party libraries, part2 #3707

Are you sure you want to change the base?

update 3rd party libraries, part2 #3707

Conversation

georgeliao commented Sep 30, 2024 • edited Loading

codecov bot commented Oct 7, 2024 • edited Loading

Codecov Report

georgeliao Oct 9, 2024

Choose a reason for hiding this comment

georgeliao commented Nov 6, 2024 • edited Loading

Saviq commented Nov 6, 2024

georgeliao commented Nov 12, 2024 • edited Loading

townsend2010 commented Nov 18, 2024

Saviq commented Nov 18, 2024

georgeliao commented Nov 19, 2024

georgeliao commented Sep 30, 2024 •

edited

Loading

codecov bot commented Oct 7, 2024 •

edited

Loading

georgeliao commented Nov 6, 2024 •

edited

Loading

georgeliao commented Nov 12, 2024 •

edited

Loading