Mount static kmod as /sbin/modprobe #4007

vigh-m · 2024-05-30T01:54:43Z

Issue number:

Description of changes:
This change mounts the static kmod built in #3981 into a customer container.

Testing done:
Tested on the host-ctr:

[ssm-user@control]$ enter-admin-container
Confirming admin container is enabled...
Waiting for admin container to start........
Entering admin container
          Welcome to Bottlerocket's admin container!
    ╱╲
   ╱┄┄╲   This container provides access to the Bottlerocket host
   │▗▖│   filesystems (see /.bottlerocket/rootfs) and contains common
  ╱│  │╲  tools for inspection and troubleshooting.  It is based on
  │╰╮╭╯│  Amazon Linux 2, and most things are in the same places you
    ╹╹    would find them on an AL2 host.

To permit more intrusive troubleshooting, including actions that mutate the
running state of the Bottlerocket host, we provide a tool called "sheltie"
(`sudo sheltie`).  When run, this tool drops you into a root shell in the
Bottlerocket host's root filesystem.
[root@admin]# ldd /usr/sbin/modprobe
  not a dynamic executable

Tested on a kubernetes container:

root@my-pod:/# ldd /usr/sbin/modprobe
  not a dynamic executable
root@my-pod:/# ldd /sbin/modprobe
  not a dynamic executable
root@my-pod:/# ldd /usr/bin/kmod
ldd: /usr/bin/kmod: No such file or directory

Pod Spec used:

apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  containers:
  - name: my-container
    image: ubuntu:noble
    command: ["/bin/sh", "-c", "while true; do echo 'Running...'; sleep 5; done"]
    securityContext:
      capabilities:
        add: ["SYS_MODULE"]
    volumeMounts:
    - name: kernel-modules
      mountPath: /lib/modules/
      readOnly: true
  volumes:
  - name: kernel-modules
    hostPath:
      path: /lib/modules/

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

bcressey · 2024-05-30T02:08:42Z

Code changes LGTM, though I would split them into one for host-ctr and one for containerd.

Does it work for Cilium? For the example pod spec, can you load a module via modprobe?

yeazelm · 2024-05-30T02:17:11Z

The code looks good, it would be awesome to include a before and after on that pod test to show the change, even better showing it loading a module after the fix to prove this fixes the issue.

vigh-m · 2024-05-30T02:43:25Z

Yeah. It loads the modules like expected.

root@my-pod:/# ln -s /usr/sbin/modprobe /usr/bin/lsmod
root@my-pod:/# lsmod | grep tables
nf_tables             307200  0
nfnetlink              20480  2 nf_conntrack_netlink,nf_tables
root@my-pod:/# modprobe -r nf_tables
root@my-pod:/# lsmod | grep tables
root@my-pod:/# modprobe nf_tables
root@my-pod:/# lsmod | grep tables
nf_tables             307200  0
nfnetlink              20480  2 nf_conntrack_netlink,nf_tables

I had validated but didn't add it to the initial notes

vigh-m · 2024-05-30T02:49:07Z

Validated via SHA512 that it's the same modprobe

On admin container

[root@admin]# sha512sum /usr/bin/kmod
b2f5451a89fc2e21a971b23086380ea3c8916b5ad753c31812b4419992e6f63687da3b35723ec1d1545039c4e946e54a01ff2555420966bcceda6bb8d0569404  /usr/bin/kmod

On rootfs

[root@admin]# sha512sum /.bottlerocket/rootfs/bin/kmod
b2f5451a89fc2e21a971b23086380ea3c8916b5ad753c31812b4419992e6f63687da3b35723ec1d1545039c4e946e54a01ff2555420966bcceda6bb8d0569404  /.bottlerocket/rootfs/bin/kmod

On container

root@my-pod:/# sha512sum /usr/sbin/modprobe
b2f5451a89fc2e21a971b23086380ea3c8916b5ad753c31812b4419992e6f63687da3b35723ec1d1545039c4e946e54a01ff2555420966bcceda6bb8d0569404  /usr/sbin/modprobe

arnaldo2792 · 2024-05-30T03:28:45Z

For the validation, you could just have checked that the modprobe binary is coming from the host via /proc/self/mountinfo.

sources/host-ctr/cmd/host-ctr/main.go

arnaldo2792 · 2024-05-30T03:33:14Z

packages/containerd/containerd-cri-base-json

@@ -102,6 102,15 @@ oci-defaults = { version = "v1", helpers = ["oci_defaults"] }
 "mode=755",
 "size=65536k"
 ]
+ },
+ {
+ "destination": "/sbin/modprobe",


One risk of this implementation is that if PATH is /usr/bin:/sbin, the modprobe binary in the container will be used instead of the injected modprobe. I guess that's a calculated risk to take, since we can't inject modprobe in every single path in PATH. Probably something to follow up for a later release is checking PATH and inject modprobe to the first path in that list.

bcressey

It would be good to validate that Cilium specifically works as expected with this change, in case there are more paths we need to bind to.

vigh-m · 2024-05-30T18:22:08Z

New changes include:

Updates to docker to inject kmod by default
Fix based on comments
Typo fixes

Co-authored-by: Arnaldo Garcia Rincon <[email protected]>

vigh-m · 2024-05-30T20:25:10Z

Fixing typo in the commit message

vigh-m · 2024-05-30T21:02:03Z

Validated by running the cilium/cilium:v1.15.5 image from DockerHub:

Locally launched container:

$ docker run -it --rm cilium/cilium:v1.15.5 sh
# ldd /usr/sbin/modprobe
	linux-vdso.so.1 (0x00007fffc838c000)
	libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f6632d16000)
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f6632ceb000)
	libcrypto.so.3 => /lib/x86_64-linux-gnu/libcrypto.so.3 (0x00007f66328a7000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f663267e000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f6632e14000)
# ls -la /usr/sbin/modprobe
lrwxrwxrwx. 1 root root 9 Aug 17  2021 /usr/sbin/modprobe -> /bin/kmod
# cat /proc/self/mountinfo  | grep kmod
#

VS container launched on bottlerocket

$ kubectl exec -it my-pod -- bash
root@my-pod:/home/cilium#
root@my-pod:/home/cilium# ldd /usr/sbin/modprobe
	not a dynamic executable
root@my-pod:/home/cilium# ls -la /usr/sbin/modprobe
lrwxrwxrwx. 1 root root 9 Aug 17  2021 /usr/sbin/modprobe -> /bin/kmod
root@my-pod:/home/cilium# ldd /bin/kmod
	not a dynamic executable
root@my-pod:/home/cilium# cat /proc/self/mountinfo | grep kmod
1281 1275 252:0 /x86_64-bottlerocket-linux-gnu/sys-root/usr/bin/kmod /usr/bin/kmod ro,relatime master:1 - ext4 /dev/root ro,seclabel,stripe=1024

I'm able to load and unload modules

root@my-pod:/home/cilium# lsmod | grep tables
nf_tables             307200  0
nfnetlink              20480  2 nf_conntrack_netlink,nf_tables
root@my-pod:/home/cilium# modprobe -r nf_tables
root@my-pod:/home/cilium# lsmod | grep tables
root@my-pod:/home/cilium# modprobe nf_tables
root@my-pod:/home/cilium# lsmod | grep tables
nf_tables             307200  0
nfnetlink              20480  2 nf_conntrack_netlink,nf_tables

vigh-m requested review from bcressey, yeazelm and arnaldo2792 May 30, 2024 01:55

containerd: update base mounts to add modprobe

8ef185c

vigh-m force-pushed the mount-modprobe branch from f0c029f to a702957 Compare May 30, 2024 03:08

yeazelm approved these changes May 30, 2024

View reviewed changes

arnaldo2792 reviewed May 30, 2024

View reviewed changes

bcressey approved these changes May 30, 2024

View reviewed changes

vigh-m force-pushed the mount-modprobe branch 2 times, most recently from bd4debd to 54d8e27 Compare May 30, 2024 17:16

vigh-m requested review from bcressey and yeazelm May 30, 2024 17:16

host-ctr: update mounts to add modprobe

822947d

vigh-m force-pushed the mount-modprobe branch from 54d8e27 to 4e8487c Compare May 30, 2024 17:25

arnaldo2792 approved these changes May 30, 2024

View reviewed changes

docker-engine: inject kmod in all containers

e718060

Co-authored-by: Arnaldo Garcia Rincon <[email protected]>

vigh-m force-pushed the mount-modprobe branch from 4e8487c to e718060 Compare May 30, 2024 20:24

bcressey approved these changes May 30, 2024

View reviewed changes

yeazelm approved these changes May 30, 2024

View reviewed changes

vigh-m merged commit 70c60c6 into bottlerocket-os:develop May 30, 2024
34 checks passed

vigh-m deleted the mount-modprobe branch May 30, 2024 21:17

larvacea mentioned this pull request May 30, 2024

1.20.x: static kmod cherry-pick #4009

Merged

yeazelm mentioned this pull request Jun 3, 2024

v1.20.1 🐫 Tracking Issue #4025

Closed

5 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Mount static kmod as /sbin/modprobe #4007

Mount static kmod as /sbin/modprobe #4007

vigh-m commented May 30, 2024

bcressey commented May 30, 2024 •

edited

Loading

yeazelm commented May 30, 2024

vigh-m commented May 30, 2024

vigh-m commented May 30, 2024

arnaldo2792 commented May 30, 2024

arnaldo2792 May 30, 2024

bcressey left a comment

vigh-m commented May 30, 2024

vigh-m commented May 30, 2024

vigh-m commented May 30, 2024

Mount static kmod as /sbin/modprobe #4007

Mount static kmod as /sbin/modprobe #4007

Conversation

vigh-m commented May 30, 2024

bcressey commented May 30, 2024 • edited Loading

yeazelm commented May 30, 2024

vigh-m commented May 30, 2024

vigh-m commented May 30, 2024

arnaldo2792 commented May 30, 2024

arnaldo2792 May 30, 2024

Choose a reason for hiding this comment

bcressey left a comment

Choose a reason for hiding this comment

vigh-m commented May 30, 2024

vigh-m commented May 30, 2024

vigh-m commented May 30, 2024

bcressey commented May 30, 2024 •

edited

Loading