Skip to content
This repository has been archived by the owner on Jan 12, 2023. It is now read-only.

Unable to delete pod #122

Open
chilu49 opened this issue Sep 8, 2021 · 10 comments
Open

Unable to delete pod #122

chilu49 opened this issue Sep 8, 2021 · 10 comments

Comments

@chilu49
Copy link

chilu49 commented Sep 8, 2021

This pod is not getting deleted after its deployment is deleted. Even manual deletion is giving below error.
Error from server (InternalError): Internal error occurred: admission webhook "k-rail.cruise-automation.github.com" attempted to modify the object, which is not supported for this operation

I have added exemption but its still not working.

kubernetes version: v1.20
@Kaezon
Copy link

Kaezon commented Sep 23, 2021 •
edited
Loading

I just ran into the same issue trying to delete a Job pod

Kubernetes version: 1.19
k-rail version: v3.5.1

@chilu49
Copy link
Author

chilu49 commented Sep 23, 2021

I installed k-rail using helm.
So, i ended up doing helm uninstall k-rail after which I was able to delete the pod.
Not sure if this works for you or not.

@chilu49
Copy link
Author

chilu49 commented Sep 23, 2021

I just ran into the same issue trying to delete a Job pod

Kubernetes version: 1.19
k-rail version: v3.5.1

I installed k-rail using helm.
So, i ended up doing helm uninstall k-rail after which I was able to delete the pod.
Not sure if this works for you or not.

@Kaezon
Copy link

Kaezon commented Sep 23, 2021

I installed k-rail using helm.
So, i ended up doing helm uninstall k-rail after which I was able to delete the pod.
Not sure if this works for you or not.

Oh, yes. I can remove k-rail to delete the pod; however, deleting k-rail every time I run a job doesn't seem like an ideal way to administrate my deployments :P

@tobymilne-haven
Copy link

I had the same issue as soon as I switched to reportonly false, in the end i hacked the helm chart, and disabled the webhook for "DELETE", that allows pods to be deleted, but i suspect rules about eviction etc wont work.

@Kaezon
Copy link

Kaezon commented Nov 1, 2021 •
edited
Loading

@chilu49 @tobymilne-haven
I created a temporary work-around in a branch: Kaezon/k-rail@9599c670d942f10547e276d8bf0056d34995b0c5

All I did was limit the webhook to processing DELETEs to CRDs. This was because it's the only thing I'm aware of that has a plugin which looks at deletes.
In the long run, I would not keep this solution in place since we probably want k-rail to be examining all requests anyways.

I'm going to see if I can figure out what the actual cause of the problem is and fix it.

@Kaezon
Copy link

Kaezon commented Nov 2, 2021 •
edited
Loading

After adding a lot of debug prints, I found what's happening at least.
It looks like k-rail is trying to attach some extra metadata to the DELETE request. Specifically "seccomp.security.alpha.kubernetes.io/pod:runtime/default"

I'm still not sure why though.

{"kind":"AdmissionReview","apiVersion":"admission.k8s.io/v1","request":{"uid":"f48f2da7-6e29-4d50-bd41-5843bd91a045","kind":{"group":"","version":"v1","kind":"Pod"},"resource":{"group":"","version":"v1","resource":"pods"},"requestKind":{"group":"","version":"v1","kind":"Pod"},"requestResource":{"group":"","version":"v1","resource":"pods"},"name":"banana-app-c74b498db-cps64","namespace":"default","operation":"DELETE","userInfo":{"username":"system:serviceaccount:argocd:argocd-server","uid":"3c627c97-ddae-4f57-baa4-3937d7abcdf4","groups":["system:serviceaccounts","system:serviceaccounts:argocd","system:authenticated"]},"object":null,"oldObject":{"kind":"Pod","apiVersion":"v1","metadata":{"name":"banana-app-c74b498db-cps64","generateName":"banana-app-c74b498db-","namespace":"default","uid":"1a4921ba-456a-48ef-9e25-33a18177222a","resourceVersion":"76855","creationTimestamp":"2021-11-02T18:04:42Z","labels":{"app":"banana","pod-template-hash":"c74b498db"},"annotations":{"cni.projectcalico.org/podIP":"10.1.9.216/32","cni.projectcalico.org/podIPs":"10.1.9.216/32","sidecar.istio.io/inject":"true"},"ownerReferences":[{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"banana-app-c74b498db","uid":"c1aa8378-137e-4c9a-a948-256236e889e4","controller":true,"blockOwnerDeletion":true}],"managedFields":[{"manager":"kube-controller-manager","operation":"Update","apiVersion":"v1","time":"2021-11-02T18:04:42Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{".":{},"f:sidecar.istio.io/inject":{}},"f:generateName":{},"f:labels":{".":{},"f:app":{},"f:pod-template-hash":{}},"f:ownerReferences":{".":{},"k:{\"uid\":\"c1aa8378-137e-4c9a-a948-256236e889e4\"}":{".":{},"f:apiVersion":{},"f:blockOwnerDeletion":{},"f:controller":{},"f:kind":{},"f:name":{},"f:uid":{}}}},"f:spec":{"f:containers":{"k:{\"name\":\"banana-app\"}":{".":{},"f:args":{},"f:image":{},"f:imagePullPolicy":{},"f:name":{},"f:ports":{".":{},"k:{\"containerPort\":8080,\"protocol\":\"TCP\"}":{".":{},"f:containerPort":{},"f:protocol":{}}},"f:resources":{".":{},"f:limits":{".":{},"f:cpu":{},"f:memory":{}},"f:requests":{".":{},"f:cpu":{},"f:memory":{}}},"f:securityContext":{".":{},"f:runAsGroup":{},"f:runAsNonRoot":{},"f:runAsUser":{}},"f:terminationMessagePath":{},"f:terminationMessagePolicy":{}}},"f:dnsPolicy":{},"f:enableServiceLinks":{},"f:restartPolicy":{},"f:schedulerName":{},"f:securityContext":{".":{},"f:runAsGroup":{},"f:runAsNonRoot":{},"f:runAsUser":{}},"f:terminationGracePeriodSeconds":{}}}},{"manager":"calico","operation":"Update","apiVersion":"v1","time":"2021-11-02T18:04:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:metadata":{"f:annotations":{"f:cni.projectcalico.org/podIP":{},"f:cni.projectcalico.org/podIPs":{}}}}},{"manager":"kubelet","operation":"Update","apiVersion":"v1","time":"2021-11-02T18:04:43Z","fieldsType":"FieldsV1","fieldsV1":{"f:status":{"f:conditions":{"k:{\"type\":\"ContainersReady\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Initialized\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}},"k:{\"type\":\"Ready\"}":{".":{},"f:lastProbeTime":{},"f:lastTransitionTime":{},"f:status":{},"f:type":{}}},"f:containerStatuses":{},"f:hostIP":{},"f:phase":{},"f:podIP":{},"f:podIPs":{".":{},"k:{\"ip\":\"10.1.9.216\"}":{".":{},"f:ip":{}}},"f:startTime":{}}}}]},"spec":{"volumes":[{"name":"default-token-l4qr4","secret":{"secretName":"default-token-l4qr4","defaultMode":420}}],"containers":[{"name":"banana-app","image":"packages.bco.cudaops.com:443/docker-virtual/hashicorp/http-echo@sha256:ba27d460cd1f22a1a4331bdf74f4fccbc025552357e8a3249c40ae216275de96","args":["-listen=:8080","-text=banana"],"ports":[{"containerPort":8080,"protocol":"TCP"}],"resources":{"limits":{"cpu":"550m","memory":"2560Mi"},"requests":{"cpu":"500m","memory":"2Gi"}},"volumeMounts":[{"name":"default-token-l4qr4","readOnly":true,"mountPath":"/var/run/secrets/kubernetes.io/serviceaccount"}],"terminationMessagePath":"/dev/termination-log","terminationMessagePolicy":"File","imagePullPolicy":"IfNotPresent","securityContext":{"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true}}],"restartPolicy":"Always","terminationGracePeriodSeconds":30,"dnsPolicy":"ClusterFirst","serviceAccountName":"default","serviceAccount":"default","nodeName":"bcostabile-barracuda","securityContext":{"runAsUser":1000,"runAsGroup":1000,"runAsNonRoot":true},"schedulerName":"default-scheduler","tolerations":[{"key":"node.kubernetes.io/not-ready","operator":"Exists","effect":"NoExecute","tolerationSeconds":300},{"key":"node.kubernetes.io/unreachable","operator":"Exists","effect":"NoExecute","tolerationSeconds":300}],"priority":0,"enableServiceLinks":true,"preemptionPolicy":"PreemptLowerPriority"},"status":{"phase":"Running","conditions":[{"type":"Initialized","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-02T18:04:42Z"},{"type":"Ready","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-02T18:04:43Z"},{"type":"ContainersReady","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-02T18:04:43Z"},{"type":"PodScheduled","status":"True","lastProbeTime":null,"lastTransitionTime":"2021-11-02T18:04:42Z"}],"hostIP":"192.168.1.189","podIP":"10.1.9.216","podIPs":[{"ip":"10.1.9.216"}],"startTime":"2021-11-02T18:04:42Z","containerStatuses":[{"name":"banana-app","state":{"running":{"startedAt":"2021-11-02T18:04:43Z"}},"lastState":{},"ready":true,"restartCount":0,"image":"sha256:a6838e9a6ff6ab3624720a7bd36152dda540ce3987714398003e14780e61478a","imageID":"packages.bco.cudaops.com:443/docker-virtual/hashicorp/http-echo@sha256:ba27d460cd1f22a1a4331bdf74f4fccbc025552357e8a3249c40ae216275de96","containerID":"containerd://a756072973f9ba3a3d4d7b5222aaf53e1cef82ada12a5e4d82d0fa8575b7f183","started":true}],"qosClass":"Burstable"}},"dryRun":false,"options":{"kind":"DeleteOptions","apiVersion":"meta.k8s.io/v1","gracePeriodSeconds":30,"propagationPolicy":"Foreground"}}}

DEBUG: Printing list of patches
{add /metadata/annotations map[seccomp.security.alpha.kubernetes.io/pod:runtime/default]}

@Kaezon
Copy link

Kaezon commented Nov 2, 2021 •
edited
Loading

Ok, a little more debugging revealed it's the pod_default_seccomp_policy plugin.
I'll look at the code there next.

DEBUG: List of patches from pod_default_seccomp_policy policy
{add /metadata/annotations map[seccomp.security.alpha.kubernetes.io/pod:runtime/default]}

@Kaezon Kaezon mentioned this issue Nov 2, 2021
Closed
@funkypenguin
Copy link
Contributor

I've found this problem as well, after enabling the pod_default_seccomp_policy. The pods were already running, and so thereafter any attempts to delete them caused the above-mentioned issue.

@mark-adams
Copy link
Contributor

👋 The k-rail project has been deprecated and is no longer under active development. We recommend taking a look at OPA Gatekeeper to see if it might meet your needs going forward.

Thanks for your contribution(s) to the project!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mark-adams @funkypenguin @Kaezon @chilu49 @tobymilne-haven